Merge branch 'dhk-1' of http://192.168.0.200/knowledge-base-bj/deerflow-server into dhk-1
This commit is contained in:
commit
d9e3a87911
129 files changed
+17888
-1703
No files matched your search
@@ -250,9 +250,18 @@ FastAPI application on port 8001 with health check at `GET /health`. Set `GATEWA
|
||||
| **Scheduled Tasks** (`/api/scheduled-tasks`) | Per-user cron-like agent tasks. `GET/POST /` - list/create owned tasks; `PATCH/DELETE /{id}` - update/delete (owner); `POST /{id}/{pause,resume,trigger}`; `GET /{id}/runs`, `GET /runs/{rid}`, run-file preview/edit/download; `GET /runs/{rid}/messages` - agent messages for the run's underlying agent run (used to inspect a stuck/running run; authorized via `_load_run`); `GET /runs/{rid}/html` - generated HTML document for an `html_page` run; `DELETE /runs/{rid}` - owner-scoped, with an admin fallback so admins can delete any user's run; publish/subscribe under `/{id}/{publish,unpublish,subscribe}` and `GET /published`. **Admin (cross-user):** `GET /admin/all` - every task; `PATCH/DELETE /admin/{id}` - edit/delete any task; `POST /admin/{id}/trigger`; `GET /admin/{id}/runs`. Admin routes require `system_role == "admin"`. |
|
||||
| **HTML Page Favorites** (`/api/html-page-favorites`) | Bookmarked generated HTML pages, owner-scoped. `GET /` - list; `POST /` `{page_id, title?, description?, tags?}` - bookmark a `scheduled_html_pages` row (extracts a reusable style/layout summary via GLM5 at favorite time); a page can only be favorited once per user (`DuplicateFavoriteError` → HTTP 409). `GET/DELETE /{id}` - read/delete (the create-task reference picker deletes favorites here). A favorite can be selected as a style reference when creating a new `html_page` task. |
|
||||
| **Public Scheduled Tasks** (`/api/public/scheduled-tasks`) | **Unauthenticated** mirror for embedding. Adds `GET /runs/{rid}/html` and `GET /{task_id}/latest-html` for the `html_page` preview; the run result keeps `result.html` (only file paths are redacted). |
|
||||
| **Knowledge** (`/api/knowledge`) | Product knowledge base, **global/shared** (no user isolation; writes record `created_by`/`updated_by` for audit). `POST /ingest/thread/{thread_id}` - sediment one thread into a note (reads messages from the checkpointer, `threads:read` owner-checked); `POST /ingest/threads/batch` - batch-sediment the caller's recent threads (`skip_existing` via `.manifest.json` dedup). **Optional model selection**: both ingest endpoints accept `model_name` (overrides `knowledge.extract_model_name` for that capture; threaded through `KnowledgeService.capture_thread` → `extract_thread_llm_multi`). **Model-error surfacing**: when the chosen model produces nothing usable (timeout / API error / non-JSON) capture still **falls back to rule-based extraction and lands a draft** (`llm_fallback:true`); the single-thread (non-background) response and the batch response (`llm_fallback` count) carry this so the UI tells the user "已改用规则抽取". The "保存到知识库" chat button stays background/fire-and-forget (passes `model_name`, no per-save error toast); the batch-import dialog is synchronous and shows the fallback count. `POST /ingest/search` - sediment search/tool results as a reference note; `POST /notes` - manual note; `GET /notes` (keyword/tag/source_type/status + pagination); `GET/PUT/DELETE /notes/{id}` (DELETE is soft = archive, `?hard=true` removes the vault file); `POST /search` - `mode=keyword\|vector\|hybrid` (vector→keyword fallback when no embedding endpoint); `GET /notes/{id}/versions` - edit history; `GET /duplicates` + `POST /merge` - dedup; `POST /reindex` - rebuild vector index; `GET /graph` - product graph data (notes + entities); `POST /export/graph` + `GET /export/files/{graph.json\|graph.html\|graph.graphml\|cypher.txt}` - obsidian-wiki `wiki-export` artifacts (self-contained `graph.html` for iframe); `POST /resolve` `{targets:[…]}` + `GET /notes/{id}/wikilinks` - resolve `[[wikilink]]` targets to a note id / vault page / missing (powers clickable wikilinks); `GET /vault-page?path=` - read a raw vault meta/hub Markdown page (no DB note). Backed by the `deerflow.knowledge` module (see below). |
|
||||
|
||||
Proxied through nginx: `/api/langgraph/*` → LangGraph, all other `/api/*` → Gateway.
|
||||
|
||||
### Admin Leaderboard Snapshots (`/api/admin/users/leaderboard*`)
|
||||
|
||||
The admin 用户排行榜 is served from **precomputed per-day snapshots** in `admin_leaderboard_daily_stats` (keyed by `(stat_date, settings_hash)`), not by aggregating raw rows on every request. A background scheduler inside the Gateway (`_leaderboard_snapshot_scheduler_loop` in `app/gateway/app.py`, singleton file-locked) ticks every `ADMIN_LEADERBOARD_BACKGROUND_INTERVAL_SECONDS` (default 300s): it prewarms recent days, then builds any `queued`/`error`/stale-`running` day. `GET /leaderboard` merges the per-day snapshots for the requested range. **Today** is rebuilt whenever its snapshot is older than `ADMIN_LEADERBOARD_TODAY_TTL_SECONDS` (default 300s); a past day, once `ready`, is treated as fresh forever.
|
||||
|
||||
Two on-demand/automatic refresh paths layer on top:
|
||||
- **Manual "刷新当天"** — `POST /leaderboard/refresh-today` force-rebuilds *today's* snapshot **synchronously** (bounded by the same semaphore/timeout the scheduler uses), then returns the merged range. Lets an admin pull the latest real-time records into the stats without waiting for the TTL. Frontend: a "刷新当天" button on `AdminUserLeaderboardPage` → `useRefreshTodayLeaderboard` (`core/admin/hooks.ts`), which primes + invalidates the `["admin-activity","leaderboard"]` query.
|
||||
- **Daily finalize of the previous day** — once per day, on/after `ADMIN_LEADERBOARD_FINALIZE_HOUR` (local Beijing, default `1` = 凌晨 1 点), the scheduler tick force-recomputes *yesterday* a single time so runs/tool-calls that only landed near midnight are fully captured. The decision is the pure `deerflow.persistence.admin_stats.finalize.should_finalize_previous_day` (status/generated_at/now/hour) — the rebuild stamps `generated_at` to today, which self-debounces it for the rest of the day (no extra state file). Tests: `tests/test_leaderboard_finalize.py`.
|
||||
|
||||
### Scheduled Task Runtime
|
||||
|
||||
`ScheduledTaskService` (`deerflow/runtime/scheduler/service.py`) polls due tasks every `poll_interval_seconds` and executes each in its own throwaway thread, serialized by a shared `_agent_run_lock`.
|
||||
@@ -273,11 +282,32 @@ Proxied through nginx: `/api/langgraph/*` → LangGraph, all other `/api/*` →
|
||||
|
||||
**HTML skills (tools the render step uses).** Two offline skills under `skills/public/` shape the generated HTML and are toggled in the skills settings page (enabled by default): `html-page-builder` (a self-contained inline-CSS design system — colors/cards/tables/KPI/responsive) and `html-charts` (inline-SVG charts that render in the no-`allow-scripts` preview iframe; optional local ECharts documented in its `assets/README.md`). A skill opts into HTML injection via `metadata.kind: html_page` in its SKILL.md frontmatter; `_collect_html_skill_instructions()` reads the bodies of **enabled** such skills and `build_generation_prompt(skill_instructions=…)` injects them, so the (raw-model) render step honors the user's skill selection. The data-gathering step and the render/repair/style passes all use the task's selected `model_name` (no silent fallback to the default model — and there is **no hardcoded GLM-5**; an unset selection falls back to config `models[0]`). To stop auxiliary LLM calls from leaking onto the default model, **`TitleMiddleware` is skipped when `is_scheduled_run` is set** (`_build_middlewares` in `lead_agent/agent.py`) — scheduled/HTML runs use throwaway threads whose titles are never shown, and `config.title.model_name` is usually unset (→ would call `models[0]`, not the chosen model). The completeness gate (`validate_html`) also requires a non-empty `<title>` and forbids external stylesheets, on top of the doctype/html/head/body + non-empty-body + no-script checks.
|
||||
|
||||
### Knowledge Base (`packages/harness/deerflow/knowledge/`)
|
||||
|
||||
Self-contained product knowledge base. **All knowledge capability lives in this one module** (harness layer, `deerflow.*` only — no `app.*` imports). Wired as `app.state.knowledge_service` in `deps.py` via `make_knowledge_service(sf, config.knowledge)` (returns `None` on the memory backend or when `knowledge.enabled=false` → router 503). Surfaced by `app/gateway/routers/knowledge.py` (`/api/knowledge`). Config: `config.yaml` → `knowledge` (`deerflow/config/knowledge_config.py`).
|
||||
|
||||
**Two stores, kept in sync on every write:**
|
||||
- **DB mirror** — tables `knowledge_notes` / `knowledge_sources` / `knowledge_tags` / `knowledge_note_tags` (`knowledge/models.py`, registered in `persistence/models/__init__.py`, auto-created by `Base.metadata.create_all`; portable types `PortableLongText`/`BeijingDateTime`). Powers listing/filter/search/audit. Global/shared — no `user_id` column; `created_by`/`updated_by` are audit-only (resolved via `get_effective_user_id()`).
|
||||
- **Markdown vault** — obsidian-wiki compatible, default `{$DEER_FLOW_HOME}/data/knowledge/obsidian-vault`. `ObsidianWikiAdapter` (`obsidian_adapter.py`) writes wiki-capture frontmatter+body, and after **every** write syncs `index.md` / `log.md` (CAPTURE/INGEST/UPDATE/ARCHIVE/EXPORT) / `hot.md` / `.manifest.json` (source-hash dedup for `skip_existing`). `markdown_vault.py` handles frontmatter build/parse, slugging, wikilink extraction and **path-traversal-proof** resolution.
|
||||
|
||||
**Module layout:** `models.py` (ORM: notes/sources/tags/note_tags + embeddings/versions/entities/relations) · `repository.py` (DB only, no LLM) · `schemas.py` (NoteDraft/SourceDraft DTOs) · `skill_templates.py` · `markdown_vault.py` · `manifest.py` · `extractor.py` (rule-based fallback) · `llm_extractor.py` (**default**: distills declarative knowledge per wiki-capture, structured JSON incl. entities/relations; falls back to `extractor.py`) · `sources.py` · `exporter.py` (`graph.json`/`graph.graphml`/`cypher.txt` + self-contained vanilla-JS `graph.html`; entity nodes + relation edges) · `chunking.py` + `embeddings.py` (OpenAI-compatible embedding client, keyword fallback) + `search.py` (keyword/vector/hybrid) · `sensitive.py` (secret/PII redaction) · `dedup.py` (Jaccard + hash dedup) · `auto_ingest.py` (background queue/worker) · `obsidian_adapter.py` · `service.py` (`KnowledgeService` orchestrator — the only thing the router touches). Tests: `tests/test_knowledge.py` (20 cases). Frontend: `frontend-web/src/core/knowledge/*` + `pages/KnowledgeBasePage.tsx` / `KnowledgeGraphPage.tsx` + chat `knowledge-save-trigger.tsx` / `knowledge-rag-toggle.tsx`; spec at `frontend-web/docs/product-knowledge-base-dev.md`.
|
||||
|
||||
**Wikilinks & multi-note capture** (`knowledge.{related_notes_enabled,related_notes_limit,multi_note_enabled}`, all default-on; `project_name` defaults to `zncm`):
|
||||
- **Navigable `[[wikilinks]]`** — `service.resolve_wikilink(s)` maps a `[[target]]` to a knowledge note (by id → `vault_path` → title) or a vault meta page (`read_vault_page`, path-traversal-proof), else `missing`. `markdown_vault.normalize_wikilink_target` canonicalizes targets (drops `|alias`, slashes, `.md`). The frontend `MarkdownRenderer` renders non-numeric `[[...]]` as clickable links from the resolver map (numeric `[[123]]` stay citation badges); note links navigate, vault links open a `VaultPageDialog`, missing links are greyed.
|
||||
- **Real related notes only (no structural meta-links)** — `extractor.related_links()` now returns `[]`: notes no longer auto-link `index` / `_meta/taxonomy` / a project hub (they were noise in every `## Related`). On capture, `service._find_related_notes` cross-links **only by shared *distinctive* tags** (project/`knowledge-base` defaults excluded; free-text title/summary matching was removed because CJK tokenization linked unrelated notes via generic words like 助手/知识图谱) and `_inject_related_notes` writes real `[[Title]]` links into `## Related` (plus note→note `related` graph edges); no distinctive tags → no related notes. The renderers omit the `## Related` section entirely when there are no related notes and no sources. The vault's own `index.md`/`_meta/taxonomy.md` still exist (obsidian scaffold) and remain resolvable for manually-written links. The router enriches `GET /notes/{id}` with `created_by_name`/`updated_by_name` (auth-provider email lookup) so the UI shows a person, not a UUID. The frontend `MarkdownRenderer` renders wiki-capture provenance markers `^[inferred]`/`^[ambiguous]` as small "推测"/"存疑" badges.
|
||||
- **LLM-extraction outcomes** — capture distinguishes three LLM outcomes: (1) **success** → declarative note(s); (2) **deliberate rejection** — the model returns `should_ingest:false` with no notes (greeting / meta chatter) → `extract_thread_llm_multi` raises `LLMRejectedIngest` and `capture_thread` raises `EmptyThreadError` (skip — nothing persisted, so worthless chats never enter the KB); (3) **genuine failure** — API error / non-JSON → returns `[]`, capture falls back to the rule-based verbatim extractor and lands the note as a **`draft`** (not `approved`), logs a WARNING, and returns `llm_fallback:true` (surfaced by the router). `llm_extractor` logs an actionable WARNING/INFO for each non-success path so extraction issues are diagnosable.
|
||||
- **Multi-note split** — `llm_extractor.extract_thread_llm_multi` lets one conversation distill into 1–4 cross-linked notes (`notes:[…]` JSON array; legacy single-object output still accepted). `capture_thread` persists each, sibling-cross-links them by title, and returns `{note, created, notes:[…]}`. Gated by `multi_note_enabled` (false → first note only). Tests: `tests/test_knowledge_wikilinks.py`.
|
||||
|
||||
**Phases 2–4 (all config-gated, defaults safe):**
|
||||
- **Phase 2 — vector RAG**: `knowledge.embedding.{enabled,base_url,api_key,model,...}` enables an OpenAI-compatible embedding endpoint; notes are chunked + embedded on write (`knowledge_embeddings`), `POST /search` supports `mode=keyword|vector|hybrid` (vector degrades to keyword when unconfigured). `KnowledgeRagMiddleware` (lead-agent chain) injects a transient `<knowledge-context>` recall before the model call, gated **per-conversation** by the `knowledge_rag_enabled` runtime flag (frontend chat toggle) falling back to global `knowledge.rag_enabled`. `POST /reindex` rebuilds the index.
|
||||
- **Phase 3 — governance**: `sensitive.py` redacts secrets/PII before persistence (`sensitive_redaction_enabled`); every update snapshots `knowledge_note_versions` (`GET /notes/{id}/versions`); `GET /duplicates` + `POST /merge` for dedup; background `AutoIngestQueue` (wired in `deps.py`, started/stopped in the gateway lifespan) auto-sediments finished conversations **off the chat path** when `auto_ingest_enabled` — enqueued from `KnowledgeRagMiddleware.after_agent`, value-judged by min-chars + `skip_existing`, landing as `draft`.
|
||||
- **Phase 4 — graph**: LLM extraction also yields entities/relations → `knowledge_entities`/`knowledge_relations`; `exporter.build_graph` adds entity nodes + note→entity/entity→entity edges (reflected in `graph.html`/`graph.json`); `GET /graph` returns product graph data.
|
||||
|
||||
### Custom Agent Persistence
|
||||
|
||||
Custom agent metadata lives in the `agents` ORM table (`deerflow.persistence.agents`). `agents.id` is the stable runtime id and the filesystem directory name under `.deer-flow/agents/{id}/`; `agents.name` is display-only, can contain Chinese characters, and is not unique. `user_id IS NULL` marks built-in agents, user-owned agents are visible only to their owner unless `published = true`, and update/delete operations require ownership. The runtime accepts `agent_id` in context/configurable, and non-default `assistant_id` values are treated as agent ids after visibility checks.
|
||||
|
||||
**Listing cache + pagination** — to keep `GET /api/agents` fast, the gallery's per-card extras (skills + tool_groups) are denormalized into two cache tables instead of reading every agent's `config.yaml` on each request: `agent_skills` (relational, one row per (agent, skill) with `position`) and `agent_extras` (one row per agent: `tool_groups` JSON + `model` + `skills_synced_at`). `config.yaml` remains authoritative for the agent runtime; these tables are a read cache, write-through on create/update and backfilled from disk by `_sync_legacy_agents` (mtime-vs-`skills_synced_at` drift check). The list endpoint enriches via `AgentStore.get_extras_for([ids])` (two batched `IN` queries). Both tables are auto-created by `Base.metadata.create_all` (no migration needed). `GET /api/agents` supports **opt-in server-side pagination**: `scope` (`mine` = owned + favorited / `square` = built-in/published public), `page`, `page_size`, returning `{agents, total, page, page_size}`; omitting `scope`/`page` preserves the legacy "return everything" behavior. Name `search` matches the raw OR desensitized name (the frontend's static term→abbrev map is mirrored server-side in `deerflow/config/agent_name_desensitize.py`); `tag_ids` is an SQL OR-filter. `POST /api/agents/private-square/list` takes the same pagination params. Repository entry point: `AgentStore.list_paginated(...)`.
|
||||
**Listing cache + pagination** — to keep `GET /api/agents` fast, the gallery's per-card extras (skills + tool_groups) are denormalized into two cache tables instead of reading every agent's `config.yaml` on each request: `agent_skills` (relational, one row per (agent, skill) with `position`) and `agent_extras` (one row per agent: `tool_groups` JSON + `model` + `skills_synced_at`). `config.yaml` remains authoritative for the agent runtime; these tables are a read cache, write-through on create/update and backfilled from disk by `_sync_legacy_agents` (mtime-vs-`skills_synced_at` drift check). The list endpoint enriches via `AgentStore.get_extras_for([ids])` (two batched `IN` queries). Both tables are auto-created by `Base.metadata.create_all` (no migration needed). `GET /api/agents` supports **opt-in server-side pagination**: `scope` (`mine` = owned + favorited / `square` = built-in/published public), `page`, `page_size`, returning `{agents, total, page, page_size}`; omitting `scope`/`page` preserves the legacy "return everything" behavior. Name `search` matches the raw OR desensitized name (the frontend's static term→abbrev map is mirrored server-side in `deerflow/config/agent_name_desensitize.py`); `tag_ids` is an SQL OR-filter; `owner` keeps only agents published by that user id (the card's 发布者 — built-in agents never match), powering the gallery's click-发布者-to-filter. `POST /api/agents/private-square/list` takes the same pagination params (incl. `owner`). Repository entry point: `AgentStore.list_paginated(...)`. Tests: `tests/test_agent_listing_cache.py::test_list_paginated_owner_filter_keeps_one_publisher`.
|
||||
|
||||
**Homepage selector (agent picker on the new-chat page)** — single admin-curated list backed by `agents.featured_order` (nullable BIGINT). NULL = not featured. Only `published = true` agents can be featured (router enforces on writes); any published agent is eligible regardless of ownership. Cleared with `order: null`. The list rendered to end users is `WHERE featured_order IS NOT NULL` + visibility-filter, sorted by `featured_order ASC`. Admins reorder by sending the desired ids to `PUT /api/agents/featured/order` (assigns indices 0..N-1). BIGINT is used rather than INT so the management UI can seed initial values from `Date.now()` (overflows INT on MySQL).
|
||||
|
||||
@@ -289,6 +319,16 @@ Two `TypeDecorator`s keep ORM columns correct across the `sqlite` / `postgres` /
|
||||
- **`PortableLongText`** — a large text column: maps to MySQL `LONGTEXT` (4 GB) and plain `TEXT` on SQLite/Postgres (already unbounded). Use for big JSON/markdown blobs that would overflow MySQL's 64 KB `TEXT` cap — e.g. `ai_writing_sessions.transcript`.
|
||||
- **`BeijingDateTime`** — stores timestamps as Beijing (UTC+8) wall-clock time. The app records time with `datetime.now(UTC)`, but MySQL `DATETIME` / SQLite have no timezone and persist whatever wall clock the driver serializes — so a raw `DateTime` column would store UTC and read 8 hours behind Beijing. `BeijingDateTime` converts UTC→Beijing on bind and re-attaches `+08:00` on result, so the database holds Beijing time while Python still gets timezone-aware datetimes (instant comparisons stay correct, including in WHERE clauses). PostgreSQL `timestamptz` stores the instant unambiguously, so values pass through unchanged there. Uses a fixed UTC+8 offset (China has no DST), so no `tzdata` dependency.
|
||||
|
||||
### Schema Migrations / Manual Column-Add SQL (`scripts/sql/`)
|
||||
|
||||
Alembic auto-migration is **disabled** — `_run_alembic_upgrade(url)` is commented out in `persistence/engine.py`. On startup only `Base.metadata.create_all` runs, which creates **new tables** but never patches **existing** ones (no new columns, no new indexes). So a column added to an ORM model appears automatically on a brand-new deployment, but is **missing on any existing MySQL database** — and code reading/writing that column will error there until the column is added by hand.
|
||||
|
||||
**Convention (required):** whenever code adds a field to an existing table (a new `mapped_column` on an ORM model), you must, in the same change:
|
||||
1. Keep the matching alembic version file under `persistence/migrations/versions/` (guarded with `_has_table`/`_has_column`), and
|
||||
2. Append the idempotent `ADD COLUMN` statement to **`scripts/sql/column_additions_mysql.sql`**, with a definition matching the ORM column exactly (type, length, nullability).
|
||||
|
||||
Operators upgrading an existing DB run the whole file (`USE <db>;` then execute) — it is idempotent (existing columns are skipped via `information_schema.columns`), so it is safe to re-run. Index-only changes go in the sibling `scripts/sql/leaderboard_indexes_mysql.sql` (same idempotent pattern).
|
||||
|
||||
### Sandbox System (`packages/harness/deerflow/sandbox/`)
|
||||
|
||||
**Interface**: Abstract `Sandbox` with `execute_command`, `read_file`, `write_file`, `list_dir`
|
||||
@@ -358,6 +398,7 @@ Two `TypeDecorator`s keep ORM columns correct across the `sqlite` / `postgres` /
|
||||
- **Loading**: `load_skills()` recursively scans `skills/{public,custom}` for `SKILL.md`, parses metadata, and reads enabled state from extensions_config.json
|
||||
- **Injection**: Enabled skills listed in agent system prompt with container paths
|
||||
- **Installation**: `POST /api/skills/install` extracts .skill ZIP archive to custom/ directory
|
||||
- **Chinese name (`name_zh`)**: a Chinese display name stored in the **`skills` DB table** (`SkillRow.name_zh`, `String(128)`, migration `20260609_01`) — never written to the on-disk SKILL.md, so it works for built-in skills too. Threaded through `SkillStore` create/ensure_legacy/update/update_admin exactly like `detail`; surfaced on `SkillResponse.name_zh` from the ownership record in `_skill_to_response`. Authorization mirrors `detail`: custom skill → owner or admin; built-in/public → admin only (row materialized on demand via `ensure_legacy`). Endpoints (in `app/gateway/routers/skills.py`): `PUT /api/skills/{name}/name-zh` (set/clear), `POST /api/skills/{name}/name-zh/generate` (LLM generates one), `POST /api/skills/name-zh/generate-missing` (一键补充: **admin-only** bulk-generate for every skill that lacks one, bounded-concurrency `asyncio.gather`). **Optional model selection**: both generate endpoints accept an optional JSON body `{model_name}` — an explicit name wins, else the curator model setting, else the main model (`_resolve_skill_name_zh_model_name`). **Model-error surfacing**: `_generate_skill_name_zh` raises `SkillNameZhModelError` when the model itself fails (build/timeout/API error) — distinct from the model returning an unusable name (→ `None`). The single endpoint maps that to `502 大模型报错,无法补全技能中文名:…`; the bulk endpoint fails fast on a model-build error (502) and otherwise carries a representative `model_error` string in `SkillNameZhBulkResponse` so the UI can say "N 个因大模型报错失败". The frontend (`skill-settings-page.tsx` list + `skill-detail-dialog.tsx` editor, plus the agent skill pickers/badges in `NewAgentPage.tsx` + `agent-card.tsx`) shows `name_zh || name`, offers per-skill "AI 生成" plus an **admin-only** header "一键补充中文名" button, and a compact model dropdown (`components/workspace/model-name-select.tsx`, default = configured model) beside each action.
|
||||
|
||||
#### Skill Curator (`packages/harness/deerflow/skills/curator.py`)
|
||||
|
||||
|
||||
@@ -6,6 +6,8 @@ import sys
|
||||
import threading
|
||||
from collections.abc import AsyncGenerator
|
||||
from contextlib import asynccontextmanager
|
||||
from datetime import datetime
|
||||
from time import perf_counter
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from fastapi import FastAPI
|
||||
@@ -30,12 +32,14 @@ from app.gateway.routers import (
|
||||
artifacts,
|
||||
assistants_compat,
|
||||
auth,
|
||||
browser_context,
|
||||
business_mapping,
|
||||
channels,
|
||||
checkpoint_migration,
|
||||
feedback,
|
||||
html_page_favorites,
|
||||
intent,
|
||||
knowledge,
|
||||
llm_metrics,
|
||||
mcp,
|
||||
memory,
|
||||
@@ -45,6 +49,7 @@ from app.gateway.routers import (
|
||||
parallel_agent_tasks,
|
||||
parallel_agents,
|
||||
public_embed,
|
||||
public_leaderboard,
|
||||
public_scheduled_tasks,
|
||||
recommend,
|
||||
recommended_questions,
|
||||
@@ -469,6 +474,91 @@ _LEADERBOARD_SNAPSHOT_STARTUP_DELAY_SECONDS = _env_float(
|
||||
"ADMIN_LEADERBOARD_BACKGROUND_STARTUP_DELAY_SECONDS",
|
||||
60.0,
|
||||
)
|
||||
# Hour (local Beijing time, 0-23) after which the scheduler force-recomputes the
|
||||
# *previous* day's snapshot once, so records that landed near midnight are fully
|
||||
# captured. Default 1 → "每天凌晨 1 点重新统计前一天".
|
||||
_LEADERBOARD_FINALIZE_HOUR = min(
|
||||
23,
|
||||
_env_int("ADMIN_LEADERBOARD_FINALIZE_HOUR", 1, minimum=0),
|
||||
)
|
||||
|
||||
|
||||
def _leaderboard_scheduler_sql_for_log(stmt) -> str:
|
||||
try:
|
||||
return str(stmt.compile(compile_kwargs={"literal_binds": True}))
|
||||
except Exception:
|
||||
return str(stmt)
|
||||
|
||||
|
||||
async def _execute_leaderboard_scheduler_query(session, step: str, stmt, **context):
|
||||
sql = _leaderboard_scheduler_sql_for_log(stmt)
|
||||
logger.info(
|
||||
"Leaderboard snapshot DB query start: step=%s stat_date=%s job=%s settings_hash=%s started_at=%s sql=%s",
|
||||
step,
|
||||
context.get("stat_date") or context.get("stat_dates") or "",
|
||||
context.get("job_key") or "",
|
||||
context.get("settings_hash") or "",
|
||||
datetime.now().isoformat(),
|
||||
sql,
|
||||
)
|
||||
start = perf_counter()
|
||||
try:
|
||||
result = await session.execute(stmt)
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"Leaderboard snapshot DB query failed: step=%s stat_date=%s job=%s settings_hash=%s elapsed_ms=%.2f",
|
||||
step,
|
||||
context.get("stat_date") or context.get("stat_dates") or "",
|
||||
context.get("job_key") or "",
|
||||
context.get("settings_hash") or "",
|
||||
(perf_counter() - start) * 1000,
|
||||
)
|
||||
raise
|
||||
logger.info(
|
||||
"Leaderboard snapshot DB query done: step=%s stat_date=%s job=%s settings_hash=%s finished_at=%s elapsed_ms=%.2f",
|
||||
step,
|
||||
context.get("stat_date") or context.get("stat_dates") or "",
|
||||
context.get("job_key") or "",
|
||||
context.get("settings_hash") or "",
|
||||
datetime.now().isoformat(),
|
||||
(perf_counter() - start) * 1000,
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
async def _time_leaderboard_scheduler_db_operation(step: str, operation, *, sql: str = "", **context):
|
||||
logger.info(
|
||||
"Leaderboard snapshot DB query start: step=%s stat_date=%s job=%s settings_hash=%s started_at=%s sql=%s",
|
||||
step,
|
||||
context.get("stat_date") or context.get("stat_dates") or "",
|
||||
context.get("job_key") or "",
|
||||
context.get("settings_hash") or "",
|
||||
datetime.now().isoformat(),
|
||||
sql,
|
||||
)
|
||||
start = perf_counter()
|
||||
try:
|
||||
result = await operation()
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"Leaderboard snapshot DB query failed: step=%s stat_date=%s job=%s settings_hash=%s elapsed_ms=%.2f",
|
||||
step,
|
||||
context.get("stat_date") or context.get("stat_dates") or "",
|
||||
context.get("job_key") or "",
|
||||
context.get("settings_hash") or "",
|
||||
(perf_counter() - start) * 1000,
|
||||
)
|
||||
raise
|
||||
logger.info(
|
||||
"Leaderboard snapshot DB query done: step=%s stat_date=%s job=%s settings_hash=%s finished_at=%s elapsed_ms=%.2f",
|
||||
step,
|
||||
context.get("stat_date") or context.get("stat_dates") or "",
|
||||
context.get("job_key") or "",
|
||||
context.get("settings_hash") or "",
|
||||
datetime.now().isoformat(),
|
||||
(perf_counter() - start) * 1000,
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def _recent_leaderboard_stat_dates(days: int) -> list[str]:
|
||||
@@ -496,21 +586,27 @@ async def _leaderboard_snapshot_due_dates(
|
||||
|
||||
cutoff = datetime.now(UTC) - timedelta(seconds=running_ttl_seconds)
|
||||
async with session_factory() as session:
|
||||
rows = (
|
||||
await session.execute(
|
||||
select(AdminLeaderboardDailyStatRow.stat_date)
|
||||
.where(
|
||||
AdminLeaderboardDailyStatRow.settings_hash == settings_hash,
|
||||
or_(
|
||||
AdminLeaderboardDailyStatRow.status.in_(["queued", "error"]),
|
||||
and_(
|
||||
AdminLeaderboardDailyStatRow.status == "running",
|
||||
AdminLeaderboardDailyStatRow.updated_at < cutoff,
|
||||
),
|
||||
stmt = (
|
||||
select(AdminLeaderboardDailyStatRow.stat_date)
|
||||
.where(
|
||||
AdminLeaderboardDailyStatRow.settings_hash == settings_hash,
|
||||
or_(
|
||||
AdminLeaderboardDailyStatRow.status.in_(["queued", "error"]),
|
||||
and_(
|
||||
AdminLeaderboardDailyStatRow.status == "running",
|
||||
AdminLeaderboardDailyStatRow.updated_at < cutoff,
|
||||
),
|
||||
)
|
||||
.order_by(AdminLeaderboardDailyStatRow.stat_date.desc())
|
||||
.limit(limit)
|
||||
),
|
||||
)
|
||||
.order_by(AdminLeaderboardDailyStatRow.stat_date.desc())
|
||||
.limit(limit)
|
||||
)
|
||||
rows = (
|
||||
await _execute_leaderboard_scheduler_query(
|
||||
session,
|
||||
"scheduler.due_dates",
|
||||
stmt,
|
||||
settings_hash=settings_hash,
|
||||
)
|
||||
).all()
|
||||
return [str(row[0]) for row in rows]
|
||||
@@ -530,9 +626,15 @@ async def _claim_leaderboard_snapshot(
|
||||
from deerflow.persistence.types import BEIJING_TZ
|
||||
|
||||
async with session_factory() as session:
|
||||
row = await session.get(
|
||||
AdminLeaderboardDailyStatRow,
|
||||
{"stat_date": stat_date, "settings_hash": settings_hash},
|
||||
row = await _time_leaderboard_scheduler_db_operation(
|
||||
"scheduler.claim.get",
|
||||
lambda: session.get(
|
||||
AdminLeaderboardDailyStatRow,
|
||||
{"stat_date": stat_date, "settings_hash": settings_hash},
|
||||
),
|
||||
sql="SELECT admin_leaderboard_daily_stats by primary key",
|
||||
stat_date=stat_date,
|
||||
settings_hash=settings_hash,
|
||||
)
|
||||
if row is None or _snapshot_is_fresh(row, stat_date):
|
||||
return False
|
||||
@@ -548,10 +650,68 @@ async def _claim_leaderboard_snapshot(
|
||||
row.message = "排行榜后台统计任务已认领。"
|
||||
row.error = None
|
||||
row.updated_at = datetime.now(UTC)
|
||||
await session.commit()
|
||||
await _time_leaderboard_scheduler_db_operation(
|
||||
"scheduler.claim.commit",
|
||||
lambda: session.commit(),
|
||||
sql="COMMIT",
|
||||
stat_date=stat_date,
|
||||
settings_hash=settings_hash,
|
||||
)
|
||||
return True
|
||||
|
||||
|
||||
async def _maybe_finalize_previous_leaderboard_day(session_factory, settings_hash: str) -> bool:
|
||||
"""Once per day after ``_LEADERBOARD_FINALIZE_HOUR``, force-recompute yesterday.
|
||||
|
||||
A ``ready`` past-day snapshot is otherwise treated as fresh forever and never
|
||||
rebuilt. But the previous day's snapshot is usually generated before midnight,
|
||||
so it can miss runs/tool-calls that only land in the DB around the day
|
||||
boundary. This re-queues yesterday (``force=True``) a single time; the rebuild
|
||||
stamps ``generated_at`` to today, which stops it from re-firing for the rest of
|
||||
the day.
|
||||
"""
|
||||
from datetime import timedelta
|
||||
|
||||
from app.gateway.routers.admin_users import (
|
||||
_as_beijing,
|
||||
_queue_daily_leaderboard_snapshots,
|
||||
)
|
||||
from deerflow.persistence.admin_stats.finalize import should_finalize_previous_day
|
||||
from deerflow.persistence.admin_stats.model import AdminLeaderboardDailyStatRow
|
||||
from deerflow.persistence.types import BEIJING_TZ
|
||||
|
||||
now = datetime.now(BEIJING_TZ)
|
||||
yesterday = (now.date() - timedelta(days=1)).isoformat()
|
||||
async with session_factory() as session:
|
||||
row = await session.get(
|
||||
AdminLeaderboardDailyStatRow,
|
||||
{"stat_date": yesterday, "settings_hash": settings_hash},
|
||||
)
|
||||
generated_at = _as_beijing(row.generated_at) if (row is not None and row.generated_at is not None) else None
|
||||
if not should_finalize_previous_day(
|
||||
status=(row.status if row is not None else None),
|
||||
generated_at=generated_at,
|
||||
now=now,
|
||||
finalize_hour=_LEADERBOARD_FINALIZE_HOUR,
|
||||
):
|
||||
return False
|
||||
queued = await _queue_daily_leaderboard_snapshots(
|
||||
session_factory,
|
||||
[yesterday],
|
||||
settings_hash,
|
||||
force=True,
|
||||
message="每日凌晨重新统计前一天完整数据,确保全部记录纳入统计。",
|
||||
)
|
||||
if queued:
|
||||
logger.info(
|
||||
"Leaderboard snapshot finalize: queued previous day %s for full recompute (hash=%s)",
|
||||
yesterday,
|
||||
settings_hash[:10],
|
||||
)
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
async def _run_leaderboard_snapshot_tick() -> tuple[int, int]:
|
||||
from app.gateway.routers.admin_users import (
|
||||
_daily_job_key,
|
||||
@@ -576,6 +736,12 @@ async def _run_leaderboard_snapshot_tick() -> tuple[int, int]:
|
||||
message="排行榜后台统计任务定期预热/补算队列。",
|
||||
)
|
||||
|
||||
# 每天凌晨(默认 1 点)后强制重算"前一天"的完整快照一次,覆盖跨午夜才落库的记录。
|
||||
try:
|
||||
await _maybe_finalize_previous_leaderboard_day(session_factory, settings_hash)
|
||||
except Exception:
|
||||
logger.exception("Leaderboard snapshot finalize check failed (non-fatal)")
|
||||
|
||||
due_dates = await _leaderboard_snapshot_due_dates(
|
||||
session_factory,
|
||||
settings_hash=settings_hash,
|
||||
@@ -1115,6 +1281,9 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
|
||||
# Uploads API is mounted at /api/threads/{thread_id}/uploads
|
||||
app.include_router(uploads.router)
|
||||
|
||||
# Optional Chrome-extension page capture API is mounted at /api/browser-context
|
||||
app.include_router(browser_context.router)
|
||||
|
||||
# Thread cleanup API is mounted at /api/threads/{thread_id}
|
||||
app.include_router(threads.router)
|
||||
|
||||
@@ -1162,10 +1331,16 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
|
||||
# surface results without a DeerFlow login.
|
||||
app.include_router(public_scheduled_tasks.router)
|
||||
|
||||
# Public read-only user leaderboard publish page (/api/public/leaderboard)
|
||||
app.include_router(public_leaderboard.router)
|
||||
|
||||
# Public embed Q&A status polling (/api/public/embed/*)
|
||||
app.include_router(public_embed.router)
|
||||
app.include_router(html_page_favorites.router)
|
||||
|
||||
# Product knowledge base API is mounted at /api/knowledge
|
||||
app.include_router(knowledge.router)
|
||||
|
||||
# System settings API is mounted at /api/system-settings
|
||||
app.include_router(system_settings.router)
|
||||
|
||||
@@ -1174,6 +1349,7 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
|
||||
|
||||
# Admin window into other users' threads / messages / memory
|
||||
app.include_router(admin_users.router)
|
||||
app.include_router(admin_users.self_router)
|
||||
|
||||
# Admin tool / skill call metrics audit + export
|
||||
app.include_router(tool_metrics.router)
|
||||
|
||||
@@ -20,6 +20,7 @@ from app.gateway.auth.disabled_mode import get_assumed_user_for_disabled_auth, i
|
||||
from app.gateway.auth.errors import AuthErrorCode, AuthErrorResponse
|
||||
from app.gateway.authz import _ALL_PERMISSIONS, AuthContext
|
||||
from app.gateway.internal_auth import INTERNAL_AUTH_HEADER_NAME, get_internal_user, is_valid_internal_auth_token
|
||||
from app.gateway.proxy_path import app_relative_path
|
||||
from deerflow.runtime.user_context import reset_current_user, set_current_user
|
||||
|
||||
# Paths that never require authentication.
|
||||
@@ -84,7 +85,9 @@ class AuthMiddleware(BaseHTTPMiddleware):
|
||||
super().__init__(app)
|
||||
|
||||
async def dispatch(self, request: Request, call_next: Callable) -> Response:
|
||||
if _is_public(request.url.path):
|
||||
# Match the application-relative path so a reverse-proxy prefix
|
||||
# (e.g. /xxx/api/public/...) does not defeat the public whitelist.
|
||||
if _is_public(app_relative_path(request)):
|
||||
return await call_next(request)
|
||||
|
||||
if is_auth_disabled():
|
||||
|
||||
@@ -8,6 +8,7 @@ import secrets
|
||||
from collections.abc import Callable
|
||||
|
||||
from app.gateway.auth.disabled_mode import is_auth_disabled
|
||||
from app.gateway.proxy_path import app_relative_path
|
||||
from fastapi import Request, Response
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.responses import JSONResponse
|
||||
@@ -37,7 +38,9 @@ def should_check_csrf(request: Request) -> bool:
|
||||
if request.method not in ("POST", "PUT", "DELETE", "PATCH"):
|
||||
return False
|
||||
|
||||
path = request.url.path
|
||||
# Match the application-relative path so a reverse-proxy prefix
|
||||
# (e.g. /xxx/api/v1/auth/me) does not defeat these whitelists.
|
||||
path = app_relative_path(request)
|
||||
stripped = path.rstrip("/")
|
||||
# Exempt /api/v1/auth/me endpoint
|
||||
if stripped == "/api/v1/auth/me":
|
||||
@@ -56,6 +59,10 @@ _AUTH_EXEMPT_PATHS: frozenset[str] = frozenset(
|
||||
{
|
||||
"/api/v1/auth/login/local",
|
||||
"/api/v1/auth/login/username",
|
||||
# Token-exchange login is sessionless on first call (no CSRF cookie yet),
|
||||
# exactly like the other login endpoints — mirror auth_middleware's
|
||||
# _PUBLIC_EXACT_PATHS, which already whitelists it.
|
||||
"/api/v1/auth/login/token",
|
||||
"/api/v1/auth/logout",
|
||||
"/api/v1/auth/register",
|
||||
"/api/v1/auth/initialize",
|
||||
@@ -69,8 +76,10 @@ def is_auth_endpoint(request: Request) -> bool:
|
||||
"""Check if the request is to an auth endpoint.
|
||||
|
||||
Auth endpoints don't need CSRF validation on first call (no token).
|
||||
Matches the application-relative path so a reverse-proxy prefix does not
|
||||
defeat the whitelist.
|
||||
"""
|
||||
return request.url.path.rstrip("/") in _AUTH_EXEMPT_PATHS
|
||||
return app_relative_path(request).rstrip("/") in _AUTH_EXEMPT_PATHS
|
||||
|
||||
|
||||
def _has_bearer_auth(request: Request) -> bool:
|
||||
@@ -82,6 +91,22 @@ def _has_bearer_auth(request: Request) -> bool:
|
||||
return scheme.lower() == "bearer" and bool(token.strip())
|
||||
|
||||
|
||||
def _has_cookie_session(request: Request) -> bool:
|
||||
"""Return True when the request carries a cookie-based session.
|
||||
|
||||
Double-Submit-Cookie CSRF only protects requests the browser authenticates
|
||||
**automatically via cookies** — that is the exact surface a cross-site
|
||||
forgery can ride. A request with no ``access_token`` session cookie has no
|
||||
session for an attacker to exploit, so enforcing CSRF on it is both
|
||||
pointless and harmful: a freshly-loaded browser (e.g. one that just cleared
|
||||
its cache/cookies) hitting the login endpoint would otherwise be rejected
|
||||
with "CSRF token missing" before it can even sign in. Such an
|
||||
unauthenticated request, if forged, is rejected by the auth layer (401)
|
||||
anyway.
|
||||
"""
|
||||
return bool(request.cookies.get("access_token"))
|
||||
|
||||
|
||||
class CSRFMiddleware(BaseHTTPMiddleware):
|
||||
"""Middleware that implements CSRF protection using Double Submit Cookie pattern."""
|
||||
|
||||
@@ -95,8 +120,11 @@ class CSRFMiddleware(BaseHTTPMiddleware):
|
||||
return await call_next(request)
|
||||
|
||||
# Double-submit-cookie CSRF applies to cookie-based browser sessions.
|
||||
# API clients using explicit Bearer tokens are not CSRF-prone.
|
||||
if should_check_csrf(request) and not _is_auth and not _has_bearer_auth(request):
|
||||
# API clients using explicit Bearer tokens are not CSRF-prone, and a
|
||||
# request with no session cookie at all (e.g. a fresh browser signing
|
||||
# in) has no session to forge — only enforce when an ``access_token``
|
||||
# cookie is actually present.
|
||||
if should_check_csrf(request) and not _is_auth and not _has_bearer_auth(request) and _has_cookie_session(request):
|
||||
cookie_token = request.cookies.get(CSRF_COOKIE_NAME)
|
||||
header_token = request.headers.get(CSRF_HEADER_NAME)
|
||||
|
||||
|
||||
@@ -143,6 +143,8 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
|
||||
from deerflow.persistence.ai_writing_sessions import (
|
||||
make_ai_writing_session_store,
|
||||
)
|
||||
from deerflow.persistence.ai_writing_sessions import (
|
||||
register_default_store as _register_ai_writing_default_store,
|
||||
)
|
||||
|
||||
@@ -159,6 +161,25 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
|
||||
app.state.memory_config_store = make_user_memory_config_store(sf)
|
||||
|
||||
from deerflow.knowledge import make_knowledge_service
|
||||
|
||||
app.state.knowledge_service = make_knowledge_service(sf, config.knowledge)
|
||||
|
||||
# Background auto-sedimentation queue (phase 3) — drains finished
|
||||
# conversations off the chat path. Worker only does work when
|
||||
# ``knowledge.auto_ingest_enabled`` is set; the queue is always wired so
|
||||
# the middleware can enqueue without a hard dependency.
|
||||
from deerflow.knowledge.auto_ingest import AutoIngestQueue, set_auto_ingest_queue
|
||||
|
||||
if app.state.knowledge_service is not None:
|
||||
_auto_ingest_queue = AutoIngestQueue(
|
||||
lambda: getattr(app.state, "knowledge_service", None),
|
||||
min_chars=config.knowledge.auto_ingest_min_chars,
|
||||
)
|
||||
_auto_ingest_queue.start()
|
||||
set_auto_ingest_queue(_auto_ingest_queue)
|
||||
app.state.knowledge_auto_ingest_queue = _auto_ingest_queue
|
||||
|
||||
from deerflow.persistence.user_prompts import make_user_prompt_store
|
||||
|
||||
app.state.user_prompt_store = make_user_prompt_store(sf)
|
||||
@@ -189,6 +210,8 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
|
||||
from deerflow.persistence.roundtable_jobs import (
|
||||
make_roundtable_job_store,
|
||||
)
|
||||
from deerflow.persistence.roundtable_jobs import (
|
||||
register_default_store as _register_roundtable_job_default_store,
|
||||
)
|
||||
|
||||
@@ -246,6 +269,12 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
from deerflow.persistence.notifications import set_notification_store
|
||||
|
||||
set_notification_store(None)
|
||||
queue = getattr(app.state, "knowledge_auto_ingest_queue", None)
|
||||
if queue is not None:
|
||||
from deerflow.knowledge.auto_ingest import set_auto_ingest_queue
|
||||
|
||||
await queue.stop()
|
||||
set_auto_ingest_queue(None)
|
||||
await close_engine()
|
||||
|
||||
|
||||
@@ -380,6 +409,14 @@ def get_memory_config_store(request: Request):
|
||||
return val
|
||||
|
||||
|
||||
def get_knowledge_service(request: Request):
|
||||
"""Return the knowledge base service (503 when disabled / memory backend)."""
|
||||
val = getattr(request.app.state, "knowledge_service", None)
|
||||
if val is None:
|
||||
raise HTTPException(status_code=503, detail="Knowledge base not available")
|
||||
return val
|
||||
|
||||
|
||||
def get_run_context(request: Request) -> RunContext:
|
||||
"""Build a :class:`RunContext` from ``app.state`` singletons.
|
||||
|
||||
|
||||
@@ -40,6 +40,13 @@ def _check_csrf(request) -> None:
|
||||
# CSRF protection is for cookie-auth browser flows.
|
||||
return
|
||||
|
||||
# No cookie-based session → nothing for a cross-site forgery to ride, so
|
||||
# CSRF is moot. Enforcing it here would reject a freshly-loaded browser
|
||||
# (just cleared its cookies) before it can authenticate; such a request is
|
||||
# rejected as unauthenticated (401) below anyway.
|
||||
if not request.cookies.get("access_token"):
|
||||
return
|
||||
|
||||
cookie_token = request.cookies.get("csrf_token")
|
||||
header_token = request.headers.get("x-csrf-token")
|
||||
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
"""Resolve the application-relative request path behind a reverse proxy.
|
||||
|
||||
A reverse proxy may forward requests under a path prefix, e.g.
|
||||
``/xxx/api/v1/auth/login/token``. FastAPI strips that prefix for **routing**
|
||||
when it is told about it via the ASGI ``root_path`` (uvicorn ``--root-path`` /
|
||||
``--proxy-headers``, or ``X-Forwarded-Prefix`` from the proxy) — which is why
|
||||
the route still matches and the request reaches a handler.
|
||||
|
||||
Starlette middleware, however, runs *before* routing and sees the full,
|
||||
prefixed ``request.url.path``. Middlewares that match that raw path against
|
||||
fixed whitelists — the CSRF-exempt auth paths and the AuthMiddleware public
|
||||
paths — then mis-match, so login and ``/api/public/*`` endpoints get wrongly
|
||||
rejected (``CSRF token missing`` / ``401``) on a prefixed deployment.
|
||||
|
||||
``app_relative_path`` returns the path with the proxy prefix removed so that
|
||||
whitelist matching is prefix-agnostic. It is a no-op when no prefix is present
|
||||
(direct deployment), and tolerates the prefix already being stripped from the
|
||||
path (it only strips when the path actually starts with the prefix).
|
||||
"""
|
||||
|
||||
from fastapi import Request
|
||||
|
||||
|
||||
def app_relative_path(request: Request) -> str:
|
||||
"""Return ``request.url.path`` with any reverse-proxy prefix removed.
|
||||
|
||||
Prefix source order: ASGI ``root_path`` (set by the server when the proxy
|
||||
forwards it), then the ``X-Forwarded-Prefix`` header (proxies that add a
|
||||
prefix without informing the ASGI server). Returns ``"/"`` for an empty
|
||||
result so downstream ``rstrip("/")`` / ``startswith`` checks stay sane.
|
||||
"""
|
||||
path = request.url.path or "/"
|
||||
prefix = (request.scope.get("root_path") or request.headers.get("x-forwarded-prefix") or "").rstrip("/")
|
||||
if prefix and path.startswith(prefix):
|
||||
stripped = path[len(prefix):]
|
||||
if not stripped:
|
||||
return "/"
|
||||
return stripped if stripped.startswith("/") else "/" + stripped
|
||||
return path
|
||||
@@ -19,8 +19,11 @@ import logging
|
||||
import os
|
||||
import re
|
||||
from collections.abc import AsyncIterator
|
||||
from contextvars import ContextVar
|
||||
from difflib import SequenceMatcher
|
||||
from hashlib import sha256
|
||||
from time import perf_counter
|
||||
from typing import Any
|
||||
|
||||
from datetime import UTC, datetime, timedelta, time as datetime_time
|
||||
|
||||
@@ -56,6 +59,7 @@ from deerflow.persistence.user.model import UserRow
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/api/admin/users", tags=["admin-users"])
|
||||
self_router = APIRouter(prefix="/api/users/me", tags=["user-qa"])
|
||||
|
||||
_LEADERBOARD_MAX_CONCURRENT = max(
|
||||
1,
|
||||
@@ -99,6 +103,119 @@ _LEADERBOARD_QUEUE_ON_REQUEST_ENABLED = os.getenv(
|
||||
"off",
|
||||
}
|
||||
_leaderboard_semaphore = asyncio.Semaphore(_LEADERBOARD_MAX_CONCURRENT)
|
||||
_LEADERBOARD_TRACE_CONTEXT: ContextVar[dict[str, object] | None] = ContextVar(
|
||||
"leaderboard_trace_context",
|
||||
default=None,
|
||||
)
|
||||
|
||||
|
||||
def _leaderboard_log_context(**context: object) -> dict[str, object] | None:
|
||||
current = _LEADERBOARD_TRACE_CONTEXT.get()
|
||||
merged: dict[str, object] = dict(current or {})
|
||||
for key, value in context.items():
|
||||
if value is None or value == "":
|
||||
continue
|
||||
merged[key] = value
|
||||
return merged or None
|
||||
|
||||
|
||||
def _leaderboard_sql_for_log(stmt) -> str:
|
||||
try:
|
||||
return str(stmt.compile(compile_kwargs={"literal_binds": True}))
|
||||
except Exception:
|
||||
return str(stmt)
|
||||
|
||||
|
||||
async def _execute_leaderboard_query(session, step: str, stmt, **context: object):
|
||||
log_context = _leaderboard_log_context(**context)
|
||||
if not log_context:
|
||||
return await session.execute(stmt)
|
||||
|
||||
started_at = datetime.now(BEIJING_TZ).isoformat()
|
||||
sql = _leaderboard_sql_for_log(stmt)
|
||||
logger.info(
|
||||
"Leaderboard snapshot DB query start: step=%s stat_date=%s job=%s settings_hash=%s started_at=%s sql=%s",
|
||||
step,
|
||||
log_context.get("stat_date") or log_context.get("stat_dates") or "",
|
||||
log_context.get("job_key") or "",
|
||||
log_context.get("settings_hash") or "",
|
||||
started_at,
|
||||
sql,
|
||||
)
|
||||
start = perf_counter()
|
||||
try:
|
||||
result = await session.execute(stmt)
|
||||
except Exception:
|
||||
elapsed_ms = (perf_counter() - start) * 1000
|
||||
logger.exception(
|
||||
"Leaderboard snapshot DB query failed: step=%s stat_date=%s job=%s settings_hash=%s elapsed_ms=%.2f",
|
||||
step,
|
||||
log_context.get("stat_date") or log_context.get("stat_dates") or "",
|
||||
log_context.get("job_key") or "",
|
||||
log_context.get("settings_hash") or "",
|
||||
elapsed_ms,
|
||||
)
|
||||
raise
|
||||
|
||||
elapsed_ms = (perf_counter() - start) * 1000
|
||||
logger.info(
|
||||
"Leaderboard snapshot DB query done: step=%s stat_date=%s job=%s settings_hash=%s finished_at=%s elapsed_ms=%.2f",
|
||||
step,
|
||||
log_context.get("stat_date") or log_context.get("stat_dates") or "",
|
||||
log_context.get("job_key") or "",
|
||||
log_context.get("settings_hash") or "",
|
||||
datetime.now(BEIJING_TZ).isoformat(),
|
||||
elapsed_ms,
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
async def _time_leaderboard_db_operation(
|
||||
step: str,
|
||||
operation,
|
||||
*,
|
||||
sql: str = "",
|
||||
**context: object,
|
||||
):
|
||||
log_context = _leaderboard_log_context(**context)
|
||||
if not log_context:
|
||||
return await operation()
|
||||
|
||||
logger.info(
|
||||
"Leaderboard snapshot DB query start: step=%s stat_date=%s job=%s settings_hash=%s started_at=%s sql=%s",
|
||||
step,
|
||||
log_context.get("stat_date") or log_context.get("stat_dates") or "",
|
||||
log_context.get("job_key") or "",
|
||||
log_context.get("settings_hash") or "",
|
||||
datetime.now(BEIJING_TZ).isoformat(),
|
||||
sql,
|
||||
)
|
||||
start = perf_counter()
|
||||
try:
|
||||
result = await operation()
|
||||
except Exception:
|
||||
elapsed_ms = (perf_counter() - start) * 1000
|
||||
logger.exception(
|
||||
"Leaderboard snapshot DB query failed: step=%s stat_date=%s job=%s settings_hash=%s elapsed_ms=%.2f",
|
||||
step,
|
||||
log_context.get("stat_date") or log_context.get("stat_dates") or "",
|
||||
log_context.get("job_key") or "",
|
||||
log_context.get("settings_hash") or "",
|
||||
elapsed_ms,
|
||||
)
|
||||
raise
|
||||
|
||||
elapsed_ms = (perf_counter() - start) * 1000
|
||||
logger.info(
|
||||
"Leaderboard snapshot DB query done: step=%s stat_date=%s job=%s settings_hash=%s finished_at=%s elapsed_ms=%.2f",
|
||||
step,
|
||||
log_context.get("stat_date") or log_context.get("stat_dates") or "",
|
||||
log_context.get("job_key") or "",
|
||||
log_context.get("settings_hash") or "",
|
||||
datetime.now(BEIJING_TZ).isoformat(),
|
||||
elapsed_ms,
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
async def _require_admin(request: Request) -> None:
|
||||
@@ -285,6 +402,13 @@ def _parse_datetime(value: str | None, fallback: datetime) -> datetime:
|
||||
raise HTTPException(status_code=400, detail=f"Invalid datetime: {value}") from exc
|
||||
|
||||
|
||||
def _parse_optional_datetime(value: str | None) -> datetime | None:
|
||||
if not value:
|
||||
return None
|
||||
parsed = _parse_datetime(value, _now())
|
||||
return parsed if parsed.tzinfo is not None else parsed.replace(tzinfo=BEIJING_TZ)
|
||||
|
||||
|
||||
def _resolve_range(days: int, since: str | None, until: str | None) -> tuple[datetime, datetime, int]:
|
||||
end = _parse_datetime(until, _now())
|
||||
start = _parse_datetime(since, end - timedelta(days=max(1, min(days, 365))))
|
||||
@@ -325,7 +449,7 @@ def _bounds_for_stat_date(stat_date: str) -> tuple[datetime, datetime]:
|
||||
|
||||
async def _try_set_session_value(session, sql: str) -> None:
|
||||
try:
|
||||
await session.execute(text(sql))
|
||||
await _execute_leaderboard_query(session, "session_setting", text(sql))
|
||||
except Exception:
|
||||
logger.debug("Failed to apply analytics session setting: %s", sql, exc_info=True)
|
||||
try:
|
||||
@@ -334,13 +458,36 @@ async def _try_set_session_value(session, sql: str) -> None:
|
||||
logger.debug("Failed to rollback after analytics session setting failure", exc_info=True)
|
||||
|
||||
|
||||
def _is_mariadb(dialect) -> bool:
|
||||
"""SQLAlchemy reports ``dialect.name == 'mysql'`` for MariaDB too."""
|
||||
if getattr(dialect, "_is_mariadb", False) or getattr(dialect, "is_mariadb", False):
|
||||
return True
|
||||
version_info = getattr(dialect, "server_version_info", None) or ()
|
||||
return any("mariadb" in str(part).lower() for part in version_info)
|
||||
|
||||
|
||||
async def _set_analytics_read_timeout(session) -> None:
|
||||
"""Best-effort per-statement read timeout for the analytics queries.
|
||||
|
||||
The right knob differs by server, and some servers have none at all:
|
||||
- MySQL >= 5.7.4: ``max_execution_time`` (milliseconds)
|
||||
- MariaDB: ``max_statement_time`` (seconds)
|
||||
- MySQL <= 5.6: no per-statement timeout variable — skip it entirely
|
||||
(5.6 raises "Unknown system variable" otherwise)
|
||||
- PostgreSQL: ``statement_timeout`` (milliseconds)
|
||||
"""
|
||||
conn = await session.connection()
|
||||
dialect = conn.dialect.name
|
||||
if dialect == "mysql":
|
||||
await _try_set_session_value(session, f"SET SESSION max_execution_time = {_LEADERBOARD_QUERY_TIMEOUT_MS}")
|
||||
dialect = conn.dialect
|
||||
name = dialect.name
|
||||
if name == "mysql":
|
||||
if _is_mariadb(dialect):
|
||||
# MariaDB: max_statement_time is in SECONDS (fractional ok).
|
||||
await _try_set_session_value(session, f"SET SESSION max_statement_time = {_LEADERBOARD_QUERY_TIMEOUT_MS / 1000}")
|
||||
elif (getattr(dialect, "server_version_info", None) or ()) >= (5, 7, 4):
|
||||
await _try_set_session_value(session, f"SET SESSION max_execution_time = {_LEADERBOARD_QUERY_TIMEOUT_MS}")
|
||||
# else: MySQL <= 5.6 has no statement-timeout variable — skip silently.
|
||||
await _try_set_session_value(session, f"SET SESSION innodb_lock_wait_timeout = {_LEADERBOARD_LOCK_WAIT_SECONDS}")
|
||||
elif dialect == "postgresql":
|
||||
elif name == "postgresql":
|
||||
await _try_set_session_value(session, f"SET LOCAL statement_timeout = {_LEADERBOARD_QUERY_TIMEOUT_MS}")
|
||||
|
||||
|
||||
@@ -661,11 +808,15 @@ async def _query_user_leaderboard(
|
||||
if user_id:
|
||||
stmt = stmt.where(UserRow.id == user_id)
|
||||
stmt = _apply_user_scope(stmt, UserRow.id, exclude_user_ids=exclude_user_ids, include_admins=include_admins)
|
||||
# 排行榜只统计真正产生过问答的用户;从未问答的账号(仅注册、零活跃)不计入榜单与人数。
|
||||
# 个人分析(指定 user_id)不受此限制,仍可查看任意单个用户。
|
||||
if user_id is None:
|
||||
stmt = stmt.where(func.coalesce(run_agg.c.qa_count, 0) > 0)
|
||||
stmt = stmt.order_by(
|
||||
desc(func.coalesce(run_agg.c.run_count, 0) + func.coalesce(tool_agg.c.tool_call_count, 0)),
|
||||
desc(func.coalesce(run_agg.c.total_tokens, 0)),
|
||||
)
|
||||
rows = (await session.execute(stmt)).all()
|
||||
rows = (await _execute_leaderboard_query(session, "users", stmt)).all()
|
||||
return [
|
||||
AdminLeaderboardUser(
|
||||
user_id=r[0],
|
||||
@@ -725,7 +876,7 @@ async def _query_call_rankings(
|
||||
include_admins=include_admins,
|
||||
include_scheduled=include_scheduled,
|
||||
)
|
||||
rows = (await session.execute(stmt)).all()
|
||||
rows = (await _execute_leaderboard_query(session, f"{field}_rankings", stmt)).all()
|
||||
return [
|
||||
AdminLeaderboardSkill(
|
||||
name=r[0] or "",
|
||||
@@ -791,7 +942,7 @@ async def _query_run_dimensions(
|
||||
include_failed=include_failed,
|
||||
include_scheduled=include_scheduled,
|
||||
)
|
||||
rows = (await session.execute(stmt)).all()
|
||||
rows = (await _execute_leaderboard_query(session, f"{field}_dimensions", stmt)).all()
|
||||
return [
|
||||
AdminLeaderboardDimension(
|
||||
name=r[0] or "",
|
||||
@@ -848,7 +999,7 @@ async def _query_model_rankings(
|
||||
stmt = stmt.where(LlmCallMetricRow.status != "error")
|
||||
if not include_scheduled:
|
||||
stmt = stmt.where(or_(LlmCallMetricRow.run_id.is_(None), LlmCallMetricRow.run_id.not_in(_scheduled_run_ids_stmt())))
|
||||
rows = (await session.execute(stmt)).all()
|
||||
rows = (await _execute_leaderboard_query(session, "model_rankings", stmt)).all()
|
||||
return [
|
||||
AdminLeaderboardDimension(
|
||||
name=r[0] or "",
|
||||
@@ -895,7 +1046,7 @@ async def _query_recent_questions(
|
||||
include_failed=include_failed,
|
||||
include_scheduled=include_scheduled,
|
||||
)
|
||||
rows = (await session.execute(stmt)).all()
|
||||
rows = (await _execute_leaderboard_query(session, "recent_questions", stmt)).all()
|
||||
questions: list[AdminLeaderboardQuestion] = []
|
||||
for r in rows:
|
||||
text = _clean_question_text(r[0]) if cleanup_system_questions else str(r[0] or "").strip()
|
||||
@@ -947,7 +1098,7 @@ async def _query_frequent_questions(
|
||||
include_failed=include_failed,
|
||||
include_scheduled=include_scheduled,
|
||||
)
|
||||
rows = (await session.execute(stmt)).all()
|
||||
rows = (await _execute_leaderboard_query(session, "frequent_questions", stmt)).all()
|
||||
groups: list[dict] = []
|
||||
for r in rows:
|
||||
text = _clean_question_text(r[0]) if cleanup_system_questions else str(r[0] or "").strip()
|
||||
@@ -1022,7 +1173,7 @@ async def _query_trends(
|
||||
include_failed=include_failed,
|
||||
include_scheduled=include_scheduled,
|
||||
)
|
||||
rows = (await session.execute(stmt)).all()
|
||||
rows = (await _execute_leaderboard_query(session, "trends", stmt)).all()
|
||||
return [
|
||||
AdminLeaderboardTrendPoint(
|
||||
date=str(r[0]),
|
||||
@@ -1071,8 +1222,8 @@ async def _query_overview(
|
||||
|
||||
total_users_stmt = select(func.count(UserRow.id))
|
||||
total_users_stmt = _apply_user_scope(total_users_stmt, UserRow.id, exclude_user_ids=exclude_user_ids, include_admins=include_admins)
|
||||
total_users = 1 if user_id else (await session.execute(total_users_stmt)).scalar_one()
|
||||
run_row = (await session.execute(
|
||||
total_users = 1 if user_id else (await _execute_leaderboard_query(session, "overview.total_users", total_users_stmt)).scalar_one()
|
||||
run_summary_stmt = (
|
||||
select(
|
||||
func.count(RunRow.run_id),
|
||||
_question_count_expr(),
|
||||
@@ -1080,16 +1231,19 @@ async def _query_overview(
|
||||
func.count(func.distinct(RunRow.user_id)),
|
||||
func.sum(case((RunRow.status == "error", 1), else_=0)),
|
||||
).where(*run_where)
|
||||
)).one()
|
||||
thread_count = (await session.execute(select(func.count(ThreadMetaRow.thread_id)).where(*thread_where))).scalar_one()
|
||||
tool_row = (await session.execute(
|
||||
)
|
||||
run_row = (await _execute_leaderboard_query(session, "overview.run_summary", run_summary_stmt)).one()
|
||||
thread_count_stmt = select(func.count(ThreadMetaRow.thread_id)).where(*thread_where)
|
||||
thread_count = (await _execute_leaderboard_query(session, "overview.thread_count", thread_count_stmt)).scalar_one()
|
||||
tool_summary_stmt = (
|
||||
select(
|
||||
func.count(ToolCallMetricRow.id),
|
||||
func.sum(case((func.nullif(ToolCallMetricRow.skill_name, "").is_not(None), 1), else_=0)),
|
||||
)
|
||||
.outerjoin(RunRow, RunRow.run_id == ToolCallMetricRow.run_id)
|
||||
.where(*tool_where)
|
||||
)).one()
|
||||
)
|
||||
tool_row = (await _execute_leaderboard_query(session, "overview.tool_summary", tool_summary_stmt)).one()
|
||||
|
||||
return AdminLeaderboardOverview(
|
||||
total_users=int(total_users or 0),
|
||||
@@ -1579,13 +1733,12 @@ async def _load_daily_snapshot_rows(
|
||||
if not stat_dates:
|
||||
return {}
|
||||
async with session_factory() as session:
|
||||
stmt = select(AdminLeaderboardDailyStatRow).where(
|
||||
AdminLeaderboardDailyStatRow.stat_date.in_(stat_dates),
|
||||
AdminLeaderboardDailyStatRow.settings_hash == settings_hash,
|
||||
)
|
||||
rows = (
|
||||
await session.execute(
|
||||
select(AdminLeaderboardDailyStatRow).where(
|
||||
AdminLeaderboardDailyStatRow.stat_date.in_(stat_dates),
|
||||
AdminLeaderboardDailyStatRow.settings_hash == settings_hash,
|
||||
)
|
||||
)
|
||||
await _execute_leaderboard_query(session, "snapshot_rows.load", stmt)
|
||||
).scalars().all()
|
||||
return {row.stat_date: row for row in rows}
|
||||
|
||||
@@ -1603,9 +1756,15 @@ async def _store_daily_snapshot(
|
||||
) -> None:
|
||||
async with session_factory() as session:
|
||||
now = datetime.now(UTC)
|
||||
row = await session.get(
|
||||
AdminLeaderboardDailyStatRow,
|
||||
{"stat_date": stat_date, "settings_hash": settings_hash},
|
||||
row = await _time_leaderboard_db_operation(
|
||||
"snapshot.store.get",
|
||||
lambda: session.get(
|
||||
AdminLeaderboardDailyStatRow,
|
||||
{"stat_date": stat_date, "settings_hash": settings_hash},
|
||||
),
|
||||
sql="SELECT admin_leaderboard_daily_stats by primary key",
|
||||
stat_date=stat_date,
|
||||
settings_hash=settings_hash,
|
||||
)
|
||||
if row is None:
|
||||
row = AdminLeaderboardDailyStatRow(
|
||||
@@ -1630,7 +1789,13 @@ async def _store_daily_snapshot(
|
||||
if generated_at is not None:
|
||||
row.generated_at = generated_at
|
||||
row.updated_at = now
|
||||
await session.commit()
|
||||
await _time_leaderboard_db_operation(
|
||||
"snapshot.store.commit",
|
||||
lambda: session.commit(),
|
||||
sql="COMMIT",
|
||||
stat_date=stat_date,
|
||||
settings_hash=settings_hash,
|
||||
)
|
||||
|
||||
|
||||
async def _queue_daily_leaderboard_snapshots(
|
||||
@@ -1646,12 +1811,17 @@ async def _queue_daily_leaderboard_snapshots(
|
||||
return queued
|
||||
async with session_factory() as session:
|
||||
now = datetime.now(UTC)
|
||||
stmt = select(AdminLeaderboardDailyStatRow).where(
|
||||
AdminLeaderboardDailyStatRow.stat_date.in_(stat_dates),
|
||||
AdminLeaderboardDailyStatRow.settings_hash == settings_hash,
|
||||
)
|
||||
rows = (
|
||||
await session.execute(
|
||||
select(AdminLeaderboardDailyStatRow).where(
|
||||
AdminLeaderboardDailyStatRow.stat_date.in_(stat_dates),
|
||||
AdminLeaderboardDailyStatRow.settings_hash == settings_hash,
|
||||
)
|
||||
await _execute_leaderboard_query(
|
||||
session,
|
||||
"snapshot_queue.load_existing",
|
||||
stmt,
|
||||
stat_dates=stat_dates,
|
||||
settings_hash=settings_hash,
|
||||
)
|
||||
).scalars().all()
|
||||
rows_by_date = {row.stat_date: row for row in rows}
|
||||
@@ -1680,7 +1850,13 @@ async def _queue_daily_leaderboard_snapshots(
|
||||
row.updated_at = now
|
||||
queued.append(stat_date)
|
||||
if queued:
|
||||
await session.commit()
|
||||
await _time_leaderboard_db_operation(
|
||||
"snapshot_queue.commit",
|
||||
lambda: session.commit(),
|
||||
sql="COMMIT",
|
||||
stat_dates=queued,
|
||||
settings_hash=settings_hash,
|
||||
)
|
||||
return queued
|
||||
|
||||
|
||||
@@ -1722,7 +1898,22 @@ async def _refresh_daily_leaderboard_snapshot(
|
||||
include_failed: bool,
|
||||
) -> None:
|
||||
acquired = False
|
||||
trace_token = _LEADERBOARD_TRACE_CONTEXT.set(
|
||||
{
|
||||
"job_key": job_key,
|
||||
"stat_date": stat_date,
|
||||
"settings_hash": settings_hash,
|
||||
}
|
||||
)
|
||||
snapshot_start = perf_counter()
|
||||
try:
|
||||
logger.info(
|
||||
"Leaderboard snapshot build start: job=%s stat_date=%s settings_hash=%s started_at=%s",
|
||||
job_key,
|
||||
stat_date,
|
||||
settings_hash,
|
||||
datetime.now(BEIJING_TZ).isoformat(),
|
||||
)
|
||||
if _LEADERBOARD_START_DELAY_SECONDS > 0:
|
||||
await asyncio.sleep(_LEADERBOARD_START_DELAY_SECONDS)
|
||||
await _store_daily_snapshot(
|
||||
@@ -1758,6 +1949,14 @@ async def _refresh_daily_leaderboard_snapshot(
|
||||
error=None,
|
||||
generated_at=generated_at,
|
||||
)
|
||||
logger.info(
|
||||
"Leaderboard snapshot build done: job=%s stat_date=%s settings_hash=%s finished_at=%s elapsed_ms=%.2f",
|
||||
job_key,
|
||||
stat_date,
|
||||
settings_hash,
|
||||
datetime.now(BEIJING_TZ).isoformat(),
|
||||
(perf_counter() - snapshot_start) * 1000,
|
||||
)
|
||||
except asyncio.TimeoutError:
|
||||
logger.warning("Daily leaderboard snapshot timed out: %s", job_key)
|
||||
await _store_daily_snapshot(
|
||||
@@ -1781,6 +1980,7 @@ async def _refresh_daily_leaderboard_snapshot(
|
||||
finally:
|
||||
if acquired:
|
||||
_leaderboard_semaphore.release()
|
||||
_LEADERBOARD_TRACE_CONTEXT.reset(trace_token)
|
||||
|
||||
|
||||
async def _load_or_schedule_daily_leaderboard(
|
||||
@@ -1978,6 +2178,72 @@ async def queue_leaderboard_snapshot_backfill(
|
||||
)
|
||||
|
||||
|
||||
@router.post("/leaderboard/refresh-today", response_model=AdminLeaderboardResponse)
|
||||
async def refresh_today_leaderboard(
|
||||
request: Request,
|
||||
days: int = Query(default=30, ge=1, le=365),
|
||||
since: str | None = Query(default=None, description="Inclusive range start, ISO datetime"),
|
||||
until: str | None = Query(default=None, description="Inclusive range end, ISO datetime"),
|
||||
exclude_user_ids: list[str] | None = Query(default=None, description="User ids excluded from this leaderboard aggregation"),
|
||||
include_scheduled: bool | None = Query(default=None, description="Include runs created by scheduled tasks"),
|
||||
include_admins: bool | None = Query(default=None, description="Include admin accounts in leaderboard aggregation"),
|
||||
include_failed: bool | None = Query(default=None, description="Include failed runs and LLM calls"),
|
||||
limit: int = Query(default=20, ge=5, le=100),
|
||||
) -> AdminLeaderboardResponse:
|
||||
"""Force a synchronous rebuild of *today's* snapshot, then return the range.
|
||||
|
||||
Today's snapshot otherwise refreshes only every ``_LEADERBOARD_TODAY_TTL_SECONDS``
|
||||
(~5 min) via the background scheduler. This endpoint lets an admin pull the
|
||||
latest real-time records into the stats on demand: it rebuilds today's daily
|
||||
snapshot immediately (bounded by the same semaphore/timeout the scheduler uses)
|
||||
and then returns the merged leaderboard for the requested range.
|
||||
"""
|
||||
await _require_admin(request)
|
||||
session_factory = get_session_factory()
|
||||
if session_factory is None:
|
||||
raise HTTPException(status_code=503, detail="Analytics requires database persistence")
|
||||
|
||||
settings = load_system_settings().leaderboard
|
||||
(
|
||||
excluded,
|
||||
resolved_include_scheduled,
|
||||
resolved_include_admins,
|
||||
resolved_include_failed,
|
||||
settings_hash,
|
||||
) = _resolve_leaderboard_snapshot_scope(
|
||||
settings,
|
||||
exclude_user_ids=exclude_user_ids,
|
||||
include_scheduled=include_scheduled,
|
||||
include_admins=include_admins,
|
||||
include_failed=include_failed,
|
||||
)
|
||||
today = _today_stat_date()
|
||||
await _refresh_daily_leaderboard_snapshot(
|
||||
_daily_job_key(today, settings_hash),
|
||||
session_factory=session_factory,
|
||||
settings=settings,
|
||||
stat_date=today,
|
||||
settings_hash=settings_hash,
|
||||
excluded=excluded,
|
||||
include_scheduled=resolved_include_scheduled,
|
||||
include_admins=resolved_include_admins,
|
||||
include_failed=resolved_include_failed,
|
||||
)
|
||||
|
||||
since_dt, until_dt, _range_days = _resolve_range(days, since, until)
|
||||
return await _load_or_schedule_daily_leaderboard(
|
||||
session_factory,
|
||||
settings,
|
||||
since_dt,
|
||||
until_dt,
|
||||
limit=limit,
|
||||
exclude_user_ids=exclude_user_ids,
|
||||
include_scheduled=include_scheduled,
|
||||
include_admins=include_admins,
|
||||
include_failed=include_failed,
|
||||
)
|
||||
|
||||
|
||||
async def _build_leaderboard_report_prompt(
|
||||
session_factory,
|
||||
settings,
|
||||
@@ -2248,22 +2514,182 @@ class AdminUserQaPageResponse(BaseModel):
|
||||
items: list[AdminUserQaItem] = Field(default_factory=list)
|
||||
|
||||
|
||||
@router.get("/{user_id}/qa", response_model=AdminUserQaPageResponse)
|
||||
async def list_user_qa(
|
||||
user_id: str,
|
||||
request: Request,
|
||||
page: int = Query(default=1, ge=1),
|
||||
page_size: int = Query(default=20, ge=1, le=100),
|
||||
q: str | None = Query(default=None, description="Search question, answer, run id, or thread id"),
|
||||
since: str | None = Query(default=None, description="Inclusive range start, ISO datetime"),
|
||||
until: str | None = Query(default=None, description="Inclusive range end, ISO datetime"),
|
||||
) -> AdminUserQaPageResponse:
|
||||
"""Return one user's Q&A runs from lightweight run summary fields."""
|
||||
await _require_admin(request)
|
||||
session_factory = get_session_factory()
|
||||
if session_factory is None:
|
||||
raise HTTPException(status_code=503, detail="Q&A audit requires database persistence")
|
||||
def _qa_sort_datetime(value: str | None) -> datetime:
|
||||
if not value:
|
||||
return datetime.min.replace(tzinfo=BEIJING_TZ)
|
||||
text_value = str(value).strip()
|
||||
if text_value.endswith("Z"):
|
||||
text_value = text_value[:-1] + "+00:00"
|
||||
try:
|
||||
parsed = datetime.fromisoformat(text_value)
|
||||
except ValueError:
|
||||
return datetime.min.replace(tzinfo=BEIJING_TZ)
|
||||
if parsed.tzinfo is None:
|
||||
return parsed.replace(tzinfo=BEIJING_TZ)
|
||||
return parsed.astimezone(BEIJING_TZ)
|
||||
|
||||
|
||||
def _qa_dedupe_key(item: AdminUserQaItem) -> tuple[str, str, str]:
|
||||
question_key = _question_similarity_key(item.question)[:160]
|
||||
answer_key = _question_similarity_key(item.answer)[:160]
|
||||
return (item.thread_id, question_key, answer_key)
|
||||
|
||||
|
||||
def _message_value(message: Any, key: str, default: Any = None) -> Any:
|
||||
if isinstance(message, dict):
|
||||
return message.get(key, default)
|
||||
return getattr(message, key, default)
|
||||
|
||||
|
||||
def _qa_message_type(message: Any) -> str:
|
||||
value = _message_value(message, "type") or _message_value(message, "role") or ""
|
||||
if value == "user":
|
||||
return "human"
|
||||
if value == "assistant":
|
||||
return "ai"
|
||||
return str(value)
|
||||
|
||||
|
||||
def _qa_message_id(message: Any, fallback: str) -> str:
|
||||
return str(_message_value(message, "id") or fallback)
|
||||
|
||||
|
||||
def _qa_message_text(message: Any) -> str:
|
||||
return _message_content_to_text(_message_value(message, "content")).strip()
|
||||
|
||||
|
||||
async def _read_thread_checkpoint_messages(checkpointer, thread_id: str) -> list[dict]:
|
||||
config = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
|
||||
try:
|
||||
checkpoint_tuple = await checkpointer.aget_tuple(config)
|
||||
except Exception:
|
||||
logger.warning("Failed to read checkpoint while listing Q&A: thread=%s", thread_id, exc_info=True)
|
||||
return []
|
||||
if checkpoint_tuple is None:
|
||||
return []
|
||||
checkpoint = getattr(checkpoint_tuple, "checkpoint", {}) or {}
|
||||
channel_values = checkpoint.get("channel_values", {}) if isinstance(checkpoint, dict) else {}
|
||||
values = serialize_channel_values(channel_values)
|
||||
messages = values.get("messages", [])
|
||||
return messages if isinstance(messages, list) else []
|
||||
|
||||
|
||||
def _qa_items_from_checkpoint_messages(
|
||||
*,
|
||||
user_id: str,
|
||||
thread_id: str,
|
||||
thread_title: str,
|
||||
assistant_id: str,
|
||||
thread_time: str,
|
||||
messages: list[dict],
|
||||
) -> list[AdminUserQaItem]:
|
||||
items: list[AdminUserQaItem] = []
|
||||
current_question = ""
|
||||
current_answer = ""
|
||||
current_id = ""
|
||||
current_index = 0
|
||||
|
||||
def flush() -> None:
|
||||
nonlocal current_question, current_answer, current_id, current_index
|
||||
question = _clean_question_text(current_question)
|
||||
if not _is_displayable_question(question):
|
||||
current_question = ""
|
||||
current_answer = ""
|
||||
current_id = ""
|
||||
return
|
||||
items.append(
|
||||
AdminUserQaItem(
|
||||
run_id=current_id or f"{thread_id}:checkpoint:{current_index}",
|
||||
thread_id=thread_id,
|
||||
thread_title=thread_title,
|
||||
assistant_id=assistant_id,
|
||||
status="checkpoint",
|
||||
question=question[:2000],
|
||||
answer=current_answer.strip()[:6000],
|
||||
created_at=thread_time,
|
||||
updated_at=thread_time,
|
||||
)
|
||||
)
|
||||
current_question = ""
|
||||
current_answer = ""
|
||||
current_id = ""
|
||||
|
||||
for index, message in enumerate(messages):
|
||||
msg_type = _qa_message_type(message)
|
||||
if msg_type == "human":
|
||||
if current_question:
|
||||
flush()
|
||||
current_question = _qa_message_text(message)
|
||||
current_answer = ""
|
||||
current_id = _qa_message_id(message, f"{thread_id}:checkpoint:{index}")
|
||||
current_index = index
|
||||
continue
|
||||
if msg_type == "ai" and current_question:
|
||||
text = _qa_message_text(message)
|
||||
if text:
|
||||
current_answer = text
|
||||
|
||||
if current_question:
|
||||
flush()
|
||||
|
||||
return items
|
||||
|
||||
|
||||
async def _list_thread_checkpoint_qa_items(
|
||||
*,
|
||||
request: Request,
|
||||
user_id: str,
|
||||
) -> list[AdminUserQaItem]:
|
||||
repo = get_thread_store(request)
|
||||
checkpointer = get_checkpointer(request)
|
||||
items: list[AdminUserQaItem] = []
|
||||
offset = 0
|
||||
page_size = 200
|
||||
while True:
|
||||
rows = await repo.search(limit=page_size, offset=offset, user_id=user_id)
|
||||
if not rows:
|
||||
break
|
||||
for row in rows:
|
||||
thread_id = str(row.get("thread_id") or "")
|
||||
if not thread_id:
|
||||
continue
|
||||
messages = await _read_thread_checkpoint_messages(checkpointer, thread_id)
|
||||
if not messages:
|
||||
continue
|
||||
metadata = row.get("metadata") if isinstance(row.get("metadata"), dict) else {}
|
||||
context = row.get("context") if isinstance(row.get("context"), dict) else {}
|
||||
assistant_id = str(
|
||||
row.get("assistant_id")
|
||||
or (context or {}).get("agent_id")
|
||||
or (metadata or {}).get("agent_id")
|
||||
or (context or {}).get("agent_name")
|
||||
or (metadata or {}).get("agent_name")
|
||||
or ""
|
||||
)
|
||||
thread_time = str(row.get("updated_at") or row.get("created_at") or "")
|
||||
items.extend(
|
||||
_qa_items_from_checkpoint_messages(
|
||||
user_id=user_id,
|
||||
thread_id=thread_id,
|
||||
thread_title=str(row.get("display_name") or ""),
|
||||
assistant_id=assistant_id,
|
||||
thread_time=thread_time,
|
||||
messages=messages,
|
||||
)
|
||||
)
|
||||
if len(rows) < page_size:
|
||||
break
|
||||
offset += len(rows)
|
||||
return items
|
||||
|
||||
|
||||
async def _list_run_summary_qa_items(
|
||||
*,
|
||||
session_factory,
|
||||
user_id: str,
|
||||
since: str | None,
|
||||
until: str | None,
|
||||
) -> list[AdminUserQaItem]:
|
||||
filters = [
|
||||
RunRow.user_id == user_id,
|
||||
RunRow.first_human_message.is_not(None),
|
||||
@@ -2272,26 +2698,9 @@ async def list_user_qa(
|
||||
if since or until:
|
||||
since_dt, until_dt, _range_days = _resolve_range(365, since, until)
|
||||
filters.append(_run_time_filter(since_dt, until_dt))
|
||||
needle = (q or "").strip()
|
||||
if needle:
|
||||
like = f"%{needle}%"
|
||||
filters.append(
|
||||
or_(
|
||||
RunRow.first_human_message.like(like),
|
||||
RunRow.last_ai_message.like(like),
|
||||
RunRow.run_id.like(like),
|
||||
RunRow.thread_id.like(like),
|
||||
)
|
||||
)
|
||||
|
||||
offset = (page - 1) * page_size
|
||||
async with session_factory() as session:
|
||||
await _set_analytics_read_timeout(session)
|
||||
total = (
|
||||
await session.execute(
|
||||
select(func.count(RunRow.run_id)).where(*filters)
|
||||
)
|
||||
).scalar_one()
|
||||
rows = (
|
||||
await session.execute(
|
||||
select(
|
||||
@@ -2316,32 +2725,152 @@ async def list_user_qa(
|
||||
)
|
||||
.where(*filters)
|
||||
.order_by(RunRow.created_at.desc(), RunRow.run_id.desc())
|
||||
.limit(page_size)
|
||||
.offset(offset)
|
||||
)
|
||||
).all()
|
||||
|
||||
return [
|
||||
AdminUserQaItem(
|
||||
run_id=r[0] or "",
|
||||
thread_id=r[1] or "",
|
||||
thread_title=r[2] or "",
|
||||
assistant_id=r[3] or "",
|
||||
status=r[4] or "",
|
||||
model_name=r[5] or "",
|
||||
question=_clean_question_text(r[6])[:2000],
|
||||
answer=str(r[7] or "").strip()[:6000],
|
||||
total_tokens=int(r[8] or 0),
|
||||
created_at=_iso(r[9]),
|
||||
updated_at=_iso(r[10]),
|
||||
)
|
||||
for r in rows
|
||||
if _is_displayable_question(r[6])
|
||||
]
|
||||
|
||||
|
||||
def _filter_qa_items(
|
||||
items: list[AdminUserQaItem],
|
||||
*,
|
||||
q: str | None,
|
||||
since: str | None,
|
||||
until: str | None,
|
||||
) -> list[AdminUserQaItem]:
|
||||
needle = (q or "").strip().lower()
|
||||
since_dt = _parse_optional_datetime(since) if since else None
|
||||
until_dt = _parse_optional_datetime(until) if until else None
|
||||
filtered: list[AdminUserQaItem] = []
|
||||
for item in items:
|
||||
item_dt = _qa_sort_datetime(item.created_at or item.updated_at)
|
||||
if since_dt and item_dt < since_dt:
|
||||
continue
|
||||
if until_dt and item_dt > until_dt:
|
||||
continue
|
||||
if needle:
|
||||
haystack = "\n".join(
|
||||
[
|
||||
item.question,
|
||||
item.answer,
|
||||
item.run_id,
|
||||
item.thread_id,
|
||||
item.thread_title,
|
||||
item.assistant_id,
|
||||
]
|
||||
).lower()
|
||||
if needle not in haystack:
|
||||
continue
|
||||
filtered.append(item)
|
||||
return filtered
|
||||
|
||||
|
||||
async def _list_user_qa_page(
|
||||
*,
|
||||
request: Request,
|
||||
user_id: str,
|
||||
page: int,
|
||||
page_size: int,
|
||||
q: str | None,
|
||||
since: str | None,
|
||||
until: str | None,
|
||||
include_checkpoints: bool = True,
|
||||
) -> AdminUserQaPageResponse:
|
||||
session_factory = get_session_factory()
|
||||
if session_factory is None:
|
||||
raise HTTPException(status_code=503, detail="Q&A audit requires database persistence")
|
||||
|
||||
items = await _list_run_summary_qa_items(
|
||||
session_factory=session_factory,
|
||||
user_id=user_id,
|
||||
since=since,
|
||||
until=until,
|
||||
)
|
||||
if include_checkpoints:
|
||||
items.extend(await _list_thread_checkpoint_qa_items(request=request, user_id=user_id))
|
||||
|
||||
by_key: dict[tuple[str, str, str], AdminUserQaItem] = {}
|
||||
for item in items:
|
||||
key = _qa_dedupe_key(item)
|
||||
existing = by_key.get(key)
|
||||
if existing is None or (existing.status == "checkpoint" and item.status != "checkpoint"):
|
||||
by_key[key] = item
|
||||
|
||||
filtered = _filter_qa_items(list(by_key.values()), q=q, since=since, until=until)
|
||||
filtered.sort(
|
||||
key=lambda item: (_qa_sort_datetime(item.created_at or item.updated_at), item.thread_id, item.run_id),
|
||||
reverse=True,
|
||||
)
|
||||
offset = (page - 1) * page_size
|
||||
return AdminUserQaPageResponse(
|
||||
user_id=user_id,
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
total=int(total or 0),
|
||||
items=[
|
||||
AdminUserQaItem(
|
||||
run_id=r[0] or "",
|
||||
thread_id=r[1] or "",
|
||||
thread_title=r[2] or "",
|
||||
assistant_id=r[3] or "",
|
||||
status=r[4] or "",
|
||||
model_name=r[5] or "",
|
||||
question=_clean_question_text(r[6])[:2000],
|
||||
answer=str(r[7] or "").strip()[:6000],
|
||||
total_tokens=int(r[8] or 0),
|
||||
created_at=_iso(r[9]),
|
||||
updated_at=_iso(r[10]),
|
||||
)
|
||||
for r in rows
|
||||
],
|
||||
total=len(filtered),
|
||||
items=filtered[offset : offset + page_size],
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{user_id}/qa", response_model=AdminUserQaPageResponse)
|
||||
async def list_user_qa(
|
||||
user_id: str,
|
||||
request: Request,
|
||||
page: int = Query(default=1, ge=1),
|
||||
page_size: int = Query(default=20, ge=1, le=100),
|
||||
q: str | None = Query(default=None, description="Search question, answer, run id, or thread id"),
|
||||
since: str | None = Query(default=None, description="Inclusive range start, ISO datetime"),
|
||||
until: str | None = Query(default=None, description="Inclusive range end, ISO datetime"),
|
||||
) -> AdminUserQaPageResponse:
|
||||
"""Return one user's Q&A runs, including older checkpoint-only threads."""
|
||||
await _require_admin(request)
|
||||
return await _list_user_qa_page(
|
||||
request=request,
|
||||
user_id=user_id,
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
q=q,
|
||||
since=since,
|
||||
until=until,
|
||||
)
|
||||
|
||||
|
||||
@self_router.get("/qa", response_model=AdminUserQaPageResponse)
|
||||
async def list_my_qa(
|
||||
request: Request,
|
||||
page: int = Query(default=1, ge=1),
|
||||
page_size: int = Query(default=20, ge=1, le=100),
|
||||
q: str | None = Query(default=None, description="Search question, answer, run id, or thread id"),
|
||||
since: str | None = Query(default=None, description="Inclusive range start, ISO datetime"),
|
||||
until: str | None = Query(default=None, description="Inclusive range end, ISO datetime"),
|
||||
) -> AdminUserQaPageResponse:
|
||||
"""Return the current user's own Q&A history."""
|
||||
user = getattr(request.state, "user", None)
|
||||
if user is None:
|
||||
user = await get_current_user_from_request(request)
|
||||
return await _list_user_qa_page(
|
||||
request=request,
|
||||
user_id=str(user.id),
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
q=q,
|
||||
since=since,
|
||||
until=until,
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -15,7 +15,7 @@ from fastapi import APIRouter, Depends, HTTPException, Query, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.gateway.auth.password import verify_password_async
|
||||
from app.gateway.deps import get_agent_store, get_config, get_tag_store
|
||||
from app.gateway.deps import get_agent_store, get_config, get_tag_store
|
||||
from deerflow.agents.lead_agent.prompt import refresh_skills_system_prompt_cache_async
|
||||
from deerflow.config.agents_api_config import get_agents_api_config
|
||||
from deerflow.config.agents_config import list_custom_agents, load_agent_config, load_agent_soul, validate_agent_id
|
||||
@@ -39,6 +39,7 @@ class AgentResponse(BaseModel):
|
||||
name: str = Field(..., description="Display name. Chinese and duplicate names are allowed.")
|
||||
description: str = Field(default="", description="Agent description")
|
||||
user_id: str | None = Field(default=None, description="Owner user id. Null means built-in.")
|
||||
owner_username: str | None = Field(default=None, description="Publisher username (email local part). Null for built-in agents.")
|
||||
published: bool = Field(default=False, description="Whether a user-created agent is public")
|
||||
is_private: bool = Field(default=False, description="Whether the agent lives in the password-gated private square instead of the public one")
|
||||
builtin: bool = Field(default=False, description="True when user_id is null")
|
||||
@@ -62,15 +63,15 @@ class AgentResponse(BaseModel):
|
||||
is_favorite: bool = Field(
|
||||
default=False,
|
||||
description="Whether the requesting user has favorited (收藏) this agent.",
|
||||
)
|
||||
|
||||
is_pinned: bool = Field(
|
||||
default=False,
|
||||
description="Whether the requesting user has pinned this agent.",
|
||||
)
|
||||
|
||||
|
||||
class AgentsListResponse(BaseModel):
|
||||
)
|
||||
|
||||
is_pinned: bool = Field(
|
||||
default=False,
|
||||
description="Whether the requesting user has pinned this agent.",
|
||||
)
|
||||
|
||||
|
||||
class AgentsListResponse(BaseModel):
|
||||
"""Response model for listing agents.
|
||||
|
||||
``total`` / ``page`` / ``page_size`` describe server-side pagination. When
|
||||
@@ -115,7 +116,7 @@ class AgentUpdateRequest(BaseModel):
|
||||
tool_groups: list[str] | None = Field(default=None, description="Updated tool group whitelist (config.yaml)")
|
||||
|
||||
|
||||
class AgentSelectorResponse(BaseModel):
|
||||
class AgentSelectorResponse(BaseModel):
|
||||
"""Response for the homepage agent selector dropdown.
|
||||
|
||||
The list is fully curated by admins (``featured_order`` on each row).
|
||||
@@ -125,10 +126,10 @@ class AgentSelectorResponse(BaseModel):
|
||||
agents: list[AgentResponse] = Field(
|
||||
default_factory=list,
|
||||
description="Admin-curated agents, sorted by featured_order ascending.",
|
||||
)
|
||||
|
||||
|
||||
class AgentFeaturedRequest(BaseModel):
|
||||
)
|
||||
|
||||
|
||||
class AgentFeaturedRequest(BaseModel):
|
||||
"""Request body for setting/clearing an agent's homepage featured rank."""
|
||||
|
||||
order: int | None = Field(
|
||||
@@ -159,6 +160,41 @@ def _is_admin_user(request: Request) -> bool:
|
||||
return getattr(user, "system_role", None) == "admin"
|
||||
|
||||
|
||||
def _email_local_part(email: str | None) -> str | None:
|
||||
"""Return the username portion (before ``@``) of an email address."""
|
||||
if not email:
|
||||
return None
|
||||
at = email.find("@")
|
||||
return email[:at] if at > 0 else email
|
||||
|
||||
|
||||
async def _resolve_usernames(user_ids: set[str]) -> dict[str, str]:
|
||||
"""Batch-resolve owner user ids to friendly usernames (email local part).
|
||||
|
||||
One lookup per distinct id; tolerates missing accounts (deleted users) and
|
||||
an uninitialised auth provider by simply omitting those ids from the map.
|
||||
"""
|
||||
user_ids = {uid for uid in user_ids if uid}
|
||||
if not user_ids:
|
||||
return {}
|
||||
try:
|
||||
from app.gateway.deps import get_local_provider
|
||||
|
||||
provider = get_local_provider()
|
||||
except Exception:
|
||||
return {}
|
||||
names: dict[str, str] = {}
|
||||
for uid in user_ids:
|
||||
try:
|
||||
user = await provider.get_user(uid)
|
||||
except Exception:
|
||||
user = None
|
||||
local = _email_local_part(getattr(user, "email", None)) if user is not None else None
|
||||
if local:
|
||||
names[uid] = local
|
||||
return names
|
||||
|
||||
|
||||
def _validate_display_name(name: str) -> str:
|
||||
normalized = name.strip()
|
||||
if not normalized:
|
||||
@@ -348,7 +384,7 @@ def _ensure_agent_dir_from_record(record: dict[str, Any]) -> None:
|
||||
logger.exception("Failed to self-heal agent dir for '%s'", agent_id)
|
||||
|
||||
|
||||
def _record_to_response(
|
||||
def _record_to_response(
|
||||
record: dict[str, Any],
|
||||
*,
|
||||
include_soul: bool = False,
|
||||
@@ -390,21 +426,21 @@ def _record_to_response(
|
||||
tool_groups=extras["tool_groups"],
|
||||
skills=extras["skills"],
|
||||
featured_order=record.get("featured_order"),
|
||||
)
|
||||
|
||||
|
||||
def _sort_agent_responses(responses: list[AgentResponse], sort: str) -> list[AgentResponse]:
|
||||
reverse = sort not in {"time_asc", "updated_asc", "name_asc"}
|
||||
if sort in {"name_asc", "name_desc"}:
|
||||
ordered = sorted(responses, key=lambda r: (r.name or "").lower(), reverse=reverse)
|
||||
elif sort in {"updated_desc", "updated_asc"}:
|
||||
ordered = sorted(responses, key=lambda r: str(r.updated_at or ""), reverse=reverse)
|
||||
else:
|
||||
ordered = sorted(responses, key=lambda r: str(r.created_at or ""), reverse=reverse)
|
||||
return sorted(ordered, key=lambda r: 0 if r.is_pinned else 1)
|
||||
|
||||
|
||||
async def _record_to_response_async(
|
||||
)
|
||||
|
||||
|
||||
def _sort_agent_responses(responses: list[AgentResponse], sort: str) -> list[AgentResponse]:
|
||||
reverse = sort not in {"time_asc", "updated_asc", "name_asc"}
|
||||
if sort in {"name_asc", "name_desc"}:
|
||||
ordered = sorted(responses, key=lambda r: (r.name or "").lower(), reverse=reverse)
|
||||
elif sort in {"updated_desc", "updated_asc"}:
|
||||
ordered = sorted(responses, key=lambda r: str(r.updated_at or ""), reverse=reverse)
|
||||
else:
|
||||
ordered = sorted(responses, key=lambda r: str(r.created_at or ""), reverse=reverse)
|
||||
return sorted(ordered, key=lambda r: 0 if r.is_pinned else 1)
|
||||
|
||||
|
||||
async def _record_to_response_async(
|
||||
record: dict[str, Any],
|
||||
*,
|
||||
include_soul: bool = False,
|
||||
@@ -478,20 +514,23 @@ async def _build_list_responses(
|
||||
_record_to_response(a, include_soul=False, include_extras=False) for a in records
|
||||
]
|
||||
ids = [r.id for r in responses]
|
||||
extras = await store.get_extras_for(ids)
|
||||
assignments = await tag_store.list_assignments("agent", ids)
|
||||
favorite_ids = await store.list_favorite_ids(user_id)
|
||||
pin_ids = await store.list_pin_ids(user_id)
|
||||
for resp in responses:
|
||||
extras = await store.get_extras_for(ids)
|
||||
assignments = await tag_store.list_assignments("agent", ids)
|
||||
favorite_ids = await store.list_favorite_ids(user_id)
|
||||
pin_ids = await store.list_pin_ids(user_id)
|
||||
usernames = await _resolve_usernames({r.user_id for r in responses if r.user_id})
|
||||
for resp in responses:
|
||||
cached = extras.get(resp.id)
|
||||
if cached is not None:
|
||||
resp.skills = cached.get("skills")
|
||||
resp.tool_groups = cached.get("tool_groups")
|
||||
resp.model = cached.get("model")
|
||||
resp.tags = assignments.get(resp.id, [])
|
||||
resp.is_favorite = resp.id in favorite_ids
|
||||
resp.is_pinned = resp.id in pin_ids
|
||||
return responses
|
||||
resp.tags = assignments.get(resp.id, [])
|
||||
resp.is_favorite = resp.id in favorite_ids
|
||||
resp.is_pinned = resp.id in pin_ids
|
||||
if resp.user_id:
|
||||
resp.owner_username = usernames.get(resp.user_id)
|
||||
return responses
|
||||
|
||||
|
||||
@router.get(
|
||||
@@ -509,14 +548,18 @@ async def list_agents(
|
||||
pattern="^(mine|square)$",
|
||||
description="Pagination scope. 'mine' = owned + favorited; 'square' = built-in/published public. Omit for the default combined visibility list.",
|
||||
),
|
||||
page: int | None = Query(default=None, ge=1, description="1-based page index. Providing this (or 'scope') enables server-side pagination."),
|
||||
page_size: int = Query(default=20, ge=1, le=200, description="Items per page when paginating."),
|
||||
sort: str = Query(
|
||||
default="time_desc",
|
||||
pattern="^(time_desc|time_asc|updated_desc|updated_asc|name_asc|name_desc)$",
|
||||
description="Sort order. Default is newest created first.",
|
||||
),
|
||||
admin: bool = Query(
|
||||
page: int | None = Query(default=None, ge=1, description="1-based page index. Providing this (or 'scope') enables server-side pagination."),
|
||||
page_size: int = Query(default=20, ge=1, le=200, description="Items per page when paginating."),
|
||||
sort: str = Query(
|
||||
default="time_desc",
|
||||
pattern="^(time_desc|time_asc|updated_desc|updated_asc|name_asc|name_desc)$",
|
||||
description="Sort order. Default is newest created first.",
|
||||
),
|
||||
owner: str | None = Query(
|
||||
default=None,
|
||||
description="Keep only agents published by this user id (the card's publisher). Built-in agents never match.",
|
||||
),
|
||||
admin: bool = Query(
|
||||
default=False,
|
||||
description=(
|
||||
"Admin-only override. When ``true`` and the caller is a system admin, "
|
||||
@@ -543,11 +586,12 @@ async def list_agents(
|
||||
scope=scope,
|
||||
search=search,
|
||||
tag_ids=tag_ids,
|
||||
page=eff_page,
|
||||
page_size=page_size,
|
||||
admin=admin,
|
||||
sort=sort,
|
||||
)
|
||||
page=eff_page,
|
||||
page_size=page_size,
|
||||
admin=admin,
|
||||
sort=sort,
|
||||
owner=owner,
|
||||
)
|
||||
responses = await _build_list_responses(store, tag_store, user_id, rows)
|
||||
return AgentsListResponse(agents=responses, total=total, page=eff_page, page_size=page_size)
|
||||
|
||||
@@ -559,11 +603,13 @@ async def list_agents(
|
||||
if search:
|
||||
needle = search.strip().lower()
|
||||
responses = [r for r in responses if needle in r.name.lower()]
|
||||
if tag_ids:
|
||||
wanted = set(tag_ids)
|
||||
responses = [r for r in responses if any(t.get("id") in wanted for t in r.tags)]
|
||||
responses = _sort_agent_responses(responses, sort)
|
||||
return AgentsListResponse(agents=responses, total=len(responses), page=1, page_size=len(responses))
|
||||
if tag_ids:
|
||||
wanted = set(tag_ids)
|
||||
responses = [r for r in responses if any(t.get("id") in wanted for t in r.tags)]
|
||||
if owner and owner.strip():
|
||||
responses = [r for r in responses if r.user_id == owner.strip()]
|
||||
responses = _sort_agent_responses(responses, sort)
|
||||
return AgentsListResponse(agents=responses, total=len(responses), page=1, page_size=len(responses))
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
@@ -576,14 +622,14 @@ async def list_agents(
|
||||
summary="Check Agent Name",
|
||||
description="Validate an agent display name. Duplicate names are allowed.",
|
||||
)
|
||||
async def check_agent_name(name: str) -> dict[str, Any]:
|
||||
_require_agents_api_enabled()
|
||||
normalized = _validate_display_name(name)
|
||||
return {"available": True, "name": normalized}
|
||||
|
||||
|
||||
@router.get(
|
||||
"/agents/selector",
|
||||
async def check_agent_name(name: str) -> dict[str, Any]:
|
||||
_require_agents_api_enabled()
|
||||
normalized = _validate_display_name(name)
|
||||
return {"available": True, "name": normalized}
|
||||
|
||||
|
||||
@router.get(
|
||||
"/agents/selector",
|
||||
response_model=AgentSelectorResponse,
|
||||
summary="Agent Homepage Selector",
|
||||
description=(
|
||||
@@ -635,12 +681,12 @@ async def get_agent(agent_id: str, request: Request) -> AgentResponse:
|
||||
# Detail view legitimately reads SOUL.md + config.yaml — hop to a
|
||||
# worker thread so the asyncio loop stays responsive under concurrent
|
||||
# detail fetches.
|
||||
response = await _record_to_response_async(record, include_soul=True, include_extras=True)
|
||||
favorite_ids = await store.list_favorite_ids(user_id)
|
||||
pin_ids = await store.list_pin_ids(user_id)
|
||||
response.is_favorite = agent_id in favorite_ids
|
||||
response.is_pinned = agent_id in pin_ids
|
||||
return response
|
||||
response = await _record_to_response_async(record, include_soul=True, include_extras=True)
|
||||
favorite_ids = await store.list_favorite_ids(user_id)
|
||||
pin_ids = await store.list_pin_ids(user_id)
|
||||
response.is_favorite = agent_id in favorite_ids
|
||||
response.is_pinned = agent_id in pin_ids
|
||||
return response
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
@@ -874,16 +920,16 @@ async def delete_agent(agent_id: str, request: Request) -> None:
|
||||
await get_tag_store(request).unassign_all("agent", agent_id)
|
||||
except Exception: # noqa: BLE001 — tag cleanup must not block deletion
|
||||
pass
|
||||
try:
|
||||
await store.delete_favorites_for_agent(agent_id)
|
||||
except Exception: # noqa: BLE001 — favorite cleanup must not block deletion
|
||||
pass
|
||||
try:
|
||||
await store.delete_pins_for_agent(agent_id)
|
||||
except Exception: # noqa: BLE001 - pin cleanup must not block deletion
|
||||
pass
|
||||
try:
|
||||
await store.delete_extras_for_agent(agent_id)
|
||||
try:
|
||||
await store.delete_favorites_for_agent(agent_id)
|
||||
except Exception: # noqa: BLE001 — favorite cleanup must not block deletion
|
||||
pass
|
||||
try:
|
||||
await store.delete_pins_for_agent(agent_id)
|
||||
except Exception: # noqa: BLE001 - pin cleanup must not block deletion
|
||||
pass
|
||||
try:
|
||||
await store.delete_extras_for_agent(agent_id)
|
||||
except Exception: # noqa: BLE001 — cache cleanup must not block deletion
|
||||
pass
|
||||
logger.info("Deleted agent '%s' from %s", agent_id, agent_dir)
|
||||
@@ -925,7 +971,7 @@ async def favorite_agent(agent_id: str, request: Request) -> None:
|
||||
summary="Unfavorite Agent",
|
||||
description="Remove an agent from the current user's favorites. No-op when not favorited.",
|
||||
)
|
||||
async def unfavorite_agent(agent_id: str, request: Request) -> None:
|
||||
async def unfavorite_agent(agent_id: str, request: Request) -> None:
|
||||
_require_agents_api_enabled()
|
||||
agent_id = _validate_storage_id(agent_id)
|
||||
|
||||
@@ -936,60 +982,60 @@ async def unfavorite_agent(agent_id: str, request: Request) -> None:
|
||||
logger.info("User '%s' unfavorited agent '%s'", user_id, agent_id)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error("Failed to unfavorite agent '%s': %s", agent_id, e, exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to unfavorite agent: {str(e)}") from e
|
||||
|
||||
|
||||
@router.post(
|
||||
"/agents/{agent_id}/pin",
|
||||
status_code=204,
|
||||
summary="Pin Agent",
|
||||
description="Pin a visible agent to the top of the current user's lists. Idempotent.",
|
||||
)
|
||||
async def pin_agent(agent_id: str, request: Request) -> None:
|
||||
_require_agents_api_enabled()
|
||||
agent_id = _validate_storage_id(agent_id)
|
||||
|
||||
try:
|
||||
store = get_agent_store(request)
|
||||
user_id = _current_user_id(request)
|
||||
record = await store.get_visible(agent_id, user_id)
|
||||
if record is None:
|
||||
raise HTTPException(status_code=404, detail=f"Agent '{agent_id}' not found")
|
||||
await store.add_pin(user_id, agent_id)
|
||||
logger.info("User '%s' pinned agent '%s'", user_id, agent_id)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error("Failed to pin agent '%s': %s", agent_id, e, exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to pin agent: {str(e)}") from e
|
||||
|
||||
|
||||
@router.delete(
|
||||
"/agents/{agent_id}/pin",
|
||||
status_code=204,
|
||||
summary="Unpin Agent",
|
||||
description="Remove the current user's pin from an agent. No-op when not pinned.",
|
||||
)
|
||||
async def unpin_agent(agent_id: str, request: Request) -> None:
|
||||
_require_agents_api_enabled()
|
||||
agent_id = _validate_storage_id(agent_id)
|
||||
|
||||
try:
|
||||
store = get_agent_store(request)
|
||||
user_id = _current_user_id(request)
|
||||
await store.remove_pin(user_id, agent_id)
|
||||
logger.info("User '%s' unpinned agent '%s'", user_id, agent_id)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error("Failed to unpin agent '%s': %s", agent_id, e, exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to unpin agent: {str(e)}") from e
|
||||
|
||||
|
||||
@router.put(
|
||||
"/agents/{agent_id}/featured",
|
||||
except Exception as e:
|
||||
logger.error("Failed to unfavorite agent '%s': %s", agent_id, e, exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to unfavorite agent: {str(e)}") from e
|
||||
|
||||
|
||||
@router.post(
|
||||
"/agents/{agent_id}/pin",
|
||||
status_code=204,
|
||||
summary="Pin Agent",
|
||||
description="Pin a visible agent to the top of the current user's lists. Idempotent.",
|
||||
)
|
||||
async def pin_agent(agent_id: str, request: Request) -> None:
|
||||
_require_agents_api_enabled()
|
||||
agent_id = _validate_storage_id(agent_id)
|
||||
|
||||
try:
|
||||
store = get_agent_store(request)
|
||||
user_id = _current_user_id(request)
|
||||
record = await store.get_visible(agent_id, user_id)
|
||||
if record is None:
|
||||
raise HTTPException(status_code=404, detail=f"Agent '{agent_id}' not found")
|
||||
await store.add_pin(user_id, agent_id)
|
||||
logger.info("User '%s' pinned agent '%s'", user_id, agent_id)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error("Failed to pin agent '%s': %s", agent_id, e, exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to pin agent: {str(e)}") from e
|
||||
|
||||
|
||||
@router.delete(
|
||||
"/agents/{agent_id}/pin",
|
||||
status_code=204,
|
||||
summary="Unpin Agent",
|
||||
description="Remove the current user's pin from an agent. No-op when not pinned.",
|
||||
)
|
||||
async def unpin_agent(agent_id: str, request: Request) -> None:
|
||||
_require_agents_api_enabled()
|
||||
agent_id = _validate_storage_id(agent_id)
|
||||
|
||||
try:
|
||||
store = get_agent_store(request)
|
||||
user_id = _current_user_id(request)
|
||||
await store.remove_pin(user_id, agent_id)
|
||||
logger.info("User '%s' unpinned agent '%s'", user_id, agent_id)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error("Failed to unpin agent '%s': %s", agent_id, e, exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to unpin agent: {str(e)}") from e
|
||||
|
||||
|
||||
@router.put(
|
||||
"/agents/{agent_id}/featured",
|
||||
response_model=AgentResponse,
|
||||
summary="Set Agent Featured Rank (admin)",
|
||||
description=(
|
||||
@@ -1031,19 +1077,20 @@ async def set_agent_featured(
|
||||
raise HTTPException(status_code=500, detail=f"Failed to set featured: {str(e)}") from e
|
||||
|
||||
|
||||
class PrivateSquareListRequest(BaseModel):
|
||||
class PrivateSquareListRequest(BaseModel):
|
||||
"""Request body for listing all private-square agents."""
|
||||
|
||||
password: str = Field(..., description="Plain-text private-square password. Verified against the admin-configured hash before the listing is returned.")
|
||||
search: str | None = Field(default=None, description="Case-insensitive fuzzy match on agent name (raw or desensitized).")
|
||||
tag_ids: list[str] | None = Field(default=None, description="Keep agents carrying any of these tags (OR).")
|
||||
page: int | None = Field(default=None, ge=1, description="1-based page index. Providing this enables server-side pagination.")
|
||||
page_size: int = Field(default=20, ge=1, le=200, description="Items per page when paginating.")
|
||||
sort: str = Field(
|
||||
default="time_desc",
|
||||
pattern="^(time_desc|time_asc|updated_desc|updated_asc|name_asc|name_desc)$",
|
||||
description="Sort order. Default is newest created first.",
|
||||
)
|
||||
page: int | None = Field(default=None, ge=1, description="1-based page index. Providing this enables server-side pagination.")
|
||||
page_size: int = Field(default=20, ge=1, le=200, description="Items per page when paginating.")
|
||||
sort: str = Field(
|
||||
default="time_desc",
|
||||
pattern="^(time_desc|time_asc|updated_desc|updated_asc|name_asc|name_desc)$",
|
||||
description="Sort order. Default is newest created first.",
|
||||
)
|
||||
owner: str | None = Field(default=None, description="Keep only agents published by this user id (the card's publisher).")
|
||||
|
||||
|
||||
class AgentPrivateUpdateRequest(BaseModel):
|
||||
@@ -1086,17 +1133,18 @@ async def list_private_square(body: PrivateSquareListRequest, request: Request)
|
||||
scope="private",
|
||||
search=body.search,
|
||||
tag_ids=body.tag_ids,
|
||||
page=body.page,
|
||||
page_size=body.page_size,
|
||||
sort=body.sort,
|
||||
)
|
||||
page=body.page,
|
||||
page_size=body.page_size,
|
||||
sort=body.sort,
|
||||
owner=body.owner,
|
||||
)
|
||||
responses = await _build_list_responses(store, tag_store, user_id, rows)
|
||||
return AgentsListResponse(agents=responses, total=total, page=body.page, page_size=body.page_size)
|
||||
|
||||
records = await store.list_private()
|
||||
responses = await _build_list_responses(store, tag_store, user_id, records)
|
||||
responses = _sort_agent_responses(responses, body.sort)
|
||||
return AgentsListResponse(agents=responses, total=len(responses), page=1, page_size=len(responses))
|
||||
responses = await _build_list_responses(store, tag_store, user_id, records)
|
||||
responses = _sort_agent_responses(responses, body.sort)
|
||||
return AgentsListResponse(agents=responses, total=len(responses), page=1, page_size=len(responses))
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
|
||||
@@ -0,0 +1,590 @@
|
||||
"""Browser page context capture endpoints.
|
||||
|
||||
This router is intentionally isolated behind ``config.yaml``:
|
||||
|
||||
browser_context:
|
||||
enabled: true
|
||||
|
||||
When disabled, mutating endpoints behave as if the integration is unavailable.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import re
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any, Literal
|
||||
from urllib.parse import urlencode
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.gateway.authz import require_permission
|
||||
from app.gateway.deps import get_checkpointer, get_config, get_thread_store
|
||||
from app.gateway.routers.thread_runs import RunCreateRequest
|
||||
from app.gateway.routers.threads import ThreadCreateRequest, create_thread
|
||||
from app.gateway.routers.uploads import _make_file_sandbox_writable
|
||||
from app.gateway.services import start_run
|
||||
from deerflow.browser_context.tasks import claim_next_browser_task, complete_browser_task, prune_browser_tasks
|
||||
from deerflow.config.app_config import AppConfig
|
||||
from deerflow.config.browser_context_config import BrowserContextConfig
|
||||
from deerflow.runtime import serialize_channel_values
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
from deerflow.sandbox.sandbox_provider import get_sandbox_provider
|
||||
from deerflow.uploads.manager import (
|
||||
UnsafeUploadPathError,
|
||||
ensure_uploads_dir,
|
||||
normalize_filename,
|
||||
open_upload_file_no_symlink,
|
||||
upload_artifact_url,
|
||||
upload_virtual_path,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/browser-context", tags=["browser-context"])
|
||||
DEFAULT_BROWSER_CONTEXT_PROMPT = "\u8bf7\u57fa\u4e8e\u521a\u521a\u53d1\u9001\u7684\u7f51\u9875\u5185\u5bb9\uff0c\u5148\u7528\u4e2d\u6587\u603b\u7ed3\u8981\u70b9\uff0c\u5e76\u7b49\u5f85\u6211\u7ee7\u7eed\u63d0\u95ee\u3002"
|
||||
|
||||
|
||||
class BrowserContextStatusResponse(BaseModel):
|
||||
enabled: bool
|
||||
max_content_chars: int
|
||||
max_title_chars: int
|
||||
max_url_chars: int
|
||||
task_result_timeout_seconds: int
|
||||
task_poll_interval_seconds: float
|
||||
task_lease_seconds: int
|
||||
tool_result_max_chars: int
|
||||
|
||||
|
||||
class BrowserContextCaptureRequest(BaseModel):
|
||||
thread_id: str | None = Field(default=None, description="Existing or caller-generated zncm thread id")
|
||||
title: str | None = Field(default=None, description="Current browser page title")
|
||||
url: str | None = Field(default=None, description="Current browser page URL")
|
||||
content: str = Field(description="Captured page content, preferably Markdown")
|
||||
content_type: Literal["markdown", "text"] = Field(default="markdown")
|
||||
prompt: str | None = Field(default=None, description="Optional first question to ask in zncm after capture")
|
||||
|
||||
|
||||
class BrowserContextFileInfo(BaseModel):
|
||||
filename: str
|
||||
size: int
|
||||
path: str
|
||||
virtual_path: str
|
||||
artifact_url: str
|
||||
|
||||
|
||||
class BrowserContextCaptureResponse(BaseModel):
|
||||
success: bool
|
||||
thread_id: str
|
||||
file: BrowserContextFileInfo
|
||||
chat_path: str
|
||||
chat_url: str
|
||||
browser_context_token: str
|
||||
prompt: str
|
||||
truncated: bool = False
|
||||
message: str
|
||||
|
||||
|
||||
class BrowserContextAskRequest(BaseModel):
|
||||
thread_id: str | None = Field(default=None, description="Existing zncm thread id for follow-up questions")
|
||||
title: str | None = Field(default=None, description="Current browser page title")
|
||||
url: str | None = Field(default=None, description="Current browser page URL")
|
||||
content: str = Field(description="Captured page content, preferably Markdown")
|
||||
content_type: Literal["markdown", "text"] = Field(default="markdown")
|
||||
question: str = Field(description="Question asked from the in-page assistant")
|
||||
|
||||
|
||||
class BrowserContextAskResponse(BaseModel):
|
||||
success: bool
|
||||
thread_id: str
|
||||
run_id: str
|
||||
answer: str
|
||||
truncated: bool = False
|
||||
message: str
|
||||
|
||||
|
||||
class BrowserContextTaskPayload(BaseModel):
|
||||
id: str
|
||||
kind: str = "fetch"
|
||||
url: str
|
||||
instruction: str = ""
|
||||
session_id: str = ""
|
||||
action: dict[str, Any] | None = None
|
||||
created_at: str
|
||||
updated_at: str
|
||||
max_content_chars: int
|
||||
|
||||
|
||||
class BrowserContextTaskNextResponse(BaseModel):
|
||||
task: BrowserContextTaskPayload | None = None
|
||||
|
||||
|
||||
class BrowserContextTaskResultRequest(BaseModel):
|
||||
kind: str = Field(default="fetch", description="Task kind the extension processed: fetch or act")
|
||||
title: str | None = Field(default=None, description="Captured page title")
|
||||
url: str | None = Field(default=None, description="Final page URL after redirects")
|
||||
content: str | None = Field(default=None, description="Captured page content")
|
||||
content_type: Literal["markdown", "text"] = Field(default="markdown")
|
||||
observation: dict[str, Any] | None = Field(default=None, description="Structured observation for an act step (elements, text, status)")
|
||||
error: str | None = Field(default=None, description="Extension-side task error")
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class BrowserContextTaskResultResponse(BaseModel):
|
||||
success: bool
|
||||
task_id: str
|
||||
status: str
|
||||
truncated: bool = False
|
||||
|
||||
|
||||
def _settings(config: AppConfig) -> BrowserContextConfig:
|
||||
value = getattr(config, "browser_context", None)
|
||||
if isinstance(value, BrowserContextConfig):
|
||||
return value
|
||||
if isinstance(value, dict):
|
||||
return BrowserContextConfig.model_validate(value)
|
||||
return BrowserContextConfig()
|
||||
|
||||
|
||||
def _require_enabled(config: AppConfig) -> BrowserContextConfig:
|
||||
settings = _settings(config)
|
||||
if not settings.enabled:
|
||||
raise HTTPException(status_code=404, detail="Browser context integration is disabled")
|
||||
return settings
|
||||
|
||||
|
||||
def _trim(value: str | None, limit: int) -> str:
|
||||
if not value:
|
||||
return ""
|
||||
return value.strip()[:limit]
|
||||
|
||||
|
||||
def _limit_utf8_bytes(value: str, max_bytes: int) -> str:
|
||||
text = value
|
||||
while len(text.encode("utf-8")) > max_bytes:
|
||||
text = text[:-1]
|
||||
return text
|
||||
|
||||
|
||||
def _filename_part(value: str, fallback: str, *, max_bytes: int) -> str:
|
||||
text = value.strip() or fallback
|
||||
# Keep unicode word chars so Chinese titles remain readable, while removing
|
||||
# path separators and shell-hostile punctuation.
|
||||
text = re.sub(r"[^\w.-]+", "-", text, flags=re.UNICODE).strip("-._")
|
||||
text = _limit_utf8_bytes(text, max_bytes)
|
||||
return text or fallback
|
||||
|
||||
|
||||
def _build_markdown(
|
||||
*,
|
||||
title: str,
|
||||
url: str,
|
||||
content: str,
|
||||
content_type: str,
|
||||
captured_at: str,
|
||||
truncated: bool,
|
||||
) -> str:
|
||||
lines = [
|
||||
f"# {title or 'Captured Web Page'}",
|
||||
"",
|
||||
f"- Source URL: {url or '(unknown)'}",
|
||||
f"- Captured at: {captured_at}",
|
||||
]
|
||||
if truncated:
|
||||
lines.append("- Note: Content was truncated by the browser_context.max_content_chars limit.")
|
||||
lines.extend(["", "---", ""])
|
||||
|
||||
if content_type == "markdown":
|
||||
lines.append(content.strip())
|
||||
else:
|
||||
lines.append("```text")
|
||||
lines.append(content.strip())
|
||||
lines.append("```")
|
||||
lines.append("")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _build_ask_prompt(
|
||||
*,
|
||||
title: str,
|
||||
url: str,
|
||||
content: str,
|
||||
content_type: str,
|
||||
captured_at: str,
|
||||
truncated: bool,
|
||||
) -> str:
|
||||
note = "页面内容已按配置截断,请只基于可见内容回答。" if truncated else "页面内容如下。"
|
||||
body = content.strip()
|
||||
if content_type == "text":
|
||||
body = f"```text\n{body}\n```"
|
||||
return "\n\n".join(
|
||||
[
|
||||
"你是 zncm 页面问答助手。请直接基于当前网页内容回答用户问题。",
|
||||
"如果网页内容不足以回答,请说明缺少哪些信息;不要编造,也不要再尝试抓取其它网页。",
|
||||
f"页面标题:{title or '未知标题'}",
|
||||
f"页面 URL:{url or '未知 URL'}",
|
||||
f"读取时间:{captured_at}",
|
||||
f"说明:{note}",
|
||||
"当前网页内容:",
|
||||
body,
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def _message_type(message: Any) -> str:
|
||||
if isinstance(message, dict):
|
||||
return str(message.get("type") or message.get("role") or "").lower()
|
||||
return str(getattr(message, "type", "") or getattr(message, "role", "") or "").lower()
|
||||
|
||||
|
||||
def _content_to_text(content: Any) -> str:
|
||||
if isinstance(content, str):
|
||||
return content.strip()
|
||||
if isinstance(content, list):
|
||||
parts: list[str] = []
|
||||
for item in content:
|
||||
if isinstance(item, str):
|
||||
parts.append(item)
|
||||
elif isinstance(item, dict):
|
||||
if isinstance(item.get("text"), str):
|
||||
parts.append(item["text"])
|
||||
elif item.get("type") == "text" and isinstance(item.get("content"), str):
|
||||
parts.append(item["content"])
|
||||
return "\n".join(part.strip() for part in parts if part and part.strip()).strip()
|
||||
return ""
|
||||
|
||||
|
||||
def _latest_ai_answer(values: dict[str, Any]) -> str:
|
||||
messages = values.get("messages")
|
||||
if not isinstance(messages, list):
|
||||
return ""
|
||||
for message in reversed(messages):
|
||||
if _message_type(message) not in {"ai", "assistant"}:
|
||||
continue
|
||||
answer = _content_to_text(message.get("content") if isinstance(message, dict) else getattr(message, "content", None))
|
||||
if answer:
|
||||
return answer
|
||||
return ""
|
||||
|
||||
|
||||
async def _sync_to_sandbox_if_needed(thread_id: str, virtual_path: str, file_path, data: bytes) -> None:
|
||||
sandbox_provider = get_sandbox_provider()
|
||||
if bool(getattr(sandbox_provider, "uses_thread_data_mounts", False)):
|
||||
return
|
||||
sandbox_id = sandbox_provider.acquire(thread_id)
|
||||
sandbox = sandbox_provider.get(sandbox_id)
|
||||
if sandbox is None:
|
||||
raise HTTPException(status_code=500, detail="Failed to acquire sandbox")
|
||||
_make_file_sandbox_writable(file_path)
|
||||
sandbox.update_file(virtual_path, data)
|
||||
|
||||
|
||||
@router.get("/status", response_model=BrowserContextStatusResponse)
|
||||
async def get_browser_context_status(
|
||||
config: AppConfig = Depends(get_config),
|
||||
) -> BrowserContextStatusResponse:
|
||||
settings = _settings(config)
|
||||
return BrowserContextStatusResponse(
|
||||
enabled=settings.enabled,
|
||||
max_content_chars=settings.max_content_chars,
|
||||
max_title_chars=settings.max_title_chars,
|
||||
max_url_chars=settings.max_url_chars,
|
||||
task_result_timeout_seconds=settings.task_result_timeout_seconds,
|
||||
task_poll_interval_seconds=settings.task_poll_interval_seconds,
|
||||
task_lease_seconds=settings.task_lease_seconds,
|
||||
tool_result_max_chars=settings.tool_result_max_chars,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/tasks/next", response_model=BrowserContextTaskNextResponse)
|
||||
@require_permission("runs", "create")
|
||||
async def claim_browser_context_task(
|
||||
request: Request,
|
||||
worker_id: str | None = None,
|
||||
config: AppConfig = Depends(get_config),
|
||||
) -> BrowserContextTaskNextResponse:
|
||||
settings = _require_enabled(config)
|
||||
prune_browser_tasks()
|
||||
task = claim_next_browser_task(
|
||||
user_id=get_effective_user_id(),
|
||||
worker_id=worker_id or "chrome-extension",
|
||||
lease_seconds=settings.task_lease_seconds,
|
||||
)
|
||||
if task is None:
|
||||
return BrowserContextTaskNextResponse(task=None)
|
||||
return BrowserContextTaskNextResponse(task=BrowserContextTaskPayload.model_validate(task))
|
||||
|
||||
|
||||
@router.post("/tasks/{task_id}/result", response_model=BrowserContextTaskResultResponse)
|
||||
@require_permission("runs", "create")
|
||||
async def submit_browser_context_task_result(
|
||||
task_id: str,
|
||||
body: BrowserContextTaskResultRequest,
|
||||
request: Request,
|
||||
config: AppConfig = Depends(get_config),
|
||||
) -> BrowserContextTaskResultResponse:
|
||||
settings = _require_enabled(config)
|
||||
error = _trim(body.error, 4000)
|
||||
content = body.content or ""
|
||||
is_act = (body.kind or "fetch") == "act"
|
||||
if not error and not is_act and not content.strip():
|
||||
raise HTTPException(status_code=400, detail="Captured page content is empty")
|
||||
if not error and is_act and not isinstance(body.observation, dict):
|
||||
raise HTTPException(status_code=400, detail="Act task result requires an observation")
|
||||
|
||||
task = complete_browser_task(
|
||||
task_id=task_id,
|
||||
user_id=get_effective_user_id(),
|
||||
title=_trim(body.title, settings.max_title_chars),
|
||||
url=_trim(body.url, settings.max_url_chars),
|
||||
content=content,
|
||||
content_type=body.content_type,
|
||||
error=error,
|
||||
metadata=body.metadata,
|
||||
observation=body.observation,
|
||||
)
|
||||
if task is None:
|
||||
raise HTTPException(status_code=404, detail=f"Browser task {task_id} not found")
|
||||
|
||||
result = task.get("result") or {}
|
||||
return BrowserContextTaskResultResponse(
|
||||
success=True,
|
||||
task_id=task_id,
|
||||
status=str(task.get("status") or "unknown"),
|
||||
truncated=bool(result.get("truncated")),
|
||||
)
|
||||
|
||||
|
||||
@router.post("/capture", response_model=BrowserContextCaptureResponse)
|
||||
@require_permission("threads", "write")
|
||||
async def capture_browser_context(
|
||||
body: BrowserContextCaptureRequest,
|
||||
request: Request,
|
||||
config: AppConfig = Depends(get_config),
|
||||
) -> BrowserContextCaptureResponse:
|
||||
settings = _require_enabled(config)
|
||||
|
||||
raw_content = (body.content or "").strip()
|
||||
if not raw_content:
|
||||
raise HTTPException(status_code=400, detail="Captured page content is empty")
|
||||
|
||||
title = _trim(body.title, settings.max_title_chars)
|
||||
url = _trim(body.url, settings.max_url_chars)
|
||||
truncated = len(raw_content) > settings.max_content_chars
|
||||
content = raw_content[: settings.max_content_chars] if truncated else raw_content
|
||||
|
||||
requested_thread_id = (body.thread_id or "").strip() or None
|
||||
if requested_thread_id:
|
||||
thread_store = get_thread_store(request)
|
||||
allowed = await thread_store.check_access(
|
||||
requested_thread_id,
|
||||
get_effective_user_id(),
|
||||
require_existing=False,
|
||||
)
|
||||
if not allowed:
|
||||
raise HTTPException(status_code=404, detail=f"Thread {requested_thread_id} not found")
|
||||
|
||||
thread_response = await create_thread(
|
||||
ThreadCreateRequest(
|
||||
thread_id=requested_thread_id,
|
||||
metadata={
|
||||
"source": "browser_context",
|
||||
"browser_context_url": url,
|
||||
"browser_context_title": title,
|
||||
},
|
||||
),
|
||||
request,
|
||||
)
|
||||
thread_id = thread_response.thread_id
|
||||
|
||||
captured_at = datetime.now(UTC).isoformat()
|
||||
markdown = _build_markdown(
|
||||
title=title,
|
||||
url=url,
|
||||
content=content,
|
||||
content_type=body.content_type,
|
||||
captured_at=captured_at,
|
||||
truncated=truncated,
|
||||
)
|
||||
data = markdown.encode("utf-8")
|
||||
|
||||
timestamp = datetime.now(UTC).strftime("%Y%m%d-%H%M%S")
|
||||
prefix = _filename_part(settings.filename_prefix, "webpage", max_bytes=40)
|
||||
title_part = _filename_part(title, "page", max_bytes=120)
|
||||
filename = normalize_filename(f"{prefix}-{timestamp}-{title_part}.md")
|
||||
|
||||
try:
|
||||
uploads_dir = ensure_uploads_dir(thread_id)
|
||||
file_path, fh = open_upload_file_no_symlink(uploads_dir, filename)
|
||||
with fh:
|
||||
fh.write(data)
|
||||
except UnsafeUploadPathError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
except Exception as exc:
|
||||
logger.exception("Failed to save browser context capture for thread %s", thread_id)
|
||||
raise HTTPException(status_code=500, detail="Failed to save captured page") from exc
|
||||
|
||||
virtual_path = upload_virtual_path(filename)
|
||||
await _sync_to_sandbox_if_needed(thread_id, virtual_path, file_path, data)
|
||||
|
||||
prompt = (body.prompt or "").strip() or DEFAULT_BROWSER_CONTEXT_PROMPT
|
||||
token = uuid4().hex
|
||||
route_base = settings.route_base.rstrip("/") or "/page/workspace"
|
||||
chat_path = f"{route_base}/chats/{thread_id}"
|
||||
query = urlencode(
|
||||
{
|
||||
"browser_context": token,
|
||||
"bc_filename": filename,
|
||||
"bc_size": str(len(data)),
|
||||
"bc_path": virtual_path,
|
||||
"bc_prompt": prompt,
|
||||
}
|
||||
)
|
||||
chat_url = f"#{chat_path}?{query}"
|
||||
|
||||
return BrowserContextCaptureResponse(
|
||||
success=True,
|
||||
thread_id=thread_id,
|
||||
file=BrowserContextFileInfo(
|
||||
filename=filename,
|
||||
size=len(data),
|
||||
path=virtual_path,
|
||||
virtual_path=virtual_path,
|
||||
artifact_url=upload_artifact_url(thread_id, filename),
|
||||
),
|
||||
chat_path=chat_path,
|
||||
chat_url=chat_url,
|
||||
browser_context_token=token,
|
||||
prompt=prompt,
|
||||
truncated=truncated,
|
||||
message="Captured page saved as a zncm upload",
|
||||
)
|
||||
|
||||
|
||||
@router.post("/ask", response_model=BrowserContextAskResponse)
|
||||
@require_permission("runs", "create")
|
||||
async def ask_browser_context(
|
||||
body: BrowserContextAskRequest,
|
||||
request: Request,
|
||||
config: AppConfig = Depends(get_config),
|
||||
) -> BrowserContextAskResponse:
|
||||
settings = _require_enabled(config)
|
||||
|
||||
raw_content = (body.content or "").strip()
|
||||
if not raw_content:
|
||||
raise HTTPException(status_code=400, detail="Captured page content is empty")
|
||||
|
||||
question = (body.question or "").strip()
|
||||
if not question:
|
||||
raise HTTPException(status_code=400, detail="Question is empty")
|
||||
|
||||
title = _trim(body.title, settings.max_title_chars)
|
||||
url = _trim(body.url, settings.max_url_chars)
|
||||
content_limit = max(1_000, min(settings.max_content_chars, settings.tool_result_max_chars))
|
||||
truncated = len(raw_content) > content_limit
|
||||
content = raw_content[:content_limit] if truncated else raw_content
|
||||
|
||||
requested_thread_id = (body.thread_id or "").strip() or None
|
||||
if requested_thread_id:
|
||||
thread_store = get_thread_store(request)
|
||||
allowed = await thread_store.check_access(
|
||||
requested_thread_id,
|
||||
get_effective_user_id(),
|
||||
require_existing=False,
|
||||
)
|
||||
if not allowed:
|
||||
raise HTTPException(status_code=404, detail=f"Thread {requested_thread_id} not found")
|
||||
|
||||
thread_response = await create_thread(
|
||||
ThreadCreateRequest(
|
||||
thread_id=requested_thread_id,
|
||||
assistant_id="lead_agent",
|
||||
metadata={
|
||||
"source": "browser_context_page_assistant",
|
||||
"browser_context_url": url,
|
||||
"browser_context_title": title,
|
||||
},
|
||||
),
|
||||
request,
|
||||
)
|
||||
thread_id = thread_response.thread_id
|
||||
captured_at = datetime.now(UTC).isoformat()
|
||||
is_follow_up = requested_thread_id is not None
|
||||
if is_follow_up:
|
||||
prompt_prefix = "\n\n".join(
|
||||
[
|
||||
"你是 zncm 页面问答助手。请继续基于本会话里已经读取的网页内容回答用户问题。",
|
||||
"如果本会话里的网页内容不足以回答,请说明缺少哪些信息;不要编造,也不要再尝试抓取其它网页。",
|
||||
f"当前页面标题:{title or '未知标题'}",
|
||||
f"当前页面 URL:{url or '未知 URL'}",
|
||||
f"本轮提问时间:{captured_at}",
|
||||
]
|
||||
)
|
||||
else:
|
||||
prompt_prefix = _build_ask_prompt(
|
||||
title=title,
|
||||
url=url,
|
||||
content=content,
|
||||
content_type=body.content_type,
|
||||
captured_at=captured_at,
|
||||
truncated=truncated,
|
||||
)
|
||||
|
||||
run_body = RunCreateRequest(
|
||||
assistant_id="lead_agent",
|
||||
input={
|
||||
"messages": [
|
||||
{
|
||||
"type": "human",
|
||||
"content": question,
|
||||
"additional_kwargs": {
|
||||
"prompt_prefix": prompt_prefix,
|
||||
},
|
||||
}
|
||||
]
|
||||
},
|
||||
metadata={
|
||||
"source": "browser_context_page_assistant",
|
||||
"browser_context_url": url,
|
||||
"browser_context_title": title,
|
||||
"browser_context_truncated": truncated,
|
||||
},
|
||||
stream_mode=["values"],
|
||||
on_disconnect="continue",
|
||||
multitask_strategy="reject",
|
||||
)
|
||||
|
||||
record = await start_run(run_body, thread_id, request)
|
||||
if record.task is not None:
|
||||
try:
|
||||
await record.task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
|
||||
checkpointer = get_checkpointer(request)
|
||||
values: dict[str, Any] = {}
|
||||
try:
|
||||
checkpoint_tuple = await checkpointer.aget_tuple({"configurable": {"thread_id": thread_id}})
|
||||
if checkpoint_tuple is not None:
|
||||
checkpoint = getattr(checkpoint_tuple, "checkpoint", {}) or {}
|
||||
values = serialize_channel_values(checkpoint.get("channel_values", {}) or {})
|
||||
except Exception:
|
||||
logger.exception("Failed to fetch browser context answer for run %s", record.run_id)
|
||||
|
||||
answer = _latest_ai_answer(values)
|
||||
if not answer:
|
||||
answer = record.error or "zncm did not return an answer."
|
||||
|
||||
return BrowserContextAskResponse(
|
||||
success=bool(answer),
|
||||
thread_id=thread_id,
|
||||
run_id=record.run_id,
|
||||
answer=answer,
|
||||
truncated=truncated,
|
||||
message="zncm page assistant answer completed",
|
||||
)
|
||||
@@ -0,0 +1,521 @@
|
||||
"""Knowledge base API (``/api/knowledge``).
|
||||
|
||||
Sediments threads / searches into an obsidian-wiki compatible vault, mirrors
|
||||
metadata into the database, and serves listing, detail, edit, delete, search
|
||||
and graph export. The knowledge base is **global / shared** — no per-user
|
||||
isolation; writes record ``created_by`` / ``updated_by`` for audit.
|
||||
|
||||
All obsidian-wiki specifics live in :mod:`deerflow.knowledge`; this router only
|
||||
adapts HTTP ↔ the :class:`~deerflow.knowledge.service.KnowledgeService`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import mimetypes
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, BackgroundTasks, HTTPException, Request
|
||||
from fastapi.responses import FileResponse
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.gateway.authz import require_auth, require_permission
|
||||
from app.gateway.deps import get_checkpointer, get_knowledge_service, get_local_provider, get_thread_store
|
||||
from app.gateway.utils import sanitize_log_param
|
||||
from deerflow.knowledge.service import EmptyThreadError
|
||||
from deerflow.runtime import serialize_channel_values
|
||||
from deerflow.runtime.references import build_reference_batches
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/api/knowledge", tags=["knowledge"])
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Request / response models
|
||||
# --------------------------------------------------------------------------- #
|
||||
class IngestThreadRequest(BaseModel):
|
||||
mode: str = Field(default="summary", description="summary | full")
|
||||
title: str | None = None
|
||||
status: str = Field(default="approved", description="draft | approved")
|
||||
tags: list[str] = Field(default_factory=list)
|
||||
include_sources: bool = True
|
||||
background: bool = Field(
|
||||
default=False,
|
||||
description="When true, sediment the thread asynchronously in a background task and return immediately.",
|
||||
)
|
||||
model_name: str | None = Field(default=None, description="可选:指定用于知识提炼的大模型名称;留空使用默认配置")
|
||||
|
||||
|
||||
class BatchIngestRequest(BaseModel):
|
||||
limit: int = Field(default=100, ge=1, le=500)
|
||||
offset: int = Field(default=0, ge=0)
|
||||
status: str = "approved"
|
||||
include_sources: bool = True
|
||||
skip_existing: bool = True
|
||||
mode: str = "summary"
|
||||
model_name: str | None = Field(default=None, description="可选:指定用于知识提炼的大模型名称;留空使用默认配置")
|
||||
|
||||
|
||||
class SearchResultItem(BaseModel):
|
||||
title: str | None = None
|
||||
url: str | None = None
|
||||
snippet: str | None = None
|
||||
tool_name: str | None = None
|
||||
score: float | None = None
|
||||
|
||||
|
||||
class IngestSearchRequest(BaseModel):
|
||||
query: str
|
||||
results: list[SearchResultItem] = Field(default_factory=list)
|
||||
thread_id: str | None = None
|
||||
message_id: str | None = None
|
||||
title: str | None = None
|
||||
tags: list[str] = Field(default_factory=list)
|
||||
status: str = "approved"
|
||||
|
||||
|
||||
class CreateNoteRequest(BaseModel):
|
||||
title: str
|
||||
content_md: str
|
||||
summary: str | None = None
|
||||
tags: list[str] = Field(default_factory=list)
|
||||
status: str = "approved"
|
||||
|
||||
|
||||
class UpdateNoteRequest(BaseModel):
|
||||
title: str | None = None
|
||||
content_md: str | None = None
|
||||
summary: str | None = None
|
||||
tags: list[str] | None = None
|
||||
status: str | None = None
|
||||
|
||||
|
||||
class SearchRequest(BaseModel):
|
||||
query: str
|
||||
mode: str = Field(default="keyword", description="keyword | vector | hybrid (vector/hybrid fall back to keyword in v1)")
|
||||
limit: int = Field(default=8, ge=1, le=50)
|
||||
|
||||
|
||||
class ResolveWikilinksRequest(BaseModel):
|
||||
targets: list[str] = Field(default_factory=list, description="`[[wikilink]]` targets to resolve to note ids / vault pages.")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Helpers
|
||||
# --------------------------------------------------------------------------- #
|
||||
async def _read_thread_values(checkpointer, thread_id: str) -> dict[str, Any]:
|
||||
"""Return serialized channel values for a thread, or raise 404/500."""
|
||||
config = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
|
||||
try:
|
||||
checkpoint_tuple = await checkpointer.aget_tuple(config)
|
||||
except Exception:
|
||||
logger.exception("Failed to read thread %s for knowledge ingest", sanitize_log_param(thread_id))
|
||||
raise HTTPException(status_code=500, detail="Failed to read thread state")
|
||||
if checkpoint_tuple is None:
|
||||
raise HTTPException(status_code=404, detail=f"Thread {thread_id} not found")
|
||||
checkpoint = getattr(checkpoint_tuple, "checkpoint", {}) or {}
|
||||
return serialize_channel_values(checkpoint.get("channel_values", {}) or {})
|
||||
|
||||
|
||||
async def _enrich_actor_names(note: dict[str, Any]) -> None:
|
||||
"""Resolve created_by/updated_by user ids to a human label (email).
|
||||
|
||||
Adds ``created_by_name`` / ``updated_by_name`` to the note dict so the UI can
|
||||
show a person instead of a raw UUID. Best-effort: unresolved ids fall back to
|
||||
``None`` and the frontend then shows a shortened id.
|
||||
"""
|
||||
provider = get_local_provider()
|
||||
cache: dict[str, str | None] = {}
|
||||
|
||||
async def name_of(uid: str | None) -> str | None:
|
||||
if not uid:
|
||||
return None
|
||||
if uid not in cache:
|
||||
try:
|
||||
user = await provider.get_user(uid)
|
||||
cache[uid] = user.email if user else None
|
||||
except Exception:
|
||||
cache[uid] = None
|
||||
return cache[uid]
|
||||
|
||||
note["created_by_name"] = await name_of(note.get("created_by"))
|
||||
note["updated_by_name"] = await name_of(note.get("updated_by"))
|
||||
|
||||
|
||||
def _messages_and_meta(values: dict[str, Any]) -> tuple[list[dict[str, Any]], str | None, list[dict[str, Any]]]:
|
||||
messages = values.get("messages")
|
||||
if not isinstance(messages, list):
|
||||
messages = []
|
||||
title = values.get("title") if isinstance(values.get("title"), str) else None
|
||||
batches = build_reference_batches(messages) if messages else []
|
||||
return messages, title, batches
|
||||
|
||||
|
||||
async def _ingest_thread_background(service, checkpointer, thread_id: str, body: IngestThreadRequest, created_by: str | None) -> None:
|
||||
"""Sediment a thread into a note without an HTTP context.
|
||||
|
||||
Runs after the response is sent (FastAPI ``BackgroundTasks``); ``created_by``
|
||||
must be captured by the request handler since the user contextvar is gone
|
||||
by the time this executes. All failures are swallowed (logged) — the user
|
||||
has already been told the save was queued.
|
||||
"""
|
||||
try:
|
||||
values = await _read_thread_values(checkpointer, thread_id)
|
||||
messages, title, batches = _messages_and_meta(values)
|
||||
if not messages:
|
||||
logger.warning("Background knowledge ingest skipped: thread %s has no messages", sanitize_log_param(thread_id))
|
||||
return
|
||||
await service.capture_thread(
|
||||
thread_id=thread_id,
|
||||
messages=messages,
|
||||
thread_title=title,
|
||||
mode=body.mode,
|
||||
title=body.title,
|
||||
status=body.status,
|
||||
tags=body.tags,
|
||||
include_sources=body.include_sources,
|
||||
reference_batches=batches,
|
||||
created_by=created_by,
|
||||
model_name=body.model_name,
|
||||
)
|
||||
except EmptyThreadError:
|
||||
logger.warning("Background knowledge ingest skipped: thread %s has no sediment-able content", sanitize_log_param(thread_id))
|
||||
except Exception:
|
||||
logger.exception("Background knowledge ingest failed for thread %s", sanitize_log_param(thread_id))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Ingest
|
||||
# --------------------------------------------------------------------------- #
|
||||
@router.post("/ingest/thread/{thread_id}")
|
||||
@require_auth
|
||||
@require_permission("threads", "read", owner_check=True)
|
||||
async def ingest_thread(thread_id: str, body: IngestThreadRequest, request: Request, background_tasks: BackgroundTasks) -> dict[str, Any]:
|
||||
"""Sediment a single thread into a knowledge note.
|
||||
|
||||
With ``background=true`` the heavy capture (read checkpoint → extract →
|
||||
write vault + DB) is deferred to a background task and the endpoint returns
|
||||
``{"queued": true}`` immediately, so the user's "保存" click never blocks.
|
||||
"""
|
||||
service = get_knowledge_service(request)
|
||||
checkpointer = get_checkpointer(request)
|
||||
if body.background:
|
||||
# Capture the user id now — the contextvar is gone once the task runs.
|
||||
background_tasks.add_task(
|
||||
_ingest_thread_background,
|
||||
service,
|
||||
checkpointer,
|
||||
thread_id,
|
||||
body,
|
||||
get_effective_user_id(),
|
||||
)
|
||||
return {"queued": True}
|
||||
values = await _read_thread_values(checkpointer, thread_id)
|
||||
messages, title, batches = _messages_and_meta(values)
|
||||
if not messages:
|
||||
raise HTTPException(status_code=400, detail="Thread has no messages to sediment")
|
||||
try:
|
||||
result = await service.capture_thread(
|
||||
thread_id=thread_id,
|
||||
messages=messages,
|
||||
thread_title=title,
|
||||
mode=body.mode,
|
||||
title=body.title,
|
||||
status=body.status,
|
||||
tags=body.tags,
|
||||
include_sources=body.include_sources,
|
||||
reference_batches=batches,
|
||||
created_by=get_effective_user_id(),
|
||||
model_name=body.model_name,
|
||||
)
|
||||
except EmptyThreadError:
|
||||
raise HTTPException(status_code=400, detail="Thread has no sediment-able content")
|
||||
return {
|
||||
"note": result["note"],
|
||||
"created": result["created"],
|
||||
"notes": result.get("notes", [result["note"]]),
|
||||
"llm_fallback": result.get("llm_fallback", False),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/ingest/threads/batch")
|
||||
@require_auth
|
||||
async def ingest_threads_batch(body: BatchIngestRequest, request: Request) -> dict[str, Any]:
|
||||
"""Batch-sediment the caller's recent threads into knowledge notes."""
|
||||
service = get_knowledge_service(request)
|
||||
checkpointer = get_checkpointer(request)
|
||||
thread_store = get_thread_store(request)
|
||||
|
||||
threads = await thread_store.search(limit=body.limit, offset=body.offset)
|
||||
created = skipped = failed = 0
|
||||
# Notes that landed via the rule-based fallback because the chosen model
|
||||
# produced nothing usable (timeout / API error / non-JSON). Surfaced so the
|
||||
# caller can tell the user "X 条因大模型异常改用了规则抽取".
|
||||
llm_fallback = 0
|
||||
items: list[dict[str, Any]] = []
|
||||
for meta in threads:
|
||||
thread_id = meta.get("thread_id")
|
||||
if not thread_id:
|
||||
continue
|
||||
try:
|
||||
values = await _read_thread_values(checkpointer, thread_id)
|
||||
messages, title, batches = _messages_and_meta(values)
|
||||
if not messages:
|
||||
skipped += 1
|
||||
items.append({"thread_id": thread_id, "note_id": None, "status": "skipped"})
|
||||
continue
|
||||
result = await service.capture_thread(
|
||||
thread_id=thread_id,
|
||||
messages=messages,
|
||||
thread_title=title or meta.get("display_name"),
|
||||
mode=body.mode,
|
||||
status=body.status,
|
||||
include_sources=body.include_sources,
|
||||
reference_batches=batches,
|
||||
created_by=get_effective_user_id(),
|
||||
skip_existing=body.skip_existing,
|
||||
model_name=body.model_name,
|
||||
)
|
||||
if result.get("llm_fallback"):
|
||||
llm_fallback += 1
|
||||
if result["created"]:
|
||||
created += 1
|
||||
items.append({"thread_id": thread_id, "note_id": result["note"]["id"], "status": "created", "llm_fallback": bool(result.get("llm_fallback"))})
|
||||
else:
|
||||
skipped += 1
|
||||
items.append({"thread_id": thread_id, "note_id": result["note"]["id"], "status": "skipped"})
|
||||
except EmptyThreadError:
|
||||
skipped += 1
|
||||
items.append({"thread_id": thread_id, "note_id": None, "status": "skipped"})
|
||||
except HTTPException:
|
||||
failed += 1
|
||||
items.append({"thread_id": thread_id, "note_id": None, "status": "failed"})
|
||||
except Exception:
|
||||
logger.exception("Batch ingest failed for thread %s", sanitize_log_param(str(thread_id)))
|
||||
failed += 1
|
||||
items.append({"thread_id": thread_id, "note_id": None, "status": "failed"})
|
||||
|
||||
return {"ok": True, "processed": len(items), "created": created, "skipped": skipped, "failed": failed, "llm_fallback": llm_fallback, "items": items}
|
||||
|
||||
|
||||
@router.post("/ingest/search")
|
||||
@require_auth
|
||||
async def ingest_search(body: IngestSearchRequest, request: Request) -> dict[str, Any]:
|
||||
"""Sediment search/tool results as a reference note."""
|
||||
service = get_knowledge_service(request)
|
||||
note = await service.capture_search(
|
||||
query=body.query,
|
||||
results=[r.model_dump() for r in body.results],
|
||||
thread_id=body.thread_id,
|
||||
message_id=body.message_id,
|
||||
title=body.title,
|
||||
tags=body.tags,
|
||||
status=body.status,
|
||||
created_by=get_effective_user_id(),
|
||||
)
|
||||
return {"note": note}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# CRUD
|
||||
# --------------------------------------------------------------------------- #
|
||||
@router.post("/notes")
|
||||
@require_auth
|
||||
async def create_note(body: CreateNoteRequest, request: Request) -> dict[str, Any]:
|
||||
service = get_knowledge_service(request)
|
||||
note = await service.create_manual_note(
|
||||
title=body.title,
|
||||
content_md=body.content_md,
|
||||
summary=body.summary,
|
||||
tags=body.tags,
|
||||
status=body.status,
|
||||
created_by=get_effective_user_id(),
|
||||
)
|
||||
return {"note": note}
|
||||
|
||||
|
||||
@router.get("/notes")
|
||||
@require_auth
|
||||
async def list_notes(
|
||||
request: Request,
|
||||
keyword: str | None = None,
|
||||
tag: str | None = None,
|
||||
source_type: str | None = None,
|
||||
status: str | None = None,
|
||||
limit: int = 20,
|
||||
offset: int = 0,
|
||||
) -> dict[str, Any]:
|
||||
service = get_knowledge_service(request)
|
||||
limit = max(1, min(limit, 100))
|
||||
offset = max(0, offset)
|
||||
return await service.list_notes(keyword=keyword, tag=tag, source_type=source_type, status=status, limit=limit, offset=offset)
|
||||
|
||||
|
||||
@router.get("/notes/{note_id}")
|
||||
@require_auth
|
||||
async def get_note(note_id: str, request: Request) -> dict[str, Any]:
|
||||
service = get_knowledge_service(request)
|
||||
note = await service.get_note(note_id)
|
||||
if note is None:
|
||||
raise HTTPException(status_code=404, detail="Knowledge note not found")
|
||||
await _enrich_actor_names(note)
|
||||
return note
|
||||
|
||||
|
||||
@router.put("/notes/{note_id}")
|
||||
@require_auth
|
||||
async def update_note(note_id: str, body: UpdateNoteRequest, request: Request) -> dict[str, Any]:
|
||||
service = get_knowledge_service(request)
|
||||
note = await service.update_note(
|
||||
note_id,
|
||||
title=body.title,
|
||||
content_md=body.content_md,
|
||||
summary=body.summary,
|
||||
tags=body.tags,
|
||||
status=body.status,
|
||||
updated_by=get_effective_user_id(),
|
||||
)
|
||||
if note is None:
|
||||
raise HTTPException(status_code=404, detail="Knowledge note not found")
|
||||
return note
|
||||
|
||||
|
||||
@router.delete("/notes/{note_id}")
|
||||
@require_auth
|
||||
async def delete_note(note_id: str, request: Request, hard: bool = False) -> dict[str, Any]:
|
||||
service = get_knowledge_service(request)
|
||||
ok = await service.delete_note(note_id, updated_by=get_effective_user_id(), hard=hard)
|
||||
if not ok:
|
||||
raise HTTPException(status_code=404, detail="Knowledge note not found")
|
||||
return {"ok": True, "note_id": note_id}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Search & graph
|
||||
# --------------------------------------------------------------------------- #
|
||||
@router.post("/search")
|
||||
@require_auth
|
||||
async def search(body: SearchRequest, request: Request) -> dict[str, Any]:
|
||||
service = get_knowledge_service(request)
|
||||
items = await service.search(body.query, mode=body.mode, limit=body.limit)
|
||||
return {"items": items}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Wikilinks ([[...]] resolution + vault meta-page viewer)
|
||||
# --------------------------------------------------------------------------- #
|
||||
@router.post("/resolve")
|
||||
@require_auth
|
||||
async def resolve_wikilinks(body: ResolveWikilinksRequest, request: Request) -> dict[str, Any]:
|
||||
"""Resolve ``[[wikilink]]`` targets to navigable destinations.
|
||||
|
||||
Each target resolves to a knowledge note (``kind=note``), a vault meta/hub
|
||||
page (``kind=vault``), or nothing (``kind=missing``). Powers clickable
|
||||
wikilinks in the knowledge UI.
|
||||
"""
|
||||
service = get_knowledge_service(request)
|
||||
return {"resolved": await service.resolve_wikilinks(body.targets)}
|
||||
|
||||
|
||||
@router.get("/notes/{note_id}/wikilinks")
|
||||
@require_auth
|
||||
async def resolve_note_wikilinks(note_id: str, request: Request) -> dict[str, Any]:
|
||||
"""Resolve every ``[[wikilink]]`` found in a note's body in one call."""
|
||||
service = get_knowledge_service(request)
|
||||
resolved = await service.resolve_note_wikilinks(note_id)
|
||||
if resolved is None:
|
||||
raise HTTPException(status_code=404, detail="Knowledge note not found")
|
||||
return {"resolved": resolved}
|
||||
|
||||
|
||||
@router.get("/vault-page")
|
||||
@require_auth
|
||||
async def get_vault_page(request: Request, path: str) -> dict[str, Any]:
|
||||
"""Return a raw vault Markdown meta/hub page (no DB note) by its path."""
|
||||
service = get_knowledge_service(request)
|
||||
page = service.read_vault_page(path)
|
||||
if page is None:
|
||||
raise HTTPException(status_code=404, detail="Vault page not found")
|
||||
return page
|
||||
|
||||
|
||||
class MergeRequest(BaseModel):
|
||||
primary_id: str
|
||||
duplicate_ids: list[str] = Field(default_factory=list)
|
||||
|
||||
|
||||
@router.get("/duplicates")
|
||||
@require_auth
|
||||
async def list_duplicates(request: Request) -> dict[str, Any]:
|
||||
"""Return suggested duplicate groups (phase 3, heuristic)."""
|
||||
service = get_knowledge_service(request)
|
||||
return {"groups": await service.find_duplicates()}
|
||||
|
||||
|
||||
@router.post("/merge")
|
||||
@require_auth
|
||||
async def merge_notes(body: MergeRequest, request: Request) -> dict[str, Any]:
|
||||
"""Merge duplicate notes into a primary note (sources moved, dups archived)."""
|
||||
service = get_knowledge_service(request)
|
||||
note = await service.merge_notes(body.primary_id, body.duplicate_ids, updated_by=get_effective_user_id())
|
||||
if note is None:
|
||||
raise HTTPException(status_code=404, detail="Primary note not found")
|
||||
return {"note": note}
|
||||
|
||||
|
||||
@router.post("/reindex")
|
||||
@require_auth
|
||||
async def reindex_embeddings(request: Request) -> dict[str, Any]:
|
||||
"""Rebuild the vector index for all notes (maintenance; no-op if embeddings off)."""
|
||||
service = get_knowledge_service(request)
|
||||
return await service.reindex_all()
|
||||
|
||||
|
||||
@router.get("/notes/{note_id}/versions")
|
||||
@require_auth
|
||||
async def list_note_versions(note_id: str, request: Request) -> dict[str, Any]:
|
||||
"""Return the edit history (pre-update snapshots) for a note."""
|
||||
service = get_knowledge_service(request)
|
||||
return {"items": await service.list_versions(note_id)}
|
||||
|
||||
|
||||
@router.get("/graph")
|
||||
@require_auth
|
||||
async def get_graph(request: Request) -> dict[str, Any]:
|
||||
"""Return product knowledge-graph data (notes + entities + relations)."""
|
||||
service = get_knowledge_service(request)
|
||||
return await service.get_graph()
|
||||
|
||||
|
||||
@router.post("/export/graph")
|
||||
@require_auth
|
||||
async def export_graph(request: Request) -> dict[str, Any]:
|
||||
service = get_knowledge_service(request)
|
||||
result = await service.export_graph()
|
||||
base = "/api/knowledge/export/files"
|
||||
return {
|
||||
"ok": True,
|
||||
"files": {
|
||||
"graph_json": f"{base}/graph.json",
|
||||
"graph_html": f"{base}/graph.html",
|
||||
"graph_graphml": f"{base}/graph.graphml",
|
||||
"cypher": f"{base}/cypher.txt",
|
||||
},
|
||||
"stats": result["stats"],
|
||||
}
|
||||
|
||||
|
||||
@router.get("/export/files/{filename}")
|
||||
@require_auth
|
||||
async def export_file(filename: str, request: Request) -> FileResponse:
|
||||
service = get_knowledge_service(request)
|
||||
path = service.export_file_path(filename)
|
||||
if path is None:
|
||||
# Lazily build the export on first access so the iframe never 404s.
|
||||
await service.export_graph()
|
||||
path = service.export_file_path(filename)
|
||||
if path is None:
|
||||
raise HTTPException(status_code=404, detail="Export file not found")
|
||||
media_type = mimetypes.guess_type(filename)[0] or "application/octet-stream"
|
||||
return FileResponse(path, media_type=media_type)
|
||||
@@ -0,0 +1,97 @@
|
||||
"""Public (unauthenticated) read-only user leaderboard.
|
||||
|
||||
Mounted under the ``/api/public/`` prefix so the auth middleware lets this route
|
||||
through without a session cookie. Powers the public "publish" page for the user
|
||||
leaderboard — a view-only snapshot with no admin controls and no mutations.
|
||||
|
||||
The snapshot honors the same system-wide leaderboard settings (excluded users,
|
||||
scheduled/admin/failed inclusion) as the authenticated admin endpoint, so any
|
||||
accounts an admin chose to hide stay hidden on the public page as well.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query
|
||||
|
||||
from app.gateway.routers.admin_users import (
|
||||
AdminLeaderboardResponse,
|
||||
_load_or_schedule_daily_leaderboard,
|
||||
_resolve_range,
|
||||
)
|
||||
from deerflow.config.system_settings import load_system_settings
|
||||
from deerflow.persistence.engine import get_session_factory
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Whitelisted under ``auth_middleware._PUBLIC_PATH_PREFIXES`` (``/api/public/``).
|
||||
router = APIRouter(prefix="/api/public/leaderboard", tags=["public-leaderboard"])
|
||||
|
||||
|
||||
def _mask_email(email: str | None) -> str:
|
||||
"""Reduce an email to its local part so the public page can show a
|
||||
recognizable handle without leaking the full address (domain).
|
||||
|
||||
The leaderboard only makes sense if it names its top users, so the local
|
||||
part is kept intentionally; the privacy gain is dropping the domain and the
|
||||
raw account id from the unauthenticated payload.
|
||||
"""
|
||||
if not email:
|
||||
return ""
|
||||
return email.split("@", 1)[0].strip() or ""
|
||||
|
||||
|
||||
def _anonymize(resp: AdminLeaderboardResponse) -> AdminLeaderboardResponse:
|
||||
"""Strip PII from a leaderboard snapshot before it leaves over the public,
|
||||
unauthenticated endpoint: email → local part, account/thread ids → opaque
|
||||
short hashes (kept stable so they still work as React keys)."""
|
||||
|
||||
def hashed(value: str | None) -> str:
|
||||
if not value:
|
||||
return ""
|
||||
return hashlib.sha1(value.encode("utf-8")).hexdigest()[:12]
|
||||
|
||||
seen_users: set[int] = set()
|
||||
for bucket in (resp.users_by_activity, resp.users_by_questions, resp.users_by_tokens):
|
||||
for user in bucket:
|
||||
if id(user) in seen_users:
|
||||
continue
|
||||
seen_users.add(id(user))
|
||||
user.email = _mask_email(user.email)
|
||||
user.user_id = hashed(user.user_id)
|
||||
|
||||
for bucket in (resp.recent_questions, resp.frequent_questions):
|
||||
for question in bucket:
|
||||
question.email = _mask_email(question.email)
|
||||
question.user_id = hashed(question.user_id)
|
||||
question.thread_id = ""
|
||||
question.run_id = ""
|
||||
|
||||
return resp
|
||||
|
||||
|
||||
@router.get("", response_model=AdminLeaderboardResponse)
|
||||
@router.get("/", response_model=AdminLeaderboardResponse)
|
||||
async def get_public_leaderboard(
|
||||
days: int = Query(default=30, ge=1, le=365),
|
||||
since: str | None = Query(default=None, description="Inclusive range start, ISO datetime"),
|
||||
until: str | None = Query(default=None, description="Inclusive range end, ISO datetime"),
|
||||
limit: int = Query(default=20, ge=5, le=100),
|
||||
) -> AdminLeaderboardResponse:
|
||||
"""Return a read-only leaderboard snapshot for the public publish page."""
|
||||
session_factory = get_session_factory()
|
||||
if session_factory is None:
|
||||
raise HTTPException(status_code=503, detail="Analytics requires database persistence")
|
||||
|
||||
settings = load_system_settings().leaderboard
|
||||
since_dt, until_dt, _range_days = _resolve_range(days, since, until)
|
||||
resp = await _load_or_schedule_daily_leaderboard(
|
||||
session_factory,
|
||||
settings,
|
||||
since_dt,
|
||||
until_dt,
|
||||
limit=limit,
|
||||
)
|
||||
return _anonymize(resp)
|
||||
@@ -56,6 +56,44 @@ def _is_admin_user(request: Request) -> bool:
|
||||
return getattr(user, "system_role", None) == "admin"
|
||||
|
||||
|
||||
def _email_local_part(email: str | None) -> str | None:
|
||||
"""Return the username portion (before ``@``) of an email address."""
|
||||
if not email:
|
||||
return None
|
||||
at = email.find("@")
|
||||
return email[:at] if at > 0 else email
|
||||
|
||||
|
||||
async def _attach_owner_usernames(responses: list["SkillResponse"]) -> list["SkillResponse"]:
|
||||
"""Resolve each custom skill's ``owner_user_id`` to a friendly username.
|
||||
|
||||
One lookup per distinct owner id; tolerates deleted accounts and an
|
||||
uninitialised auth provider by leaving ``owner_username`` as None.
|
||||
"""
|
||||
owner_ids = {r.owner_user_id for r in responses if r.owner_user_id}
|
||||
if not owner_ids:
|
||||
return responses
|
||||
try:
|
||||
from app.gateway.deps import get_local_provider
|
||||
|
||||
provider = get_local_provider()
|
||||
except Exception:
|
||||
return responses
|
||||
names: dict[str, str] = {}
|
||||
for uid in owner_ids:
|
||||
try:
|
||||
user = await provider.get_user(uid)
|
||||
except Exception:
|
||||
user = None
|
||||
local = _email_local_part(getattr(user, "email", None)) if user is not None else None
|
||||
if local:
|
||||
names[uid] = local
|
||||
for resp in responses:
|
||||
if resp.owner_user_id:
|
||||
resp.owner_username = names.get(resp.owner_user_id)
|
||||
return responses
|
||||
|
||||
|
||||
def _can_view_skill_source(
|
||||
category: SkillCategory,
|
||||
owner_user_id: str | None,
|
||||
@@ -111,6 +149,7 @@ class SkillResponse(BaseModel):
|
||||
"""Response model for skill information."""
|
||||
|
||||
name: str = Field(..., description="Name of the skill")
|
||||
name_zh: str | None = Field(default=None, description="用户配置或大模型生成的中文名称")
|
||||
description: str = Field(..., description="Description of what the skill does")
|
||||
license: str | None = Field(None, description="License information")
|
||||
category: SkillCategory = Field(..., description="Category of the skill (public or custom)")
|
||||
@@ -125,6 +164,7 @@ class SkillResponse(BaseModel):
|
||||
featured: bool = Field(default=False, description="Whether this skill is pinned to the top of the list (sort only)")
|
||||
source: str = Field(default="upload", description="Origin: 'agent' for agent-created, 'upload' for user-uploaded")
|
||||
owner_user_id: str | None = Field(default=None, description="Custom skill owner (null = built-in or legacy)")
|
||||
owner_username: str | None = Field(default=None, description="Publisher username (email local part). Null for built-in skills.")
|
||||
published: bool = Field(default=False, description="Whether the owner has made this skill visible to other users")
|
||||
builtin: bool = Field(default=False, description="True when owner_user_id is null (no human owner)")
|
||||
detail: str | None = Field(default=None, description="Human-authored long-form detail text (Markdown), editable by owner/admin")
|
||||
@@ -267,11 +307,13 @@ def _skill_to_response(
|
||||
published = False
|
||||
builtin = skill.category != SkillCategory.CUSTOM
|
||||
detail: str | None = None
|
||||
name_zh: str | None = None
|
||||
if ownership is not None:
|
||||
owner_user_id = ownership.get("owner_user_id")
|
||||
published = bool(ownership.get("published", False))
|
||||
builtin = bool(ownership.get("builtin", owner_user_id is None))
|
||||
detail = ownership.get("detail")
|
||||
name_zh = ownership.get("name_zh")
|
||||
|
||||
display = None if display_override is _UNSET else display_override
|
||||
if display_override is _UNSET:
|
||||
@@ -284,6 +326,7 @@ def _skill_to_response(
|
||||
|
||||
return SkillResponse(
|
||||
name=skill.name,
|
||||
name_zh=name_zh,
|
||||
description=skill.description,
|
||||
license=skill.license,
|
||||
category=skill.category,
|
||||
@@ -697,7 +740,8 @@ async def _attach_and_filter_tags(
|
||||
resp.tags = assignments.get(resp.name, [])
|
||||
if search:
|
||||
needle = search.strip().lower()
|
||||
responses = [r for r in responses if needle in r.name.lower()]
|
||||
# Match the English name OR the Chinese display name so 中文 search works.
|
||||
responses = [r for r in responses if needle in r.name.lower() or (r.name_zh or "").lower().find(needle) >= 0]
|
||||
if tag_ids:
|
||||
wanted = set(tag_ids)
|
||||
responses = [r for r in responses if any(t.get("id") in wanted for t in r.tags)]
|
||||
@@ -726,7 +770,7 @@ async def list_skills(
|
||||
config: AppConfig = Depends(get_config),
|
||||
skill_store: SkillStore = Depends(get_skill_store),
|
||||
tag_store: TagStore = Depends(get_tag_store),
|
||||
search: str | None = Query(default=None, description="Case-insensitive fuzzy match on skill name"),
|
||||
search: str | None = Query(default=None, description="Case-insensitive fuzzy match on skill name or Chinese name (name_zh)"),
|
||||
tag_ids: list[str] | None = Query(default=None, description="Keep skills carrying any of these tags (OR)"),
|
||||
sort: str = Query(
|
||||
default="time_desc",
|
||||
@@ -786,6 +830,7 @@ async def list_skills(
|
||||
)
|
||||
)
|
||||
responses = await _attach_and_filter_tags(tag_store, responses, search, tag_ids)
|
||||
responses = await _attach_owner_usernames(responses)
|
||||
responses = _sort_skill_responses(responses, sort)
|
||||
return SkillsListResponse(skills=responses)
|
||||
except HTTPException:
|
||||
@@ -1176,6 +1221,7 @@ async def list_custom_skills(
|
||||
display_sample=adapter.get("sample") if adapter else None,
|
||||
)
|
||||
)
|
||||
responses = await _attach_owner_usernames(responses)
|
||||
return SkillsListResponse(skills=responses)
|
||||
except Exception as e:
|
||||
logger.error("Failed to list custom skills: %s", e, exc_info=True)
|
||||
@@ -2309,6 +2355,324 @@ async def update_skill_detail(
|
||||
raise HTTPException(status_code=500, detail=f"Failed to update skill detail: {str(e)}")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 技能中文名称(用户可配置 + 大模型自动生成 + 一键补充)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_SKILL_NAME_ZH_SYSTEM_PROMPT = (
|
||||
"你是一名中文产品命名助手。根据给定技能的英文名与功能描述,生成一个简洁、自然、"
|
||||
"易懂的中文名称。要求:2 到 10 个汉字;准确概括技能用途;不要包含标点、引号、英文、"
|
||||
"解释或多余文字;只输出中文名称本身。"
|
||||
)
|
||||
|
||||
|
||||
class SkillNameZhModelError(Exception):
|
||||
"""The LLM call backing skill-name generation failed (a *model-side* problem).
|
||||
|
||||
Distinguishes a genuine model failure (timeout / API error / unbuildable
|
||||
model) — which must be surfaced to the user as "大模型报错导致无法补全" — from
|
||||
the model merely returning an unusable name (callers get ``None`` instead).
|
||||
"""
|
||||
|
||||
|
||||
def _clean_generated_name_zh(raw: str) -> str | None:
|
||||
"""Normalise an LLM-produced Chinese name: first line, strip quotes/punct, cap length."""
|
||||
if not raw:
|
||||
return None
|
||||
text = str(raw).strip()
|
||||
# Take the first non-empty line only.
|
||||
for line in text.splitlines():
|
||||
line = line.strip()
|
||||
if line:
|
||||
text = line
|
||||
break
|
||||
# Strip wrapping quotes / common trailing punctuation.
|
||||
text = text.strip().strip("\"'“”‘’`「」《》【】()()[]").strip()
|
||||
text = text.rstrip("。.,,、!!??;;::")
|
||||
if not text:
|
||||
return None
|
||||
# Keep it short — guard against the model returning a sentence.
|
||||
if len(text) > 20:
|
||||
text = text[:20]
|
||||
return text or None
|
||||
|
||||
|
||||
def _resolve_skill_name_zh_model_name(app_config: AppConfig, model_name: str | None) -> str | None:
|
||||
"""Pick the model name for skill-name generation.
|
||||
|
||||
An explicit ``model_name`` (user-selected in the UI) wins; otherwise fall
|
||||
back to the curator/security-scan model setting (itself falling back to the
|
||||
main model when unset).
|
||||
"""
|
||||
if model_name and model_name.strip():
|
||||
return model_name.strip()
|
||||
if app_config.skill_evolution and app_config.skill_evolution.curator:
|
||||
return app_config.skill_evolution.curator.model_name
|
||||
return None
|
||||
|
||||
|
||||
async def _generate_skill_name_zh(skill: Skill, app_config: AppConfig, model: Any = None, *, model_name: str | None = None) -> str | None:
|
||||
"""Generate a Chinese display name for ``skill`` via the LLM.
|
||||
|
||||
An explicit ``model_name`` lets the caller pick the model; otherwise it
|
||||
reuses the curator/security-scan model setting (falls back to the main
|
||||
model). Returns ``None`` only when the model replies but the output cannot be
|
||||
normalised into a short Chinese name. Raises :class:`SkillNameZhModelError`
|
||||
when the model itself fails (cannot be built / times out / API error), so the
|
||||
caller can tell the user it was a *model* problem.
|
||||
"""
|
||||
user_prompt = (
|
||||
f"技能英文名:{skill.name}\n"
|
||||
f"功能描述:{skill.description or '(无描述)'}\n\n"
|
||||
"请为该技能起一个中文名称。"
|
||||
)
|
||||
if model is None:
|
||||
try:
|
||||
from deerflow.models import create_chat_model
|
||||
|
||||
resolved = _resolve_skill_name_zh_model_name(app_config, model_name)
|
||||
model = create_chat_model(name=resolved, thinking_enabled=False, app_config=app_config)
|
||||
except Exception as e:
|
||||
logger.warning("Skill name_zh model build failed for %s", skill.name, exc_info=True)
|
||||
raise SkillNameZhModelError(str(e) or "无法初始化所选大模型") from e
|
||||
try:
|
||||
response = await asyncio.wait_for(
|
||||
model.ainvoke(
|
||||
[
|
||||
{"role": "system", "content": _SKILL_NAME_ZH_SYSTEM_PROMPT},
|
||||
{"role": "user", "content": user_prompt},
|
||||
],
|
||||
config={"run_name": "skill_name_zh"},
|
||||
),
|
||||
timeout=60,
|
||||
)
|
||||
except asyncio.TimeoutError as e:
|
||||
logger.warning("Skill name_zh LLM call timed out for %s", skill.name)
|
||||
raise SkillNameZhModelError("大模型响应超时(60 秒)") from e
|
||||
except Exception as e:
|
||||
logger.warning("Skill name_zh LLM call failed for %s", skill.name, exc_info=True)
|
||||
raise SkillNameZhModelError(str(e) or "大模型调用失败") from e
|
||||
return _clean_generated_name_zh(str(getattr(response, "content", "") or ""))
|
||||
|
||||
|
||||
async def _persist_skill_name_zh(
|
||||
skill: Skill,
|
||||
name_zh: str | None,
|
||||
skill_store: SkillStore,
|
||||
user_id: str,
|
||||
*,
|
||||
is_admin: bool,
|
||||
) -> str | None:
|
||||
"""Persist ``name_zh`` into the ``skills`` DB table (never the on-disk SKILL.md).
|
||||
|
||||
Authorization mirrors the ``detail`` annotation: a custom skill is editable by
|
||||
its owner or an admin; a built-in / public skill only by an admin (its DB row
|
||||
is materialized on demand). Returns the stored value.
|
||||
"""
|
||||
if skill.category == SkillCategory.CUSTOM:
|
||||
_record, mode = await _get_editable_skill_or_raise(skill_store, skill.name, user_id, is_admin=is_admin)
|
||||
if mode == "admin":
|
||||
updated = await skill_store.update_admin(skill.name, {"name_zh": name_zh})
|
||||
else:
|
||||
updated = await skill_store.update(skill.name, user_id, {"name_zh": name_zh})
|
||||
else:
|
||||
if not is_admin:
|
||||
raise HTTPException(status_code=403, detail="Only admin users can edit built-in skill Chinese names")
|
||||
if await skill_store.get_any(skill.name) is None:
|
||||
await skill_store.ensure_legacy({"name": skill.name, "owner_user_id": None, "published": True})
|
||||
updated = await skill_store.update_admin(skill.name, {"name_zh": name_zh})
|
||||
|
||||
if updated is None:
|
||||
raise HTTPException(status_code=500, detail="Failed to persist skill Chinese name")
|
||||
return updated.get("name_zh")
|
||||
|
||||
|
||||
class SkillNameZhRequest(BaseModel):
|
||||
"""Request body for setting a skill's Chinese display name."""
|
||||
|
||||
name_zh: str = Field(default="", description="中文名称;传空字符串表示清空")
|
||||
|
||||
|
||||
class SkillNameZhResponse(BaseModel):
|
||||
"""Response model for a single skill's Chinese-name update."""
|
||||
|
||||
name: str = Field(..., description="Skill name")
|
||||
name_zh: str | None = Field(default=None, description="中文名称(为空表示已清空)")
|
||||
|
||||
|
||||
class SkillNameZhBulkResponse(BaseModel):
|
||||
"""Response model for the one-click bulk fill of missing Chinese names."""
|
||||
|
||||
generated: list[SkillNameZhResponse] = Field(default_factory=list, description="本次成功生成中文名的技能")
|
||||
failed: list[str] = Field(default_factory=list, description="生成失败的技能英文名")
|
||||
skipped: int = Field(default=0, description="已有中文名而被跳过的技能数")
|
||||
model_error: str | None = Field(default=None, description="若有技能因大模型报错而失败,这里给出代表性的报错信息")
|
||||
|
||||
|
||||
class GenerateSkillNameZhRequest(BaseModel):
|
||||
"""Optional body for skill-name generation: pick the model to use."""
|
||||
|
||||
model_name: str | None = Field(default=None, description="可选:指定用于生成的大模型名称;留空使用默认配置")
|
||||
|
||||
|
||||
def _can_edit_skill_name_zh(skill: Skill, record: dict[str, Any] | None, user_id: str, *, is_admin: bool) -> bool:
|
||||
"""Whether ``user_id`` may set this skill's Chinese name (mirrors detail auth)."""
|
||||
if is_admin:
|
||||
return True
|
||||
if skill.category != SkillCategory.CUSTOM:
|
||||
return False
|
||||
return record is not None and record.get("owner_user_id") == user_id
|
||||
|
||||
|
||||
@router.put(
|
||||
"/skills/{skill_name}/name-zh",
|
||||
response_model=SkillNameZhResponse,
|
||||
summary="Set Skill Chinese Name",
|
||||
description="设置技能的中文名称(存数据库,不改源文件)。自定义技能由其所有者或管理员编辑;内置/公开技能仅管理员可编辑。传空字符串表示清空,回退到使用英文名。",
|
||||
)
|
||||
async def set_skill_name_zh(
|
||||
skill_name: str,
|
||||
body: SkillNameZhRequest,
|
||||
request: Request,
|
||||
config: AppConfig = Depends(get_config),
|
||||
skill_store: SkillStore = Depends(get_skill_store),
|
||||
) -> SkillNameZhResponse:
|
||||
try:
|
||||
skill_name = skill_name.replace("\r\n", "").replace("\n", "")
|
||||
user_id = _current_user_id(request)
|
||||
is_admin = _is_admin_user(request)
|
||||
skills = get_or_new_skill_storage(app_config=config).load_skills(enabled_only=False)
|
||||
skill = next((s for s in skills if s.name == skill_name), None)
|
||||
if skill is None:
|
||||
raise HTTPException(status_code=404, detail=f"Skill '{skill_name}' not found")
|
||||
name_zh = body.name_zh.strip() or None
|
||||
stored = await _persist_skill_name_zh(skill, name_zh, skill_store, user_id, is_admin=is_admin)
|
||||
return SkillNameZhResponse(name=skill_name, name_zh=stored)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error("Failed to set name_zh for %s: %s", skill_name, e, exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to set skill Chinese name: {str(e)}")
|
||||
|
||||
|
||||
@router.post(
|
||||
"/skills/{skill_name}/name-zh/generate",
|
||||
response_model=SkillNameZhResponse,
|
||||
summary="Generate Skill Chinese Name",
|
||||
description="用大模型为该技能生成一个中文名称并保存到数据库。可在请求体里通过 model_name 指定使用的大模型。",
|
||||
)
|
||||
async def generate_skill_name_zh(
|
||||
skill_name: str,
|
||||
request: Request,
|
||||
body: GenerateSkillNameZhRequest | None = None,
|
||||
config: AppConfig = Depends(get_config),
|
||||
skill_store: SkillStore = Depends(get_skill_store),
|
||||
) -> SkillNameZhResponse:
|
||||
try:
|
||||
body = body or GenerateSkillNameZhRequest()
|
||||
skill_name = skill_name.replace("\r\n", "").replace("\n", "")
|
||||
user_id = _current_user_id(request)
|
||||
is_admin = _is_admin_user(request)
|
||||
skills = get_or_new_skill_storage(app_config=config).load_skills(enabled_only=False)
|
||||
skill = next((s for s in skills if s.name == skill_name), None)
|
||||
if skill is None:
|
||||
raise HTTPException(status_code=404, detail=f"Skill '{skill_name}' not found")
|
||||
try:
|
||||
name_zh = await _generate_skill_name_zh(skill, config, model_name=body.model_name)
|
||||
except SkillNameZhModelError as e:
|
||||
raise HTTPException(status_code=502, detail=f"大模型报错,无法补全技能中文名:{e}")
|
||||
if not name_zh:
|
||||
raise HTTPException(status_code=502, detail="大模型未能生成有效的中文名称,请稍后重试或手动填写")
|
||||
stored = await _persist_skill_name_zh(skill, name_zh, skill_store, user_id, is_admin=is_admin)
|
||||
return SkillNameZhResponse(name=skill_name, name_zh=stored)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error("Failed to generate name_zh for %s: %s", skill_name, e, exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to generate skill Chinese name: {str(e)}")
|
||||
|
||||
|
||||
@router.post(
|
||||
"/skills/name-zh/generate-missing",
|
||||
response_model=SkillNameZhBulkResponse,
|
||||
summary="Bulk Generate Missing Skill Chinese Names",
|
||||
description="管理员一键为库中所有尚未配置中文名称的技能生成中文名(存数据库)。可在请求体里通过 model_name 指定使用的大模型。",
|
||||
)
|
||||
async def generate_missing_skill_names_zh(
|
||||
request: Request,
|
||||
body: GenerateSkillNameZhRequest | None = None,
|
||||
config: AppConfig = Depends(get_config),
|
||||
skill_store: SkillStore = Depends(get_skill_store),
|
||||
) -> SkillNameZhBulkResponse:
|
||||
try:
|
||||
from deerflow.models import create_chat_model
|
||||
|
||||
body = body or GenerateSkillNameZhRequest()
|
||||
user_id = _current_user_id(request)
|
||||
is_admin = _is_admin_user(request)
|
||||
# 一键补充技能中文名仅对管理员开放。
|
||||
if not is_admin:
|
||||
raise HTTPException(status_code=403, detail="仅管理员可一键补充技能中文名")
|
||||
skills = get_or_new_skill_storage(app_config=config).load_skills(enabled_only=False)
|
||||
|
||||
# Admin sees every row, so existing name_zh values come from the full set.
|
||||
records = {r["name"]: r for r in await skill_store.list_all()}
|
||||
|
||||
pending: list[Skill] = []
|
||||
skipped = 0
|
||||
for skill in skills:
|
||||
record = records.get(skill.name)
|
||||
if record and record.get("name_zh"):
|
||||
skipped += 1
|
||||
continue
|
||||
pending.append(skill)
|
||||
|
||||
generated: list[SkillNameZhResponse] = []
|
||||
failed: list[str] = []
|
||||
model_error: str | None = None
|
||||
if pending:
|
||||
# Build the model once and reuse it for every skill. A build failure
|
||||
# is a model problem — fail fast with a clear message rather than
|
||||
# silently producing zero results.
|
||||
try:
|
||||
resolved_name = _resolve_skill_name_zh_model_name(config, body.model_name)
|
||||
model = create_chat_model(name=resolved_name, thinking_enabled=False, app_config=config)
|
||||
except Exception as e:
|
||||
logger.warning("Failed to build model for bulk name_zh generation", exc_info=True)
|
||||
raise HTTPException(status_code=502, detail=f"大模型报错,无法补全技能中文名:{str(e) or '无法初始化所选大模型'}")
|
||||
|
||||
# Generate concurrently with a small bound to avoid overloading the model.
|
||||
semaphore = asyncio.Semaphore(4)
|
||||
|
||||
async def _one(skill: Skill) -> tuple[Skill, str | None, str | None]:
|
||||
async with semaphore:
|
||||
try:
|
||||
return skill, await _generate_skill_name_zh(skill, config, model=model), None
|
||||
except SkillNameZhModelError as e:
|
||||
return skill, None, str(e)
|
||||
|
||||
results = await asyncio.gather(*[_one(s) for s in pending])
|
||||
for skill, name_zh, err in results:
|
||||
if err and model_error is None:
|
||||
model_error = err
|
||||
if not name_zh:
|
||||
failed.append(skill.name)
|
||||
continue
|
||||
try:
|
||||
stored = await _persist_skill_name_zh(skill, name_zh, skill_store, user_id, is_admin=is_admin)
|
||||
generated.append(SkillNameZhResponse(name=skill.name, name_zh=stored))
|
||||
except HTTPException:
|
||||
# Permission / persistence issue for this one skill — skip it.
|
||||
failed.append(skill.name)
|
||||
|
||||
return SkillNameZhBulkResponse(generated=generated, failed=failed, skipped=skipped, model_error=model_error)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error("Failed to bulk-generate skill Chinese names: %s", e, exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to bulk-generate skill Chinese names: {str(e)}")
|
||||
|
||||
|
||||
class SkillPinRequest(BaseModel):
|
||||
pinned: bool = Field(..., description="Whether to pin or unpin the skill")
|
||||
|
||||
|
||||
@@ -10,90 +10,90 @@ gateway:
|
||||
port: 8001
|
||||
|
||||
models:
|
||||
- name: deepseek-chat
|
||||
display_name: DeepSeek Chat
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: deepseek-chat
|
||||
api_key: sk-3fee562f834a4c08bb0d471ef44cb445
|
||||
base_url: https://api.deepseek.com
|
||||
request_timeout: 600.0
|
||||
max_retries: 2
|
||||
max_tokens: 4096
|
||||
temperature: 1.0
|
||||
supports_vision: false
|
||||
supports_thinking: true
|
||||
# - name: deepseek-chat
|
||||
# display_name: DeepSeek Chat
|
||||
# use: langchain_openai:ChatOpenAI
|
||||
# model: deepseek-chat
|
||||
# api_key: sk-1520028668cb4cc388ed8315ff1e856e
|
||||
# base_url: https://api.deepseek.com
|
||||
# request_timeout: 600.0
|
||||
# max_retries: 2
|
||||
# max_tokens: 4096
|
||||
# temperature: 1.0
|
||||
# supports_vision: false
|
||||
# supports_thinking: true
|
||||
|
||||
- name: deepseek-v4-flash
|
||||
display_name: DeepSeek V4 Flash
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: deepseek-v4-flash
|
||||
api_key: sk-3fee562f834a4c08bb0d471ef44cb445
|
||||
base_url: https://api.deepseek.com
|
||||
request_timeout: 600.0
|
||||
max_retries: 2
|
||||
max_tokens: 4096
|
||||
temperature: 1.0
|
||||
supports_vision: false
|
||||
supports_thinking: true
|
||||
# - name: deepseek-v4-flash
|
||||
# display_name: DeepSeek V4 Flash
|
||||
# use: langchain_openai:ChatOpenAI
|
||||
# model: deepseek-v4-flash
|
||||
# api_key: sk-e1c22b7931dc446f8b67e57878edd105
|
||||
# base_url: https://api.deepseek.com
|
||||
# request_timeout: 600.0
|
||||
# max_retries: 2
|
||||
# max_tokens: 4096
|
||||
# temperature: 1.0
|
||||
# supports_vision: false
|
||||
# supports_thinking: true
|
||||
|
||||
- name: deepseek-v4-pro
|
||||
display_name: DeepSeek V4 Pro
|
||||
# - name: deepseek-v4-pro
|
||||
# display_name: DeepSeek V4 Pro
|
||||
# use: langchain_openai:ChatOpenAI
|
||||
# model: deepseek-v4-pro
|
||||
# api_key: sk-e1c22b7931dc446f8b67e57878edd105
|
||||
# base_url: https://api.deepseek.com
|
||||
# request_timeout: 600.0
|
||||
# max_retries: 2
|
||||
# max_tokens: 4096
|
||||
# temperature: 1.0
|
||||
# supports_vision: false
|
||||
# supports_thinking: true
|
||||
# - name: MiniMax-M2.7
|
||||
# display_name: MiniMax-M2.7
|
||||
# use: langchain_openai:ChatOpenAI
|
||||
# model: MiniMax-M2.7
|
||||
# api_key: sk-cp-mrsAGtzDpUl0TpOBPVoiRG_pWF1kDDN9_mDAnONBs6HIPcNUexSKtSbx2QOxfKBvJT6Xw6hDTV6hdkJYFWfNA8UtHQ08TiGJiosdRoW6q1sGb6Cb7D85NHk
|
||||
# base_url: https://api.minimaxi.com/v1
|
||||
# max_tokens: 4096
|
||||
# temperature: 1.0
|
||||
# supports_vision: true
|
||||
# supports_thinking: true
|
||||
# - name: glm-5.1
|
||||
# display_name: glm-5.1
|
||||
# use: langchain_openai:ChatOpenAI
|
||||
# model: glm-5.1
|
||||
# api_key: 75121e4bed7640d5b71ac95cf8b4310a.RA3GyRBAR3kv7VrR
|
||||
# base_url: https://open.bigmodel.cn/api/paas/v4/
|
||||
# max_tokens: 4096
|
||||
# temperature: 1.0
|
||||
# supports_vision: true
|
||||
# supports_thinking: true
|
||||
# - name: deepseek-ai/DeepSeek-OCR
|
||||
# display_name: deepseek-ai/DeepSeek-OCR
|
||||
# use: langchain_openai:ChatOpenAI
|
||||
# model: deepseek-ai/DeepSeek-OCR
|
||||
# api_key: sk-beshfxfxfrllsczpqfrgjbugfbpzxgqgedbehceytpcvvtyb
|
||||
# base_url: https://api.siliconflow.cn/v1/
|
||||
# max_tokens: 4096
|
||||
# temperature: 1.0
|
||||
# supports_vision: true
|
||||
# supports_thinking: true
|
||||
# - name: zai-org/GLM-5-FP8
|
||||
# display_name: zai-org/GLM-5-FP8
|
||||
# use: langchain_openai:ChatOpenAI
|
||||
# model: zai-org/GLM-5-FP8
|
||||
# api_key: modalresearch_gWu46FDpVLYZWPVEA3b_2fq8Jw2UJiKMgEm_Rxmn1SA
|
||||
# base_url: https://api.us-west-2.modal.direct/v1
|
||||
# request_timeout: 600.0
|
||||
# max_retries: 2
|
||||
# max_tokens: 4096
|
||||
# temperature: 1.0
|
||||
# supports_vision: true
|
||||
# supports_thinking: true
|
||||
- name: deepseek-ai/DeepSeek-V4-Flash
|
||||
display_name: deepseek-ai/DeepSeek-V4-Flash
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: deepseek-v4-pro
|
||||
api_key: sk-e1c22b7931dc446f8b67e57878edd105
|
||||
base_url: https://api.deepseek.com
|
||||
request_timeout: 600.0
|
||||
max_retries: 2
|
||||
max_tokens: 4096
|
||||
temperature: 1.0
|
||||
supports_vision: false
|
||||
supports_thinking: true
|
||||
- name: MiniMax-M2.7
|
||||
display_name: MiniMax-M2.7
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: MiniMax-M2.7
|
||||
api_key: sk-cp-mrsAGtzDpUl0TpOBPVoiRG_pWF1kDDN9_mDAnONBs6HIPcNUexSKtSbx2QOxfKBvJT6Xw6hDTV6hdkJYFWfNA8UtHQ08TiGJiosdRoW6q1sGb6Cb7D85NHk
|
||||
base_url: https://api.minimaxi.com/v1
|
||||
max_tokens: 4096
|
||||
temperature: 1.0
|
||||
supports_vision: true
|
||||
supports_thinking: true
|
||||
- name: glm-5.1
|
||||
display_name: glm-5.1
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: glm-5.1
|
||||
api_key: 75121e4bed7640d5b71ac95cf8b4310a.RA3GyRBAR3kv7VrR
|
||||
base_url: https://open.bigmodel.cn/api/paas/v4/
|
||||
max_tokens: 4096
|
||||
temperature: 1.0
|
||||
supports_vision: true
|
||||
supports_thinking: true
|
||||
- name: deepseek-ai/DeepSeek-OCR
|
||||
display_name: deepseek-ai/DeepSeek-OCR
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: deepseek-ai/DeepSeek-OCR
|
||||
api_key: sk-beshfxfxfrllsczpqfrgjbugfbpzxgqgedbehceytpcvvtyb
|
||||
base_url: https://api.siliconflow.cn/v1/
|
||||
max_tokens: 4096
|
||||
temperature: 1.0
|
||||
supports_vision: true
|
||||
supports_thinking: true
|
||||
- name: zai-org/GLM-5-FP8
|
||||
display_name: zai-org/GLM-5-FP8
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: zai-org/GLM-5-FP8
|
||||
api_key: modalresearch_gWu46FDpVLYZWPVEA3b_2fq8Jw2UJiKMgEm_Rxmn1SA
|
||||
base_url: https://api.us-west-2.modal.direct/v1
|
||||
request_timeout: 600.0
|
||||
max_retries: 2
|
||||
max_tokens: 4096
|
||||
temperature: 1.0
|
||||
supports_vision: true
|
||||
supports_thinking: true
|
||||
- name: Qwen/Qwen3.6-27B
|
||||
display_name: Qwen/Qwen3.6-27B
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: Qwen/Qwen3.6-27B
|
||||
model: deepseek-ai/DeepSeek-V4-Flash
|
||||
api_key: sk-ighylafootabtuthwueirmoycccyqtdgbifyhozduvolkdyt
|
||||
base_url: https://api.siliconflow.cn/v1/
|
||||
max_tokens: 4096
|
||||
@@ -122,6 +122,26 @@ uploads:
|
||||
auto_convert_documents: true
|
||||
pdf_converter: auto
|
||||
|
||||
# Chrome 插件「当前网页发送到 Deer」入口。默认关闭,打开后才启用
|
||||
# /api/browser-context/capture。它只保存当前页 Markdown 到线程 uploads,
|
||||
# 不会改变普通聊天、上传或工具调用流程。
|
||||
browser_context:
|
||||
enabled: true
|
||||
max_content_chars: 200000
|
||||
max_title_chars: 300
|
||||
max_url_chars: 2048
|
||||
filename_prefix: webpage
|
||||
route_base: /page/workspace
|
||||
task_result_timeout_seconds: 60
|
||||
task_poll_interval_seconds: 1
|
||||
task_lease_seconds: 120
|
||||
tool_result_max_chars: 60000
|
||||
# 交互式页面操作(browser_act:点击/输入/滚动/等待/跳转)。关闭后只保留只读抓取。
|
||||
actions_enabled: true
|
||||
action_result_timeout_seconds: 45
|
||||
max_observed_elements: 150
|
||||
observation_text_max_chars: 12000
|
||||
|
||||
title:
|
||||
enabled: true
|
||||
max_words: 6
|
||||
@@ -139,6 +159,7 @@ title:
|
||||
|
||||
tool_groups:
|
||||
- name: web
|
||||
- name: browser
|
||||
- name: file:read
|
||||
- name: file:write
|
||||
- name: bash
|
||||
@@ -154,6 +175,14 @@ tools:
|
||||
use: deerflow.community.jina_ai.tools:web_fetch_tool
|
||||
timeout: 15
|
||||
|
||||
- name: browser_fetch_page
|
||||
group: browser
|
||||
use: deerflow.tools.builtins.browser_context_tool:browser_fetch_page_tool
|
||||
|
||||
- name: browser_act
|
||||
group: browser
|
||||
use: deerflow.tools.builtins.browser_act_tool:browser_act_tool
|
||||
|
||||
- name: ls
|
||||
group: file:read
|
||||
use: deerflow.sandbox.tools:ls_tool
|
||||
@@ -223,7 +252,7 @@ auth_login:
|
||||
|
||||
database:
|
||||
backend: sqlite
|
||||
sqlite_dir: .deer-flow/data
|
||||
sqlite_dir: /app/backend/.deer-flow/data
|
||||
pool_size: 20 # 基础连接池(postgres/mysql 生效;sqlite 忽略)
|
||||
max_overflow: 20 # 高峰期可临时超出 pool_size 的额外连接数,真实上限 = pool_size + max_overflow
|
||||
pool_timeout: 30 # 池满时等待空闲连接的秒数,超时报错
|
||||
@@ -231,7 +260,7 @@ database:
|
||||
|
||||
checkpointer:
|
||||
type: sqlite
|
||||
connection_string: .deer-flow/data/deerflow.db
|
||||
connection_string: /app/backend/.deer-flow/data/deerflow.db
|
||||
|
||||
# 记忆子系统:跨会话持久化用户画像与 agent 私有记忆。
|
||||
# 三层覆盖优先级(高->低):
|
||||
@@ -313,3 +342,39 @@ log_system:
|
||||
enabled: true
|
||||
url: "https://your-log-system.example.com"
|
||||
label: "日志系统"
|
||||
|
||||
# 产品内知识库 —— obsidian-wiki 兼容 vault。知识元数据写入数据库,正文
|
||||
# content_md 写入 Markdown vault(index.md/log.md/hot.md/.manifest.json/frontmatter/
|
||||
# wikilinks/wiki-export 图谱产物)。知识库为全局共享,不做 user_id 隔离,
|
||||
# 写入时记录 created_by / updated_by 作为审计。
|
||||
knowledge:
|
||||
enabled: true
|
||||
provider: obsidian_wiki
|
||||
vault_dir: data/knowledge/obsidian-vault
|
||||
project_name: zncm
|
||||
auto_ingest_enabled: false
|
||||
auto_ingest_min_chars: 300
|
||||
# 用大模型按 wiki-capture 规范提炼陈述式知识(而非堆砌对话原文)。
|
||||
# 关闭则回退到规则抽取;extract_model_name 留空时用 models[0]。
|
||||
llm_extract_enabled: true
|
||||
extract_model_name: ""
|
||||
extract_max_input_chars: 12000
|
||||
max_thread_messages: 80
|
||||
max_source_items: 20
|
||||
rag_enabled: false
|
||||
rag_limit: 5
|
||||
# 阶段三:写入前脱敏(token/密钥/手机号/身份证/邮箱);重复检测相似度阈值。
|
||||
sensitive_redaction_enabled: true
|
||||
dedup_similarity_threshold: 0.82
|
||||
export:
|
||||
enabled: true
|
||||
graph_html_enabled: true
|
||||
graph_json_enabled: true
|
||||
query:
|
||||
use_hot_md: true
|
||||
use_index_md: true
|
||||
use_frontmatter_scan: true
|
||||
embedding:
|
||||
enabled: false
|
||||
provider: local
|
||||
model: ""
|
||||
@@ -299,6 +299,16 @@ def _build_middlewares(
|
||||
if not is_bootstrap and not is_writing_setup:
|
||||
middlewares.append(MemoryMiddleware(agent_name=agent_name, memory_config=resolved_app_config.memory, injection_enabled=memory_injection_enabled))
|
||||
|
||||
# KnowledgeRagMiddleware — inject knowledge-base recall (phase 2). Gated
|
||||
# per-conversation by the ``knowledge_rag_enabled`` runtime flag, defaulting
|
||||
# to the global ``knowledge.rag_enabled`` config. No-op when knowledge is
|
||||
# disabled. Skipped in bootstrap/writing-setup like MemoryMiddleware.
|
||||
knowledge_cfg = getattr(resolved_app_config, "knowledge", None)
|
||||
if not is_bootstrap and not is_writing_setup and knowledge_cfg is not None and knowledge_cfg.enabled:
|
||||
from deerflow.agents.middlewares.knowledge_rag_middleware import KnowledgeRagMiddleware
|
||||
|
||||
middlewares.append(KnowledgeRagMiddleware(limit=knowledge_cfg.rag_limit, default_enabled=knowledge_cfg.rag_enabled))
|
||||
|
||||
# Add ViewImageMiddleware only for the NATIVE-vision path (main model
|
||||
# ingests raw image data). In delegated mode the view_image tool already
|
||||
# returns a text description and stores nothing in `viewed_images`, so the
|
||||
|
||||
+190
@@ -0,0 +1,190 @@
|
||||
"""KnowledgeRagMiddleware — inject knowledge-base recall before the model call.
|
||||
|
||||
Phase 2 of the product knowledge base. When enabled (per-conversation runtime
|
||||
flag ``knowledge_rag_enabled``, falling back to the global
|
||||
``knowledge.rag_enabled`` config), the latest user question is used to retrieve
|
||||
the most relevant knowledge notes, which are injected as a transient
|
||||
``<knowledge-context>`` system message — not persisted to state, so it never
|
||||
leaks to the frontend transcript.
|
||||
|
||||
Retrieval is hybrid (vector + keyword) when an embedding endpoint is
|
||||
configured, else keyword/wiki-query layered search. Failures are swallowed: RAG
|
||||
must never break a conversation.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Any, override
|
||||
|
||||
from langchain.agents import AgentState
|
||||
from langchain.agents.middleware import AgentMiddleware
|
||||
from langchain_core.messages import HumanMessage
|
||||
from langgraph.runtime import Runtime
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_TAG = "knowledge-context"
|
||||
|
||||
|
||||
def _coerce_text(content: Any) -> str:
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
if isinstance(content, list):
|
||||
parts = []
|
||||
for block in content:
|
||||
if isinstance(block, str):
|
||||
parts.append(block)
|
||||
elif isinstance(block, dict) and block.get("type") == "text":
|
||||
parts.append(str(block.get("text", "")))
|
||||
return " ".join(parts)
|
||||
return str(content or "")
|
||||
|
||||
|
||||
def _latest_human_text(messages: list) -> str:
|
||||
for msg in reversed(messages or []):
|
||||
if isinstance(msg, HumanMessage):
|
||||
return _coerce_text(msg.content).strip()
|
||||
return ""
|
||||
|
||||
|
||||
class KnowledgeRagMiddleware(AgentMiddleware[AgentState]):
|
||||
"""Inject knowledge-base recall as transient system context."""
|
||||
|
||||
state_schema = AgentState
|
||||
|
||||
def __init__(self, *, limit: int = 5, default_enabled: bool = False) -> None:
|
||||
super().__init__()
|
||||
self._limit = limit
|
||||
self._default_enabled = default_enabled
|
||||
self._service = None # lazy
|
||||
|
||||
def _get_service(self):
|
||||
if self._service is not None:
|
||||
return self._service
|
||||
try:
|
||||
from deerflow.config import get_app_config
|
||||
from deerflow.knowledge import make_knowledge_service
|
||||
from deerflow.persistence.engine import get_session_factory
|
||||
|
||||
config = get_app_config()
|
||||
self._service = make_knowledge_service(get_session_factory(), config.knowledge)
|
||||
except Exception:
|
||||
logger.exception("KnowledgeRagMiddleware: failed to build knowledge service")
|
||||
self._service = None
|
||||
return self._service
|
||||
|
||||
def _enabled_for(self, request: Any) -> bool:
|
||||
runtime = getattr(request, "runtime", None)
|
||||
if runtime is not None and getattr(runtime, "context", None):
|
||||
flag = runtime.context.get("knowledge_rag_enabled")
|
||||
if flag is not None:
|
||||
return bool(flag)
|
||||
return self._default_enabled
|
||||
|
||||
async def _build_context_message(self, request: Any) -> HumanMessage | None:
|
||||
if not self._enabled_for(request):
|
||||
return None
|
||||
service = self._get_service()
|
||||
if service is None:
|
||||
return None
|
||||
query = _latest_human_text(getattr(request, "messages", None) or [])
|
||||
if not query:
|
||||
return None
|
||||
try:
|
||||
result = await service.build_rag_context(query, limit=self._limit)
|
||||
except Exception:
|
||||
logger.exception("KnowledgeRagMiddleware: recall failed (non-fatal)")
|
||||
return None
|
||||
text = result.get("text") if isinstance(result, dict) else ""
|
||||
if not text:
|
||||
return None
|
||||
fenced = (
|
||||
f"<{_TAG}>\n"
|
||||
"[系统提示:以下是知识库召回的背景资料,不是用户的新输入。回答时优先参考,"
|
||||
"并在使用到时标注来源 note_id。]\n\n"
|
||||
f"{text}\n"
|
||||
f"</{_TAG}>"
|
||||
)
|
||||
return HumanMessage(content=fenced)
|
||||
|
||||
@override
|
||||
async def awrap_model_call(self, request: Any, handler: Any) -> Any:
|
||||
msg = await self._build_context_message(request)
|
||||
if msg is not None:
|
||||
request = request.override(messages=[*request.messages, msg])
|
||||
return await handler(request)
|
||||
|
||||
@override
|
||||
def wrap_model_call(self, request: Any, handler: Any) -> Any:
|
||||
# Sync path: best-effort. The gateway runtime is async, so this is rarely
|
||||
# hit; run the async recall in a private loop when there is none.
|
||||
if not self._enabled_for(request):
|
||||
return handler(request)
|
||||
try:
|
||||
loop = asyncio.get_event_loop()
|
||||
if loop.is_running():
|
||||
# Cannot block a running loop from sync code; skip injection.
|
||||
return handler(request)
|
||||
msg = loop.run_until_complete(self._build_context_message(request))
|
||||
except Exception:
|
||||
msg = None
|
||||
if msg is not None:
|
||||
request = request.override(messages=[*request.messages, msg])
|
||||
return handler(request)
|
||||
|
||||
# ---- after_agent: enqueue background auto-sedimentation -------------
|
||||
def _resolve_thread_id(self, runtime: Runtime | None) -> str | None:
|
||||
if runtime is not None and getattr(runtime, "context", None):
|
||||
tid = runtime.context.get("thread_id")
|
||||
if tid:
|
||||
return str(tid)
|
||||
try:
|
||||
from langgraph.config import get_config
|
||||
|
||||
return get_config().get("configurable", {}).get("thread_id")
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
@override
|
||||
def after_agent(self, state: AgentState, runtime: Runtime) -> dict | None:
|
||||
"""When auto-ingest is on, hand the finished turn to the background queue."""
|
||||
try:
|
||||
from deerflow.config import get_app_config
|
||||
|
||||
knowledge_cfg = getattr(get_app_config(), "knowledge", None)
|
||||
if knowledge_cfg is None or not knowledge_cfg.enabled or not knowledge_cfg.auto_ingest_enabled:
|
||||
return None
|
||||
|
||||
from deerflow.knowledge.auto_ingest import AutoIngestItem, get_auto_ingest_queue
|
||||
|
||||
queue = get_auto_ingest_queue()
|
||||
if queue is None:
|
||||
return None
|
||||
|
||||
messages = state.get("messages") if isinstance(state, dict) else getattr(state, "messages", None)
|
||||
if not messages:
|
||||
return None
|
||||
|
||||
from deerflow.runtime.references import build_reference_batches
|
||||
from deerflow.runtime.serialization import serialize_lc_object
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
|
||||
serialized = [serialize_lc_object(m) for m in messages]
|
||||
thread_id = self._resolve_thread_id(runtime)
|
||||
if not thread_id:
|
||||
return None
|
||||
title = state.get("title") if isinstance(state, dict) else getattr(state, "title", None)
|
||||
queue.enqueue(
|
||||
AutoIngestItem(
|
||||
thread_id=thread_id,
|
||||
user_id=get_effective_user_id(),
|
||||
messages=serialized,
|
||||
thread_title=title if isinstance(title, str) else None,
|
||||
reference_batches=build_reference_batches(serialized),
|
||||
)
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("KnowledgeRagMiddleware.after_agent enqueue failed (non-fatal)")
|
||||
return None
|
||||
@@ -0,0 +1,2 @@
|
||||
"""Browser-backed page context helpers."""
|
||||
|
||||
@@ -0,0 +1,303 @@
|
||||
"""Small file-backed task queue for browser page fetch requests."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
|
||||
from deerflow.config.runtime_paths import runtime_home
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_LOCK = threading.RLock()
|
||||
_TASK_ID_RE = re.compile(r"^[a-f0-9]{32}$")
|
||||
_TERMINAL_STATUSES = {"done", "error", "expired"}
|
||||
|
||||
|
||||
def tasks_dir() -> Path:
|
||||
"""Return the browser-context task directory, creating it if needed."""
|
||||
|
||||
path = runtime_home() / "browser-context" / "tasks"
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
return path
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(UTC)
|
||||
|
||||
|
||||
def _iso(value: datetime | None = None) -> str:
|
||||
return (value or _now()).isoformat()
|
||||
|
||||
|
||||
def _parse_iso(value: Any) -> datetime | None:
|
||||
if not isinstance(value, str) or not value:
|
||||
return None
|
||||
try:
|
||||
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
return None
|
||||
if parsed.tzinfo is None:
|
||||
return parsed.replace(tzinfo=UTC)
|
||||
return parsed.astimezone(UTC)
|
||||
|
||||
|
||||
def _task_path(task_id: str) -> Path | None:
|
||||
if not _TASK_ID_RE.fullmatch(task_id or ""):
|
||||
return None
|
||||
return tasks_dir() / f"{task_id}.json"
|
||||
|
||||
|
||||
def _read_task(path: Path) -> dict[str, Any] | None:
|
||||
try:
|
||||
with path.open("r", encoding="utf-8") as fh:
|
||||
data = json.load(fh)
|
||||
except FileNotFoundError:
|
||||
return None
|
||||
except Exception:
|
||||
logger.warning("Ignoring unreadable browser task file: %s", path, exc_info=True)
|
||||
return None
|
||||
return data if isinstance(data, dict) else None
|
||||
|
||||
|
||||
def _write_task(path: Path, data: dict[str, Any]) -> None:
|
||||
tmp = path.with_name(f"{path.name}.{uuid4().hex}.tmp")
|
||||
with tmp.open("w", encoding="utf-8") as fh:
|
||||
json.dump(data, fh, ensure_ascii=False, separators=(",", ":"))
|
||||
tmp.replace(path)
|
||||
|
||||
|
||||
def _public_task(data: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
"id": data.get("id"),
|
||||
"kind": data.get("kind") or "fetch",
|
||||
"url": data.get("url") or "",
|
||||
"instruction": data.get("instruction") or "",
|
||||
"session_id": data.get("session_id") or "",
|
||||
"action": data.get("action") or None,
|
||||
"created_at": data.get("created_at") or "",
|
||||
"updated_at": data.get("updated_at") or "",
|
||||
"max_content_chars": int(data.get("max_content_chars") or 200_000),
|
||||
}
|
||||
|
||||
|
||||
def _expire_if_needed(path: Path, data: dict[str, Any], now: datetime) -> dict[str, Any]:
|
||||
if data.get("status") in _TERMINAL_STATUSES:
|
||||
return data
|
||||
expires_at = _parse_iso(data.get("expires_at"))
|
||||
if expires_at is not None and now >= expires_at:
|
||||
data = dict(data)
|
||||
data["status"] = "expired"
|
||||
data["error"] = "Browser task expired before the extension returned a result."
|
||||
data["updated_at"] = _iso(now)
|
||||
_write_task(path, data)
|
||||
return data
|
||||
|
||||
|
||||
def prune_browser_tasks(max_age_seconds: int = 86_400) -> int:
|
||||
"""Delete old terminal tasks and return the number removed."""
|
||||
|
||||
cutoff = _now() - timedelta(seconds=max(60, int(max_age_seconds)))
|
||||
removed = 0
|
||||
with _LOCK:
|
||||
for path in tasks_dir().glob("*.json"):
|
||||
data = _read_task(path)
|
||||
if not data or data.get("status") not in _TERMINAL_STATUSES:
|
||||
continue
|
||||
updated_at = _parse_iso(data.get("updated_at")) or _parse_iso(data.get("created_at"))
|
||||
if updated_at is None or updated_at < cutoff:
|
||||
try:
|
||||
path.unlink()
|
||||
removed += 1
|
||||
except OSError:
|
||||
logger.debug("Failed to delete old browser task: %s", path, exc_info=True)
|
||||
return removed
|
||||
|
||||
|
||||
def create_browser_task(
|
||||
*,
|
||||
user_id: str,
|
||||
url: str,
|
||||
instruction: str = "",
|
||||
timeout_seconds: int = 60,
|
||||
max_content_chars: int = 200_000,
|
||||
metadata: dict[str, Any] | None = None,
|
||||
kind: str = "fetch",
|
||||
session_id: str = "",
|
||||
action: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Create a browser task for the current user's extension to pick up.
|
||||
|
||||
``kind`` is ``"fetch"`` for one-shot page captures (the legacy behavior) or
|
||||
``"act"`` for an interactive observe/act step driven by ``action`` within a
|
||||
long-lived ``session_id`` (a tab kept open by the extension).
|
||||
"""
|
||||
|
||||
now = _now()
|
||||
task_id = uuid4().hex
|
||||
task = {
|
||||
"id": task_id,
|
||||
"user_id": str(user_id or "default"),
|
||||
"status": "pending",
|
||||
"kind": kind if kind in {"fetch", "act"} else "fetch",
|
||||
"url": url,
|
||||
"instruction": instruction or "",
|
||||
"session_id": str(session_id or ""),
|
||||
"action": action or None,
|
||||
"created_at": _iso(now),
|
||||
"updated_at": _iso(now),
|
||||
"expires_at": _iso(now + timedelta(seconds=max(5, int(timeout_seconds)))),
|
||||
"max_content_chars": max(1_000, int(max_content_chars)),
|
||||
"worker_id": None,
|
||||
"claimed_at": None,
|
||||
"result": None,
|
||||
"error": None,
|
||||
"metadata": metadata or {},
|
||||
}
|
||||
path = tasks_dir() / f"{task_id}.json"
|
||||
with _LOCK:
|
||||
_write_task(path, task)
|
||||
return task
|
||||
|
||||
|
||||
def get_browser_task(*, task_id: str, user_id: str) -> dict[str, Any] | None:
|
||||
"""Read a task if it belongs to the requested user."""
|
||||
|
||||
path = _task_path(task_id)
|
||||
if path is None:
|
||||
return None
|
||||
with _LOCK:
|
||||
data = _read_task(path)
|
||||
if not data or str(data.get("user_id") or "") != str(user_id or "default"):
|
||||
return None
|
||||
return _expire_if_needed(path, data, _now())
|
||||
|
||||
|
||||
def claim_next_browser_task(
|
||||
*,
|
||||
user_id: str,
|
||||
worker_id: str,
|
||||
lease_seconds: int = 120,
|
||||
) -> dict[str, Any] | None:
|
||||
"""Claim the oldest pending task for a user."""
|
||||
|
||||
now = _now()
|
||||
user_id = str(user_id or "default")
|
||||
worker_id = (worker_id or "chrome-extension").strip()[:128] or "chrome-extension"
|
||||
lease_seconds = max(10, int(lease_seconds))
|
||||
|
||||
with _LOCK:
|
||||
paths = sorted(tasks_dir().glob("*.json"), key=lambda p: p.stat().st_mtime)
|
||||
for path in paths:
|
||||
data = _read_task(path)
|
||||
if not data or str(data.get("user_id") or "") != user_id:
|
||||
continue
|
||||
|
||||
data = _expire_if_needed(path, data, now)
|
||||
status = data.get("status")
|
||||
if status == "running":
|
||||
updated_at = _parse_iso(data.get("updated_at")) or now
|
||||
if (now - updated_at).total_seconds() <= lease_seconds:
|
||||
continue
|
||||
status = "pending"
|
||||
data = dict(data)
|
||||
data["status"] = status
|
||||
|
||||
if status != "pending":
|
||||
continue
|
||||
|
||||
data = dict(data)
|
||||
data["status"] = "running"
|
||||
data["worker_id"] = worker_id
|
||||
data["claimed_at"] = _iso(now)
|
||||
data["updated_at"] = _iso(now)
|
||||
_write_task(path, data)
|
||||
return _public_task(data)
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def complete_browser_task(
|
||||
*,
|
||||
task_id: str,
|
||||
user_id: str,
|
||||
title: str = "",
|
||||
url: str = "",
|
||||
content: str = "",
|
||||
content_type: str = "markdown",
|
||||
error: str = "",
|
||||
metadata: dict[str, Any] | None = None,
|
||||
observation: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any] | None:
|
||||
"""Mark a browser task complete or failed.
|
||||
|
||||
For ``act`` tasks the extension returns a structured ``observation`` (final
|
||||
URL/title, interactive element list, page text) instead of a captured
|
||||
markdown document; it is stored alongside the result verbatim.
|
||||
"""
|
||||
|
||||
path = _task_path(task_id)
|
||||
if path is None:
|
||||
return None
|
||||
|
||||
now = _now()
|
||||
with _LOCK:
|
||||
data = _read_task(path)
|
||||
if not data or str(data.get("user_id") or "") != str(user_id or "default"):
|
||||
return None
|
||||
|
||||
data = dict(data)
|
||||
if error:
|
||||
data["status"] = "error"
|
||||
data["error"] = str(error).strip()[:4000]
|
||||
data["result"] = None
|
||||
else:
|
||||
raw_content = str(content or "").strip()
|
||||
max_content_chars = max(1_000, int(data.get("max_content_chars") or 200_000))
|
||||
truncated = len(raw_content) > max_content_chars
|
||||
data["status"] = "done"
|
||||
data["error"] = None
|
||||
data["result"] = {
|
||||
"title": str(title or "").strip()[:500],
|
||||
"url": str(url or data.get("url") or "").strip()[:4096],
|
||||
"content": raw_content[:max_content_chars] if truncated else raw_content,
|
||||
"content_type": content_type if content_type in {"markdown", "text"} else "markdown",
|
||||
"truncated": truncated,
|
||||
"observation": observation if isinstance(observation, dict) else None,
|
||||
"captured_at": _iso(now),
|
||||
"metadata": metadata or {},
|
||||
}
|
||||
data["updated_at"] = _iso(now)
|
||||
_write_task(path, data)
|
||||
return data
|
||||
|
||||
|
||||
async def wait_for_browser_task_result(
|
||||
*,
|
||||
task_id: str,
|
||||
user_id: str,
|
||||
timeout_seconds: int,
|
||||
poll_interval_seconds: float = 1.0,
|
||||
) -> dict[str, Any] | None:
|
||||
"""Wait for a browser task to reach a terminal state."""
|
||||
|
||||
deadline = time.monotonic() + max(1, int(timeout_seconds))
|
||||
interval = max(0.25, float(poll_interval_seconds))
|
||||
while True:
|
||||
task = get_browser_task(task_id=task_id, user_id=user_id)
|
||||
if task is None:
|
||||
return None
|
||||
if task.get("status") in _TERMINAL_STATUSES:
|
||||
return task
|
||||
if time.monotonic() >= deadline:
|
||||
return task
|
||||
await asyncio.sleep(interval)
|
||||
|
||||
@@ -10,11 +10,13 @@ from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
from deerflow.config.acp_config import ACPAgentConfig, load_acp_config_from_dict
|
||||
from deerflow.config.agents_api_config import AgentsApiConfig, load_agents_api_config_from_dict
|
||||
from deerflow.config.browser_context_config import BrowserContextConfig
|
||||
from deerflow.config.checkpointer_config import CheckpointerConfig, load_checkpointer_config_from_dict
|
||||
from deerflow.config.database_config import DatabaseConfig
|
||||
from deerflow.config.extensions_config import ExtensionsConfig
|
||||
from deerflow.config.gateway_config import GatewayConfig
|
||||
from deerflow.config.guardrails_config import GuardrailsConfig, load_guardrails_config_from_dict
|
||||
from deerflow.config.knowledge_config import KnowledgeConfig
|
||||
from deerflow.config.log_system_config import LogSystemConfig
|
||||
from deerflow.config.login_config import AuthLoginConfig
|
||||
from deerflow.config.memory_config import MemoryConfig, load_memory_config_from_dict
|
||||
@@ -133,6 +135,8 @@ class AppConfig(BaseModel):
|
||||
run_events: RunEventsConfig = Field(default_factory=RunEventsConfig, description="Run event storage configuration")
|
||||
checkpointer: CheckpointerConfig | None = Field(default=None, description="Checkpointer configuration")
|
||||
stream_bridge: StreamBridgeConfig | None = Field(default=None, description="Stream bridge configuration")
|
||||
browser_context: BrowserContextConfig = Field(default_factory=BrowserContextConfig, description="Optional browser page capture integration")
|
||||
knowledge: KnowledgeConfig = Field(default_factory=KnowledgeConfig, description="Product knowledge base (obsidian-wiki compatible vault) configuration")
|
||||
|
||||
@classmethod
|
||||
def resolve_config_path(cls, config_path: str | None = None) -> Path:
|
||||
|
||||
+74
@@ -0,0 +1,74 @@
|
||||
"""Configuration for browser page context capture."""
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class BrowserContextConfig(BaseModel):
|
||||
"""Controls the optional Chrome-extension page capture integration."""
|
||||
|
||||
enabled: bool = Field(
|
||||
default=False,
|
||||
description="Enable /api/browser-context endpoints for Chrome extension page capture.",
|
||||
)
|
||||
max_content_chars: int = Field(
|
||||
default=200_000,
|
||||
ge=1_000,
|
||||
description="Maximum captured page text stored per request. Longer content is truncated.",
|
||||
)
|
||||
max_title_chars: int = Field(
|
||||
default=300,
|
||||
ge=20,
|
||||
description="Maximum title length accepted from the browser extension.",
|
||||
)
|
||||
max_url_chars: int = Field(
|
||||
default=2_048,
|
||||
ge=100,
|
||||
description="Maximum source URL length accepted from the browser extension.",
|
||||
)
|
||||
filename_prefix: str = Field(
|
||||
default="webpage",
|
||||
description="Prefix used for Markdown files created from captured pages.",
|
||||
)
|
||||
route_base: str = Field(
|
||||
default="/page/workspace",
|
||||
description="Frontend route base used when returning chat paths to the extension.",
|
||||
)
|
||||
task_result_timeout_seconds: int = Field(
|
||||
default=60,
|
||||
ge=5,
|
||||
description="Maximum time the browser_fetch_page tool waits for the extension result.",
|
||||
)
|
||||
task_poll_interval_seconds: float = Field(
|
||||
default=1.0,
|
||||
ge=0.25,
|
||||
description="How often the backend tool checks for a returned browser task result.",
|
||||
)
|
||||
task_lease_seconds: int = Field(
|
||||
default=120,
|
||||
ge=10,
|
||||
description="How long a claimed browser task is reserved before another extension may retry it.",
|
||||
)
|
||||
tool_result_max_chars: int = Field(
|
||||
default=60_000,
|
||||
ge=1_000,
|
||||
description="Maximum page characters returned from browser_fetch_page into the model context.",
|
||||
)
|
||||
actions_enabled: bool = Field(
|
||||
default=True,
|
||||
description="Enable the interactive browser_act tool (click/type/scroll/wait/navigate on the user's page).",
|
||||
)
|
||||
action_result_timeout_seconds: int = Field(
|
||||
default=45,
|
||||
ge=5,
|
||||
description="Maximum time the browser_act tool waits for the extension to perform one step and return an observation.",
|
||||
)
|
||||
max_observed_elements: int = Field(
|
||||
default=150,
|
||||
ge=10,
|
||||
description="Maximum number of interactive elements returned in a browser_act observation.",
|
||||
)
|
||||
observation_text_max_chars: int = Field(
|
||||
default=12_000,
|
||||
ge=500,
|
||||
description="Maximum page text characters returned alongside a browser_act observation.",
|
||||
)
|
||||
+110
-110
@@ -3,7 +3,7 @@
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any, Literal
|
||||
from typing import Any, Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
@@ -53,115 +53,115 @@ class McpServerConfig(BaseModel):
|
||||
model_config = ConfigDict(extra="allow", populate_by_name=True)
|
||||
|
||||
|
||||
class SkillDisplayTableColumn(BaseModel):
|
||||
"""A configured table column for structured skill result display."""
|
||||
|
||||
label: str = Field(description="Column label shown in the frontend table")
|
||||
field: str = Field(description="Field name or dot path read from each result item")
|
||||
width: str | None = Field(default=None, description="Optional CSS width for this column")
|
||||
model_config = ConfigDict(extra="allow", populate_by_name=True)
|
||||
|
||||
|
||||
class SkillDisplayResultSetConfig(BaseModel):
|
||||
"""Display adapter for one result array inside a skill response."""
|
||||
|
||||
key: str = Field(description="Stable key for this result set")
|
||||
label: str | None = Field(default=None, description="Human-readable result set label")
|
||||
enabled: bool = Field(default=True, description="Whether this result set is enabled")
|
||||
mode: Literal["citation", "list", "table", "card", "custom", "none"] = Field(
|
||||
default="citation",
|
||||
description="How the frontend should render this result set",
|
||||
)
|
||||
result_path: str = Field(default="results", description="Dot path to the array of result items", alias="resultPath")
|
||||
title_field: str | None = Field(default=None, description="Field used as item title", alias="titleField")
|
||||
summary_field: str | None = Field(default=None, description="Field used as short summary/snippet", alias="summaryField")
|
||||
content_field: str | None = Field(default=None, description="Field used as full item content", alias="contentField")
|
||||
source_field: str | None = Field(default=None, description="Field used as source label", alias="sourceField")
|
||||
author_field: str | None = Field(default=None, description="Field used as author/byline label", alias="authorField")
|
||||
type_field: str | None = Field(default=None, description="Field used as source/type badge", alias="typeField")
|
||||
time_field: str | None = Field(default=None, description="Field used as publication/update time", alias="timeField")
|
||||
score_field: str | None = Field(default=None, description="Field used as relevance score", alias="scoreField")
|
||||
url_field: str | None = Field(default=None, description="Field containing a direct URL", alias="urlField")
|
||||
id_field: str | None = Field(default=None, description="Field containing an ID for template URLs", alias="idField")
|
||||
link_strategy: Literal["auto", "url", "id"] = Field(
|
||||
default="auto",
|
||||
description="Whether links use a URL field, an ID template, or automatic fallback",
|
||||
alias="linkStrategy",
|
||||
)
|
||||
link_template: str | None = Field(default=None, description="Template used when linkStrategy is id", alias="linkTemplate")
|
||||
citation_token: Literal["doubleBracket", "singleBracket"] = Field(
|
||||
default="singleBracket",
|
||||
description="Citation token style expected in model output",
|
||||
alias="citationToken",
|
||||
)
|
||||
max_items: int = Field(default=100, ge=1, le=100, description="Maximum displayed items", alias="maxItems")
|
||||
custom_instruction: str = Field(
|
||||
default="",
|
||||
description="User-authored instruction for custom rendering schema generation",
|
||||
alias="customInstruction",
|
||||
)
|
||||
table_columns: list[SkillDisplayTableColumn] = Field(
|
||||
default_factory=list,
|
||||
description="Columns used when mode is table",
|
||||
alias="tableColumns",
|
||||
)
|
||||
model_config = ConfigDict(extra="allow", populate_by_name=True)
|
||||
|
||||
|
||||
class SkillDisplayConfig(BaseModel):
|
||||
"""Frontend display adapter for structured skill results."""
|
||||
|
||||
enabled: bool = Field(default=False, description="Whether result display adaptation is enabled")
|
||||
mode: Literal["citation", "list", "table", "card", "custom", "none"] = Field(
|
||||
default="none",
|
||||
description="How the frontend should render this skill's structured results",
|
||||
)
|
||||
result_path: str = Field(default="results", description="Dot path to the array of result items", alias="resultPath")
|
||||
title_field: str | None = Field(default=None, description="Field used as item title", alias="titleField")
|
||||
summary_field: str | None = Field(default=None, description="Field used as short summary/snippet", alias="summaryField")
|
||||
content_field: str | None = Field(default=None, description="Field used as full item content", alias="contentField")
|
||||
source_field: str | None = Field(default=None, description="Field used as source label", alias="sourceField")
|
||||
author_field: str | None = Field(default=None, description="Field used as author/byline label", alias="authorField")
|
||||
type_field: str | None = Field(default=None, description="Field used as source/type badge", alias="typeField")
|
||||
time_field: str | None = Field(default=None, description="Field used as publication/update time", alias="timeField")
|
||||
score_field: str | None = Field(default=None, description="Field used as relevance score", alias="scoreField")
|
||||
url_field: str | None = Field(default=None, description="Field containing a direct URL", alias="urlField")
|
||||
id_field: str | None = Field(default=None, description="Field containing an ID for template URLs", alias="idField")
|
||||
link_strategy: Literal["auto", "url", "id"] = Field(
|
||||
default="auto",
|
||||
description="Whether links use a URL field, an ID template, or automatic fallback",
|
||||
alias="linkStrategy",
|
||||
)
|
||||
link_template: str | None = Field(default=None, description="Template used when linkStrategy is id", alias="linkTemplate")
|
||||
citation_token: Literal["doubleBracket", "singleBracket"] = Field(
|
||||
default="doubleBracket",
|
||||
description="Citation token style expected in model output",
|
||||
alias="citationToken",
|
||||
)
|
||||
max_items: int = Field(default=8, ge=1, le=100, description="Maximum displayed items", alias="maxItems")
|
||||
custom_instruction: str = Field(
|
||||
default="",
|
||||
description="User-authored instruction for custom rendering schema generation",
|
||||
alias="customInstruction",
|
||||
)
|
||||
table_columns: list[SkillDisplayTableColumn] = Field(
|
||||
default_factory=list,
|
||||
description="Columns used when mode is table",
|
||||
alias="tableColumns",
|
||||
)
|
||||
result_sets: list[SkillDisplayResultSetConfig] = Field(
|
||||
default_factory=list,
|
||||
description="Optional per-result-array display adapters",
|
||||
alias="resultSets",
|
||||
)
|
||||
model_config = ConfigDict(extra="allow", populate_by_name=True)
|
||||
|
||||
|
||||
class SkillStateConfig(BaseModel):
|
||||
"""Configuration for a single skill's state."""
|
||||
|
||||
enabled: bool = Field(default=True, description="Whether this skill is enabled")
|
||||
display: SkillDisplayConfig | None = Field(default=None, description="Optional frontend display adapter")
|
||||
class SkillDisplayTableColumn(BaseModel):
|
||||
"""A configured table column for structured skill result display."""
|
||||
|
||||
label: str = Field(description="Column label shown in the frontend table")
|
||||
field: str = Field(description="Field name or dot path read from each result item")
|
||||
width: str | None = Field(default=None, description="Optional CSS width for this column")
|
||||
model_config = ConfigDict(extra="allow", populate_by_name=True)
|
||||
|
||||
|
||||
class SkillDisplayResultSetConfig(BaseModel):
|
||||
"""Display adapter for one result array inside a skill response."""
|
||||
|
||||
key: str = Field(description="Stable key for this result set")
|
||||
label: str | None = Field(default=None, description="Human-readable result set label")
|
||||
enabled: bool = Field(default=True, description="Whether this result set is enabled")
|
||||
mode: Literal["citation", "list", "table", "card", "custom", "none"] = Field(
|
||||
default="citation",
|
||||
description="How the frontend should render this result set",
|
||||
)
|
||||
result_path: str = Field(default="results", description="Dot path to the array of result items", alias="resultPath")
|
||||
title_field: str | None = Field(default=None, description="Field used as item title", alias="titleField")
|
||||
summary_field: str | None = Field(default=None, description="Field used as short summary/snippet", alias="summaryField")
|
||||
content_field: str | None = Field(default=None, description="Field used as full item content", alias="contentField")
|
||||
source_field: str | None = Field(default=None, description="Field used as source label", alias="sourceField")
|
||||
author_field: str | None = Field(default=None, description="Field used as author/byline label", alias="authorField")
|
||||
type_field: str | None = Field(default=None, description="Field used as source/type badge", alias="typeField")
|
||||
time_field: str | None = Field(default=None, description="Field used as publication/update time", alias="timeField")
|
||||
score_field: str | None = Field(default=None, description="Field used as relevance score", alias="scoreField")
|
||||
url_field: str | None = Field(default=None, description="Field containing a direct URL", alias="urlField")
|
||||
id_field: str | None = Field(default=None, description="Field containing an ID for template URLs", alias="idField")
|
||||
link_strategy: Literal["auto", "url", "id"] = Field(
|
||||
default="auto",
|
||||
description="Whether links use a URL field, an ID template, or automatic fallback",
|
||||
alias="linkStrategy",
|
||||
)
|
||||
link_template: str | None = Field(default=None, description="Template used when linkStrategy is id", alias="linkTemplate")
|
||||
citation_token: Literal["doubleBracket", "singleBracket"] = Field(
|
||||
default="singleBracket",
|
||||
description="Citation token style expected in model output",
|
||||
alias="citationToken",
|
||||
)
|
||||
max_items: int = Field(default=100, ge=1, le=100, description="Maximum displayed items", alias="maxItems")
|
||||
custom_instruction: str = Field(
|
||||
default="",
|
||||
description="User-authored instruction for custom rendering schema generation",
|
||||
alias="customInstruction",
|
||||
)
|
||||
table_columns: list[SkillDisplayTableColumn] = Field(
|
||||
default_factory=list,
|
||||
description="Columns used when mode is table",
|
||||
alias="tableColumns",
|
||||
)
|
||||
model_config = ConfigDict(extra="allow", populate_by_name=True)
|
||||
|
||||
|
||||
class SkillDisplayConfig(BaseModel):
|
||||
"""Frontend display adapter for structured skill results."""
|
||||
|
||||
enabled: bool = Field(default=False, description="Whether result display adaptation is enabled")
|
||||
mode: Literal["citation", "list", "table", "card", "custom", "none"] = Field(
|
||||
default="none",
|
||||
description="How the frontend should render this skill's structured results",
|
||||
)
|
||||
result_path: str = Field(default="results", description="Dot path to the array of result items", alias="resultPath")
|
||||
title_field: str | None = Field(default=None, description="Field used as item title", alias="titleField")
|
||||
summary_field: str | None = Field(default=None, description="Field used as short summary/snippet", alias="summaryField")
|
||||
content_field: str | None = Field(default=None, description="Field used as full item content", alias="contentField")
|
||||
source_field: str | None = Field(default=None, description="Field used as source label", alias="sourceField")
|
||||
author_field: str | None = Field(default=None, description="Field used as author/byline label", alias="authorField")
|
||||
type_field: str | None = Field(default=None, description="Field used as source/type badge", alias="typeField")
|
||||
time_field: str | None = Field(default=None, description="Field used as publication/update time", alias="timeField")
|
||||
score_field: str | None = Field(default=None, description="Field used as relevance score", alias="scoreField")
|
||||
url_field: str | None = Field(default=None, description="Field containing a direct URL", alias="urlField")
|
||||
id_field: str | None = Field(default=None, description="Field containing an ID for template URLs", alias="idField")
|
||||
link_strategy: Literal["auto", "url", "id"] = Field(
|
||||
default="auto",
|
||||
description="Whether links use a URL field, an ID template, or automatic fallback",
|
||||
alias="linkStrategy",
|
||||
)
|
||||
link_template: str | None = Field(default=None, description="Template used when linkStrategy is id", alias="linkTemplate")
|
||||
citation_token: Literal["doubleBracket", "singleBracket"] = Field(
|
||||
default="doubleBracket",
|
||||
description="Citation token style expected in model output",
|
||||
alias="citationToken",
|
||||
)
|
||||
max_items: int = Field(default=8, ge=1, le=100, description="Maximum displayed items", alias="maxItems")
|
||||
custom_instruction: str = Field(
|
||||
default="",
|
||||
description="User-authored instruction for custom rendering schema generation",
|
||||
alias="customInstruction",
|
||||
)
|
||||
table_columns: list[SkillDisplayTableColumn] = Field(
|
||||
default_factory=list,
|
||||
description="Columns used when mode is table",
|
||||
alias="tableColumns",
|
||||
)
|
||||
result_sets: list[SkillDisplayResultSetConfig] = Field(
|
||||
default_factory=list,
|
||||
description="Optional per-result-array display adapters",
|
||||
alias="resultSets",
|
||||
)
|
||||
model_config = ConfigDict(extra="allow", populate_by_name=True)
|
||||
|
||||
|
||||
class SkillStateConfig(BaseModel):
|
||||
"""Configuration for a single skill's state."""
|
||||
|
||||
enabled: bool = Field(default=True, description="Whether this skill is enabled")
|
||||
display: SkillDisplayConfig | None = Field(default=None, description="Optional frontend display adapter")
|
||||
|
||||
|
||||
class ExtensionsConfig(BaseModel):
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
"""Configuration for the product knowledge base (obsidian-wiki compatible)."""
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class KnowledgeExportConfig(BaseModel):
|
||||
"""Knowledge graph export switches (``wiki-export`` compatible output)."""
|
||||
|
||||
enabled: bool = Field(default=True, description="Master switch for graph export.")
|
||||
graph_html_enabled: bool = Field(default=True, description="Generate wiki-export/graph.html.")
|
||||
graph_json_enabled: bool = Field(default=True, description="Generate wiki-export/graph.json.")
|
||||
|
||||
|
||||
class KnowledgeQueryConfig(BaseModel):
|
||||
"""Layered query strategy flags (``wiki-query`` style)."""
|
||||
|
||||
use_hot_md: bool = Field(default=True, description="Consult hot.md when searching.")
|
||||
use_index_md: bool = Field(default=True, description="Consult index.md when searching.")
|
||||
use_frontmatter_scan: bool = Field(default=True, description="Scan frontmatter when searching.")
|
||||
|
||||
|
||||
class KnowledgeEmbeddingConfig(BaseModel):
|
||||
"""Vector embedding config (phase 2).
|
||||
|
||||
Vector RAG is opt-in: set ``enabled: true`` and point ``base_url`` at an
|
||||
OpenAI-compatible ``/embeddings`` endpoint. When disabled or unreachable,
|
||||
search transparently falls back to keyword scoring, so the feature degrades
|
||||
gracefully on offline deployments without an embedding model.
|
||||
"""
|
||||
|
||||
enabled: bool = Field(default=False, description="Enable vector embeddings.")
|
||||
provider: str = Field(default="openai", description="Embedding provider (openai-compatible).")
|
||||
model: str = Field(default="", description="Embedding model name, e.g. text-embedding-3-small / bge-m3.")
|
||||
base_url: str = Field(default="", description="OpenAI-compatible base URL (…/v1). Empty disables vector mode.")
|
||||
api_key: str = Field(default="", description="API key for the embedding endpoint ($ENV resolved like other config).")
|
||||
dimensions: int | None = Field(default=None, description="Optional embedding dimensions hint sent to the endpoint.")
|
||||
batch_size: int = Field(default=32, description="Texts per embedding request.")
|
||||
timeout_seconds: int = Field(default=30, description="HTTP timeout per embedding request.")
|
||||
|
||||
|
||||
class KnowledgeConfig(BaseModel):
|
||||
"""Product knowledge base configuration.
|
||||
|
||||
The first release ships a global, shared, obsidian-wiki compatible vault.
|
||||
Knowledge metadata is mirrored into the database for fast listing/search,
|
||||
while ``content_md`` is written to a Markdown vault that stays compatible
|
||||
with ``ar9av/obsidian-wiki`` (index.md / log.md / hot.md / .manifest.json,
|
||||
frontmatter, wikilinks, wiki-export graph artifacts).
|
||||
"""
|
||||
|
||||
enabled: bool = Field(default=True, description="Enable the knowledge base feature.")
|
||||
provider: str = Field(default="obsidian_wiki", description="Knowledge organization provider.")
|
||||
vault_dir: str = Field(
|
||||
default="data/knowledge/obsidian-vault",
|
||||
description="Vault directory (relative to DEER_FLOW_HOME, or absolute).",
|
||||
)
|
||||
project_name: str = Field(default="zncm", description="Project namespace for projects/<name>/ pages.")
|
||||
auto_ingest_enabled: bool = Field(default=False, description="Auto-sediment conversations in the background (phase 3).")
|
||||
auto_ingest_min_chars: int = Field(default=300, description="Minimum characters before auto-ingest considers a turn.")
|
||||
llm_extract_enabled: bool = Field(default=True, description="Distill notes with an LLM (wiki-capture style) instead of the rule-based fallback.")
|
||||
extract_model_name: str | None = Field(default=None, description="Model (from models[]) used to distill knowledge. Falls back to models[0] when unset.")
|
||||
extract_max_input_chars: int = Field(default=12000, description="Max characters of conversation transcript fed to the extraction LLM.")
|
||||
max_thread_messages: int = Field(default=80, description="Maximum thread messages to scan when capturing.")
|
||||
max_source_items: int = Field(default=20, description="Maximum source items recorded per note.")
|
||||
related_notes_enabled: bool = Field(default=True, description="On capture, find existing notes on the same topic and link them under ## Related (real cross-references instead of placeholders).")
|
||||
related_notes_limit: int = Field(default=3, description="Maximum real related notes linked into each captured note.")
|
||||
multi_note_enabled: bool = Field(default=True, description="Allow the LLM extractor to split one conversation into multiple cross-linked knowledge notes.")
|
||||
rag_enabled: bool = Field(default=False, description="Enable RAG injection before agent runs (phase 2).")
|
||||
rag_limit: int = Field(default=5, description="Maximum knowledge snippets injected per query.")
|
||||
sensitive_redaction_enabled: bool = Field(default=True, description="Redact secrets/PII (tokens, keys, phone, id-card, email) before persisting notes (phase 3).")
|
||||
dedup_similarity_threshold: float = Field(default=0.82, description="Title/summary similarity above which two notes are flagged as duplicates (phase 3).")
|
||||
entity_extraction_enabled: bool = Field(default=True, description="Extract entities/relations during LLM distillation and feed them into the graph (phase 4).")
|
||||
export: KnowledgeExportConfig = Field(default_factory=KnowledgeExportConfig, description="Graph export configuration.")
|
||||
query: KnowledgeQueryConfig = Field(default_factory=KnowledgeQueryConfig, description="Layered query configuration.")
|
||||
embedding: KnowledgeEmbeddingConfig = Field(default_factory=KnowledgeEmbeddingConfig, description="Embedding configuration.")
|
||||
@@ -0,0 +1,41 @@
|
||||
"""Product knowledge base module (obsidian-wiki compatible).
|
||||
|
||||
Self-contained: knowledge ingestion, storage (DB mirror + Markdown vault),
|
||||
search and graph export all live here. The Gateway only depends on
|
||||
:class:`KnowledgeService` via :func:`make_knowledge_service`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from deerflow.knowledge.obsidian_adapter import ObsidianWikiAdapter
|
||||
from deerflow.knowledge.repository import KnowledgeRepository
|
||||
from deerflow.knowledge.service import EmptyThreadError, KnowledgeService
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
from deerflow.config.knowledge_config import KnowledgeConfig
|
||||
|
||||
__all__ = [
|
||||
"EmptyThreadError",
|
||||
"KnowledgeRepository",
|
||||
"KnowledgeService",
|
||||
"ObsidianWikiAdapter",
|
||||
"make_knowledge_service",
|
||||
]
|
||||
|
||||
|
||||
def make_knowledge_service(
|
||||
session_factory: async_sessionmaker[AsyncSession] | None,
|
||||
config: KnowledgeConfig,
|
||||
) -> KnowledgeService | None:
|
||||
"""Build the knowledge service.
|
||||
|
||||
Returns ``None`` when persistence is memory-only (no session factory) or
|
||||
the feature is disabled — the router then responds 503.
|
||||
"""
|
||||
if session_factory is None or not config.enabled:
|
||||
return None
|
||||
return KnowledgeService(KnowledgeRepository(session_factory), config)
|
||||
@@ -0,0 +1,125 @@
|
||||
"""Background auto-sediment queue (phase 3).
|
||||
|
||||
A single asyncio worker drains a bounded queue of finished conversations and
|
||||
sediments the worthwhile ones into the knowledge base — entirely off the chat
|
||||
request path, so it never adds latency to a reply. Enqueue happens from
|
||||
``KnowledgeRagMiddleware.after_agent`` when ``knowledge.auto_ingest_enabled``.
|
||||
|
||||
Value judgment is intentionally cheap (min assistant chars + dedup via
|
||||
``skip_existing``); the LLM still distills the kept conversations via the normal
|
||||
``capture_thread`` path.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import logging
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_QUEUE: AutoIngestQueue | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class _QueueUser:
|
||||
"""Minimal CurrentUser stand-in (only ``.id`` is needed for context)."""
|
||||
|
||||
id: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class AutoIngestItem:
|
||||
thread_id: str
|
||||
user_id: str
|
||||
messages: list[dict[str, Any]]
|
||||
thread_title: str | None = None
|
||||
reference_batches: list[dict[str, Any]] = field(default_factory=list)
|
||||
|
||||
|
||||
class AutoIngestQueue:
|
||||
"""Bounded background queue + single worker for auto-sedimentation."""
|
||||
|
||||
def __init__(self, service_provider: Callable[[], Any], *, min_chars: int = 300, maxsize: int = 256) -> None:
|
||||
self._provider = service_provider
|
||||
self._min_chars = min_chars
|
||||
self._queue: asyncio.Queue[AutoIngestItem] = asyncio.Queue(maxsize=maxsize)
|
||||
self._task: asyncio.Task | None = None
|
||||
|
||||
def start(self) -> None:
|
||||
if self._task is None or self._task.done():
|
||||
self._task = asyncio.create_task(self._worker(), name="knowledge-auto-ingest")
|
||||
logger.info("Knowledge auto-ingest worker started")
|
||||
|
||||
async def stop(self) -> None:
|
||||
if self._task is not None:
|
||||
self._task.cancel()
|
||||
with contextlib.suppress(asyncio.CancelledError):
|
||||
await self._task
|
||||
self._task = None
|
||||
|
||||
def enqueue(self, item: AutoIngestItem) -> None:
|
||||
try:
|
||||
self._queue.put_nowait(item)
|
||||
except asyncio.QueueFull:
|
||||
logger.warning("Knowledge auto-ingest queue full; dropping thread %s", item.thread_id)
|
||||
|
||||
def _worth_ingesting(self, item: AutoIngestItem) -> bool:
|
||||
chars = 0
|
||||
for m in item.messages:
|
||||
if str(m.get("type") or m.get("role") or "").lower() == "ai":
|
||||
content = m.get("content")
|
||||
chars += len(content) if isinstance(content, str) else len(str(content or ""))
|
||||
return chars >= self._min_chars
|
||||
|
||||
async def _process(self, item: AutoIngestItem) -> None:
|
||||
service = self._provider()
|
||||
if service is None:
|
||||
return
|
||||
if not self._worth_ingesting(item):
|
||||
logger.debug("Auto-ingest skipped (below min chars) thread=%s", item.thread_id)
|
||||
return
|
||||
from deerflow.runtime.user_context import reset_current_user, set_current_user
|
||||
|
||||
token = set_current_user(_QueueUser(id=item.user_id))
|
||||
try:
|
||||
result = await service.capture_thread(
|
||||
thread_id=item.thread_id,
|
||||
messages=item.messages,
|
||||
thread_title=item.thread_title,
|
||||
status="draft", # auto-sedimented notes land as drafts for review
|
||||
include_sources=True,
|
||||
reference_batches=item.reference_batches,
|
||||
created_by=item.user_id,
|
||||
skip_existing=True,
|
||||
)
|
||||
if result.get("created"):
|
||||
logger.info("Auto-ingested thread %s -> note %s", item.thread_id, result["note"]["id"])
|
||||
except Exception:
|
||||
logger.exception("Auto-ingest failed for thread %s", item.thread_id)
|
||||
finally:
|
||||
reset_current_user(token)
|
||||
|
||||
async def _worker(self) -> None:
|
||||
while True:
|
||||
item = await self._queue.get()
|
||||
try:
|
||||
await self._process(item)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception:
|
||||
logger.exception("Auto-ingest worker error")
|
||||
finally:
|
||||
self._queue.task_done()
|
||||
|
||||
|
||||
def set_auto_ingest_queue(queue: AutoIngestQueue | None) -> None:
|
||||
global _QUEUE
|
||||
_QUEUE = queue
|
||||
|
||||
|
||||
def get_auto_ingest_queue() -> AutoIngestQueue | None:
|
||||
return _QUEUE
|
||||
@@ -0,0 +1,60 @@
|
||||
"""Markdown chunking for embedding/index (phase 2).
|
||||
|
||||
Splits a note body into overlapping chunks along Markdown structure (headings →
|
||||
paragraphs) so each embedded chunk is semantically coherent and bounded in
|
||||
size. Frontmatter is assumed already stripped by the caller.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
_HEADING_RE = re.compile(r"^#{1,6}\s+", re.MULTILINE)
|
||||
|
||||
|
||||
def _split_paragraphs(text: str) -> list[str]:
|
||||
return [p.strip() for p in re.split(r"\n\s*\n", text) if p.strip()]
|
||||
|
||||
|
||||
def chunk_markdown(content: str, *, max_chars: int = 800, overlap: int = 120) -> list[str]:
|
||||
"""Return a list of text chunks for ``content``.
|
||||
|
||||
Sections are cut at Markdown headings; long sections are further split into
|
||||
~``max_chars`` windows with ``overlap`` characters of carry-over.
|
||||
"""
|
||||
text = (content or "").strip()
|
||||
if not text:
|
||||
return []
|
||||
|
||||
# Split into heading-delimited sections, keeping the heading with its body.
|
||||
sections: list[str] = []
|
||||
last = 0
|
||||
for m in _HEADING_RE.finditer(text):
|
||||
if m.start() > last:
|
||||
sections.append(text[last : m.start()].strip())
|
||||
last = m.start()
|
||||
sections.append(text[last:].strip())
|
||||
sections = [s for s in sections if s]
|
||||
|
||||
chunks: list[str] = []
|
||||
for section in sections:
|
||||
if len(section) <= max_chars:
|
||||
chunks.append(section)
|
||||
continue
|
||||
# Pack paragraphs into windows, then hard-split anything still too long.
|
||||
buf = ""
|
||||
for para in _split_paragraphs(section):
|
||||
if len(buf) + len(para) + 2 <= max_chars:
|
||||
buf = f"{buf}\n\n{para}".strip()
|
||||
else:
|
||||
if buf:
|
||||
chunks.append(buf)
|
||||
if len(para) <= max_chars:
|
||||
buf = para
|
||||
else:
|
||||
for i in range(0, len(para), max_chars - overlap):
|
||||
chunks.append(para[i : i + max_chars])
|
||||
buf = ""
|
||||
if buf:
|
||||
chunks.append(buf)
|
||||
return [c for c in chunks if c.strip()]
|
||||
@@ -0,0 +1,75 @@
|
||||
"""Duplicate detection for knowledge notes (phase 3).
|
||||
|
||||
First-release heuristic: token Jaccard over ``title + summary + tags`` plus an
|
||||
exact ``content_md`` hash match. Cheap, deterministic and embedding-free so it
|
||||
works on offline deployments. Returns *suggestions*; merging is an explicit,
|
||||
user-driven action.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from deerflow.knowledge.manifest import content_hash
|
||||
|
||||
_TOKEN_RE = re.compile(r"[\w一-鿿]+", re.UNICODE)
|
||||
|
||||
|
||||
def _tokens(*texts: str) -> set[str]:
|
||||
out: set[str] = set()
|
||||
for t in texts:
|
||||
out.update(m.lower() for m in _TOKEN_RE.findall(t or ""))
|
||||
return out
|
||||
|
||||
|
||||
def _jaccard(a: set[str], b: set[str]) -> float:
|
||||
if not a or not b:
|
||||
return 0.0
|
||||
inter = len(a & b)
|
||||
union = len(a | b)
|
||||
return inter / union if union else 0.0
|
||||
|
||||
|
||||
def find_duplicate_groups(notes: list[dict[str, Any]], *, threshold: float = 0.82, max_notes: int = 800) -> list[dict[str, Any]]:
|
||||
"""Return groups ``{primary, duplicates:[{id,title,score,reason}]}``.
|
||||
|
||||
``primary`` is the most recently updated note in each cluster.
|
||||
"""
|
||||
notes = notes[:max_notes]
|
||||
feats: list[tuple[dict[str, Any], set[str], str]] = []
|
||||
for n in notes:
|
||||
toks = _tokens(n.get("title") or "", n.get("summary") or "", " ".join(n.get("tags") or []))
|
||||
feats.append((n, toks, content_hash(n.get("content_md") or "")))
|
||||
|
||||
used: set[str] = set()
|
||||
groups: list[dict[str, Any]] = []
|
||||
for i in range(len(feats)):
|
||||
ni, ti, hi = feats[i]
|
||||
if ni["id"] in used:
|
||||
continue
|
||||
dups: list[dict[str, Any]] = []
|
||||
for j in range(i + 1, len(feats)):
|
||||
nj, tj, hj = feats[j]
|
||||
if nj["id"] in used:
|
||||
continue
|
||||
if hi and hi == hj:
|
||||
score, reason = 1.0, "identical-content"
|
||||
else:
|
||||
score = _jaccard(ti, tj)
|
||||
reason = "similar-title-summary"
|
||||
if score >= threshold:
|
||||
dups.append({"id": nj["id"], "title": nj.get("title"), "score": round(score, 4), "reason": reason})
|
||||
used.add(nj["id"])
|
||||
if dups:
|
||||
used.add(ni["id"])
|
||||
# Primary = newest by updated_at.
|
||||
cluster = [ni, *[next(f[0] for f in feats if f[0]["id"] == d["id"]) for d in dups]]
|
||||
primary = max(cluster, key=lambda x: x.get("updated_at") or "")
|
||||
others = [
|
||||
{"id": c["id"], "title": c.get("title"), "score": next((d["score"] for d in dups if d["id"] == c["id"]), 1.0)}
|
||||
for c in cluster
|
||||
if c["id"] != primary["id"]
|
||||
]
|
||||
groups.append({"primary": {"id": primary["id"], "title": primary.get("title")}, "duplicates": others})
|
||||
return groups
|
||||
@@ -0,0 +1,79 @@
|
||||
"""Embedding generation for vector RAG (phase 2).
|
||||
|
||||
Talks to an OpenAI-compatible ``/embeddings`` endpoint configured under
|
||||
``knowledge.embedding``. Everything here is **best-effort**: if embeddings are
|
||||
disabled, unconfigured, or the endpoint errors, callers get an empty result and
|
||||
the search layer falls back to keyword scoring — so vector RAG never breaks a
|
||||
deployment that has no embedding model.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import math
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from deerflow.config.knowledge_config import KnowledgeEmbeddingConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def cosine_similarity(a: list[float], b: list[float]) -> float:
|
||||
"""Cosine similarity of two equal-length vectors (0 when degenerate)."""
|
||||
if not a or not b or len(a) != len(b):
|
||||
return 0.0
|
||||
dot = sum(x * y for x, y in zip(a, b, strict=False))
|
||||
na = math.sqrt(sum(x * x for x in a))
|
||||
nb = math.sqrt(sum(y * y for y in b))
|
||||
if na == 0 or nb == 0:
|
||||
return 0.0
|
||||
return dot / (na * nb)
|
||||
|
||||
|
||||
class EmbeddingClient:
|
||||
"""Thin async client over an OpenAI-compatible embeddings endpoint."""
|
||||
|
||||
def __init__(self, config: KnowledgeEmbeddingConfig) -> None:
|
||||
self.config = config
|
||||
|
||||
def is_enabled(self) -> bool:
|
||||
return bool(self.config.enabled and self.config.base_url and self.config.model)
|
||||
|
||||
async def embed_texts(self, texts: list[str]) -> list[list[float]]:
|
||||
"""Return one vector per input text; ``[]`` on any failure/disabled."""
|
||||
if not self.is_enabled() or not texts:
|
||||
return []
|
||||
try:
|
||||
import httpx
|
||||
except Exception: # pragma: no cover - httpx is a core dep
|
||||
logger.warning("httpx unavailable; knowledge embeddings disabled")
|
||||
return []
|
||||
|
||||
url = self.config.base_url.rstrip("/") + "/embeddings"
|
||||
headers = {"Content-Type": "application/json"}
|
||||
if self.config.api_key:
|
||||
headers["Authorization"] = f"Bearer {self.config.api_key}"
|
||||
|
||||
vectors: list[list[float]] = []
|
||||
batch = max(1, self.config.batch_size)
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=self.config.timeout_seconds) as client:
|
||||
for start in range(0, len(texts), batch):
|
||||
chunk = texts[start : start + batch]
|
||||
payload: dict = {"model": self.config.model, "input": chunk}
|
||||
if self.config.dimensions:
|
||||
payload["dimensions"] = self.config.dimensions
|
||||
resp = await client.post(url, headers=headers, json=payload)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
items = sorted(data.get("data", []), key=lambda d: d.get("index", 0))
|
||||
vectors.extend([item.get("embedding", []) for item in items])
|
||||
except Exception:
|
||||
logger.exception("Knowledge embedding request failed; falling back to keyword search")
|
||||
return []
|
||||
return vectors
|
||||
|
||||
async def embed_query(self, text: str) -> list[float]:
|
||||
vecs = await self.embed_texts([text])
|
||||
return vecs[0] if vecs else []
|
||||
@@ -0,0 +1,227 @@
|
||||
"""obsidian-wiki ``wiki-export`` compatible graph generation.
|
||||
|
||||
Builds a knowledge graph from the notes (nodes) plus their wikilinks and shared
|
||||
tags (edges) and writes the four native artifacts into ``<vault>/wiki-export/``:
|
||||
``graph.json``, ``graph.graphml``, ``cypher.txt`` and a self-contained
|
||||
``graph.html`` (no external CDN, safe to iframe offline).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import html
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from deerflow.knowledge.markdown_vault import extract_wikilinks
|
||||
|
||||
|
||||
def _vault_key(vault_path: str | None) -> str | None:
|
||||
if not vault_path:
|
||||
return None
|
||||
key = vault_path.replace("\\", "/")
|
||||
if key.endswith(".md"):
|
||||
key = key[:-3]
|
||||
return key
|
||||
|
||||
|
||||
def build_graph(
|
||||
notes: list[dict[str, Any]],
|
||||
entities: list[dict[str, Any]] | None = None,
|
||||
relations: list[dict[str, Any]] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Return ``{nodes, edges}`` from notes (+ optional entities/relations).
|
||||
|
||||
Note nodes come from ``notes`` (id, title, tags, vault_path, content_md);
|
||||
when ``entities``/``relations`` are supplied (phase 4), entity nodes and
|
||||
note→entity / entity→entity edges are added on top of the wikilink and
|
||||
shared-tag edges.
|
||||
"""
|
||||
nodes = []
|
||||
by_key: dict[str, str] = {}
|
||||
note_ids: set[str] = set()
|
||||
for n in notes:
|
||||
nodes.append({"id": n["id"], "type": "note", "label": n.get("title") or n["id"], "category": n.get("source_type")})
|
||||
note_ids.add(n["id"])
|
||||
key = _vault_key(n.get("vault_path"))
|
||||
if key:
|
||||
by_key[key] = n["id"]
|
||||
|
||||
# Only surface entities that are still mentioned by a LIVE note. Entities whose
|
||||
# only referencing notes were deleted/archived would otherwise linger forever
|
||||
# as orphan nodes ("graph doesn't follow my deletions").
|
||||
mentioned: set[str] = set()
|
||||
for rel in relations or []:
|
||||
if rel.get("from_note_id") in note_ids and rel.get("to_entity_id"):
|
||||
mentioned.add(rel["to_entity_id"])
|
||||
|
||||
entity_ids: set[str] = set()
|
||||
for ent in entities or []:
|
||||
if ent["id"] not in mentioned:
|
||||
continue
|
||||
nodes.append({"id": ent["id"], "type": "entity", "label": ent.get("name") or ent["id"], "category": ent.get("entity_type")})
|
||||
entity_ids.add(ent["id"])
|
||||
|
||||
edges: list[dict[str, Any]] = []
|
||||
edge_seen: set[tuple[str, str, str]] = set()
|
||||
|
||||
def add_edge(a: str, b: str, label: str, weight: float) -> None:
|
||||
if a == b:
|
||||
return
|
||||
key = (min(a, b), max(a, b), label)
|
||||
if key in edge_seen:
|
||||
return
|
||||
edge_seen.add(key)
|
||||
edges.append({"id": f"rel_{len(edges) + 1}", "source": a, "target": b, "label": label, "weight": weight})
|
||||
|
||||
# Wikilink edges.
|
||||
for n in notes:
|
||||
for target in extract_wikilinks(n.get("content_md") or ""):
|
||||
tid = by_key.get(target.replace("\\", "/").removesuffix(".md"))
|
||||
if tid:
|
||||
add_edge(n["id"], tid, "links", 2)
|
||||
|
||||
# Shared-tag edges.
|
||||
tag_to_notes: dict[str, list[str]] = {}
|
||||
for n in notes:
|
||||
for tag in n.get("tags") or []:
|
||||
tag_to_notes.setdefault(tag.lower(), []).append(n["id"])
|
||||
for tag, ids in tag_to_notes.items():
|
||||
if len(ids) < 2 or len(ids) > 50: # skip ubiquitous tags (e.g. knowledge-base)
|
||||
continue
|
||||
for i in range(len(ids)):
|
||||
for j in range(i + 1, len(ids)):
|
||||
add_edge(ids[i], ids[j], "related", 1)
|
||||
|
||||
# Entity / relation edges (phase 4) — only when both endpoints exist as nodes.
|
||||
valid = note_ids | entity_ids
|
||||
for rel in relations or []:
|
||||
src = rel.get("from_note_id") or rel.get("from_entity_id")
|
||||
tgt = rel.get("to_note_id") or rel.get("to_entity_id")
|
||||
if src in valid and tgt in valid:
|
||||
add_edge(src, tgt, rel.get("relation_type") or "related", float(rel.get("weight") or 1))
|
||||
|
||||
return {"nodes": nodes, "edges": edges}
|
||||
|
||||
|
||||
def to_graphml(graph: dict[str, Any]) -> str:
|
||||
lines = [
|
||||
'<?xml version="1.0" encoding="UTF-8"?>',
|
||||
'<graphml xmlns="http://graphml.graphdrawing.org/xmlns">',
|
||||
' <key id="label" for="node" attr.name="label" attr.type="string"/>',
|
||||
' <key id="elabel" for="edge" attr.name="label" attr.type="string"/>',
|
||||
' <graph edgedefault="undirected">',
|
||||
]
|
||||
for n in graph["nodes"]:
|
||||
lines.append(f' <node id="{html.escape(str(n["id"]))}"><data key="label">{html.escape(str(n["label"]))}</data></node>')
|
||||
for e in graph["edges"]:
|
||||
lines.append(
|
||||
f' <edge id="{html.escape(str(e["id"]))}" source="{html.escape(str(e["source"]))}" '
|
||||
f'target="{html.escape(str(e["target"]))}"><data key="elabel">{html.escape(str(e["label"]))}</data></edge>'
|
||||
)
|
||||
lines += [" </graph>", "</graphml>"]
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def to_cypher(graph: dict[str, Any]) -> str:
|
||||
lines = []
|
||||
for n in graph["nodes"]:
|
||||
label = str(n["label"]).replace("'", "\\'")
|
||||
lines.append(f"CREATE (:Note {{id:'{n['id']}', title:'{label}'}});")
|
||||
for e in graph["edges"]:
|
||||
rel = str(e["label"]).upper()
|
||||
lines.append(f"MATCH (a:Note {{id:'{e['source']}'}}),(b:Note {{id:'{e['target']}'}}) CREATE (a)-[:{rel} {{weight:{e['weight']}}}]->(b);")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def to_html(graph: dict[str, Any]) -> str:
|
||||
"""Self-contained force-directed graph (canvas + vanilla JS, no CDN)."""
|
||||
data_json = json.dumps(graph, ensure_ascii=False)
|
||||
return _HTML_TEMPLATE.replace("__GRAPH_DATA__", data_json)
|
||||
|
||||
|
||||
def export_all(
|
||||
vault_root: Path | str,
|
||||
notes: list[dict[str, Any]],
|
||||
*,
|
||||
entities: list[dict[str, Any]] | None = None,
|
||||
relations: list[dict[str, Any]] | None = None,
|
||||
write_html: bool = True,
|
||||
write_json: bool = True,
|
||||
) -> dict[str, Any]:
|
||||
"""Generate graph artifacts under ``<vault>/wiki-export/`` and return stats."""
|
||||
out_dir = Path(vault_root).resolve() / "wiki-export"
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
graph = build_graph(notes, entities, relations)
|
||||
written: list[str] = []
|
||||
if write_json:
|
||||
(out_dir / "graph.json").write_text(json.dumps(graph, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
written.append("graph.json")
|
||||
(out_dir / "graph.graphml").write_text(to_graphml(graph), encoding="utf-8")
|
||||
written.append("graph.graphml")
|
||||
(out_dir / "cypher.txt").write_text(to_cypher(graph), encoding="utf-8")
|
||||
written.append("cypher.txt")
|
||||
if write_html:
|
||||
(out_dir / "graph.html").write_text(to_html(graph), encoding="utf-8")
|
||||
written.append("graph.html")
|
||||
return {
|
||||
"files": written,
|
||||
"stats": {"nodes": len(graph["nodes"]), "edges": len(graph["edges"])},
|
||||
"graph": graph,
|
||||
}
|
||||
|
||||
|
||||
_HTML_TEMPLATE = """<!DOCTYPE html>
|
||||
<html lang="zh">
|
||||
<head>
|
||||
<meta charset="utf-8"/>
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1"/>
|
||||
<title>Knowledge Graph</title>
|
||||
<style>
|
||||
html,body{margin:0;height:100%;background:#0b1020;color:#e7ecff;font-family:system-ui,Segoe UI,Arial,sans-serif;overflow:hidden}
|
||||
#hud{position:fixed;top:10px;left:12px;font-size:12px;opacity:.8;z-index:2}
|
||||
#tip{position:fixed;padding:4px 8px;background:#111a36;border:1px solid #2a3a66;border-radius:4px;font-size:12px;pointer-events:none;display:none;z-index:3}
|
||||
canvas{display:block}
|
||||
#empty{position:fixed;inset:0;display:flex;align-items:center;justify-content:center;opacity:.6}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div id="hud"></div>
|
||||
<div id="tip"></div>
|
||||
<canvas id="c"></canvas>
|
||||
<script>
|
||||
const GRAPH = __GRAPH_DATA__;
|
||||
const canvas = document.getElementById('c'), ctx = canvas.getContext('2d');
|
||||
const hud = document.getElementById('hud'), tip = document.getElementById('tip');
|
||||
let W=0,H=0;
|
||||
function resize(){W=canvas.width=innerWidth;H=canvas.height=innerHeight;}
|
||||
addEventListener('resize',resize);resize();
|
||||
hud.textContent = GRAPH.nodes.length + ' 个知识节点 · ' + GRAPH.edges.length + ' 条关系';
|
||||
if(!GRAPH.nodes.length){document.body.insertAdjacentHTML('beforeend','<div id=\"empty\">暂无知识图谱数据</div>');}
|
||||
const idx={};GRAPH.nodes.forEach((n,i)=>{idx[n.id]=n;n.x=W/2+Math.cos(i)*200*Math.random()+(Math.random()-.5)*200;n.y=H/2+Math.sin(i)*200*Math.random()+(Math.random()-.5)*200;n.vx=0;n.vy=0;});
|
||||
const edges=GRAPH.edges.map(e=>({s:idx[e.source],t:idx[e.target],w:e.w||e.weight||1})).filter(e=>e.s&&e.t);
|
||||
let cam={x:0,y:0,k:1},drag=null,pan=null;
|
||||
function step(){
|
||||
for(const n of GRAPH.nodes){for(const m of GRAPH.nodes){if(n===m)continue;let dx=n.x-m.x,dy=n.y-m.y,d2=dx*dx+dy*dy+0.01,f=1200/d2;n.vx+=dx*f*0.0006;n.vy+=dy*f*0.0006;}}
|
||||
for(const e of edges){let dx=e.t.x-e.s.x,dy=e.t.y-e.s.y,d=Math.sqrt(dx*dx+dy*dy)||1,f=(d-110)*0.01;e.s.vx+=dx/d*f;e.s.vy+=dy/d*f;e.t.vx-=dx/d*f;e.t.vy-=dy/d*f;}
|
||||
for(const n of GRAPH.nodes){n.vx+=(W/2-n.x)*0.0008;n.vy+=(H/2-n.y)*0.0008;n.x+=n.vx*=0.85;n.y+=n.vy*=0.85;}
|
||||
}
|
||||
function draw(){
|
||||
ctx.setTransform(cam.k,0,0,cam.k,cam.x,cam.y);ctx.clearRect(-cam.x/cam.k,-cam.y/cam.k,W/cam.k,H/cam.k);
|
||||
ctx.strokeStyle='rgba(120,150,255,.25)';
|
||||
for(const e of edges){ctx.lineWidth=e.w*0.6;ctx.beginPath();ctx.moveTo(e.s.x,e.s.y);ctx.lineTo(e.t.x,e.t.y);ctx.stroke();}
|
||||
for(const n of GRAPH.nodes){const r=6;ctx.fillStyle='#5b8cff';ctx.beginPath();ctx.arc(n.x,n.y,r,0,7);ctx.fill();
|
||||
ctx.fillStyle='rgba(231,236,255,.85)';ctx.font='11px system-ui';ctx.fillText((n.label||'').slice(0,18),n.x+9,n.y+3);}
|
||||
}
|
||||
function loop(){step();draw();requestAnimationFrame(loop);}loop();
|
||||
function toWorld(x,y){return {x:(x-cam.x)/cam.k,y:(y-cam.y)/cam.k};}
|
||||
canvas.addEventListener('mousedown',ev=>{const p=toWorld(ev.clientX,ev.clientY);let hit=null;
|
||||
for(const n of GRAPH.nodes){if((n.x-p.x)**2+(n.y-p.y)**2<100){hit=n;break;}}if(hit)drag=hit;else pan={x:ev.clientX,y:ev.clientY,cx:cam.x,cy:cam.y};});
|
||||
addEventListener('mousemove',ev=>{const p=toWorld(ev.clientX,ev.clientY);if(drag){drag.x=p.x;drag.y=p.y;drag.vx=drag.vy=0;}else if(pan){cam.x=pan.cx+ev.clientX-pan.x;cam.y=pan.cy+ev.clientY-pan.y;}
|
||||
let hov=null;for(const n of GRAPH.nodes){if((n.x-p.x)**2+(n.y-p.y)**2<100){hov=n;break;}}
|
||||
if(hov){tip.style.display='block';tip.style.left=(ev.clientX+12)+'px';tip.style.top=(ev.clientY+12)+'px';tip.textContent=hov.label;}else{tip.style.display='none';}});
|
||||
addEventListener('mouseup',()=>{drag=null;pan=null;});
|
||||
canvas.addEventListener('wheel',ev=>{ev.preventDefault();const s=ev.deltaY<0?1.1:0.9;cam.k*=s;cam.x=ev.clientX-(ev.clientX-cam.x)*s;cam.y=ev.clientY-(ev.clientY-cam.y)*s;},{passive:false});
|
||||
</script>
|
||||
</body>
|
||||
</html>"""
|
||||
@@ -0,0 +1,262 @@
|
||||
"""Extract knowledge notes from threads / searches / tool results.
|
||||
|
||||
First-release strategy is rule-based (wiki-capture aligned, §11 of the dev doc):
|
||||
keep high-value content (user questions, final assistant answers, cited
|
||||
sources), drop noise (empty/greeting/intermediate reasoning), and write a
|
||||
*declarative* note rather than a chat transcript. An LLM extractor can later
|
||||
replace :func:`extract_thread` while keeping the same :class:`NoteDraft` output.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from deerflow.knowledge.schemas import NoteDraft, SourceDraft
|
||||
|
||||
# Greeting / low-value lines filtered from extracted turns.
|
||||
_GREETING_RE = re.compile(r"^(你好|您好|hi|hello|hey|thanks?|谢谢|好的|ok|okay)[\s!!。.,,~]*$", re.IGNORECASE)
|
||||
_WS_RE = re.compile(r"\s+")
|
||||
|
||||
|
||||
def _one_line(text: str, limit: int = 300) -> str:
|
||||
"""Collapse whitespace and clip ``text`` to a single line."""
|
||||
collapsed = _WS_RE.sub(" ", text or "").strip()
|
||||
return collapsed[:limit]
|
||||
|
||||
|
||||
def message_text(msg: dict[str, Any]) -> str:
|
||||
"""Return the plain-text content of a serialized LangChain message."""
|
||||
content = msg.get("content")
|
||||
if isinstance(content, str):
|
||||
return content.strip()
|
||||
if isinstance(content, list):
|
||||
parts: list[str] = []
|
||||
for block in content:
|
||||
if isinstance(block, str):
|
||||
parts.append(block)
|
||||
elif isinstance(block, dict):
|
||||
if block.get("type") in (None, "text") and isinstance(block.get("text"), str):
|
||||
parts.append(block["text"])
|
||||
return "\n".join(p for p in parts if p).strip()
|
||||
return ""
|
||||
|
||||
|
||||
def _msg_type(msg: dict[str, Any]) -> str:
|
||||
return str(msg.get("type") or msg.get("role") or "").lower()
|
||||
|
||||
|
||||
def split_turns(messages: list[dict[str, Any]], *, max_messages: int = 80) -> list[dict[str, str]]:
|
||||
"""Split messages into ``{user, ai}`` turns keyed by human messages.
|
||||
|
||||
For each turn we keep the last substantial AI message (with text body).
|
||||
"""
|
||||
msgs = messages[-max_messages:] if max_messages and len(messages) > max_messages else list(messages)
|
||||
turns: list[dict[str, str]] = []
|
||||
current: dict[str, str] | None = None
|
||||
for msg in msgs:
|
||||
mtype = _msg_type(msg)
|
||||
text = message_text(msg)
|
||||
if mtype == "human":
|
||||
if current and (current.get("user") or current.get("ai")):
|
||||
turns.append(current)
|
||||
current = {"user": text, "ai": ""}
|
||||
elif mtype == "ai" and text:
|
||||
if current is None:
|
||||
current = {"user": "", "ai": text}
|
||||
else:
|
||||
current["ai"] = text # last substantial AI message wins
|
||||
if current and (current.get("user") or current.get("ai")):
|
||||
turns.append(current)
|
||||
# Drop turns that are pure greetings with no answer.
|
||||
cleaned = []
|
||||
for t in turns:
|
||||
user = (t.get("user") or "").strip()
|
||||
ai = (t.get("ai") or "").strip()
|
||||
if not ai and _GREETING_RE.match(user):
|
||||
continue
|
||||
if not user and not ai:
|
||||
continue
|
||||
cleaned.append({"user": user, "ai": ai})
|
||||
return cleaned
|
||||
|
||||
|
||||
def _auto_title(thread_title: str | None, turns: list[dict[str, str]]) -> str:
|
||||
if thread_title and thread_title.strip() and thread_title.strip().lower() != "untitled":
|
||||
return thread_title.strip()[:200]
|
||||
for t in turns:
|
||||
if t.get("user"):
|
||||
first = t["user"].splitlines()[0].strip()
|
||||
return (first[:40] + ("…" if len(first) > 40 else "")) or "知识笔记"
|
||||
return "知识笔记"
|
||||
|
||||
|
||||
def _summary(turns: list[dict[str, str]], *, limit: int = 280) -> str:
|
||||
"""Simple rule-based summary: the first answer, clipped."""
|
||||
for t in turns:
|
||||
if t.get("ai"):
|
||||
text = re.sub(r"\s+", " ", t["ai"]).strip()
|
||||
return text[:limit] + ("…" if len(text) > limit else "")
|
||||
for t in turns:
|
||||
if t.get("user"):
|
||||
text = re.sub(r"\s+", " ", t["user"]).strip()
|
||||
return text[:limit] + ("…" if len(text) > limit else "")
|
||||
return ""
|
||||
|
||||
|
||||
def related_links(project_name: str | None) -> list[str]: # noqa: ARG001
|
||||
"""Structural wikilinks every note used to carry.
|
||||
|
||||
Product decision: notes no longer auto-link the vault meta/hub pages
|
||||
(``index`` / ``_meta/taxonomy`` / the project hub) — they were noise in every
|
||||
note's ``## Related``. Only real, content-specific related notes (injected by
|
||||
the service layer, which has DB access) are linked now. Kept as a function
|
||||
(returns empty) so existing callers and the wiki-capture contract stay intact.
|
||||
"""
|
||||
return []
|
||||
|
||||
|
||||
# Backwards-compatible internal alias.
|
||||
_related_links = related_links
|
||||
|
||||
|
||||
def _related_section_md(related_lines: list[str], source_lines: list[str]) -> str:
|
||||
"""Render the ``## Related`` section, or ``""`` when there is nothing to show."""
|
||||
if not related_lines and not source_lines:
|
||||
return ""
|
||||
inner = "\n".join(related_lines)
|
||||
if source_lines:
|
||||
sources_block = "### Sources\n\n" + "\n".join(source_lines)
|
||||
inner = (inner + "\n\n" + sources_block) if inner else sources_block
|
||||
return f"## Related\n\n{inner}\n"
|
||||
|
||||
|
||||
def _build_body(
|
||||
*,
|
||||
title: str,
|
||||
turns: list[dict[str, str]],
|
||||
mode: str,
|
||||
project_name: str | None,
|
||||
sources: list[SourceDraft],
|
||||
) -> str:
|
||||
"""Render the wiki-capture 5-section declarative body."""
|
||||
full = mode == "full"
|
||||
answer_clip = 4000 if full else 800
|
||||
|
||||
context_lines = []
|
||||
for t in turns:
|
||||
if t.get("user"):
|
||||
q = re.sub(r"\s+", " ", t["user"]).strip()
|
||||
context_lines.append(f"- {q[:300]}")
|
||||
context = "\n".join(context_lines[:8]) or "- 来自一次对话沉淀。^[inferred]"
|
||||
|
||||
finding_lines = []
|
||||
for i, t in enumerate(turns, 1):
|
||||
if t.get("ai"):
|
||||
ans = t["ai"].strip()
|
||||
ans = ans if full else (ans[:answer_clip] + ("…" if len(ans) > answer_clip else ""))
|
||||
finding_lines.append(f"### 结论 {i}\n\n{ans}")
|
||||
finding = "\n\n".join(finding_lines) or "_暂无可提炼的结论。_"
|
||||
|
||||
reasoning = "本页由对话自动沉淀而成,结论来自助手最终回答;过程性推理已省略。判断与归纳部分标记为 ^[inferred]。"
|
||||
|
||||
implications = "- 后续可在相关问答中参考本页知识。\n- 如内容过期或冲突,请编辑或归档。"
|
||||
|
||||
related_lines = [f"- {link}" for link in _related_links(project_name)]
|
||||
source_lines = [f"- {s.title or s.url or '来源'}" + (f" — {s.url}" if s.url else "") for s in sources[:20]]
|
||||
related_section = _related_section_md(related_lines, source_lines)
|
||||
|
||||
return (
|
||||
f"# {title}\n\n"
|
||||
f"## Context\n\n{context}\n\n"
|
||||
f"## Finding / Decision\n\n{finding}\n\n"
|
||||
f"## Reasoning\n\n{reasoning}\n\n"
|
||||
f"## Implications\n\n{implications}\n\n"
|
||||
f"{related_section}"
|
||||
)
|
||||
|
||||
|
||||
def extract_thread(
|
||||
messages: list[dict[str, Any]],
|
||||
*,
|
||||
thread_id: str,
|
||||
thread_title: str | None = None,
|
||||
mode: str = "summary",
|
||||
title: str | None = None,
|
||||
tags: list[str] | None = None,
|
||||
project_name: str | None = "zncm",
|
||||
sources: list[SourceDraft] | None = None,
|
||||
max_messages: int = 80,
|
||||
) -> NoteDraft | None:
|
||||
"""Build a :class:`NoteDraft` from thread messages, or ``None`` if empty."""
|
||||
turns = split_turns(messages, max_messages=max_messages)
|
||||
if not turns:
|
||||
return None
|
||||
final_title = (title or "").strip() or _auto_title(thread_title, turns)
|
||||
summary = _summary(turns)
|
||||
sources = sources or []
|
||||
body = _build_body(title=final_title, turns=turns, mode=mode, project_name=project_name, sources=sources)
|
||||
base_tags = list(tags or [])
|
||||
for default in (project_name or "zncm", "knowledge-base"):
|
||||
if default and default not in base_tags:
|
||||
base_tags.append(default)
|
||||
return NoteDraft(
|
||||
title=final_title,
|
||||
summary=summary,
|
||||
content_md=body,
|
||||
category="synthesis",
|
||||
source_type="thread",
|
||||
source_id=thread_id,
|
||||
tags=base_tags,
|
||||
confidence=0.6,
|
||||
related=_related_links(project_name),
|
||||
sources=sources,
|
||||
source_key=f"thread:{thread_id}",
|
||||
)
|
||||
|
||||
|
||||
def extract_search(
|
||||
query: str,
|
||||
sources: list[SourceDraft],
|
||||
*,
|
||||
thread_id: str | None = None,
|
||||
project_name: str | None = "zncm",
|
||||
title: str | None = None,
|
||||
tags: list[str] | None = None,
|
||||
) -> NoteDraft:
|
||||
"""Build a reference-category :class:`NoteDraft` from search/tool results."""
|
||||
final_title = (title or "").strip() or f"检索:{query[:40]}"
|
||||
body_lines = [f"# {final_title}\n", "## Context\n", f"- 检索关键词:{query}\n", "## Finding / Decision\n"]
|
||||
for s in sources:
|
||||
label = s.title or s.url or "来源"
|
||||
line = f"- **{label}**"
|
||||
if s.url:
|
||||
line += f" — {s.url}"
|
||||
if s.snippet:
|
||||
line += "\n - " + _one_line(s.snippet)
|
||||
body_lines.append(line)
|
||||
body_lines += [
|
||||
"\n## Reasoning\n",
|
||||
"本页汇总外部检索来源,供后续问答引用。来源可信度需结合原文判断。^[ambiguous]\n",
|
||||
"## Implications\n",
|
||||
"- 引用前请核对原始来源。\n",
|
||||
"## Related\n",
|
||||
*[f"- {link}" for link in _related_links(project_name)],
|
||||
]
|
||||
base_tags = list(tags or [])
|
||||
for default in (project_name or "zncm", "knowledge-base", "reference"):
|
||||
if default and default not in base_tags:
|
||||
base_tags.append(default)
|
||||
return NoteDraft(
|
||||
title=final_title,
|
||||
summary=f"检索“{query}”的来源汇总,共 {len(sources)} 条。",
|
||||
content_md="\n".join(body_lines),
|
||||
category="references",
|
||||
source_type="search",
|
||||
source_id=f"search:{query[:120]}",
|
||||
tags=base_tags,
|
||||
confidence=0.4,
|
||||
related=_related_links(project_name),
|
||||
sources=sources,
|
||||
source_key=None,
|
||||
)
|
||||
+354
@@ -0,0 +1,354 @@
|
||||
"""LLM-based knowledge extraction (wiki-capture style).
|
||||
|
||||
Turns a conversation into a *declarative* knowledge note instead of dumping the
|
||||
raw chat. Follows the obsidian-wiki ``wiki-capture`` skill: classify the note,
|
||||
keep only high-value findings/decisions, drop greetings / intermediate
|
||||
reasoning / verbatim transcripts, mark inferred content, and emit structured
|
||||
JSON that we render into the Context / Finding / Reasoning / Implications /
|
||||
Related body.
|
||||
|
||||
Falls back to the rule-based :func:`deerflow.knowledge.extractor.extract_thread`
|
||||
when the LLM is unavailable or returns something unusable, so capture never
|
||||
hard-fails.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from deerflow.knowledge import extractor
|
||||
from deerflow.knowledge.schemas import NoteDraft, SourceDraft
|
||||
from deerflow.knowledge.skill_templates import WIKI_CAPTURE_GUIDANCE
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_VALID_CATEGORIES = {"synthesis", "concepts", "references", "decision", "journal"}
|
||||
|
||||
|
||||
class LLMRejectedIngest(Exception):
|
||||
"""The LLM judged the conversation as having no knowledge value.
|
||||
|
||||
Raised (instead of returning ``[]``) when the model explicitly sets
|
||||
``should_ingest: false`` with no notes — so callers can *skip* ingestion
|
||||
entirely rather than falling back to the rule-based verbatim dump.
|
||||
"""
|
||||
|
||||
|
||||
def _is_falsey(value: Any) -> bool:
|
||||
if value is False:
|
||||
return True
|
||||
if isinstance(value, str):
|
||||
return value.strip().lower() in ("false", "0", "no")
|
||||
return False
|
||||
|
||||
_SYSTEM_PROMPT = """\
|
||||
你是知识库整理专家,遵循 obsidian-wiki 的 wiki-capture 规范,把一次对话提炼成\
|
||||
**陈述式知识(declarative knowledge)**,而不是聊天流水账。
|
||||
|
||||
要求:
|
||||
- 只保留高价值内容:明确结论、方案、决策、关键事实、被引用的来源。
|
||||
- 丢弃:寒暄、客套、中间推理过程、重复内容、无意义的原文堆砌。
|
||||
- 不要写“用户问了什么、AI 回答了什么”,要写“结论是什么、为什么成立”。
|
||||
- 推断内容用 ^[inferred] 标记,不确定/冲突用 ^[ambiguous] 标记。
|
||||
- 分类 category 取值之一:synthesis(分析/方案/结论)、concepts(概念/框架)、\
|
||||
references(外部来源汇总)、decision(架构/设计决策)、journal(会话纪要)。
|
||||
- 用中文输出。
|
||||
|
||||
一次对话可能只讲一个主题,也可能覆盖多个相互独立的主题。请把它拆成 1~4 页知识:
|
||||
- 默认只产出 **1 页**;只有当对话明确包含**多个彼此独立、可各自成文**的主题时才拆成多页。
|
||||
- 每一页聚焦单一主题,标题与 summary 必须彼此区分,不要把同一结论拆到多页。
|
||||
- 不值得沉淀时返回空的 notes 数组。
|
||||
|
||||
只输出一个 JSON 对象,不要任何额外文字或 Markdown 代码块,结构如下:
|
||||
{
|
||||
"should_ingest": true,
|
||||
"reason": "为什么值得/不值得沉淀",
|
||||
"notes": [
|
||||
{
|
||||
"title": "简洁的知识标题(不超过 40 字)",
|
||||
"summary": "1-2 句话概括这页知识",
|
||||
"category": "synthesis",
|
||||
"context": "问题背景或触发场景,1-3 句",
|
||||
"findings": ["结论/要点(陈述式,可含 ^[inferred])", "..."],
|
||||
"reasoning": "为什么成立、取舍、不确定性",
|
||||
"implications": ["后续建议/风险/下一步", "..."],
|
||||
"tags": ["标签1", "标签2"],
|
||||
"entities": [{"name": "实体名", "type": "technology|concept|person|org|product|other"}],
|
||||
"relations": [{"from": "实体A", "to": "实体B", "type": "uses|relates_to|part_of|depends_on"}],
|
||||
"confidence": 0.0
|
||||
}
|
||||
]
|
||||
}"""
|
||||
|
||||
|
||||
def _build_transcript(turns: list[dict[str, str]], sources: list[SourceDraft], *, max_chars: int) -> str:
|
||||
parts: list[str] = []
|
||||
for i, t in enumerate(turns, 1):
|
||||
if t.get("user"):
|
||||
parts.append(f"[用户 {i}] {t['user']}")
|
||||
if t.get("ai"):
|
||||
parts.append(f"[助手 {i}] {t['ai']}")
|
||||
if sources:
|
||||
parts.append("\n[来源]")
|
||||
for s in sources[:20]:
|
||||
label = s.title or s.url or s.tool_name or "来源"
|
||||
parts.append(f"- {label}" + (f" ({s.url})" if s.url else ""))
|
||||
text = "\n\n".join(parts)
|
||||
if len(text) > max_chars:
|
||||
text = text[:max_chars] + "\n…(已截断)"
|
||||
return text
|
||||
|
||||
|
||||
def _parse_json(raw: str) -> dict[str, Any] | None:
|
||||
if not raw:
|
||||
return None
|
||||
text = raw.strip()
|
||||
# Strip ```json fences if present.
|
||||
fence = re.search(r"```(?:json)?\s*(.*?)```", text, re.DOTALL)
|
||||
if fence:
|
||||
text = fence.group(1).strip()
|
||||
# Otherwise grab the outermost { ... }.
|
||||
if not text.startswith("{"):
|
||||
start = text.find("{")
|
||||
end = text.rfind("}")
|
||||
if start >= 0 and end > start:
|
||||
text = text[start : end + 1]
|
||||
try:
|
||||
data = json.loads(text)
|
||||
return data if isinstance(data, dict) else None
|
||||
except json.JSONDecodeError:
|
||||
logger.warning("Knowledge LLM returned non-JSON output; falling back to rule-based")
|
||||
return None
|
||||
|
||||
|
||||
def _render_body(data: dict[str, Any], title: str, related: list[str], sources: list[SourceDraft]) -> str:
|
||||
def _bullets(items: Any) -> str:
|
||||
if isinstance(items, str):
|
||||
items = [items]
|
||||
if not isinstance(items, list):
|
||||
return ""
|
||||
return "\n".join(f"- {str(x).strip()}" for x in items if str(x).strip())
|
||||
|
||||
context = str(data.get("context") or "").strip() or "- 来自一次对话沉淀。^[inferred]"
|
||||
findings = _bullets(data.get("findings")) or "_暂无可提炼的结论。_"
|
||||
reasoning = str(data.get("reasoning") or "").strip() or "(未提供推理说明)"
|
||||
implications = _bullets(data.get("implications")) or "- 如内容过期或冲突,请编辑或归档。"
|
||||
|
||||
related_lines = [f"- {link}" for link in related]
|
||||
source_lines = [f"- {s.title or s.url or '来源'}" + (f" — {s.url}" if s.url else "") for s in sources[:20]]
|
||||
related_section = extractor._related_section_md(related_lines, source_lines)
|
||||
|
||||
return (
|
||||
f"# {title}\n\n"
|
||||
f"## Context\n\n{context}\n\n"
|
||||
f"## Finding / Decision\n\n{findings}\n\n"
|
||||
f"## Reasoning\n\n{reasoning}\n\n"
|
||||
f"## Implications\n\n{implications}\n\n"
|
||||
f"{related_section}"
|
||||
)
|
||||
|
||||
|
||||
def _notes_from_data(data: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
"""Normalize the LLM JSON into a list of per-note dicts.
|
||||
|
||||
Accepts the multi-note shape ``{"notes": [...]}`` as well as the legacy
|
||||
single-note shape (``{"title": ..., "findings": ...}``) for backward
|
||||
compatibility. Notes without any title/findings are dropped.
|
||||
"""
|
||||
raw_notes = data.get("notes")
|
||||
if isinstance(raw_notes, list):
|
||||
candidates = [n for n in raw_notes if isinstance(n, dict)]
|
||||
elif data.get("title") or data.get("findings") or data.get("context"):
|
||||
candidates = [data] # legacy single-object output
|
||||
else:
|
||||
candidates = []
|
||||
return [n for n in candidates if str(n.get("title") or "").strip() or n.get("findings")]
|
||||
|
||||
|
||||
def _build_note_draft(
|
||||
note_data: dict[str, Any],
|
||||
*,
|
||||
thread_id: str,
|
||||
thread_title: str | None,
|
||||
turns: list[dict[str, str]],
|
||||
title_override: str | None,
|
||||
tags: list[str] | None,
|
||||
project_name: str | None,
|
||||
sources: list[SourceDraft],
|
||||
source_key_suffix: str = "",
|
||||
) -> NoteDraft:
|
||||
"""Render one note dict into a :class:`NoteDraft`."""
|
||||
category = str(note_data.get("category") or "synthesis").strip().lower()
|
||||
if category not in _VALID_CATEGORIES:
|
||||
category = "synthesis"
|
||||
|
||||
final_title = (title_override or "").strip() or str(note_data.get("title") or "").strip() or extractor._auto_title(thread_title, turns)
|
||||
summary = str(note_data.get("summary") or "").strip()
|
||||
related = extractor.related_links(project_name)
|
||||
body = _render_body(note_data, final_title, related, sources)
|
||||
|
||||
base_tags = list(tags or [])
|
||||
llm_tags = note_data.get("tags") if isinstance(note_data.get("tags"), list) else []
|
||||
for tag in [*llm_tags, project_name or "zncm", "knowledge-base"]:
|
||||
tag = str(tag).strip()
|
||||
if tag and tag not in base_tags:
|
||||
base_tags.append(tag)
|
||||
|
||||
try:
|
||||
confidence = float(note_data.get("confidence") or 0.7)
|
||||
except (TypeError, ValueError):
|
||||
confidence = 0.7
|
||||
|
||||
entities = note_data.get("entities") if isinstance(note_data.get("entities"), list) else []
|
||||
relations = note_data.get("relations") if isinstance(note_data.get("relations"), list) else []
|
||||
|
||||
return NoteDraft(
|
||||
title=final_title,
|
||||
summary=summary or final_title,
|
||||
content_md=body,
|
||||
category=category,
|
||||
source_type="thread",
|
||||
source_id=thread_id,
|
||||
tags=base_tags,
|
||||
confidence=max(0.0, min(confidence, 1.0)),
|
||||
related=related,
|
||||
sources=sources,
|
||||
entities=[e for e in entities if isinstance(e, dict) and e.get("name")],
|
||||
relations=[r for r in relations if isinstance(r, dict) and r.get("from") and r.get("to")],
|
||||
source_key=f"thread:{thread_id}{source_key_suffix}",
|
||||
)
|
||||
|
||||
|
||||
async def extract_thread_llm_multi(
|
||||
messages: list[dict[str, Any]],
|
||||
*,
|
||||
thread_id: str,
|
||||
thread_title: str | None = None,
|
||||
mode: str = "summary",
|
||||
title: str | None = None,
|
||||
tags: list[str] | None = None,
|
||||
project_name: str | None = "zncm",
|
||||
sources: list[SourceDraft] | None = None,
|
||||
max_messages: int = 80,
|
||||
model_name: str | None = None,
|
||||
max_input_chars: int = 12000,
|
||||
allow_multi: bool = True,
|
||||
max_notes: int = 4,
|
||||
) -> list[NoteDraft]:
|
||||
"""Distill a thread into one *or several* cross-linkable knowledge notes.
|
||||
|
||||
The LLM may split a conversation that covers multiple independent topics into
|
||||
several declarative notes. Returns ``[]`` when there is nothing to sediment
|
||||
or the LLM path fails — callers should fall back to the rule-based extractor.
|
||||
When ``allow_multi`` is false only the first note is kept.
|
||||
"""
|
||||
turns = extractor.split_turns(messages, max_messages=max_messages)
|
||||
if not turns:
|
||||
return []
|
||||
sources = sources or []
|
||||
|
||||
try:
|
||||
from langchain_core.messages import HumanMessage, SystemMessage
|
||||
|
||||
from deerflow.models import create_chat_model
|
||||
|
||||
model = create_chat_model(name=model_name or None, thinking_enabled=False)
|
||||
transcript = _build_transcript(turns, sources, max_chars=max_input_chars)
|
||||
prompt = [
|
||||
SystemMessage(content=_SYSTEM_PROMPT + "\n\n" + WIKI_CAPTURE_GUIDANCE),
|
||||
HumanMessage(content=f"以下是一次对话,请按规范提炼为知识 JSON:\n\n{transcript}"),
|
||||
]
|
||||
response = await model.ainvoke(prompt)
|
||||
raw = response.content if isinstance(response.content, str) else str(response.content)
|
||||
data = _parse_json(raw)
|
||||
except Exception:
|
||||
logger.exception("[knowledge] thread %s — LLM extraction call failed; using rule-based fallback", thread_id)
|
||||
return []
|
||||
|
||||
if not data:
|
||||
logger.warning("[knowledge] thread %s — LLM returned no parsable JSON; using rule-based fallback", thread_id)
|
||||
return []
|
||||
|
||||
note_dicts = _notes_from_data(data)
|
||||
if not note_dicts:
|
||||
reason = str(data.get("reason") or "")[:200]
|
||||
if _is_falsey(data.get("should_ingest")):
|
||||
# The model deliberately judged this conversation worthless (greeting,
|
||||
# meta chatter, no findings). Skip ingestion entirely — do NOT fall
|
||||
# back to the rule-based verbatim dump.
|
||||
logger.warning("[knowledge] thread %s SKIPPED — model judged not worth ingesting: %s", thread_id, reason)
|
||||
raise LLMRejectedIngest(reason or "no knowledge value")
|
||||
logger.warning(
|
||||
"[knowledge] thread %s — LLM produced no notes (should_ingest=%s, reason=%r); using rule-based fallback",
|
||||
thread_id,
|
||||
data.get("should_ingest"),
|
||||
reason,
|
||||
)
|
||||
return []
|
||||
if not allow_multi:
|
||||
note_dicts = note_dicts[:1]
|
||||
else:
|
||||
note_dicts = note_dicts[: max(1, max_notes)]
|
||||
|
||||
drafts: list[NoteDraft] = []
|
||||
for i, nd in enumerate(note_dicts):
|
||||
# The user-supplied title override only applies to a single-note result;
|
||||
# multi-note splits keep each note's own LLM title.
|
||||
override = title if (len(note_dicts) == 1) else None
|
||||
suffix = "" if len(note_dicts) == 1 else f"#{i}"
|
||||
drafts.append(
|
||||
_build_note_draft(
|
||||
nd,
|
||||
thread_id=thread_id,
|
||||
thread_title=thread_title,
|
||||
turns=turns,
|
||||
title_override=override,
|
||||
tags=tags,
|
||||
project_name=project_name,
|
||||
sources=sources,
|
||||
source_key_suffix=suffix,
|
||||
)
|
||||
)
|
||||
return drafts
|
||||
|
||||
|
||||
async def extract_thread_llm(
|
||||
messages: list[dict[str, Any]],
|
||||
*,
|
||||
thread_id: str,
|
||||
thread_title: str | None = None,
|
||||
mode: str = "summary",
|
||||
title: str | None = None,
|
||||
tags: list[str] | None = None,
|
||||
project_name: str | None = "zncm",
|
||||
sources: list[SourceDraft] | None = None,
|
||||
max_messages: int = 80,
|
||||
model_name: str | None = None,
|
||||
max_input_chars: int = 12000,
|
||||
) -> NoteDraft | None:
|
||||
"""Distill a thread into a single :class:`NoteDraft` (backward-compatible).
|
||||
|
||||
Thin wrapper over :func:`extract_thread_llm_multi` that returns the first
|
||||
note, or ``None`` when nothing was produced.
|
||||
"""
|
||||
try:
|
||||
drafts = await extract_thread_llm_multi(
|
||||
messages,
|
||||
thread_id=thread_id,
|
||||
thread_title=thread_title,
|
||||
mode=mode,
|
||||
title=title,
|
||||
tags=tags,
|
||||
project_name=project_name,
|
||||
sources=sources,
|
||||
max_messages=max_messages,
|
||||
model_name=model_name,
|
||||
max_input_chars=max_input_chars,
|
||||
allow_multi=False,
|
||||
)
|
||||
except LLMRejectedIngest:
|
||||
return None
|
||||
return drafts[0] if drafts else None
|
||||
@@ -0,0 +1,74 @@
|
||||
""".manifest.json read/write for the vault.
|
||||
|
||||
The manifest records, per page, the source key, content hash, vault path and
|
||||
update time — so batch ingest can skip already-sedimented threads (delta/dedup)
|
||||
exactly like obsidian-wiki's native ``.manifest.json``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
def content_hash(text: str) -> str:
|
||||
"""Stable sha256 hex digest of ``text`` (used for dedup)."""
|
||||
return hashlib.sha256((text or "").encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
class Manifest:
|
||||
"""JSON manifest stored at ``<vault>/.manifest.json``."""
|
||||
|
||||
def __init__(self, root: Path | str) -> None:
|
||||
self.path = Path(root).resolve() / ".manifest.json"
|
||||
|
||||
def load(self) -> dict[str, Any]:
|
||||
if not self.path.exists():
|
||||
return {"version": 1, "updated": None, "pages": {}, "sources": {}}
|
||||
try:
|
||||
data = json.loads(self.path.read_text(encoding="utf-8"))
|
||||
except (json.JSONDecodeError, OSError):
|
||||
return {"version": 1, "updated": None, "pages": {}, "sources": {}}
|
||||
data.setdefault("pages", {})
|
||||
data.setdefault("sources", {})
|
||||
return data
|
||||
|
||||
def save(self, data: dict[str, Any]) -> None:
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
self.path.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
|
||||
def record_page(
|
||||
self,
|
||||
*,
|
||||
note_id: str,
|
||||
vault_path: str,
|
||||
source_key: str | None,
|
||||
hash_value: str,
|
||||
updated: str,
|
||||
) -> None:
|
||||
data = self.load()
|
||||
data["pages"][note_id] = {
|
||||
"vault_path": vault_path,
|
||||
"source": source_key,
|
||||
"hash": hash_value,
|
||||
"updated": updated,
|
||||
}
|
||||
if source_key:
|
||||
data["sources"][source_key] = {"note_id": note_id, "hash": hash_value, "updated": updated}
|
||||
data["updated"] = updated
|
||||
self.save(data)
|
||||
|
||||
def remove_page(self, note_id: str) -> None:
|
||||
data = self.load()
|
||||
page = data["pages"].pop(note_id, None)
|
||||
if page and page.get("source"):
|
||||
data["sources"].pop(page["source"], None)
|
||||
self.save(data)
|
||||
|
||||
def has_source(self, source_key: str) -> bool:
|
||||
return source_key in self.load().get("sources", {})
|
||||
|
||||
def get_source(self, source_key: str) -> dict[str, Any] | None:
|
||||
return self.load().get("sources", {}).get(source_key)
|
||||
+213
@@ -0,0 +1,213 @@
|
||||
"""obsidian-wiki compatible Markdown vault writer/reader.
|
||||
|
||||
Responsibilities (dev doc §10 ``markdown_vault.py``):
|
||||
|
||||
- Generate obsidian-wiki compatible frontmatter.
|
||||
- Produce safe, path-traversal-proof file names/paths.
|
||||
- Write/update/read Markdown note files.
|
||||
- Parse frontmatter + extract ``[[wikilinks]]``.
|
||||
|
||||
The vault layout follows §8 of the dev doc and stays compatible with
|
||||
``ar9av/obsidian-wiki`` (concepts/ references/ synthesis/ journal/ projects/…,
|
||||
plus index.md / log.md / hot.md / .manifest.json / _meta/taxonomy.md /
|
||||
wiki-export/).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import unicodedata
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
_WIKILINK_RE = re.compile(r"\[\[([^\]|]+)(?:\|[^\]]+)?\]\]")
|
||||
_FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n?(.*)$", re.DOTALL)
|
||||
_UNSAFE_CHARS_RE = re.compile(r"[^\w一-鿿-]+")
|
||||
|
||||
# Top-level vault directories that must exist (obsidian-wiki native layout).
|
||||
_VAULT_DIRS = ("concepts", "entities", "skills", "references", "synthesis", "journal", "projects", "_meta", "wiki-export")
|
||||
|
||||
|
||||
def slugify(title: str, *, max_len: int = 60) -> str:
|
||||
"""Return a filesystem-safe slug (ASCII + CJK), never empty."""
|
||||
text = unicodedata.normalize("NFKC", (title or "").strip()).lower()
|
||||
text = text.replace(" ", "-")
|
||||
text = _UNSAFE_CHARS_RE.sub("-", text).strip("-")
|
||||
text = re.sub(r"-{2,}", "-", text)
|
||||
if len(text) > max_len:
|
||||
text = text[:max_len].rstrip("-")
|
||||
return text or "note"
|
||||
|
||||
|
||||
def extract_wikilinks(text: str) -> list[str]:
|
||||
"""Return the targets of every ``[[wikilink]]`` in ``text`` (de-duplicated)."""
|
||||
seen: list[str] = []
|
||||
for match in _WIKILINK_RE.finditer(text or ""):
|
||||
target = match.group(1).strip()
|
||||
if target and target not in seen:
|
||||
seen.append(target)
|
||||
return seen
|
||||
|
||||
|
||||
def normalize_wikilink_target(target: str) -> str:
|
||||
"""Canonicalize a ``[[wikilink]]`` target for resolution/comparison.
|
||||
|
||||
Drops any ``|alias`` part, normalizes slashes, strips a leading ``/`` and a
|
||||
trailing ``.md`` — so ``[[index]]``, ``index.md`` and ``/index`` all map to
|
||||
the same canonical ``index`` key used by the index/graph/resolver layers.
|
||||
"""
|
||||
t = (target or "").strip()
|
||||
if not t:
|
||||
return ""
|
||||
t = t.split("|", 1)[0].strip()
|
||||
t = t.replace("\\", "/").lstrip("/")
|
||||
if t.lower().endswith(".md"):
|
||||
t = t[:-3]
|
||||
return t
|
||||
|
||||
|
||||
class VaultPathError(ValueError):
|
||||
"""Raised when a requested vault path escapes the vault root."""
|
||||
|
||||
|
||||
class MarkdownVault:
|
||||
"""Reads/writes Markdown files inside a single vault root directory."""
|
||||
|
||||
def __init__(self, root: Path | str) -> None:
|
||||
self.root = Path(root).resolve()
|
||||
|
||||
# -- path safety ------------------------------------------------------
|
||||
def resolve(self, rel_path: str) -> Path:
|
||||
"""Resolve ``rel_path`` under the vault root, blocking traversal."""
|
||||
rel = str(rel_path).replace("\\", "/").lstrip("/")
|
||||
candidate = (self.root / rel).resolve()
|
||||
try:
|
||||
candidate.relative_to(self.root)
|
||||
except ValueError as exc:
|
||||
raise VaultPathError(f"Path escapes vault root: {rel_path!r}") from exc
|
||||
return candidate
|
||||
|
||||
def note_relpath(self, *, category: str, project_name: str | None, slug: str, note_id: str) -> str:
|
||||
"""Build a vault-relative path for a note.
|
||||
|
||||
Project knowledge goes under ``projects/<project>/<category>/`` per
|
||||
wiki-capture; a short id suffix guarantees uniqueness.
|
||||
"""
|
||||
from deerflow.knowledge.skill_templates import category_to_dir
|
||||
|
||||
cat_dir = category_to_dir(category)
|
||||
safe_slug = slugify(slug)
|
||||
suffix = note_id.split("_")[-1][:8]
|
||||
filename = f"{safe_slug}-{suffix}.md"
|
||||
if project_name:
|
||||
return f"projects/{slugify(project_name)}/{cat_dir}/{filename}"
|
||||
return f"{cat_dir}/{filename}"
|
||||
|
||||
# -- scaffold ---------------------------------------------------------
|
||||
def ensure_scaffold(self, *, project_name: str | None = None) -> None:
|
||||
"""Create the vault directory tree + base files if missing."""
|
||||
self.root.mkdir(parents=True, exist_ok=True)
|
||||
for d in _VAULT_DIRS:
|
||||
(self.root / d).mkdir(parents=True, exist_ok=True)
|
||||
if project_name:
|
||||
base = self.root / "projects" / slugify(project_name)
|
||||
for d in ("concepts", "references", "synthesis", "journal"):
|
||||
(base / d).mkdir(parents=True, exist_ok=True)
|
||||
# Base vault files.
|
||||
index = self.root / "index.md"
|
||||
if not index.exists():
|
||||
index.write_text("# Knowledge Index\n\n_页面索引。每次写入新页面后自动更新。_\n\n", encoding="utf-8")
|
||||
log = self.root / "log.md"
|
||||
if not log.exists():
|
||||
log.write_text("# Activity Log\n\n_CAPTURE / INGEST / QUERY / EXPORT 记录。_\n\n", encoding="utf-8")
|
||||
hot = self.root / "hot.md"
|
||||
if not hot.exists():
|
||||
hot.write_text("# Hot\n\n_近期活动摘要。_\n\n", encoding="utf-8")
|
||||
taxonomy = self.root / "_meta" / "taxonomy.md"
|
||||
if not taxonomy.exists():
|
||||
taxonomy.write_text(
|
||||
"# Taxonomy\n\n- synthesis — 多步骤分析、方案、结论\n- concepts — 概念、框架、模型\n"
|
||||
"- references — 外部来源、搜索结果、文章资料\n- decision — 架构或设计决策\n- journal — 完整会话摘要\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
# -- frontmatter ------------------------------------------------------
|
||||
@staticmethod
|
||||
def build_frontmatter(
|
||||
*,
|
||||
note_id: str,
|
||||
title: str,
|
||||
category: str,
|
||||
tags: list[str],
|
||||
sources: list[str],
|
||||
summary: str,
|
||||
created: str,
|
||||
updated: str,
|
||||
base_confidence: float,
|
||||
lifecycle: str,
|
||||
provenance: dict[str, float] | None = None,
|
||||
relationships: list[dict[str, str]] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Build an obsidian-wiki / wiki-capture compatible frontmatter dict."""
|
||||
created_date = (created or "")[:10]
|
||||
return {
|
||||
"id": note_id,
|
||||
"title": title,
|
||||
"category": category,
|
||||
"tags": list(tags or []),
|
||||
"sources": list(sources or []),
|
||||
"created": created,
|
||||
"updated": updated,
|
||||
"summary": summary or "",
|
||||
"provenance": provenance or {"extracted": 0.7, "inferred": 0.2, "ambiguous": 0.1},
|
||||
"base_confidence": round(float(base_confidence or 0.0), 2),
|
||||
"lifecycle": lifecycle,
|
||||
"lifecycle_changed": created_date,
|
||||
"relationships": relationships or [],
|
||||
}
|
||||
|
||||
# -- IO ---------------------------------------------------------------
|
||||
def write_note(self, rel_path: str, frontmatter: dict[str, Any], body: str) -> Path:
|
||||
"""Write ``frontmatter`` + ``body`` to ``rel_path`` (creates dirs)."""
|
||||
path = self.resolve(rel_path)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
fm = yaml.safe_dump(frontmatter, allow_unicode=True, sort_keys=False, default_flow_style=False)
|
||||
path.write_text(f"---\n{fm}---\n\n{body.strip()}\n", encoding="utf-8")
|
||||
return path
|
||||
|
||||
def read_note(self, rel_path: str) -> tuple[dict[str, Any], str] | None:
|
||||
"""Return ``(frontmatter, body)`` for a note, or ``None`` if missing."""
|
||||
path = self.resolve(rel_path)
|
||||
if not path.exists():
|
||||
return None
|
||||
return self.parse(path.read_text(encoding="utf-8"))
|
||||
|
||||
@staticmethod
|
||||
def parse(text: str) -> tuple[dict[str, Any], str]:
|
||||
"""Split raw Markdown into ``(frontmatter dict, body)``."""
|
||||
match = _FRONTMATTER_RE.match(text or "")
|
||||
if not match:
|
||||
return {}, text or ""
|
||||
try:
|
||||
fm = yaml.safe_load(match.group(1)) or {}
|
||||
except yaml.YAMLError:
|
||||
fm = {}
|
||||
if not isinstance(fm, dict):
|
||||
fm = {}
|
||||
return fm, match.group(2) or ""
|
||||
|
||||
def delete_note(self, rel_path: str) -> None:
|
||||
"""Remove a note file if present (used on hard delete; archive keeps it)."""
|
||||
path = self.resolve(rel_path)
|
||||
if path.exists():
|
||||
path.unlink()
|
||||
|
||||
|
||||
def now_iso_with_tz(dt: datetime) -> str:
|
||||
"""ISO-8601 string with timezone offset (frontmatter ``created``/``updated``)."""
|
||||
if dt.tzinfo is None:
|
||||
return dt.isoformat()
|
||||
return dt.isoformat(timespec="seconds")
|
||||
@@ -0,0 +1,141 @@
|
||||
"""ORM models for the product knowledge base.
|
||||
|
||||
Tables mirror the obsidian-wiki vault into the database so the product can
|
||||
list, filter, search and audit notes quickly without re-parsing Markdown. The
|
||||
vault Markdown remains the source-of-truth for ``content_md`` / frontmatter /
|
||||
graph; these rows are an index + audit layer.
|
||||
|
||||
The knowledge base is **global / shared** — there is no ``user_id`` isolation.
|
||||
Writes record ``created_by`` / ``updated_by`` for auditing only.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from sqlalchemy import Float, ForeignKey, Integer, String, Text, UniqueConstraint
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from deerflow.persistence.base import Base
|
||||
from deerflow.persistence.types import BeijingDateTime, PortableJSON, PortableLongText
|
||||
|
||||
|
||||
class KnowledgeNoteRow(Base):
|
||||
"""Knowledge note main table (mirrors ``knowledge_notes``)."""
|
||||
|
||||
__tablename__ = "knowledge_notes"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
title: Mapped[str] = mapped_column(String(512))
|
||||
summary: Mapped[str | None] = mapped_column(Text)
|
||||
content_md: Mapped[str] = mapped_column(PortableLongText())
|
||||
vault_path: Mapped[str | None] = mapped_column(String(1024), index=True)
|
||||
source_type: Mapped[str] = mapped_column(String(64), index=True)
|
||||
source_id: Mapped[str | None] = mapped_column(String(256), index=True)
|
||||
status: Mapped[str] = mapped_column(String(32), default="approved", index=True)
|
||||
confidence: Mapped[float] = mapped_column(Float, default=0.0)
|
||||
created_by: Mapped[str | None] = mapped_column(String(128), index=True)
|
||||
updated_by: Mapped[str | None] = mapped_column(String(128))
|
||||
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), default=lambda: datetime.now(UTC), index=True)
|
||||
updated_at: Mapped[datetime] = mapped_column(BeijingDateTime(), default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC), index=True)
|
||||
|
||||
|
||||
class KnowledgeSourceRow(Base):
|
||||
"""Knowledge source table (mirrors ``knowledge_sources``)."""
|
||||
|
||||
__tablename__ = "knowledge_sources"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
note_id: Mapped[str] = mapped_column(String(64), ForeignKey("knowledge_notes.id", ondelete="CASCADE"), index=True)
|
||||
source_type: Mapped[str] = mapped_column(String(64))
|
||||
thread_id: Mapped[str | None] = mapped_column(String(128), index=True)
|
||||
message_id: Mapped[str | None] = mapped_column(String(128))
|
||||
tool_name: Mapped[str | None] = mapped_column(String(128))
|
||||
title: Mapped[str | None] = mapped_column(String(512))
|
||||
url: Mapped[str | None] = mapped_column(Text)
|
||||
snippet: Mapped[str | None] = mapped_column(Text)
|
||||
raw_json: Mapped[str | None] = mapped_column(PortableLongText())
|
||||
created_by: Mapped[str | None] = mapped_column(String(128))
|
||||
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), default=lambda: datetime.now(UTC))
|
||||
|
||||
|
||||
class KnowledgeTagRow(Base):
|
||||
"""Tag table (mirrors ``knowledge_tags``)."""
|
||||
|
||||
__tablename__ = "knowledge_tags"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
name: Mapped[str] = mapped_column(String(128), unique=True, index=True)
|
||||
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), default=lambda: datetime.now(UTC))
|
||||
|
||||
|
||||
class KnowledgeNoteTagRow(Base):
|
||||
"""Note ↔ tag join table (mirrors ``knowledge_note_tags``)."""
|
||||
|
||||
__tablename__ = "knowledge_note_tags"
|
||||
__table_args__ = (UniqueConstraint("note_id", "tag_id", name="uq_knowledge_note_tags"),)
|
||||
|
||||
note_id: Mapped[str] = mapped_column(String(64), ForeignKey("knowledge_notes.id", ondelete="CASCADE"), primary_key=True)
|
||||
tag_id: Mapped[str] = mapped_column(String(64), ForeignKey("knowledge_tags.id", ondelete="CASCADE"), primary_key=True)
|
||||
|
||||
|
||||
class KnowledgeEmbeddingRow(Base):
|
||||
"""Per-chunk embedding index (phase 2, mirrors ``knowledge_embeddings``).
|
||||
|
||||
The vector is stored as a JSON float array; first-release similarity is
|
||||
computed in Python (no external vector DB). ``vector_id`` is reserved for a
|
||||
future external store mapping.
|
||||
"""
|
||||
|
||||
__tablename__ = "knowledge_embeddings"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
note_id: Mapped[str] = mapped_column(String(64), ForeignKey("knowledge_notes.id", ondelete="CASCADE"), index=True)
|
||||
chunk_index: Mapped[int] = mapped_column(Integer, default=0)
|
||||
content: Mapped[str] = mapped_column(Text)
|
||||
vector: Mapped[list] = mapped_column(PortableJSON(), default=list)
|
||||
vector_id: Mapped[str | None] = mapped_column(String(256))
|
||||
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), default=lambda: datetime.now(UTC))
|
||||
|
||||
|
||||
class KnowledgeNoteVersionRow(Base):
|
||||
"""Immutable snapshot of a note before each update (phase 3 versioning)."""
|
||||
|
||||
__tablename__ = "knowledge_note_versions"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
note_id: Mapped[str] = mapped_column(String(64), ForeignKey("knowledge_notes.id", ondelete="CASCADE"), index=True)
|
||||
title: Mapped[str] = mapped_column(String(512))
|
||||
summary: Mapped[str | None] = mapped_column(Text)
|
||||
content_md: Mapped[str] = mapped_column(PortableLongText())
|
||||
status: Mapped[str] = mapped_column(String(32))
|
||||
tags_json: Mapped[list] = mapped_column(PortableJSON(), default=list)
|
||||
edited_by: Mapped[str | None] = mapped_column(String(128))
|
||||
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), default=lambda: datetime.now(UTC), index=True)
|
||||
|
||||
|
||||
class KnowledgeEntityRow(Base):
|
||||
"""Knowledge entity (phase 4, mirrors ``knowledge_entities``)."""
|
||||
|
||||
__tablename__ = "knowledge_entities"
|
||||
__table_args__ = (UniqueConstraint("name", "entity_type", name="uq_knowledge_entities_name_type"),)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
name: Mapped[str] = mapped_column(String(256), index=True)
|
||||
entity_type: Mapped[str | None] = mapped_column(String(64))
|
||||
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), default=lambda: datetime.now(UTC))
|
||||
|
||||
|
||||
class KnowledgeRelationRow(Base):
|
||||
"""Relation between notes/entities (phase 4, mirrors ``knowledge_relations``)."""
|
||||
|
||||
__tablename__ = "knowledge_relations"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
from_note_id: Mapped[str | None] = mapped_column(String(64), index=True)
|
||||
to_note_id: Mapped[str | None] = mapped_column(String(64), index=True)
|
||||
from_entity_id: Mapped[str | None] = mapped_column(String(64), index=True)
|
||||
to_entity_id: Mapped[str | None] = mapped_column(String(64), index=True)
|
||||
relation_type: Mapped[str] = mapped_column(String(128))
|
||||
weight: Mapped[float] = mapped_column(Float, default=1.0)
|
||||
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), default=lambda: datetime.now(UTC))
|
||||
+254
@@ -0,0 +1,254 @@
|
||||
"""ObsidianWikiAdapter — the obsidian-wiki vault facade.
|
||||
|
||||
Owns the Markdown vault: writes notes with wiki-capture frontmatter/body, keeps
|
||||
``index.md`` / ``log.md`` / ``hot.md`` / ``.manifest.json`` in sync after every
|
||||
write, and produces wiki-export graph artifacts. Output stays compatible with
|
||||
``ar9av/obsidian-wiki`` without ever invoking its CLI.
|
||||
|
||||
This class is storage-only (no DB, no LLM). The :class:`KnowledgeService`
|
||||
combines it with the repository + extractor.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from deerflow.knowledge import exporter
|
||||
from deerflow.knowledge.manifest import Manifest, content_hash
|
||||
from deerflow.knowledge.markdown_vault import MarkdownVault
|
||||
from deerflow.knowledge.schemas import NoteDraft
|
||||
|
||||
_INDEX_PAGES_HEADER = "## Pages"
|
||||
_HOT_HEADER = "## Recent"
|
||||
_HOT_MAX = 20
|
||||
|
||||
|
||||
class ObsidianWikiAdapter:
|
||||
"""Read/write an obsidian-wiki compatible vault."""
|
||||
|
||||
def __init__(self, vault_root: Path | str, *, project_name: str = "zncm", export_html: bool = True, export_json: bool = True) -> None:
|
||||
self.vault = MarkdownVault(vault_root)
|
||||
self.root = self.vault.root
|
||||
self.manifest = Manifest(self.root)
|
||||
self.project_name = project_name
|
||||
self.export_html = export_html
|
||||
self.export_json = export_json
|
||||
|
||||
def ensure_ready(self) -> None:
|
||||
self.vault.ensure_scaffold(project_name=self.project_name)
|
||||
|
||||
# -- writes -----------------------------------------------------------
|
||||
def write_note(
|
||||
self,
|
||||
note_id: str,
|
||||
draft: NoteDraft,
|
||||
*,
|
||||
created: str,
|
||||
updated: str,
|
||||
status: str,
|
||||
) -> str:
|
||||
"""Write the note file, sync vault metadata, return the vault path."""
|
||||
self.ensure_ready()
|
||||
rel = self.vault.note_relpath(category=draft.category, project_name=self.project_name, slug=draft.title, note_id=note_id)
|
||||
sources_fm = self._frontmatter_sources(draft)
|
||||
relationships = [{"target": link, "type": "related"} for link in (draft.related or [])[:8]]
|
||||
frontmatter = self.vault.build_frontmatter(
|
||||
note_id=note_id,
|
||||
title=draft.title,
|
||||
category=draft.category,
|
||||
tags=draft.tags,
|
||||
sources=sources_fm,
|
||||
summary=draft.summary,
|
||||
created=created,
|
||||
updated=updated,
|
||||
base_confidence=draft.confidence,
|
||||
lifecycle=status,
|
||||
relationships=relationships,
|
||||
)
|
||||
self.vault.write_note(rel, frontmatter, draft.content_md)
|
||||
self.manifest.record_page(
|
||||
note_id=note_id,
|
||||
vault_path=rel,
|
||||
source_key=draft.source_key,
|
||||
hash_value=content_hash(draft.content_md),
|
||||
updated=updated,
|
||||
)
|
||||
self._update_index(note_id, rel, draft.title, draft.summary)
|
||||
self._append_log("CAPTURE" if draft.source_type in ("thread", "search", "tool") else "INGEST", note_id, draft.title, updated)
|
||||
self._update_hot(draft.title, rel, updated)
|
||||
return rel
|
||||
|
||||
def update_note_file(
|
||||
self,
|
||||
note_id: str,
|
||||
vault_path: str | None,
|
||||
*,
|
||||
title: str,
|
||||
content_md: str,
|
||||
tags: list[str],
|
||||
status: str,
|
||||
summary: str,
|
||||
category: str,
|
||||
created: str,
|
||||
updated: str,
|
||||
) -> str:
|
||||
"""Rewrite a note file in place (frontmatter refreshed), sync metadata."""
|
||||
self.ensure_ready()
|
||||
rel = vault_path
|
||||
if not rel:
|
||||
rel = self.vault.note_relpath(category=category, project_name=self.project_name, slug=title, note_id=note_id)
|
||||
# Preserve original `sources`/`provenance`/`category` frontmatter when present.
|
||||
existing = self.vault.read_note(rel)
|
||||
sources_fm = existing[0].get("sources", []) if existing else []
|
||||
provenance = existing[0].get("provenance") if existing else None
|
||||
relationships = existing[0].get("relationships") if existing else []
|
||||
effective_category = (existing[0].get("category") if existing else None) or category
|
||||
frontmatter = self.vault.build_frontmatter(
|
||||
note_id=note_id,
|
||||
title=title,
|
||||
category=effective_category,
|
||||
tags=tags,
|
||||
sources=sources_fm,
|
||||
summary=summary,
|
||||
created=created,
|
||||
updated=updated,
|
||||
base_confidence=float((existing[0].get("base_confidence") if existing else 0.0) or 0.0),
|
||||
lifecycle=status,
|
||||
provenance=provenance,
|
||||
relationships=relationships,
|
||||
)
|
||||
self.vault.write_note(rel, frontmatter, content_md)
|
||||
self.manifest.record_page(
|
||||
note_id=note_id,
|
||||
vault_path=rel,
|
||||
source_key=self.manifest.load().get("pages", {}).get(note_id, {}).get("source"),
|
||||
hash_value=content_hash(content_md),
|
||||
updated=updated,
|
||||
)
|
||||
self._update_index(note_id, rel, title, summary)
|
||||
self._append_log("UPDATE", note_id, title, updated)
|
||||
self._update_hot(title, rel, updated)
|
||||
return rel
|
||||
|
||||
def archive_note(self, note_id: str, vault_path: str | None, *, updated: str) -> None:
|
||||
"""Mark a note archived in its frontmatter + log (soft delete)."""
|
||||
if vault_path:
|
||||
parsed = self.vault.read_note(vault_path)
|
||||
if parsed:
|
||||
fm, body = parsed
|
||||
fm["lifecycle"] = "archived"
|
||||
fm["updated"] = updated
|
||||
self.vault.write_note(vault_path, fm, body)
|
||||
self._remove_from_index(note_id)
|
||||
self._append_log("ARCHIVE", note_id, "", updated)
|
||||
|
||||
def export_graph(
|
||||
self,
|
||||
notes: list[dict[str, Any]],
|
||||
*,
|
||||
entities: list[dict[str, Any]] | None = None,
|
||||
relations: list[dict[str, Any]] | None = None,
|
||||
updated: str,
|
||||
) -> dict[str, Any]:
|
||||
self.ensure_ready()
|
||||
result = exporter.export_all(
|
||||
self.root,
|
||||
notes,
|
||||
entities=entities,
|
||||
relations=relations,
|
||||
write_html=self.export_html,
|
||||
write_json=self.export_json,
|
||||
)
|
||||
self._append_log("EXPORT", "graph", f"nodes={result['stats']['nodes']} edges={result['stats']['edges']}", updated)
|
||||
return result
|
||||
|
||||
def export_file_path(self, filename: str) -> Path | None:
|
||||
allowed = {"graph.json", "graph.graphml", "cypher.txt", "graph.html"}
|
||||
if filename not in allowed:
|
||||
return None
|
||||
path = (self.root / "wiki-export" / filename).resolve()
|
||||
try:
|
||||
path.relative_to(self.root)
|
||||
except ValueError:
|
||||
return None
|
||||
return path if path.exists() else None
|
||||
|
||||
# -- frontmatter source serialization ---------------------------------
|
||||
@staticmethod
|
||||
def _frontmatter_sources(draft: NoteDraft) -> list[str]:
|
||||
out: list[str] = []
|
||||
if draft.source_type == "thread" and draft.source_id:
|
||||
out.append(f"thread:{draft.source_id}")
|
||||
for s in draft.sources[:20]:
|
||||
if s.url:
|
||||
out.append(f"url:{s.url}")
|
||||
elif s.thread_id:
|
||||
out.append(f"thread:{s.thread_id}")
|
||||
elif s.tool_name:
|
||||
out.append(f"tool:{s.tool_name}")
|
||||
# De-dup preserving order.
|
||||
seen: set[str] = set()
|
||||
deduped = []
|
||||
for item in out:
|
||||
if item not in seen:
|
||||
seen.add(item)
|
||||
deduped.append(item)
|
||||
return deduped
|
||||
|
||||
# -- index.md ---------------------------------------------------------
|
||||
def _index_path(self) -> Path:
|
||||
return self.root / "index.md"
|
||||
|
||||
def _update_index(self, note_id: str, rel: str, title: str, summary: str | None) -> None:
|
||||
path = self._index_path()
|
||||
text = path.read_text(encoding="utf-8") if path.exists() else "# Knowledge Index\n\n"
|
||||
link = rel[:-3] if rel.endswith(".md") else rel
|
||||
summary_txt = (summary or "").strip().replace("\n", " ")
|
||||
if len(summary_txt) > 120:
|
||||
summary_txt = summary_txt[:120] + "…"
|
||||
line = f"- [[{link}|{title}]]" + (f" — {summary_txt}" if summary_txt else "") + f" <!-- id:{note_id} -->"
|
||||
lines = text.splitlines()
|
||||
# Drop any existing line for this note id.
|
||||
lines = [ln for ln in lines if f"<!-- id:{note_id} -->" not in ln]
|
||||
if _INDEX_PAGES_HEADER not in text:
|
||||
lines.append("")
|
||||
lines.append(_INDEX_PAGES_HEADER)
|
||||
lines.append("")
|
||||
# Insert after the Pages header.
|
||||
try:
|
||||
insert_at = lines.index(_INDEX_PAGES_HEADER) + 1
|
||||
except ValueError:
|
||||
insert_at = len(lines)
|
||||
lines.insert(insert_at, line)
|
||||
path.write_text("\n".join(lines).rstrip() + "\n", encoding="utf-8")
|
||||
|
||||
def _remove_from_index(self, note_id: str) -> None:
|
||||
path = self._index_path()
|
||||
if not path.exists():
|
||||
return
|
||||
lines = [ln for ln in path.read_text(encoding="utf-8").splitlines() if f"<!-- id:{note_id} -->" not in ln]
|
||||
path.write_text("\n".join(lines).rstrip() + "\n", encoding="utf-8")
|
||||
|
||||
# -- log.md -----------------------------------------------------------
|
||||
def _append_log(self, action: str, note_id: str, label: str, updated: str) -> None:
|
||||
path = self.root / "log.md"
|
||||
text = path.read_text(encoding="utf-8") if path.exists() else "# Activity Log\n\n"
|
||||
entry = f"- {updated} {action} {note_id}" + (f" — {label}" if label else "")
|
||||
path.write_text(text.rstrip() + "\n" + entry + "\n", encoding="utf-8")
|
||||
|
||||
# -- hot.md -----------------------------------------------------------
|
||||
def _update_hot(self, title: str, rel: str, updated: str) -> None:
|
||||
path = self.root / "hot.md"
|
||||
link = rel[:-3] if rel.endswith(".md") else rel
|
||||
new_line = f"- {updated} [[{link}|{title}]]"
|
||||
existing_lines: list[str] = []
|
||||
if path.exists():
|
||||
for ln in path.read_text(encoding="utf-8").splitlines():
|
||||
if ln.startswith("- ") and "[[" in ln:
|
||||
existing_lines.append(ln)
|
||||
existing_lines = [ln for ln in existing_lines if f"[[{link}" not in ln]
|
||||
existing_lines.insert(0, new_line)
|
||||
existing_lines = existing_lines[:_HOT_MAX]
|
||||
body = "# Hot\n\n_近期活动摘要。_\n\n" + _HOT_HEADER + "\n\n" + "\n".join(existing_lines) + "\n"
|
||||
path.write_text(body, encoding="utf-8")
|
||||
@@ -0,0 +1,528 @@
|
||||
"""Database read/write for the knowledge base.
|
||||
|
||||
This layer never calls an LLM and never touches the Markdown vault — it only
|
||||
persists/queries the relational mirror (notes, sources, tags). The vault and
|
||||
extraction live in :mod:`deerflow.knowledge.markdown_vault` and
|
||||
:mod:`deerflow.knowledge.extractor`; orchestration lives in the service.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import delete, func, or_, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
from deerflow.knowledge.models import (
|
||||
KnowledgeEmbeddingRow,
|
||||
KnowledgeEntityRow,
|
||||
KnowledgeNoteRow,
|
||||
KnowledgeNoteTagRow,
|
||||
KnowledgeNoteVersionRow,
|
||||
KnowledgeRelationRow,
|
||||
KnowledgeSourceRow,
|
||||
KnowledgeTagRow,
|
||||
)
|
||||
from deerflow.knowledge.schemas import SourceDraft
|
||||
|
||||
|
||||
def _new_id(prefix: str) -> str:
|
||||
return f"{prefix}_{uuid.uuid4().hex[:24]}"
|
||||
|
||||
|
||||
def _iso(value: Any) -> Any:
|
||||
if isinstance(value, datetime):
|
||||
return value.isoformat()
|
||||
return value
|
||||
|
||||
|
||||
class KnowledgeRepository:
|
||||
"""SQLAlchemy-backed repository for knowledge notes/sources/tags."""
|
||||
|
||||
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
|
||||
self._sf = session_factory
|
||||
|
||||
# -- serialization ----------------------------------------------------
|
||||
@staticmethod
|
||||
def _note_to_dict(row: KnowledgeNoteRow) -> dict[str, Any]:
|
||||
d = row.to_dict()
|
||||
d["created_at"] = _iso(d.get("created_at"))
|
||||
d["updated_at"] = _iso(d.get("updated_at"))
|
||||
return d
|
||||
|
||||
@staticmethod
|
||||
def _source_to_dict(row: KnowledgeSourceRow) -> dict[str, Any]:
|
||||
d = row.to_dict()
|
||||
d["created_at"] = _iso(d.get("created_at"))
|
||||
return d
|
||||
|
||||
# -- tags -------------------------------------------------------------
|
||||
async def _get_or_create_tags(self, session: AsyncSession, names: list[str]) -> list[KnowledgeTagRow]:
|
||||
cleaned = []
|
||||
seen: set[str] = set()
|
||||
for raw in names or []:
|
||||
name = (raw or "").strip()
|
||||
if not name or name.lower() in seen:
|
||||
continue
|
||||
seen.add(name.lower())
|
||||
cleaned.append(name)
|
||||
if not cleaned:
|
||||
return []
|
||||
existing = {
|
||||
r.name.lower(): r
|
||||
for r in (await session.execute(select(KnowledgeTagRow).where(func.lower(KnowledgeTagRow.name).in_([n.lower() for n in cleaned])))).scalars()
|
||||
}
|
||||
result: list[KnowledgeTagRow] = []
|
||||
for name in cleaned:
|
||||
row = existing.get(name.lower())
|
||||
if row is None:
|
||||
row = KnowledgeTagRow(id=_new_id("kbtag"), name=name, created_at=datetime.now(UTC))
|
||||
session.add(row)
|
||||
await session.flush()
|
||||
existing[name.lower()] = row
|
||||
result.append(row)
|
||||
return result
|
||||
|
||||
async def _tags_for(self, session: AsyncSession, note_id: str) -> list[str]:
|
||||
stmt = (
|
||||
select(KnowledgeTagRow.name)
|
||||
.join(KnowledgeNoteTagRow, KnowledgeNoteTagRow.tag_id == KnowledgeTagRow.id)
|
||||
.where(KnowledgeNoteTagRow.note_id == note_id)
|
||||
.order_by(KnowledgeTagRow.name)
|
||||
)
|
||||
return list((await session.execute(stmt)).scalars())
|
||||
|
||||
async def _tags_for_many(self, session: AsyncSession, note_ids: list[str]) -> dict[str, list[str]]:
|
||||
if not note_ids:
|
||||
return {}
|
||||
stmt = (
|
||||
select(KnowledgeNoteTagRow.note_id, KnowledgeTagRow.name)
|
||||
.join(KnowledgeTagRow, KnowledgeNoteTagRow.tag_id == KnowledgeTagRow.id)
|
||||
.where(KnowledgeNoteTagRow.note_id.in_(note_ids))
|
||||
.order_by(KnowledgeTagRow.name)
|
||||
)
|
||||
out: dict[str, list[str]] = {nid: [] for nid in note_ids}
|
||||
for note_id, name in (await session.execute(stmt)).all():
|
||||
out.setdefault(note_id, []).append(name)
|
||||
return out
|
||||
|
||||
async def _replace_note_tags(self, session: AsyncSession, note_id: str, names: list[str]) -> None:
|
||||
await session.execute(delete(KnowledgeNoteTagRow).where(KnowledgeNoteTagRow.note_id == note_id))
|
||||
tags = await self._get_or_create_tags(session, names)
|
||||
for tag in tags:
|
||||
session.add(KnowledgeNoteTagRow(note_id=note_id, tag_id=tag.id))
|
||||
|
||||
# -- notes ------------------------------------------------------------
|
||||
async def create_note(
|
||||
self,
|
||||
*,
|
||||
title: str,
|
||||
summary: str | None,
|
||||
content_md: str,
|
||||
source_type: str,
|
||||
source_id: str | None,
|
||||
status: str,
|
||||
confidence: float = 0.0,
|
||||
vault_path: str | None = None,
|
||||
tags: list[str] | None = None,
|
||||
created_by: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
note_id = _new_id("kb")
|
||||
now = datetime.now(UTC)
|
||||
row = KnowledgeNoteRow(
|
||||
id=note_id,
|
||||
title=title[:512],
|
||||
summary=summary,
|
||||
content_md=content_md,
|
||||
vault_path=vault_path,
|
||||
source_type=source_type,
|
||||
source_id=source_id,
|
||||
status=status,
|
||||
confidence=confidence,
|
||||
created_by=created_by,
|
||||
updated_by=created_by,
|
||||
created_at=now,
|
||||
updated_at=now,
|
||||
)
|
||||
async with self._sf() as session:
|
||||
session.add(row)
|
||||
await session.flush()
|
||||
await self._replace_note_tags(session, note_id, tags or [])
|
||||
await session.commit()
|
||||
await session.refresh(row)
|
||||
d = self._note_to_dict(row)
|
||||
d["tags"] = await self._tags_for(session, note_id)
|
||||
return d
|
||||
|
||||
async def set_vault_path(self, note_id: str, vault_path: str) -> None:
|
||||
async with self._sf() as session:
|
||||
row = await session.get(KnowledgeNoteRow, note_id)
|
||||
if row is None:
|
||||
return
|
||||
row.vault_path = vault_path
|
||||
await session.commit()
|
||||
|
||||
async def update_note(
|
||||
self,
|
||||
note_id: str,
|
||||
*,
|
||||
title: str | None = None,
|
||||
summary: str | None = None,
|
||||
content_md: str | None = None,
|
||||
status: str | None = None,
|
||||
tags: list[str] | None = None,
|
||||
vault_path: str | None = None,
|
||||
updated_by: str | None = None,
|
||||
) -> dict[str, Any] | None:
|
||||
async with self._sf() as session:
|
||||
row = await session.get(KnowledgeNoteRow, note_id)
|
||||
if row is None:
|
||||
return None
|
||||
if title is not None:
|
||||
row.title = title[:512]
|
||||
if summary is not None:
|
||||
row.summary = summary
|
||||
if content_md is not None:
|
||||
row.content_md = content_md
|
||||
if status is not None:
|
||||
row.status = status
|
||||
if vault_path is not None:
|
||||
row.vault_path = vault_path
|
||||
if updated_by is not None:
|
||||
row.updated_by = updated_by
|
||||
row.updated_at = datetime.now(UTC)
|
||||
if tags is not None:
|
||||
await self._replace_note_tags(session, note_id, tags)
|
||||
await session.commit()
|
||||
await session.refresh(row)
|
||||
d = self._note_to_dict(row)
|
||||
d["tags"] = await self._tags_for(session, note_id)
|
||||
return d
|
||||
|
||||
async def set_status(self, note_id: str, status: str, *, updated_by: str | None = None) -> dict[str, Any] | None:
|
||||
return await self.update_note(note_id, status=status, updated_by=updated_by)
|
||||
|
||||
async def get_note(self, note_id: str, *, with_sources: bool = True) -> dict[str, Any] | None:
|
||||
async with self._sf() as session:
|
||||
row = await session.get(KnowledgeNoteRow, note_id)
|
||||
if row is None:
|
||||
return None
|
||||
d = self._note_to_dict(row)
|
||||
d["tags"] = await self._tags_for(session, note_id)
|
||||
if with_sources:
|
||||
stmt = select(KnowledgeSourceRow).where(KnowledgeSourceRow.note_id == note_id).order_by(KnowledgeSourceRow.created_at)
|
||||
d["sources"] = [self._source_to_dict(r) for r in (await session.execute(stmt)).scalars()]
|
||||
return d
|
||||
|
||||
async def find_by_vault_path(self, vault_path: str) -> dict[str, Any] | None:
|
||||
"""Return the first (non-archived) note whose vault_path matches.
|
||||
|
||||
``vault_path`` is matched both with and without the ``.md`` suffix, since
|
||||
wikilink targets are stored without it while ``vault_path`` keeps it.
|
||||
"""
|
||||
stripped = vault_path[:-3] if vault_path.endswith(".md") else vault_path
|
||||
candidates = {vault_path, stripped, f"{stripped}.md"}
|
||||
async with self._sf() as session:
|
||||
stmt = (
|
||||
select(KnowledgeNoteRow)
|
||||
.where(KnowledgeNoteRow.vault_path.in_(list(candidates)))
|
||||
.where(KnowledgeNoteRow.status != "archived")
|
||||
.limit(1)
|
||||
)
|
||||
row = (await session.execute(stmt)).scalars().first()
|
||||
if row is None:
|
||||
return None
|
||||
d = self._note_to_dict(row)
|
||||
d["tags"] = await self._tags_for(session, row.id)
|
||||
return d
|
||||
|
||||
async def find_by_title(self, title: str) -> dict[str, Any] | None:
|
||||
"""Return the first (non-archived) note with a case-insensitive title match."""
|
||||
name = (title or "").strip()
|
||||
if not name:
|
||||
return None
|
||||
async with self._sf() as session:
|
||||
stmt = (
|
||||
select(KnowledgeNoteRow)
|
||||
.where(func.lower(KnowledgeNoteRow.title) == name.lower())
|
||||
.where(KnowledgeNoteRow.status != "archived")
|
||||
.order_by(KnowledgeNoteRow.updated_at.desc())
|
||||
.limit(1)
|
||||
)
|
||||
row = (await session.execute(stmt)).scalars().first()
|
||||
if row is None:
|
||||
return None
|
||||
d = self._note_to_dict(row)
|
||||
d["tags"] = await self._tags_for(session, row.id)
|
||||
return d
|
||||
|
||||
async def find_by_source(self, source_type: str, source_id: str) -> dict[str, Any] | None:
|
||||
"""Return the first (non-archived) note for a (source_type, source_id)."""
|
||||
async with self._sf() as session:
|
||||
stmt = (
|
||||
select(KnowledgeNoteRow)
|
||||
.where(KnowledgeNoteRow.source_type == source_type, KnowledgeNoteRow.source_id == source_id)
|
||||
.where(KnowledgeNoteRow.status != "archived")
|
||||
.limit(1)
|
||||
)
|
||||
row = (await session.execute(stmt)).scalars().first()
|
||||
return self._note_to_dict(row) if row else None
|
||||
|
||||
async def list_notes(
|
||||
self,
|
||||
*,
|
||||
keyword: str | None = None,
|
||||
tag: str | None = None,
|
||||
source_type: str | None = None,
|
||||
status: str | None = None,
|
||||
include_archived: bool = False,
|
||||
limit: int = 20,
|
||||
offset: int = 0,
|
||||
) -> tuple[list[dict[str, Any]], int]:
|
||||
async with self._sf() as session:
|
||||
stmt = select(KnowledgeNoteRow)
|
||||
count_stmt = select(func.count()).select_from(KnowledgeNoteRow)
|
||||
|
||||
conditions = []
|
||||
if status:
|
||||
conditions.append(KnowledgeNoteRow.status == status)
|
||||
elif not include_archived:
|
||||
conditions.append(KnowledgeNoteRow.status != "archived")
|
||||
if source_type:
|
||||
conditions.append(KnowledgeNoteRow.source_type == source_type)
|
||||
if keyword:
|
||||
like = f"%{keyword}%"
|
||||
conditions.append(or_(KnowledgeNoteRow.title.like(like), KnowledgeNoteRow.summary.like(like), KnowledgeNoteRow.content_md.like(like)))
|
||||
if tag:
|
||||
tag_subq = (
|
||||
select(KnowledgeNoteTagRow.note_id)
|
||||
.join(KnowledgeTagRow, KnowledgeNoteTagRow.tag_id == KnowledgeTagRow.id)
|
||||
.where(func.lower(KnowledgeTagRow.name) == tag.lower())
|
||||
)
|
||||
conditions.append(KnowledgeNoteRow.id.in_(tag_subq))
|
||||
|
||||
for cond in conditions:
|
||||
stmt = stmt.where(cond)
|
||||
count_stmt = count_stmt.where(cond)
|
||||
|
||||
total = (await session.execute(count_stmt)).scalar_one()
|
||||
stmt = stmt.order_by(KnowledgeNoteRow.updated_at.desc()).limit(limit).offset(offset)
|
||||
rows = list((await session.execute(stmt)).scalars())
|
||||
ids = [r.id for r in rows]
|
||||
tags_map = await self._tags_for_many(session, ids)
|
||||
items = []
|
||||
for r in rows:
|
||||
d = self._note_to_dict(r)
|
||||
d["tags"] = tags_map.get(r.id, [])
|
||||
items.append(d)
|
||||
return items, int(total)
|
||||
|
||||
async def list_all(self, *, include_archived: bool = False) -> list[dict[str, Any]]:
|
||||
"""Return every note (light fields + tags) for graph export."""
|
||||
async with self._sf() as session:
|
||||
stmt = select(KnowledgeNoteRow)
|
||||
if not include_archived:
|
||||
stmt = stmt.where(KnowledgeNoteRow.status != "archived")
|
||||
rows = list((await session.execute(stmt)).scalars())
|
||||
tags_map = await self._tags_for_many(session, [r.id for r in rows])
|
||||
out = []
|
||||
for r in rows:
|
||||
d = self._note_to_dict(r)
|
||||
d["tags"] = tags_map.get(r.id, [])
|
||||
out.append(d)
|
||||
return out
|
||||
|
||||
# -- embeddings (phase 2) --------------------------------------------
|
||||
async def replace_embeddings(self, note_id: str, chunks: list[tuple[int, str, list[float]]]) -> None:
|
||||
"""Replace all embedding rows for a note with ``(index, content, vector)``."""
|
||||
async with self._sf() as session:
|
||||
await session.execute(delete(KnowledgeEmbeddingRow).where(KnowledgeEmbeddingRow.note_id == note_id))
|
||||
for idx, content, vector in chunks:
|
||||
session.add(
|
||||
KnowledgeEmbeddingRow(
|
||||
id=_new_id("kbemb"),
|
||||
note_id=note_id,
|
||||
chunk_index=idx,
|
||||
content=content,
|
||||
vector=list(vector),
|
||||
created_at=datetime.now(UTC),
|
||||
)
|
||||
)
|
||||
await session.commit()
|
||||
|
||||
async def delete_embeddings(self, note_id: str) -> None:
|
||||
async with self._sf() as session:
|
||||
await session.execute(delete(KnowledgeEmbeddingRow).where(KnowledgeEmbeddingRow.note_id == note_id))
|
||||
await session.commit()
|
||||
|
||||
async def all_embeddings(self) -> list[dict[str, Any]]:
|
||||
"""Return every embedding chunk joined to its (non-archived) note."""
|
||||
async with self._sf() as session:
|
||||
stmt = (
|
||||
select(KnowledgeEmbeddingRow, KnowledgeNoteRow.title, KnowledgeNoteRow.status)
|
||||
.join(KnowledgeNoteRow, KnowledgeNoteRow.id == KnowledgeEmbeddingRow.note_id)
|
||||
.where(KnowledgeNoteRow.status != "archived")
|
||||
)
|
||||
out = []
|
||||
for emb, title, _status in (await session.execute(stmt)).all():
|
||||
out.append({"note_id": emb.note_id, "chunk_index": emb.chunk_index, "content": emb.content, "vector": emb.vector or [], "title": title})
|
||||
return out
|
||||
|
||||
async def has_embeddings(self) -> bool:
|
||||
async with self._sf() as session:
|
||||
row = (await session.execute(select(KnowledgeEmbeddingRow.id).limit(1))).first()
|
||||
return row is not None
|
||||
|
||||
# -- versions (phase 3) ----------------------------------------------
|
||||
async def add_version(self, note: dict[str, Any], *, edited_by: str | None) -> None:
|
||||
async with self._sf() as session:
|
||||
session.add(
|
||||
KnowledgeNoteVersionRow(
|
||||
id=_new_id("kbver"),
|
||||
note_id=note["id"],
|
||||
title=note.get("title") or "",
|
||||
summary=note.get("summary"),
|
||||
content_md=note.get("content_md") or "",
|
||||
status=note.get("status") or "approved",
|
||||
tags_json=note.get("tags") or [],
|
||||
edited_by=edited_by,
|
||||
created_at=datetime.now(UTC),
|
||||
)
|
||||
)
|
||||
await session.commit()
|
||||
|
||||
async def list_versions(self, note_id: str) -> list[dict[str, Any]]:
|
||||
async with self._sf() as session:
|
||||
stmt = select(KnowledgeNoteVersionRow).where(KnowledgeNoteVersionRow.note_id == note_id).order_by(KnowledgeNoteVersionRow.created_at.desc())
|
||||
out = []
|
||||
for r in (await session.execute(stmt)).scalars():
|
||||
d = r.to_dict()
|
||||
d["tags"] = d.pop("tags_json", [])
|
||||
d["created_at"] = _iso(d.get("created_at"))
|
||||
out.append(d)
|
||||
return out
|
||||
|
||||
# -- entities & relations (phase 4) ----------------------------------
|
||||
async def get_or_create_entity(self, name: str, entity_type: str | None) -> dict[str, Any]:
|
||||
name = (name or "").strip()
|
||||
async with self._sf() as session:
|
||||
stmt = select(KnowledgeEntityRow).where(func.lower(KnowledgeEntityRow.name) == name.lower())
|
||||
row = (await session.execute(stmt)).scalars().first()
|
||||
if row is None:
|
||||
row = KnowledgeEntityRow(id=_new_id("kbent"), name=name, entity_type=entity_type, created_at=datetime.now(UTC))
|
||||
session.add(row)
|
||||
await session.commit()
|
||||
await session.refresh(row)
|
||||
return {"id": row.id, "name": row.name, "entity_type": row.entity_type}
|
||||
|
||||
async def add_relation(
|
||||
self,
|
||||
*,
|
||||
from_note_id: str | None = None,
|
||||
to_note_id: str | None = None,
|
||||
from_entity_id: str | None = None,
|
||||
to_entity_id: str | None = None,
|
||||
relation_type: str,
|
||||
weight: float = 1.0,
|
||||
) -> None:
|
||||
async with self._sf() as session:
|
||||
session.add(
|
||||
KnowledgeRelationRow(
|
||||
id=_new_id("kbrel"),
|
||||
from_note_id=from_note_id,
|
||||
to_note_id=to_note_id,
|
||||
from_entity_id=from_entity_id,
|
||||
to_entity_id=to_entity_id,
|
||||
relation_type=relation_type,
|
||||
weight=weight,
|
||||
created_at=datetime.now(UTC),
|
||||
)
|
||||
)
|
||||
await session.commit()
|
||||
|
||||
async def clear_note_relations(self, note_id: str) -> None:
|
||||
async with self._sf() as session:
|
||||
await session.execute(delete(KnowledgeRelationRow).where(KnowledgeRelationRow.from_note_id == note_id))
|
||||
await session.commit()
|
||||
|
||||
async def list_note_entities(self, note_id: str) -> list[dict[str, Any]]:
|
||||
"""Return the entities mentioned by a note (for note-detail display)."""
|
||||
async with self._sf() as session:
|
||||
stmt = (
|
||||
select(KnowledgeEntityRow.name, KnowledgeEntityRow.entity_type)
|
||||
.join(KnowledgeRelationRow, KnowledgeRelationRow.to_entity_id == KnowledgeEntityRow.id)
|
||||
.where(KnowledgeRelationRow.from_note_id == note_id)
|
||||
.order_by(KnowledgeEntityRow.name)
|
||||
)
|
||||
seen: set[str] = set()
|
||||
out: list[dict[str, Any]] = []
|
||||
for name, etype in (await session.execute(stmt)).all():
|
||||
key = (name or "").lower()
|
||||
if not name or key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
out.append({"name": name, "entity_type": etype})
|
||||
return out
|
||||
|
||||
async def prune_orphan_entities(self) -> int:
|
||||
"""Delete entity rows no longer referenced by any relation. Returns count."""
|
||||
async with self._sf() as session:
|
||||
referenced: set[str] = set()
|
||||
for col in (KnowledgeRelationRow.from_entity_id, KnowledgeRelationRow.to_entity_id):
|
||||
for (eid,) in (await session.execute(select(col).where(col.isnot(None)))).all():
|
||||
referenced.add(eid)
|
||||
all_ids = list((await session.execute(select(KnowledgeEntityRow.id))).scalars())
|
||||
orphans = [eid for eid in all_ids if eid not in referenced]
|
||||
if orphans:
|
||||
await session.execute(delete(KnowledgeEntityRow).where(KnowledgeEntityRow.id.in_(orphans)))
|
||||
await session.commit()
|
||||
return len(orphans)
|
||||
|
||||
async def list_entities(self) -> list[dict[str, Any]]:
|
||||
async with self._sf() as session:
|
||||
rows = (await session.execute(select(KnowledgeEntityRow))).scalars()
|
||||
return [{"id": r.id, "name": r.name, "entity_type": r.entity_type} for r in rows]
|
||||
|
||||
async def list_relations(self) -> list[dict[str, Any]]:
|
||||
async with self._sf() as session:
|
||||
rows = (await session.execute(select(KnowledgeRelationRow))).scalars()
|
||||
return [
|
||||
{
|
||||
"id": r.id,
|
||||
"from_note_id": r.from_note_id,
|
||||
"to_note_id": r.to_note_id,
|
||||
"from_entity_id": r.from_entity_id,
|
||||
"to_entity_id": r.to_entity_id,
|
||||
"relation_type": r.relation_type,
|
||||
"weight": r.weight,
|
||||
}
|
||||
for r in rows
|
||||
]
|
||||
|
||||
# -- sources ----------------------------------------------------------
|
||||
async def add_sources(self, note_id: str, sources: list[SourceDraft], *, created_by: str | None = None) -> None:
|
||||
if not sources:
|
||||
return
|
||||
import json
|
||||
|
||||
async with self._sf() as session:
|
||||
for src in sources:
|
||||
session.add(
|
||||
KnowledgeSourceRow(
|
||||
id=_new_id("kbsrc"),
|
||||
note_id=note_id,
|
||||
source_type=src.source_type,
|
||||
thread_id=src.thread_id,
|
||||
message_id=src.message_id,
|
||||
tool_name=src.tool_name,
|
||||
title=(src.title or "")[:512] or None,
|
||||
url=src.url,
|
||||
snippet=src.snippet,
|
||||
raw_json=json.dumps(src.raw, ensure_ascii=False) if src.raw else None,
|
||||
created_by=created_by,
|
||||
created_at=datetime.now(UTC),
|
||||
)
|
||||
)
|
||||
await session.commit()
|
||||
@@ -0,0 +1,75 @@
|
||||
"""Internal domain models for the knowledge module.
|
||||
|
||||
These are plain dataclasses passed between the extractor, the vault writer and
|
||||
the repository. The Gateway router defines its own Pydantic request/response
|
||||
models on top of the dicts the service returns.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
# Allowed note source types (see product-knowledge-base-dev.md §7.1).
|
||||
SOURCE_TYPES = ("thread", "search", "tool", "file", "manual", "report")
|
||||
# Allowed lifecycle states.
|
||||
STATUSES = ("draft", "approved", "archived")
|
||||
# obsidian-wiki wiki-capture categories.
|
||||
CATEGORIES = ("synthesis", "concepts", "references", "decision", "journal")
|
||||
|
||||
|
||||
@dataclass
|
||||
class SourceDraft:
|
||||
"""A normalized knowledge source (search result, tool call, thread, …)."""
|
||||
|
||||
source_type: str
|
||||
thread_id: str | None = None
|
||||
message_id: str | None = None
|
||||
tool_name: str | None = None
|
||||
title: str | None = None
|
||||
url: str | None = None
|
||||
snippet: str | None = None
|
||||
raw: dict | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class NoteDraft:
|
||||
"""A knowledge note ready to be persisted + written to the vault.
|
||||
|
||||
Produced by the extractor (from a thread / search / tool result) or built
|
||||
directly for a manual note. ``content_md`` here is the *body* — the vault
|
||||
writer prepends obsidian-wiki frontmatter.
|
||||
"""
|
||||
|
||||
title: str
|
||||
summary: str
|
||||
content_md: str
|
||||
category: str = "synthesis"
|
||||
source_type: str = "manual"
|
||||
source_id: str | None = None
|
||||
status: str = "approved"
|
||||
confidence: float = 0.0
|
||||
tags: list[str] = field(default_factory=list)
|
||||
key_points: list[str] = field(default_factory=list)
|
||||
actions: list[str] = field(default_factory=list)
|
||||
related: list[str] = field(default_factory=list)
|
||||
sources: list[SourceDraft] = field(default_factory=list)
|
||||
# Phase 4: extracted graph signal. entities=[{name,type}],
|
||||
# relations=[{from,to,type}] (names refer to entities or this note's title).
|
||||
entities: list[dict] = field(default_factory=list)
|
||||
relations: list[dict] = field(default_factory=list)
|
||||
# Stable identity hint used for manifest dedup (e.g. ``thread:thread_id``).
|
||||
source_key: str | None = None
|
||||
|
||||
|
||||
def normalize_status(status: str | None) -> str:
|
||||
"""Coerce an arbitrary status string to a valid lifecycle state."""
|
||||
if status in STATUSES:
|
||||
return status
|
||||
return "approved"
|
||||
|
||||
|
||||
def normalize_source_type(source_type: str | None) -> str:
|
||||
"""Coerce an arbitrary source-type string to a valid value."""
|
||||
if source_type in SOURCE_TYPES:
|
||||
return source_type
|
||||
return "manual"
|
||||
@@ -0,0 +1,160 @@
|
||||
"""Knowledge search: keyword / vector / hybrid (phase 1 + 2).
|
||||
|
||||
- ``keyword`` — title>tags>summary>body scoring over the DB mirror (always works).
|
||||
- ``vector`` — cosine over per-chunk embeddings (requires a configured
|
||||
embedding endpoint; otherwise transparently falls back to ``keyword``).
|
||||
- ``hybrid`` — min-max-normalized blend of keyword + vector.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from deerflow.knowledge.embeddings import cosine_similarity
|
||||
from deerflow.knowledge.repository import KnowledgeRepository
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from deerflow.knowledge.embeddings import EmbeddingClient
|
||||
|
||||
_TOKEN_RE = re.compile(r"[\w一-鿿]+", re.UNICODE)
|
||||
|
||||
|
||||
def _tokens(text: str) -> list[str]:
|
||||
return [t.lower() for t in _TOKEN_RE.findall(text or "")]
|
||||
|
||||
|
||||
def _snippet(content: str, terms: list[str], *, width: int = 160) -> str:
|
||||
body = re.sub(r"\s+", " ", content or "").strip()
|
||||
low = body.lower()
|
||||
for term in terms:
|
||||
idx = low.find(term)
|
||||
if idx >= 0:
|
||||
start = max(0, idx - width // 3)
|
||||
end = min(len(body), start + width)
|
||||
prefix = "…" if start > 0 else ""
|
||||
suffix = "…" if end < len(body) else ""
|
||||
return f"{prefix}{body[start:end]}{suffix}"
|
||||
return body[:width] + ("…" if len(body) > width else "")
|
||||
|
||||
|
||||
async def _keyword_scores(repo: KnowledgeRepository, query: str, *, candidate_cap: int) -> dict[str, tuple[float, dict[str, Any]]]:
|
||||
terms = _tokens(query)
|
||||
if not terms:
|
||||
return {}
|
||||
candidates, _total = await repo.list_notes(limit=candidate_cap, offset=0)
|
||||
scored: dict[str, tuple[float, dict[str, Any]]] = {}
|
||||
for note in candidates:
|
||||
title = (note.get("title") or "").lower()
|
||||
summary = (note.get("summary") or "").lower()
|
||||
content = (note.get("content_md") or "").lower()
|
||||
tags = " ".join(note.get("tags") or []).lower()
|
||||
score = 0.0
|
||||
for term in terms:
|
||||
if term in title:
|
||||
score += 3.0
|
||||
if term in tags:
|
||||
score += 2.0
|
||||
if term in summary:
|
||||
score += 2.0
|
||||
score += min(content.count(term), 5) * 0.5
|
||||
if score > 0:
|
||||
scored[note["id"]] = (score, note)
|
||||
return scored
|
||||
|
||||
|
||||
async def _vector_scores(
|
||||
repo: KnowledgeRepository,
|
||||
query: str,
|
||||
embedding_client: EmbeddingClient,
|
||||
) -> dict[str, tuple[float, dict[str, Any]]]:
|
||||
qvec = await embedding_client.embed_query(query)
|
||||
if not qvec:
|
||||
return {}
|
||||
embeddings = await repo.all_embeddings()
|
||||
best: dict[str, tuple[float, str, str]] = {} # note_id -> (score, title, snippet)
|
||||
for emb in embeddings:
|
||||
sim = cosine_similarity(qvec, emb.get("vector") or [])
|
||||
if sim <= 0:
|
||||
continue
|
||||
nid = emb["note_id"]
|
||||
if nid not in best or sim > best[nid][0]:
|
||||
best[nid] = (sim, emb.get("title") or "", emb.get("content") or "")
|
||||
return {nid: (score, {"id": nid, "title": title, "_chunk": chunk}) for nid, (score, title, chunk) in best.items()}
|
||||
|
||||
|
||||
def _normalize(scores: dict[str, float]) -> dict[str, float]:
|
||||
if not scores:
|
||||
return {}
|
||||
mx = max(scores.values())
|
||||
if mx <= 0:
|
||||
return {k: 0.0 for k in scores}
|
||||
return {k: v / mx for k, v in scores.items()}
|
||||
|
||||
|
||||
async def search_notes(
|
||||
repo: KnowledgeRepository,
|
||||
query: str,
|
||||
*,
|
||||
mode: str = "keyword",
|
||||
limit: int = 8,
|
||||
candidate_cap: int = 500,
|
||||
embedding_client: EmbeddingClient | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Score notes against ``query`` and return the top ``limit`` matches."""
|
||||
terms = _tokens(query)
|
||||
if not terms:
|
||||
return []
|
||||
|
||||
want_vector = mode in ("vector", "hybrid") and embedding_client is not None and embedding_client.is_enabled()
|
||||
keyword = await _keyword_scores(repo, query, candidate_cap=candidate_cap) if mode != "vector" or not want_vector else {}
|
||||
vector = await _vector_scores(repo, query, embedding_client) if want_vector else {}
|
||||
|
||||
# Pure vector (when available); else keyword; hybrid blends both.
|
||||
if mode == "vector" and want_vector:
|
||||
combined = {nid: (sc, meta) for nid, (sc, meta) in vector.items()}
|
||||
ranked = sorted(combined.items(), key=lambda kv: kv[1][0], reverse=True)[:limit]
|
||||
return await _materialize(repo, ranked, terms)
|
||||
|
||||
if mode == "hybrid" and want_vector:
|
||||
kw_norm = _normalize({k: v[0] for k, v in keyword.items()})
|
||||
vec_norm = _normalize({k: v[0] for k, v in vector.items()})
|
||||
ids = set(kw_norm) | set(vec_norm)
|
||||
blended: dict[str, tuple[float, dict[str, Any]]] = {}
|
||||
for nid in ids:
|
||||
score = 0.5 * kw_norm.get(nid, 0.0) + 0.5 * vec_norm.get(nid, 0.0)
|
||||
meta = keyword.get(nid, (0, None))[1] or vector.get(nid, (0, {}))[1]
|
||||
blended[nid] = (score, meta)
|
||||
ranked = sorted(blended.items(), key=lambda kv: kv[1][0], reverse=True)[:limit]
|
||||
return await _materialize(repo, ranked, terms)
|
||||
|
||||
# keyword (default, and fallback when vector unavailable)
|
||||
ranked = sorted(keyword.items(), key=lambda kv: kv[1][0], reverse=True)[:limit]
|
||||
if not ranked and vector: # mode=vector requested but keyword empty: use vector
|
||||
ranked = sorted(vector.items(), key=lambda kv: kv[1][0], reverse=True)[:limit]
|
||||
return await _materialize(repo, ranked, terms)
|
||||
|
||||
|
||||
async def _materialize(repo: KnowledgeRepository, ranked: list[tuple[str, tuple[float, dict[str, Any]]]], terms: list[str]) -> list[dict[str, Any]]:
|
||||
max_score = ranked[0][1][0] if ranked else 1.0
|
||||
results: list[dict[str, Any]] = []
|
||||
for nid, (score, meta) in ranked:
|
||||
note = meta if (meta and meta.get("content_md")) else await repo.get_note(nid, with_sources=False)
|
||||
if note is None:
|
||||
continue
|
||||
snippet_src = note.get("content_md") or meta.get("_chunk") or note.get("summary") or ""
|
||||
results.append(
|
||||
{
|
||||
"note_id": nid,
|
||||
"chunk_id": None,
|
||||
"title": note.get("title") or meta.get("title"),
|
||||
"summary": note.get("summary"),
|
||||
"snippet": _snippet(snippet_src, terms),
|
||||
"score": round(score / max_score, 4) if max_score else 0.0,
|
||||
"tags": note.get("tags") or [],
|
||||
"source_type": note.get("source_type"),
|
||||
"updated_at": note.get("updated_at"),
|
||||
"sources": [],
|
||||
}
|
||||
)
|
||||
return results
|
||||
@@ -0,0 +1,54 @@
|
||||
"""Sensitive-information redaction for knowledge content (phase 3).
|
||||
|
||||
Best-effort masking of secrets / PII before a note is persisted: API keys,
|
||||
bearer tokens, generic ``key=secret`` assignments, Chinese mobile numbers,
|
||||
ID-card numbers and email addresses. Conservative by design — it favours not
|
||||
mangling legitimate prose over catching every possible secret.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
_MASK = "[REDACTED]"
|
||||
|
||||
# Order matters: more specific patterns first.
|
||||
_PATTERNS: list[tuple[str, re.Pattern[str]]] = [
|
||||
("openai_key", re.compile(r"\bsk-[A-Za-z0-9]{20,}\b")),
|
||||
("aws_key", re.compile(r"\b(?:AKIA|ASIA)[A-Z0-9]{16}\b")),
|
||||
("github_token", re.compile(r"\bgh[pousr]_[A-Za-z0-9]{20,}\b")),
|
||||
("bearer", re.compile(r"(?i)\bBearer\s+[A-Za-z0-9._\-]{16,}")),
|
||||
# key/secret/password/token assignment: keep the field name, mask the value.
|
||||
("assignment", re.compile(r"(?i)\b(api[_-]?key|secret|password|passwd|token|access[_-]?token)\b\s*[:=]\s*[\"']?[^\s\"']{6,}[\"']?")),
|
||||
("cn_id_card", re.compile(r"\b\d{17}[\dXx]\b")),
|
||||
("cn_mobile", re.compile(r"(?<!\d)1[3-9]\d{9}(?!\d)")),
|
||||
("email", re.compile(r"\b[A-Za-z0-9._%+\-]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,}\b")),
|
||||
]
|
||||
|
||||
|
||||
def redact(text: str | None) -> tuple[str, list[str]]:
|
||||
"""Return ``(redacted_text, hit_kinds)``.
|
||||
|
||||
``hit_kinds`` lists which detectors fired (for logging/audit). When ``text``
|
||||
is falsy it is returned unchanged with no hits.
|
||||
"""
|
||||
if not text:
|
||||
return text or "", []
|
||||
hits: list[str] = []
|
||||
out = text
|
||||
for kind, pattern in _PATTERNS:
|
||||
def _sub(m: re.Match[str], _kind: str = kind) -> str:
|
||||
# For assignments keep the field name + delimiter, mask only the value.
|
||||
if _kind == "assignment":
|
||||
head = re.split(r"[:=]", m.group(0), maxsplit=1)[0]
|
||||
sep = ":" if ":" in m.group(0) else "="
|
||||
return f"{head}{sep} {_MASK}"
|
||||
if _kind == "bearer":
|
||||
return f"Bearer {_MASK}"
|
||||
return _MASK
|
||||
|
||||
new_out, n = pattern.subn(_sub, out)
|
||||
if n:
|
||||
hits.append(kind)
|
||||
out = new_out
|
||||
return out, hits
|
||||
@@ -0,0 +1,689 @@
|
||||
"""KnowledgeService — high-level orchestration for the knowledge base.
|
||||
|
||||
Ties together the repository (DB mirror), the :class:`ObsidianWikiAdapter`
|
||||
(Markdown vault + index/log/hot/manifest + graph export) and the extractor
|
||||
(thread/search → :class:`NoteDraft`). The Gateway router only talks to this
|
||||
service; everything obsidian-wiki specific stays inside the ``knowledge``
|
||||
module.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from deerflow.config.knowledge_config import KnowledgeConfig
|
||||
from deerflow.config.runtime_paths import runtime_home
|
||||
from deerflow.knowledge import extractor, llm_extractor
|
||||
from deerflow.knowledge.chunking import chunk_markdown
|
||||
from deerflow.knowledge.embeddings import EmbeddingClient
|
||||
from deerflow.knowledge.markdown_vault import MarkdownVault, VaultPathError, extract_wikilinks, normalize_wikilink_target
|
||||
from deerflow.knowledge.obsidian_adapter import ObsidianWikiAdapter
|
||||
from deerflow.knowledge.repository import KnowledgeRepository
|
||||
from deerflow.knowledge.schemas import NoteDraft, SourceDraft, normalize_source_type, normalize_status
|
||||
from deerflow.knowledge.search import search_notes
|
||||
from deerflow.knowledge.sources import from_reference_batches, from_search_results
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class EmptyThreadError(ValueError):
|
||||
"""Raised when a thread has no sediment-able content."""
|
||||
|
||||
|
||||
def _resolve_vault_dir(vault_dir: str) -> Path:
|
||||
path = Path(vault_dir)
|
||||
if path.is_absolute():
|
||||
return path.resolve()
|
||||
return (runtime_home() / vault_dir).resolve()
|
||||
|
||||
|
||||
def _inject_related_notes(body: str, links: list[str]) -> str:
|
||||
"""Insert real related-note ``[[wikilinks]]`` into the Related block.
|
||||
|
||||
Notes carry only content-specific related notes now (the structural meta-links
|
||||
were dropped), so this is the sole writer of ``## Related`` links. When the
|
||||
body has no ``## Related`` section yet, one is appended.
|
||||
"""
|
||||
if not links:
|
||||
return body
|
||||
bullets = "\n".join(f"- {link}" for link in links)
|
||||
marker = "## Related\n"
|
||||
idx = body.find(marker)
|
||||
if idx == -1:
|
||||
return body.rstrip() + "\n\n## Related\n\n" + bullets + "\n"
|
||||
insert_at = idx + len(marker)
|
||||
return body[:insert_at] + "\n" + bullets + "\n" + body[insert_at:]
|
||||
|
||||
|
||||
class KnowledgeService:
|
||||
"""Application-facing knowledge base API."""
|
||||
|
||||
def __init__(self, repo: KnowledgeRepository, config: KnowledgeConfig) -> None:
|
||||
self.repo = repo
|
||||
self.config = config
|
||||
self.project_name = config.project_name
|
||||
vault_root = _resolve_vault_dir(config.vault_dir)
|
||||
self.adapter = ObsidianWikiAdapter(
|
||||
vault_root,
|
||||
project_name=config.project_name,
|
||||
export_html=config.export.graph_html_enabled,
|
||||
export_json=config.export.graph_json_enabled,
|
||||
)
|
||||
self.embedding_client = EmbeddingClient(config.embedding)
|
||||
try:
|
||||
self.adapter.ensure_ready()
|
||||
except Exception: # pragma: no cover - vault init best-effort
|
||||
logger.exception("Failed to initialize knowledge vault at %s", vault_root)
|
||||
|
||||
# -- internal helpers -------------------------------------------------
|
||||
def _redact_draft(self, draft: NoteDraft) -> NoteDraft:
|
||||
"""Mask secrets/PII in a draft before persistence (phase 3)."""
|
||||
if not self.config.sensitive_redaction_enabled:
|
||||
return draft
|
||||
from deerflow.knowledge.sensitive import redact
|
||||
|
||||
draft.content_md, body_hits = redact(draft.content_md)
|
||||
draft.summary, _ = redact(draft.summary)
|
||||
for src in draft.sources:
|
||||
if src.snippet:
|
||||
src.snippet, _ = redact(src.snippet)
|
||||
if body_hits:
|
||||
logger.info("Knowledge redaction applied to note '%s': %s", draft.title, ",".join(sorted(set(body_hits))))
|
||||
return draft
|
||||
|
||||
async def _find_related_notes(self, draft: NoteDraft) -> list[dict[str, Any]]:
|
||||
"""Find existing notes to cross-link under ## Related — by **shared tags**.
|
||||
|
||||
Matching is strictly tag-based: a candidate must share at least one of the
|
||||
draft's *distinctive* tags (the per-note project / ``knowledge-base``
|
||||
defaults are dropped, else everything matches everything). Free-text
|
||||
title/summary matching was removed because CJK tokenization made generic
|
||||
words (助手 / 知识图谱 / 页面) link unrelated notes. No distinctive tags →
|
||||
no related notes (better empty than noisy). Best-effort.
|
||||
"""
|
||||
if not self.config.related_notes_enabled:
|
||||
return []
|
||||
default_tags = {t.lower() for t in (self.project_name or "", "knowledge-base", "zncm") if t}
|
||||
distinctive = [t for t in (draft.tags or []) if t and t.lower() not in default_tags]
|
||||
if not distinctive:
|
||||
return []
|
||||
own_title = (draft.title or "").strip().lower()
|
||||
# Rank candidates by how many distinctive tags they share with the draft.
|
||||
shared: dict[str, dict[str, Any]] = {}
|
||||
for tag in distinctive:
|
||||
try:
|
||||
items, _total = await self.repo.list_notes(tag=tag, limit=50, offset=0)
|
||||
except Exception:
|
||||
logger.exception("Related-notes tag lookup failed for tag '%s'", tag)
|
||||
continue
|
||||
for it in items:
|
||||
nid = it.get("id")
|
||||
title = (it.get("title") or "").strip()
|
||||
if not nid or not title or title.lower() == own_title:
|
||||
continue
|
||||
entry = shared.setdefault(nid, {"id": nid, "title": title, "count": 0})
|
||||
entry["count"] += 1
|
||||
ranked = sorted(shared.values(), key=lambda e: e["count"], reverse=True)
|
||||
limit = max(1, self.config.related_notes_limit)
|
||||
return [{"id": e["id"], "title": e["title"]} for e in ranked[:limit]]
|
||||
|
||||
async def _persist_draft(self, draft: NoteDraft, *, created_by: str | None) -> dict[str, Any]:
|
||||
"""Create a DB row, write the vault file, attach sources, return note."""
|
||||
draft = self._redact_draft(draft)
|
||||
related_notes = await self._find_related_notes(draft)
|
||||
if related_notes:
|
||||
real_links = [f"[[{n['title']}]]" for n in related_notes]
|
||||
draft.content_md = _inject_related_notes(draft.content_md, real_links)
|
||||
draft.related = list(dict.fromkeys([*real_links, *draft.related]))
|
||||
note = await self.repo.create_note(
|
||||
title=draft.title,
|
||||
summary=draft.summary,
|
||||
content_md=draft.content_md,
|
||||
source_type=draft.source_type,
|
||||
source_id=draft.source_id,
|
||||
status=draft.status,
|
||||
confidence=draft.confidence,
|
||||
tags=draft.tags,
|
||||
created_by=created_by,
|
||||
)
|
||||
note_id = note["id"]
|
||||
try:
|
||||
rel = self.adapter.write_note(note_id, draft, created=note["created_at"], updated=note["updated_at"], status=draft.status)
|
||||
await self.repo.set_vault_path(note_id, rel)
|
||||
note["vault_path"] = rel
|
||||
except Exception:
|
||||
logger.exception("Failed to write knowledge note %s to vault", note_id)
|
||||
if draft.sources:
|
||||
await self.repo.add_sources(note_id, draft.sources, created_by=created_by)
|
||||
await self._reindex_embeddings(note_id, draft.content_md)
|
||||
await self._store_entities(note_id, draft)
|
||||
# Record note→note "related" edges (after _store_entities, which clears
|
||||
# this note's relations) so the graph reflects the cross-links too.
|
||||
for rel_note in related_notes:
|
||||
try:
|
||||
await self.repo.add_relation(from_note_id=note_id, to_note_id=rel_note["id"], relation_type="related")
|
||||
except Exception:
|
||||
logger.exception("Failed to link related note %s -> %s", note_id, rel_note["id"])
|
||||
return await self.repo.get_note(note_id) or note
|
||||
|
||||
async def _store_entities(self, note_id: str, draft: NoteDraft) -> None:
|
||||
"""Persist extracted entities/relations and link them to the note (phase 4)."""
|
||||
if not self.config.entity_extraction_enabled or not draft.entities:
|
||||
return
|
||||
try:
|
||||
await self.repo.clear_note_relations(note_id)
|
||||
name_to_id: dict[str, str] = {}
|
||||
for ent in draft.entities:
|
||||
name = str(ent.get("name") or "").strip()
|
||||
if not name:
|
||||
continue
|
||||
row = await self.repo.get_or_create_entity(name, ent.get("type"))
|
||||
name_to_id[name.lower()] = row["id"]
|
||||
# note -> entity "mentions"
|
||||
await self.repo.add_relation(from_note_id=note_id, to_entity_id=row["id"], relation_type="mentions")
|
||||
for rel in draft.relations:
|
||||
a = name_to_id.get(str(rel.get("from") or "").strip().lower())
|
||||
b = name_to_id.get(str(rel.get("to") or "").strip().lower())
|
||||
if a and b and a != b:
|
||||
await self.repo.add_relation(from_entity_id=a, to_entity_id=b, relation_type=str(rel.get("type") or "relates_to"))
|
||||
except Exception:
|
||||
logger.exception("Failed to store entities/relations for note %s", note_id)
|
||||
|
||||
async def _reindex_embeddings(self, note_id: str, content_md: str) -> None:
|
||||
"""Best-effort: (re)build the per-chunk embedding index for a note."""
|
||||
if not self.embedding_client.is_enabled():
|
||||
return
|
||||
try:
|
||||
chunks = chunk_markdown(content_md)
|
||||
if not chunks:
|
||||
await self.repo.delete_embeddings(note_id)
|
||||
return
|
||||
vectors = await self.embedding_client.embed_texts(chunks)
|
||||
if not vectors or len(vectors) != len(chunks):
|
||||
return
|
||||
await self.repo.replace_embeddings(note_id, [(i, c, v) for i, (c, v) in enumerate(zip(chunks, vectors, strict=False))])
|
||||
except Exception:
|
||||
logger.exception("Failed to (re)index embeddings for note %s", note_id)
|
||||
|
||||
# -- capture ----------------------------------------------------------
|
||||
async def capture_thread(
|
||||
self,
|
||||
*,
|
||||
thread_id: str,
|
||||
messages: list[dict[str, Any]],
|
||||
thread_title: str | None = None,
|
||||
mode: str = "summary",
|
||||
title: str | None = None,
|
||||
status: str = "approved",
|
||||
tags: list[str] | None = None,
|
||||
include_sources: bool = True,
|
||||
reference_batches: list[dict[str, Any]] | None = None,
|
||||
created_by: str | None = None,
|
||||
skip_existing: bool = False,
|
||||
model_name: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Sediment a thread into a knowledge note.
|
||||
|
||||
``model_name`` optionally overrides the configured extraction model for
|
||||
this one capture (falls back to ``config.extract_model_name``).
|
||||
|
||||
Returns ``{"note": <note>, "created": bool}``. Raises
|
||||
:class:`EmptyThreadError` when there is nothing worth keeping.
|
||||
"""
|
||||
# Always-on entry log: proves a capture was triggered for this thread,
|
||||
# regardless of how it ends (success / skip / fallback). Lets you tell
|
||||
# "capture never ran" apart from "logs not visible" via one grep.
|
||||
logger.info(
|
||||
"[knowledge] capturing thread %s (msgs=%d, llm_extract=%s, skip_existing=%s)",
|
||||
thread_id,
|
||||
len(messages or []),
|
||||
self.config.llm_extract_enabled,
|
||||
skip_existing,
|
||||
)
|
||||
if skip_existing:
|
||||
existing = await self.repo.find_by_source("thread", thread_id)
|
||||
if existing is None and self.adapter.manifest.has_source(f"thread:{thread_id}"):
|
||||
existing = await self.repo.find_by_source("thread", thread_id)
|
||||
if existing:
|
||||
logger.info("[knowledge] thread %s already captured (note %s); skipping", thread_id, existing.get("id"))
|
||||
return {"note": existing, "created": False}
|
||||
|
||||
sources = []
|
||||
if include_sources and reference_batches:
|
||||
sources = from_reference_batches(reference_batches, thread_id=thread_id, max_items=self.config.max_source_items)
|
||||
|
||||
drafts: list[NoteDraft] = []
|
||||
# Prefer LLM distillation (wiki-capture style) so notes are declarative
|
||||
# knowledge, not raw chat. A single conversation may split into several
|
||||
# cross-linked notes. Falls back to the rule-based extractor (always one
|
||||
# note) when the LLM is disabled/unavailable or returns nothing usable.
|
||||
llm_attempted = self.config.llm_extract_enabled
|
||||
llm_fallback = False
|
||||
if llm_attempted:
|
||||
try:
|
||||
drafts = await llm_extractor.extract_thread_llm_multi(
|
||||
messages,
|
||||
thread_id=thread_id,
|
||||
thread_title=thread_title,
|
||||
mode=mode,
|
||||
title=title,
|
||||
tags=tags,
|
||||
project_name=self.project_name,
|
||||
sources=sources,
|
||||
max_messages=self.config.max_thread_messages,
|
||||
model_name=model_name or self.config.extract_model_name,
|
||||
max_input_chars=self.config.extract_max_input_chars,
|
||||
allow_multi=self.config.multi_note_enabled,
|
||||
)
|
||||
except llm_extractor.LLMRejectedIngest as exc:
|
||||
# The model judged this conversation worthless (greeting / meta
|
||||
# chatter). Skip ingestion entirely — do NOT dump the raw chat.
|
||||
raise EmptyThreadError(f"Thread {thread_id} judged not worth ingesting: {exc}") from exc
|
||||
if not drafts:
|
||||
# The LLM path was supposed to run but produced nothing usable
|
||||
# (API error / non-JSON). We will fall back to the rule-based
|
||||
# extractor, which yields a low-quality verbatim dump with no
|
||||
# entities/tags — flag it so it lands as a draft.
|
||||
llm_fallback = True
|
||||
if not drafts:
|
||||
single = extractor.extract_thread(
|
||||
messages,
|
||||
thread_id=thread_id,
|
||||
thread_title=thread_title,
|
||||
mode=mode,
|
||||
title=title,
|
||||
tags=tags,
|
||||
project_name=self.project_name,
|
||||
sources=sources,
|
||||
max_messages=self.config.max_thread_messages,
|
||||
)
|
||||
drafts = [single] if single else []
|
||||
if not drafts:
|
||||
logger.info("[knowledge] thread %s has no sediment-able content; skipping", thread_id)
|
||||
raise EmptyThreadError(f"Thread {thread_id} has no sediment-able content")
|
||||
|
||||
# When the LLM was enabled but silently fell back to rule-based extraction,
|
||||
# the note is a verbatim dump (no distillation / entities / tags). Land it
|
||||
# as a *draft* for review instead of silently approving it, and log loudly
|
||||
# so the extraction failure is diagnosable (see also the WARNING/EXCEPTION
|
||||
# logged inside llm_extractor.extract_thread_llm_multi).
|
||||
effective_status = status
|
||||
if llm_fallback:
|
||||
logger.warning(
|
||||
"[knowledge] thread %s — capture fell back to rule-based extraction "
|
||||
"(LLM distillation produced nothing usable); landing as DRAFT. "
|
||||
"Check extraction model '%s' / earlier [knowledge] logs for the cause.",
|
||||
thread_id,
|
||||
self.config.extract_model_name or "models[0]",
|
||||
)
|
||||
if normalize_status(status) == "approved":
|
||||
effective_status = "draft"
|
||||
|
||||
normalized = normalize_status(effective_status)
|
||||
for draft in drafts:
|
||||
draft.status = normalized
|
||||
# Cross-link sibling notes from the same conversation by title, so the
|
||||
# split pages reference each other (resolved once all are persisted).
|
||||
if len(drafts) > 1:
|
||||
for i, draft in enumerate(drafts):
|
||||
siblings = [f"[[{o.title}]]" for j, o in enumerate(drafts) if j != i and o.title]
|
||||
if siblings:
|
||||
draft.content_md = _inject_related_notes(draft.content_md, siblings)
|
||||
draft.related = list(dict.fromkeys([*siblings, *draft.related]))
|
||||
|
||||
notes = [await self._persist_draft(draft, created_by=created_by) for draft in drafts]
|
||||
logger.info(
|
||||
"[knowledge] thread %s captured → %d note(s) [%s], status=%s, llm_fallback=%s",
|
||||
thread_id,
|
||||
len(notes),
|
||||
", ".join(n.get("title", "") for n in notes),
|
||||
normalized,
|
||||
llm_fallback,
|
||||
)
|
||||
return {"note": notes[0], "created": True, "notes": notes, "llm_fallback": llm_fallback}
|
||||
|
||||
async def capture_search(
|
||||
self,
|
||||
*,
|
||||
query: str,
|
||||
results: list[dict[str, Any]] | None,
|
||||
thread_id: str | None = None,
|
||||
message_id: str | None = None,
|
||||
title: str | None = None,
|
||||
tags: list[str] | None = None,
|
||||
status: str = "approved",
|
||||
created_by: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Sediment search/tool results into a reference note."""
|
||||
sources = from_search_results(query, results, thread_id=thread_id, message_id=message_id, max_items=self.config.max_source_items)
|
||||
draft = extractor.extract_search(query, sources, thread_id=thread_id, project_name=self.project_name, title=title, tags=tags)
|
||||
draft.status = normalize_status(status)
|
||||
return await self._persist_draft(draft, created_by=created_by)
|
||||
|
||||
async def create_manual_note(
|
||||
self,
|
||||
*,
|
||||
title: str,
|
||||
content_md: str,
|
||||
tags: list[str] | None = None,
|
||||
status: str = "approved",
|
||||
summary: str | None = None,
|
||||
created_by: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Create a note directly from product-edited Markdown."""
|
||||
# Strip any frontmatter the user may have pasted; vault writer adds its own.
|
||||
_, body = MarkdownVault.parse(content_md)
|
||||
body = body or content_md
|
||||
base_tags = list(tags or [])
|
||||
for default in (self.project_name, "knowledge-base"):
|
||||
if default and default not in base_tags:
|
||||
base_tags.append(default)
|
||||
draft = NoteDraft(
|
||||
title=title.strip() or "知识笔记",
|
||||
summary=(summary or body.strip().splitlines()[0] if body.strip() else "")[:280],
|
||||
content_md=body,
|
||||
category="synthesis",
|
||||
source_type="manual",
|
||||
source_id=None,
|
||||
status=normalize_status(status),
|
||||
tags=base_tags,
|
||||
confidence=0.5,
|
||||
related=extractor.related_links(self.project_name),
|
||||
)
|
||||
return await self._persist_draft(draft, created_by=created_by)
|
||||
|
||||
# -- CRUD -------------------------------------------------------------
|
||||
async def list_notes(
|
||||
self,
|
||||
*,
|
||||
keyword: str | None = None,
|
||||
tag: str | None = None,
|
||||
source_type: str | None = None,
|
||||
status: str | None = None,
|
||||
limit: int = 20,
|
||||
offset: int = 0,
|
||||
) -> dict[str, Any]:
|
||||
items, total = await self.repo.list_notes(
|
||||
keyword=keyword,
|
||||
tag=tag,
|
||||
source_type=normalize_source_type(source_type) if source_type else None,
|
||||
status=status,
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
)
|
||||
return {"items": items, "total": total}
|
||||
|
||||
async def get_note(self, note_id: str) -> dict[str, Any] | None:
|
||||
note = await self.repo.get_note(note_id)
|
||||
if note is not None:
|
||||
note["entities"] = await self.repo.list_note_entities(note_id)
|
||||
return note
|
||||
|
||||
async def update_note(
|
||||
self,
|
||||
note_id: str,
|
||||
*,
|
||||
title: str | None = None,
|
||||
content_md: str | None = None,
|
||||
tags: list[str] | None = None,
|
||||
status: str | None = None,
|
||||
summary: str | None = None,
|
||||
updated_by: str | None = None,
|
||||
) -> dict[str, Any] | None:
|
||||
current = await self.repo.get_note(note_id, with_sources=False)
|
||||
if current is None:
|
||||
return None
|
||||
# Snapshot the pre-edit state for version history (phase 3).
|
||||
try:
|
||||
await self.repo.add_version(current, edited_by=updated_by)
|
||||
except Exception:
|
||||
logger.exception("Failed to snapshot version for note %s", note_id)
|
||||
body = content_md
|
||||
if body is not None:
|
||||
_, parsed = MarkdownVault.parse(body)
|
||||
body = parsed or body
|
||||
note = await self.repo.update_note(
|
||||
note_id,
|
||||
title=title,
|
||||
summary=summary,
|
||||
content_md=body,
|
||||
status=normalize_status(status) if status else None,
|
||||
tags=tags,
|
||||
updated_by=updated_by,
|
||||
)
|
||||
if note is None:
|
||||
return None
|
||||
try:
|
||||
rel = self.adapter.update_note_file(
|
||||
note_id,
|
||||
current.get("vault_path"),
|
||||
title=note["title"],
|
||||
content_md=note["content_md"],
|
||||
tags=note.get("tags") or [],
|
||||
status=note["status"],
|
||||
summary=note.get("summary") or "",
|
||||
category="synthesis",
|
||||
created=note["created_at"],
|
||||
updated=note["updated_at"],
|
||||
)
|
||||
if rel and rel != current.get("vault_path"):
|
||||
await self.repo.set_vault_path(note_id, rel)
|
||||
except Exception:
|
||||
logger.exception("Failed to update knowledge note %s in vault", note_id)
|
||||
if note.get("content_md") is not None:
|
||||
await self._reindex_embeddings(note_id, note["content_md"])
|
||||
return await self.repo.get_note(note_id)
|
||||
|
||||
async def list_versions(self, note_id: str) -> list[dict[str, Any]]:
|
||||
return await self.repo.list_versions(note_id)
|
||||
|
||||
async def delete_note(self, note_id: str, *, updated_by: str | None = None, hard: bool = False) -> bool:
|
||||
"""Soft delete (archive) by default; hard delete removes the vault file."""
|
||||
current = await self.repo.get_note(note_id, with_sources=False)
|
||||
if current is None:
|
||||
return False
|
||||
note = await self.repo.set_status(note_id, "archived", updated_by=updated_by)
|
||||
try:
|
||||
self.adapter.archive_note(note_id, current.get("vault_path"), updated=(note or current)["updated_at"])
|
||||
if hard and current.get("vault_path"):
|
||||
self.adapter.vault.delete_note(current["vault_path"])
|
||||
self.adapter.manifest.remove_page(note_id)
|
||||
await self.repo.delete_embeddings(note_id)
|
||||
except Exception:
|
||||
logger.exception("Failed to archive knowledge note %s in vault", note_id)
|
||||
# Drop this note's entity mentions + any now-orphaned entities so the graph
|
||||
# follows the deletion (no lingering nodes from removed notes).
|
||||
try:
|
||||
await self.repo.clear_note_relations(note_id)
|
||||
pruned = await self.repo.prune_orphan_entities()
|
||||
if pruned:
|
||||
logger.info("[knowledge] deleted note %s → pruned %d orphan entities", note_id, pruned)
|
||||
except Exception:
|
||||
logger.exception("Failed to prune relations/entities for deleted note %s", note_id)
|
||||
return True
|
||||
|
||||
# -- wikilinks --------------------------------------------------------
|
||||
async def resolve_wikilink(self, target: str) -> dict[str, Any]:
|
||||
"""Resolve a single ``[[wikilink]]`` target to a navigable destination.
|
||||
|
||||
Returns one of:
|
||||
- ``{"kind": "note", "note_id", "title", "exists": True}`` — a knowledge note
|
||||
- ``{"kind": "vault", "vault_path", "title", "exists": True}`` — a vault meta/hub page
|
||||
- ``{"kind": "missing", "exists": False}`` — nothing matches
|
||||
|
||||
Resolution order: note id → note vault_path → note title → vault file.
|
||||
"""
|
||||
norm = normalize_wikilink_target(target)
|
||||
base = {"target": target, "normalized": norm}
|
||||
if not norm:
|
||||
return {**base, "kind": "missing", "exists": False}
|
||||
|
||||
# 1) Direct note id (e.g. "kb_abc...").
|
||||
if norm.startswith("kb_"):
|
||||
note = await self.repo.get_note(norm, with_sources=False)
|
||||
if note and note.get("status") != "archived":
|
||||
return {**base, "kind": "note", "note_id": note["id"], "title": note["title"], "exists": True}
|
||||
|
||||
# 2) Note by vault path (links from index.md / hot.md / structural hub).
|
||||
note = await self.repo.find_by_vault_path(norm)
|
||||
if note:
|
||||
return {**base, "kind": "note", "note_id": note["id"], "title": note["title"], "exists": True}
|
||||
|
||||
# 3) Note by title (the real related-note cross-links use [[Title]]).
|
||||
note = await self.repo.find_by_title(norm) or await self.repo.find_by_title(norm.split("/")[-1])
|
||||
if note:
|
||||
return {**base, "kind": "note", "note_id": note["id"], "title": note["title"], "exists": True}
|
||||
|
||||
# 4) A vault meta/hub page that has no DB note (index, _meta/taxonomy, hub…).
|
||||
page = self.read_vault_page(norm)
|
||||
if page is not None:
|
||||
return {**base, "kind": "vault", "vault_path": page["path"], "title": page["title"], "exists": True}
|
||||
|
||||
return {**base, "kind": "missing", "exists": False}
|
||||
|
||||
async def resolve_wikilinks(self, targets: list[str]) -> dict[str, dict[str, Any]]:
|
||||
"""Resolve many targets at once; returns ``{original_target: resolution}``."""
|
||||
out: dict[str, dict[str, Any]] = {}
|
||||
for target in dict.fromkeys(t for t in targets if t and t.strip()):
|
||||
out[target] = await self.resolve_wikilink(target)
|
||||
return out
|
||||
|
||||
async def resolve_note_wikilinks(self, note_id: str) -> dict[str, dict[str, Any]] | None:
|
||||
"""Extract every ``[[wikilink]]`` in a note's body and resolve them.
|
||||
|
||||
Returns ``{target: resolution}`` or ``None`` when the note is missing.
|
||||
Numeric ``[[123]]`` citation badges are skipped (handled by the renderer).
|
||||
"""
|
||||
note = await self.repo.get_note(note_id, with_sources=False)
|
||||
if note is None:
|
||||
return None
|
||||
targets = [t for t in extract_wikilinks(note.get("content_md") or "") if not t.isdigit()]
|
||||
return await self.resolve_wikilinks(targets)
|
||||
|
||||
def read_vault_page(self, path: str) -> dict[str, Any] | None:
|
||||
"""Read a raw vault Markdown page (meta/hub pages without a DB note).
|
||||
|
||||
Returns ``{"path", "title", "content_md"}`` or ``None`` if the page does
|
||||
not exist or the path escapes the vault root.
|
||||
"""
|
||||
norm = normalize_wikilink_target(path)
|
||||
if not norm:
|
||||
return None
|
||||
rel = f"{norm}.md"
|
||||
try:
|
||||
parsed = self.adapter.vault.read_note(rel)
|
||||
except VaultPathError:
|
||||
return None
|
||||
if parsed is None:
|
||||
return None
|
||||
fm, body = parsed
|
||||
title = str(fm.get("title") or "").strip() or norm.split("/")[-1]
|
||||
return {"path": norm, "title": title, "content_md": body}
|
||||
|
||||
# -- search & graph ---------------------------------------------------
|
||||
async def search(self, query: str, *, mode: str = "keyword", limit: int = 8) -> list[dict[str, Any]]:
|
||||
return await search_notes(self.repo, query, mode=mode, limit=limit, embedding_client=self.embedding_client)
|
||||
|
||||
async def build_rag_context(self, query: str, *, limit: int = 5) -> dict[str, Any]:
|
||||
"""Retrieve top knowledge for ``query`` and format an injectable block.
|
||||
|
||||
Returns ``{"text": <system context or "">, "items": [...]}``. Uses hybrid
|
||||
search when embeddings are available, else keyword (wiki-query style).
|
||||
"""
|
||||
mode = "hybrid" if self.embedding_client.is_enabled() else "keyword"
|
||||
items = await search_notes(self.repo, query, mode=mode, limit=limit, embedding_client=self.embedding_client)
|
||||
if not items:
|
||||
return {"text": "", "items": []}
|
||||
lines = [
|
||||
"以下是产品知识库中与本问题相关的资料。回答时优先参考;若使用其中信息,请在回答中标注来源(note_id)。",
|
||||
"",
|
||||
]
|
||||
for i, it in enumerate(items, 1):
|
||||
lines.append(f"[知识 {i}] {it.get('title') or ''}")
|
||||
if it.get("snippet"):
|
||||
lines.append(it["snippet"])
|
||||
lines.append(f"来源:{it['note_id']}")
|
||||
lines.append("")
|
||||
return {"text": "\n".join(lines).strip(), "items": items}
|
||||
|
||||
# -- dedup & merge (phase 3) -----------------------------------------
|
||||
async def find_duplicates(self) -> list[dict[str, Any]]:
|
||||
from deerflow.knowledge.dedup import find_duplicate_groups
|
||||
|
||||
notes = await self.repo.list_all()
|
||||
return find_duplicate_groups(notes, threshold=self.config.dedup_similarity_threshold)
|
||||
|
||||
async def merge_notes(self, primary_id: str, duplicate_ids: list[str], *, updated_by: str | None = None) -> dict[str, Any] | None:
|
||||
"""Merge ``duplicate_ids`` into ``primary_id``.
|
||||
|
||||
Sources from each duplicate are re-attached to the primary, a wikilink
|
||||
+ "Merged from" note is appended to the primary body, the duplicates are
|
||||
archived (soft delete), and a ``merged_into`` relation is recorded.
|
||||
"""
|
||||
primary = await self.repo.get_note(primary_id)
|
||||
if primary is None:
|
||||
return None
|
||||
merged_titles: list[str] = []
|
||||
for dup_id in duplicate_ids:
|
||||
if dup_id == primary_id:
|
||||
continue
|
||||
dup = await self.repo.get_note(dup_id)
|
||||
if dup is None:
|
||||
continue
|
||||
# Move sources.
|
||||
for src in dup.get("sources") or []:
|
||||
await self.repo.add_sources(
|
||||
primary_id,
|
||||
[
|
||||
SourceDraft(
|
||||
source_type=src.get("source_type") or "tool",
|
||||
thread_id=src.get("thread_id"),
|
||||
message_id=src.get("message_id"),
|
||||
tool_name=src.get("tool_name"),
|
||||
title=src.get("title"),
|
||||
url=src.get("url"),
|
||||
snippet=src.get("snippet"),
|
||||
)
|
||||
],
|
||||
created_by=updated_by,
|
||||
)
|
||||
await self.repo.add_relation(from_note_id=dup_id, to_note_id=primary_id, relation_type="merged_into")
|
||||
await self.delete_note(dup_id, updated_by=updated_by)
|
||||
merged_titles.append(dup.get("title") or dup_id)
|
||||
|
||||
if merged_titles:
|
||||
appended = primary["content_md"].rstrip() + "\n\n## Merged\n\n" + "\n".join(f"- {t}" for t in merged_titles) + "\n"
|
||||
await self.update_note(primary_id, content_md=appended, updated_by=updated_by)
|
||||
return await self.repo.get_note(primary_id)
|
||||
|
||||
async def reindex_all(self) -> dict[str, int]:
|
||||
"""Rebuild embeddings for every non-archived note (admin/maintenance)."""
|
||||
notes = await self.repo.list_all()
|
||||
count = 0
|
||||
for note in notes:
|
||||
await self._reindex_embeddings(note["id"], note.get("content_md") or "")
|
||||
count += 1
|
||||
return {"reindexed": count, "embedding_enabled": int(self.embedding_client.is_enabled())}
|
||||
|
||||
async def export_graph(self) -> dict[str, Any]:
|
||||
from deerflow.utils.time import now_iso
|
||||
|
||||
notes = await self.repo.list_all()
|
||||
entities = await self.repo.list_entities() if self.config.entity_extraction_enabled else []
|
||||
relations = await self.repo.list_relations() if self.config.entity_extraction_enabled else []
|
||||
result = self.adapter.export_graph(notes, entities=entities, relations=relations, updated=now_iso())
|
||||
return {"files": result["files"], "stats": result["stats"]}
|
||||
|
||||
async def get_graph(self) -> dict[str, Any]:
|
||||
"""Return product graph data ``{nodes, edges}`` (notes + entities)."""
|
||||
from deerflow.knowledge import exporter
|
||||
|
||||
notes = await self.repo.list_all()
|
||||
entities = await self.repo.list_entities() if self.config.entity_extraction_enabled else []
|
||||
relations = await self.repo.list_relations() if self.config.entity_extraction_enabled else []
|
||||
return exporter.build_graph(notes, entities, relations)
|
||||
|
||||
def export_file_path(self, filename: str) -> Path | None:
|
||||
return self.adapter.export_file_path(filename)
|
||||
+51
@@ -0,0 +1,51 @@
|
||||
"""Built-in obsidian-wiki skill templates.
|
||||
|
||||
Mirrors the key intent of ``ar9av/obsidian-wiki`` skills so the backend can
|
||||
write vault content that stays compatible with the original tooling without
|
||||
shelling out to its CLI:
|
||||
|
||||
- ``wiki-capture`` — how a conversation becomes a *declarative* knowledge note
|
||||
(frontmatter schema + Context/Finding/Reasoning/Implications/Related body).
|
||||
- ``wiki-query`` — layered retrieval (hot.md → index.md → frontmatter → body).
|
||||
- ``wiki-export`` — graph artifacts (graph.json / graph.html / graphml / cypher).
|
||||
|
||||
These are intentionally lightweight: the structures here are what
|
||||
:mod:`deerflow.knowledge.markdown_vault` and :mod:`deerflow.knowledge.extractor`
|
||||
produce, and what the first-release LLM prompt (future enhancement) should
|
||||
target.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# Category → vault sub-directory (wiki-capture page taxonomy, §8 of the dev doc).
|
||||
CATEGORY_DIRS = {
|
||||
"synthesis": "synthesis",
|
||||
"concepts": "concepts",
|
||||
"concept": "concepts",
|
||||
"references": "references",
|
||||
"reference": "references",
|
||||
"source": "references",
|
||||
"decision": "synthesis",
|
||||
"journal": "journal",
|
||||
"session": "journal",
|
||||
}
|
||||
|
||||
# Body section headings (wiki-capture). Declarative knowledge, never a chat log.
|
||||
BODY_SECTIONS = ("Context", "Finding / Decision", "Reasoning", "Implications", "Related")
|
||||
|
||||
# Inline provenance markers reused from wiki-capture.
|
||||
INFERRED_MARKER = "^[inferred]"
|
||||
AMBIGUOUS_MARKER = "^[ambiguous]"
|
||||
|
||||
WIKI_CAPTURE_GUIDANCE = """\
|
||||
Capture *declarative* knowledge, not a chat transcript. Classify the note as one
|
||||
of: synthesis, concept, source, decision, session. Prefer at least two
|
||||
[[wikilinks]]. Mark inferred content with ^[inferred] and uncertain/conflicting
|
||||
content with ^[ambiguous]. Project-specific knowledge belongs under
|
||||
projects/<project-name>/...
|
||||
"""
|
||||
|
||||
|
||||
def category_to_dir(category: str | None) -> str:
|
||||
"""Map a wiki-capture category to its vault sub-directory."""
|
||||
return CATEGORY_DIRS.get((category or "").strip().lower(), "synthesis")
|
||||
@@ -0,0 +1,82 @@
|
||||
"""Normalize knowledge sources from threads / searches / tool results.
|
||||
|
||||
Source items come from a few shapes: backend reference-batches (built by
|
||||
``deerflow.runtime.references``), raw search-result payloads, and tool-call
|
||||
messages. They are all normalized into :class:`SourceDraft` so the repository
|
||||
and frontend see one consistent structure.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from deerflow.knowledge.schemas import SourceDraft
|
||||
|
||||
|
||||
def _clip(value: Any, limit: int = 2000) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
if not text:
|
||||
return None
|
||||
return text[:limit]
|
||||
|
||||
|
||||
def from_reference_batches(batches: list[dict[str, Any]] | None, *, thread_id: str | None = None, max_items: int = 20) -> list[SourceDraft]:
|
||||
"""Convert backend reference-batches into :class:`SourceDraft`."""
|
||||
out: list[SourceDraft] = []
|
||||
seen: set[str] = set()
|
||||
for batch in batches or []:
|
||||
msg_id = batch.get("assistant_message_id") or batch.get("id")
|
||||
for src in batch.get("sources", []) or []:
|
||||
if not isinstance(src, dict):
|
||||
continue
|
||||
url = _clip(src.get("url"), 1024)
|
||||
title = _clip(src.get("title"), 512) or _clip(src.get("source"), 512) or "来源"
|
||||
dedup_key = url or f"{title}:{_clip(src.get('snippet'), 80)}"
|
||||
if dedup_key in seen:
|
||||
continue
|
||||
seen.add(dedup_key)
|
||||
out.append(
|
||||
SourceDraft(
|
||||
source_type="search" if url else "tool",
|
||||
thread_id=thread_id,
|
||||
message_id=_clip(msg_id, 128),
|
||||
tool_name=_clip(src.get("skillName"), 128),
|
||||
title=title,
|
||||
url=url,
|
||||
snippet=_clip(src.get("snippet")),
|
||||
raw=None,
|
||||
)
|
||||
)
|
||||
if len(out) >= max_items:
|
||||
return out
|
||||
return out
|
||||
|
||||
|
||||
def from_search_results(
|
||||
query: str,
|
||||
results: list[dict[str, Any]] | None,
|
||||
*,
|
||||
thread_id: str | None = None,
|
||||
message_id: str | None = None,
|
||||
max_items: int = 20,
|
||||
) -> list[SourceDraft]:
|
||||
"""Convert raw search results (title/url/snippet) into :class:`SourceDraft`."""
|
||||
out: list[SourceDraft] = []
|
||||
for item in (results or [])[:max_items]:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
out.append(
|
||||
SourceDraft(
|
||||
source_type="search",
|
||||
thread_id=thread_id,
|
||||
message_id=message_id,
|
||||
tool_name=_clip(item.get("tool_name") or item.get("tool"), 128),
|
||||
title=_clip(item.get("title"), 512) or query[:200],
|
||||
url=_clip(item.get("url"), 1024),
|
||||
snippet=_clip(item.get("snippet") or item.get("content")),
|
||||
raw={k: item.get(k) for k in ("score",) if item.get(k) is not None} or None,
|
||||
)
|
||||
)
|
||||
return out
|
||||
+47
@@ -0,0 +1,47 @@
|
||||
"""Pure scheduling helpers for the admin leaderboard daily snapshots.
|
||||
|
||||
Kept import-light on purpose (only stdlib ``datetime``) so the decision logic
|
||||
is unit-testable without standing up the FastAPI app, the DB engine, or the
|
||||
ORM. The background scheduler in ``app/gateway/app.py`` wires this to the real
|
||||
snapshot rows.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
|
||||
def should_finalize_previous_day(
|
||||
*,
|
||||
status: str | None,
|
||||
generated_at: datetime | None,
|
||||
now: datetime,
|
||||
finalize_hour: int,
|
||||
) -> bool:
|
||||
"""Decide whether *yesterday's* snapshot should be force-recomputed once.
|
||||
|
||||
The previous day's snapshot is normally built before midnight, so it can
|
||||
miss runs / tool calls whose rows only land in the database shortly before
|
||||
or after the day boundary. Once per day, on/after ``finalize_hour`` (local
|
||||
Beijing time), we recompute it a single time so the full day is captured.
|
||||
|
||||
``now`` and ``generated_at`` are expected to be timezone-aware Beijing
|
||||
datetimes (``generated_at`` may be ``None`` for legacy rows).
|
||||
|
||||
Returns ``True`` only when all hold:
|
||||
- it is already on/after ``finalize_hour`` today;
|
||||
- a snapshot row exists (``status`` is not ``None`` — a missing day is
|
||||
left to the normal prewarm/queue path);
|
||||
- that snapshot is not already ``queued``/``running`` (a finalize already
|
||||
in flight, or normal processing);
|
||||
- it was generated before today (i.e. not yet finalized for this day).
|
||||
"""
|
||||
if status is None:
|
||||
return False
|
||||
if now.hour < finalize_hour:
|
||||
return False
|
||||
if status in ("queued", "running"):
|
||||
return False
|
||||
if generated_at is not None and generated_at.date() >= now.date():
|
||||
return False
|
||||
return True
|
||||
+45
-42
@@ -119,18 +119,19 @@ class AgentStore(ABC):
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def list_paginated(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
scope: str | None,
|
||||
search: str | None,
|
||||
tag_ids: list[str] | None,
|
||||
page: int,
|
||||
page_size: int,
|
||||
admin: bool = False,
|
||||
sort: str = "time_desc",
|
||||
) -> tuple[list[dict[str, Any]], int]:
|
||||
async def list_paginated(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
scope: str | None,
|
||||
search: str | None,
|
||||
tag_ids: list[str] | None,
|
||||
page: int,
|
||||
page_size: int,
|
||||
admin: bool = False,
|
||||
sort: str = "time_desc",
|
||||
owner: str | None = None,
|
||||
) -> tuple[list[dict[str, Any]], int]:
|
||||
"""Return one page of agents plus the total matching count.
|
||||
|
||||
``scope``:
|
||||
@@ -142,13 +143,15 @@ class AgentStore(ABC):
|
||||
agent), but still honors ``search`` / ``tag_ids``. Caller authorizes.
|
||||
|
||||
``search`` matches the raw name OR its desensitized display form.
|
||||
``tag_ids`` keeps agents carrying ANY of the tags (OR). Both filters,
|
||||
the COUNT, and LIMIT/OFFSET all execute in SQL.
|
||||
``tag_ids`` keeps agents carrying ANY of the tags (OR). ``owner`` keeps
|
||||
only agents published by that user (``user_id`` equality) — built-in
|
||||
agents (no owner) never match. All filters, the COUNT, and LIMIT/OFFSET
|
||||
all execute in SQL.
|
||||
|
||||
Returns ``(rows, total)`` where ``rows`` is at most ``page_size`` items
|
||||
for the 1-based ``page``.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
for the 1-based ``page``.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
# --- Listing cache (skills / tool_groups / model) ---
|
||||
|
||||
@@ -199,28 +202,28 @@ class AgentStore(ABC):
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def delete_favorites_for_agent(self, agent_id: str) -> None:
|
||||
"""Drop every user's favorite of ``agent_id`` (called on agent delete)."""
|
||||
raise NotImplementedError
|
||||
|
||||
# --- Per-user pins ("置顶") ---
|
||||
|
||||
@abstractmethod
|
||||
async def add_pin(self, user_id: str, agent_id: str) -> None:
|
||||
"""Pin ``agent_id`` for ``user_id``. Idempotent."""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def remove_pin(self, user_id: str, agent_id: str) -> bool:
|
||||
"""Remove a per-user pin. Returns ``False`` when none existed."""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def list_pin_ids(self, user_id: str) -> set[str]:
|
||||
"""Return the set of agent ids pinned by ``user_id``."""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def delete_pins_for_agent(self, agent_id: str) -> None:
|
||||
"""Drop every user's pin of ``agent_id`` (called on agent delete)."""
|
||||
raise NotImplementedError
|
||||
async def delete_favorites_for_agent(self, agent_id: str) -> None:
|
||||
"""Drop every user's favorite of ``agent_id`` (called on agent delete)."""
|
||||
raise NotImplementedError
|
||||
|
||||
# --- Per-user pins ("置顶") ---
|
||||
|
||||
@abstractmethod
|
||||
async def add_pin(self, user_id: str, agent_id: str) -> None:
|
||||
"""Pin ``agent_id`` for ``user_id``. Idempotent."""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def remove_pin(self, user_id: str, agent_id: str) -> bool:
|
||||
"""Remove a per-user pin. Returns ``False`` when none existed."""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def list_pin_ids(self, user_id: str) -> set[str]:
|
||||
"""Return the set of agent ids pinned by ``user_id``."""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def delete_pins_for_agent(self, agent_id: str) -> None:
|
||||
"""Drop every user's pin of ``agent_id`` (called on agent delete)."""
|
||||
raise NotImplementedError
|
||||
+62
-59
@@ -21,9 +21,9 @@ def _copy(data: dict[str, Any]) -> dict[str, Any]:
|
||||
|
||||
class MemoryAgentStore(AgentStore):
|
||||
def __init__(self) -> None:
|
||||
self._agents: dict[str, dict[str, Any]] = {}
|
||||
self._favorites: set[tuple[str, str]] = set()
|
||||
self._pins: set[tuple[str, str]] = set()
|
||||
self._agents: dict[str, dict[str, Any]] = {}
|
||||
self._favorites: set[tuple[str, str]] = set()
|
||||
self._pins: set[tuple[str, str]] = set()
|
||||
# Listing cache mirrors (see AgentExtrasRow / AgentSkillRow):
|
||||
self._skills: dict[str, list[str]] = {}
|
||||
self._extras: dict[str, dict[str, Any]] = {}
|
||||
@@ -186,51 +186,54 @@ class MemoryAgentStore(AgentStore):
|
||||
return bool(agent.get("is_private"))
|
||||
return self._is_visible(agent, user_id)
|
||||
|
||||
async def list_paginated(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
scope: str | None,
|
||||
async def list_paginated(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
scope: str | None,
|
||||
search: str | None,
|
||||
tag_ids: list[str] | None,
|
||||
page: int,
|
||||
page_size: int,
|
||||
admin: bool = False,
|
||||
sort: str = "time_desc",
|
||||
) -> tuple[list[dict[str, Any]], int]:
|
||||
tag_ids: list[str] | None,
|
||||
page: int,
|
||||
page_size: int,
|
||||
admin: bool = False,
|
||||
sort: str = "time_desc",
|
||||
owner: str | None = None,
|
||||
) -> tuple[list[dict[str, Any]], int]:
|
||||
# NOTE: the in-memory store has no tag-assignment data, so ``tag_ids``
|
||||
# is ignored here. Tag-filter behavior is covered by the SQL repo tests.
|
||||
owner_key = owner.strip() if owner and owner.strip() else None
|
||||
matched = [
|
||||
a
|
||||
for a in self._agents.values()
|
||||
if self._matches_scope(a, user_id=user_id, scope=scope, admin=admin)
|
||||
and (not search or not search.strip() or name_matches(a.get("name") or "", search))
|
||||
and (owner_key is None or a.get("user_id") == owner_key)
|
||||
]
|
||||
if sort == "time_asc":
|
||||
matched.sort(key=lambda a: (a.get("created_at") or "", a.get("id") or ""))
|
||||
elif sort == "updated_desc":
|
||||
matched.sort(
|
||||
key=lambda a: (a.get("updated_at") or "", a.get("id") or ""),
|
||||
reverse=True,
|
||||
)
|
||||
elif sort == "updated_asc":
|
||||
matched.sort(key=lambda a: (a.get("updated_at") or "", a.get("id") or ""))
|
||||
elif sort == "name_asc":
|
||||
matched.sort(key=lambda a: ((a.get("name") or "").lower(), a.get("id") or ""))
|
||||
elif sort == "name_desc":
|
||||
matched.sort(
|
||||
key=lambda a: ((a.get("name") or "").lower(), a.get("id") or ""),
|
||||
reverse=True,
|
||||
)
|
||||
else:
|
||||
matched.sort(
|
||||
key=lambda a: (a.get("created_at") or "", a.get("id") or ""),
|
||||
reverse=True,
|
||||
)
|
||||
pin_ids = {agent_id for uid, agent_id in self._pins if uid == user_id}
|
||||
matched.sort(key=lambda a: (0 if a.get("id") in pin_ids else 1,))
|
||||
total = len(matched)
|
||||
start = max(page - 1, 0) * page_size
|
||||
if sort == "time_asc":
|
||||
matched.sort(key=lambda a: (a.get("created_at") or "", a.get("id") or ""))
|
||||
elif sort == "updated_desc":
|
||||
matched.sort(
|
||||
key=lambda a: (a.get("updated_at") or "", a.get("id") or ""),
|
||||
reverse=True,
|
||||
)
|
||||
elif sort == "updated_asc":
|
||||
matched.sort(key=lambda a: (a.get("updated_at") or "", a.get("id") or ""))
|
||||
elif sort == "name_asc":
|
||||
matched.sort(key=lambda a: ((a.get("name") or "").lower(), a.get("id") or ""))
|
||||
elif sort == "name_desc":
|
||||
matched.sort(
|
||||
key=lambda a: ((a.get("name") or "").lower(), a.get("id") or ""),
|
||||
reverse=True,
|
||||
)
|
||||
else:
|
||||
matched.sort(
|
||||
key=lambda a: (a.get("created_at") or "", a.get("id") or ""),
|
||||
reverse=True,
|
||||
)
|
||||
pin_ids = {agent_id for uid, agent_id in self._pins if uid == user_id}
|
||||
matched.sort(key=lambda a: (0 if a.get("id") in pin_ids else 1,))
|
||||
total = len(matched)
|
||||
start = max(page - 1, 0) * page_size
|
||||
page_rows = matched[start : start + page_size]
|
||||
return [_copy(a) for a in page_rows], total
|
||||
|
||||
@@ -275,23 +278,23 @@ class MemoryAgentStore(AgentStore):
|
||||
async def list_favorite_ids(self, user_id: str) -> set[str]:
|
||||
return {agent_id for (uid, agent_id) in self._favorites if uid == user_id}
|
||||
|
||||
async def delete_favorites_for_agent(self, agent_id: str) -> None:
|
||||
self._favorites = {(uid, aid) for (uid, aid) in self._favorites if aid != agent_id}
|
||||
|
||||
# --- Pins ("置顶") ---
|
||||
|
||||
async def add_pin(self, user_id: str, agent_id: str) -> None:
|
||||
self._pins.add((user_id, agent_id))
|
||||
|
||||
async def remove_pin(self, user_id: str, agent_id: str) -> bool:
|
||||
key = (user_id, agent_id)
|
||||
if key not in self._pins:
|
||||
return False
|
||||
self._pins.remove(key)
|
||||
return True
|
||||
|
||||
async def list_pin_ids(self, user_id: str) -> set[str]:
|
||||
return {agent_id for uid, agent_id in self._pins if uid == user_id}
|
||||
|
||||
async def delete_pins_for_agent(self, agent_id: str) -> None:
|
||||
self._pins = {(uid, aid) for (uid, aid) in self._pins if aid != agent_id}
|
||||
async def delete_favorites_for_agent(self, agent_id: str) -> None:
|
||||
self._favorites = {(uid, aid) for (uid, aid) in self._favorites if aid != agent_id}
|
||||
|
||||
# --- Pins ("置顶") ---
|
||||
|
||||
async def add_pin(self, user_id: str, agent_id: str) -> None:
|
||||
self._pins.add((user_id, agent_id))
|
||||
|
||||
async def remove_pin(self, user_id: str, agent_id: str) -> bool:
|
||||
key = (user_id, agent_id)
|
||||
if key not in self._pins:
|
||||
return False
|
||||
self._pins.remove(key)
|
||||
return True
|
||||
|
||||
async def list_pin_ids(self, user_id: str) -> set[str]:
|
||||
return {agent_id for uid, agent_id in self._pins if uid == user_id}
|
||||
|
||||
async def delete_pins_for_agent(self, agent_id: str) -> None:
|
||||
self._pins = {(uid, aid) for (uid, aid) in self._pins if aid != agent_id}
|
||||
+90
-86
@@ -6,19 +6,19 @@ from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
|
||||
from sqlalchemy import and_, case, delete, func, or_, select
|
||||
from sqlalchemy import and_, case, delete, func, or_, select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
from deerflow.config.agent_name_desensitize import source_terms_for_needle
|
||||
from deerflow.persistence.agents.base import AgentStore
|
||||
from deerflow.persistence.agents.model import (
|
||||
AgentExtrasRow,
|
||||
AgentFavoriteRow,
|
||||
AgentPinRow,
|
||||
AgentRow,
|
||||
AgentSkillRow,
|
||||
)
|
||||
AgentExtrasRow,
|
||||
AgentFavoriteRow,
|
||||
AgentPinRow,
|
||||
AgentRow,
|
||||
AgentSkillRow,
|
||||
)
|
||||
from deerflow.persistence.tags.model import TagAssignmentRow
|
||||
|
||||
|
||||
@@ -278,18 +278,19 @@ class AgentRepository(AgentStore):
|
||||
)
|
||||
return AgentRow.id.in_(subq)
|
||||
|
||||
async def list_paginated(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
scope: str | None,
|
||||
async def list_paginated(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
scope: str | None,
|
||||
search: str | None,
|
||||
tag_ids: list[str] | None,
|
||||
page: int,
|
||||
page_size: int,
|
||||
admin: bool = False,
|
||||
sort: str = "time_desc",
|
||||
) -> tuple[list[dict[str, Any]], int]:
|
||||
tag_ids: list[str] | None,
|
||||
page: int,
|
||||
page_size: int,
|
||||
admin: bool = False,
|
||||
sort: str = "time_desc",
|
||||
owner: str | None = None,
|
||||
) -> tuple[list[dict[str, Any]], int]:
|
||||
filters = []
|
||||
scope_filter = self._scope_filter(user_id=user_id, scope=scope, admin=admin)
|
||||
if scope_filter is not None:
|
||||
@@ -298,30 +299,33 @@ class AgentRepository(AgentStore):
|
||||
filters.append(self._search_clause(search))
|
||||
if tag_ids:
|
||||
filters.append(self._tag_clause(tag_ids))
|
||||
if owner and owner.strip():
|
||||
# Filter to one publisher. Built-in agents (user_id IS NULL) never match.
|
||||
filters.append(AgentRow.user_id == owner.strip())
|
||||
|
||||
count_stmt = select(func.count()).select_from(AgentRow)
|
||||
pin_subq = select(AgentPinRow.agent_id).where(AgentPinRow.user_id == user_id)
|
||||
pin_rank = case((AgentRow.id.in_(pin_subq), 0), else_=1)
|
||||
|
||||
if sort == "time_asc":
|
||||
order_by = (AgentRow.created_at.asc(), AgentRow.id.asc())
|
||||
elif sort == "updated_desc":
|
||||
order_by = (AgentRow.updated_at.desc(), AgentRow.id.asc())
|
||||
elif sort == "updated_asc":
|
||||
order_by = (AgentRow.updated_at.asc(), AgentRow.id.asc())
|
||||
elif sort == "name_asc":
|
||||
order_by = (func.lower(AgentRow.name).asc(), AgentRow.id.asc())
|
||||
elif sort == "name_desc":
|
||||
order_by = (func.lower(AgentRow.name).desc(), AgentRow.id.asc())
|
||||
else:
|
||||
order_by = (AgentRow.created_at.desc(), AgentRow.id.asc())
|
||||
|
||||
page_stmt = (
|
||||
select(AgentRow)
|
||||
.order_by(pin_rank.asc(), *order_by)
|
||||
.limit(page_size)
|
||||
.offset(max(page - 1, 0) * page_size)
|
||||
)
|
||||
pin_subq = select(AgentPinRow.agent_id).where(AgentPinRow.user_id == user_id)
|
||||
pin_rank = case((AgentRow.id.in_(pin_subq), 0), else_=1)
|
||||
|
||||
if sort == "time_asc":
|
||||
order_by = (AgentRow.created_at.asc(), AgentRow.id.asc())
|
||||
elif sort == "updated_desc":
|
||||
order_by = (AgentRow.updated_at.desc(), AgentRow.id.asc())
|
||||
elif sort == "updated_asc":
|
||||
order_by = (AgentRow.updated_at.asc(), AgentRow.id.asc())
|
||||
elif sort == "name_asc":
|
||||
order_by = (func.lower(AgentRow.name).asc(), AgentRow.id.asc())
|
||||
elif sort == "name_desc":
|
||||
order_by = (func.lower(AgentRow.name).desc(), AgentRow.id.asc())
|
||||
else:
|
||||
order_by = (AgentRow.created_at.desc(), AgentRow.id.asc())
|
||||
|
||||
page_stmt = (
|
||||
select(AgentRow)
|
||||
.order_by(pin_rank.asc(), *order_by)
|
||||
.limit(page_size)
|
||||
.offset(max(page - 1, 0) * page_size)
|
||||
)
|
||||
if filters:
|
||||
where = and_(*filters)
|
||||
count_stmt = count_stmt.where(where)
|
||||
@@ -441,49 +445,49 @@ class AgentRepository(AgentStore):
|
||||
)
|
||||
return {row[0] for row in result.all()}
|
||||
|
||||
async def delete_favorites_for_agent(self, agent_id: str) -> None:
|
||||
async with self._sf() as session:
|
||||
await session.execute(delete(AgentFavoriteRow).where(AgentFavoriteRow.agent_id == agent_id))
|
||||
await session.commit()
|
||||
|
||||
# --- Pins ("置顶") ---
|
||||
|
||||
async def add_pin(self, user_id: str, agent_id: str) -> None:
|
||||
async with self._sf() as session:
|
||||
existing = (
|
||||
await session.execute(
|
||||
select(AgentPinRow.id)
|
||||
.where(AgentPinRow.user_id == user_id, AgentPinRow.agent_id == agent_id)
|
||||
.limit(1)
|
||||
)
|
||||
).scalar_one_or_none()
|
||||
if existing is not None:
|
||||
return
|
||||
session.add(AgentPinRow(id=uuid4().hex, user_id=user_id, agent_id=agent_id))
|
||||
try:
|
||||
await session.commit()
|
||||
except IntegrityError:
|
||||
await session.rollback()
|
||||
|
||||
async def remove_pin(self, user_id: str, agent_id: str) -> bool:
|
||||
async with self._sf() as session:
|
||||
result = await session.execute(
|
||||
delete(AgentPinRow).where(
|
||||
AgentPinRow.user_id == user_id,
|
||||
AgentPinRow.agent_id == agent_id,
|
||||
)
|
||||
)
|
||||
await session.commit()
|
||||
return (result.rowcount or 0) > 0
|
||||
|
||||
async def list_pin_ids(self, user_id: str) -> set[str]:
|
||||
async with self._sf() as session:
|
||||
result = await session.execute(
|
||||
select(AgentPinRow.agent_id).where(AgentPinRow.user_id == user_id)
|
||||
)
|
||||
return {row[0] for row in result.all()}
|
||||
|
||||
async def delete_pins_for_agent(self, agent_id: str) -> None:
|
||||
async with self._sf() as session:
|
||||
await session.execute(delete(AgentPinRow).where(AgentPinRow.agent_id == agent_id))
|
||||
await session.commit()
|
||||
async def delete_favorites_for_agent(self, agent_id: str) -> None:
|
||||
async with self._sf() as session:
|
||||
await session.execute(delete(AgentFavoriteRow).where(AgentFavoriteRow.agent_id == agent_id))
|
||||
await session.commit()
|
||||
|
||||
# --- Pins ("置顶") ---
|
||||
|
||||
async def add_pin(self, user_id: str, agent_id: str) -> None:
|
||||
async with self._sf() as session:
|
||||
existing = (
|
||||
await session.execute(
|
||||
select(AgentPinRow.id)
|
||||
.where(AgentPinRow.user_id == user_id, AgentPinRow.agent_id == agent_id)
|
||||
.limit(1)
|
||||
)
|
||||
).scalar_one_or_none()
|
||||
if existing is not None:
|
||||
return
|
||||
session.add(AgentPinRow(id=uuid4().hex, user_id=user_id, agent_id=agent_id))
|
||||
try:
|
||||
await session.commit()
|
||||
except IntegrityError:
|
||||
await session.rollback()
|
||||
|
||||
async def remove_pin(self, user_id: str, agent_id: str) -> bool:
|
||||
async with self._sf() as session:
|
||||
result = await session.execute(
|
||||
delete(AgentPinRow).where(
|
||||
AgentPinRow.user_id == user_id,
|
||||
AgentPinRow.agent_id == agent_id,
|
||||
)
|
||||
)
|
||||
await session.commit()
|
||||
return (result.rowcount or 0) > 0
|
||||
|
||||
async def list_pin_ids(self, user_id: str) -> set[str]:
|
||||
async with self._sf() as session:
|
||||
result = await session.execute(
|
||||
select(AgentPinRow.agent_id).where(AgentPinRow.user_id == user_id)
|
||||
)
|
||||
return {row[0] for row in result.all()}
|
||||
|
||||
async def delete_pins_for_agent(self, agent_id: str) -> None:
|
||||
async with self._sf() as session:
|
||||
await session.execute(delete(AgentPinRow).where(AgentPinRow.agent_id == agent_id))
|
||||
await session.commit()
|
||||
+42
@@ -0,0 +1,42 @@
|
||||
"""Add name_zh column to skills table.
|
||||
|
||||
Revision ID: 20260609_01
|
||||
Revises: 20260608_01
|
||||
Create Date: 2026-06-09
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy import inspect
|
||||
|
||||
|
||||
revision = "20260609_01"
|
||||
down_revision = "20260608_01"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def _has_table(table_name: str) -> bool:
|
||||
inspector = inspect(op.get_bind())
|
||||
return table_name in inspector.get_table_names()
|
||||
|
||||
|
||||
def _has_column(table_name: str, column_name: str) -> bool:
|
||||
if not _has_table(table_name):
|
||||
return False
|
||||
inspector = inspect(op.get_bind())
|
||||
return any(col["name"] == column_name for col in inspector.get_columns(table_name))
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# ``skills`` may not exist yet on a brand-new DB — create_all handles that
|
||||
# case with the column already present, so only patch existing tables.
|
||||
if _has_table("skills") and not _has_column("skills", "name_zh"):
|
||||
op.add_column("skills", sa.Column("name_zh", sa.String(length=128), nullable=True))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
if _has_column("skills", "name_zh"):
|
||||
op.drop_column("skills", "name_zh")
|
||||
+30
@@ -0,0 +1,30 @@
|
||||
"""Merge the skill_name_zh and chain-stages migration heads.
|
||||
|
||||
Two lineages both descend from ``20260608_01`` (admin_leaderboard_daily_stats):
|
||||
``20260608_02`` … ``20260608_06`` (the chain-stages / parallel-agent branch) and
|
||||
``20260609_01`` (skills.name_zh), which branched directly off ``20260608_01``
|
||||
instead of off the latest head. That left the graph with two heads and made
|
||||
``alembic upgrade head`` fail with "Multiple head revisions are present". This
|
||||
no-op merge revision joins both lineages so ``upgrade head`` resolves to a single
|
||||
head again.
|
||||
|
||||
Revision ID: 20260609_02
|
||||
Revises: 20260608_06, 20260609_01
|
||||
Create Date: 2026-06-10
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
revision = "20260609_02"
|
||||
down_revision = ("20260608_06", "20260609_01")
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
pass
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
pass
|
||||
+18
@@ -25,6 +25,16 @@ from deerflow.persistence.admin_stats.model import AdminLeaderboardDailyStatRow
|
||||
from deerflow.persistence.business_mapping.model import BusinessMappingRow
|
||||
from deerflow.persistence.memory_config.model import UserMemoryConfigRow
|
||||
from deerflow.persistence.feedback.model import FeedbackRow
|
||||
from deerflow.knowledge.models import (
|
||||
KnowledgeEmbeddingRow,
|
||||
KnowledgeEntityRow,
|
||||
KnowledgeNoteRow,
|
||||
KnowledgeNoteTagRow,
|
||||
KnowledgeNoteVersionRow,
|
||||
KnowledgeRelationRow,
|
||||
KnowledgeSourceRow,
|
||||
KnowledgeTagRow,
|
||||
)
|
||||
from deerflow.persistence.llm_metrics.model import LlmCallMetricRow
|
||||
from deerflow.persistence.models.run_event import RunEventRow
|
||||
from deerflow.persistence.notifications.model import NotificationRow
|
||||
@@ -63,6 +73,14 @@ __all__ = [
|
||||
"AgentSkillRow",
|
||||
"AdminLeaderboardDailyStatRow",
|
||||
"BusinessMappingRow",
|
||||
"KnowledgeEmbeddingRow",
|
||||
"KnowledgeEntityRow",
|
||||
"KnowledgeNoteRow",
|
||||
"KnowledgeNoteTagRow",
|
||||
"KnowledgeNoteVersionRow",
|
||||
"KnowledgeRelationRow",
|
||||
"KnowledgeSourceRow",
|
||||
"KnowledgeTagRow",
|
||||
"LlmCallMetricRow",
|
||||
"NotificationRow",
|
||||
"ParallelAgentTaskRow",
|
||||
|
||||
+6
@@ -70,6 +70,7 @@ class MemorySkillStore(SkillStore):
|
||||
"owner_user_id": data.get("owner_user_id"),
|
||||
"published": bool(data.get("published", False)),
|
||||
"detail": data.get("detail"),
|
||||
"name_zh": data.get("name_zh"),
|
||||
"created_at": _now(),
|
||||
"updated_at": _now(),
|
||||
}
|
||||
@@ -85,6 +86,7 @@ class MemorySkillStore(SkillStore):
|
||||
"owner_user_id": data.get("owner_user_id"),
|
||||
"published": bool(data.get("published", True)),
|
||||
"detail": data.get("detail"),
|
||||
"name_zh": data.get("name_zh"),
|
||||
"created_at": _now(),
|
||||
"updated_at": _now(),
|
||||
}
|
||||
@@ -102,6 +104,8 @@ class MemorySkillStore(SkillStore):
|
||||
record["published"] = bool(data["published"])
|
||||
if "detail" in data:
|
||||
record["detail"] = data["detail"]
|
||||
if "name_zh" in data:
|
||||
record["name_zh"] = data["name_zh"]
|
||||
record["updated_at"] = _now()
|
||||
return _with_derived(_copy(record)) # type: ignore[arg-type]
|
||||
|
||||
@@ -115,6 +119,8 @@ class MemorySkillStore(SkillStore):
|
||||
record["owner_user_id"] = data["owner_user_id"]
|
||||
if "detail" in data:
|
||||
record["detail"] = data["detail"]
|
||||
if "name_zh" in data:
|
||||
record["name_zh"] = data["name_zh"]
|
||||
record["updated_at"] = _now()
|
||||
return _with_derived(_copy(record)) # type: ignore[arg-type]
|
||||
|
||||
|
||||
@@ -29,6 +29,10 @@ class SkillRow(Base):
|
||||
# kept in the DB so admins can annotate built-in skills without rewriting
|
||||
# the shared on-disk SKILL.md file.
|
||||
detail: Mapped[str | None] = mapped_column(Text, nullable=True, default=None)
|
||||
# User-configured or LLM-generated Chinese display name. Kept in the DB (not
|
||||
# the on-disk SKILL.md) so it can be set for built-in skills too, and so the
|
||||
# shared source file is never rewritten.
|
||||
name_zh: Mapped[str | None] = mapped_column(String(128), nullable=True, default=None)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, default=lambda: datetime.now(UTC))
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC))
|
||||
|
||||
|
||||
@@ -64,6 +64,7 @@ class SkillRepository(SkillStore):
|
||||
owner_user_id=data.get("owner_user_id"),
|
||||
published=bool(data.get("published", False)),
|
||||
detail=data.get("detail"),
|
||||
name_zh=data.get("name_zh"),
|
||||
created_at=now,
|
||||
updated_at=now,
|
||||
)
|
||||
@@ -83,6 +84,7 @@ class SkillRepository(SkillStore):
|
||||
owner_user_id=data.get("owner_user_id"),
|
||||
published=bool(data.get("published", True)),
|
||||
detail=data.get("detail"),
|
||||
name_zh=data.get("name_zh"),
|
||||
created_at=now,
|
||||
updated_at=now,
|
||||
)
|
||||
@@ -105,6 +107,8 @@ class SkillRepository(SkillStore):
|
||||
row.published = bool(data["published"])
|
||||
if "detail" in data:
|
||||
row.detail = data["detail"]
|
||||
if "name_zh" in data:
|
||||
row.name_zh = data["name_zh"]
|
||||
row.updated_at = datetime.now(UTC)
|
||||
await session.commit()
|
||||
await session.refresh(row)
|
||||
@@ -121,6 +125,8 @@ class SkillRepository(SkillStore):
|
||||
row.owner_user_id = data["owner_user_id"]
|
||||
if "detail" in data:
|
||||
row.detail = data["detail"]
|
||||
if "name_zh" in data:
|
||||
row.name_zh = data["name_zh"]
|
||||
row.updated_at = datetime.now(UTC)
|
||||
await session.commit()
|
||||
await session.refresh(row)
|
||||
|
||||
+302
-295
@@ -1,295 +1,302 @@
|
||||
"""Async checkpointer factory.
|
||||
|
||||
Provides an **async context manager** for long-running async servers that need
|
||||
proper resource cleanup.
|
||||
|
||||
Supported backends: memory, sqlite, postgres.
|
||||
|
||||
Usage (e.g. FastAPI lifespan)::
|
||||
|
||||
from deerflow.runtime.checkpointer.async_provider import make_checkpointer
|
||||
|
||||
async with make_checkpointer() as checkpointer:
|
||||
app.state.checkpointer = checkpointer # InMemorySaver if not configured
|
||||
|
||||
For sync usage see :mod:`deerflow.runtime.checkpointer.provider`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import logging
|
||||
from collections.abc import AsyncIterator
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from langgraph.types import Checkpointer
|
||||
|
||||
from deerflow.config.app_config import AppConfig, get_app_config
|
||||
from deerflow.runtime.checkpointer.provider import (
|
||||
POSTGRES_CONN_REQUIRED,
|
||||
POSTGRES_INSTALL,
|
||||
SQLITE_INSTALL,
|
||||
)
|
||||
from deerflow.runtime.store._sqlite_utils import ensure_sqlite_parent_dir, resolve_sqlite_conn_str
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _csv_env(name: str) -> list[str]:
|
||||
raw = os.getenv(name, "")
|
||||
return [part.strip() for part in raw.split(",") if part.strip()]
|
||||
|
||||
|
||||
def _dual_read_enabled(config) -> bool:
|
||||
if getattr(config, "dual_read_enabled", False):
|
||||
return True
|
||||
return os.getenv("DEER_FLOW_CHECKPOINT_DUAL_READ", "").strip().lower() in {"1", "true", "yes", "on"}
|
||||
|
||||
|
||||
def _use_postgres_enabled(config) -> bool:
|
||||
raw = os.getenv("DEER_FLOW_USE_PG_CHECKPOINTER")
|
||||
if raw:
|
||||
return raw.strip().lower() not in {"0", "false", "no", "off"}
|
||||
return bool(getattr(config, "use_postgres", True))
|
||||
|
||||
|
||||
def _dual_read_paths(config) -> list[str]:
|
||||
paths = list(getattr(config, "dual_read_sqlite_paths", []) or [])
|
||||
paths.extend(_csv_env("DEER_FLOW_CHECKPOINT_SQLITE_PATHS"))
|
||||
resolved: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for path in paths:
|
||||
conn_str = resolve_sqlite_conn_str(path)
|
||||
if conn_str == ":memory:" or conn_str.startswith("file:"):
|
||||
logger.warning("Skipping unsupported checkpoint migration SQLite path: %s", path)
|
||||
continue
|
||||
if conn_str not in seen:
|
||||
resolved.append(conn_str)
|
||||
seen.add(conn_str)
|
||||
return resolved
|
||||
|
||||
|
||||
def _sqlite_conn_str_for_postgres_fallback(config) -> str:
|
||||
configured = getattr(config, "sqlite_connection_string", None)
|
||||
if configured:
|
||||
return configured
|
||||
paths = _dual_read_paths(config)
|
||||
return paths[0] if paths else "store.db"
|
||||
|
||||
|
||||
@contextlib.asynccontextmanager
|
||||
async def _pg_pool_saver(conn_str: str) -> AsyncIterator[Checkpointer]:
|
||||
"""Pool-backed ``AsyncPostgresSaver``, resilient to idle-closed connections.
|
||||
|
||||
``AsyncPostgresSaver.from_conn_string`` opens **one** long-lived connection.
|
||||
对远程 Postgres(本项目 47.88.25.99 这类),空闲几分钟后连接会被服务端 / NAT /
|
||||
连接池中间件断开,而 psycopg **不会自动重连** —— 下一次 checkpoint 操作就抛
|
||||
``OperationalError: the connection is closed``,整个 run 直接失败(实测:圆桌第二步
|
||||
完成后闲置 ~6 分钟,用户再人工干预 → leader run 读 checkpoint 时撞死连接 → 前端
|
||||
"(无文本输出)")。
|
||||
|
||||
用带健康检查的连接池根治:``check=AsyncConnectionPool.check_connection`` 在把连接交给
|
||||
调用方前先验活,坏连接直接丢弃重建;``max_idle`` 主动回收久闲连接,赶在远程 idle 超时
|
||||
之前。这样上层永远拿不到失效连接。``prepare_threshold=0`` 关预备语句,兼容 pgbouncer
|
||||
一类连接池(8432 非标准端口很可能就是)。
|
||||
"""
|
||||
try:
|
||||
from langgraph.checkpoint.postgres.aio import AsyncPostgresSaver
|
||||
except ImportError as exc:
|
||||
raise ImportError(POSTGRES_INSTALL) from exc
|
||||
from psycopg_pool import AsyncConnectionPool
|
||||
|
||||
try:
|
||||
max_size = int(os.getenv("DEER_FLOW_PG_POOL_MAX", "20"))
|
||||
except ValueError:
|
||||
max_size = 20
|
||||
|
||||
pool = AsyncConnectionPool(
|
||||
conninfo=conn_str,
|
||||
min_size=1,
|
||||
max_size=max_size,
|
||||
open=False,
|
||||
kwargs={"autocommit": True, "prepare_threshold": 0},
|
||||
check=AsyncConnectionPool.check_connection,
|
||||
max_idle=120.0,
|
||||
max_lifetime=1800.0,
|
||||
)
|
||||
await pool.open(wait=True)
|
||||
try:
|
||||
saver = AsyncPostgresSaver(pool)
|
||||
await saver.setup()
|
||||
logger.info("Checkpointer: Postgres connection pool ready (max_size=%d, health-checked)", max_size)
|
||||
yield saver
|
||||
finally:
|
||||
await pool.close()
|
||||
|
||||
|
||||
@contextlib.asynccontextmanager
|
||||
async def _sqlite_checkpointer_from_conn_str(raw: str) -> AsyncIterator[Checkpointer]:
|
||||
try:
|
||||
from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver
|
||||
except ImportError as exc:
|
||||
raise ImportError(SQLITE_INSTALL) from exc
|
||||
|
||||
conn_str = resolve_sqlite_conn_str(raw)
|
||||
await asyncio.to_thread(ensure_sqlite_parent_dir, conn_str)
|
||||
async with AsyncSqliteSaver.from_conn_string(conn_str) as saver:
|
||||
await saver.setup()
|
||||
yield saver
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Async factory
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@contextlib.asynccontextmanager
|
||||
async def _async_checkpointer(config) -> AsyncIterator[Checkpointer]:
|
||||
"""Async context manager that constructs and tears down a checkpointer."""
|
||||
if config.type == "memory":
|
||||
from langgraph.checkpoint.memory import InMemorySaver
|
||||
|
||||
yield InMemorySaver()
|
||||
return
|
||||
|
||||
if config.type == "sqlite":
|
||||
async with _sqlite_checkpointer_from_conn_str(config.connection_string or "store.db") as saver:
|
||||
yield saver
|
||||
return
|
||||
|
||||
if config.type == "postgres":
|
||||
if not _use_postgres_enabled(config):
|
||||
conn_str = _sqlite_conn_str_for_postgres_fallback(config)
|
||||
logger.info("Checkpointer: use_postgres=false; using SQLite fallback (%s)", conn_str)
|
||||
async with _sqlite_checkpointer_from_conn_str(conn_str) as saver:
|
||||
yield saver
|
||||
return
|
||||
|
||||
if not config.connection_string:
|
||||
raise ValueError(POSTGRES_CONN_REQUIRED)
|
||||
|
||||
# 连接池 + 健康检查,根治远程 Postgres 单连接空闲被关导致的 run 崩溃(见 _pg_pool_saver)。
|
||||
async with _pg_pool_saver(config.connection_string) as saver:
|
||||
if not _dual_read_enabled(config):
|
||||
yield saver
|
||||
return
|
||||
|
||||
from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver
|
||||
|
||||
from deerflow.runtime.checkpointer.checkpoint_migration import (
|
||||
DualReadCheckpointer,
|
||||
SQLiteCheckpointSource,
|
||||
)
|
||||
|
||||
sources: list[SQLiteCheckpointSource] = []
|
||||
for sqlite_path in _dual_read_paths(config):
|
||||
if not Path(sqlite_path).exists():
|
||||
logger.info("Checkpoint dual-read source does not exist, skipping: %s", sqlite_path)
|
||||
continue
|
||||
ctx = AsyncSqliteSaver.from_conn_string(sqlite_path)
|
||||
source_saver = await ctx.__aenter__()
|
||||
await source_saver.setup()
|
||||
sources.append(
|
||||
SQLiteCheckpointSource(
|
||||
path=sqlite_path,
|
||||
saver=source_saver,
|
||||
context=ctx,
|
||||
progress_path=Path(sqlite_path).with_suffix(Path(sqlite_path).suffix + ".pg-migration.json"),
|
||||
)
|
||||
)
|
||||
|
||||
if not sources:
|
||||
logger.info("Checkpoint dual-read enabled but no SQLite sources were opened; using Postgres only")
|
||||
yield saver
|
||||
return
|
||||
|
||||
proxy = DualReadCheckpointer(
|
||||
saver,
|
||||
sources,
|
||||
)
|
||||
try:
|
||||
yield proxy
|
||||
finally:
|
||||
await proxy.aclose_sources()
|
||||
return
|
||||
|
||||
raise ValueError(f"Unknown checkpointer type: {config.type!r}")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Public async context manager
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@contextlib.asynccontextmanager
|
||||
async def _async_checkpointer_from_database(db_config) -> AsyncIterator[Checkpointer]:
|
||||
"""Async context manager that constructs a checkpointer from unified DatabaseConfig."""
|
||||
if db_config.backend == "memory":
|
||||
from langgraph.checkpoint.memory import InMemorySaver
|
||||
|
||||
yield InMemorySaver()
|
||||
return
|
||||
|
||||
if db_config.backend == "sqlite":
|
||||
try:
|
||||
from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver
|
||||
except ImportError as exc:
|
||||
raise ImportError(SQLITE_INSTALL) from exc
|
||||
|
||||
conn_str = db_config.checkpointer_sqlite_path
|
||||
ensure_sqlite_parent_dir(conn_str)
|
||||
async with AsyncSqliteSaver.from_conn_string(conn_str) as saver:
|
||||
await saver.setup()
|
||||
yield saver
|
||||
return
|
||||
|
||||
if db_config.backend == "postgres":
|
||||
if not db_config.postgres_url:
|
||||
raise ValueError("database.postgres_url is required for the postgres backend")
|
||||
|
||||
# 连接池 + 健康检查(见 _pg_pool_saver),避免单连接空闲被关。
|
||||
async with _pg_pool_saver(db_config.postgres_url) as saver:
|
||||
yield saver
|
||||
return
|
||||
|
||||
raise ValueError(f"Unknown database backend: {db_config.backend!r}")
|
||||
|
||||
|
||||
@contextlib.asynccontextmanager
|
||||
async def make_checkpointer(app_config: AppConfig | None = None) -> AsyncIterator[Checkpointer]:
|
||||
"""Async context manager that yields a checkpointer for the caller's lifetime.
|
||||
Resources are opened on enter and closed on exit -- no global state::
|
||||
|
||||
async with make_checkpointer(app_config) as checkpointer:
|
||||
app.state.checkpointer = checkpointer
|
||||
|
||||
Yields an ``InMemorySaver`` when no checkpointer is configured in *config.yaml*.
|
||||
|
||||
Priority:
|
||||
1. Legacy ``checkpointer:`` config section (backward compatible)
|
||||
2. Unified ``database:`` config section
|
||||
3. Default InMemorySaver
|
||||
"""
|
||||
|
||||
if app_config is None:
|
||||
app_config = get_app_config()
|
||||
|
||||
# Legacy: standalone checkpointer config takes precedence
|
||||
if app_config.checkpointer is not None:
|
||||
async with _async_checkpointer(app_config.checkpointer) as saver:
|
||||
yield saver
|
||||
return
|
||||
|
||||
# Unified database config
|
||||
db_config = getattr(app_config, "database", None)
|
||||
if db_config is not None and db_config.backend != "memory":
|
||||
async with _async_checkpointer_from_database(db_config) as saver:
|
||||
yield saver
|
||||
return
|
||||
|
||||
# Default: in-memory
|
||||
from langgraph.checkpoint.memory import InMemorySaver
|
||||
|
||||
yield InMemorySaver()
|
||||
"""Async checkpointer factory.
|
||||
|
||||
Provides an **async context manager** for long-running async servers that need
|
||||
proper resource cleanup.
|
||||
|
||||
Supported backends: memory, sqlite, postgres.
|
||||
|
||||
Usage (e.g. FastAPI lifespan)::
|
||||
|
||||
from deerflow.runtime.checkpointer.async_provider import make_checkpointer
|
||||
|
||||
async with make_checkpointer() as checkpointer:
|
||||
app.state.checkpointer = checkpointer # InMemorySaver if not configured
|
||||
|
||||
For sync usage see :mod:`deerflow.runtime.checkpointer.provider`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import logging
|
||||
from collections.abc import AsyncIterator
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from langgraph.types import Checkpointer
|
||||
|
||||
from deerflow.config.app_config import AppConfig, get_app_config
|
||||
from deerflow.runtime.checkpointer.provider import (
|
||||
POSTGRES_CONN_REQUIRED,
|
||||
POSTGRES_INSTALL,
|
||||
SQLITE_INSTALL,
|
||||
)
|
||||
from deerflow.runtime.store._sqlite_utils import ensure_sqlite_parent_dir, resolve_sqlite_conn_str
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _csv_env(name: str) -> list[str]:
|
||||
raw = os.getenv(name, "")
|
||||
return [part.strip() for part in raw.split(",") if part.strip()]
|
||||
|
||||
|
||||
def _dual_read_enabled(config) -> bool:
|
||||
if getattr(config, "dual_read_enabled", False):
|
||||
return True
|
||||
return os.getenv("DEER_FLOW_CHECKPOINT_DUAL_READ", "").strip().lower() in {"1", "true", "yes", "on"}
|
||||
|
||||
|
||||
def _use_postgres_enabled(config) -> bool:
|
||||
raw = os.getenv("DEER_FLOW_USE_PG_CHECKPOINTER")
|
||||
if raw:
|
||||
return raw.strip().lower() not in {"0", "false", "no", "off"}
|
||||
return bool(getattr(config, "use_postgres", True))
|
||||
|
||||
|
||||
def _dual_read_paths(config) -> list[str]:
|
||||
paths = list(getattr(config, "dual_read_sqlite_paths", []) or [])
|
||||
paths.extend(_csv_env("DEER_FLOW_CHECKPOINT_SQLITE_PATHS"))
|
||||
resolved: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for path in paths:
|
||||
conn_str = resolve_sqlite_conn_str(path)
|
||||
if conn_str == ":memory:" or conn_str.startswith("file:"):
|
||||
logger.warning("Skipping unsupported checkpoint migration SQLite path: %s", path)
|
||||
continue
|
||||
if conn_str not in seen:
|
||||
resolved.append(conn_str)
|
||||
seen.add(conn_str)
|
||||
return resolved
|
||||
|
||||
|
||||
def _sqlite_conn_str_for_postgres_fallback(config) -> str:
|
||||
configured = getattr(config, "sqlite_connection_string", None)
|
||||
if configured:
|
||||
return configured
|
||||
paths = _dual_read_paths(config)
|
||||
return paths[0] if paths else "store.db"
|
||||
|
||||
|
||||
def _pool_size_env(name: str, default: int) -> int:
|
||||
raw = os.getenv(name, "").strip()
|
||||
if not raw:
|
||||
return default
|
||||
try:
|
||||
return max(0, int(raw))
|
||||
except ValueError:
|
||||
logger.warning("Invalid %s=%r; using default %d", name, raw, default)
|
||||
return default
|
||||
|
||||
|
||||
@contextlib.asynccontextmanager
|
||||
async def _postgres_saver_from_pool(conn_string: str) -> AsyncIterator[Checkpointer]:
|
||||
"""Yield an ``AsyncPostgresSaver`` backed by a self-healing connection pool.
|
||||
|
||||
``AsyncPostgresSaver.from_conn_string`` opens a *single* long-lived
|
||||
connection. If that one connection dies (Postgres restart, idle timeout,
|
||||
network blip, or a prior broken pipeline), every later checkpointer call
|
||||
fails permanently with "the connection is closed" until the process is
|
||||
restarted. A pool with ``check=check_connection`` validates each connection
|
||||
on checkout and transparently replaces dead ones, so the runtime recovers
|
||||
on its own.
|
||||
"""
|
||||
from langgraph.checkpoint.postgres.aio import AsyncPostgresSaver
|
||||
from psycopg.rows import dict_row
|
||||
from psycopg_pool import AsyncConnectionPool
|
||||
|
||||
pool = AsyncConnectionPool(
|
||||
conninfo=conn_string,
|
||||
min_size=_pool_size_env("DEER_FLOW_PG_POOL_MIN", 1),
|
||||
max_size=_pool_size_env("DEER_FLOW_PG_POOL_MAX", 10),
|
||||
max_idle=300.0,
|
||||
max_lifetime=3600.0,
|
||||
check=AsyncConnectionPool.check_connection,
|
||||
kwargs={"autocommit": True, "prepare_threshold": 0, "row_factory": dict_row},
|
||||
open=False,
|
||||
)
|
||||
await pool.open(wait=True)
|
||||
try:
|
||||
saver = AsyncPostgresSaver(pool)
|
||||
await saver.setup()
|
||||
yield saver
|
||||
finally:
|
||||
await pool.close()
|
||||
|
||||
|
||||
@contextlib.asynccontextmanager
|
||||
async def _sqlite_checkpointer_from_conn_str(raw: str) -> AsyncIterator[Checkpointer]:
|
||||
try:
|
||||
from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver
|
||||
except ImportError as exc:
|
||||
raise ImportError(SQLITE_INSTALL) from exc
|
||||
|
||||
conn_str = resolve_sqlite_conn_str(raw)
|
||||
await asyncio.to_thread(ensure_sqlite_parent_dir, conn_str)
|
||||
async with AsyncSqliteSaver.from_conn_string(conn_str) as saver:
|
||||
await saver.setup()
|
||||
yield saver
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Async factory
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@contextlib.asynccontextmanager
|
||||
async def _async_checkpointer(config) -> AsyncIterator[Checkpointer]:
|
||||
"""Async context manager that constructs and tears down a checkpointer."""
|
||||
if config.type == "memory":
|
||||
from langgraph.checkpoint.memory import InMemorySaver
|
||||
|
||||
yield InMemorySaver()
|
||||
return
|
||||
|
||||
if config.type == "sqlite":
|
||||
async with _sqlite_checkpointer_from_conn_str(config.connection_string or "store.db") as saver:
|
||||
yield saver
|
||||
return
|
||||
|
||||
if config.type == "postgres":
|
||||
if not _use_postgres_enabled(config):
|
||||
conn_str = _sqlite_conn_str_for_postgres_fallback(config)
|
||||
logger.info("Checkpointer: use_postgres=false; using SQLite fallback (%s)", conn_str)
|
||||
async with _sqlite_checkpointer_from_conn_str(conn_str) as saver:
|
||||
yield saver
|
||||
return
|
||||
|
||||
try:
|
||||
from langgraph.checkpoint.postgres.aio import AsyncPostgresSaver # noqa: F401
|
||||
except ImportError as exc:
|
||||
raise ImportError(POSTGRES_INSTALL) from exc
|
||||
|
||||
if not config.connection_string:
|
||||
raise ValueError(POSTGRES_CONN_REQUIRED)
|
||||
|
||||
async with _postgres_saver_from_pool(config.connection_string) as saver:
|
||||
if not _dual_read_enabled(config):
|
||||
yield saver
|
||||
return
|
||||
|
||||
from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver
|
||||
|
||||
from deerflow.runtime.checkpointer.checkpoint_migration import (
|
||||
DualReadCheckpointer,
|
||||
SQLiteCheckpointSource,
|
||||
)
|
||||
|
||||
sources: list[SQLiteCheckpointSource] = []
|
||||
for sqlite_path in _dual_read_paths(config):
|
||||
if not Path(sqlite_path).exists():
|
||||
logger.info("Checkpoint dual-read source does not exist, skipping: %s", sqlite_path)
|
||||
continue
|
||||
ctx = AsyncSqliteSaver.from_conn_string(sqlite_path)
|
||||
source_saver = await ctx.__aenter__()
|
||||
await source_saver.setup()
|
||||
sources.append(
|
||||
SQLiteCheckpointSource(
|
||||
path=sqlite_path,
|
||||
saver=source_saver,
|
||||
context=ctx,
|
||||
progress_path=Path(sqlite_path).with_suffix(Path(sqlite_path).suffix + ".pg-migration.json"),
|
||||
)
|
||||
)
|
||||
|
||||
if not sources:
|
||||
logger.info("Checkpoint dual-read enabled but no SQLite sources were opened; using Postgres only")
|
||||
yield saver
|
||||
return
|
||||
|
||||
proxy = DualReadCheckpointer(
|
||||
saver,
|
||||
sources,
|
||||
)
|
||||
try:
|
||||
yield proxy
|
||||
finally:
|
||||
await proxy.aclose_sources()
|
||||
return
|
||||
|
||||
raise ValueError(f"Unknown checkpointer type: {config.type!r}")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Public async context manager
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@contextlib.asynccontextmanager
|
||||
async def _async_checkpointer_from_database(db_config) -> AsyncIterator[Checkpointer]:
|
||||
"""Async context manager that constructs a checkpointer from unified DatabaseConfig."""
|
||||
if db_config.backend == "memory":
|
||||
from langgraph.checkpoint.memory import InMemorySaver
|
||||
|
||||
yield InMemorySaver()
|
||||
return
|
||||
|
||||
if db_config.backend == "sqlite":
|
||||
try:
|
||||
from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver
|
||||
except ImportError as exc:
|
||||
raise ImportError(SQLITE_INSTALL) from exc
|
||||
|
||||
conn_str = db_config.checkpointer_sqlite_path
|
||||
ensure_sqlite_parent_dir(conn_str)
|
||||
async with AsyncSqliteSaver.from_conn_string(conn_str) as saver:
|
||||
await saver.setup()
|
||||
yield saver
|
||||
return
|
||||
|
||||
if db_config.backend == "postgres":
|
||||
try:
|
||||
from langgraph.checkpoint.postgres.aio import AsyncPostgresSaver # noqa: F401
|
||||
except ImportError as exc:
|
||||
raise ImportError(POSTGRES_INSTALL) from exc
|
||||
|
||||
if not db_config.postgres_url:
|
||||
raise ValueError("database.postgres_url is required for the postgres backend")
|
||||
|
||||
async with _postgres_saver_from_pool(db_config.postgres_url) as saver:
|
||||
yield saver
|
||||
return
|
||||
|
||||
raise ValueError(f"Unknown database backend: {db_config.backend!r}")
|
||||
|
||||
|
||||
@contextlib.asynccontextmanager
|
||||
async def make_checkpointer(app_config: AppConfig | None = None) -> AsyncIterator[Checkpointer]:
|
||||
"""Async context manager that yields a checkpointer for the caller's lifetime.
|
||||
Resources are opened on enter and closed on exit -- no global state::
|
||||
|
||||
async with make_checkpointer(app_config) as checkpointer:
|
||||
app.state.checkpointer = checkpointer
|
||||
|
||||
Yields an ``InMemorySaver`` when no checkpointer is configured in *config.yaml*.
|
||||
|
||||
Priority:
|
||||
1. Legacy ``checkpointer:`` config section (backward compatible)
|
||||
2. Unified ``database:`` config section
|
||||
3. Default InMemorySaver
|
||||
"""
|
||||
|
||||
if app_config is None:
|
||||
app_config = get_app_config()
|
||||
|
||||
# Legacy: standalone checkpointer config takes precedence
|
||||
if app_config.checkpointer is not None:
|
||||
async with _async_checkpointer(app_config.checkpointer) as saver:
|
||||
yield saver
|
||||
return
|
||||
|
||||
# Unified database config
|
||||
db_config = getattr(app_config, "database", None)
|
||||
if db_config is not None and db_config.backend != "memory":
|
||||
async with _async_checkpointer_from_database(db_config) as saver:
|
||||
yield saver
|
||||
return
|
||||
|
||||
# Default: in-memory
|
||||
from langgraph.checkpoint.memory import InMemorySaver
|
||||
|
||||
yield InMemorySaver()
|
||||
+38
-2
@@ -28,6 +28,42 @@ from deerflow.runtime.store.provider import POSTGRES_CONN_REQUIRED, POSTGRES_STO
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _pool_size_env(name: str, default: int) -> int:
|
||||
import os
|
||||
|
||||
raw = os.getenv(name, "").strip()
|
||||
if not raw:
|
||||
return default
|
||||
try:
|
||||
return max(0, int(raw))
|
||||
except ValueError:
|
||||
logger.warning("Invalid %s=%r; using default %d", name, raw, default)
|
||||
return default
|
||||
|
||||
|
||||
def _postgres_pool_config() -> dict:
|
||||
"""Pool config for the Postgres store.
|
||||
|
||||
A single long-lived connection (the default ``from_conn_string`` behavior)
|
||||
dies permanently on a server restart / idle timeout / network blip — every
|
||||
later store call then fails with "server closed the connection
|
||||
unexpectedly". A pool with ``check=check_connection`` validates each
|
||||
connection on checkout and replaces dead ones, so the runtime self-heals.
|
||||
Extra keys (``check``/``max_idle``/``max_lifetime``) are passed straight
|
||||
through to ``AsyncConnectionPool`` by ``from_conn_string``.
|
||||
"""
|
||||
from psycopg_pool import AsyncConnectionPool
|
||||
|
||||
return {
|
||||
"min_size": _pool_size_env("DEER_FLOW_PG_POOL_MIN", 1),
|
||||
"max_size": _pool_size_env("DEER_FLOW_PG_POOL_MAX", 10),
|
||||
"check": AsyncConnectionPool.check_connection,
|
||||
"max_idle": 300.0,
|
||||
"max_lifetime": 3600.0,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Internal backend factory
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -71,9 +107,9 @@ async def _async_store(config) -> AsyncIterator[BaseStore]:
|
||||
if not config.connection_string:
|
||||
raise ValueError(POSTGRES_CONN_REQUIRED)
|
||||
|
||||
async with AsyncPostgresStore.from_conn_string(config.connection_string) as store:
|
||||
async with AsyncPostgresStore.from_conn_string(config.connection_string, pool_config=_postgres_pool_config()) as store:
|
||||
await store.setup()
|
||||
logger.info("Store: using AsyncPostgresStore")
|
||||
logger.info("Store: using AsyncPostgresStore (pooled, self-healing)")
|
||||
yield store
|
||||
return
|
||||
|
||||
|
||||
+225
@@ -0,0 +1,225 @@
|
||||
"""Interactive browser action tool backed by the user's Chrome extension.
|
||||
|
||||
Unlike ``browser_fetch_page`` (a one-shot read), this tool drives an
|
||||
*observe -> act* loop on a live page the extension keeps open:
|
||||
|
||||
1. The model calls ``browser_act`` with an action (navigate/click/type/...).
|
||||
2. The extension performs the action on the session tab, then re-scans the page
|
||||
and returns an *observation*: the final URL/title, an indexed list of
|
||||
interactive elements, and the visible page text.
|
||||
3. The model reads the observation, picks the next element by its index, and
|
||||
calls ``browser_act`` again with the same ``session_id``.
|
||||
|
||||
The element ``index`` values are only valid for the most recent observation of a
|
||||
session; always act on indices from the latest result.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
from urllib.parse import urlparse
|
||||
from uuid import uuid4
|
||||
|
||||
from langchain.tools import tool
|
||||
|
||||
from deerflow.browser_context.tasks import create_browser_task, wait_for_browser_task_result
|
||||
from deerflow.config import get_app_config
|
||||
from deerflow.config.browser_context_config import BrowserContextConfig
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
|
||||
_ACTIONS = {"navigate", "click", "type", "scroll", "wait", "extract", "observe", "close"}
|
||||
_NEEDS_INDEX = {"click", "type"}
|
||||
|
||||
|
||||
def _settings() -> BrowserContextConfig:
|
||||
value = getattr(get_app_config(), "browser_context", None)
|
||||
if isinstance(value, BrowserContextConfig):
|
||||
return value
|
||||
if isinstance(value, dict):
|
||||
return BrowserContextConfig.model_validate(value)
|
||||
return BrowserContextConfig()
|
||||
|
||||
|
||||
def _validate_url(raw_url: str) -> str | None:
|
||||
url = (raw_url or "").strip()
|
||||
parsed = urlparse(url)
|
||||
if parsed.scheme not in {"http", "https"} or not parsed.netloc:
|
||||
return None
|
||||
return url
|
||||
|
||||
|
||||
def _format_elements(elements: list[dict[str, Any]] | None) -> str:
|
||||
if not elements:
|
||||
return "(no interactive elements detected)"
|
||||
lines: list[str] = []
|
||||
for el in elements:
|
||||
idx = el.get("index")
|
||||
tag = str(el.get("tag") or "?")
|
||||
el_type = str(el.get("type") or "").strip()
|
||||
label = str(el.get("label") or "").strip().replace("\n", " ")
|
||||
if len(label) > 120:
|
||||
label = label[:120] + "…"
|
||||
descriptor = f"{tag}" + (f"[{el_type}]" if el_type else "")
|
||||
extra = ""
|
||||
if el.get("value"):
|
||||
value = str(el["value"]).strip().replace("\n", " ")
|
||||
extra = f' value="{value[:60]}"'
|
||||
lines.append(f"[{idx}] {descriptor} \"{label}\"{extra}".rstrip())
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _format_observation(action: str, observation: dict[str, Any] | None, session_id: str, settings: BrowserContextConfig) -> str:
|
||||
observation = observation or {}
|
||||
status = str(observation.get("action_status") or "ok")
|
||||
message = str(observation.get("message") or "").strip()
|
||||
url = str(observation.get("url") or "").strip() or "(unknown)"
|
||||
title = str(observation.get("title") or "").strip() or "(untitled)"
|
||||
elements = observation.get("elements") if isinstance(observation.get("elements"), list) else []
|
||||
text = str(observation.get("text") or "").strip()
|
||||
text_limit = max(500, int(settings.observation_text_max_chars))
|
||||
text_truncated = len(text) > text_limit
|
||||
if text_truncated:
|
||||
text = text[:text_limit]
|
||||
|
||||
parts = [
|
||||
f"Browser session: {session_id}",
|
||||
f"Last action: {action} -> {status}",
|
||||
]
|
||||
if message:
|
||||
parts.append(f"Note: {message}")
|
||||
parts.extend(
|
||||
[
|
||||
f"URL: {url}",
|
||||
f"Title: {title}",
|
||||
"",
|
||||
"Interactive elements (use the [index] with click/type):",
|
||||
_format_elements(elements),
|
||||
"",
|
||||
"Page text:",
|
||||
text or "(empty page text)",
|
||||
]
|
||||
)
|
||||
if text_truncated:
|
||||
parts.append("\n[Page text truncated by browser_context.observation_text_max_chars]")
|
||||
parts.append(
|
||||
"\nNext step: call browser_act again with this session_id and the index of the element to act on. "
|
||||
"When the task is finished, call browser_act with action=\"close\" to release the tab."
|
||||
)
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
@tool("browser_act", parse_docstring=True)
|
||||
async def browser_act_tool(
|
||||
action: str,
|
||||
session_id: str = "",
|
||||
url: str = "",
|
||||
index: int = -1,
|
||||
text: str = "",
|
||||
submit: bool = False,
|
||||
wait_for: str = "",
|
||||
wait_seconds: int = 0,
|
||||
) -> str:
|
||||
"""Drive an interactive observe/act loop on a page in the user's Chrome browser.
|
||||
|
||||
Use this to operate a real web page on the user's behalf — log in, fill forms,
|
||||
click through navigation, and read pages that need the user's session or local
|
||||
network. Start every task with ``action="navigate"`` and a URL to open a tab
|
||||
and get the first observation, then keep calling this tool with the same
|
||||
``session_id``, choosing elements by the ``index`` shown in the observation.
|
||||
Call ``action="close"`` when done to free the tab.
|
||||
|
||||
Args:
|
||||
action: One of navigate, click, type, scroll, wait, extract, observe, close.
|
||||
session_id: Session id from a previous observation. Leave empty to start a new session (requires navigate + url).
|
||||
url: Target http(s) URL. Required for navigate (and to start a new session).
|
||||
index: Index of the interactive element to act on, taken from the latest observation. Required for click and type.
|
||||
text: Text to enter for the type action.
|
||||
submit: For type, press Enter / submit the form after typing.
|
||||
wait_for: For wait, a CSS selector to wait for before observing.
|
||||
wait_seconds: Optional extra seconds to wait for this step's result. Zero uses the configured default.
|
||||
"""
|
||||
|
||||
settings = _settings()
|
||||
if not settings.enabled:
|
||||
return "browser_act is disabled by config.yaml: browser_context.enabled is false."
|
||||
if not settings.actions_enabled:
|
||||
return "browser_act is disabled by config.yaml: browser_context.actions_enabled is false."
|
||||
|
||||
action = (action or "").strip().lower()
|
||||
if action not in _ACTIONS:
|
||||
return f"browser_act: unknown action {action!r}. Use one of: {', '.join(sorted(_ACTIONS))}."
|
||||
|
||||
session_id = (session_id or "").strip()
|
||||
safe_url = _validate_url(url) if url else None
|
||||
|
||||
if action == "navigate" and safe_url is None:
|
||||
return "browser_act navigate requires an exact http(s) URL, for example https://example.com/page."
|
||||
|
||||
if not session_id:
|
||||
if action != "navigate":
|
||||
return "browser_act: no session_id yet. Start with action=\"navigate\" and a url to open the page first."
|
||||
session_id = uuid4().hex
|
||||
|
||||
if action in _NEEDS_INDEX and index < 0:
|
||||
return f"browser_act {action} requires a non-negative element index from the latest observation."
|
||||
if action == "type" and not text:
|
||||
return "browser_act type requires the text to enter."
|
||||
|
||||
action_payload: dict[str, Any] = {"type": action}
|
||||
if safe_url is not None:
|
||||
action_payload["url"] = safe_url
|
||||
if index >= 0:
|
||||
action_payload["index"] = int(index)
|
||||
if text:
|
||||
action_payload["text"] = text
|
||||
if submit:
|
||||
action_payload["submit"] = True
|
||||
if wait_for.strip():
|
||||
action_payload["wait_for"] = wait_for.strip()[:300]
|
||||
|
||||
configured_timeout = max(5, int(settings.action_result_timeout_seconds))
|
||||
requested_timeout = int(wait_seconds or configured_timeout)
|
||||
timeout = min(max(5, requested_timeout), configured_timeout)
|
||||
user_id = get_effective_user_id()
|
||||
|
||||
task = create_browser_task(
|
||||
user_id=user_id,
|
||||
url=safe_url or "",
|
||||
instruction="",
|
||||
kind="act",
|
||||
session_id=session_id,
|
||||
action=action_payload,
|
||||
timeout_seconds=timeout + int(settings.task_lease_seconds),
|
||||
max_content_chars=settings.max_content_chars,
|
||||
metadata={"source": "browser_act_tool"},
|
||||
)
|
||||
|
||||
completed = await wait_for_browser_task_result(
|
||||
task_id=task["id"],
|
||||
user_id=user_id,
|
||||
timeout_seconds=timeout,
|
||||
poll_interval_seconds=settings.task_poll_interval_seconds,
|
||||
)
|
||||
if completed is None:
|
||||
return f"Browser task {task['id']} was not found after it was created."
|
||||
|
||||
status = completed.get("status")
|
||||
if status == "error":
|
||||
return f"browser_act session {session_id}: step failed: {completed.get('error') or 'unknown error'}"
|
||||
if status == "expired":
|
||||
return (
|
||||
f"browser_act session {session_id}: the Chrome extension did not respond in time. "
|
||||
"Check that Deer Web Clipper is installed, the same user is logged in, and task polling is enabled."
|
||||
)
|
||||
if status != "done":
|
||||
return (
|
||||
f"browser_act session {session_id}: step is still {status or 'pending'} after {timeout} seconds. "
|
||||
"Keep the Deer Web Clipper extension running with task polling enabled, then try again."
|
||||
)
|
||||
|
||||
if action == "close":
|
||||
return f"browser_act session {session_id}: tab closed."
|
||||
|
||||
result = completed.get("result") or {}
|
||||
observation = result.get("observation") if isinstance(result.get("observation"), dict) else None
|
||||
return _format_observation(action, observation, session_id, settings)
|
||||
+117
@@ -0,0 +1,117 @@
|
||||
"""Browser extension backed page fetch tool."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from langchain.tools import tool
|
||||
|
||||
from deerflow.browser_context.tasks import create_browser_task, wait_for_browser_task_result
|
||||
from deerflow.config import get_app_config
|
||||
from deerflow.config.browser_context_config import BrowserContextConfig
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
|
||||
|
||||
def _settings() -> BrowserContextConfig:
|
||||
value = getattr(get_app_config(), "browser_context", None)
|
||||
if isinstance(value, BrowserContextConfig):
|
||||
return value
|
||||
if isinstance(value, dict):
|
||||
return BrowserContextConfig.model_validate(value)
|
||||
return BrowserContextConfig()
|
||||
|
||||
|
||||
def _validate_url(raw_url: str) -> str | None:
|
||||
url = (raw_url or "").strip()
|
||||
parsed = urlparse(url)
|
||||
if parsed.scheme not in {"http", "https"} or not parsed.netloc:
|
||||
return None
|
||||
return url
|
||||
|
||||
|
||||
@tool("browser_fetch_page", parse_docstring=True)
|
||||
async def browser_fetch_page_tool(url: str, question: str = "", wait_seconds: int = 0) -> str:
|
||||
"""Ask the user's paired Chrome extension to fetch a page and return visible page text.
|
||||
|
||||
Use this when the user explicitly asks to use the browser extension or asks
|
||||
for data from a page that may require their browser session, login state, or
|
||||
local network access. The URL must be an exact http(s) URL provided by the
|
||||
user. If the user did not provide a URL, ask for the URL instead of guessing.
|
||||
|
||||
Args:
|
||||
url: Exact http(s) URL to open in the user's Chrome browser.
|
||||
question: Optional instruction describing what data the user wants from the page.
|
||||
wait_seconds: Optional seconds to wait for the extension result. Zero uses the configured default.
|
||||
"""
|
||||
|
||||
settings = _settings()
|
||||
if not settings.enabled:
|
||||
return "browser_fetch_page is disabled by config.yaml: browser_context.enabled is false."
|
||||
|
||||
safe_url = _validate_url(url)
|
||||
if safe_url is None:
|
||||
return "browser_fetch_page requires an exact http(s) URL, for example https://example.com/page."
|
||||
|
||||
configured_timeout = max(5, int(settings.task_result_timeout_seconds))
|
||||
requested_timeout = int(wait_seconds or configured_timeout)
|
||||
timeout = min(max(5, requested_timeout), configured_timeout)
|
||||
user_id = get_effective_user_id()
|
||||
|
||||
task = create_browser_task(
|
||||
user_id=user_id,
|
||||
url=safe_url,
|
||||
instruction=(question or "").strip(),
|
||||
timeout_seconds=timeout + int(settings.task_lease_seconds),
|
||||
max_content_chars=settings.max_content_chars,
|
||||
metadata={"source": "browser_fetch_page_tool"},
|
||||
)
|
||||
|
||||
completed = await wait_for_browser_task_result(
|
||||
task_id=task["id"],
|
||||
user_id=user_id,
|
||||
timeout_seconds=timeout,
|
||||
poll_interval_seconds=settings.task_poll_interval_seconds,
|
||||
)
|
||||
if completed is None:
|
||||
return f"Browser task {task['id']} was not found after it was created."
|
||||
|
||||
status = completed.get("status")
|
||||
if status == "error":
|
||||
return f"Browser task {task['id']} failed: {completed.get('error') or 'unknown error'}"
|
||||
if status == "expired":
|
||||
return (
|
||||
f"Browser task {task['id']} expired before the Chrome extension returned a result. "
|
||||
"Check that Deer Web Clipper is installed, configured with the same access token, and task polling is enabled."
|
||||
)
|
||||
if status != "done":
|
||||
return (
|
||||
f"Browser task {task['id']} is still {status or 'pending'} after {timeout} seconds. "
|
||||
"Keep the Deer Web Clipper extension running with task polling enabled, then try again."
|
||||
)
|
||||
|
||||
result = completed.get("result") or {}
|
||||
content = str(result.get("content") or "").strip()
|
||||
tool_limit = max(1_000, int(settings.tool_result_max_chars))
|
||||
tool_truncated = len(content) > tool_limit
|
||||
if tool_truncated:
|
||||
content = content[:tool_limit]
|
||||
|
||||
notes: list[str] = []
|
||||
if result.get("truncated"):
|
||||
notes.append("The browser capture was truncated by browser_context.max_content_chars.")
|
||||
if tool_truncated:
|
||||
notes.append("The tool response was truncated by browser_context.tool_result_max_chars.")
|
||||
|
||||
parts = [
|
||||
"Browser page fetched through the user's Chrome extension.",
|
||||
f"Task ID: {task['id']}",
|
||||
f"Title: {result.get('title') or '(untitled)'}",
|
||||
f"URL: {result.get('url') or safe_url}",
|
||||
]
|
||||
if question:
|
||||
parts.extend(["", "Requested extraction:", question.strip()])
|
||||
if notes:
|
||||
parts.extend(["", "Notes:", "\n".join(f"- {note}" for note in notes)])
|
||||
parts.extend(["", "Page content:", content or "(empty page text)"])
|
||||
return "\n".join(parts)
|
||||
|
||||
@@ -11,6 +11,11 @@ from deerflow.tools.builtins.tool_search import reset_deferred_registry
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
BROWSER_CONTEXT_TOOL_USE = "deerflow.tools.builtins.browser_context_tool:browser_fetch_page_tool"
|
||||
BROWSER_ACT_TOOL_USE = "deerflow.tools.builtins.browser_act_tool:browser_act_tool"
|
||||
BROWSER_CONTEXT_TOOL_USES = {BROWSER_CONTEXT_TOOL_USE, BROWSER_ACT_TOOL_USE}
|
||||
BROWSER_CONTEXT_TOOL_NAMES = {"browser_fetch_page", "browser_act"}
|
||||
|
||||
BUILTIN_TOOLS = [
|
||||
present_file_tool,
|
||||
ask_clarification_tool,
|
||||
@@ -34,6 +39,14 @@ def _is_host_bash_tool(tool: object) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def _is_browser_context_tool(tool: object) -> bool:
|
||||
"""Return True if the tool config represents the browser extension bridge."""
|
||||
|
||||
use = getattr(tool, "use", None)
|
||||
name = getattr(tool, "name", None)
|
||||
return use in BROWSER_CONTEXT_TOOL_USES or name in BROWSER_CONTEXT_TOOL_NAMES
|
||||
|
||||
|
||||
def get_available_tools(
|
||||
groups: list[str] | None = None,
|
||||
include_mcp: bool = True,
|
||||
@@ -64,6 +77,18 @@ def get_available_tools(
|
||||
config = app_config or get_app_config()
|
||||
tool_configs = [tool for tool in config.tools if groups is None or tool.group in groups]
|
||||
|
||||
browser_context_config = getattr(config, "browser_context", None)
|
||||
if not getattr(browser_context_config, "enabled", False):
|
||||
tool_configs = [tool for tool in tool_configs if not _is_browser_context_tool(tool)]
|
||||
elif not getattr(browser_context_config, "actions_enabled", True):
|
||||
# browser_context is on, but interactive page actions are disabled — keep
|
||||
# the read-only browser_fetch_page tool, drop only browser_act.
|
||||
tool_configs = [
|
||||
tool
|
||||
for tool in tool_configs
|
||||
if getattr(tool, "use", None) != BROWSER_ACT_TOOL_USE and getattr(tool, "name", None) != "browser_act"
|
||||
]
|
||||
|
||||
# Do not expose host bash by default when LocalSandboxProvider is active.
|
||||
if not is_host_bash_allowed(config):
|
||||
tool_configs = [tool for tool in tool_configs if not _is_host_bash_tool(tool)]
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
-- ============================================================================
|
||||
-- 给【已存在的表】补列(ADD COLUMN)—— MySQL 手动执行版
|
||||
-- ============================================================================
|
||||
--
|
||||
-- 背景:
|
||||
-- 本项目的 alembic 自动迁移是【关闭】的(engine.py 中
|
||||
-- `# await _run_alembic_upgrade(url)` 被注释)。`Base.metadata.create_all`
|
||||
-- 只会建【新表】,对【已存在的表】既不会补建索引、也不会补加新列。
|
||||
-- 因此当我们在某个 ORM model 上新增了一列时,【全新部署】会随建表一并带上,
|
||||
-- 但【已有数据的 MySQL 部署】上这一列并不会自动出现,代码读写该列会报错。
|
||||
--
|
||||
-- 约定(重要):
|
||||
-- 今后只要有代码给某张【已存在的表】新增字段(在 ORM model 上加列),就必须
|
||||
-- 把对应的 `ADD COLUMN` 语句以下面的【幂等】写法补进本文件,并在迁移目录里
|
||||
-- 保留对应的 alembic 版本文件,二者保持一致。运维在升级老库时整文件执行本脚本
|
||||
-- 即可补齐所有缺失的列。
|
||||
--
|
||||
-- 用法:
|
||||
-- 1) 先选中目标库(与 database.mysql_url 里的库名一致):
|
||||
-- USE your_deerflow_db;
|
||||
-- 2) 直接整文件执行即可。本脚本是【幂等】的:已存在的列会跳过,不会报错,
|
||||
-- 可重复执行。
|
||||
--
|
||||
-- 注意:
|
||||
-- * 列定义必须与 ORM model / alembic migration 完全一致(类型、长度、可空性)。
|
||||
-- * 新增列一律建议 NULL 或带 DEFAULT,避免在大表上回填导致长时间锁表。
|
||||
-- * MySQL 8.0 在线 DDL 默认 ALGORITHM=INPLACE 一般不阻塞写入,但仍建议低峰执行。
|
||||
-- ============================================================================
|
||||
|
||||
-- ---- 通用幂等封装:仅当列不存在时才新增 --------------------------------------
|
||||
DELIMITER //
|
||||
DROP PROCEDURE IF EXISTS _add_column_if_missing //
|
||||
CREATE PROCEDURE _add_column_if_missing(
|
||||
IN p_table VARCHAR(64),
|
||||
IN p_column VARCHAR(64),
|
||||
IN p_definition VARCHAR(255) -- 列定义,如 'VARCHAR(128) NULL DEFAULT NULL'
|
||||
)
|
||||
BEGIN
|
||||
DECLARE v_exists INT DEFAULT 0;
|
||||
SELECT COUNT(*) INTO v_exists
|
||||
FROM information_schema.columns
|
||||
WHERE table_schema = DATABASE()
|
||||
AND table_name = p_table
|
||||
AND column_name = p_column;
|
||||
|
||||
IF v_exists = 0 THEN
|
||||
SET @ddl = CONCAT('ALTER TABLE `', p_table, '` ADD COLUMN `', p_column, '` ', p_definition);
|
||||
PREPARE stmt FROM @ddl;
|
||||
EXECUTE stmt;
|
||||
DEALLOCATE PREPARE stmt;
|
||||
SELECT CONCAT('ADDED ', p_table, '.', p_column) AS result;
|
||||
ELSE
|
||||
SELECT CONCAT('SKIPPED ', p_table, '.', p_column, ' (already exists)') AS result;
|
||||
END IF;
|
||||
END //
|
||||
DELIMITER ;
|
||||
|
||||
-- ---- skills ------------------------------------------------------------------
|
||||
-- name_zh: 技能的中文显示名(用户配置或 LLM 生成),对应 migration 20260609_01。
|
||||
-- ORM: SkillRow.name_zh = String(128), nullable=True。
|
||||
CALL _add_column_if_missing('skills', 'name_zh', 'VARCHAR(128) NULL DEFAULT NULL');
|
||||
-- detail: 技能的长文介绍,对应 migration 20260518_01。
|
||||
-- ORM: SkillRow.detail = Text, nullable=True。
|
||||
CALL _add_column_if_missing('skills', 'detail', 'TEXT NULL DEFAULT NULL');
|
||||
|
||||
-- ---- 清理临时存储过程 --------------------------------------------------------
|
||||
DROP PROCEDURE IF EXISTS _add_column_if_missing;
|
||||
|
||||
-- ---- 执行后自检:确认本批次涉及的列都已存在 ---------------------------------
|
||||
SELECT table_name, column_name
|
||||
FROM information_schema.columns
|
||||
WHERE table_schema = DATABASE()
|
||||
AND (table_name = 'skills' AND column_name IN ('name_zh', 'detail'))
|
||||
ORDER BY table_name, column_name;
|
||||
@@ -0,0 +1,158 @@
|
||||
"""Tests for the interactive browser_act tool and its act-task queue support."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
|
||||
import pytest
|
||||
|
||||
from deerflow.browser_context import tasks as task_queue
|
||||
from deerflow.config.browser_context_config import BrowserContextConfig
|
||||
from deerflow.tools.builtins import browser_act_tool as act_mod
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _isolated_task_home(tmp_path, monkeypatch):
|
||||
"""Point the file-backed task queue at a throwaway directory."""
|
||||
monkeypatch.setenv("DEER_FLOW_HOME", str(tmp_path))
|
||||
yield
|
||||
|
||||
|
||||
def _enable_actions(monkeypatch, *, enabled=True, actions_enabled=True):
|
||||
cfg = BrowserContextConfig(
|
||||
enabled=enabled,
|
||||
actions_enabled=actions_enabled,
|
||||
action_result_timeout_seconds=5,
|
||||
task_poll_interval_seconds=0.25,
|
||||
task_lease_seconds=30,
|
||||
)
|
||||
monkeypatch.setattr(act_mod, "_settings", lambda: cfg)
|
||||
return cfg
|
||||
|
||||
|
||||
# --- task queue layer -------------------------------------------------------
|
||||
|
||||
|
||||
def test_act_task_round_trip_carries_session_and_observation():
|
||||
task = task_queue.create_browser_task(
|
||||
user_id="u1",
|
||||
url="https://example.com",
|
||||
kind="act",
|
||||
session_id="sess-1",
|
||||
action={"type": "navigate", "url": "https://example.com"},
|
||||
)
|
||||
assert task["kind"] == "act"
|
||||
assert task["session_id"] == "sess-1"
|
||||
|
||||
claimed = task_queue.claim_next_browser_task(user_id="u1", worker_id="w1")
|
||||
assert claimed is not None
|
||||
assert claimed["kind"] == "act"
|
||||
assert claimed["session_id"] == "sess-1"
|
||||
assert claimed["action"] == {"type": "navigate", "url": "https://example.com"}
|
||||
|
||||
observation = {
|
||||
"action_status": "ok",
|
||||
"url": "https://example.com/home",
|
||||
"title": "Home",
|
||||
"elements": [{"index": 0, "tag": "button", "label": "Login"}],
|
||||
"text": "welcome",
|
||||
}
|
||||
done = task_queue.complete_browser_task(
|
||||
task_id=task["id"],
|
||||
user_id="u1",
|
||||
observation=observation,
|
||||
)
|
||||
assert done is not None
|
||||
assert done["status"] == "done"
|
||||
assert done["result"]["observation"] == observation
|
||||
|
||||
|
||||
def test_fetch_task_default_kind_is_fetch():
|
||||
task = task_queue.create_browser_task(user_id="u1", url="https://example.com")
|
||||
assert task["kind"] == "fetch"
|
||||
claimed = task_queue.claim_next_browser_task(user_id="u1", worker_id="w1")
|
||||
assert claimed["kind"] == "fetch"
|
||||
assert claimed["session_id"] == ""
|
||||
assert claimed["action"] is None
|
||||
|
||||
|
||||
# --- tool validation --------------------------------------------------------
|
||||
|
||||
|
||||
async def _invoke(**kwargs) -> str:
|
||||
return await act_mod.browser_act_tool.ainvoke(kwargs)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_browser_act_disabled_when_actions_off(monkeypatch):
|
||||
_enable_actions(monkeypatch, enabled=True, actions_enabled=False)
|
||||
out = await _invoke(action="navigate", url="https://example.com")
|
||||
assert "actions_enabled is false" in out
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_browser_act_unknown_action(monkeypatch):
|
||||
_enable_actions(monkeypatch)
|
||||
out = await _invoke(action="frobnicate")
|
||||
assert "unknown action" in out
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_browser_act_navigate_requires_url(monkeypatch):
|
||||
_enable_actions(monkeypatch)
|
||||
out = await _invoke(action="navigate")
|
||||
assert "requires an exact http(s) URL" in out
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_browser_act_non_navigate_needs_session(monkeypatch):
|
||||
_enable_actions(monkeypatch)
|
||||
out = await _invoke(action="click", index=0)
|
||||
assert "no session_id yet" in out
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_browser_act_click_requires_index(monkeypatch):
|
||||
_enable_actions(monkeypatch)
|
||||
out = await _invoke(action="click", session_id="sess-1")
|
||||
assert "requires a non-negative element index" in out
|
||||
|
||||
|
||||
# --- tool success path (simulated extension) --------------------------------
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_browser_act_returns_formatted_observation(monkeypatch):
|
||||
_enable_actions(monkeypatch)
|
||||
|
||||
async def fake_extension():
|
||||
# Wait for the tool to create the pending act task, then complete it.
|
||||
for _ in range(40):
|
||||
claimed = task_queue.claim_next_browser_task(user_id="default", worker_id="ext")
|
||||
if claimed is not None:
|
||||
task_queue.complete_browser_task(
|
||||
task_id=claimed["id"],
|
||||
user_id="default",
|
||||
observation={
|
||||
"action_status": "ok",
|
||||
"url": "https://example.com/home",
|
||||
"title": "Home Page",
|
||||
"elements": [
|
||||
{"index": 0, "tag": "input", "type": "search", "label": "Search", "value": ""},
|
||||
{"index": 1, "tag": "button", "label": "Go"},
|
||||
],
|
||||
"text": "Welcome to the home page.",
|
||||
},
|
||||
)
|
||||
return
|
||||
await asyncio.sleep(0.1)
|
||||
|
||||
ext = asyncio.create_task(fake_extension())
|
||||
out = await _invoke(action="navigate", url="https://example.com")
|
||||
await ext
|
||||
|
||||
assert "Browser session:" in out
|
||||
assert "Home Page" in out
|
||||
assert "[0] input[search]" in out
|
||||
assert "[1] button" in out
|
||||
assert "Welcome to the home page." in out
|
||||
@@ -1,76 +1,76 @@
|
||||
config_version: 8
|
||||
log_level: info
|
||||
models:
|
||||
- name: deepseek-chat
|
||||
display_name: DeepSeek Chat
|
||||
# - name: deepseek-chat
|
||||
# display_name: DeepSeek Chat
|
||||
# use: langchain_openai:ChatOpenAI
|
||||
# model: deepseek-chat
|
||||
# api_key: sk-9808fb2375534112ac275541aec14b9b
|
||||
# base_url: https://api.deepseek.com
|
||||
# request_timeout: 600.0
|
||||
# max_retries: 2
|
||||
# max_tokens: 4096
|
||||
# temperature: 1.0
|
||||
# supports_vision: false
|
||||
# supports_thinking: true
|
||||
# - name: zai-org/GLM-5-FP8
|
||||
# display_name: zai-org/GLM-5-FP8
|
||||
# use: langchain_openai:ChatOpenAI
|
||||
# model: zai-org/GLM-5-FP8
|
||||
# api_key: modalresearch_gWu46FDpVLYZWPVEA3b_2fq8Jw2UJiKMgEm_Rxmn1SA
|
||||
# base_url: https://api.us-west-2.modal.direct/v1
|
||||
# request_timeout: 600.0
|
||||
# max_retries: 2
|
||||
# max_tokens: 4096
|
||||
# temperature: 1.0
|
||||
# supports_vision: true
|
||||
# supports_thinking: true
|
||||
# - name: deepseek-v4-pro
|
||||
# display_name: DeepSeek V4 Pro
|
||||
# use: langchain_openai:ChatOpenAI
|
||||
# model: deepseek-v4-pro
|
||||
# api_key: sk-e1c22b7931dc446f8b67e57878edd105
|
||||
# base_url: https://api.deepseek.com
|
||||
# request_timeout: 600.0
|
||||
# max_retries: 2
|
||||
# max_tokens: 4096
|
||||
# temperature: 1.0
|
||||
# supports_vision: false
|
||||
# supports_thinking: true
|
||||
# - name: minimax-m2.7
|
||||
# display_name: MiniMax M2.7
|
||||
# use: langchain_openai:ChatOpenAI
|
||||
# model: MiniMax-M2.7
|
||||
# api_key: sk-cp-mrsAGtzDpUl0TpOBPVoiRG_pWF1kDDN9_mDAnONBs6HIPcNUexSKtSbx2QOxfKBvJT6Xw6hDTV6hdkJYFWfNA8UtHQ08TiGJiosdRoW6q1sGb6Cb7D85NHk
|
||||
# base_url: https://api.minimaxi.com
|
||||
# max_tokens: 4096
|
||||
# temperature: 1.0
|
||||
# supports_vision: true
|
||||
# supports_thinking: true
|
||||
# - name: minimax-m2.5
|
||||
# display_name: MiniMax M2.5
|
||||
# use: langchain_openai:ChatOpenAI
|
||||
# model: MiniMax-M2.5
|
||||
# api_key: sk-cp-mrsAGtzDpUl0TpOBPVoiRG_pWF1kDDN9_mDAnONBs6HIPcNUexSKtSbx2QOxfKBvJT6Xw6hDTV6hdkJYFWfNA8UtHQ08TiGJiosdRoW6q1sGb6Cb7D85NHk
|
||||
# base_url: https://api.minimaxi.com
|
||||
# max_tokens: 4096
|
||||
# temperature: 1.0
|
||||
# supports_vision: true
|
||||
# supports_thinking: true
|
||||
# - name: deepseek-ai/DeepSeek-OCR
|
||||
# display_name: deepseek-ai/DeepSeek-OCR
|
||||
# use: langchain_openai:ChatOpenAI
|
||||
# model: deepseek-ai/DeepSeek-OCR
|
||||
# api_key: sk-beshfxfxfrllsczpqfrgjbugfbpzxgqgedbehceytpcvvtyb
|
||||
# base_url: https://api.siliconflow.cn/v1/
|
||||
# max_tokens: 4096
|
||||
# temperature: 1.0
|
||||
# supports_vision: true
|
||||
# supports_thinking: true
|
||||
- name: deepseek-ai/DeepSeek-V4-Flash
|
||||
display_name: deepseek-ai/DeepSeek-V4-Flash
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: deepseek-chat
|
||||
api_key: sk-9808fb2375534112ac275541aec14b9b
|
||||
base_url: https://api.deepseek.com
|
||||
request_timeout: 600.0
|
||||
max_retries: 2
|
||||
max_tokens: 4096
|
||||
temperature: 1.0
|
||||
supports_vision: false
|
||||
supports_thinking: true
|
||||
- name: zai-org/GLM-5-FP8
|
||||
display_name: zai-org/GLM-5-FP8
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: zai-org/GLM-5-FP8
|
||||
api_key: modalresearch_gWu46FDpVLYZWPVEA3b_2fq8Jw2UJiKMgEm_Rxmn1SA
|
||||
base_url: https://api.us-west-2.modal.direct/v1
|
||||
request_timeout: 600.0
|
||||
max_retries: 2
|
||||
max_tokens: 4096
|
||||
temperature: 1.0
|
||||
supports_vision: true
|
||||
supports_thinking: true
|
||||
- name: deepseek-v4-pro
|
||||
display_name: DeepSeek V4 Pro
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: deepseek-v4-pro
|
||||
api_key: sk-e1c22b7931dc446f8b67e57878edd105
|
||||
base_url: https://api.deepseek.com
|
||||
request_timeout: 600.0
|
||||
max_retries: 2
|
||||
max_tokens: 4096
|
||||
temperature: 1.0
|
||||
supports_vision: false
|
||||
supports_thinking: true
|
||||
- name: minimax-m2.7
|
||||
display_name: MiniMax M2.7
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: MiniMax-M2.7
|
||||
api_key: sk-cp-mrsAGtzDpUl0TpOBPVoiRG_pWF1kDDN9_mDAnONBs6HIPcNUexSKtSbx2QOxfKBvJT6Xw6hDTV6hdkJYFWfNA8UtHQ08TiGJiosdRoW6q1sGb6Cb7D85NHk
|
||||
base_url: https://api.minimaxi.com
|
||||
max_tokens: 4096
|
||||
temperature: 1.0
|
||||
supports_vision: true
|
||||
supports_thinking: true
|
||||
- name: minimax-m2.5
|
||||
display_name: MiniMax M2.5
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: MiniMax-M2.5
|
||||
api_key: sk-cp-mrsAGtzDpUl0TpOBPVoiRG_pWF1kDDN9_mDAnONBs6HIPcNUexSKtSbx2QOxfKBvJT6Xw6hDTV6hdkJYFWfNA8UtHQ08TiGJiosdRoW6q1sGb6Cb7D85NHk
|
||||
base_url: https://api.minimaxi.com
|
||||
max_tokens: 4096
|
||||
temperature: 1.0
|
||||
supports_vision: true
|
||||
supports_thinking: true
|
||||
- name: deepseek-ai/DeepSeek-OCR
|
||||
display_name: deepseek-ai/DeepSeek-OCR
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: deepseek-ai/DeepSeek-OCR
|
||||
api_key: sk-beshfxfxfrllsczpqfrgjbugfbpzxgqgedbehceytpcvvtyb
|
||||
base_url: https://api.siliconflow.cn/v1/
|
||||
max_tokens: 4096
|
||||
temperature: 1.0
|
||||
supports_vision: true
|
||||
supports_thinking: true
|
||||
- name: Qwen/Qwen3.6-27B
|
||||
display_name: Qwen/Qwen3.6-27B
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: Qwen/Qwen3.6-27B
|
||||
model: deepseek-ai/DeepSeek-V4-Flash
|
||||
api_key: sk-qwwwvjompnzxqhndctlwmmomvqdgaqqqgxvibfxchsikvgod
|
||||
base_url: https://api.siliconflow.cn/v1/
|
||||
max_tokens: 4096
|
||||
|
||||
Reference in new issue
Block a user