sync update from LAN repo (source ji @ 85078a31)
This commit is contained in:
1 parent
5d6627af50
commit
e602d88695
864 files changed
+99150
-6346
No files matched your search
@@ -4,6 +4,41 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
||||
|
||||
## Project Overview
|
||||
|
||||
### Ordinary Q&A map visualization
|
||||
|
||||
The ordinary lead-agent prompt includes `_ORDINARY_QA_VISUALIZATION_SECTION` only for normal, non-bootstrap, non-writing, non-notebook, non-scheduled runs and defines the highest-priority output-carrier gate. Only an explicit current-request ask for HTML, a webpage/page, dashboard, large-screen view, or `.html` file permits reading `page-report-generation`, calling `page_report_template`, or writing HTML. “Draw a flowchart/chart/map/mind map” and “make a visualization” never imply HTML: ordinary flowcharts must be inline Mermaid, ordinary charts keep the established message-inline path, and requested maps emit strict JSON in a `map-json` fence. A mind map is emitted only for an explicit current-request ask for a mind map/knowledge map (or `analysis-mindmap`); ordinary analyses, summaries, outlines, flowcharts, and generic JSON must never auto-upgrade into one. Its strict `mindmap-json` contract is a bounded rooted tree (`id`, `parentId`, `label`, semantic `kind`, optional detail/source refs), targeting 12–24 nodes with a hard cap of 36, at most four layers including the center topic, at most six top-level dimensions and three children beneath every later node. Models must consolidate related facts into `detail`/`sourceRefs`, not make one node per report bullet, datum, paragraph, or reference. An unfinished dedicated `mindmap-json` fence (and compatible historical `mindmap` fence) remains an ordinary streaming code block; only the completed, valid object lazy-renders as a compact left-to-right message card. Clicking a parent node, or its +/− control at the outgoing connection point, expands/collapses its child tree and the canvas uses an explicit visible edge stroke. The full dialog keeps its close and view tools in the top-right and a layout-derived all-map preview in the bottom-right; the message preview also permits pan, wheel zoom, and zoom/reset controls. The message card's icon-only download control opens PNG/SVG/JSON choices. Generic JSON never invokes this renderer. Ambiguity defaults to inline output. Once the HTML gate passes, delivery is mandatory as a sandbox-generated `/mnt/user-data/outputs/*.html` artifact opened with `present_files`; a full inline `html` fence is forbidden as a substitute. The shared frontend `MarkdownContent` keeps its exact legacy branch when no map-shaped fence exists; dedicated `map-json` fences and complete generic `json` fences that contain map fields and pass the full map schema are validated and lazy-rendered from `components/workspace/visualizations/`. Map/loading/error cards use the full message-content width so their edges align with the step panel. Non-map and incomplete generic JSON retain the ordinary code-block path.
|
||||
|
||||
HTML artifacts are single-page by default. Omitted `preferred_page_count` is normalized to `1` during direct validation. A count greater than one is allowed only when the current user directly asks for multiple pages or gives a page count; content length and report chapter count never imply multi-page intent. When a template is explicitly requested, prepare strictly excludes profiles from the wrong page-count mode, persists `requested_page_count` and `page_mode`, and validate reloads them. Validation rejects multiple detected page views (`data-page`, tabpanels, `.report-page`, or `.view`) in single-page mode, or a single view when multi-page was explicitly requested.
|
||||
|
||||
Ordinary HTML generation is from scratch by default: do not read `page-template-library`, call catalog/prepare, or select a bundled layout unless the current user explicitly asks to use a template. The agent writes the final path directly under `/mnt/user-data/outputs/`, which lets frontend live preview open on the first `write_file`; it then validates and presents the same file without a bash/cp copy. Single-page HTML means a scrollable responsive document, not a fixed single-screen canvas. Direct validate derives `layout_mode` and page count from `page_brief` / `preferred_page_count`; it rejects `html/body overflow:hidden`, fixed 1920×1080 viewport scaling, undersized G6 containers, and long LR diagrams. Explicit template requests may use the shared `light-document` / `dark-document` profile backed by `report-document-responsive-template.html`; prepare creates its editable `working_copy` directly in outputs and persists the same constraints.
|
||||
|
||||
### Generated HTML artifact visualization runtimes
|
||||
|
||||
The same normal lead-agent prompt includes `_HTML_ARTIFACT_VISUALIZATION_SECTION`, but its contents are executable only after the explicit HTML-intent gate above passes; visualization content alone never implies this intent. New generated HTML explicitly loads browser distributions from Gateway `/api/public/html-runtime/`: `g6.min.js`, `echarts.min.js`, and then `deerflow-runtime.js`; it can also load the packaged country/region map catalog: `china-provinces`, `us-states`, `japan-prefectures`, and `taiwan-counties`. The files are package resources under `packages/harness/deerflow/html_runtime/assets/` and ship with the harness wheel, so HTML execution does not depend on frontend Vite/node_modules assets, Leaflet, map tiles, or a public CDN. `config.yaml -> html_runtime.base_url` is the browser-reachable absolute runtime base written into generated source (localhost default; deployments must replace it with the real Gateway origin). Flowcharts use paired `data-df-flow` JSON blocks and AntV G6, not Mermaid; charts use `data-df-chart`; maps use ECharts Canvas-backed `data-df-geo` with the local GeoJSON catalog. `data-df-chart` must be strict JSON: no JavaScript callbacks or function-valued colors, with literal colors/data-item styles and string tooltip formatters instead. `deerflow-runtime.js` initializes all three contracts, opens a detail popover for described regions/markers, and exposes catalog URLs through its public dynamic response. Map large screens use a centered map stage with narrow side rails and bottom summaries, rather than cards layered across a full-page map background. Frontend runtime replacement is bypassed whenever this explicit Gateway runtime is present. This remains separate from ordinary message rendering; ordinary `map-json` keeps its existing frontend tile renderer and `VITE_MAP_TILE_URL` configuration, and ordinary `flow-json`/`chart-json` must not be reintroduced.
|
||||
|
||||
The G6 runtime must not mutate author layout by auto-switching long LR graphs to TB or by increasing the target element height after render. Layout quality is decided before delivery: keep a flow to at most eight nodes where possible; otherwise split it or declare TB with at least 600px. Each flow gets its own document row and a parent that can grow naturally, preventing canvas overlap with later sections.
|
||||
|
||||
Markdown-to-HTML conversion must inventory source Mermaid/flowcharts, charts, and maps before editing. Direct `page_report_template validate` derives `required_visualizations` from its `required_features` and `page_brief`; an explicit-template prepare also persists a machine-readable `visualization_contract` in the per-output brief, which validate reloads and merges with its arguments. Required flowcharts fail validation unless the HTML contains `g6.min.js`, `deerflow-runtime.js`, a `.df-flow` container, and `data-df-flow`; embedded Mermaid and inline `new G6.Graph(...)` are errors. Required charts use ECharts + `data-df-chart`; required maps use ECharts + `data-df-geo`, an `.df-geo-map` container, and one of the four local GeoJSON catalog entries. Leaflet, `data-df-map`, map tiles, and manual `echarts.init` are validation errors for generated HTML. A failed report must be repaired and revalidated before `present_files`.
|
||||
|
||||
G6 nodes use light semantic rounded cards with matching borders, dark high-contrast text, centered type-prefixed labels, subtle shadows, and non-animated initial rendering (avoids a faded first frame). More than seven nodes automatically switch an `LR` request to layered `TB` layout and expand a short container to 720px, preventing long workflows from shrinking into illegible shapes; drag/zoom remain enabled.
|
||||
|
||||
### 问答记忆运行时总开关
|
||||
|
||||
问答记忆由 `system_settings.json -> memory.enabled` 控制,默认 `false`,并且是问答运行时的权威总开关。关闭时 `lead_agent` 不挂载 `MemoryMiddleware`,工具注册不暴露 memory/hindsight 工具,prompt 不读取记忆,因此也不会触发 V1 的延迟提取模型调用。`config.yaml -> memory` 仅保留 provider/version/限额等参数,不能自行启用记忆。管理员通过 `GET/PUT /api/system-settings/memory` 和「设置 → 问答记忆」修改开关;下一次创建的问答立即按新值构建。
|
||||
|
||||
### 项目空间(固定问答模式)
|
||||
|
||||
- `app/gateway/routers/analysis_projects.py`:用户私有项目 CRUD 和项目下新建/查询对话;更新请求不允许 `mode`,对话归属不通过可编辑 metadata 保存。
|
||||
- 项目线程 metadata 的 `analysis_project_id` / `analysis_project_mode` / `analysis_project_name` 仅用于最近对话的类型、项目名与路由展示;创建、列表修复和每次运行都由服务端覆盖,授权与真实归属仍以 `analysis_project_threads` 绑定为准。
|
||||
- `deerflow.persistence.analysis_projects`:`analysis_projects` / `analysis_project_threads` / additive `analysis_project_configs`,经 models 注册和 Gateway deps 初始化自动建表。配置表保存项目默认智能体、策略 Lens、报告结构快照及每条对话所选智能体,避免修改旧表破坏已有部署。
|
||||
- `app/gateway/analysis_projects.py::prepare_project_run` 在共享 `start_run` 前置读取绑定、校验所有者及知识库、覆盖运行模式;运行知识范围是项目自身普通/助手知识库与该线程已绑定智能体普通知识库的去重并集,不能只取智能体配置而丢掉项目知识库;`authz.py` 同时保护直接读取线程。不要将 `analysis_project` 加入公共 context 白名单。
|
||||
- `deerflow.agents.analysis_project` 提供三个模式的系统约束和固定方案模板;方案复用 deep_research_report 的 fixed structure,thread_harvester 在方案项目中额外采集服务端检索的引用。普通写作仍使用原有 adaptive 配置。
|
||||
- 方案生成首份报告对用户保持两步语义:「正常问答理解需求 → 结束问答并生成方案」。第一步应自然交流并先做语义级意图理解:复述真实目标、面向对象、可观察的成功结果与非目标/边界,推断必须标为暂定;后续消息增量更新整体理解,最新明确要求覆盖较早的冲突表述。仅在关键条件缺失时少量追问;信息足够后输出自然段式的简短「意图理解」,以及唯一的 `solution-intent-json` HTML 注释快照和末尾就绪标记,不得呈现为表单式访谈或额外确认阶段。快照的 `goal` / `audience` / `success_outcome` / `scope` / `constraints` / `deliverable` / `acceptance_criteria` 必须是有界非空字符串,`open_questions` 必须是有界字符串数组且只有为空时可就绪。`prepare_project_run` 删除客户端伪造的 `solution_requirements_confirmed` / `solution_intent`,只解析检查点中最新用户轮次后的最新助手回复;只有可读摘要、合法快照、末尾就绪标记和当前用户确认标记同时成立时才确认,并把归一化快照作为服务端权威 `solution_intent` 注入运行配置。`services.inject_validated_analysis_project_context` 负责绕过公共白名单安全注入这些服务端字段,确认后的系统提示把转义后的快照交给最终生成。已有 `report.md` 仅用于后续编辑的兼容解锁;`deep_research_report` 在首份方案未确认时必须拒绝执行。不要把该闸门退化为仅靠前端按钮或提示词的约定。
|
||||
- 方案阶段的关键追问必须调用现有 `ask_clarification`:普通方案需求第一步至少给出 1 个最关键的可点击决策问题,一个调用只问一个决策点,提供 2–4 个稳定选项(含「其他/我补充」),总量限制 1–3 个,由 `ClarificationMiddleware` 继续输出既有 `{question, clarification_type, context, options, allow_custom, allow_multiple}` 载荷供前端点击。独立问题可同轮多次调用;不要用普通 Markdown 问句替代工具卡,也不要在用户回答前只输出快照/就绪标记。另有服务器校验的强制生成令牌 `<!-- solution-requirements:force -->`:只认本轮最后一条 user/human 消息,仍清除客户端伪造的 `solution_force_generate` / `solution_requirements_confirmed` / `solution_intent`。强制生成可复用 checkpoint 中最近一个合法但未就绪的快照(允许 `open_questions` 非空;没有则为 `None`),prompt 必须结合完整对话推断缺项、显式标注暂定假设并禁止再次澄清。确认/强制生成首报轮以 `WritingModeReportMiddleware(require_pipeline=True)` 做运行时兜底:纯文本“已生成”或澄清调用会被确定性改写为 `deep_research_report`;任何完成提醒/循环成功判定都必须同时看到固定成功工具结果和状态中真实登记的 `/mnt/user-data/outputs/report.md` artifact,被意图门禁拒绝的 ToolMessage 绝不能触发“已完成”提醒。当前用户轮次里只要管线返回结果却没有 artifact,就输出确定性的失败/重试提示并结束,禁止自动重试到循环上限;下一次用户点击「重新生成方案」才开始新调用。
|
||||
- 方案项目最终落盘的 `report.md` 禁止 `[1]` / `【1】` 等数字引用标记。`deep_research_report_tool` 在 solution 模式下同时注入无编号引用要求并对最终 Markdown 做确定性清理:正文删除数字引用标记,参考来源改为无编号列表,代码围栏和行内代码保持原样;普通写作与深度研究不走该分支。后续方案修改的系统提示也必须维持这项约束。
|
||||
- 前端 `frontend-web/src/pages/AnalysisProjectsPage.tsx` 和 `pages/analysis-projects/ProjectConversation.tsx` 嵌入现有 WorkspaceSidebar,复用消息流、输入框、研究进度和产物画布;策略项目默认打开真实消息流,用户问答、reasoning 与技能/工具执行步骤必须在运行中可见,结构化看板仍可一键切换。`StrategyBoard.tsx` 只解析对用户可见的「意图理解 / 多视角策略分析与辩论 / 完整策略方案」Markdown,不读取 reasoning/tool payload。完整策略方案以可导出的文档结构呈现。方案设置复用 `WorkflowConfigDialog` 管理报告结构。
|
||||
- 回归测试:`tests/test_analysis_projects.py`(持久化、权限、模式锁定、运行配置与模板)。
|
||||
|
||||
DeerFlow is a LangGraph-based AI super agent system with a full-stack architecture. The backend provides a "super agent" with sandbox execution, persistent memory, subagent delegation, and extensible tool integration - all operating in per-thread isolated environments.
|
||||
|
||||
**Architecture**:
|
||||
@@ -77,7 +112,7 @@ When making code changes, you MUST update the relevant documentation:
|
||||
|
||||
### Workflow Studio stream and canvas compatibility
|
||||
|
||||
- `app/gateway/workflow_agent_runner.py` is the boundary between LangGraph token chunks and durable workflow events. Provider reasoning in `AIMessage.additional_kwargs.reasoning_content`/`reasoning` is emitted to the UI as balanced inline `<think>` blocks; only visible answer content becomes the node's `text` output for downstream nodes. It must also emit every original `[AIMessage|ToolMessage, metadata]` tuple as `node.message.data.messages` with no whitelist, preview or length crop. Tool results and `additional_kwargs.clarification` are needed by the Studio's DeerFlow-compatible message UI and must survive durable SSE replay.
|
||||
- `PatchedChatOpenAI` accepts provider response fields `reasoning_content` and `reasoning`, normalizing both into `AIMessage.additional_kwargs.reasoning_content`. For OpenAI-compatible models that declare `supports_thinking: true` without an explicit `when_thinking_enabled`, `create_chat_model` sends `extra_body.enable_thinking=true` whenever the frontend enables thinking. `app/gateway/workflow_agent_runner.py` is the boundary between LangGraph token chunks and durable workflow events. Provider reasoning in `AIMessage.additional_kwargs.reasoning_content`/`reasoning` is emitted to the UI as balanced inline `<think>` blocks; only visible answer content becomes the node's `text` output for downstream nodes. It must also emit every original `[AIMessage|ToolMessage, metadata]` tuple as `node.message.data.messages` with no whitelist, preview or length crop. Tool results and `additional_kwargs.clarification` are needed by the Studio's DeerFlow-compatible message UI and must survive durable SSE replay.
|
||||
- An agent `ask_clarification` ToolMessage pauses the workflow at that agent node (`awaiting_input`), not as a terminal run. The pause records the source `nodeRunId`/`toolCallId`; resuming turns the submitted reply into the next user message on the same `wf-{run}-{node}` LangGraph thread. Keep the raw ToolMessage authoritative; do not invent a second card payload.
|
||||
- `app/gateway/workflow_proposal_planner.py` converts the workflow controller's selected catalog roles into per-agent `mission`/`deliverable`/`scope`/`handoff` contracts. Only a selected visible `agentId` may contribute a contract; every omitted or malformed field falls back to a deterministic role-aware contract. Preserve this prompt-level worker boundary when adding strategies: researchers must not silently become final writers, and sequential roles must receive their upstream handoff binding plus an explicit next handoff.
|
||||
- `app/gateway/routers/workflows_coze_compat.py` must return both `creator.self: true` and editable VCS metadata for an owned standalone workflow. Coze Playground derives its preview/read-only mode from those fields, so omitting `creator.self` disables node dragging even when the standalone page passes `readonly={false}`.
|
||||
@@ -213,7 +248,7 @@ Independent of Workflow Studio and the existing roundtable. Plan: repo-root `doc
|
||||
- **Config**: `config.yaml → report_collaboration` (`deerflow.config.report_collaboration_config.ReportCollaborationConfig`). Default **off** (`enabled` / `worker_enabled`).
|
||||
- **Contracts**: `app/report_collaboration/contracts/` — snake_case Pydantic models matching `frontend-web/src/report-collaboration/api/types.ts`. SSE envelope uses camelCase (`eventId` / `sessionId` / `runId`). `node_run_id` is always `${node_id}-attempt{attempt}`.
|
||||
- **Persistence**: `deerflow.persistence.report_collaboration` (`report_collaboration_*` tables, migrations `20260905_01` / `20260906_01` / `20260907_01`). SQL + memory stores; one active run per session via `active_dedupe_key`; command/message/run/version idempotency keys refuse cross-session reuse; event `seq` taken from the run row. Session delete removes children explicitly.
|
||||
- **Gateway**: `/api/report-collaboration` (`app/gateway/routers/report_collaboration.py`). `enabled=false` → 503. Plan/run writes enqueue durable jobs and return immediately. Pre-run `POST /sessions/{id}/messages` runs a rule-based `IntentRouterAgent` + `RequirementResolver` (`app/report_collaboration/requirement/`): writes a requirement revision and clarification messages, never auto-starts planning. `POST /sessions/{id}/plan-requests` then runs `PlanProposalService` (`app/report_collaboration/planning/`): catalog projection (no SOUL/keys) → selector → 2–3 strategy RoleDemand graphs → assembler binds catalog agents/tools/quality gates → validator (acyclic DAG, artifact types, allowlisted tools). Select and start-run stay independent commands. `POST /runs` freezes role models, report-structure template and run budget. Rewrite / apply / restore are live (`reporting/`). `GET /runs/{id}/stream` is a read-only durable SSE tail (`execution/live_hub.py`): persist then publish, replay + `after_seq` / `Last-Event-ID`, heartbeat comments, token-delta coalesce before seq assignment. Disconnect does **not** cancel the run. Lifespan mounts `ReportCollaborationDispatcher` only when `enabled` **and** `worker_enabled` are both true; default both false.
|
||||
- **Gateway**: `/api/report-collaboration` (`app/gateway/routers/report_collaboration.py`). `enabled=false` → 503. Pre-run intent dialogue uses the separately seeded built-in agent `report-collaboration-intent` through the shared real `/api/intent/init` + `/api/intent/stream` LangGraph thread path. The browser consumes upstream `messages-tuple` frames incrementally (text, provider reasoning and tool steps) and renders them in the normal `MessageList`; this path must never be replaced with `PlannerAgent.ainvoke()` or another wait-for-completion wrapper. `POST /sessions/{id}/intent-turns` receives only the completed streamed turn for durable mirroring; it does not invoke an intent model. An `INTENT_READY` terminal summary is converted to the report requirement and automatically fulfilled through `PlanProposalService`, with no first-plan confirmation click. The old pre-run `/messages` path returns `USE_STREAMING_INTENT`. `POST /sessions/{id}/plan-requests` remains for regeneration/compatibility. Workflow design after intent confirmation is LLM-generated, assembled against the safe agent catalog and validated (acyclic DAG, artifact types, allowlisted tools). Selecting a plan and starting a run remain independent commands. `POST /runs` freezes role models, report-structure template and run budget. Rewrite / apply / restore are live (`reporting/`). `GET /runs/{id}/stream` is a read-only durable SSE tail (`execution/live_hub.py`): persist then publish, replay + `after_seq` / `Last-Event-ID`, heartbeat comments, token-delta coalesce before seq assignment. Disconnect does **not** cancel the run. Lifespan mounts `ReportCollaborationDispatcher` only when `enabled` **and** `worker_enabled` are both true; default both false.
|
||||
- **Runtime**: AgentScope adapters live in `app/report_collaboration/agentscope_runtime/` (not the publishable harness). `ConversationReplica` maps AgentScope 2.0.5 events 1:1 onto collaboration SSE types; TeamSay is `HINT_BLOCK`. `DeerFlowChatModelAdapter` wraps `create_chat_model` (no second Provider config); role models are frozen onto node `agent_snapshot` at `POST /runs` and missing/incapable models fail with 422 before the run is created. Fallback is recorded on the snapshot plus a durable `heartbeat` audit event. Nodes complete only after a validated artifact (`app/report_collaboration/execution/completion.py`). `TaskLedger` (`execution/ledger.py`) owns ready waves, attempts and terminal node status; AgentScope / coordinator suggestions never pick the next node. A writer-bearing run is `completed` only after `ReportFinalizer` publishes a formal version. `WaveExecutor` + `QualityRoleKernel` (`execution/quality_roles.py`) remain for fixtures: Verifier marks stale/contested claims, Analyst/Writer only cite supported IDs, Reviewer `revise` replans only the target write node plus review. Production worker injects `AgentScopeMemberKernel` (`agentscope_runtime/member_kernel.py`) with DeerFlow tools and per-run budget. `QualityGate` (`execution/quality_gate.py`) is the claim-level gate: conflicts/missing units, unverified facts, coverage gaps, and unmapped review targets fail before `validated`; `ArtifactBoard.supersede_previous` hides old attempts from downstream. `ReportCollaborationLiveHub` (`execution/live_hub.py`) fans out persisted envelopes to SSE viewers; `PublishingReportCollaborationStore` sanitizes secrets/privacy/hidden reasoning without truncating visible tool results, and writes redaction + event audits. Dispatcher (`execution/dispatcher.py`) atomically claims queued/recovering/expired-lease runs, heartbeats via `renew_lease`, two-phase cancels (`cancel_requested` → `finalize_cancel`), and recovers orphans (`execution/recovery.py`): keep completed work, mark leftover in-flight `interrupted`/`superseded`, open a new `planned` attempt. Late artifacts after cancel/reclaim are `superseded` and must not change terminal run status. Tests drive `dispatch_once()`; do not rely on the background loop. Per-run budgets (`security/budget.py`) fail-closed; per-user active-run caps are enforced at `create_run`.
|
||||
- **Quality eval**: `app/report_collaboration/eval/` (`RC-BE-018`) — 20 annotated tasks (policy/market/enterprise/event/comparison ×4) with required angles, key facts, forbidden fabrications, templates and review rules. Deterministic scorer (no LLM judge) gates citation/angle/template/directed-mod; gold reports must pass, canned failures (including a roundtable-style essay) must fail and stay reproducible. Recommended runtime knobs live in `eval/thresholds.py` and match current `ReportCollaborationConfig` defaults (`worker_enabled` remains false). Tests: `tests/test_report_collaboration_eval.py`.
|
||||
- **Runtime intervention**: `app/report_collaboration/interventions/` owns run-time natural-language commands. `RuntimeIntentRouterAgent` uses explicit rules first and a tool-free coordinator-model fallback only for ambiguity; its Pydantic decision cannot dispatch or mutate anything. `ImpactAnalyzer` validates server-side node/agent ownership, resolves angle-specific lineage, and enforces confidence/cost/parallel-branch confirmation (`intent_confirmation_threshold`, default `0.8`). `RuntimeCommandService` mirrors the human message, emits frozen command events, CAS-claims accepted commands, and applies them only through `TaskLedger` safe points. Rework persists command overlays, supersedes affected attempts/artifacts, creates monotonic attempts, and prevents late output from entering downstream. Waiting-node answers resume only those nodes. `rerun_node` without a legal target does not expand to the whole graph; requirement patches apply only after rework succeeds; stuck `executing` commands are reclaimed on takeover or lease timeout; `command.cancelled` is a durable SSE event. In-run `POST /sessions/{id}/messages` returns 409 (`RUN_ACTIVE`). Formal rewrite candidates and version apply/restore live in `reporting/` (`RC-BE-015`).
|
||||
@@ -301,6 +336,8 @@ from deerflow.config import get_app_config
|
||||
|
||||
**Strategy-selection functional agents.** `app/gateway/routers/_strategy_selection_seed.py` seeds nine stable folder-backed agents from `_strategy_selection_seed_assets/` before `_sync_legacy_agents`: intent, governance/diplomacy/economy/security/public-opinion/law-policy angles, judge, and solution editor. Seeding is missing-directory-only and must never overwrite administrator changes. The global `strategy_agent_roles` registry classifies them independently of projects: intent/judge/solution are fixed, unique roles that may be configured but never added, disabled, reordered, or deleted; angle roles are a shared, ordered catalog that administrators may add, edit, disable (soft delete), restore, and reorder through `/api/strategy-agents`. Public `GET /api/strategy-agents` returns enabled identity/description metadata only; `include_disabled=true` or `include_config=true` is admin-only. Admin writes are `POST /api/strategy-agents/angles`, `PUT /api/strategy-agents/{id}`, and `DELETE /api/strategy-agents/{id}`; DELETE retains the folder-backed agent and role row to preserve historical references. Dynamic angles remain built-in folder-backed agents under the reserved `strategy-selection-angle-` id namespace, with `web` + `file:read`; a blank SOUL on create receives a safe role template. All strategy skill writes validate that names exist, are active, and are globally visible; generic agent CRUD cannot create or hard-delete reserved strategy IDs, and parallel execution rejects disabled roles. Angle configs reuse `policy-knowledge`; intent reuses `intent-clarification-policy`; the judge has no retrieval skill; the solution editor gets only `file:write`. Keep the SOUL contracts neutral and evidence-aware, with explicit de-escalation/civilian-protection framing and hard bans on operational combat guidance and targeted political manipulation. Tests: `tests/test_strategy_selection_seed.py`, `tests/test_strategy_agent_management.py`.
|
||||
|
||||
`POST /api/parallel-agents/run/stream` must remain observable during silent skill calls. Each running seat emits an `agent_progress` heartbeat every five seconds with the latest translated tool/reasoning/body phase and monotonic elapsed seconds; an upstream iterator that produces no frame for 180 seconds is closed and that seat emits `agent_error`, while the other seats continue. Set the terminal flag before queuing `agent_done` / clarification so a late heartbeat cannot reopen a completed card. The strategy workbench renders the intent Q&A history, readable skill execution steps and live heartbeat on the affected agent card; judge and synthesis traces use the same step translation. Tests: `tests/test_roundtable_model_fallback.py` and frontend `src/strategy-selection/lib.test.ts`.
|
||||
|
||||
**ThreadState** (`packages/harness/deerflow/agents/thread_state.py`):
|
||||
- Extends `AgentState` with: `sandbox`, `thread_data`, `title`, `artifacts`, `todos`, `uploaded_files`, `viewed_images`
|
||||
- Uses custom reducers: `merge_artifacts` (deduplicate), `merge_viewed_images` (merge/clear)
|
||||
@@ -332,7 +369,8 @@ Lead-agent middlewares are assembled in strict append order across `packages/har
|
||||
15. **DeferredToolFilterMiddleware** - Hides deferred tool schemas from the bound model until tool search is enabled (optional)
|
||||
16. **SubagentLimitMiddleware** - Truncates excess `task` tool calls from model response to enforce `MAX_CONCURRENT_SUBAGENTS` limit (optional, if `subagent_enabled`)
|
||||
17. **LoopDetectionMiddleware** - Detects repeated tool-call loops; hard-stop responses clear both structured `tool_calls` and raw provider tool-call metadata before forcing a final text answer
|
||||
18. **ClarificationMiddleware** - Intercepts `ask_clarification` tool calls, interrupts via `Command(goto=END)` (must be last)
|
||||
18. **ContextWindowGuardMiddleware** - Final transient request-size guard; enforces the selected model's `max_input_tokens` after every context/tool injection
|
||||
19. **ClarificationMiddleware** - Intercepts `ask_clarification` tool calls, interrupts via `Command(goto=END)` (must be last)
|
||||
8a. **ToolMetricsMiddleware** - Appended immediately after `ToolErrorHandlingMiddleware` in `_build_runtime_middlewares`, so it sits *deepest* in the `wrap_tool_call` chain. Records one `tool_call_metrics` row per tool/skill/MCP invocation ? timing duration and bucketing failures into a coarse `error_category` (rate_limited / ip_blocked / auth_error / timeout / network_error / not_found / empty_result / tool_error). It catches both raised exceptions (before #8 converts them) and *self-handled* errors returned as error `ToolMessage`s (search tools that swallow HTTP 429/403). It also captures the targeted Agent Skill into `tool_call_metrics.skill_name`: the lead agent invokes a skill by `read_file`-ing its `/mnt/skills/<category>/<name>/SKILL.md` path (per the `<skill_system>` prompt block ? there is no dedicated "use skill" tool), so `skill_name` is recovered by scanning every tool call's string arguments for the `skills/{public,custom}/{name}/` path shape (`_SKILL_PATH_RE`), plus the explicit `name` argument of `skill_view`/`skill_manage`/`skill_archive`. `skill_view`-specific success detection handles its swallowed failures (`Skill 'x' not found.` returned as a plain string rather than raised). Regression coverage: `tests/test_tool_metrics_middleware.py`. Writes are best-effort fire-and-forget. **Per-run skill merging**: `SqlToolMetricsStore.record` upserts skill rows by `(run_id, skill_name)` instead of inserting one row per call ? a skill touched by several tool calls / retries inside one run collapses to a single row. Any success in the run wins (a skill that failed then finally succeeded is recorded as one `success`); a skill that only ever failed yields one `error` row whose `error_message` accumulates every attempt's message; durations are summed. Non-skill tool calls, and skill calls with no `run_id`, keep one row each. The read?merge?write is guarded by a per-event-loop `asyncio.Lock`. Surfaced by the admin `GET /api/admin/tool-metrics` router (which has a `kind=skill` filter and a per-skill `/summary` breakdown).
|
||||
9. **PromptPrefixMiddleware** - Injects the admin-configured prompt prefix into the model request only. The prefix is tagged onto the first human message's `additional_kwargs['prompt_prefix']` by the Gateway (`app.gateway.services`) and prepended to content here at call time, so the persisted/displayed message stays prefix-free
|
||||
10. **SummarizationMiddleware** (`DeerFlowSummarizationMiddleware`) - Context reduction when approaching token limits (optional, if enabled). **Non-destructive**: overrides the stock `before_model` (which persists a `RemoveMessage(REMOVE_ALL_MESSAGES)`) to a no-op and instead compresses **transiently** in `wrap_model_call` ? the model request is replaced with `[summary, *recent]` while the full original conversation stays in state (so the frontend keeps rendering the user's real Q&A). The `summary` message is `hide_from_ui` and never persisted; summaries are cached **per-thread in a process-wide, thread-safe LRU** (`_GLOBAL_SUMMARY_CACHE`, bounded to 512 threads, sticky boundary). The cache is module-level — **not** on the middleware instance — because `make_lead_agent` rebuilds a fresh middleware on every run; an instance cache would be cold each turn and force a blocking summary LLM call on every turn of a long conversation. Keyed by `thread_id` (resolved like `ThreadDataMiddleware`), so a summary computed on one turn is reused on the next, paying for an LLM call only on the first threshold crossing or a genuine re-compaction. **Threshold-first gating**: `_plan_compression` returns passthrough (no compression at all) whenever the *full* current conversation is still under the trigger — the cache/summary substitution only engages once the real context crosses the configured trigger. **Stickiness headroom** (`_apply_stickiness_headroom`, `_KEEP_HEADROOM_FRACTION=0.75`): after the `keep` policy picks a cutoff, the kept suffix is forced below ~75% of *every* trigger before summarizing. Without it a `keep` in *messages* (e.g. 20) can itself exceed a *token* trigger (e.g. 60000) when recent messages are large (tool outputs / pasted docs), leaving the "compressed" context still over the trigger so it re-summarizes on every follow-up question; the headroom makes the sticky boundary actually stick (re-compaction only after genuinely new content). When the genuine (blocking) summarize path runs, it emits a `{"type":"context_compacting","message":"正在压缩上下文…"}` event on LangGraph's `custom` stream channel (the cheap reuse path stays silent); the chat frontend's `onCustomEvent` renders it as a transient toast. Tests: `tests/test_summarization_middleware.py`
|
||||
@@ -344,7 +382,8 @@ Lead-agent middlewares are assembled in strict append order across `packages/har
|
||||
16. **DeferredToolFilterMiddleware** - Hides deferred tool schemas from the bound model until tool search is enabled (optional)
|
||||
17. **SubagentLimitMiddleware** - Truncates excess `task` tool calls from model response to enforce `MAX_CONCURRENT_SUBAGENTS` limit (optional, if `subagent_enabled`)
|
||||
18. **LoopDetectionMiddleware** - Detects repeated tool-call loops; hard-stop responses clear both structured `tool_calls` and raw provider tool-call metadata before forcing a final text answer
|
||||
19. **ClarificationMiddleware** - Intercepts `ask_clarification` tool calls, interrupts via `Command(goto=END)` (must be last)
|
||||
19. **ContextWindowGuardMiddleware** - Final transient request-size guard; enforces the selected model's `max_input_tokens` after every context/tool injection
|
||||
20. **ClarificationMiddleware** - Intercepts `ask_clarification` tool calls, interrupts via `Command(goto=END)` (must be last)
|
||||
|
||||
### Configuration System
|
||||
|
||||
@@ -395,16 +434,17 @@ FastAPI application on port 8001 with health checks at `GET|HEAD /health`, `/api
|
||||
| **Memory** (`/api/memory`) | `GET /` - memory data; `POST /reload` - force reload; `GET /config` - config; `GET /status` - config + data |
|
||||
| **Agents** (`/api/agents`) | `GET /` - list built-in, owned, and published agents (optional `search` + repeatable `tag_ids` multi-select filter, OR-matched; each item carries `tags` and `featured_order`); `POST /` - create user-owned agent; `GET/PUT/DELETE /{id}` - read/update/delete by stable agent id; `GET /check` - validate non-empty display name (duplicates allowed); `GET /selector` - homepage agent picker (single admin-curated list, visibility-filtered, sorted by `featured_order` asc); `PUT /{id}/featured` - **admin** sets/clears the homepage `featured_order` (must be `published=true` to feature; passing `order: null` clears unconditionally); `PUT /featured/order` - **admin** bulk-reorders the homepage list (body `{agent_ids: [...]}` ? indices become 0..N-1) |
|
||||
| **Tags** (`/api/tags`) | `GET /` - list tags (optional `tag_type` filter; open to all); `POST /` - create tag (admin); `PUT /{id}` / `DELETE /{id}` - update/delete tag (admin); `GET /assignments/{target_type}/{target_id}` - tags on a resource; `PUT /assignments/{target_type}/{target_id}` - replace a resource's tags (admin). Tags are typed `agent`/`skill`/`scheduled_task` and may only tag resources of their own type. |
|
||||
| **Uploads** (`/api/threads/{id}/uploads`) | `POST /` - upload files (auto-converts PDF/PPT/Excel/Word); `GET /list` - list; `DELETE /{filename}` - delete |
|
||||
| **Upload policy** (`/api/upload-policy`) | Database-backed global singleton. `GET /` is authenticated; `PUT /` and `POST /reset` are admin-only atomic replacements with revision/audit metadata. Startup seeds the real database row with 20 files, 100 MiB total, documents/images 10 MiB each, spreadsheets 5 MiB, and large-file budgets 12k/32k chars. The validated row is also published to the harness runtime registry without introducing an `app.*` import. |
|
||||
| **Uploads** (`/api/threads/{id}/uploads`) | `POST /` reads the latest database policy, enforces category extensions/sizes/total/count, validates signatures/OOXML containers/text encoding, then commits the whole staged batch or rolls it back; optional document conversion uses `<original-name>.<ext>.md`. `GET /limits` returns the full effective policy plus compatibility fields; `GET /list` lists; `DELETE /{filename}` deletes. |
|
||||
| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; unexpected failures are logged server-side and return a generic 500 detail |
|
||||
| **Threads search** (`POST /threads/search`, LangGraph-compat) | Conversation listing. **System threads are excluded by default**: rows whose `metadata.system` is truthy (scheduler runs, roundtable workers) never crowd the user's list; a caller whose `metadata` filter explicitly references `thread_type`/`system` (e.g. studio notebooks) opts out of the exclusion. New `query` field = case-insensitive title (`display_name`) substring search, used by the chats page's server-side pagination. `POST /threads/count` takes the same request body (limit/offset ignored) and returns `{total}` under identical filtering/exclusion semantics ? powers the chats page's ??/??? display without fetching rows (`ThreadMetaStore.count`). `ThreadMetaStore.search` gained `exclude_system`/`query` params; JSON-blob filters batch-scan until the requested `limit`/`offset` window is full (so pagination over a heavily-filtered set never under-fills), with a `thread_id` tie-breaker on the `updated_at DESC` ordering for stable OFFSET paging. Tests: `tests/test_thread_meta_search.py` |
|
||||
| **Artifacts** (`/api/threads/{id}/artifacts`, `/api/artifact-library`) | `GET /{path}` - serve artifacts; active content types (`text/html`, `application/xhtml+xml`, `image/svg+xml`) are always forced as download attachments to reduce XSS risk; `?download=true` still forces download for other file types. `GET /api/artifact-library` scans every current-user normal thread's authoritative `outputs/` directory (including files omitted from `present_files`), excludes uploads and image/audio/video media, joins the owning conversation title/agent route metadata, and exposes search/filter/pagination with a bounded 15-second per-user cache plus `refresh=true` bypass. Frontend route `/page/workspace/library` links each item back to the source chat with `?artifact=<path>` so the chat opens the existing artifact panel directly. |
|
||||
| **Artifacts** (`/api/threads/{id}/artifacts`, `/api/artifact-library`, `/api/strategy-project-artifacts/{id}/download`) | `GET /{path}` streams thread artifacts; active content types (`text/html`, `application/xhtml+xml`, `image/svg+xml`) are always forced as download attachments to reduce XSS risk; `?download=true` still forces download for other file types. `GET /api/threads/{id}/artifact-preview?path=` returns bounded text, using streaming statistics plus deterministic head/middle/tail samples above the database threshold. `GET /api/artifact-library` combines every current-user normal thread's authoritative `outputs/` files with completed strategy-selection syntheses projected from the owner-scoped `parallel_agent_tasks.messages` snapshot. Strategy items use `source_type=strategy`, carry `strategy_project_id` / `strategy_name` / `strategy_stale`, and explicitly distinguish `strategy_source_kind=strategy_selection` from an `analysis_project_mode=strategy` thread (`analysis_project`); virtual items also expose `download_url`, so clients never infer routing from sentinel thread/path values. Search/source filters cover both. `workflowStale=true` keeps the last completed historical synthesis visible and downloadable. The dedicated strategy download route revalidates current-user ownership and snapshot completion and serves Markdown without pretending the virtual item is a thread artifact. Uploads and image/audio/video media remain excluded; list results keep the bounded 15-second per-user cache plus `refresh=true` bypass. |
|
||||
| **Word export** (`POST /api/writing/export/docx`) | Shared Markdown → DOCX generator used by conversation, artifact, scheduled-task, Deep Research, and AI-writing download surfaces. `app/gateway/word_export.py` builds OOXML with no Python conversion dependency: A4 portrait, 3.7/3.5/2.8/2.6 cm margins, real multilevel heading numbering (`一、` / `(一)` / `1.`), two-character first-line indent on heading levels 1–3, formal body/table styles, page numbers beginning at 1 with odd pages left-aligned and even pages right-aligned, and embedded `app/gateway/assets/fonts/方正小标宋简体.ttf`. Before numbering it removes complete manual heading prefixes (including `2.1` / `2.1.1`) and drops Markdown horizontal rules from Word output. The ordinary-Q&A Lead Agent can emit the matching Markdown heading forms and avoid unordered bullet lists when the admin enables `prompt_prefix.ordinary_qa_markdown_format_enabled` in `system_settings.json`; the switch defaults off, while writing setup, writing mode, notebook, scheduled runs, and SOUL-driven custom agents keep their independent prompts. The font loader validates its internal family, SHA-256, and OpenType embedding permission (`OS/2.fsType`) before export. Tests: `tests/test_word_export.py`, `tests/test_es_query_routing_prompt.py`. |
|
||||
| **Deep Research** (`/api/deep-research`) | Durable, user-owned research sessions/jobs with replayable SSE events and hidden-thread artifacts. `basic`/`quick` use the compact runner; `detailed` creates bounded subtopics, searches/writes sections concurrently, and assembles a report; `deep` recursively searches follow-up questions under a deterministic breadth/depth query budget. `multi_agent` is a LangGraph 1.x editor/researcher/writer/reviewer graph: its initial plan emits `awaiting_input`, the recorder stores the interrupt and releases the lease, and `POST /jobs/{id}/resume` rebuilds the graph from the persisted approved/revised plan. A bounded `custom_outline` is frozen into the session; headings/list entries deterministically supply the detailed and multi-agent plan, and the shared writer receives it only as a guarded content constraint. Completed reports support owner-scoped follow-up (`GET /sessions/{id}/messages`, `POST /sessions/{id}/chat`): context is only the current report, selected sources and recent messages; source-id citations are persisted, while `allow_new_research=true` is explicitly rejected. Owners may change that evidence set only after the job stops via `PATCH /sessions/{id}/sources/{source_id}`; this applies to future follow-ups rather than rewriting a completed report. Optional report illustrations are gated by `config.yaml -> deep_research.image_generation`: the Harness `OpenAICompatibleImageProvider` requires base64 output, the shared runner step writes only bounded `research-image-01..04.(png|jpg|webp)` artifacts and Markdown uses private `deep-research://` links. `GET /sessions/{id}/artifacts/{name}` checks session ownership before serving an image; an unavailable provider yields a warning but never fails the text report. Runners live in `packages/harness/deerflow/agents/deep_research/runners/`, may never import `app.*`, and use only the `AdapterBundle`. |
|
||||
| **Deep Research** (`/api/deep-research`) | Durable, user-owned research sessions/jobs with replayable SSE events and hidden-thread artifacts. `basic`/`quick` use the compact runner; `detailed` creates bounded subtopics, searches/writes sections concurrently, and assembles a report; `deep` recursively searches follow-up questions under a deterministic breadth/depth query budget. `multi_agent` is a LangGraph 1.x editor/researcher/writer/reviewer graph: its initial plan emits `awaiting_input`, the recorder stores the interrupt and releases the lease, and `POST /jobs/{id}/resume` rebuilds the graph from the persisted approved/revised plan. A bounded `custom_outline` is frozen into the session; headings/list entries deterministically supply the detailed and multi-agent plan, and the shared writer receives it only as a guarded content constraint. When `custom_outline` is empty (the workbench default — the user did not pick a report structure), chat collection first calls `POST /sessions/{id}/intent-outline` after the collector turn ends: the chosen model plans a Markdown skeleton from the topic plus collected materials, the outline is frozen into the session snapshot and shown in the「开始撰写」form for editing. If that step is skipped or fails, `basic`/`quick` still call `generate_intent_outline` at write time, emit `plan_created` (`source: intent`), then write through the same shared writer; generation failure is a recoverable `intent_outline_fallback` warning and does not abort the report. Completed reports support owner-scoped follow-up (`GET /sessions/{id}/messages`, `POST /sessions/{id}/chat`): context is only the current report, selected sources and recent messages; source-id citations are persisted, while `allow_new_research=true` is explicitly rejected. Owners may change that evidence set only after the job stops via `PATCH /sessions/{id}/sources/{source_id}`; this applies to future follow-ups rather than rewriting a completed report. Optional report illustrations are gated by `config.yaml -> deep_research.image_generation`: the Harness `OpenAICompatibleImageProvider` requires base64 output, the shared runner step writes only bounded `research-image-01..04.(png|jpg|webp)` artifacts and Markdown uses private `deep-research://` links. `GET /sessions/{id}/artifacts/{name}` checks session ownership before serving an image; an unavailable provider yields a warning but never fails the text report. Runners live in `packages/harness/deerflow/agents/deep_research/runners/`, may never import `app.*`, and use only the `AdapterBundle`. |
|
||||
| **Gateway auth** | Owner-scoped Deep Research routes await the platform's asynchronous `get_current_user` resolver before they read or write persistence, ensuring each session and job operation receives a concrete user ID. Deep Research artifacts are always resolved from a whitelisted filename via `/mnt/user-data/outputs/<name>` before mapping to a host path, preserving the sandbox guard on native Windows and Linux hosts. |
|
||||
| **Suggestions** (`/api/threads/{id}/suggestions`) | `POST /` - generate follow-up questions; rich list/block model content is normalized before JSON parsing |
|
||||
| **Thread Runs** (`/api/threads/{id}/runs`) | `POST /` - create background run; `POST /stream` - create + SSE stream; `POST /wait` - create + block; `GET /` - list runs; `GET /{rid}` - run details; `POST /{rid}/cancel` - cancel; `GET /{rid}/join` - join SSE; `GET /{rid}/messages` - paginated messages `{data, has_more}`; `GET /{rid}/events` - full event stream; `GET /../messages` - thread messages with feedback; `GET /../token-usage` - aggregate tokens |
|
||||
| **Feedback** (`/api/threads/{id}/runs/{rid}/feedback`) | `PUT /` - upsert feedback; `DELETE /` - delete user feedback; `POST /` - create feedback; `GET /` - list feedback; `GET /stats` - aggregate stats; `DELETE /{fid}` - delete specific |
|
||||
| **Answer feedback** (`/api/threads/{id}/messages/{mid}/feedback`) | Message-scoped `PUT`/`DELETE`, `GET /feedback-context`, and batched `POST /api/threads/{id}/feedback-contexts`; persists excellent/poor ratings, six stable poor-reason codes and optional detail idempotently per user/thread/message/agent. Legacy run-scoped feedback routes remain compatible. Admin-only `GET /api/agents/{agent_id}/feedback-score` uses the project-wide `system_role == "admin"` rule (and preserves auth-disabled mode) and returns a Bayesian user-feedback answer-quality score, raw task-success rate, weighted response-quality dimensions and reason counts; it is explicitly not a complete agent-capability evaluation. |
|
||||
| **Runs** (`/api/runs`) | `POST /stream` - stateless run + SSE; `POST /wait` - stateless run + block; `GET /{rid}/messages` - paginated messages by run_id `{data, has_more}` (cursor: `after_seq`/`before_seq`); `GET /{rid}/feedback` - list feedback by run_id |
|
||||
| **Open Chat** (`/api/open/chat`) | **Unauthenticated** open Q&A entry for external systems (the `/api/open/` prefix is whitelisted in `auth_middleware._PUBLIC_PATH_PREFIXES` + CSRF-exempt in `csrf_middleware.should_check_csrf`). `POST /chat` `{message \| messages[], model_name?, agent_name?, agent_id?, thinking_enabled=false, strip_think=true, thread_id?, recursion_limit=100}` → resolves the agent by **Chinese display name** (`agent_name`, matched against each `.deer-flow/agents/*/config.yaml` `name` via `_resolve_agent_id`; explicit `agent_id` wins if given), builds the lead agent via `make_lead_agent` (so the chosen **agent's** SOUL/tools/skills all apply), `ainvoke`s it once (stateless — no checkpointer, no thread persistence; runs under the `"default"` user bucket), and returns **only the final** `{answer, model_name, agent_name, agent_id, thinking_enabled, thread_id}` (no intermediate tool-call/process). **`POST /chat/stream`** is the **SSE streaming variant** (same request body) for **long tasks** — the blocking `/chat` produces no downstream bytes for the whole run, so a front nginx `proxy_read_timeout` (default 60s) cuts it off with a **504**; the stream keeps the connection warm (a `: keepalive` SSE comment every `_HEARTBEAT_SECONDS=15`s via a background producer-task + queue, plus `event: delta` AI-text increments and `event: progress`), and ends with **`event: result`** carrying the same final `{answer, …}` JSON (or `event: error`). Clients only need to read `event: result`; intermediate events are ignorable. Both endpoints share `_build_run_context` (config/agent/thread). `GET /agents` lists `{name, id}` so callers know what `agent_name` to send. **Thinking off is reliable**: when `thinking_enabled=false` the route also sets `thinking_force_disabled=true` so the lead agent calls `create_chat_model(force_disable_thinking=True)`, which injects **three** OpenAI-compatible "thinking off" shapes into `extra_body` — top-level `enable_thinking=false` (**SiliconFlow / DashScope hybrid models** like `deepseek-ai/DeepSeek-V4-*` on `api.siliconflow.cn` — the knob the nested `chat_template_kwargs` shape does NOT reach), `chat_template_kwargs.enable_thinking=false` (vLLM/Qwen native), and `thinking.type="disabled"` (Zhipu GLM) — even for models that only declare `supports_thinking: true`. For the **web-UI path** (which sends `thinking_enabled=false` but NOT `force_disable`), SiliconFlow models need `when_thinking_disabled: {extra_body: {enable_thinking: false}}` in their `config.yaml` entry (added to `deepseek-ai/DeepSeek-V4-Flash`) for thinking-off to take effect. Sets `is_scheduled_run=true` internally to skip the auto-title LLM call. **Generated-file return (`include_files=true`, default on)**: an agent that produces its report as a file (writes to `/mnt/user-data/outputs/` + `present_files`) returns only a short summary in `answer`; to recover the real deliverable both endpoints now also return `files: [{name, virtual_path, encoding(text|base64), content}]` — `_collect_output_files` scans the run's `state['thread_data']['outputs_path']` (the open thread is stateless so artifacts can't be fetched by thread, but the endpoint runs server-side and the sandbox already wrote the files to disk), text decoded as utf-8 else base64, capped by `_MAX_OUTPUT_FILES=20` / `_MAX_OUTPUT_FILE_BYTES=5MB`. `/chat` puts `files` on the response; `/chat/stream` puts it in the `event: result` payload. A run with no answer **and** no files now 502s ("…文本答案或文件"). Router `app/gateway/routers/open_chat.py`; standalone caller `test_open_chat_api.py` (repo root, streaming by default); tests `tests/test_open_chat.py`. |
|
||||
| **Open 3qfx Ask** (`/api/open/3qfx/ask`) | **Unauthenticated** open entry that, given just a **任务id**, directly kicks off the 3qfx(三情分析)「第一次问答」as a **real persisted conversation** (sibling of `/api/open/chat`; same `/api/open/` auth-whitelist + CSRF-exempt). `POST /ask` `{task_id, message?, model_name?, thinking_enabled=false}` → (1) resolves the `3Q` agent from `app.state.business_mapping_store.get_mapping("3Q")` (the **same** agent the frontend deep-link `goPath=3qfx` uses; 503 if unconfigured); (2) **fetches the task detail server-side** (consumer `cop-task-detail`) and builds the opening text **identical to the frontend** `buildOpeningText` 3qfx branch — `任务名称:…\n任务内容:…\n任务id(task_id)为{task_id}` (an explicit `message` overrides and **skips** the fetch); (3) creates a fresh thread tagged `metadata.{taskId, agent_id}` (the exact filter `useTaskThreads` uses, and `metadata.taskId` makes it task-deeplink-shared / creator-path-resolved) and **starts a background persisted run** via `start_run(..., enforce_access=False)` (writes the checkpointer, runs under the `"default"` bucket); (4) **returns immediately** `{thread_id, run_id, agent_id, status, opening_text}` — the caller just needs "started", then watches the run in the 3qfx 任务工作区 conversation list. `start_run` gained a keyword-only `enforce_access` (default `True`); the open entry passes `False` so the admin-configured agent isn't rejected by the per-(missing)-login visibility check. **Consumer fetch config** lives in `config.yaml → task_deeplink`: `cop_task_detail_url` (full URL, appends `?taskId=`), `cop_task_detail_authorization` (default `Bearer admin`), `cop_task_detail_test` + `mock.cop_task_detail` (offline fake when the upstream is unreachable — on in this deployment). Router `app/gateway/routers/open_3qfx.py`; tests `tests/test_open_3qfx.py`. |
|
||||
@@ -419,14 +459,14 @@ FastAPI application on port 8001 with health checks at `GET|HEAD /health`, `/api
|
||||
| **HTML Page Favorites** (`/api/html-page-favorites`) | Bookmarked generated HTML pages, owner-scoped. `GET /` - list; `POST /` `{page_id, title?, description?, tags?}` - bookmark a `scheduled_html_pages` row (extracts a reusable style/layout summary via GLM5 at favorite time); a page can only be favorited once per user (`DuplicateFavoriteError` ? HTTP 409). `GET/DELETE /{id}` - read/delete (the create-task reference picker deletes favorites here). A favorite can be selected as a style reference when creating a new `html_page` task. |
|
||||
| **Public Scheduled Tasks** (`/api/public/scheduled-tasks`) | **Unauthenticated** mirror for embedding. Adds `GET /runs/{rid}/html` and `GET /{task_id}/latest-html` for the `html_page` preview; the run result keeps `result.html` (only file paths are redacted). |
|
||||
| **Light Apps** (`/api/light-apps`) | Light-app registry (????? / ????), **global/shared** (no user scoping; `created_by`/`updated_by` audit-only). Two kinds: `window_open` (launched in a new tab from the ???? card grid) and `iframe` (embedded under a parent sidebar menu via `mount_location` and dynamically injected into the sidebar). Both carry `route_params` ? a list of `{key, paramType: login\|theme, source, customValue, themeValue}` descriptors the **frontend** resolves at launch time (login params read from `localStorage.userInfo`; theme params use a literal value) and appends to `url` as query string. `GET /` - list (filters `app_type`/`mount_location`/`enabled_only`/`search`); `GET /{app_id}` - read one (both open to any authenticated user); `POST /`, `PUT /{app_id}`, `DELETE /{app_id}` - **admin only**; iframe apps must carry a `mount_location` (400 otherwise); the unique `name` collides as HTTP 409. Backed by `deerflow.persistence.light_apps` (table `light_apps`, auto-created by `create_all`); store wired in `deps.py` as `app.state.light_app_store`. Tests: `tests/test_light_apps.py`. |
|
||||
| **Menu Overrides** (`/api/menu-overrides`) | Sidebar-menu overrides (菜单管理), **global/shared** (no user scoping; `updated_by` audit-only). The static menu tree (icons/routes/`adminOnly`) lives in the **frontend** code (`core/page-layout/sidebar-menu.ts`); iframe light-apps are injected at render time. This router persists a per-node **overlay** the frontend merges onto that tree: `parent_id` (re-parent, moves the whole subtree; NULL = top level), `sort_order` (reorder among siblings), `label` (rename; NULL keeps code label), `disabled` (停用 — hides the node **and its subtree**). A node with no row renders exactly as code defines it, so new code menus / newly-registered iframe apps appear automatically and are immediately configurable. `GET /` - list all overrides (open to any authenticated user); `PUT /` - **admin only**, atomically **replaces the whole set** (`{overrides: [...]}`); rejects re-parenting that would exceed **3 levels** or form a cycle (`_validate_depth`, HTTP 400). Backed by `deerflow.persistence.menu_overrides` (table `menu_overrides`, auto-created by `create_all`); store wired in `deps.py` as `app.state.menu_override_store`. Frontend: merge logic `core/page-layout/menu-overrides.ts` (`applyMenuOverrides` in `use-sidebar-menu.ts`), client `strategy-components/api/menu-overrides.ts`, admin page `pages/MenuManagementPage.tsx` (@dnd-kit drag-reorder/re-parent + 「移动到」parent selector + ↑↓ arrows + 改名/停用), entry in the bottom-left settings dropdown (`components/page-sidebar.tsx`, admin-only). Tests: `tests/test_menu_overrides.py`, `frontend-web/src/core/page-layout/menu-overrides.test.ts`. |
|
||||
| **Compact Menu Overrides** (`/api/compact-menu-overrides`) | Compact/sidebar-summary left-nav overlay (简介模式菜单), **global/shared**. Two independent sets keyed by `menu_type` (`A` default / `B`); saving one type does not wipe the other. Type B rows use a `B::` node_id prefix so they stay unique on databases that still use `node_id` as the sole PK. `GET ?menu_type=` lists one type (open); `PUT {menu_type, overrides}` admin-only replaces that type. Frontend: `MenuManagementPage` 简介模式 type switcher + `runtime-config.js` `VITE_COMPACT_MENU_TYPE`. Tests: `tests/test_compact_menu_overrides.py`. |
|
||||
| **Menu Overrides** (`/api/menu-overrides`) | Sidebar-menu overrides (菜单管理). `menu_overrides` is the administrator-maintained shared **default**; `user_menu_override_configs` stores one complete personal snapshot per user. `GET /` returns `overrides` (effective), `default_overrides`, `user_overrides`, `has_user_override`, and `editable_scope`. `PUT /` is scope-aware: administrators replace the sparse default overlay, while ordinary users replace only their own complete snapshot; an empty personal payload (or `DELETE /`) restores defaults. Personal rows win for known `node_id`s and default rows absent from the snapshot flow through, so newly-added menus follow administrator configuration without overwriting existing personal choices. The static tree (icons/routes/`adminOnly`) remains frontend-owned; iframe light-app nodes are supported; re-parenting is validated against cycles and the three-level cap. Both tables are auto-created by `create_all`. Frontend: `core/page-layout/menu-overrides.ts`, `use-sidebar-menu.ts`, `strategy-components/api/menu-overrides.ts`, and `pages/MenuManagementPage.tsx`; the settings entry is available to every user and labels the administrator view as default management. Tests: `tests/test_menu_overrides.py`, `frontend-web/src/core/page-layout/menu-overrides.test.ts`. |
|
||||
| **Compact Menu Overrides** (`/api/compact-menu-overrides`) | Compact/sidebar-summary menu overrides (简介模式菜单). `compact_menu_overrides` remains the administrator default for independent type `A` / `B` layouts; `user_compact_menu_override_configs` stores a complete snapshot per `(user_id, menu_type)`. Scope-aware `GET`/`PUT`/`DELETE` use the same effective/default/personal contract as the full menu: existing snapshot nodes stay personal and unseen new default nodes flow through. Type B default rows retain the historical `B::` storage prefix; public node ids never contain it. Frontend editing is fixed to the deployment's `runtime-config.js` `VITE_COMPACT_MENU_TYPE`. Tests: `tests/test_compact_menu_overrides.py`, `frontend-web/src/core/page-layout/compact-menu-overrides.test.ts`. |
|
||||
| **Sensitive Words** (`/api/sensitive-words`) | Sensitive-word / desensitize map (敏感词管理 / 脱敏词), **global/shared** (no user scoping; `updated_by` audit-only). Each row maps an original `word` → `replacement`, with `enabled` (停用 parks a word without deleting) + `sort_order`. The **frontend** ships a seed map in code (`src/lib/desensitize.ts` `SEED_DESENSITIZE_MAP`) used as the default on a fresh deployment; once an admin saves, this table is the source of truth and the frontend loads it at startup (`DesensitizeWordsLoader` → `replaceDesensitizeMap`, **enabled** rows only). `GET /` - list all words (open to any authenticated user — every page desensitizes); `PUT /` - **admin OR the `lqq` account** (`_require_manager`: `system_role == "admin"` or email `@`-prefix `== "lqq"`), atomically **replaces the whole set** (`{words: [...]}`, de-duped by `word`, last wins). Backed by `deerflow.persistence.sensitive_words` (table `sensitive_words`, PK `word` `String(191)`, auto-created by `create_all`); store wired in `deps.py` as `app.state.sensitive_word_store`. Frontend: client `strategy-components/api/sensitive-words.ts`, page `pages/SensitiveWordsPage.tsx` (route `/page/strategy/admin/sensitive-words`, entry in the bottom-left settings dropdown — **gated on username `lqq`, not admin role**, present in both sidebar modes; 「导入种子词」 merges missing seed entries). Tests: `tests/test_sensitive_words.py`. |
|
||||
| **Sentiment Agent Proxy** (`/api/sentiment-agent`) | BFF for the frontend 舆情分析 virtual agent. `POST /stream` accepts the AG-UI body plus `apiUrl`/`authUsername`/`authPassword` from `runtime-config.js`, forwards to that URL with TLS verify off, and byte-pipes upstream SSE. Connection/HTTP failures return the full error in `detail`; mid-stream failures emit AG-UI `RUN_ERROR`. Distinct from `skill-packages/sentiment-analysis.skill`. Tests: `tests/test_sentiment_agent_proxy.py`. |
|
||||
| **Dashboard Sessions** (`/api/dashboard-sessions`) | 「大屏绘制智能体」独立页面会话,**按外部 `task_id` 存取,不按 user 分权**(open read + shared writes;`user_id` audit-only)。一个 task 一行(upsert):`transcript`(聊天记录 JSON)+ `report_json`(最新结构化数据,可解析 = 「有效结果」)。`GET /by-task/{task_id}` - 取该 task 会话(无则 404);`PUT /by-task/{task_id}` - upsert(`{transcript?, report_json?}`,只覆盖传入项)。Backed by `deerflow.persistence.dashboard_sessions`(table `dashboard_sessions`, auto-created by `create_all`,无迁移);store wired in `deps.py` as `app.state.dashboard_session_store`。前端独立页 `roundtable-planning/pages/DashboardAgentPage.tsx`(路由 `/page/strategy/dashboard-agent?taskId=`,侧边栏「问答管理 → 大屏绘制」)让智能体分析任务报告 JSON → report-json → 渲染 `DashboardApp`。Tests: `tests/test_dashboard_sessions.py`。 |
|
||||
| **Task Reports** (`/api/task-reports`) | 任务级研判情报报告详情,**按 `task_id` 共享、不按 user 分权**(写入者只留 `updated_by` 审计)。`GET /by-task/{task_id}` 始终返回原 Consumer 兼容外壳 `{state:"200", msg:"操作成功!", data:[...]}`(未保存为 `data: []`);`POST /by-task/{task_id}` 首次保存,已有报告返回 409;`PUT /by-task/{task_id}` 原子替换完整 `data` 数组。`POST /import/by-task/{task_id}` 面向 TaskCOP 导入页,接收 `sq-report-mock.json` 兼容的 `data`/`records` 数组并完整覆盖目标任务;路径中的选定任务 id 优先于文件内的 `taskId`,避免误写。每一项严格是 `{id, taskId, createTime, updateTime, sessionId, contentJson, categoryType}`;`contentJson` 以 JSON **字符串**原样落在 `records_json`,避免破坏大屏按类目解析的输入格式。保存或导入非空 `data` 后,router 会通过 `app.state.taskcop_task_store.mark_analysis_completed` 将同 id 的 TaskCOP 任务升级至状态 `25`,使旧任务列表显示并筛选为“已完成”;普通保存时不存在匹配任务的报告仍可独立保存,导入则要求目标任务存在。Backed by `deerflow.persistence.task_reports`(table `task_reports`, auto-created by `create_all`);store wired in `deps.py` as `app.state.task_report_store`。前端客户端 `strategy-components/api/task-reports.ts`,`core/auth/dashboard-input.ts` 与 rwfx 3q 完成检查均直接读此接口,不再依赖 Consumer 或本地 sq-report mock。Tests: `tests/test_task_reports.py`。 |
|
||||
| **Task Buttons** (`/api/task-buttons`) | Task-workspace configurable jump buttons (按钮管理), **global/shared** (no user scoping; `updated_by` audit-only). Each row = one jump button in a task deep-link workspace's bottom 快捷跳转 bar: `{id, business (rwfx/3qfx/xdfx), label, link_type (url/business/purpose/host), target, append_task_id, id_param (editable query-param name, e.g. id/task_id), id_kind (task/action — only xdfx differs, its deep-link id is an 行动id), open_mode (blank/self), login_params, append_auth, auth_token_param, auth_name_param, enabled, sort_order}`. **Two independent, coexisting login-append mechanisms for `url` buttons** — both resolved frontend-side: (1) `login_params` (JSON column `login_params_json`, `PortableJSON`, migration `20260625_01`) = a list of `{key, source, custom_value}` resolved at click time (`source` = a `userInfo` field or `other`→literal `custom_value`) and appended to the target URL — mirrors the light-app login params; empty = nothing appended. (2) `append_auth` + `auth_token_param` + `auth_name_param` (columns, migration `20260625_02`, default off) — appends `?{auth_token_param}={token1}&{auth_name_param}={username}` so the target system can identify the current user; gated on **token login** — the frontend only stashes `token1` (the `?authToken=`) + the raw `getTokenInfo` username (`login.token1`/`login.username`) when login went through `/login/token`, so a non-token-login session appends neither. The **frontend** holds code-seed defaults (rwfx's 4 historical buttons built from the live `task_deeplink` config); only when this table is empty do the workspace + admin page fall back to them, else this table is the sole source of truth. `GET /` - list all (open to any authenticated user — the workspace renders them); `PUT /` - **admin only** (`_require_admin`), atomically **replaces the whole set** (`{buttons: [...]}`, de-duped by `id`, last wins). Backed by `deerflow.persistence.task_buttons` (table `task_buttons`, auto-created by `create_all`); store wired in `deps.py` as `app.state.task_button_store`. Frontend: helpers `core/auth/task-buttons.ts`, client `strategy-components/api/task-buttons.ts`, admin page `pages/TaskButtonsPage.tsx` (route `/page/strategy/admin/task-buttons`, entry「按钮管理」in the bottom-left settings dropdown, admin-only, both sidebar modes; per-business tabs + 「导入默认」). Tests: `tests/test_task_buttons.py`. |
|
||||
| **Fixed Questions** (`/api/fixed-questions`) | Administrator-managed exact-match fixed Q&A (问题配置), **global/shared**. Each row is `{id, question, answer, enabled, tokens_per_second, sort_order}`; stream speed is configurable per row from `1–1000 token/s` and defaults to `100`. `GET /` and whole-set atomic `PUT /` are both admin-only. At the start of an external chat run, `services._find_fixed_answer` checks the latest eligible plain-text human message with exact, case/space/punctuation-sensitive equality. Enabled matches switch the run to `make_fixed_answer_agent`: a local `BaseChatModel` tokenizes with `cl100k_base`, replays the answer in rate-limited token chunks, and keeps the existing LangGraph SSE `messages`/`values` protocol, run journal, checkpoint history, deterministic local title, and refresh behavior intact while no provider/model request or model-concurrency slot is used. File/multimodal/context-prefixed turns, `canvas_agent`, `ai_writing`, writing mode, resume commands, and internal orchestration/background child runs bypass matching. Storage failures fail open to the normal model path. Backed by `deerflow.persistence.fixed_questions` (table `fixed_questions`, SHA-256 indexed lookup plus exact original-string collision check, migrations `20260821_01` + `20260821_02`), wired as `app.state.fixed_question_store`. Frontend: `strategy-components/api/fixed-questions.ts`, admin page `pages/FixedQuestionsPage.tsx`, route `/page/strategy/admin/fixed-questions`, entries in both full-layout and compact-workspace settings menus. Tests: `tests/test_fixed_questions.py`. |
|
||||
| **Fixed Questions** (`/api/fixed-questions`) | Administrator-managed exact-match fixed Q&A (问题配置), **global/shared**. Each row is `{id, question, answer, enabled, tokens_per_second, sort_order}`; stream speed is configurable per row from `1–1000 token/s` and defaults to `100`. `GET /` and whole-set atomic `PUT /` are both admin-only. At the start of an external chat run, `services._find_fixed_answer` checks the latest eligible plain-text human message with exact, case/space/punctuation-sensitive equality. Enabled matches switch the run to `make_fixed_answer_agent`: a local `BaseChatModel` uses a lossless standard-library-only word/CJK/punctuation splitter for rate-limited chunks. It must never initialize a model tokenizer such as `tiktoken.get_encoding`, because an empty tokenizer cache can trigger a public vocabulary download and break offline deployments. The local graph keeps the existing LangGraph SSE `messages`/`values` protocol, run journal, checkpoint history, deterministic local title, and refresh behavior intact while no provider/model request or model-concurrency slot is used. File/multimodal/context-prefixed turns, `canvas_agent`, `ai_writing`, writing mode, resume commands, and internal orchestration/background child runs bypass matching. Storage failures fail open to the normal model path. Backed by `deerflow.persistence.fixed_questions` (table `fixed_questions`, SHA-256 indexed lookup plus exact original-string collision check, migrations `20260821_01` + `20260821_02`), wired as `app.state.fixed_question_store`. Frontend: `strategy-components/api/fixed-questions.ts`, admin page `pages/FixedQuestionsPage.tsx`, route `/page/strategy/admin/fixed-questions`, entries in both full-layout and compact-workspace settings menus. Tests: `tests/test_fixed_questions.py`. |
|
||||
| **Knowledge** (`/api/knowledge`) | Product knowledge base, **global/shared** (no user isolation; writes record `created_by`/`updated_by` for audit). `POST /ingest/thread/{thread_id}` - sediment one thread into a note (reads messages from the checkpointer, `threads:read` owner-checked); `POST /ingest/threads/batch` - batch-sediment the caller's recent threads (`skip_existing` via `.manifest.json` dedup). **Defaults to `background=true`**: the slow per-thread extraction loop is deferred to a FastAPI background task and the endpoint returns `{queued:true, processed:N}` immediately (each thread's transcript is truncated to `max_input_chars` before the LLM call). Running the loop inline for a large batch outlives the nginx gateway timeout (504), which the frontend's `parseResponse` surfaces as a generic "Request failed"; `background=false` keeps the blocking path (full per-thread `created/skipped/failed` tally) for small batches/scripts. Shared loop: `_run_batch_ingest`. **Optional model selection**: both ingest endpoints accept `model_name` (overrides `knowledge.extract_model_name` for that capture; threaded through `KnowledgeService.capture_thread` ? `extract_thread_llm_multi`). **Model-error surfacing**: when the chosen model produces nothing usable (timeout / API error / non-JSON) capture still **falls back to rule-based extraction and lands a draft** (`llm_fallback:true`); the single-thread (non-background) response and the batch response (`llm_fallback` count) carry this so the UI tells the user "???????". The "??????" chat button stays background/fire-and-forget (passes `model_name`, no per-save error toast); the batch-import dialog is synchronous and shows the fallback count. `POST /ingest/file` - **import an uploaded file** (multipart: `file` + optional `title`/`tags`/`status`/`model_name`) into wiki note(s): the file is converted to Markdown (md/txt read directly, pdf/word/ppt/excel via `convert_file_to_markdown`), distilled by the same wiki-capture LLM pipeline as thread capture (`source_type="file"`), and falls back to a verbatim **draft** note when the model returns nothing usable (`llm_fallback:true`); `POST /ingest/search` - sediment search/tool results as a reference note; `POST /notes` - manual note; `GET /notes` (keyword/tag/source_type/status + pagination); `GET/PUT/DELETE /notes/{id}` (DELETE is soft = archive, `?hard=true` removes the vault file); `POST /search` - `mode=keyword\|vector\|hybrid` (vector?keyword fallback when no embedding endpoint); `GET /notes/{id}/versions` - edit history; `GET /duplicates` + `POST /merge` - dedup; `POST /reindex` - rebuild vector index; `GET /graph` - product graph data (notes + entities, graph-blacklist filtered); `GET/POST /graph/blacklist` + `DELETE /graph/blacklist/{id}` - graph node blacklist (labels = entity names / note titles, case-insensitive match; blacklisted nodes and their edges are dropped from `/graph` and the wiki-export artifacts; table `knowledge_graph_blacklist`, auto-created); `POST /export/graph` + `GET /export/files/{graph.json\|graph.html\|graph.graphml\|cypher.txt}` - obsidian-wiki `wiki-export` artifacts (self-contained `graph.html` for iframe); `POST /resolve` `{targets:[?]}` + `GET /notes/{id}/wikilinks` - resolve `[[wikilink]]` targets to a note id / vault page / missing (powers clickable wikilinks); `GET /vault-page?path=` - read a raw vault meta/hub Markdown page (no DB note). `GET/POST /folders` + `PUT/DELETE /folders/{id}` - user-managed directories (notes carry a `folder` path; `GET /notes?folder=` filters a dir + sub-dirs); `GET/POST /extract-templates` + `GET/PUT/DELETE /extract-templates/{id}` - editable extraction-prompt templates (thread/document ingest accept `template_id`; capture/PUT-note accept `folder`). Backed by the `deerflow.knowledge` module (see below). |
|
||||
|
||||
Proxied through nginx: `/api/langgraph/*` ? LangGraph, all other `/api/*` ? Gateway.
|
||||
@@ -563,18 +603,26 @@ frontend can hide or show the entry consistently; `PUT` is admin-only and
|
||||
partially updates `reference_mode_enabled` and/or `cleanup_words`. The normal
|
||||
workspace input box, iframe chat toolbar, send context, right reference panel,
|
||||
and appearance settings all use this flag, while already locked citation
|
||||
threads and selected LLMWiki knowledge bases still render their source
|
||||
documents in the reference panel. Tests:
|
||||
threads still render the generic reference panel. A non-empty selected
|
||||
LLMWiki/assistant knowledge-space set instead carries an explicit Wiki reference
|
||||
contract: the frontend filters every batch to those selected server-authored ids
|
||||
and renders a flat Wiki table of contents plus inline Wiki reader, never the
|
||||
generic reference-card component. Projects enable this only when their project
|
||||
or bound-agent knowledge ids are actually non-empty. Tests:
|
||||
`tests/test_prompt_prefix.py` and
|
||||
`frontend-web/src/pages/iframe-chat-mode-buttons.test.ts`.
|
||||
|
||||
### Agent Module Tag Config (zzJC / zzXD 按类型聚合)
|
||||
|
||||
Backs the zzJC / zzXD agent-chat left sidebar (「按类型聚合智能体 + 聊天记录」). An admin picks which agent **tags** belong to a module; the module's left sidebar then shows only agents under those tags (empty/unconfigured = show all visible agents). Stored in `system_settings.json` → `agent_modules` (`AgentModuleSettings`: `modules: {module_key → [tag_id, ...]}`); pure helpers `get_module_tag_ids` / `set_module_tag_ids` (de-dupe + strip). Endpoints (in `app/gateway/routers/system_settings.py`): `GET /api/system-settings/agent-modules/{key}` (readable by any user — the sidebar needs it to filter), `PUT /api/system-settings/agent-modules/{key}` (admin only, full replacement of the tag-id list). Module keys are frontend-defined (`AGENT_MODULES` in `components/workspace/agents/agent-module-sidebar.tsx`, keyed by the module's landing agent id: `6bf`→`zzjc`, `10d69…`→`zzxd`). Tests: `tests/test_agent_modules.py`. Frontend: `core/system-settings` (`useAgentModuleConfig`/`useUpdateAgentModuleConfig`), admin `AgentModuleSettingsDialog` (gear button in the agent-chat header).
|
||||
Backs the zzJC / zzXD agent-chat left sidebar (「按类型聚合智能体 + 聊天记录」). An admin picks which agent **tags** belong to a module. The module's sidebar then shows only agents under those tags (empty/unconfigured = show all visible agents). Stored in `system_settings.json` → `agent_modules` (`AgentModuleSettings`: `modules: {module_key → [tag_id, ...]}`); pure helpers `get_module_tag_ids` / `set_module_tag_ids` (de-dupe + strip). Endpoints (in `app/gateway/routers/system_settings.py`): `GET /api/system-settings/agent-modules/{key}` (readable by any user — the sidebar needs it to filter), `PUT /api/system-settings/agent-modules/{key}` (full replacement; admin only). Module keys are frontend-defined (`AGENT_MODULES` in `components/workspace/agents/agent-module-sidebar.tsx`). Tests: `tests/test_agent_modules.py`. Frontend: `core/system-settings` (`useAgentModuleConfig`/`useUpdateAgentModuleConfig`) and `AgentModuleSettingsDialog`.
|
||||
|
||||
### wx common-agent category
|
||||
|
||||
The fixed-deep-blue wx full-chat route (`#/page/workspace/wx/chats/new`) hides the global header and page sidebar while retaining its workspace common-agent panel. When its optional category is unset, the panel preserves the original recommendation logic (pinned → mine → square, deduplicated). A saved category selects one Agent Gallery category: `mine`, `sentiment`, or `square:<square_id>` (for example the configured `默认广场`). It is stored in `system_settings.json` → `wx_common_agents` (`WxCommonAgentsSettings`), validated against the effective publish-square list, and exposed by `GET`/`PUT /api/system-settings/wx-common-agents`. The `wx` account and administrators can update or clear it; all users can read it. The frontend uses `useWxCommonAgents` and displays the settings gear only to wx. Tests: `tests/test_wx_common_agents.py`.
|
||||
|
||||
### Business-mapping authorization
|
||||
|
||||
`/api/business-mapping` remains globally readable. Updates to 3Q, 6BF, and 7BF require `system_role == "admin"`; the 8BF public business-chain mapping is intentionally different and can be updated only by the trusted authenticated account whose email prefix is `lqq`. Do not rely on a username supplied in an external login URL. The frontend follows the same email-prefix rule when deciding whether to render the 8BF configuration button, while the router (`app/gateway/routers/business_mapping.py`) remains the enforcement point. Tests: `tests/test_business_mapping.py`.
|
||||
`/api/business-mapping` remains globally readable. Seeded rows include the main codes (3Q / 6BF / 7BF / 8BF / AFX) plus independent slots `3q-1…3q-5`, `6-1…6-4`, and `7-1…7-4` (empty agent/chain until configured on the 配置管理 page). Updates to 3Q, 6BF, 7BF, and those independent slots require `system_role == "admin"`; the 8BF public business-chain mapping is intentionally different and can be updated only by the trusted authenticated account whose email prefix is `lqq`. Do not rely on a username supplied in an external login URL. The frontend follows the same email-prefix rule when deciding whether to render the 8BF configuration button, while the router (`app/gateway/routers/business_mapping.py`) remains the enforcement point. Tests: `tests/test_business_mapping.py`.
|
||||
|
||||
### Position-collaboration foundation
|
||||
|
||||
@@ -831,6 +879,7 @@ MySQL index script synchronized.
|
||||
- **Location**: `deer-flow/skills/{public,custom}/`
|
||||
- **Format**: Directory with `SKILL.md` (YAML frontmatter: name, description, license, allowed-tools)
|
||||
- **Loading**: `load_skills()` recursively scans `skills/{public,custom}` for `SKILL.md`, parses metadata, and reads enabled state from extensions_config.json. **A directory containing `SKILL.md` is a self-contained leaf** — discovery does not descend into its subdirectories, so a skill *package* that bundles its own `examples/`/`references/` sub-skills (each with a SKILL.md) surfaces as **one** skill, not a swarm of phantom entries.
|
||||
- **Online/offline lifecycle**: reuse `Skill.enabled`; do not add a parallel lifecycle flag and do not substitute `published` or curator `state`. `PUT /api/skills/{name}` is a global switch: administrators may toggle every skill, while an ordinary user may toggle only a non-built-in `custom` skill whose persisted `owner_user_id` matches that user. It patches only `skills.<name>.enabled` in the raw `extensions_config.json`, preserving MCP placeholders and unrelated/unknown fields, then reloads configuration and refreshes the lead prompt cache. Writes use atomic replacement on ordinary files and an `fsync` in-place fallback for writable Docker single-file bind mounts; supported launchers instead keep the runtime file under a writable persisted config directory. Offline skills remain on disk and in the full catalogue (`load_skills(enabled_only=False)`), and existing agent/workflow references are deliberately retained so bringing the skill online restores them. Every model-facing path must use the enabled catalogue: prompt injection, `search_skills`, `skill_list`, `skill_view`, subagents, AI-writing/deep-research/roundtable discovery, and Workflow run-start resource validation. Direct sandbox access is enforced by a copied online-only view under writable `DEER_FLOW_HOME`; `/mnt/skills` in LocalSandbox and AIO/remote sandboxes must point only to that view, never to the canonical tree. AIO creation/reuse attests the view marker and fails closed when an older provisioner ignores the requested mount; explicitly declared internal support libraries (such as `task-report-lib`) enter the view only while a dependent skill is online. Path-sensitive `DEER_FLOW_*` variables and `PYTHONPATH` are removed from LocalSandbox bash while non-path settings such as `DEER_FLOW_GATEWAY_PORT` remain. LocalSandbox host bash still shares the backend UID and is not an OS isolation boundary; production requiring hostile-code isolation must use an attested AIO sandbox. Frontend controls are in the administrator 技能管理 list and, for owners, 「技能 → 技能列表 → 我的」; offline rows and retained agent references remain visible with an 已下线 badge and can be removed, while new selectors exclude them.
|
||||
- **Discovery↔delete consistency**: a skill is keyed for discovery by its frontmatter `name`, but its on-disk directory name may differ (hand-dropped / mis-extracted package). `LocalSkillStorage._resolve_skill_dir(name, category)` resolves the *discovered* directory (canonical `<category>/<name>` fast-path, else scan-by-frontmatter-name); `custom_skill_exists`/`read_custom_skill`/`delete_custom_skill` all go through it, so **anything that gets listed can also be read and deleted** regardless of directory name (the old code assumed `custom/<name>/` and 404'd "Custom skill 'x' not found." on any non-canonical layout). Delete is path-guarded to never escape the `custom/` root. Tests: `tests/test_skill_storage_delete.py`.
|
||||
- **Injection**: Enabled skills listed in agent system prompt with container paths
|
||||
- **Configurable `es_query` routing prompt**: `config.yaml → skills.es_query_routing`
|
||||
@@ -1028,9 +1077,11 @@ scheduler runs once per deployment. Per run:
|
||||
|
||||
### vLLM Provider (`packages/harness/deerflow/models/vllm_provider.py`)
|
||||
|
||||
- `VllmChatModel` subclasses `langchain_openai:ChatOpenAI` for vLLM 0.19.0 OpenAI-compatible endpoints
|
||||
- Preserves vLLM's non-standard assistant `reasoning` field on full responses, streaming deltas, and follow-up tool-call turns
|
||||
- Designed for configs that enable thinking through `extra_body.chat_template_kwargs.enable_thinking` on vLLM 0.19.0 Qwen reasoning models, while accepting the older `thinking` alias
|
||||
- `VllmChatModel` subclasses `langchain_openai:ChatOpenAI` for vLLM OpenAI-compatible endpoints
|
||||
- Thinking text is accepted from either `reasoning` (newer vLLM parsers) or `reasoning_content` (older vLLM / DeepSeek-style). The UI still reads canonical `additional_kwargs.reasoning_content`. Follow-up turns echo only the wire field that actually arrived, so other ChatOpenAI models are unchanged
|
||||
- Shared helpers live in `deerflow/models/reasoning_fields.py` and are also used by `PatchedChatOpenAI` (the default `langchain_openai:ChatOpenAI` patch)
|
||||
- Designed for configs that enable thinking through `extra_body.chat_template_kwargs.enable_thinking` on vLLM Qwen reasoning models, while accepting the older `thinking` alias
|
||||
- Per-model switch: set only that entry's `use: deerflow.models.vllm_provider:VllmChatModel`. Do not change SiliconFlow / DeepSeek entries that already use `langchain_openai:ChatOpenAI`
|
||||
|
||||
### IM Channels System (`app/channels/`)
|
||||
|
||||
@@ -1140,7 +1191,47 @@ Both can be modified at runtime via Gateway API endpoints or `DeerFlowClient` me
|
||||
|
||||
### Conditional WeKnora LLMWiki
|
||||
|
||||
Ordinary knowledge-base cards expose an owner/admin-only name/description edit dialog in `WeKnoraKnowledgeWorkspace.tsx`, reusing `useLlmWikiActions().update` and `PATCH /api/llmwiki/knowledge-bases/{id}`. The route trims names, rejects blank or reserved conversation-deposit names, writes to WeKnora before updating local metadata, and retains existing write authorization. A successful remote list is authoritative for every list scope, including `personal`: absent remote IDs are hidden without deleting mappings/references. Upstream failures remain errors. The grid also filters cached `remote_status=missing` rows and no longer renders stale-record cleanup cards. Regression coverage: `tests/test_llmwiki_knowledge_management.py`.
|
||||
**Selected knowledge-space policy:** if either `llmwiki_knowledge_base_ids` or
|
||||
`assistant_knowledge_base_ids` is non-empty, Gateway sets
|
||||
`knowledge_space_strict`. This is server policy, not a caller preference:
|
||||
after ordinary context merging, Gateway overwrites both final
|
||||
`configurable` and `context` with the authorized knowledge ids,
|
||||
`knowledge_space_strict=true`, and the merged exclusion list, so a request
|
||||
cannot pre-seed `false`, forged ids, or `excluded_tools=[]` to reopen external
|
||||
retrieval. The lead agent must fail closed: no Web/browser
|
||||
tools, MCP, external retrieval skills, ACP, deep-research pipeline, or
|
||||
subagents. Only the pre-retrieved selected-space context and the read-only
|
||||
`llmwiki_search` compatibility tool may retrieve knowledge; that tool supports
|
||||
both ordinary and assistant Wiki stores and cannot accept IDs from the model.
|
||||
Ordinary Wiki retrieval runs vector and local lexical channels concurrently,
|
||||
uses RRF, reports raw vector threshold diagnostics, and continues serving a
|
||||
coherent last-successful snapshot after a later sync is cancelled/failed.
|
||||
`config.yaml` defaults the Wiki encoder to SiliconFlow `BAAI/bge-m3`; keep
|
||||
`SILICONFLOW_API_KEY` outside Git. The frontend derives four starter questions
|
||||
from the selected space names and hides generic recommendations while these
|
||||
questions are present. Design and remaining graph/profile phases are in
|
||||
`docs/KNOWLEDGE_SPACE_RETRIEVAL_AND_WIKI_EXPANSION_ZH.md`.
|
||||
|
||||
**Embedding ownership:** WeKnora's per-knowledge-base `embedding_model_id`
|
||||
indexes uploaded source documents; `llmwiki.local_wiki_index.embedding` indexes
|
||||
the generated Wiki-page mirror inside DeerFlow. Do not couple these two model
|
||||
choices. `GET /api/llmwiki/embedding-models` returns a strict credential-free
|
||||
projection of WeKnora Embedding models, and KB creation must validate and pass
|
||||
the selected active model id. When the caller omits a choice, retain WeKnora's
|
||||
active-default behavior. Existing KBs cannot change this binding in place, and
|
||||
cross-KB WeKnora searches should use model-compatible knowledge bases.
|
||||
|
||||
Wiki-enabled knowledge bases accept two bounded, user-owned prompt extensions:
|
||||
`wiki_config.content_instructions` (generated-page language, structure and
|
||||
emphasis) and `wiki_config.extraction_instructions` (domain entities/concepts
|
||||
to prioritize). The create/edit dialogs expose both values; Gateway accepts
|
||||
only those two fields and `WeKnoraClient` merges edits into the remote
|
||||
`wiki_config` so `synthesis_model_id`, extraction granularity and WeKnora-owned
|
||||
runtime knobs are never dropped. WeKnora's upload/reparse Wiki ingest is the
|
||||
actual consumer: saving instructions affects later ingests and does not itself
|
||||
rewrite existing pages.
|
||||
|
||||
Ordinary knowledge-base cards expose an owner/admin-only name/description/Wiki-instructions edit dialog in `WeKnoraKnowledgeWorkspace.tsx`, reusing `useLlmWikiActions().update` and `PATCH /api/llmwiki/knowledge-bases/{id}`. The route trims names, rejects blank or reserved conversation-deposit names, writes to WeKnora before updating local metadata, and retains existing write authorization. Default test questions are derived from the authoritative remote display name. The editor omits `test_questions` unless the user actually changes that field, and `knowledge_profile.generated_test_questions_name` recognizes only the exact legacy four-question template as automatic; genuinely edited lists remain custom. `_mapping_view` must resolve both questions and `test_questions_customized` from a profile row containing the current remote name, so a rename immediately replaces old-name defaults without a migration. A successful remote list is authoritative for every list scope, including `personal`: absent remote IDs are hidden without deleting mappings/references. Upstream failures remain errors. The grid also filters cached `remote_status=missing` rows and no longer renders stale-record cleanup cards. Regression coverage: `tests/test_llmwiki_knowledge_management.py`, `tests/test_llmwiki_weknora.py`.
|
||||
|
||||
`llmwiki.weknora.api_base_url` is the provider switch: non-empty enables WeKnora, empty preserves the legacy LLMWiki UI. WeKnora is independently deployed and owns PostgreSQL/Redis/object storage/vector storage/parsers/models; DeerFlow config contains only `api_base_url` and optional `web_base_url`. DeerFlow deliberately sends no WeKnora credential, so the WeKnora endpoint must be restricted to a trusted network and allow DeerFlow traffic itself or through its reverse proxy. There is intentionally no `default_embedding_model_id`: `WeKnoraClient` queries `/api/v1/models` and uses WeKnora's active Embedding model when creating a KB; Wiki creation additionally resolves an active KnowledgeQA model inside WeKnora for synthesis. The native BFF is `app/gateway/routers/llmwiki.py`; ownership/publication mappings live in `deerflow.persistence.llmwiki`; the browser only sees DeerFlow mapping IDs. Knowledge-base owners publish immediately (`private` -> `published`) without a review queue, and can unpublish ordinary bases. The conversation-deposit KB is system-owned, always published, shared by all DeerFlow users, excluded from My Knowledge, and immutable through user/admin detail operations; recognition requires both the reserved name and `system` owner. Existing same-name remote mappings are adopted in place as the system deposit, preserving mapping ids and document links. Its user-facing description, generated Markdown source and tags use `cmzs` branding; list/detail reads synchronize stale remote descriptions, and search results, source previews and embedded-detail JSON recursively replace legacy brand text in historical conversation-deposit payloads. Deposit requests must reference a thread accessible to the current account. Personal knowledge-base list/search views are scoped to the current DeerFlow account even for admins; frontend LLMWiki caches include the live account id, and admin write power does not make other users' private KBs appear as owned. Only `system_role === "admin"` sees the WeKnora connection-config tab. Knowledge cards navigate to a dedicated WeKnora-style detail route. Wiki-enabled bases render and default to `Wiki -> documents -> graph`; standard bases omit Wiki and render `documents -> graph`. The detail page supports sequential multi-file upload with an aggregate result for writable ordinary bases, hides it for public read-only/system-deposit bases, and uses a generic loading label while the embed session starts. It also supports URL and published manual-Markdown imports, browses real Wiki pages/content, proxies authenticated original-file previews, pages through `/chunks/{knowledge_id}`, and renders the Wiki page graph from `/knowledgebase/{kb_id}/wiki/graph`; document details use a right sheet rather than the removed KB modal. The credential-free proxy uses one WeKnora service identity, so per-user separation is enforced by DeerFlow mappings while all remote KBs physically share that identity's WeKnora workspace. WeKnora iframe detail pages first call `/api/llmwiki/knowledge-bases/{id}/weknora/session` through the normal DeerFlow API auth path, then load `/platform/knowledge-bases/{weknora_id}` with a signed `deerflow_weknora_embed` cookie for `/platform`, `/assets`, `/locales`, root assets, and API calls. Injected fetch/XHR rewriting sends all iframe `/api/v1/*`, including colliding `/api/v1/auth/*`, through `/api/llmwiki/weknora-embed/api/v1/*`; Gateway auth/CSRF bypass only applies to those signed iframe proxy requests, the proxy still revalidates the mapped KB, and cookie security honors `X-Forwarded-Proto` for HTTPS reverse proxies. `llmwiki_search` is conditionally registered in `tools.py`, is read-only, and rechecks the current DeerFlow user, agent `llmwiki_knowledge_base_ids`, and thread-context selection on every call. Source previews prefer the cleaned Wiki/LLMWiki page text when a retrieved chunk maps to a Wiki page, with raw split chunks as the fallback. Agent bindings use an inline WeKnora-style searchable/grouped multi-select in the standard agent create/detail UI; new chats use the same interaction pattern as a compact composer dropdown, and selections are written to thread metadata/context and restored on reopen. The DeerFlow frontend is native React and the WeKnora frontend is not modified or embedded. Full deployment and behavior notes: `docs/LLMWIKI_WEKNORA_INTEGRATION_ZH.md`. Tests: `tests/test_llmwiki_weknora.py`.
|
||||
|
||||
@@ -1300,13 +1391,28 @@ link can become `#/login/<user>?password=…`. Regression coverage:
|
||||
|
||||
### File Upload
|
||||
|
||||
Multi-file upload with automatic document conversion:
|
||||
- Endpoint: `POST /api/threads/{thread_id}/uploads`
|
||||
- Supports: PDF, PPT, Excel, Word documents (converted via `markitdown`)
|
||||
- Rejects directory inputs before copying so uploads stay all-or-nothing
|
||||
- Reuses one conversion worker per request when called from an active event loop
|
||||
- Files stored in thread-isolated directories
|
||||
- Agent receives uploaded file list via `UploadsMiddleware`
|
||||
Chat upload configuration is a database-backed global policy, not a static
|
||||
`config.yaml` size tuple. `deps.py` creates and seeds the singleton row, the
|
||||
Gateway loads it for every request, and `deerflow.uploads.policy` exposes only a
|
||||
validated runtime copy to the harness. The administrator page under settings
|
||||
edits or resets the same row. The built-in defaults are 20 files / 100 MiB total;
|
||||
documents and images are 10 MiB per file, and spreadsheets are 5 MiB.
|
||||
|
||||
- Endpoint: `POST /api/threads/{thread_id}/uploads`; policy endpoints:
|
||||
`GET/PUT /api/upload-policy`, `POST /api/upload-policy/reset`
|
||||
- Supported policy formats: TXT/Markdown, PDF, DOC/DOCX, PPT/PPTX, JPG/JPEG/PNG,
|
||||
and XLS/XLSX/CSV; the server restricts edits to parsers it can validate safely
|
||||
- The whole batch is streamed to a private staging directory, checked for count,
|
||||
total size, per-category size, extension, signature/container and encoding,
|
||||
then atomically committed; validation or sandbox-sync failure restores old files
|
||||
- Optional PDF/PPT/Excel/Word conversion always runs off the event loop with
|
||||
bounded concurrency and a fixed extracted-text ceiling; companions are named
|
||||
`report.docx.md`, with legacy `report.md` lookup retained for old uploads
|
||||
- Files stay in thread-isolated directories; `UploadsMiddleware` injects bounded
|
||||
JSON metadata, not unbounded bodies or repeated converted companions
|
||||
- Range-free large local reads return streaming full-file statistics and
|
||||
deterministic head/middle/tail samples. Explicit line reads, `grep`, artifact
|
||||
previews, image dimensions/derivatives and final model context all have hard caps
|
||||
|
||||
See [docs/FILE_UPLOAD.md](docs/FILE_UPLOAD.md) for details.
|
||||
|
||||
@@ -1336,7 +1442,25 @@ Automatic conversation summarization when approaching token limits:
|
||||
when it has already been folded into the hidden summary. This keeps strict
|
||||
OpenAI-compatible gateways from rejecting a tool-continuation call with
|
||||
`No user query found in message`.
|
||||
- LangChain keeps the agent prompt in `ModelRequest.system_message`, separate
|
||||
from `ModelRequest.messages`. Compression must merge its hidden summary into
|
||||
that dedicated slot and remove every `SystemMessage` from the ordinary
|
||||
message list. Strict Qwen-style adapters accept only one system message at
|
||||
index zero and otherwise return HTTP 400 (`system message must be at the
|
||||
beginning`). Keep this normalization adapter-independent; the payload-level
|
||||
OpenAI-compatible normalizer is a second line of defence and also merges
|
||||
adjacent leading system messages.
|
||||
- Keeps recent messages while summarizing older ones.
|
||||
- **The model input hard guard is independent and always on when configured.**
|
||||
Set `models[].max_input_tokens` (or `profile.max_input_tokens`) to the safe
|
||||
input ceiling; `max_tokens` is output-only. `ContextWindowGuardMiddleware`
|
||||
is mounted at the innermost end of both lead-agent and subagent model-call
|
||||
chains, after memory/RAG/image/custom injections. It counts the complete
|
||||
request (system prompt, messages/reasoning, tool schemas and response format),
|
||||
then transiently removes old complete turns and truncates oversized bodies as
|
||||
a last resort without mutating checkpoint state. Provider context-length
|
||||
errors are non-retriable so a tokenizer mismatch cannot trigger repeated
|
||||
submission of the same oversized request.
|
||||
|
||||
Tests: `tests/test_summarization_toggle.py` (gating + whitelist),
|
||||
`tests/test_summarization_middleware.py` (cache behavior).
|
||||
@@ -1575,9 +1699,11 @@ successful full scan, matching embedding fingerprint, no failed pages,
|
||||
`ready_page_count == local_page_count`, and vectors for a non-empty library.
|
||||
Ready mappings read DeerFlow's `llmwiki_wiki_pages` /
|
||||
`llmwiki_wiki_vectors` / `llmwiki_wiki_sync_states` tables; incomplete mappings
|
||||
call only WeKnora `/knowledgebase/{id}/wiki/pages?query=...` plus Wiki-page
|
||||
detail. Never call `/knowledge-search` from internal search, RAG, or the
|
||||
`llmwiki_search` tool, and never return raw chunks as the fallback. Core code lives under
|
||||
in the Wiki-page retrieval service and `llmwiki_search` tool call only WeKnora
|
||||
`/knowledgebase/{id}/wiki/pages?query=...` plus Wiki-page detail and never use
|
||||
raw chunks as a silent fallback. The pre-answer RAG orchestrator separately
|
||||
runs the explicit, authorized `/knowledge-search` raw-chunk channel described
|
||||
under **Knowledge retrieval diagnostics** below. Core code lives under
|
||||
`deerflow.integrations.weknora.local_index` and
|
||||
`deerflow.persistence.llmwiki_index`; preserve the harness → app import
|
||||
firewall. Gateway composition injects the store/search/sync services through
|
||||
@@ -1591,8 +1717,10 @@ existing vectors.
|
||||
Vectors are normalized little-endian float32 BLOBs. A page mirror and all of
|
||||
its vectors are replaced in one transaction only after strict batched
|
||||
embedding succeeds. Never delete the old generation before a remote embedding
|
||||
call and never fall back to WeKnora raw chunks while the local feature flag is
|
||||
enabled. NumPy matrices are per-mapping LRU snapshots invalidated by
|
||||
call. The local Wiki-vector service must not silently substitute WeKnora raw
|
||||
chunks when embedding fails; the pre-answer orchestrator's separately labelled,
|
||||
authorized raw-chunk channel is independent rather than that fallback. NumPy
|
||||
matrices are per-mapping LRU snapshots invalidated by
|
||||
`index_revision`; large decode/matrix work runs through `asyncio.to_thread`.
|
||||
`embedding.dimensions` is a local response-size invariant and fingerprint
|
||||
input, not an outbound API field: fixed-dimension BGE/OpenAI-compatible
|
||||
@@ -1608,6 +1736,50 @@ page is successfully rebuilt. `WikiRetrievalService` partitions mixed requests
|
||||
per mapping, so ready libraries stay local while incomplete libraries use that
|
||||
Wiki-only fallback.
|
||||
|
||||
Library-level overview requests (for example, “此知识库都讲了什么” or “这个知识空间有哪些内容”) are
|
||||
recognized once by `local_index.overview` and shared by local vector and remote Wiki-page retrieval. A completed
|
||||
local index returns representative page bodies without embedding the broad query or applying `min_similarity`; an
|
||||
incomplete index asks the remote page API for its directory rather than attempting a wording match that can return
|
||||
zero. In both paths directory, index, and summary pages are ineligible, and a page summary must never substitute
|
||||
for its body. `AssistantKnowledgeRepository` implements the same user-facing overview semantics independently for
|
||||
assistant knowledge bases.
|
||||
|
||||
**Knowledge retrieval diagnostics.** `app.gateway.llmwiki_rag.build_llmwiki_rag_context` is the single
|
||||
pre-answer retrieval path for both normal Q&A and `POST /api/llmwiki/retrieval-debug`; do not build a second
|
||||
debug-only ranking implementation. The debug endpoint accepts explicit ordinary and assistant knowledge-base ids
|
||||
and returns the bounded `knowledge_retrieval_trace`: selected/authorized spaces, actual retrieval mode, channel
|
||||
candidate counts, vector threshold diagnostics, warnings/fallback, zero-result reason, timings, and the exact
|
||||
bounded evidence made available to the model. A normal run stores the same trace on the latest `HumanMessage`,
|
||||
publishes it in the initial SSE `metadata` event, and exposes durable pre-retrieval plus model-driven
|
||||
`llmwiki_search` tool traces through `GET /api/threads/{thread_id}/reference-batches` under `retrieval_traces`.
|
||||
Keep trace results bounded to the same eight sources and 1400 characters per source as the answer prompt; never
|
||||
expose provider credentials or unrelated library contents. If a client omits selection keys on a later turn,
|
||||
recover the persisted thread selection; an explicitly present empty list must continue to clear the scope.
|
||||
|
||||
Ordinary pre-retrieval searches both generated Wiki page bodies/sections and authorized WeKnora raw document
|
||||
chunks. The raw channel must be called only with remote ids derived from already-authorized DeerFlow mappings,
|
||||
must reject every returned row whose remote id is not in that authorized map, and must allowlist/bound source
|
||||
fields before adding them to prompts or traces. Run raw-chunk search concurrently with Wiki retrieval; one channel
|
||||
failing marks the trace partial but must not discard successful evidence from the other. A ready local Wiki snapshot
|
||||
uses vector + lexical channels with RRF; incomplete mappings use lexical Wiki recall and the remote Wiki-page API;
|
||||
Wiki and raw result lists are fused without comparing provider score scales. Raw provider scores are channel scores,
|
||||
not cosine similarity. Assistant knowledge runs vector and keyword/raw-chunk recall together and fuses both ranked
|
||||
lists with weighted RRF; exact literal identifiers such as device or incident codes take priority, so imported raw
|
||||
chunks remain searchable even when compatible Wiki vectors exist. This is not guaranteed pure-vector retrieval on
|
||||
every turn. The same pre-answer orchestrator performs bounded graph expansion for ordinary Wiki links and imported
|
||||
assistant entity relations whenever the selected repository exposes `get_adjacent_pages`. It uses up to four directly
|
||||
recalled Wiki pages as
|
||||
seeds (two per authorized mapping), traverses one hop by default and at most two hops for explicit multi-hop
|
||||
wording or low-confidence vector diagnostics, inspects at most 24 adjacency rows, and accepts at most six pages
|
||||
(three per mapping). Every returned page is rechecked against the authorized local mapping ids, must contain a
|
||||
real eligible Wiki body, and carries a local-only slug/title path plus source/chunk provenance. Decayed walk scores
|
||||
rank graph candidates; fixed graph slots merge them with Wiki/raw evidence without comparing unrelated score
|
||||
scales. Trace fields include graph seeds/candidates/accepted/used, requested/executed hops, paths, depth reason and
|
||||
`timings_ms.graph_expansion`; capability is dynamic. Graph failure marks the trace partial and must never discard
|
||||
successful Wiki, raw-chunk, or assistant evidence. Assistant relation traversal resolves both endpoints through the
|
||||
entities' `wiki_page_id`, and requires the relation, both entities, and both pages to be active and inside the same
|
||||
selected assistant base; it must never follow cross-base relations.
|
||||
|
||||
The frontend must keep Wiki citations visibly distinct from raw-document
|
||||
citations while hiding provider-brand wording from user-facing labels, status
|
||||
text, and errors. Any click on a Wiki source card opens the shared right-side
|
||||
@@ -1680,12 +1852,19 @@ conversation-deposit mapping. Never point this page at the authenticated
|
||||
`/api/llmwiki` reads or weaken the published-only gate merely because the
|
||||
mapping id is hard to guess.
|
||||
|
||||
Owners manually force a full sync/vectorization through `POST
|
||||
/api/llmwiki/knowledge-bases/{id}/vectorize`; status is available through `GET
|
||||
Owners manually request a metadata-first incremental sync/vectorization through
|
||||
`POST /api/llmwiki/knowledge-bases/{id}/sync`; the older
|
||||
`POST /api/llmwiki/knowledge-bases/{id}/vectorize` route is a compatibility
|
||||
alias. Status is available through `GET
|
||||
/api/llmwiki/knowledge-bases/{id}/wiki-index-status`. That response includes
|
||||
aggregate `progress` and per-page `pages` details; current-scan progress is
|
||||
derived from `active_sync_id` / `last_seen_sync_id`, while each page exposes its
|
||||
index state, vector count, timestamps, and bounded error. The Wiki management
|
||||
index state, vector count, timestamps, and bounded error. `last_sync` is the
|
||||
persisted, lease-fenced terminal result and includes the sync id/mode/status and
|
||||
added/updated/skipped/deleted/failed counters. Frequent progress polling must
|
||||
pass `include_pages=false` so it reads only persisted aggregate state; the Wiki
|
||||
management view keeps the default detailed response when it needs per-page
|
||||
rows. The Wiki management
|
||||
page must show the progress bar and a status badge beside every article. It
|
||||
also exports all remote processed Wiki pages via `GET
|
||||
/api/llmwiki/knowledge-bases/{id}/wiki/export.xlsx`. Directory export must keep
|
||||
@@ -1788,12 +1967,76 @@ custom bases. Deletion removes the custom base's Wiki pages, vectors, entities,
|
||||
relations, import audit rows, and package files; the global base is never
|
||||
deletable because it owns full-library aggregation.
|
||||
|
||||
Assistant-knowledge reads keep the landing and article paths size-bounded.
|
||||
`list_bases()` is metadata/statistics-only and must never call the lazy global
|
||||
alignment pass: imports align their snapshot before commit, while content reads
|
||||
and search retain `_ensure_base_aligned`. The Wiki directory calls
|
||||
`GET /api/assistant-knowledge/wiki-pages?include_content=false`; repository
|
||||
`load_only` projection must exclude both `content_markdown` and `search_text`
|
||||
(response-only deletion is insufficient), and the frontend loads only the
|
||||
active slug through `GET /wiki-pages/{slug}`. The explicit index sorts into the
|
||||
first directory batch; that batch unlocks the reader immediately while later
|
||||
500-row batches append in the background. A URL-provided slug must fetch in
|
||||
parallel with the directory. An existing base with zero sources and zero Wiki
|
||||
pages is a valid, reachable state: keep the Wiki management surface mounted and
|
||||
show upload/package-import actions instead of requiring a document or redirecting
|
||||
away. The ordinary Wiki empty reader must likewise link writable users to the
|
||||
manual Wiki editor. Do not enable global pages, sources, or import-job queries
|
||||
while the landing route has no `base_id`.
|
||||
|
||||
The assistant-base switcher is a hot metadata path. `list_bases()` must load all
|
||||
four counters with one grouped statement rather than four serial counts per
|
||||
base, and a metadata-only rename must not rescan content tables. Assistant
|
||||
vector search validates base ids without loading counters, checks vector
|
||||
fingerprint/dimensions before encoding, scans only vector blobs plus minimal
|
||||
page identity in matrix batches, and hydrates bodies only for final top results.
|
||||
Chinese lexical fallback must extract bounded meaningful subphrases rather than
|
||||
treating the entire no-space question as one SQL token.
|
||||
|
||||
The shared WeKnora runtime client retains its authenticated session and caches
|
||||
the upstream knowledge-base directory for 15 seconds, coalescing concurrent
|
||||
scope requests. Knowledge-base and document mutations invalidate that cache;
|
||||
generation checks prevent an older in-flight list response from repopulating it.
|
||||
|
||||
Local Wiki synchronization is metadata-first. A WeKnora page-list item that
|
||||
contains the body is authoritative for hashing and must not trigger a detail
|
||||
GET. For legacy list shapes without a body, only matching explicit `version` or
|
||||
`updated_at` markers may skip the detail GET; missing markers must fail safe to
|
||||
the detail API. A content-identical page still refreshes category/parent/path,
|
||||
source/chunk references, page version and remote timestamps under the same
|
||||
mapping lease, without replacing vectors. Missing pages are retired only after
|
||||
a complete paginated scan and a final owner/sync-id lease fence. A bodyless
|
||||
metadata-only scan is not a complete graph snapshot: if the provider graph call
|
||||
also fails, preserve the prior graph rather than replacing it with partial
|
||||
edges.
|
||||
|
||||
Both Gateway document APIs and successful content mutations through the
|
||||
embedded WeKnora proxy schedule one per-mapping debounced retry sequence with
|
||||
bounded 1/8/30/90/300-second backoff. Never cancel an in-flight sync to debounce
|
||||
a newer mutation; wake the coordinator, update the local generation, and let
|
||||
the database lease deduplicate workers. While WeKnora reports processing, poll
|
||||
only the cheap knowledge-base state and retain both a post-processing scan and
|
||||
one short quiet confirmation. Before retiring missing pages or replacing graph
|
||||
edges, the sync service must re-read processing state; processing or an
|
||||
unverifiable state produces a retryable partial result and preserves the prior
|
||||
page/vector/graph snapshot. The periodic scheduler reads the remote
|
||||
knowledge-base directory once per tick, skips unchanged/processing libraries
|
||||
(including forced reconciliation), and retains a 10-minute fallback.
|
||||
|
||||
### Wiki package and local encoder invariants (2026-09)
|
||||
|
||||
Ordinary knowledge detail landing is globally configured by `ordinary_knowledge_landing` (`wiki` default / `weknora`), with admin-only PUT `/api/system-settings/ordinary-knowledge-landing`. Explicit document/graph/article links bypass this default. Wiki reading and the embedded WeKnora view provide reciprocal navigation; `?view=weknora` explicitly selects the iframe and must not loop back to the default Wiki route.
|
||||
|
||||
- `knowledge_transfer.sync_to_global` is called after successful explicit local
|
||||
Wiki vectorization; it skips conversation/system mappings. Do not enable a new
|
||||
Wiki vectorization; it skips conversation/system mappings. Concurrent triggers
|
||||
for one mapping are coalesced inside a Gateway process around the complete
|
||||
check/create/import sequence. Cross-worker safety lives at the repository
|
||||
transaction boundary: every assistant import takes the target base row
|
||||
`FOR UPDATE` before graph alignment or dependent snapshot writes, preserving
|
||||
one lock order across workers. Retry only recognized transient lock-wait,
|
||||
deadlock and serialization-contention errors; let the failed transaction roll
|
||||
back fully, then replay the whole package with bounded backoff. Cancellation
|
||||
and non-contention failures must propagate unchanged. Do not enable a new
|
||||
watcher or automatic remote-write scheduler as a side effect.
|
||||
- `wiki_packages` exposes file-backed assistant exports and restores into empty
|
||||
ordinary Wiki libraries. Validate vectors before any remote mutation; recreate
|
||||
@@ -1815,8 +2058,8 @@ Ordinary knowledge detail landing is globally configured by `ordinary_knowledge_
|
||||
pages need backfill, not a forced rebuild of every page.
|
||||
FastEmbed is a standard backend dependency. Encoder configuration, directory
|
||||
upload and warmup live only in the administrator global settings dialog.
|
||||
A runtime `system_settings.json` path override wins over config.yaml; uploaded
|
||||
files are streamed into `.deer-flow/models/bge-m3`, never the source tree.
|
||||
When the effective provider is local, its `system_settings.json` path override wins over config.yaml; uploaded
|
||||
files are streamed into immutable `.deer-flow/models/bge-m3-revisions/<revision>` directories, never the source tree.
|
||||
Encoder warmup is explicitly started from the settings UI/API and must never block application startup
|
||||
or turn a model-load failure into a DeerFlow service failure. Local provider
|
||||
is offline-only: require local_model_path or DEERFLOW_BGE_M3_MODEL_PATH and
|
||||
@@ -1824,6 +2067,21 @@ Ordinary knowledge detail landing is globally configured by `ordinary_knowledge_
|
||||
`local_process_isolation=true` by default: inference runs in the bundled JSONL
|
||||
worker subprocess, and shutdown must terminate that worker without waiting on
|
||||
model teardown.
|
||||
- The administrator encoder override is an explicit persisted mode, not an
|
||||
inference from whether `model_path` happens to be non-empty. `configured`
|
||||
uses the current `config.yaml` provider (including OpenAI-compatible);
|
||||
saving/uploading a local directory sets `local` plus a new model revision,
|
||||
while `/api/llmwiki/local-encoder/use-configured` switches back. Every switch
|
||||
hot-rebuilds embedding/cache/search/sync/retrieval as one graph and updates
|
||||
the harness runtime registration. Status must remain readable for a remote
|
||||
provider. The scheduler stays alive as a lightweight config watcher but only
|
||||
scans when effective `enabled && auto_sync`; keep the default/portable release
|
||||
configs at `auto_sync=true` because asynchronous Wiki generation can finish
|
||||
after the event-triggered first scan. A changed local model revision must
|
||||
change the embedding fingerprint so vectors from replaced weights are never
|
||||
reused. Uploaded weights live in immutable revision directories. A replaced
|
||||
embedding generation gets a bounded grace period for in-flight work, then
|
||||
rejects stragglers, drains real thread work, and releases its model process.
|
||||
- Every successful mapping snapshot is mirrored to the global assistant base,
|
||||
including empty snapshots so deleted source pages are retired. Re-import is
|
||||
source-replacement semantics: absent Wiki/chunk/entity/relation pages must not
|
||||
@@ -1832,6 +2090,13 @@ Ordinary knowledge detail landing is globally configured by `ordinary_knowledge_
|
||||
duplicate synthetic entity/relation articles. Store original Wiki metadata
|
||||
separately from assistant workflow status. Count Wiki pages separately from
|
||||
raw document chunks.
|
||||
- Assistant graph retrieval resolves each active relation through both active
|
||||
endpoint entities' `wiki_page_id` values and their active Wiki pages. Every
|
||||
relation, entity, seed page, and neighbor page must match the requested
|
||||
`base_id`; never traverse a corrupt cross-base edge. Pre-retrieval uses the
|
||||
same bounded one/two-hop walk as ordinary Wiki, labels graph evidence as
|
||||
assistant evidence, and lets a directly recalled non-seed page remain a
|
||||
traversal frontier without duplicating it in the answer context.
|
||||
- The global base aligns entities by normalized display name and aliases across
|
||||
source records, and aligns content Wiki pages by normalized title. Do not use
|
||||
source slug as the global canonical identity. Merge unique knowledge blocks
|
||||
@@ -1858,3 +2123,58 @@ configured SQLite backend; the PostgreSQL checkpointer schema is not part of the
|
||||
Gauss compatibility contract. Treat both `postgresql` and `opengauss` dialect
|
||||
names as timezone-aware PostgreSQL-family engines in portable types and
|
||||
dialect-specific session settings.
|
||||
|
||||
## 2026-09-14 我的项目与知识库验收修复
|
||||
|
||||
- 简洁模式的「我的项目」列表、策略工作台和普通问答项目恢复工作区功能导航;窄屏提供展开菜单按钮。
|
||||
- 普通问答先创建具名项目,再在项目内创建多个对话。列表读取 `/api/analysis-projects` 的 `qa` 项目,不再读取、标记或自动归入全局普通聊天。
|
||||
- 项目设置更新保留既有知识库绑定。原「项目空间」与官方入口继续可用;我的项目下问答路由为 `qa/projects/groups/*`。
|
||||
- 策略生成卡片统一先显示思考/执行过程,再显示正文;评判、方案及展开视图沿用相同顺序。
|
||||
- 完成评判但尚未勾选策略的项目可以从方案列表进入,在工作台选择后生成;已保存方案在任务修改后仍可打开。旧列表请求不再覆盖新请求状态。
|
||||
- 知识库管理页读取上游最新目录,避免远端删除后命中旧缓存;检索调用仍共享短期目录缓存。
|
||||
- 普通用户访问尚未初始化的助手知识库时显示明确空态及刷新按钮。
|
||||
|
||||
验证:后端相关回归 273 项、前端策略/项目回归 31 项通过;真实 BAAI/bge-m3 编码、指定库向量检索与草稿隔离通过;本地隔离服务的问答项目创建/分组、真实问答、方案整理、助手库创建/编辑/文件入库/向量检索/详情均通过。完整验收记录见仓库 `docs/ji-knowledge-projects-acceptance.md`。不在源码或文档中保存接口密钥。
|
||||
|
||||
2026-09-14 日常使用复查:补修项目历史入口与移动侧栏、默认智能体同步、项目加载重试、助手知识检索过期响应隔离及无结果指引;更新 ScrollArea 的 React 19 兼容修复。后端再次 273 项通过,前端 33 项通过。范围及限制见仓库 docs/daily-usage-audit-20260914.md。
|
||||
|
||||
### 知识库扩展与统计检索(2026-09-20)
|
||||
|
||||
- `llmwiki_knowledge_bases.test_questions` / `retrieval_settings` 保存每库测试问题与检索策略;`NULL` 测试问题表示使用 `knowledge_profile.default_test_questions`,显式空数组表示禁用。列表/详情统一通过 `_mapping_view` 返回解析后的配置。
|
||||
- `deerflow.integrations.weknora.local_index.advanced` 是问答与调试页共享的权威实现。实体扩展只允许一跳、数量有上限,来源为别名/Wiki 链接/Markdown 链接;扩展查询各自运行现有 Wiki-only 混合检索,再进行 RRF 融合。不得把扩展改成无限图遍历,也不得绕过当前用户的 mapping-id 授权。
|
||||
- 时间区间与计数在后端确定性计算,模型不得自行数检索结果。优先按结构化 `source_refs` 的稳定来源标识去重;没有可识别来源记录时返回 `unit=wiki_page`。日期优先读取来源字段/元数据和 Wiki frontmatter,远端创建/更新时间只是带 `date_source` 的后备。区间统计排除无日期记录并返回 `undated_excluded_count`。
|
||||
- `POST /api/llmwiki/knowledge-bases/{id}/retrieval-debug` 与 `llmwiki_search` 使用同一个 `advanced_search`。调试响应必须保留 query plan、statistics、references、diagnostics 和 request id,供 `/page/workspace/knowledge/{id}/retrieval-debug` 及浏览器 Network 面板排错。
|
||||
- 回归测试:`tests/test_llmwiki_advanced_retrieval.py`;同时运行 `test_llmwiki_wiki_retrieval.py`、`test_llmwiki_vector_search.py`、`test_llmwiki_local_index_repository.py` 和 `test_llmwiki_weknora.py`。
|
||||
|
||||
### 知识库标题权重与引用切片(2026-09-22)
|
||||
|
||||
- 检索排序必须把原始用户查询的标题/别名命中作为高置信信号,不能让实体扩展词
|
||||
冒充原始查询。统一实现位于 `deerflow.integrations.weknora.local_index.ranking`;
|
||||
普通 Wiki 的向量、关键词、远端 Wiki、原文 chunk 以及助手知识库融合均保留
|
||||
`title_match_score`,标题信号以有界乘数并入 RRF/相关性分数,不能作为绝对主排序键。
|
||||
普通 Wiki 的全检索链标题加成由
|
||||
`llmwiki.local_wiki_index.title_boost_enabled/weight` 控制;关闭或设为 0 必须恢复
|
||||
原排序。
|
||||
- 引用详情必须同时展示本次命中的切片和文档/Wiki 正文。`/api/llmwiki/sources/context`
|
||||
返回 `focus_chunk_id` 及邻近 `chunks`;前端 `WeKnoraSourceDrawer` 使用精确焦点切片
|
||||
作为详情顶部的「本次检索命中切片」,不能只重复文档级摘要。
|
||||
- 对话页的后续问题保留生成结果,但只允许在消息列表底部渲染。`MessageList` 上报底部
|
||||
状态,`ChatPage`、`AgentChatPage` 与分析项目对话通过 `InputBox` 的
|
||||
`followupsDisplayAllowed` 控制;上滑隐藏,回到底部恢复。
|
||||
### Uploaded-file context and bounded reads
|
||||
|
||||
- `deerflow.uploads.inspection` performs streaming whole-file statistics and
|
||||
deterministic head/middle/tail sampling. It reads limits from the harness
|
||||
runtime upload-policy registry and never imports `app.*`.
|
||||
- `read_file` uses provider-level bounded range reads; both LocalSandbox and
|
||||
AioSandbox implement them without fetching the complete file. A range-free
|
||||
large local-file read returns the structured inspection payload.
|
||||
- `UploadsMiddleware` emits a size-bounded, JSON-encoded `<uploaded_files>`
|
||||
block. Current files may include a short outline; historical files are
|
||||
metadata-only and generated Markdown companions are excluded.
|
||||
- Converted companions use collision-free `<original-name>.<ext>.md` names and
|
||||
extracted Markdown has a fixed character ceiling. Conversion always runs off
|
||||
the event loop with bounded worker concurrency.
|
||||
- `grep` scans configured upload sizes with total-byte, time, result and line
|
||||
bounds. `ContextWindowGuardMiddleware` infers common provider window fields
|
||||
and applies a conservative fallback for upload-bearing turns only.
|
||||
Reference in new issue
Block a user