sync update from LAN repo (source master-nj @ 067eb808)

This commit is contained in:
gozeros committed 2026-11-10 14:04:09 +08:00
1 parent 1244fb1f2f
commit 1d5dd8fc90
182 files changed
+18973 -769

No files matched your search

+50 -4
View File
@@ -1,5 +1,11 @@
# DeerFlow Backend
普通知识库默认详情页面:管理员可在知识库顶部选择“本系统 Wiki 索引”或“嵌入 WeKnora”,对应 `ordinary_knowledge_landing`(默认 `wiki`)。Wiki 库首次进入默认展示本系统索引,右上角“打开 WeKnora”切换到内嵌管理页;内嵌页可返回索引。非 Wiki 库仍进入 WeKnora 文档页。设置通过 `/api/system-settings/ordinary-knowledge-landing` 保存,仅管理员可修改;指定文章、文档、图谱的直达链接优先于默认设置。
助手知识库可独立于 WeKnora 使用:管理员进入助手知识库,选择或创建库后点击“上传文件生成 Wiki”。支持 Markdown/TXT/CSV、PDF、Word、Excel、PowerPoint(单文件 100 MB,解析正文上限 400 万字符)。文件先写入运行目录 `assistant-knowledge/files/{job_id}`,后台分段调用已配置的语言模型生成分类 Wiki、实体关系,再通过 DeerFlow 编码器对正文分段向量化,保存到助手库并汇入知识梳理总库。无需配置 WeKnora URL;配置后仍可直接上传,也保留 WeKnora 数据包导入。文件处理沿用管理员写入权限,所有用户可以浏览和搜索。
导入记录显示解析、Wiki 生成和向量化阶段;失败或服务中断后可“重试 / 恢复处理”,复用已落盘的 Wiki 和相同模型指纹的向量。离线编码器须预先放置模型并在管理员设置中配置;模型或编码器不可用会明确失败,不把摘要冒充 Wiki 或虚构向量。扫描件是否可解析取决于部署的文档解析/OCR 能力。
DeerFlow is a LangGraph-based AI super agent with sandbox execution, persistent memory, and extensible tool integration. The backend enables AI agents to execute code, browse the web, manage files, delegate tasks to subagents, and retain context across conversations - all in isolated, per-thread environments.
---
@@ -45,6 +51,14 @@ Health probes are always unauthenticated, even when API authentication is enable
## Core Components
### 项目空间
前端入口为 `/page/workspace/projects`。项目作为现有问答的分类容器,保留工作区左侧聊天记录。创建项目时选择问答、策略智能或方案生成,创建后不可修改模式;可设置默认智能体,并在新建每条项目对话时另选智能体。方案生成复用「工作流配置 · 报告结构」的结构库和现有写作研究管线;策略智能支持可配置 Lens、多视角独立分析、策略辩论与综合决策。
Gateway `/api/analysis-projects` 提供列表、创建、查询、更新及 `/{id}/threads` 新建和查询对话。项目与对话归属存储在 `analysis_projects` / `analysis_project_threads`,智能体、Lens、报告结构等扩展配置存储在 `analysis_project_configs`;这些表随数据库初始化自动创建,内存部署不跨重启保存。项目按登录用户隔离,模式和归属不接受客户端元数据覆盖。策略智能可在无知识库时基于用户条件推演,但必须标出假设与资料缺口。方案生成调用现有 `deep_research_report` 研究管线,使用项目选择的固定或弹性报告结构,复用方案画布和后续修改流程。
验证:`python -m pytest tests/test_analysis_projects.py tests/test_deep_research_report_tool.py tests/test_writing_mode_report_middleware.py -q`。
### Lead Agent
The single LangGraph agent (`lead_agent`) is the runtime entry point, created via `make_lead_agent(config)`. It combines:
@@ -58,7 +72,7 @@ The single LangGraph agent (`lead_agent`) is the runtime entry point, created vi
### Workflow Studio runtime
- Candidate workflow graphs remain editable after they are loaded into the Coze-compatible canvas; the compatibility response explicitly marks the current owner as editable so node dragging is not mistaken for a read-only preview.
- Workflow-agent SSE forwards provider `reasoning_content` as an inline `<think>…</think>` trace for the UI while keeping the executable node output limited to the visible answer text. Every native LangGraph `[AIMessage|ToolMessage, metadata]` frame is additionally persisted and sent as `node.message.data.messages` without projection or length cropping, so the Studio can replay DeerFlow's tool-result, search-result and `ask_clarification` cards exactly after reconnecting.
- OpenAI-compatible model responses accept both `reasoning_content` and `reasoning`; both are normalized into the existing reasoning display path. Workflow-agent SSE forwards provider reasoning as an inline `<think>…</think>` trace for the UI while keeping the executable node output limited to the visible answer text. Every native LangGraph `[AIMessage|ToolMessage, metadata]` frame is additionally persisted and sent as `node.message.data.messages` without projection or length cropping, so the Studio can replay DeerFlow's tool-result, search-result and `ask_clarification` cards exactly after reconnecting.
### AgentScope report collaboration (in progress)
@@ -73,7 +87,7 @@ Middlewares execute in strict order, each handling a specific concern:
| 1 | **ThreadDataMiddleware** | Creates per-thread isolated directories (workspace, uploads, outputs) |
| 2 | **UploadsMiddleware** | Injects newly uploaded files into conversation context |
| 3 | **SandboxMiddleware** | Acquires sandbox environment for code execution |
| 4 | **SummarizationMiddleware** | Reduces context when approaching token limits (optional) |
| 4 | **SummarizationMiddleware** | Reduces context near token limits and folds the hidden summary into one leading system message for strict model adapters (optional) |
| 5 | **TodoListMiddleware** | Tracks multi-step tasks in plan mode (optional) |
| 6 | **TitleMiddleware** | Auto-generates conversation titles after first exchange |
| 7 | **MemoryMiddleware** | Queues conversations for async memory extraction |
@@ -88,6 +102,7 @@ Per-thread isolated execution with virtual path translation:
- **Providers**: `LocalSandboxProvider` (filesystem) and `AioSandboxProvider` (Docker, in community/)
- **Virtual paths**: `/mnt/user-data/{workspace,uploads,outputs}` → thread-specific physical directories
- **Skills path**: `/mnt/skills` → `deer-flow/skills/` directory
- **Portable configured mounts**: a relative `sandbox.mounts[*].host_path` is resolved against the directory containing the loaded `config.yaml`, not the process working directory. Providers still receive an absolute host path, so `knowledge/knowledge_base` can be committed and the deployment can be cloned or moved without editing a machine-specific path; already-absolute mount paths are preserved.
- **Skills loading**: Recursively discovers nested `SKILL.md` files under `skills/{public,custom}` and preserves nested container paths
- **Conditional WeKnora LLMWiki**: a non-empty `llmwiki.weknora.api_base_url` enables the DeerFlow-native WeKnora BFF and UI; an empty address preserves the existing LLMWiki page. WeKnora remains independently deployed and owns its database, Redis, storage, parsers and models. DeerFlow keeps only the API/admin addresses, user/publication mappings, agent knowledge-base bindings and per-conversation selections; it stores and sends no WeKnora credential, so the endpoint must be restricted to a trusted network and allow DeerFlow traffic. Owners publish directly to the public LLMWiki without an approval queue for ordinary bases; the system-owned conversation-deposit base is always published, shared by all users, excluded from My Knowledge, immutable through user/admin detail operations, and recognized by both its reserved name and `system` owner. An existing same-name remote mapping is adopted in place as the system deposit so its mapping id and document links stay valid. Its user-facing knowledge-base description and generated document metadata use the `cmzs` brand; list/detail reads synchronize stale remote descriptions, while conversation-deposit search results, source previews and embedded-detail JSON rewrite legacy brand text to `cmzs`. Deposit requests must reference a thread accessible to the current account. Personal knowledge-base lists are scoped to the current DeerFlow account even for admins, while frontend LLMWiki query caches also include the active account id so switching accounts cannot reuse another account's list/detail data. Admins remain the only role that sees the WeKnora connection-config tab. Knowledge cards open a dedicated WeKnora-style detail route: Wiki-enabled bases default to Wiki, then documents and graph; standard bases hide Wiki and default to documents. Writable ordinary knowledge-base details expose a native multi-file upload action that submits files sequentially and reports aggregate success/failure; read-only public bases and the system deposit base do not. It supports file/URL/manual-Markdown imports, Wiki page browsing, document preview, real parsed chunks and Wiki graph nodes/edges. New chats use a compact WeKnora-style searchable/grouped multi-select inside the composer, while agent create/edit pages bind spaces through an inline panel. A single proxy service identity means DeerFlow users are isolated logically while their WeKnora entities share that identity's workspace. The read-only `llmwiki_search` tool reauthorizes the current DeerFlow user on every call and renders credential-free source previews; source previews prefer cleaned Wiki/LLMWiki text when a retrieved chunk maps to a Wiki page, with raw chunks as the fallback. The detail page first creates a signed iframe session through the normal DeerFlow API auth path, then points the iframe directly at the proxied WeKnora `/platform/...` route using the signed `deerflow_weknora_embed` cookie; global auth/CSRF bypass is limited to those signed iframe proxy requests, and HTTPS cookie attributes honor `X-Forwarded-Proto` behind reverse proxies. The iframe proxy keeps WeKnora's root-relative `/platform`, `/assets`, `/locales`, and `/api/v1/*` paths, but is registered last through `llmwiki.proxy_router` so it cannot intercept first-party Gateway APIs such as `POST /api/v1/auth/login/username`. See [docs/LLMWIKI_WEKNORA_INTEGRATION_ZH.md](docs/LLMWIKI_WEKNORA_INTEGRATION_ZH.md).
- **Enterprise Research workbench (temporarily disabled)**: the independent `/api/enterprise-research` router, its report executor, and its ORM table registration are intentionally inactive. The implementation remains in the repository, but startup does not create or alter `enterprise_research_tasks` or `enterprise_research_report_jobs`; this keeps the active Deep Research workflow independent of the paused workbench and avoids MySQL schema errors.
@@ -465,6 +480,10 @@ Custom agents are indexed in the `agents` database table. The `id` column is the
The built-in `forced-research-responder` (「强制检索输出助手」) is seeded at Gateway startup. Unlike an ordinary skill-enabled custom agent, it has a runtime-enforced per-turn collection gate: the model cannot finish a response until a retrieval/knowledge/MCP tool or a skill Python script returns usable information. It then answers directly in the user's requested chat format rather than generating an artifact. The agent may create or edit only temporary `.py` files below `/mnt/user-data/workspace/` and may execute `.py` skill scripts through `bash`; writes to `outputs`, Markdown/document creation, `present_files`, general shell commands, and skill mutation are denied. Administrators can edit its model and skill allowlist from agent management; the runtime safety policy remains fixed. Regression coverage: `tests/test_forced_research_agent.py`.
The strategy-selection workbench starts with nine folder-backed functional agents: one intent agent, six independent angle agents (governance, diplomacy, economy/supply-chain, security risk, public opinion, and law/policy), one judge, and one final solution editor. Defaults ship in `app/gateway/routers/_strategy_selection_seed_assets/` and are copied only when an agent directory is absent, before `_sync_legacy_agents`; existing administrator edits are never overwritten. A global `strategy_agent_roles` registry owns classification, lifecycle state, and ordering independently of projects. Intent, judge, and solution are fixed singleton roles: administrators may update their model, SOUL, and skills but cannot add, disable, reorder, or delete them. Angles form one ordered catalog shared by every project; administrators may add, update, disable, restore, and reorder them. `GET /api/strategy-agents` is the public enabled catalog and omits model/SOUL/skills; `include_disabled=true` or `include_config=true` requires an administrator. Admin writes use `POST /api/strategy-agents/angles`, `PUT /api/strategy-agents/{id}`, and `DELETE /api/strategy-agents/{id}`; deletion is a soft disable so folder configuration and historical project references remain intact. Dynamic angles use the reserved `strategy-selection-angle-` namespace, receive `web` + `file:read`, and get a safe default SOUL if the submitted SOUL is blank. Model names and selected skills are validated, skills must be active and globally visible, generic agent CRUD cannot create or hard-delete reserved strategy agents, and parallel execution rejects disabled roles. The shipped angle agents use `policy-knowledge`, intent uses `intent-clarification-policy`, judging is input-only, and the solution editor has file-write access for `/mnt/user-data/outputs/综合策略方案.md`. Every default SOUL requires neutral evidence handling, de-escalation and civilian protection, and prohibits operational combat guidance or targeted political manipulation.
`POST /api/parallel-agents/run/stream` keeps reasoning and the deliverable in separate per-agent frames. Provider structured reasoning and inline `<think>...</think>` content are emitted only as `reasoning_delta`; visible answer text is emitted as `delta`, and `agent_done.content` contains only the visible answer. Consumers must not append `reasoning_delta` to strategy output or the final document. A final `is_llm_error_message` fallback, empty/thinking-only response, or punctuation-only provider degeneration is emitted as `agent_error`, never as a successful `agent_done`. Regression coverage: `tests/test_strategy_selection_seed.py`, `tests/test_strategy_agent_management.py`, and `tests/test_roundtable_model_fallback.py`.
### AI Writing Intranet Retrieval
AI 写作的「知识库检索」会直连 `ai_writing.intranet_search_url` 内网 ES/知识库接口,并默认跳过代理环境变量和 HTTPS 证书校验,适配自签名或内部 CA 证书的内网服务。需要强校验时可在 `config.yaml` 里设置 `ai_writing.intranet_verify_ssl: true`。通用检索里的非内置技能在 `ai_writing.researcher_skill_agent: true` 时会用真实 agent 运行时执行技能:以 `ai-writing-researcher` 身份读取该技能的 `SKILL.md`,并按技能要求通过 `bash` 运行脚本,弱模型也会在本轮任务里收到明确的读文件和脚本执行指令。
@@ -537,6 +556,13 @@ the configured model has an explicit, proxy-supported thinking control. This
prevents DeerFlow's connection check from sending vendor-specific `thinking`
fields that a generic OpenAI-compatible gateway may reject.
OpenAI-compatible gateways that return vendor errors inside an HTTP 200
response (for example, a non-zero `base_resp.status_code` with empty/null
`choices`) are treated as upstream failures in both streaming and
non-streaming calls. DeerFlow retries transient upstream responses instead of
accepting an empty assistant turn, while genuine quota/balance failures are
classified as non-retriable and surfaced immediately instead of being retried.
### Running
**Full Application** (from project root):
@@ -910,6 +936,13 @@ the file through the existing thread-artifact endpoint.
远端已经删除的知识库会在下次列表刷新时自动隐藏,包括个人、公共列表和知识库选择器;页面每 15 秒自动刷新,也可以手动点击「刷新」。远端服务请求失败时展示错误,不把连接故障当成删除,不自动删除本地映射或已有引用。
新建知识库时会通过 `GET /api/llmwiki/embedding-models` 读取 WeKnora 中可用的
Embedding 模型,并把所选 `embedding_model_id` 明确传给创建接口。该模型只负责
WeKnora 中上传原文的分块向量化;它和下文 DeerFlow 自己的 Wiki 镜像编码器是两套
独立配置,因此选择 OpenAI/OpenAI-compatible 原文编码器时不要求同时启用本地
BGE-M3。模型列表响应只暴露名称、provider、状态等白名单字段,不返回密钥或连接
参数。模型绑定只对新建库生效;需要跨知识库检索时,各库应使用相同或兼容的向量模型。
## DeerFlow-local LLMWiki Wiki vector index
LLMWiki can mirror processed WeKnora Wiki pages into DeerFlow and search only
@@ -1158,13 +1191,23 @@ llmwiki:
本地编码依赖随标准后端安装。将完整 BGE-M3 模型目录提前放到离线服务器,并通过
`local_model_path` 或 `DEERFLOW_BGE_M3_MODEL_PATH` 指向它。管理员也可在
「设置和更多 → 设置 → 本地编码器」中选择服务器目录,或从浏览器上传完整模型目录;
「设置和更多 → 设置 → Wiki 镜像编码器」中选择服务器目录,或从浏览器上传完整模型目录;
页面配置持久化到运行目录的 `system_settings.json`,上传文件落到
`.deer-flow/models/bge-m3`,均不进入源码仓库。系统强制 `local_files_only`,
`.deer-flow/models/bge-m3-revisions/<revision>` 的不可变版本目录,均不进入源码仓库。系统强制 `local_files_only`,
**不会访问互联网或下载模型**。管理员可在同一设置页点击「一键启动编码器」异步加载;该操作不挂在应用启动流程,加载失败不会
中断 DeerFlow 主服务。默认由独立低优先级子进程承载模型内存,正常停止 DeerFlow
时会一并回收该子进程;编码进程异常退出只会将当前向量任务标记失败。模型目录应包含
`onnx/model.onnx`(官方权重约 2.3 GB)及对应 tokenizer/config 文件。权重不进入业务源码仓库。
运行时编码器模式持久化为 `configured` 或 `local`。默认 `configured` 严格使用
`config.yaml` 中的 OpenAI、OpenAI-compatible 或 local provider;保存服务器目录或
上传模型会显式切到 `local` 并热重建 embedding/search/sync/retrieval 服务,不需要
重启 Gateway。`POST /api/llmwiki/local-encoder/use-configured` 可立即恢复配置文件中的
provider。`GET /api/llmwiki/local-encoder/status` 在远程 provider 下也正常返回当前
provider/mode,不再把“不是 local”当成状态接口错误。每次激活本地模型都会生成新的
模型修订指纹,避免同一路径替换权重后误复用旧向量。切换后的旧编码器会给在途任务
保留短暂宽限期,随后等待当前编码调用结束并自动回收模型进程。
浏览器目录上传采用磁盘暂存和 4 MB 分块复制,页面显示上传进度;经 Nginx 等反向
代理部署时仍需把请求体上限和读写超时配置到可承载完整模型目录。
相同模型指纹、相同内容的页面复用现有向量;短正文/摘要至少保留一个非空分段。
@@ -1217,3 +1260,6 @@ checkpointer:
normalized to the openGauss dialect. Do not configure the LangGraph
PostgreSQL checkpointer against GaussDB: its migrations are PostgreSQL-specific.
For a production image, build with `--build-arg UV_EXTRAS=gauss`.
# 问答记忆开关
问答记忆默认关闭。关闭时,普通问答不会加载记忆中间件、不会注入历史记忆,也不会在回答结束后调用模型提取或写入记忆。管理员可在「设置 → 问答记忆」中显式开启;开关保存在运行目录的 `system_settings.json`,新部署或没有该字段时始终按关闭处理。