sync update from LAN repo (source master-yx @ cf505220)

This commit is contained in:
gozeros committed 2026-10-21 15:11:11 +08:00
1 parent e243ec3a46
commit 19d3cb96dc
114 files changed
+14787 -451

No files matched your search

+32 -3
View File
@@ -79,7 +79,7 @@ Middlewares execute in strict order, each handling a specific concern:
| 1 | **ThreadDataMiddleware** | Creates per-thread isolated directories (workspace, uploads, outputs) |
| 2 | **UploadsMiddleware** | Injects newly uploaded files into conversation context |
| 3 | **SandboxMiddleware** | Acquires sandbox environment for code execution |
| 4 | **SummarizationMiddleware** | Reduces context when approaching token limits (optional) |
| 4 | **SummarizationMiddleware** | Reduces context near token limits and folds the hidden summary into one leading system message for strict model adapters (optional) |
| 5 | **TodoListMiddleware** | Tracks multi-step tasks in plan mode (optional) |
| 6 | **TitleMiddleware** | Auto-generates conversation titles after first exchange |
| 7 | **MemoryMiddleware** | Queues conversations for async memory extraction |
@@ -94,6 +94,7 @@ Per-thread isolated execution with virtual path translation:
- **Providers**: `LocalSandboxProvider` (filesystem) and `AioSandboxProvider` (Docker, in community/)
- **Virtual paths**: `/mnt/user-data/{workspace,uploads,outputs}` → thread-specific physical directories
- **Skills path**: `/mnt/skills` → `deer-flow/skills/` directory
- **Portable configured mounts**: a relative `sandbox.mounts[*].host_path` is resolved against the directory containing the loaded `config.yaml`, not the process working directory. Providers still receive an absolute host path, so `knowledge/knowledge_base` can be committed and the deployment can be cloned or moved without editing a machine-specific path; already-absolute mount paths are preserved.
- **Skills loading**: Recursively discovers nested `SKILL.md` files under `skills/{public,custom}` and preserves nested container paths
- **Conditional WeKnora LLMWiki**: a non-empty `llmwiki.weknora.api_base_url` enables the DeerFlow-native WeKnora BFF and UI; an empty address preserves the existing LLMWiki page. WeKnora remains independently deployed and owns its database, Redis, storage, parsers and models. DeerFlow keeps only the API/admin addresses, user/publication mappings, agent knowledge-base bindings and per-conversation selections; it stores and sends no WeKnora credential, so the endpoint must be restricted to a trusted network and allow DeerFlow traffic. Owners publish directly to the public LLMWiki without an approval queue for ordinary bases; the system-owned conversation-deposit base is always published, shared by all users, excluded from My Knowledge, immutable through user/admin detail operations, and recognized by both its reserved name and `system` owner. An existing same-name remote mapping is adopted in place as the system deposit so its mapping id and document links stay valid. Its user-facing knowledge-base description and generated document metadata use the `cmzs` brand; list/detail reads synchronize stale remote descriptions, while conversation-deposit search results, source previews and embedded-detail JSON rewrite legacy brand text to `cmzs`. Deposit requests must reference a thread accessible to the current account. Personal knowledge-base lists are scoped to the current DeerFlow account even for admins, while frontend LLMWiki query caches also include the active account id so switching accounts cannot reuse another account's list/detail data. Admins remain the only role that sees the WeKnora connection-config tab. Knowledge cards open a dedicated WeKnora-style detail route: Wiki-enabled bases default to Wiki, then documents and graph; standard bases hide Wiki and default to documents. Writable ordinary knowledge-base details expose a native multi-file upload action that submits files sequentially and reports aggregate success/failure; read-only public bases and the system deposit base do not. It supports file/URL/manual-Markdown imports, Wiki page browsing, document preview, real parsed chunks and Wiki graph nodes/edges. New chats use a compact WeKnora-style searchable/grouped multi-select inside the composer, while agent create/edit pages bind spaces through an inline panel. A single proxy service identity means DeerFlow users are isolated logically while their WeKnora entities share that identity's workspace. The read-only `llmwiki_search` tool reauthorizes the current DeerFlow user on every call and renders credential-free source previews; source previews prefer cleaned Wiki/LLMWiki text when a retrieved chunk maps to a Wiki page, with raw chunks as the fallback. The detail page first creates a signed iframe session through the normal DeerFlow API auth path, then points the iframe directly at the proxied WeKnora `/platform/...` route using the signed `deerflow_weknora_embed` cookie; global auth/CSRF bypass is limited to those signed iframe proxy requests, and HTTPS cookie attributes honor `X-Forwarded-Proto` behind reverse proxies. The iframe proxy keeps WeKnora's root-relative `/platform`, `/assets`, `/locales`, and `/api/v1/*` paths, but is registered last through `llmwiki.proxy_router` so it cannot intercept first-party Gateway APIs such as `POST /api/v1/auth/login/username`. See [docs/LLMWIKI_WEKNORA_INTEGRATION_ZH.md](docs/LLMWIKI_WEKNORA_INTEGRATION_ZH.md).
- **Enterprise Research workbench (temporarily disabled)**: the independent `/api/enterprise-research` router, its report executor, and its ORM table registration are intentionally inactive. The implementation remains in the repository, but startup does not create or alter `enterprise_research_tasks` or `enterprise_research_report_jobs`; this keeps the active Deep Research workflow independent of the paused workbench and avoids MySQL schema errors.
@@ -471,6 +472,10 @@ Custom agents are indexed in the `agents` database table. The `id` column is the
The built-in `forced-research-responder` (「强制检索输出助手」) is seeded at Gateway startup. Unlike an ordinary skill-enabled custom agent, it has a runtime-enforced per-turn collection gate: the model cannot finish a response until a retrieval/knowledge/MCP tool or a skill Python script returns usable information. It then answers directly in the user's requested chat format rather than generating an artifact. The agent may create or edit only temporary `.py` files below `/mnt/user-data/workspace/` and may execute `.py` skill scripts through `bash`; writes to `outputs`, Markdown/document creation, `present_files`, general shell commands, and skill mutation are denied. Administrators can edit its model and skill allowlist from agent management; the runtime safety policy remains fixed. Regression coverage: `tests/test_forced_research_agent.py`.
The strategy-selection workbench starts with nine folder-backed functional agents: one intent agent, six independent angle agents (governance, diplomacy, economy/supply-chain, security risk, public opinion, and law/policy), one judge, and one final solution editor. Defaults ship in `app/gateway/routers/_strategy_selection_seed_assets/` and are copied only when an agent directory is absent, before `_sync_legacy_agents`; existing administrator edits are never overwritten. A global `strategy_agent_roles` registry owns classification, lifecycle state, and ordering independently of projects. Intent, judge, and solution are fixed singleton roles: administrators may update their model, SOUL, and skills but cannot add, disable, reorder, or delete them. Angles form one ordered catalog shared by every project; administrators may add, update, disable, restore, and reorder them. `GET /api/strategy-agents` is the public enabled catalog and omits model/SOUL/skills; `include_disabled=true` or `include_config=true` requires an administrator. Admin writes use `POST /api/strategy-agents/angles`, `PUT /api/strategy-agents/{id}`, and `DELETE /api/strategy-agents/{id}`; deletion is a soft disable so folder configuration and historical project references remain intact. Dynamic angles use the reserved `strategy-selection-angle-` namespace, receive `web` + `file:read`, and get a safe default SOUL if the submitted SOUL is blank. Model names and selected skills are validated, skills must be active and globally visible, generic agent CRUD cannot create or hard-delete reserved strategy agents, and parallel execution rejects disabled roles. The shipped angle agents use `policy-knowledge`, intent uses `intent-clarification-policy`, judging is input-only, and the solution editor has file-write access for `/mnt/user-data/outputs/综合策略方案.md`. Every default SOUL requires neutral evidence handling, de-escalation and civilian protection, and prohibits operational combat guidance or targeted political manipulation.
`POST /api/parallel-agents/run/stream` keeps reasoning and the deliverable in separate per-agent frames. Provider structured reasoning and inline `<think>...</think>` content are emitted only as `reasoning_delta`; visible answer text is emitted as `delta`, and `agent_done.content` contains only the visible answer. Consumers must not append `reasoning_delta` to strategy output or the final document. A final `is_llm_error_message` fallback, empty/thinking-only response, or punctuation-only provider degeneration is emitted as `agent_error`, never as a successful `agent_done`. Regression coverage: `tests/test_strategy_selection_seed.py`, `tests/test_strategy_agent_management.py`, and `tests/test_roundtable_model_fallback.py`.
### AI Writing Intranet Retrieval
AI 写作的「知识库检索」会直连 `ai_writing.intranet_search_url` 内网 ES/知识库接口,并默认跳过代理环境变量和 HTTPS 证书校验,适配自签名或内部 CA 证书的内网服务。需要强校验时可在 `config.yaml` 里设置 `ai_writing.intranet_verify_ssl: true`。通用检索里的非内置技能在 `ai_writing.researcher_skill_agent: true` 时会用真实 agent 运行时执行技能:以 `ai-writing-researcher` 身份读取该技能的 `SKILL.md`,并按技能要求通过 `bash` 运行脚本,弱模型也会在本轮任务里收到明确的读文件和脚本执行指令。
@@ -543,6 +548,13 @@ the configured model has an explicit, proxy-supported thinking control. This
prevents DeerFlow's connection check from sending vendor-specific `thinking`
fields that a generic OpenAI-compatible gateway may reject.
OpenAI-compatible gateways that return vendor errors inside an HTTP 200
response (for example, a non-zero `base_resp.status_code` with empty/null
`choices`) are treated as upstream failures in both streaming and
non-streaming calls. DeerFlow retries transient upstream responses instead of
accepting an empty assistant turn, while genuine quota/balance failures are
classified as non-retriable and surfaced immediately instead of being retried.
### Running
**Full Application** (from project root):
@@ -916,6 +928,13 @@ the file through the existing thread-artifact endpoint.
远端已经删除的知识库会在下次列表刷新时自动隐藏,包括个人、公共列表和知识库选择器;页面每 15 秒自动刷新,也可以手动点击「刷新」。远端服务请求失败时展示错误,不把连接故障当成删除,不自动删除本地映射或已有引用。
新建知识库时会通过 `GET /api/llmwiki/embedding-models` 读取 WeKnora 中可用的
Embedding 模型,并把所选 `embedding_model_id` 明确传给创建接口。该模型只负责
WeKnora 中上传原文的分块向量化;它和下文 DeerFlow 自己的 Wiki 镜像编码器是两套
独立配置,因此选择 OpenAI/OpenAI-compatible 原文编码器时不要求同时启用本地
BGE-M3。模型列表响应只暴露名称、provider、状态等白名单字段,不返回密钥或连接
参数。模型绑定只对新建库生效;需要跨知识库检索时,各库应使用相同或兼容的向量模型。
## DeerFlow-local LLMWiki Wiki vector index
LLMWiki can mirror processed WeKnora Wiki pages into DeerFlow and search only
@@ -1164,13 +1183,23 @@ llmwiki:
本地编码依赖随标准后端安装。将完整 BGE-M3 模型目录提前放到离线服务器,并通过
`local_model_path` 或 `DEERFLOW_BGE_M3_MODEL_PATH` 指向它。管理员也可在
「设置和更多 → 设置 → 本地编码器」中选择服务器目录,或从浏览器上传完整模型目录;
「设置和更多 → 设置 → Wiki 镜像编码器」中选择服务器目录,或从浏览器上传完整模型目录;
页面配置持久化到运行目录的 `system_settings.json`,上传文件落到
`.deer-flow/models/bge-m3`,均不进入源码仓库。系统强制 `local_files_only`,
`.deer-flow/models/bge-m3-revisions/<revision>` 的不可变版本目录,均不进入源码仓库。系统强制 `local_files_only`,
**不会访问互联网或下载模型**。管理员可在同一设置页点击「一键启动编码器」异步加载;该操作不挂在应用启动流程,加载失败不会
中断 DeerFlow 主服务。默认由独立低优先级子进程承载模型内存,正常停止 DeerFlow
时会一并回收该子进程;编码进程异常退出只会将当前向量任务标记失败。模型目录应包含
`onnx/model.onnx`(官方权重约 2.3 GB)及对应 tokenizer/config 文件。权重不进入业务源码仓库。
运行时编码器模式持久化为 `configured` 或 `local`。默认 `configured` 严格使用
`config.yaml` 中的 OpenAI、OpenAI-compatible 或 local provider;保存服务器目录或
上传模型会显式切到 `local` 并热重建 embedding/search/sync/retrieval 服务,不需要
重启 Gateway。`POST /api/llmwiki/local-encoder/use-configured` 可立即恢复配置文件中的
provider。`GET /api/llmwiki/local-encoder/status` 在远程 provider 下也正常返回当前
provider/mode,不再把“不是 local”当成状态接口错误。每次激活本地模型都会生成新的
模型修订指纹,避免同一路径替换权重后误复用旧向量。切换后的旧编码器会给在途任务
保留短暂宽限期,随后等待当前编码调用结束并自动回收模型进程。
浏览器目录上传采用磁盘暂存和 4 MB 分块复制,页面显示上传进度;经 Nginx 等反向
代理部署时仍需把请求体上限和读写超时配置到可承载完整模型目录。
相同模型指纹、相同内容的页面复用现有向量;短正文/摘要至少保留一个非空分段。