Merge remote-tracking branch 'origin/master' into ji
# Conflicts: # offline-backend-20260512/backend/app/gateway/app.py
This commit is contained in:
commit
67d12929fc
309 files changed
+49676
-5740
No files matched your search
@@ -1,5 +1,11 @@
|
||||
# DeerFlow Backend
|
||||
|
||||
普通知识库默认详情页面:管理员可在知识库顶部选择“本系统 Wiki 索引”或“嵌入 WeKnora”,对应 `ordinary_knowledge_landing`(默认 `wiki`)。Wiki 库首次进入默认展示本系统索引,右上角“打开 WeKnora”切换到内嵌管理页;内嵌页可返回索引。非 Wiki 库仍进入 WeKnora 文档页。设置通过 `/api/system-settings/ordinary-knowledge-landing` 保存,仅管理员可修改;指定文章、文档、图谱的直达链接优先于默认设置。
|
||||
|
||||
助手知识库可独立于 WeKnora 使用:管理员进入助手知识库,选择或创建库后点击“上传文件生成 Wiki”。支持 Markdown/TXT/CSV、PDF、Word、Excel、PowerPoint(单文件 100 MB,解析正文上限 400 万字符)。文件先写入运行目录 `assistant-knowledge/files/{job_id}`,后台分段调用已配置的语言模型生成分类 Wiki、实体关系,再通过 DeerFlow 编码器对正文分段向量化,保存到助手库并汇入知识梳理总库。无需配置 WeKnora URL;配置后仍可直接上传,也保留 WeKnora 数据包导入。文件处理沿用管理员写入权限,所有用户可以浏览和搜索。
|
||||
|
||||
导入记录显示解析、Wiki 生成和向量化阶段;失败或服务中断后可“重试 / 恢复处理”,复用已落盘的 Wiki 和相同模型指纹的向量。离线编码器须预先放置模型并在管理员设置中配置;模型或编码器不可用会明确失败,不把摘要冒充 Wiki 或虚构向量。扫描件是否可解析取决于部署的文档解析/OCR 能力。
|
||||
|
||||
DeerFlow is a LangGraph-based AI super agent with sandbox execution, persistent memory, and extensible tool integration. The backend enables AI agents to execute code, browse the web, manage files, delegate tasks to subagents, and retain context across conversations - all in isolated, per-thread environments.
|
||||
|
||||
---
|
||||
@@ -60,6 +66,10 @@ The single LangGraph agent (`lead_agent`) is the runtime entry point, created vi
|
||||
- Candidate workflow graphs remain editable after they are loaded into the Coze-compatible canvas; the compatibility response explicitly marks the current owner as editable so node dragging is not mistaken for a read-only preview.
|
||||
- Workflow-agent SSE forwards provider `reasoning_content` as an inline `<think>…</think>` trace for the UI while keeping the executable node output limited to the visible answer text. Every native LangGraph `[AIMessage|ToolMessage, metadata]` frame is additionally persisted and sent as `node.message.data.messages` without projection or length cropping, so the Studio can replay DeerFlow's tool-result, search-result and `ask_clarification` cards exactly after reconnecting.
|
||||
|
||||
### AgentScope report collaboration (in progress)
|
||||
|
||||
The independent `/api/report-collaboration` workbench supports durable run-time intervention commands and immutable formal report versions. Its production worker now runs a real AgentScope 2.0.5 `Agent` per TaskLedger assignment; AgentScope owns the member ReAct/tool/structured-output loop, while TaskLedger alone owns DAG readiness, retries and completion. Ready nodes in one wave run concurrently up to `max_parallel_tasks`; cancellation and member exceptions close agent runs as terminal states, and lease recovery remains independent of in-memory AgentScope objects. Retrieval uses the deployment's configured DeerFlow `web_search`/`web_fetch` tools through the frozen role permission snapshot. Every visible AgentScope reply, TeamSay handoff, tool call and tool result is persisted immediately into the existing SSE/message protocol; hidden reasoning and system prompts are not exposed. A constrained intent agent classifies explicit answers, angle additions, re-search, re-analysis, node reruns, section rewrites, polishing, report questions, replanning, and cancellation. Server-side impact analysis validates node/agent ownership and computes the minimum downstream lineage; low-confidence, high-cost, or multi-branch changes require confirmation. Commands are CAS-claimed and applied at TaskLedger safe points, where old attempts and artifacts become `superseded` so late results cannot contaminate the revised report. Run creation freezes the selected `/api/report-structures` row and all server-enforced budgets. Per-user active-run limits and per-run model-call, retrieval, source-byte, token, cost, node-time, and total-time limits fail closed. User-visible event payloads are centrally redacted before durable persistence without size truncation; external source content is explicitly untrusted, and hidden reasoning/system prompts are never exposed. Append-only audits cover agent selection, tools, sources, user commands, confirmations, report versions, redactions, and budget failures; owner-scoped reads are available at `GET /runs/{id}/audits`. Temporary Markdown is emitted as `report.delta`; only a template/citation/coverage-clean report with an independent passing review is atomically persisted and emitted as `report.version.created` before the run becomes completed. Section rewrites are replacement candidates; apply and restore append a new head version and never destroy history. Configure `runtime_mode`, concurrency and limits under `report_collaboration`; both the feature and worker remain disabled by default. Install the `report-collaboration` project extra when enabling the worker.
|
||||
|
||||
### Middleware Chain
|
||||
|
||||
Middlewares execute in strict order, each handling a specific concern:
|
||||
@@ -169,7 +179,7 @@ FastAPI application providing REST endpoints for frontend integration:
|
||||
| `DELETE /cop/tasks/{task_id}` | TaskCOP compatibility API (authenticated): delete one task and its task-scoped situation-report document |
|
||||
| `GET /taskAnalyseSearch/cop-task-three-list` | TaskCOP compatibility API (authenticated): paginated task list for the situation-overview sentiment page (`content`, `taskStatus`, `taskDirection`, `startTime`, `endTime`) |
|
||||
| `GET/POST/PUT /api/task-reports/by-task/{task_id}` | Shared task-scoped situation-report detail. Query, first save, and full update all use the legacy-compatible `{state, msg, data:[{id, taskId, createTime, updateTime, sessionId, contentJson, categoryType}]}` shape; `contentJson` is preserved as a JSON string. Saving a non-empty report automatically moves the matching TaskCOP task to status `25` (the legacy list's “已完成” state). |
|
||||
| `POST /api/task-reports/import/by-task/{task_id}` | TaskCOP detail import (authenticated): accept a `sq-report-mock.json`-compatible `data`/`records` array and atomically replace the selected task's full report. The path task id overrides task ids inside the file. |
|
||||
| `POST /api/task-reports/import/by-task/{task_id}` | TaskCOP detail import (authenticated): accept a `sq-report-mock.json`-compatible `data`/`records` array and atomically replace the selected task's full report. The path task id overrides task ids inside the file. The agentfx built-in agent fills that file with the `task-report-build` convert skill in one call (search JSON → 14 categories; `task-report-{enemy,our,env,judge}` re-run a single dashboard page) then imports via `task-report-import`. |
|
||||
| `POST /api/sentiment-agent/stream` | Authenticated BFF for the frontend sentiment-analysis virtual agent. Forwards the AG-UI JSON body to the `apiUrl` supplied from `runtime-config.js` (Basic Auth from the same payload). TLS certificate verification is off. Upstream SSE is piped through unchanged; connection/HTTP failures return the complete error in `detail`. |
|
||||
|
||||
The separately deployed TaskCOP Vue app uses `POST /api/v1/auth/login/username` directly (its `username`, `userId`, or `yUserId` URL parameter identifies the user), so this compatibility flow does not depend on the retired Consumer login service. It calls the Gateway address configured as `b1ConsumerUrl` directly; it does not use a Vite `/api` reverse proxy. An empty TaskCOP store remains empty; task data is introduced through normal creation or the JSON import API. The Gateway CORS middleware always merges local Vite origins `http://localhost|127.0.0.1:{5173,5174,3000,8080}`, including when `GATEWAY_CORS_ORIGINS` is set for another frontend. This keeps a Vite server that falls back from 5173 to 5174 (or the reverse) working after a backend restart. For a deployment, set `GATEWAY_CORS_ORIGINS` to the exact additional frontend origins that are allowed to call the Gateway.
|
||||
@@ -914,13 +924,12 @@ LLMWiki can mirror processed WeKnora Wiki pages into DeerFlow and search only
|
||||
processed Wiki content. A knowledge base uses exact NumPy cosine similarity
|
||||
only after its whole Wiki has completed one coherent vectorization pass; until
|
||||
then internal search calls WeKnora's Wiki-page search/list API and reads the
|
||||
matched Wiki pages. It never falls back to raw documents or chunks. The local
|
||||
vector feature is disabled by default. Configure the three
|
||||
`llmwiki.local_wiki_index.embedding` values (`model`, `base_url`, and
|
||||
`api_key`), then change only `enabled` to `true` in `config.yaml`. Manual
|
||||
vectorization is the default. Set `auto_sync: true` only when periodic
|
||||
background synchronization is wanted; the scheduler waits one full interval
|
||||
after startup before its first scan, so boot does not trigger a rebuild.
|
||||
matched Wiki pages. It never falls back to raw documents or chunks. Configure
|
||||
`llmwiki.local_wiki_index.embedding`, then enable the local index in
|
||||
`config.yaml`. `auto_sync` defaults to true: the scheduler performs one
|
||||
immediate incremental scan at startup and then scans every configured interval
|
||||
(30 seconds by default). Wiki changes are content-hash checked, so unchanged
|
||||
pages reuse their vectors instead of being encoded again.
|
||||
`database_batch_size` caps each vector insert statement (default `50`) while
|
||||
the page replacement remains atomic, preventing a large Wiki from flooding
|
||||
the SQL connection in one statement.
|
||||
@@ -1123,45 +1132,67 @@ The database schema is introduced by Alembic revision `20260901_01`.
|
||||
|
||||
### Wiki 流转与本地小批量编码
|
||||
|
||||
技能归纳现在上传清理后的知识正文,等待 WeKnora 完成 Wiki 分析,再建立本地
|
||||
所有登录用户都可在「技能 → 归纳到知识库」中多选自己可见的技能,并归纳到一个
|
||||
自己可写的 WeKnora Wiki 知识库;用户只能看到和维护自己发起的归纳记录,管理员
|
||||
接口仍可查看全量记录。技能归纳上传清理后的知识正文,等待 WeKnora 完成 Wiki 分析,再建立本地
|
||||
Wiki 向量并通过落盘导出包自动沉淀到「知识梳理」。内置技能没有数据库所有权记录
|
||||
也可以归纳。若解析结束却没有 Wiki,会提示检查生成模型和资料;文档解析完成
|
||||
不等于 Wiki 生成成功,例如生成服务余额不足仍可能表现为文档已完成。
|
||||
|
||||
管理员可以在普通知识库直接生成向量数据包,每次操作有独立后台记录和下载入口。
|
||||
助手知识库支持上传包、异步导入和导出;新建空的普通 Wiki 库可上传助手导出包。
|
||||
助手知识库支持上传包、异步导入和导出;新建空的普通 Wiki 库可上传助手导出包。管理员可编辑助手知识库名称与描述,也可删除自建助手知识库及其 Wiki、向量、实体、关系、导入记录和落盘数据包;承担全库沉淀的「知识梳理总库」允许编辑但禁止删除。
|
||||
包保留正文、别名/标签、目录分类、双向链接、实体关系及 float32 向量。
|
||||
回流时通过 WeKnora API 重建目录/页面,向量进入 DeerFlow 本地 Wiki 索引,
|
||||
**不直接写 WeKnora 的底层向量或图数据库**。普通与助手 Wiki 使用同一个阅读组件。
|
||||
|
||||
小批量 CPU 编码可配置如下,无需额外部署 HTTP 编码服务:
|
||||
|
||||
```bash
|
||||
uv sync --extra local-embedding
|
||||
```
|
||||
|
||||
```yaml
|
||||
llmwiki:
|
||||
local_wiki_index:
|
||||
enabled: true
|
||||
auto_sync: false
|
||||
auto_sync: true
|
||||
sync_interval_seconds: 30
|
||||
embedding:
|
||||
provider: local
|
||||
model: bge-embedding-m3
|
||||
dimensions: 1024
|
||||
batch_size: 2
|
||||
local_threads: 2
|
||||
local_process_isolation: true
|
||||
# 离线部署时可指定已下载的 BAAI/bge-m3 模型目录:
|
||||
# local_model_path: /opt/models/bge-m3
|
||||
```
|
||||
|
||||
首次使用需要预下载官方 ONNX 权重(约 2.3 GB),默认持久缓存位于
|
||||
`~/.cache/deerflow/embeddings`,也可设置 `local_cache_dir`。权重不进入业务源码仓库。
|
||||
本地编码依赖随标准后端安装。将完整 BGE-M3 模型目录提前放到离线服务器,并通过
|
||||
`local_model_path` 或 `DEERFLOW_BGE_M3_MODEL_PATH` 指向它。管理员也可在
|
||||
「设置和更多 → 设置 → 本地编码器」中选择服务器目录,或从浏览器上传完整模型目录;
|
||||
页面配置持久化到运行目录的 `system_settings.json`,上传文件落到
|
||||
`.deer-flow/models/bge-m3`,均不进入源码仓库。系统强制 `local_files_only`,
|
||||
**不会访问互联网或下载模型**。管理员可在同一设置页点击「一键启动编码器」异步加载;该操作不挂在应用启动流程,加载失败不会
|
||||
中断 DeerFlow 主服务。默认由独立低优先级子进程承载模型内存,正常停止 DeerFlow
|
||||
时会一并回收该子进程;编码进程异常退出只会将当前向量任务标记失败。模型目录应包含
|
||||
`onnx/model.onnx`(官方权重约 2.3 GB)及对应 tokenizer/config 文件。权重不进入业务源码仓库。
|
||||
浏览器目录上传采用磁盘暂存和 4 MB 分块复制,页面显示上传进度;经 Nginx 等反向
|
||||
代理部署时仍需把请求体上限和读写超时配置到可承载完整模型目录。
|
||||
相同模型指纹、相同内容的页面复用现有向量;短正文/摘要至少保留一个非空分段。
|
||||
数据包导入不重新编码,模型指纹或维度不一致时明确拒绝,不能只按维度混用模型。
|
||||
问答编码查询本身仍需可用的同一模型。WeKnora 的原始文档向量不能当作生成后
|
||||
Wiki 正文的向量使用,两种文本需要分别建立各自的首次索引。
|
||||
|
||||
Wiki 页面新增、编辑、删除及文档上传/重解析都会立即触发一次增量检查;WeKnora
|
||||
后台稍后才完成的 Wiki 抽取由 30 秒周期补扫接管。每次成功向量化后自动把该普通
|
||||
知识库的最新完整快照同步到「知识梳理总库」,包括服务重启前已有库和新建库;来源
|
||||
页面被删除时也会在总库中退役,不保留失效 Wiki。
|
||||
|
||||
「知识梳理总库」不是按来源简单堆叠:导入时会规范化全/半角、空白、标点及常见实体
|
||||
类型,并利用实体别名对齐同一人物、组织、产品或概念;不同来源中同标题的实体 Wiki
|
||||
只保留一个规范页面,正文按去重后的知识块合并,来源与实体/关系贡献仍分别留痕。
|
||||
重复上传同一个 WeKnora 导出包时,以包内稳定知识库 ID 识别同一来源并执行快照替换,
|
||||
不会再产生一批新的片段副本。已有历史重复数据会在总库首次读取、检索、导出或下次
|
||||
同步时自动归并。助手知识检索排除目录、索引和摘要页;关键词只查标题与知识正文,
|
||||
向量检索返回实际命中的正文片段,问答上下文和检索卡片均不再用摘要代替知识。
|
||||
|
||||
大包采用流式 JSONL 落盘、同盘临时 SQLite 分阶段读取及异步导入,避免一次性将
|
||||
整个包加载进内存。部署需预留包、暂存索引和业务数据库空间,并配置代理上传大小
|
||||
与超时。当前不宣称断点续传或 GB 级压力验收已完成。
|
||||
|
||||
Reference in new issue
Block a user