Merge branch 'master' of http://192.168.0.200/knowledge-base-bj/deerflow-code into master-nj
Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # frontend-web/src/components/workspace/workspace-nav-chat-list.tsx # frontend-web/src/pages/WorkspaceRoutes.tsx # offline-backend-20260512/backend/app/gateway/app.py # offline-backend-20260512/backend/app/gateway/deps.py # offline-backend-20260512/backend/pyproject.toml
This commit is contained in:
commit
cf9fe58905
109 files changed
+18863
-982
No files matched your search
@@ -162,6 +162,7 @@ FastAPI application providing REST endpoints for frontend integration:
|
||||
| `POST /api/threads/{id}/uploads` | Upload files (auto-converts PDF/PPT/Excel/Word to Markdown, rejects directory paths) |
|
||||
| `GET /api/threads/{id}/uploads/list` | List uploaded files |
|
||||
| `DELETE /api/threads/{id}` | Delete DeerFlow-managed local thread data after LangGraph thread deletion; unexpected failures are logged server-side and return a generic 500 detail |
|
||||
| `GET /api/artifact-library` | List the current user's generated conversation files across normal threads, with filename/conversation search, file-kind filtering, pagination, and an optional cache-bypassing refresh. The filesystem `outputs/` directories remain authoritative; uploads and image/audio/video files are excluded. |
|
||||
| `GET /api/threads/{id}/artifacts/{path}` | Serve generated artifacts |
|
||||
| `GET/POST /api/deep-research/*` | Durable Deep Research sessions and jobs. `quick`/`basic` run the compact report flow; `detailed` plans subtopics and writes independent sections; `deep` recursively investigates bounded follow-up branches. All modes reuse DeerFlow's configured material provider, LLM factory, hidden-thread artifacts, and SSE event log. Deep Research session/job/event/source/message writes serialise their rows in the writer transaction, rather than doing a post-commit ORM refresh or readback; this prevents MySQL read/write-splitting replica lag from failing “start writing”, dispatcher claim, report persistence, or follow-up history. Whole-document artifact/report rewrites likewise create their reversible version snapshots from writer-local ORM data and never refresh after commit, so replica lag cannot restore an otherwise successful rewrite. A report is written to the session and confirmed before its Job becomes `completed`; an unconfirmed write is retried and never exposed as a false completion. Retrying an already-created queued job re-nudges the dispatcher, recovering an interrupted HTTP response without creating a second report. If progress-event persistence, durable replay, or live fan-out is briefly unavailable, the worker/stream logs the degradation and continues or reconnects rather than failing the whole job. Whole-report rewrite accepts an optional `reportOutline` Markdown template; the durable job persists it and applies it as the chapter structure constraint. Optional report illustrations use a separately configured OpenAI-compatible image endpoint and are served only through the owning session's authenticated artifact route. |
|
||||
| `/api/enterprise-research/*` | Temporarily disabled. Its router and persistence initialization are not registered, so these endpoints return 404 and no enterprise-research tables are created or migrated during startup. |
|
||||
@@ -903,19 +904,24 @@ without deleting the audit row. DeerFlow stores only the artifact reference
|
||||
never the Markdown body and never a task-system delivery. Continue to preview
|
||||
the file through the existing thread-artifact endpoint.
|
||||
|
||||
## 普通知识库管理
|
||||
|
||||
在「普通知识库」的「我的知识库」或「公共 LLMWiki」中,知识库所有者和管理员可以点击卡片上的「编辑」,修改名称和描述(描述可清空)。保存后同步更新知识服务和本地记录,并刷新列表;系统「对话沉淀」库不提供编辑入口。
|
||||
|
||||
远端已经删除的知识库会在下次列表刷新时自动隐藏,包括个人、公共列表和知识库选择器;页面每 15 秒自动刷新,也可以手动点击「刷新」。远端服务请求失败时展示错误,不把连接故障当成删除,不自动删除本地映射或已有引用。
|
||||
|
||||
## DeerFlow-local LLMWiki Wiki vector index
|
||||
|
||||
LLMWiki can mirror processed WeKnora Wiki pages into DeerFlow and search only
|
||||
processed Wiki content. A knowledge base uses exact NumPy cosine similarity
|
||||
only after its whole Wiki has completed one coherent vectorization pass; until
|
||||
then internal search calls WeKnora's Wiki-page search/list API and reads the
|
||||
matched Wiki pages. It never falls back to raw documents or chunks. The local
|
||||
vector feature is disabled by default. Configure the three
|
||||
`llmwiki.local_wiki_index.embedding` values (`model`, `base_url`, and
|
||||
`api_key`), then change only `enabled` to `true` in `config.yaml`. Manual
|
||||
vectorization is the default. Set `auto_sync: true` only when periodic
|
||||
background synchronization is wanted; the scheduler waits one full interval
|
||||
after startup before its first scan, so boot does not trigger a rebuild.
|
||||
matched Wiki pages. It never falls back to raw documents or chunks. Configure
|
||||
`llmwiki.local_wiki_index.embedding`, then enable the local index in
|
||||
`config.yaml`. `auto_sync` defaults to true: the scheduler performs one
|
||||
immediate incremental scan at startup and then scans every configured interval
|
||||
(30 seconds by default). Wiki changes are content-hash checked, so unchanged
|
||||
pages reuse their vectors instead of being encoded again.
|
||||
`database_batch_size` caps each vector insert statement (default `50`) while
|
||||
the page replacement remains atomic, preventing a large Wiki from flooding
|
||||
the SQL connection in one statement.
|
||||
@@ -992,8 +998,10 @@ report-structure column for databases stamped by either old variant.
|
||||
Published knowledge bases also have a login-free, chrome-free reader at
|
||||
`/#/embed/knowledge/{mapping_id}` for direct third-party iframe use. With no
|
||||
`wikiSlug`, it selects the Wiki **索引** page first (recognized by title, page
|
||||
type, or `index`/`home`/`readme` slug), pins that page to the top of the public
|
||||
directory, and otherwise falls back to an overview/root page;
|
||||
type, or `index`/`home`/`readme` slug) and pins that page to the top of the
|
||||
public directory. If WeKnora has pages but no explicit index page, the reader
|
||||
creates a virtual `__index__` landing page from the public directory; empty
|
||||
dynamic-index groups also fall back to those page summaries;
|
||||
`?wikiSlug=...` deep-links one page. The reader supports
|
||||
`theme=light|dark`, `bg`, and `text` embed appearance parameters. Index-article
|
||||
catalogue sections (page-type counts, nested categories, page summaries) are
|
||||
@@ -1009,8 +1017,14 @@ only from the read-only `/api/public/llmwiki/knowledge-bases/{id}/...` namespace
|
||||
which rechecks `publication_status=published` on every request and denies the
|
||||
system conversation-deposit knowledge base. Private or unpublished bases are
|
||||
never exposed by this route. Anonymous consumers can discover the available
|
||||
public Wiki libraries through `GET /api/public/llmwiki/knowledge-bases`; each
|
||||
entry deliberately contains only the DeerFlow mapping `id` and display `name`.
|
||||
public Wiki libraries through `GET /api/public/llmwiki/knowledge-bases`. The
|
||||
endpoint validates each published mapping against WeKnora, omits stale remote
|
||||
mappings, and returns safe comprehensive metadata: description/type,
|
||||
document/chunk/processing/share counts, non-secret configuration and
|
||||
timestamps, Wiki page/type/status counts plus explicit/generated index mode and
|
||||
default embed route, and aggregate local-vector state/counts/timestamps. It
|
||||
never returns the WeKnora id, tenant/creator identity, storage credentials, API
|
||||
credentials, raw vectors, or per-page indexing errors.
|
||||
Protected `resource://` and supported object-storage images are fetched through
|
||||
the current mapping instead of being exposed as upstream URLs. Authenticated
|
||||
readers use `GET /api/llmwiki/knowledge-bases/{id}/files?file_path=...`; public
|
||||
@@ -1081,3 +1095,125 @@ uploads are parsed directly; legacy DOC/XLS/PPT and ODT/ODS/ODP files are
|
||||
normalized through an isolated LibreOffice profile before Markdown extraction.
|
||||
Run `./OFFLINE_INSTALL.sh --office-check` from the offline bundle to generate
|
||||
and re-parse Word, Excel, and PowerPoint samples without requiring a database.
|
||||
|
||||
## Skill knowledge distillation and assistant knowledge
|
||||
|
||||
The Gateway now exposes an administrator-only skill distillation pipeline at
|
||||
`/api/skill-knowledge`. Each request can send up to 200 skills to one writable
|
||||
WeKnora Wiki knowledge base; multi-base distillation is done by repeated
|
||||
operations so the local binding remains `skill_name + target_id`. The scanner
|
||||
never reads or uploads code/script files, strips Markdown code fences, blocks
|
||||
credential-like files and content, safely inspects ZIP members, and converts
|
||||
supported Office/PDF/spreadsheet files off the event loop. Jobs use database
|
||||
claims and renewable leases, stage snapshots before activation, retain the
|
||||
prior active version on failure, support per-item retry, and keep per-binding
|
||||
contribution records for safe detach/delete. Direct assistant-base targets,
|
||||
non-Wiki/FAQ/conversation/readonly targets, and direct Neo4j writes are rejected
|
||||
for this product version.
|
||||
|
||||
`/api/assistant-knowledge` manages DeerFlow-native assistant knowledge bases.
|
||||
There is one global full-library base named `知识梳理总库`, plus admin-created
|
||||
custom assistant bases. Reads and search are available to authenticated users;
|
||||
global initialization, custom-base creation, WeKnora export/import, source
|
||||
reimport, and source deletion are admin-only. Assistant bases only accept
|
||||
WeKnora export packages or imports triggered from a WeKnora mapping; direct
|
||||
skill import and arbitrary local-file import return 410. Importing into a custom
|
||||
assistant base also mirrors the same package into the global base, where
|
||||
canonical entities and relations are aligned across all imported sources.
|
||||
The database schema is introduced by Alembic revision `20260901_01`.
|
||||
|
||||
### Wiki 流转与本地小批量编码
|
||||
|
||||
所有登录用户都可在「技能 → 归纳到知识库」中多选自己可见的技能,并归纳到一个
|
||||
自己可写的 WeKnora Wiki 知识库;用户只能看到和维护自己发起的归纳记录,管理员
|
||||
接口仍可查看全量记录。技能归纳上传清理后的知识正文,等待 WeKnora 完成 Wiki 分析,再建立本地
|
||||
Wiki 向量并通过落盘导出包自动沉淀到「知识梳理」。内置技能没有数据库所有权记录
|
||||
也可以归纳。若解析结束却没有 Wiki,会提示检查生成模型和资料;文档解析完成
|
||||
不等于 Wiki 生成成功,例如生成服务余额不足仍可能表现为文档已完成。
|
||||
|
||||
管理员可以在普通知识库直接生成向量数据包,每次操作有独立后台记录和下载入口。
|
||||
助手知识库支持上传包、异步导入和导出;新建空的普通 Wiki 库可上传助手导出包。管理员可编辑助手知识库名称与描述,也可删除自建助手知识库及其 Wiki、向量、实体、关系、导入记录和落盘数据包;承担全库沉淀的「知识梳理总库」允许编辑但禁止删除。
|
||||
包保留正文、别名/标签、目录分类、双向链接、实体关系及 float32 向量。
|
||||
回流时通过 WeKnora API 重建目录/页面,向量进入 DeerFlow 本地 Wiki 索引,
|
||||
**不直接写 WeKnora 的底层向量或图数据库**。普通与助手 Wiki 使用同一个阅读组件。
|
||||
|
||||
小批量 CPU 编码可配置如下,无需额外部署 HTTP 编码服务:
|
||||
|
||||
```yaml
|
||||
llmwiki:
|
||||
local_wiki_index:
|
||||
enabled: true
|
||||
auto_sync: true
|
||||
sync_interval_seconds: 30
|
||||
embedding:
|
||||
provider: local
|
||||
model: bge-embedding-m3
|
||||
dimensions: 1024
|
||||
batch_size: 2
|
||||
local_threads: 2
|
||||
local_process_isolation: true
|
||||
# 离线部署时可指定已下载的 BAAI/bge-m3 模型目录:
|
||||
# local_model_path: /opt/models/bge-m3
|
||||
```
|
||||
|
||||
本地编码依赖随标准后端安装。将完整 BGE-M3 模型目录提前放到离线服务器,并通过
|
||||
`local_model_path` 或 `DEERFLOW_BGE_M3_MODEL_PATH` 指向它。管理员也可在
|
||||
「设置和更多 → 设置 → 本地编码器」中选择服务器目录,或从浏览器上传完整模型目录;
|
||||
页面配置持久化到运行目录的 `system_settings.json`,上传文件落到
|
||||
`.deer-flow/models/bge-m3`,均不进入源码仓库。系统强制 `local_files_only`,
|
||||
**不会访问互联网或下载模型**。管理员可在同一设置页点击「一键启动编码器」异步加载;该操作不挂在应用启动流程,加载失败不会
|
||||
中断 DeerFlow 主服务。默认由独立低优先级子进程承载模型内存,正常停止 DeerFlow
|
||||
时会一并回收该子进程;编码进程异常退出只会将当前向量任务标记失败。模型目录应包含
|
||||
`onnx/model.onnx`(官方权重约 2.3 GB)及对应 tokenizer/config 文件。权重不进入业务源码仓库。
|
||||
浏览器目录上传采用磁盘暂存和 4 MB 分块复制,页面显示上传进度;经 Nginx 等反向
|
||||
代理部署时仍需把请求体上限和读写超时配置到可承载完整模型目录。
|
||||
相同模型指纹、相同内容的页面复用现有向量;短正文/摘要至少保留一个非空分段。
|
||||
数据包导入不重新编码,模型指纹或维度不一致时明确拒绝,不能只按维度混用模型。
|
||||
问答编码查询本身仍需可用的同一模型。WeKnora 的原始文档向量不能当作生成后
|
||||
Wiki 正文的向量使用,两种文本需要分别建立各自的首次索引。
|
||||
|
||||
Wiki 页面新增、编辑、删除及文档上传/重解析都会立即触发一次增量检查;WeKnora
|
||||
后台稍后才完成的 Wiki 抽取由 30 秒周期补扫接管。每次成功向量化后自动把该普通
|
||||
知识库的最新完整快照同步到「知识梳理总库」,包括服务重启前已有库和新建库;来源
|
||||
页面被删除时也会在总库中退役,不保留失效 Wiki。
|
||||
|
||||
「知识梳理总库」不是按来源简单堆叠:导入时会规范化全/半角、空白、标点及常见实体
|
||||
类型,并利用实体别名对齐同一人物、组织、产品或概念;不同来源中同标题的实体 Wiki
|
||||
只保留一个规范页面,正文按去重后的知识块合并,来源与实体/关系贡献仍分别留痕。
|
||||
重复上传同一个 WeKnora 导出包时,以包内稳定知识库 ID 识别同一来源并执行快照替换,
|
||||
不会再产生一批新的片段副本。已有历史重复数据会在总库首次读取、检索、导出或下次
|
||||
同步时自动归并。助手知识检索排除目录、索引和摘要页;关键词只查标题与知识正文,
|
||||
向量检索返回实际命中的正文片段,问答上下文和检索卡片均不再用摘要代替知识。
|
||||
|
||||
大包采用流式 JSONL 落盘、同盘临时 SQLite 分阶段读取及异步导入,避免一次性将
|
||||
整个包加载进内存。部署需预留包、暂存索引和业务数据库空间,并配置代理上传大小
|
||||
与超时。当前不宣称断点续传或 GB 级压力验收已完成。
|
||||
|
||||
## openGauss / GaussDB application database
|
||||
|
||||
The application ORM can use a PG-compatible openGauss/GaussDB database through
|
||||
the official `opengauss-sqlalchemy` async dialect. Install the optional driver
|
||||
set before enabling it:
|
||||
|
||||
```bash
|
||||
uv sync --extra gauss
|
||||
```
|
||||
|
||||
Configure the business database independently from the LangGraph checkpointer:
|
||||
|
||||
```yaml
|
||||
database:
|
||||
backend: gauss
|
||||
gauss_url: $GAUSS_DATABASE_URL
|
||||
|
||||
checkpointer:
|
||||
type: sqlite
|
||||
connection_string: .deer-flow/data/deerflow.db
|
||||
```
|
||||
|
||||
`GAUSS_DATABASE_URL` should normally be
|
||||
`opengauss+asyncpg://user:password@host:26000/deerflow`. Plain
|
||||
`opengauss://`, `gaussdb://`, and PostgreSQL-style URLs are accepted and
|
||||
normalized to the openGauss dialect. Do not configure the LangGraph
|
||||
PostgreSQL checkpointer against GaussDB: its migrations are PostgreSQL-specific.
|
||||
For a production image, build with `--build-arg UV_EXTRAS=gauss`.
|
||||
Reference in new issue
Block a user