feat: add local LLMWiki wiki vector index
This commit is contained in:
1 parent
4c9f75f31e
commit
3785345353
48 files changed
+3650
-152
No files matched your search
@@ -746,3 +746,38 @@ without deleting the audit row. DeerFlow stores only the artifact reference
|
||||
(`threadId` + sandbox `path`, plus the optional session/node/name/MIME fields),
|
||||
never the Markdown body and never a task-system delivery. Continue to preview
|
||||
the file through the existing thread-artifact endpoint.
|
||||
|
||||
## DeerFlow-local LLMWiki Wiki vector index
|
||||
|
||||
LLMWiki can mirror processed WeKnora Wiki pages into DeerFlow and search only
|
||||
processed Wiki content. A knowledge base uses exact NumPy cosine similarity
|
||||
only after its whole Wiki has completed one coherent vectorization pass; until
|
||||
then internal search calls WeKnora's Wiki-page search/list API and reads the
|
||||
matched Wiki pages. It never falls back to raw documents or chunks. The local
|
||||
vector feature is disabled by default. Configure
|
||||
`llmwiki.local_wiki_index.embedding` first, then enable `enabled`,
|
||||
`sync_enabled`, and `internal_search_enabled` in `config.yaml`.
|
||||
|
||||
The migration creates `llmwiki_wiki_pages`, `llmwiki_wiki_vectors`, and
|
||||
`llmwiki_wiki_sync_states`. Wiki Markdown and normalized float32 vectors are
|
||||
stored in DeerFlow's configured SQL database. Multi-worker sync is protected by
|
||||
database leases; searches use revision-checked per-knowledge-base LRU snapshots
|
||||
after the full-library readiness gate succeeds. Knowledge-base owners can click
|
||||
**手动向量化** in Wiki management (`POST
|
||||
/api/llmwiki/knowledge-bases/{id}/vectorize`) and follow its status at
|
||||
`GET /api/llmwiki/knowledge-bases/{id}/wiki-index-status`. Existing chunk
|
||||
citations remain readable while all new retrieval citations open Wiki pages.
|
||||
|
||||
Wiki management also provides **导出 Excel**. `GET
|
||||
/api/llmwiki/knowledge-bases/{id}/wiki/export.xlsx` exports every processed Wiki
|
||||
page, including its full directory path, current directory, parent-directory
|
||||
path, one column per directory level, metadata, and complete Markdown content
|
||||
(split across continuation columns when Excel's per-cell limit is reached).
|
||||
|
||||
Authenticated operations are available below `/api/llmwiki/wiki-index` and
|
||||
`POST /api/llmwiki/wiki-vector-search`. The service-to-service endpoints below
|
||||
`/api/external/llmwiki/wiki` require `X-API-Key`, are rate-limited, and only
|
||||
expose mappings that are both internally published and explicitly marked
|
||||
`external_search_enabled`; the system `对话沉淀` mapping is always denied.
|
||||
See `docs/LLMWIKI_DEERFLOW_LOCAL_VECTOR_INDEX_ZH.md` for configuration,
|
||||
security boundaries, rollout, and API contracts.
|
||||
Reference in new issue
Block a user