feat: add local LLMWiki wiki vector index

This commit is contained in:
Administrator committed 2026-08-24 00:49:31 +08:00
1 parent 4c9f75f31e
commit 3785345353
48 files changed
+3650 -152

No files matched your search

@@ -746,3 +746,38 @@ without deleting the audit row. DeerFlow stores only the artifact reference
(`threadId` + sandbox `path`, plus the optional session/node/name/MIME fields),
never the Markdown body and never a task-system delivery. Continue to preview
the file through the existing thread-artifact endpoint.
## DeerFlow-local LLMWiki Wiki vector index
LLMWiki can mirror processed WeKnora Wiki pages into DeerFlow and search only
processed Wiki content. A knowledge base uses exact NumPy cosine similarity
only after its whole Wiki has completed one coherent vectorization pass; until
then internal search calls WeKnora's Wiki-page search/list API and reads the
matched Wiki pages. It never falls back to raw documents or chunks. The local
vector feature is disabled by default. Configure
`llmwiki.local_wiki_index.embedding` first, then enable `enabled`,
`sync_enabled`, and `internal_search_enabled` in `config.yaml`.
The migration creates `llmwiki_wiki_pages`, `llmwiki_wiki_vectors`, and
`llmwiki_wiki_sync_states`. Wiki Markdown and normalized float32 vectors are
stored in DeerFlow's configured SQL database. Multi-worker sync is protected by
database leases; searches use revision-checked per-knowledge-base LRU snapshots
after the full-library readiness gate succeeds. Knowledge-base owners can click
**手动向量化** in Wiki management (`POST
/api/llmwiki/knowledge-bases/{id}/vectorize`) and follow its status at
`GET /api/llmwiki/knowledge-bases/{id}/wiki-index-status`. Existing chunk
citations remain readable while all new retrieval citations open Wiki pages.
Wiki management also provides **导出 Excel**. `GET
/api/llmwiki/knowledge-bases/{id}/wiki/export.xlsx` exports every processed Wiki
page, including its full directory path, current directory, parent-directory
path, one column per directory level, metadata, and complete Markdown content
(split across continuation columns when Excel's per-cell limit is reached).
Authenticated operations are available below `/api/llmwiki/wiki-index` and
`POST /api/llmwiki/wiki-vector-search`. The service-to-service endpoints below
`/api/external/llmwiki/wiki` require `X-API-Key`, are rate-limited, and only
expose mappings that are both internally published and explicitly marked
`external_search_enabled`; the system `对话沉淀` mapping is always denied.
See `docs/LLMWIKI_DEERFLOW_LOCAL_VECTOR_INDEX_ZH.md` for configuration,
security boundaries, rollout, and API contracts.