feat: add local LLMWiki wiki vector index

This commit is contained in:
Administrator committed 2026-08-24 00:49:31 +08:00
1 parent 4c9f75f31e
commit 3785345353
48 files changed
+3650 -152

No files matched your search

@@ -1329,3 +1329,52 @@ candidates fail with 409 rather than overwrite a later edit. Ambiguous or
ordinary evidence questions must remain non-mutating, and cancellation or a
stream error must never save a partial report. The rewrite prompt receives only
the selected range plus the selected evidence set, never a new web search.
### DeerFlow-local LLMWiki Wiki index
`llmwiki.local_wiki_index` is an opt-in local-vector subsystem. WeKnora is the
Wiki generation/sync source and the Wiki-only fallback while an entire mapping
has not completed vectorization. The readiness gate requires `state=idle`, a
successful full scan, matching embedding fingerprint, no failed pages,
`ready_page_count == local_page_count`, and vectors for a non-empty library.
Ready mappings read DeerFlow's `llmwiki_wiki_pages` /
`llmwiki_wiki_vectors` / `llmwiki_wiki_sync_states` tables; incomplete mappings
call only WeKnora `/knowledgebase/{id}/wiki/pages?query=...` plus Wiki-page
detail. Never call `/knowledge-search` from internal search, RAG, or the
`llmwiki_search` tool, and never return raw chunks as the fallback. Core code lives under
`deerflow.integrations.weknora.local_index` and
`deerflow.persistence.llmwiki_index`; preserve the harness → app import
firewall. Gateway composition injects the store/search/sync services through
`local_index.runtime`, and the lease-protected scheduler starts only when both
`enabled` and `sync_enabled` are true.
Vectors are normalized little-endian float32 BLOBs. A page mirror and all of
its vectors are replaced in one transaction only after strict batched
embedding succeeds. Never delete the old generation before a remote embedding
call and never fall back to WeKnora raw chunks while the local feature flag is
enabled. NumPy matrices are per-mapping LRU snapshots invalidated by
`index_revision`; large decode/matrix work runs through `asyncio.to_thread`.
During a fingerprint rebuild the mapping uses the Wiki-page API until every
page is successfully rebuilt. `WikiRetrievalService` partitions mixed requests
per mapping, so ready libraries stay local while incomplete libraries use that
Wiki-only fallback.
Owners manually force a full sync/vectorization through `POST
/api/llmwiki/knowledge-bases/{id}/vectorize`; status is available through `GET
/api/llmwiki/knowledge-bases/{id}/wiki-index-status`. The Wiki management page
also exports all remote processed Wiki pages via `GET
/api/llmwiki/knowledge-bases/{id}/wiki/export.xlsx`. Directory export must keep
the full path, current directory, parent path, and dynamic per-level columns;
long Markdown is split into Excel-safe continuation columns instead of being
truncated. Workbook creation runs in `asyncio.to_thread`.
Internal search authorization always starts from DeerFlow mapping ids and is
rechecked by the Gateway. External endpoints use the isolated
`/api/external/llmwiki/wiki` namespace and `X-API-Key`; require published +
external-enabled + index-enabled on every request and response, and hard-deny
the `system` / `对话沉淀` mapping. Never return remote WeKnora ids, provenance
refs, credentials, or vector bytes externally. Regression tests:
`tests/test_llmwiki_local_index_repository.py`,
`tests/test_llmwiki_wiki_sync.py`, `tests/test_llmwiki_vector_search.py`, and
`tests/test_external_llmwiki_search.py`, and
`tests/test_llmwiki_wiki_retrieval.py`.