feat: add local LLMWiki wiki vector index
This commit is contained in:
1 parent
4c9f75f31e
commit
3785345353
48 files changed
+3650
-152
No files matched your search
@@ -1329,3 +1329,52 @@ candidates fail with 409 rather than overwrite a later edit. Ambiguous or
|
||||
ordinary evidence questions must remain non-mutating, and cancellation or a
|
||||
stream error must never save a partial report. The rewrite prompt receives only
|
||||
the selected range plus the selected evidence set, never a new web search.
|
||||
|
||||
### DeerFlow-local LLMWiki Wiki index
|
||||
|
||||
`llmwiki.local_wiki_index` is an opt-in local-vector subsystem. WeKnora is the
|
||||
Wiki generation/sync source and the Wiki-only fallback while an entire mapping
|
||||
has not completed vectorization. The readiness gate requires `state=idle`, a
|
||||
successful full scan, matching embedding fingerprint, no failed pages,
|
||||
`ready_page_count == local_page_count`, and vectors for a non-empty library.
|
||||
Ready mappings read DeerFlow's `llmwiki_wiki_pages` /
|
||||
`llmwiki_wiki_vectors` / `llmwiki_wiki_sync_states` tables; incomplete mappings
|
||||
call only WeKnora `/knowledgebase/{id}/wiki/pages?query=...` plus Wiki-page
|
||||
detail. Never call `/knowledge-search` from internal search, RAG, or the
|
||||
`llmwiki_search` tool, and never return raw chunks as the fallback. Core code lives under
|
||||
`deerflow.integrations.weknora.local_index` and
|
||||
`deerflow.persistence.llmwiki_index`; preserve the harness → app import
|
||||
firewall. Gateway composition injects the store/search/sync services through
|
||||
`local_index.runtime`, and the lease-protected scheduler starts only when both
|
||||
`enabled` and `sync_enabled` are true.
|
||||
|
||||
Vectors are normalized little-endian float32 BLOBs. A page mirror and all of
|
||||
its vectors are replaced in one transaction only after strict batched
|
||||
embedding succeeds. Never delete the old generation before a remote embedding
|
||||
call and never fall back to WeKnora raw chunks while the local feature flag is
|
||||
enabled. NumPy matrices are per-mapping LRU snapshots invalidated by
|
||||
`index_revision`; large decode/matrix work runs through `asyncio.to_thread`.
|
||||
During a fingerprint rebuild the mapping uses the Wiki-page API until every
|
||||
page is successfully rebuilt. `WikiRetrievalService` partitions mixed requests
|
||||
per mapping, so ready libraries stay local while incomplete libraries use that
|
||||
Wiki-only fallback.
|
||||
|
||||
Owners manually force a full sync/vectorization through `POST
|
||||
/api/llmwiki/knowledge-bases/{id}/vectorize`; status is available through `GET
|
||||
/api/llmwiki/knowledge-bases/{id}/wiki-index-status`. The Wiki management page
|
||||
also exports all remote processed Wiki pages via `GET
|
||||
/api/llmwiki/knowledge-bases/{id}/wiki/export.xlsx`. Directory export must keep
|
||||
the full path, current directory, parent path, and dynamic per-level columns;
|
||||
long Markdown is split into Excel-safe continuation columns instead of being
|
||||
truncated. Workbook creation runs in `asyncio.to_thread`.
|
||||
|
||||
Internal search authorization always starts from DeerFlow mapping ids and is
|
||||
rechecked by the Gateway. External endpoints use the isolated
|
||||
`/api/external/llmwiki/wiki` namespace and `X-API-Key`; require published +
|
||||
external-enabled + index-enabled on every request and response, and hard-deny
|
||||
the `system` / `对话沉淀` mapping. Never return remote WeKnora ids, provenance
|
||||
refs, credentials, or vector bytes externally. Regression tests:
|
||||
`tests/test_llmwiki_local_index_repository.py`,
|
||||
`tests/test_llmwiki_wiki_sync.py`, `tests/test_llmwiki_vector_search.py`, and
|
||||
`tests/test_external_llmwiki_search.py`, and
|
||||
`tests/test_llmwiki_wiki_retrieval.py`.
|
||||
Reference in new issue
Block a user