feat: improve knowledge sync and entity alignment
This commit is contained in:
1 parent
7dbac9cc56
commit
dea4697ea0
38 files changed
+3742
-634
No files matched your search
@@ -1550,10 +1550,12 @@ detail. Never call `/knowledge-search` from internal search, RAG, or the
|
||||
`deerflow.integrations.weknora.local_index` and
|
||||
`deerflow.persistence.llmwiki_index`; preserve the harness → app import
|
||||
firewall. Gateway composition injects the store/search/sync services through
|
||||
`local_index.runtime`. `enabled` turns on local vector search and manual
|
||||
vectorization; `auto_sync` is the separate opt-in for the lease-protected
|
||||
scheduler. The scheduler sleeps for one full configured interval before its
|
||||
first scan, so application startup must never trigger immediate vectorization.
|
||||
`local_index.runtime`. `enabled` turns on local vector search and `auto_sync`
|
||||
defaults to the lease-protected background scheduler. The scheduler performs
|
||||
an immediate incremental scan at startup, then rescans every configured
|
||||
interval so new WeKnora libraries and asynchronously generated Wiki pages are
|
||||
picked up without a manual vectorization click. Unchanged content must reuse
|
||||
existing vectors.
|
||||
|
||||
Vectors are normalized little-endian float32 BLOBs. A page mirror and all of
|
||||
its vectors are replaced in one transaction only after strict batched
|
||||
@@ -1735,7 +1737,9 @@ snapshot remains authoritative on failure. A binding may have only one active
|
||||
item (`active_dedupe_key` plus atomic lease claims). Skill distillation targets
|
||||
only writable WeKnora Wiki knowledge bases in this product version; reject direct
|
||||
assistant-base targets, non-Wiki/FAQ/conversation/readonly targets, and direct
|
||||
Neo4j writes. Detach/delete operates on source contributions, not canonical rows,
|
||||
Neo4j writes. The create/list/resync/retry/detach flow is available to ordinary
|
||||
users for their own visible skills, writable knowledge bases, jobs and bindings;
|
||||
administrator-only auto-sync/review APIs remain privileged. Detach/delete operates on source contributions, not canonical rows,
|
||||
and shared entities/relations are retired only when their final contribution is
|
||||
gone.
|
||||
|
||||
@@ -1745,7 +1749,10 @@ administrators own all mutations. Assistant bases only accept WeKnora export
|
||||
packages or imports triggered from a WeKnora mapping. Do not add direct skill
|
||||
imports or arbitrary local-file imports to assistant knowledge. Importing into a
|
||||
custom assistant base must also mirror the package into the global base for
|
||||
cross-source entity alignment.
|
||||
cross-source entity alignment. Admins may rename any assistant base and delete
|
||||
custom bases. Deletion removes the custom base's Wiki pages, vectors, entities,
|
||||
relations, import audit rows, and package files; the global base is never
|
||||
deletable because it owns full-library aggregation.
|
||||
|
||||
### Wiki package and local encoder invariants (2026-09)
|
||||
|
||||
@@ -1762,16 +1769,42 @@ cross-source entity alignment.
|
||||
- Preserve normalized float32 base64 bytes exactly. Same dimension alone is not
|
||||
model compatibility. Assistant vector retrieval is fingerprint-gated cosine
|
||||
search, and any keyword fallback must be labeled as such.
|
||||
Preserve a valid zero section_index. Export/search vectors belonging to the
|
||||
current Wiki revision's import job, not stale contributions from other sources.
|
||||
Preserve a valid zero section_index. Export/search vectors belonging to each
|
||||
current source snapshot (`source.last_import_job_id`), not merely the page's
|
||||
current revision: one aligned page may intentionally contain current vectors
|
||||
from several sources.
|
||||
- The `local` embedding provider uses FastEmbed with official BAAI/bge-m3 ONNX,
|
||||
1024 dimensions, bounded CPU batches and a per-client async lock. Run inference
|
||||
off the event loop. Existing page vectors are reusable; zero-vector nonempty
|
||||
pages need backfill, not a forced rebuild of every page.
|
||||
FastEmbed is a standard backend dependency. Encoder configuration, directory
|
||||
upload and warmup live only in the administrator global settings dialog.
|
||||
A runtime `system_settings.json` path override wins over config.yaml; uploaded
|
||||
files are streamed into `.deer-flow/models/bge-m3`, never the source tree.
|
||||
Encoder warmup is explicitly started from the settings UI/API and must never block application startup
|
||||
or turn a model-load failure into a DeerFlow service failure. Local provider
|
||||
is offline-only: require local_model_path or DEERFLOW_BGE_M3_MODEL_PATH and
|
||||
never let the application download model weights. Keep
|
||||
`local_process_isolation=true` by default: inference runs in the bundled JSONL
|
||||
worker subprocess, and shutdown must terminate that worker without waiting on
|
||||
model teardown.
|
||||
- Every successful mapping snapshot is mirrored to the global assistant base,
|
||||
including empty snapshots so deleted source pages are retired. Re-import is
|
||||
source-replacement semantics: absent Wiki/chunk/entity/relation pages must not
|
||||
remain visible merely because they existed in an older source snapshot.
|
||||
- Provider graph nodes bind to their original wiki_slug. They must not create
|
||||
duplicate synthetic entity/relation articles. Store original Wiki metadata
|
||||
separately from assistant workflow status. Count Wiki pages separately from
|
||||
raw document chunks.
|
||||
- The global base aligns entities by normalized display name and aliases across
|
||||
source records, and aligns content Wiki pages by normalized title. Do not use
|
||||
source slug as the global canonical identity. Merge unique knowledge blocks
|
||||
while retaining source contributions/origins. Uploaded packages must rebind
|
||||
the temporary upload source to the stable WeKnora knowledge-base id in the
|
||||
package, so re-upload is source-replacement rather than append. Retrieval
|
||||
candidates exclude summary/index/directory pages; keyword matching, vector
|
||||
snippets, citations and prompts must use knowledge body/sections, never page
|
||||
summary as a substitute.
|
||||
- Migration `20260903_01` adds nullable page metadata and merges the pre-existing
|
||||
knowledge/workflow Alembic heads. Regression coverage is in
|
||||
`tests/test_wiki_package_roundtrip.py`, `test_llmwiki_wiki_sync.py` and
|
||||
|
||||
Reference in new issue
Block a user