Merge branch 'master' of http://192.168.0.200/knowledge-base-bj/deerflow-code into master-nj
Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # frontend-web/src/components/workspace/workspace-nav-chat-list.tsx # frontend-web/src/pages/WorkspaceRoutes.tsx # offline-backend-20260512/backend/app/gateway/app.py # offline-backend-20260512/backend/app/gateway/deps.py # offline-backend-20260512/backend/pyproject.toml
This commit is contained in:
commit
cf9fe58905
109 files changed
+18863
-982
No files matched your search
@@ -396,7 +396,7 @@ FastAPI application on port 8001 with health checks at `GET|HEAD /health`, `/api
|
||||
| **Uploads** (`/api/threads/{id}/uploads`) | `POST /` - upload files (auto-converts PDF/PPT/Excel/Word); `GET /list` - list; `DELETE /{filename}` - delete |
|
||||
| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; unexpected failures are logged server-side and return a generic 500 detail |
|
||||
| **Threads search** (`POST /threads/search`, LangGraph-compat) | Conversation listing. **System threads are excluded by default**: rows whose `metadata.system` is truthy (scheduler runs, roundtable workers) never crowd the user's list; a caller whose `metadata` filter explicitly references `thread_type`/`system` (e.g. studio notebooks) opts out of the exclusion. New `query` field = case-insensitive title (`display_name`) substring search, used by the chats page's server-side pagination. `POST /threads/count` takes the same request body (limit/offset ignored) and returns `{total}` under identical filtering/exclusion semantics ? powers the chats page's ??/??? display without fetching rows (`ThreadMetaStore.count`). `ThreadMetaStore.search` gained `exclude_system`/`query` params; JSON-blob filters batch-scan until the requested `limit`/`offset` window is full (so pagination over a heavily-filtered set never under-fills), with a `thread_id` tie-breaker on the `updated_at DESC` ordering for stable OFFSET paging. Tests: `tests/test_thread_meta_search.py` |
|
||||
| **Artifacts** (`/api/threads/{id}/artifacts`) | `GET /{path}` - serve artifacts; active content types (`text/html`, `application/xhtml+xml`, `image/svg+xml`) are always forced as download attachments to reduce XSS risk; `?download=true` still forces download for other file types |
|
||||
| **Artifacts** (`/api/threads/{id}/artifacts`, `/api/artifact-library`) | `GET /{path}` - serve artifacts; active content types (`text/html`, `application/xhtml+xml`, `image/svg+xml`) are always forced as download attachments to reduce XSS risk; `?download=true` still forces download for other file types. `GET /api/artifact-library` scans every current-user normal thread's authoritative `outputs/` directory (including files omitted from `present_files`), excludes uploads and image/audio/video media, joins the owning conversation title/agent route metadata, and exposes search/filter/pagination with a bounded 15-second per-user cache plus `refresh=true` bypass. Frontend route `/page/workspace/library` links each item back to the source chat with `?artifact=<path>` so the chat opens the existing artifact panel directly. |
|
||||
| **Word export** (`POST /api/writing/export/docx`) | Shared Markdown → DOCX generator used by conversation, artifact, scheduled-task, Deep Research, and AI-writing download surfaces. `app/gateway/word_export.py` builds OOXML with no Python conversion dependency: A4 portrait, 3.7/3.5/2.8/2.6 cm margins, real multilevel heading numbering (`一、` / `(一)` / `1.`), two-character first-line indent on heading levels 1–3, formal body/table styles, page numbers beginning at 1 with odd pages left-aligned and even pages right-aligned, and embedded `app/gateway/assets/fonts/方正小标宋简体.ttf`. Before numbering it removes complete manual heading prefixes (including `2.1` / `2.1.1`) and drops Markdown horizontal rules from Word output. The ordinary-Q&A Lead Agent can emit the matching Markdown heading forms and avoid unordered bullet lists when the admin enables `prompt_prefix.ordinary_qa_markdown_format_enabled` in `system_settings.json`; the switch defaults off, while writing setup, writing mode, notebook, scheduled runs, and SOUL-driven custom agents keep their independent prompts. The font loader validates its internal family, SHA-256, and OpenType embedding permission (`OS/2.fsType`) before export. Tests: `tests/test_word_export.py`, `tests/test_es_query_routing_prompt.py`. |
|
||||
| **Deep Research** (`/api/deep-research`) | Durable, user-owned research sessions/jobs with replayable SSE events and hidden-thread artifacts. `basic`/`quick` use the compact runner; `detailed` creates bounded subtopics, searches/writes sections concurrently, and assembles a report; `deep` recursively searches follow-up questions under a deterministic breadth/depth query budget. `multi_agent` is a LangGraph 1.x editor/researcher/writer/reviewer graph: its initial plan emits `awaiting_input`, the recorder stores the interrupt and releases the lease, and `POST /jobs/{id}/resume` rebuilds the graph from the persisted approved/revised plan. A bounded `custom_outline` is frozen into the session; headings/list entries deterministically supply the detailed and multi-agent plan, and the shared writer receives it only as a guarded content constraint. Completed reports support owner-scoped follow-up (`GET /sessions/{id}/messages`, `POST /sessions/{id}/chat`): context is only the current report, selected sources and recent messages; source-id citations are persisted, while `allow_new_research=true` is explicitly rejected. Owners may change that evidence set only after the job stops via `PATCH /sessions/{id}/sources/{source_id}`; this applies to future follow-ups rather than rewriting a completed report. Optional report illustrations are gated by `config.yaml -> deep_research.image_generation`: the Harness `OpenAICompatibleImageProvider` requires base64 output, the shared runner step writes only bounded `research-image-01..04.(png|jpg|webp)` artifacts and Markdown uses private `deep-research://` links. `GET /sessions/{id}/artifacts/{name}` checks session ownership before serving an image; an unavailable provider yields a warning but never fails the text report. Runners live in `packages/harness/deerflow/agents/deep_research/runners/`, may never import `app.*`, and use only the `AdapterBundle`. |
|
||||
| **Gateway auth** | Owner-scoped Deep Research routes await the platform's asynchronous `get_current_user` resolver before they read or write persistence, ensuring each session and job operation receives a concrete user ID. Deep Research artifacts are always resolved from a whitelisted filename via `/mnt/user-data/outputs/<name>` before mapping to a host path, preserving the sandbox guard on native Windows and Linux hosts. |
|
||||
@@ -1126,6 +1126,8 @@ Both can be modified at runtime via Gateway API endpoints or `DeerFlowClient` me
|
||||
|
||||
### Conditional WeKnora LLMWiki
|
||||
|
||||
Ordinary knowledge-base cards expose an owner/admin-only name/description edit dialog in `WeKnoraKnowledgeWorkspace.tsx`, reusing `useLlmWikiActions().update` and `PATCH /api/llmwiki/knowledge-bases/{id}`. The route trims names, rejects blank or reserved conversation-deposit names, writes to WeKnora before updating local metadata, and retains existing write authorization. A successful remote list is authoritative for every list scope, including `personal`: absent remote IDs are hidden without deleting mappings/references. Upstream failures remain errors. The grid also filters cached `remote_status=missing` rows and no longer renders stale-record cleanup cards. Regression coverage: `tests/test_llmwiki_knowledge_management.py`.
|
||||
|
||||
`llmwiki.weknora.api_base_url` is the provider switch: non-empty enables WeKnora, empty preserves the legacy LLMWiki UI. WeKnora is independently deployed and owns PostgreSQL/Redis/object storage/vector storage/parsers/models; DeerFlow config contains only `api_base_url` and optional `web_base_url`. DeerFlow deliberately sends no WeKnora credential, so the WeKnora endpoint must be restricted to a trusted network and allow DeerFlow traffic itself or through its reverse proxy. There is intentionally no `default_embedding_model_id`: `WeKnoraClient` queries `/api/v1/models` and uses WeKnora's active Embedding model when creating a KB; Wiki creation additionally resolves an active KnowledgeQA model inside WeKnora for synthesis. The native BFF is `app/gateway/routers/llmwiki.py`; ownership/publication mappings live in `deerflow.persistence.llmwiki`; the browser only sees DeerFlow mapping IDs. Knowledge-base owners publish immediately (`private` -> `published`) without a review queue, and can unpublish ordinary bases. The conversation-deposit KB is system-owned, always published, shared by all DeerFlow users, excluded from My Knowledge, and immutable through user/admin detail operations; recognition requires both the reserved name and `system` owner. Existing same-name remote mappings are adopted in place as the system deposit, preserving mapping ids and document links. Its user-facing description, generated Markdown source and tags use `cmzs` branding; list/detail reads synchronize stale remote descriptions, and search results, source previews and embedded-detail JSON recursively replace legacy brand text in historical conversation-deposit payloads. Deposit requests must reference a thread accessible to the current account. Personal knowledge-base list/search views are scoped to the current DeerFlow account even for admins; frontend LLMWiki caches include the live account id, and admin write power does not make other users' private KBs appear as owned. Only `system_role === "admin"` sees the WeKnora connection-config tab. Knowledge cards navigate to a dedicated WeKnora-style detail route. Wiki-enabled bases render and default to `Wiki -> documents -> graph`; standard bases omit Wiki and render `documents -> graph`. The detail page supports sequential multi-file upload with an aggregate result for writable ordinary bases, hides it for public read-only/system-deposit bases, and uses a generic loading label while the embed session starts. It also supports URL and published manual-Markdown imports, browses real Wiki pages/content, proxies authenticated original-file previews, pages through `/chunks/{knowledge_id}`, and renders the Wiki page graph from `/knowledgebase/{kb_id}/wiki/graph`; document details use a right sheet rather than the removed KB modal. The credential-free proxy uses one WeKnora service identity, so per-user separation is enforced by DeerFlow mappings while all remote KBs physically share that identity's WeKnora workspace. WeKnora iframe detail pages first call `/api/llmwiki/knowledge-bases/{id}/weknora/session` through the normal DeerFlow API auth path, then load `/platform/knowledge-bases/{weknora_id}` with a signed `deerflow_weknora_embed` cookie for `/platform`, `/assets`, `/locales`, root assets, and API calls. Injected fetch/XHR rewriting sends all iframe `/api/v1/*`, including colliding `/api/v1/auth/*`, through `/api/llmwiki/weknora-embed/api/v1/*`; Gateway auth/CSRF bypass only applies to those signed iframe proxy requests, the proxy still revalidates the mapped KB, and cookie security honors `X-Forwarded-Proto` for HTTPS reverse proxies. `llmwiki_search` is conditionally registered in `tools.py`, is read-only, and rechecks the current DeerFlow user, agent `llmwiki_knowledge_base_ids`, and thread-context selection on every call. Source previews prefer the cleaned Wiki/LLMWiki page text when a retrieved chunk maps to a Wiki page, with raw split chunks as the fallback. Agent bindings use an inline WeKnora-style searchable/grouped multi-select in the standard agent create/detail UI; new chats use the same interaction pattern as a compact composer dropdown, and selections are written to thread metadata/context and restored on reopen. The DeerFlow frontend is native React and the WeKnora frontend is not modified or embedded. Full deployment and behavior notes: `docs/LLMWIKI_WEKNORA_INTEGRATION_ZH.md`. Tests: `tests/test_llmwiki_weknora.py`.
|
||||
|
||||
The embedded detail proxy hides WeKnora's outer `.main > .aside_box` application sidebar and expands the route outlet, but deliberately keeps the Wiki page's internal directory/index sidebar. Do not add an HTML `sandbox` attribute to this trusted, permission-filtered detail iframe: a sandboxed ancestor causes Chromium's built-in Blob PDF viewer to render the browser-blocked page. The Gateway follows upstream redirects only for `/knowledge/{id}/preview` and `/knowledge-bases/{id}/files`, keeping PDF bytes and protected Wiki images inside the signed proxy instead of sending the browser to WeKnora object storage.
|
||||
@@ -1565,10 +1567,12 @@ detail. Never call `/knowledge-search` from internal search, RAG, or the
|
||||
`deerflow.integrations.weknora.local_index` and
|
||||
`deerflow.persistence.llmwiki_index`; preserve the harness → app import
|
||||
firewall. Gateway composition injects the store/search/sync services through
|
||||
`local_index.runtime`. `enabled` turns on local vector search and manual
|
||||
vectorization; `auto_sync` is the separate opt-in for the lease-protected
|
||||
scheduler. The scheduler sleeps for one full configured interval before its
|
||||
first scan, so application startup must never trigger immediate vectorization.
|
||||
`local_index.runtime`. `enabled` turns on local vector search and `auto_sync`
|
||||
defaults to the lease-protected background scheduler. The scheduler performs
|
||||
an immediate incremental scan at startup, then rescans every configured
|
||||
interval so new WeKnora libraries and asynchronously generated Wiki pages are
|
||||
picked up without a manual vectorization click. Unchanged content must reuse
|
||||
existing vectors.
|
||||
|
||||
Vectors are normalized little-endian float32 BLOBs. A page mirror and all of
|
||||
its vectors are replaced in one transaction only after strict batched
|
||||
@@ -1628,8 +1632,10 @@ same DOM value, so a fetched-and-revoked object URL is not acceptable.
|
||||
Third-party pages embed published Wiki content through the chrome-free route
|
||||
`/#/embed/knowledge/{mapping_id}` (optional exact-page `wikiSlug`). It defaults
|
||||
to a recognized Wiki index page (title `索引`/`index`, page type `index`, or an
|
||||
`index`/`home`/`readme` slug), promotes that page to the first public-directory
|
||||
entry, then falls back to an overview/root page when no index exists. It retains
|
||||
`index`/`home`/`readme` slug) and promotes that page to the first public-directory
|
||||
entry. When upstream pages exist without an explicit index shell, the frontend
|
||||
injects a virtual `__index__` page and builds its catalogue from the page list;
|
||||
an all-empty dynamic index response must also fall back to that page list. It retains
|
||||
directory navigation and supports the shared public-embed `theme` / `bg` /
|
||||
`text` appearance inputs. WeKnora dynamically renders index page-type counts,
|
||||
nested category headings, page summaries, and `data-slug` links even though the
|
||||
@@ -1645,9 +1651,14 @@ the loaded page title, slug, or alias and rewrites them to exact embedded
|
||||
`wikiSlug` routes. Preserve WeKnora's `wiki-content-link` interaction: primary
|
||||
brand color, medium weight, dashed underline becoming solid on hover, focus
|
||||
outline, and in-reader SPA navigation rather than a page reload. `GET
|
||||
/api/public/llmwiki/knowledge-bases` is the
|
||||
anonymous discovery endpoint and returns only each published ordinary mapping's
|
||||
`id` and `name` (plus the collection total). All anonymous reader calls use that
|
||||
/api/public/llmwiki/knowledge-bases` is the anonymous discovery endpoint. It
|
||||
validates published ordinary mappings against WeKnora, drops stale mappings,
|
||||
and returns safe comprehensive metadata from the WeKnora list/detail/page APIs:
|
||||
document/chunk/processing/share counts, non-secret configuration and timestamps,
|
||||
Wiki page type/status counts and explicit/generated index entry, plus aggregate
|
||||
local-vector state/counts/timestamps. Never expose remote WeKnora ids,
|
||||
tenant/creator identity, storage credentials, API credentials, raw vectors, or
|
||||
per-page indexing errors. All anonymous reader calls use that
|
||||
public namespace and the `/knowledge-bases/{id}` Wiki-page children. Those
|
||||
endpoints must resolve with a synthetic anonymous user and `is_admin=False`,
|
||||
explicitly require `publication_status=published`, and hard-deny the system
|
||||
@@ -1726,3 +1737,105 @@ before MarkItDown extraction, avoiding concurrent profile locks. The image
|
||||
build and `OFFLINE_INSTALL.sh --office-check` both run
|
||||
`scripts/office_runtime_selfcheck.py`; that smoke test creates and re-parses
|
||||
minimal DOCX/XLSX/PPTX files and does not need the application database.
|
||||
|
||||
## Skill/assistant knowledge invariants
|
||||
|
||||
Skill distillation is implemented in `deerflow.skill_knowledge` and
|
||||
`deerflow.persistence.skill_knowledge`; native assistant bases are in
|
||||
`deerflow.assistant_knowledge` and
|
||||
`deerflow.persistence.assistant_knowledge`. Gateway composition, remote adapters,
|
||||
the lease dispatcher, and APIs remain in `app.gateway` so the harness-to-app
|
||||
import firewall is preserved.
|
||||
|
||||
Never execute a skill or read/upload/summarize its code and script files.
|
||||
Knowledge fingerprints exclude those files and Markdown code fences. All
|
||||
resyncs must stage and validate the knowledge digest before activation; the old
|
||||
snapshot remains authoritative on failure. A binding may have only one active
|
||||
item (`active_dedupe_key` plus atomic lease claims). Skill distillation targets
|
||||
only writable WeKnora Wiki knowledge bases in this product version; reject direct
|
||||
assistant-base targets, non-Wiki/FAQ/conversation/readonly targets, and direct
|
||||
Neo4j writes. The create/list/resync/retry/detach flow is available to ordinary
|
||||
users for their own visible skills, writable knowledge bases, jobs and bindings;
|
||||
administrator-only auto-sync/review APIs remain privileged. Detach/delete operates on source contributions, not canonical rows,
|
||||
and shared entities/relations are retired only when their final contribution is
|
||||
gone.
|
||||
|
||||
Assistant knowledge has one global full-library base (`知识梳理总库`) and any
|
||||
number of admin-created custom bases. Ordinary users have read/search access and
|
||||
administrators own all mutations. Assistant bases only accept WeKnora export
|
||||
packages or imports triggered from a WeKnora mapping. Do not add direct skill
|
||||
imports or arbitrary local-file imports to assistant knowledge. Importing into a
|
||||
custom assistant base must also mirror the package into the global base for
|
||||
cross-source entity alignment. Admins may rename any assistant base and delete
|
||||
custom bases. Deletion removes the custom base's Wiki pages, vectors, entities,
|
||||
relations, import audit rows, and package files; the global base is never
|
||||
deletable because it owns full-library aggregation.
|
||||
|
||||
### Wiki package and local encoder invariants (2026-09)
|
||||
|
||||
- `knowledge_transfer.sync_to_global` is called after successful explicit local
|
||||
Wiki vectorization; it skips conversation/system mappings. Do not enable a new
|
||||
watcher or automatic remote-write scheduler as a side effect.
|
||||
- `wiki_packages` exposes file-backed assistant exports and restores into empty
|
||||
ordinary Wiki libraries. Validate vectors before any remote mutation; recreate
|
||||
folder IDs from category paths using provider APIs (a raw category_path is
|
||||
discarded by WeKnora without a matching folder_id). Preserve page metadata and
|
||||
rebuild backlinks after restoring all pages.
|
||||
- `PackageArchive` is a replayable, disk-backed JSONL staging index. Always close
|
||||
it in finally. Runtime imports must not use the legacy in-memory read_package.
|
||||
- Preserve normalized float32 base64 bytes exactly. Same dimension alone is not
|
||||
model compatibility. Assistant vector retrieval is fingerprint-gated cosine
|
||||
search, and any keyword fallback must be labeled as such.
|
||||
Preserve a valid zero section_index. Export/search vectors belonging to each
|
||||
current source snapshot (`source.last_import_job_id`), not merely the page's
|
||||
current revision: one aligned page may intentionally contain current vectors
|
||||
from several sources.
|
||||
- The `local` embedding provider uses FastEmbed with official BAAI/bge-m3 ONNX,
|
||||
1024 dimensions, bounded CPU batches and a per-client async lock. Run inference
|
||||
off the event loop. Existing page vectors are reusable; zero-vector nonempty
|
||||
pages need backfill, not a forced rebuild of every page.
|
||||
FastEmbed is a standard backend dependency. Encoder configuration, directory
|
||||
upload and warmup live only in the administrator global settings dialog.
|
||||
A runtime `system_settings.json` path override wins over config.yaml; uploaded
|
||||
files are streamed into `.deer-flow/models/bge-m3`, never the source tree.
|
||||
Encoder warmup is explicitly started from the settings UI/API and must never block application startup
|
||||
or turn a model-load failure into a DeerFlow service failure. Local provider
|
||||
is offline-only: require local_model_path or DEERFLOW_BGE_M3_MODEL_PATH and
|
||||
never let the application download model weights. Keep
|
||||
`local_process_isolation=true` by default: inference runs in the bundled JSONL
|
||||
worker subprocess, and shutdown must terminate that worker without waiting on
|
||||
model teardown.
|
||||
- Every successful mapping snapshot is mirrored to the global assistant base,
|
||||
including empty snapshots so deleted source pages are retired. Re-import is
|
||||
source-replacement semantics: absent Wiki/chunk/entity/relation pages must not
|
||||
remain visible merely because they existed in an older source snapshot.
|
||||
- Provider graph nodes bind to their original wiki_slug. They must not create
|
||||
duplicate synthetic entity/relation articles. Store original Wiki metadata
|
||||
separately from assistant workflow status. Count Wiki pages separately from
|
||||
raw document chunks.
|
||||
- The global base aligns entities by normalized display name and aliases across
|
||||
source records, and aligns content Wiki pages by normalized title. Do not use
|
||||
source slug as the global canonical identity. Merge unique knowledge blocks
|
||||
while retaining source contributions/origins. Uploaded packages must rebind
|
||||
the temporary upload source to the stable WeKnora knowledge-base id in the
|
||||
package, so re-upload is source-replacement rather than append. Retrieval
|
||||
candidates exclude summary/index/directory pages; keyword matching, vector
|
||||
snippets, citations and prompts must use knowledge body/sections, never page
|
||||
summary as a substitute.
|
||||
- Migration `20260903_01` adds nullable page metadata and merges the pre-existing
|
||||
knowledge/workflow Alembic heads. Regression coverage is in
|
||||
`tests/test_wiki_package_roundtrip.py`, `test_llmwiki_wiki_sync.py` and
|
||||
`test_assistant_knowledge_repository.py`.
|
||||
|
||||
## openGauss / GaussDB persistence invariant
|
||||
|
||||
`database.backend=gauss` is an application-ORM backend, parallel to `mysql`.
|
||||
It normalizes `database.gauss_url` to `opengauss+asyncpg://` and requires the
|
||||
`gauss` optional dependency (`asyncpg` plus `opengauss-sqlalchemy`). Never route
|
||||
a Gauss URL through SQLAlchemy's generic PostgreSQL dialect: the wire protocol
|
||||
may connect, but dialect initialization and schema DDL can fail against the
|
||||
Gauss server/version behavior. Keep the LangGraph checkpointer on the separately
|
||||
configured SQLite backend; the PostgreSQL checkpointer schema is not part of the
|
||||
Gauss compatibility contract. Treat both `postgresql` and `opengauss` dialect
|
||||
names as timezone-aware PostgreSQL-family engines in portable types and
|
||||
dialect-specific session settings.
|
||||
Reference in new issue
Block a user