Merge remote-tracking branch 'origin/master' into ji
# Conflicts: # .gitignore # frontend-web/src/components/workspace/artifacts/context.tsx # frontend-web/src/components/workspace/messages/message-group.tsx # offline-backend-20260512/backend/CLAUDE.md # offline-backend-20260512/backend/README.md # offline-backend-20260512/backend/app/gateway/app.py # offline-backend-20260512/backend/app/gateway/deps.py # offline-backend-20260512/backend/app/gateway/routers/llmwiki.py
This commit is contained in:
commit
e818c24769
262 files changed
+29527
-5236
No files matched your search
@@ -41,29 +41,10 @@ DeerFlow is a LangGraph-based AI super agent with sandbox execution, persistent
|
||||
|
||||
Health probes are always unauthenticated, even when API authentication is enabled: `GET|HEAD /health`, `/api/health`, and `/api/langgraph/health` all return the Gateway health status without requiring a cookie or bearer token.
|
||||
|
||||
Browser authentication is page-context aware. Normal application tabs share the account stored in `localStorage`; `embed=1` and `isolate=1` pages intentionally keep their own account in per-tab `sessionStorage`. Frontend API calls send that selected account as an explicit `Authorization: Bearer` token, and both Gateway and LangGraph authentication give the header priority over the origin-wide session cookie. This prevents a login in another tab or iframe from making the current page query threads as the wrong account. Reissuing a token for the same normalized username is not treated as an account switch; a real switch clears account-scoped frontend state, opens a fresh chat when necessary, and does not reload the whole browser page. Bare username login is domain-agnostic and deterministically reuses the oldest matching account if legacy synthetic email domains coexist; an explicitly supplied email still selects that exact address.
|
||||
|
||||
---
|
||||
|
||||
## Core Components
|
||||
|
||||
### Administrator fixed questions
|
||||
|
||||
Administrators can manage exact-match question/answer pairs from the frontend
|
||||
「问题配置」 page. Each pair has its own enabled switch and configurable fixed-answer
|
||||
streaming speed (`1–1000 token/s`, default `100 token/s`). For an enabled pair,
|
||||
a plain-text external chat turn whose text exactly equals the configured
|
||||
question is answered by a local streaming graph and never sent to a model
|
||||
provider. The normal LangGraph SSE protocol and checkpoint persistence are
|
||||
preserved, so the fixed response appears in live chat and conversation history.
|
||||
Multimodal/file turns, AI Writing/Canvas flows, and internal orchestration child
|
||||
runs bypass this lookup and continue through their normal model pipeline.
|
||||
|
||||
The admin-only configuration API is `GET|PUT /api/fixed-questions`; `PUT`
|
||||
atomically replaces the full set. Rows are stored in `fixed_questions` (Alembic
|
||||
revisions `20260821_01` and `20260821_02`) and are looked up with a SHA-256 index plus an exact
|
||||
original-text comparison.
|
||||
|
||||
### Lead Agent
|
||||
|
||||
The single LangGraph agent (`lead_agent`) is the runtime entry point, created via `make_lead_agent(config)`. It combines:
|
||||
@@ -100,7 +81,9 @@ Per-thread isolated execution with virtual path translation:
|
||||
- **Skills path**: `/mnt/skills` → `deer-flow/skills/` directory
|
||||
- **Skills loading**: Recursively discovers nested `SKILL.md` files under `skills/{public,custom}` and preserves nested container paths
|
||||
- **Conditional WeKnora LLMWiki**: a non-empty `llmwiki.weknora.api_base_url` enables the DeerFlow-native WeKnora BFF and UI; an empty address preserves the existing LLMWiki page. WeKnora remains independently deployed and owns its database, Redis, storage, parsers and models. DeerFlow keeps only the API/admin addresses, user/publication mappings, agent knowledge-base bindings and per-conversation selections; it stores and sends no WeKnora credential, so the endpoint must be restricted to a trusted network and allow DeerFlow traffic. Owners publish directly to the public LLMWiki without an approval queue for ordinary bases; the system-owned conversation-deposit base is always published, shared by all users, excluded from My Knowledge, immutable through user/admin detail operations, and recognized by both its reserved name and `system` owner. An existing same-name remote mapping is adopted in place as the system deposit so its mapping id and document links stay valid. Its user-facing knowledge-base description and generated document metadata use the `cmzs` brand; list/detail reads synchronize stale remote descriptions, while conversation-deposit search results, source previews and embedded-detail JSON rewrite legacy brand text to `cmzs`. Deposit requests must reference a thread accessible to the current account. Personal knowledge-base lists are scoped to the current DeerFlow account even for admins, while frontend LLMWiki query caches also include the active account id so switching accounts cannot reuse another account's list/detail data. Admins remain the only role that sees the WeKnora connection-config tab. Knowledge cards open a dedicated WeKnora-style detail route: Wiki-enabled bases default to Wiki, then documents and graph; standard bases hide Wiki and default to documents. Writable ordinary knowledge-base details expose a native multi-file upload action that submits files sequentially and reports aggregate success/failure; read-only public bases and the system deposit base do not. It supports file/URL/manual-Markdown imports, Wiki page browsing, document preview, real parsed chunks and Wiki graph nodes/edges. New chats use a compact WeKnora-style searchable/grouped multi-select inside the composer, while agent create/edit pages bind spaces through an inline panel. A single proxy service identity means DeerFlow users are isolated logically while their WeKnora entities share that identity's workspace. The read-only `llmwiki_search` tool reauthorizes the current DeerFlow user on every call and renders credential-free source previews; source previews prefer cleaned Wiki/LLMWiki text when a retrieved chunk maps to a Wiki page, with raw chunks as the fallback. The detail page first creates a signed iframe session through the normal DeerFlow API auth path, then points the iframe directly at the proxied WeKnora `/platform/...` route using the signed `deerflow_weknora_embed` cookie; global auth/CSRF bypass is limited to those signed iframe proxy requests, and HTTPS cookie attributes honor `X-Forwarded-Proto` behind reverse proxies. The iframe proxy keeps WeKnora's root-relative `/platform`, `/assets`, `/locales`, and `/api/v1/*` paths, but is registered last through `llmwiki.proxy_router` so it cannot intercept first-party Gateway APIs such as `POST /api/v1/auth/login/username`. See [docs/LLMWIKI_WEKNORA_INTEGRATION_ZH.md](docs/LLMWIKI_WEKNORA_INTEGRATION_ZH.md).
|
||||
- **Enterprise Research workbench (temporarily disabled)**: the independent `/api/enterprise-research` router, its report executor, and its ORM table registration are intentionally inactive. The implementation remains in the repository, but startup does not create or alter `enterprise_research_tasks` or `enterprise_research_report_jobs`; this keeps the active Deep Research workflow independent of the paused workbench and avoids MySQL schema errors.
|
||||
- **Embedded WeKnora detail chrome**: the proxy hides WeKnora's application sidebar (`.main > .aside_box`) and expands the detail outlet to the full iframe width, while preserving the Wiki page's own index/navigation panel.
|
||||
- **Embedded WeKnora knowledge-base switching**: the original breadcrumb picker works inside the iframe, but lists only the signed-in user's own and published knowledge bases. Every switch is reauthorized through DeerFlow and receives a fresh short-lived iframe session, so it never exposes the upstream service-admin knowledge-base list.
|
||||
- **Prefixed WeKnora deployment**: the frontend may be published under a separate prefix such as `/magentweb`, while embedded WeKnora detail traffic is served by the Gateway under `/deerflow`. The Gateway rewrites HTML root assets, WeKnora API URLs, redirects, and Vite dynamic dependency maps so page assets stay under `/deerflow`; the prefixed proxy routes must be registered before the unprefixed fallback. The signed embed cookie covers `/deerflow`, and auth/CSRF recognition includes all supported static directories.
|
||||
- **File-write safety**: `str_replace` serializes read-modify-write per `(sandbox.id, path)` so isolated sandboxes keep concurrency even when virtual paths match
|
||||
- **Tools**: `bash`, `ls`, `read_file`, `write_file`, `str_replace` (`bash` is disabled by default when using `LocalSandboxProvider`; use `AioSandboxProvider` for isolated shell access)
|
||||
@@ -170,7 +153,8 @@ FastAPI application providing REST endpoints for frontend integration:
|
||||
| `GET /api/threads/{id}/uploads/list` | List uploaded files |
|
||||
| `DELETE /api/threads/{id}` | Delete DeerFlow-managed local thread data after LangGraph thread deletion; unexpected failures are logged server-side and return a generic 500 detail |
|
||||
| `GET /api/threads/{id}/artifacts/{path}` | Serve generated artifacts |
|
||||
| `GET/POST /api/deep-research/*` | Durable Deep Research sessions and jobs. `quick`/`basic` run the compact report flow; `detailed` plans subtopics and writes independent sections; `deep` recursively investigates bounded follow-up branches. All modes reuse DeerFlow's configured material provider, LLM factory, hidden-thread artifacts, and SSE event log. Optional report illustrations use a separately configured OpenAI-compatible image endpoint and are served only through the owning session's authenticated artifact route. |
|
||||
| `GET/POST /api/deep-research/*` | Durable Deep Research sessions and jobs. `quick`/`basic` run the compact report flow; `detailed` plans subtopics and writes independent sections; `deep` recursively investigates bounded follow-up branches. All modes reuse DeerFlow's configured material provider, LLM factory, hidden-thread artifacts, and SSE event log. Deep Research session/job/event/source/message writes serialise their rows in the writer transaction, rather than doing a post-commit ORM refresh or readback; this prevents MySQL read/write-splitting replica lag from failing “start writing”, dispatcher claim, report persistence, or follow-up history. Whole-document artifact/report rewrites likewise create their reversible version snapshots from writer-local ORM data and never refresh after commit, so replica lag cannot restore an otherwise successful rewrite. A report is written to the session and confirmed before its Job becomes `completed`; an unconfirmed write is retried and never exposed as a false completion. Retrying an already-created queued job re-nudges the dispatcher, recovering an interrupted HTTP response without creating a second report. If progress-event persistence, durable replay, or live fan-out is briefly unavailable, the worker/stream logs the degradation and continues or reconnects rather than failing the whole job. Whole-report rewrite accepts an optional `reportOutline` Markdown template; the durable job persists it and applies it as the chapter structure constraint. Optional report illustrations use a separately configured OpenAI-compatible image endpoint and are served only through the owning session's authenticated artifact route. |
|
||||
| `/api/enterprise-research/*` | Temporarily disabled. Its router and persistence initialization are not registered, so these endpoints return 404 and no enterprise-research tables are created or migrated during startup. |
|
||||
| `POST /api/ai-writing/sample/extract` | Extract text from Word/PDF/Markdown/TXT samples for AI-writing imitation mode |
|
||||
| `POST /cop/saveSuperiorTask` | TaskCOP compatibility API (DeerFlow Bearer/session authentication required): create a situation-overview task with a sequential four-digit id (`0001`, `0002`, ...) and return the legacy `{state, msg, data}` envelope |
|
||||
| `POST /api/taskcop/import/tasks` | TaskCOP compatibility API (authenticated): import tasks from a JSON array or `tasks`/`records` envelope; every row's supplied `id`/`taskId` is authoritative and a duplicate id completely replaces the stored task |
|
||||
@@ -199,6 +183,11 @@ the right-side `report.md` sandbox can write smoothly, while bounded,
|
||||
persisted `report_chunk` events remain reconnect checkpoints. The durable log
|
||||
is still authoritative: a browser reconnect, slow consumer, or another worker
|
||||
falls back to `?after=<seq>` replay rather than starting a second model call.
|
||||
Reasoning exposed by providers either through structured reasoning fields or
|
||||
inline `<think>...</think>` text is separated from report deltas, checkpoints,
|
||||
and the persisted Markdown projection. It may be forwarded as an ephemeral
|
||||
live `report_thinking` frame for the message-list thought trace, but is never
|
||||
persisted or written to the sandbox; only answer Markdown reaches the sandbox.
|
||||
The short non-streaming control-plane calls used to plan queries, sections, and
|
||||
reviews are independently capped at 45 seconds (maximum 120 seconds). If query
|
||||
or section planning times out, the runner emits a visible recoverable warning
|
||||
@@ -245,17 +234,43 @@ per-user concurrency slot.
|
||||
Saved `draft` sessions do not consume a concurrency slot; the two-job limit
|
||||
only applies to sessions with a `running` or `awaiting_input` durable job.
|
||||
|
||||
Whole-report rewrite builds its requirement plan locally from the selected
|
||||
style and instruction, publishes that plan immediately, and then opens only
|
||||
one model stream for the actual Markdown writer. This avoids making users wait
|
||||
for a separate model-thinking/planning pass before sandbox output begins.
|
||||
`POST /api/deep-research/sessions/{id}/report-variants` is the structural
|
||||
regeneration boundary: it copies the completed report and selected evidence
|
||||
into a new session (including both saved `[来源:id]` and legacy streamed
|
||||
`[[source:id]]` citation markers remapped to the cloned ids). The following
|
||||
durable writer is explicitly queued with `operation=generate`: it does not send
|
||||
the inherited report to the model as text to edit, and instead writes a fresh
|
||||
report from the complete selected evidence set plus the newly confirmed
|
||||
outline. Model reasoning remains a live SSE event before the first Markdown
|
||||
token. New output uses display-ready `[来源:id]` citations; validation repairs
|
||||
only unique near-complete `drs_src_*` ids (for example a provider-truncated
|
||||
suffix) and still rejects genuinely unknown evidence ids.
|
||||
The report-only model stream requests an 8192-token completion budget so a
|
||||
normal multi-section report and its reference list do not stop at the common
|
||||
4096-token provider default; malformed trailing citation syntax is rejected
|
||||
rather than committed as a partial document.
|
||||
The original report therefore remains independently available in history even
|
||||
if the new report is cancelled or fails.
|
||||
|
||||
#### Optional report illustrations
|
||||
|
||||
DeerFlow's stock configuration includes image recognition but no image-generation
|
||||
provider. Deep Research therefore keeps illustrations disabled until an operator
|
||||
fills `config.yaml -> deep_research.image_generation` with an
|
||||
OpenAI-compatible image endpoint, API key, and model. When configured, the
|
||||
capabilities endpoint enables the page's “生成报告配图” switch. A selected job
|
||||
requests at most four base64 image results, writes them as protected
|
||||
`research-image-*.png|jpg|webp` artifacts, injects `deep-research://` links into
|
||||
the Markdown report, and emits image SSE events. A failed/unconfigured image
|
||||
provider only emits a recoverable warning; the text report still completes.
|
||||
image endpoint, API key, and model. `provider: openai_compatible` uses the
|
||||
standard `/images/generations` protocol and requires `b64_json`; `provider:
|
||||
dashscope_native` supports Qwen Image 3.0 and Wan 2.7 on DashScope / Token Plan
|
||||
with the native multimodal-generation protocol. Native provider image URLs are
|
||||
downloaded immediately, so both providers produce protected local artifacts.
|
||||
When configured, the capabilities endpoint enables the page's “生成报告配图” switch.
|
||||
A selected job writes at most four `research-image-*.png|jpg|webp` artifacts,
|
||||
injects `deep-research://` links into the Markdown report, and emits image SSE
|
||||
events. A failed/unconfigured image provider only emits a recoverable warning;
|
||||
the text report still completes.
|
||||
|
||||
Completed reports also expose `GET /api/deep-research/sessions/{id}/messages`
|
||||
and `POST /api/deep-research/sessions/{id}/chat`. Follow-up answers are grounded
|
||||
@@ -632,6 +647,12 @@ the rest of the visual tokens; the optional `overrides` object only exposes
|
||||
must use the `custom-theme-` prefix. Disabled or removed custom themes cannot
|
||||
remain the system default and automatically fall back to `light`.
|
||||
|
||||
Compact top-bar shortcuts use the same appearance endpoint. In addition to
|
||||
login and theme query parameters, each shortcut may include ordered keyless
|
||||
`path` parameters. The frontend resolves each selected login field and appends
|
||||
it as an encoded path segment (including inside a `#/…` hash route); password
|
||||
remains an ordinary login query parameter with the key `password`.
|
||||
|
||||
---
|
||||
|
||||
## Technology Stack
|
||||
@@ -774,12 +795,12 @@ part of a Wiki reference card opens one right-side drawer (rather than a second
|
||||
detail dialog), renders the complete page as Markdown, and reads the remote
|
||||
WeKnora Wiki page directly when the local index is disabled. Even if a Wiki
|
||||
citation carries a legacy document id, the drawer does not mix in the uploaded
|
||||
original-file preview. A single page can
|
||||
be downloaded as Markdown or Word from the drawer. **Open in Wiki** navigates
|
||||
to the exact DeerFlow knowledge-base mapping and Wiki slug. Wiki search-result
|
||||
cards use the same drawer interaction. Exact Wiki links now open a lightweight
|
||||
native reader at `knowledge/{mapping_id}/wiki/view?wikiSlug=...`; it fetches
|
||||
only the selected page and never creates a WeKnora iframe session. The former
|
||||
original-file preview. A single page can be downloaded as Markdown or Word from
|
||||
the drawer. **Open in Wiki** navigates to the exact DeerFlow knowledge-base
|
||||
mapping and Wiki slug. Wiki search-result cards use the same drawer interaction.
|
||||
Exact Wiki links now open a lightweight native reader at
|
||||
`knowledge/{mapping_id}/wiki/view?wikiSlug=...`; it fetches only the selected
|
||||
page and never creates a WeKnora iframe session. The former
|
||||
`knowledge/{mapping_id}?tab=wiki&wikiSlug=...` links redirect to the reader
|
||||
before waiting for the LLMWiki runtime check. Drawer navigation stays in the
|
||||
same SPA and passes the already-loaded page as route state for immediate paint.
|
||||
@@ -802,3 +823,27 @@ expose mappings that are both internally published and explicitly marked
|
||||
`external_search_enabled`; the system `对话沉淀` mapping is always denied.
|
||||
See `docs/LLMWIKI_DEERFLOW_LOCAL_VECTOR_INDEX_ZH.md` for configuration,
|
||||
security boundaries, rollout, and API contracts.
|
||||
|
||||
## Document rewrite fallback
|
||||
|
||||
Whole-document rewrite commits the artifact/report before it attempts to store
|
||||
the optional reversible before-image. If that version-snapshot write fails, the
|
||||
rewritten content remains available and Gateway logs the exception; it never
|
||||
restores the original solely because undo history is unavailable. The durable
|
||||
result marks `versionSnapshotFailed`, allowing the sandbox to enable its normal
|
||||
manual-save action without showing the user an error notification.
|
||||
|
||||
The optional rewrite-history hydration endpoints scope results to the
|
||||
authenticated job owner. They deliberately return an empty list rather than
|
||||
performing a second thread-owner check, so a legacy guest/SSO display alias
|
||||
cannot make an otherwise usable sandbox report fail with a history-only 404.
|
||||
|
||||
Artifact and rewrite path resolution uses the persisted thread creator's
|
||||
directory when metadata exists, after the route's normal ownership check. This
|
||||
prevents a missing request context from incorrectly looking under
|
||||
`users/default` and returning 404 for an existing sandbox file.
|
||||
|
||||
The final rewrite commit uses a deliberately short sibling temporary filename,
|
||||
which keeps long Windows artifact paths below the path-length limit. It also
|
||||
recreates a missing `outputs` parent before retrying, so transient
|
||||
sandbox-directory recreation cannot discard a completed rewrite.
|
||||
Reference in new issue
Block a user