Merge remote-tracking branch 'origin/master' into ji

# Conflicts:
#	.gitignore
#	frontend-web/src/components/workspace/artifacts/context.tsx
#	frontend-web/src/components/workspace/messages/message-group.tsx
#	offline-backend-20260512/backend/CLAUDE.md
#	offline-backend-20260512/backend/README.md
#	offline-backend-20260512/backend/app/gateway/app.py
#	offline-backend-20260512/backend/app/gateway/deps.py
#	offline-backend-20260512/backend/app/gateway/routers/llmwiki.py
This commit is contained in:
吉鑫升 committed 2026-08-24 14:21:19 +08:00
commit e818c24769
262 files changed
+29527 -5236

No files matched your search

+77 -32
View File
@@ -41,29 +41,10 @@ DeerFlow is a LangGraph-based AI super agent with sandbox execution, persistent
Health probes are always unauthenticated, even when API authentication is enabled: `GET|HEAD /health`, `/api/health`, and `/api/langgraph/health` all return the Gateway health status without requiring a cookie or bearer token.
Browser authentication is page-context aware. Normal application tabs share the account stored in `localStorage`; `embed=1` and `isolate=1` pages intentionally keep their own account in per-tab `sessionStorage`. Frontend API calls send that selected account as an explicit `Authorization: Bearer` token, and both Gateway and LangGraph authentication give the header priority over the origin-wide session cookie. This prevents a login in another tab or iframe from making the current page query threads as the wrong account. Reissuing a token for the same normalized username is not treated as an account switch; a real switch clears account-scoped frontend state, opens a fresh chat when necessary, and does not reload the whole browser page. Bare username login is domain-agnostic and deterministically reuses the oldest matching account if legacy synthetic email domains coexist; an explicitly supplied email still selects that exact address.
---
## Core Components
### Administrator fixed questions
Administrators can manage exact-match question/answer pairs from the frontend
「问题配置」 page. Each pair has its own enabled switch and configurable fixed-answer
streaming speed (`1–1000 token/s`, default `100 token/s`). For an enabled pair,
a plain-text external chat turn whose text exactly equals the configured
question is answered by a local streaming graph and never sent to a model
provider. The normal LangGraph SSE protocol and checkpoint persistence are
preserved, so the fixed response appears in live chat and conversation history.
Multimodal/file turns, AI Writing/Canvas flows, and internal orchestration child
runs bypass this lookup and continue through their normal model pipeline.
The admin-only configuration API is `GET|PUT /api/fixed-questions`; `PUT`
atomically replaces the full set. Rows are stored in `fixed_questions` (Alembic
revisions `20260821_01` and `20260821_02`) and are looked up with a SHA-256 index plus an exact
original-text comparison.
### Lead Agent
The single LangGraph agent (`lead_agent`) is the runtime entry point, created via `make_lead_agent(config)`. It combines:
@@ -100,7 +81,9 @@ Per-thread isolated execution with virtual path translation:
- **Skills path**: `/mnt/skills` → `deer-flow/skills/` directory
- **Skills loading**: Recursively discovers nested `SKILL.md` files under `skills/{public,custom}` and preserves nested container paths
- **Conditional WeKnora LLMWiki**: a non-empty `llmwiki.weknora.api_base_url` enables the DeerFlow-native WeKnora BFF and UI; an empty address preserves the existing LLMWiki page. WeKnora remains independently deployed and owns its database, Redis, storage, parsers and models. DeerFlow keeps only the API/admin addresses, user/publication mappings, agent knowledge-base bindings and per-conversation selections; it stores and sends no WeKnora credential, so the endpoint must be restricted to a trusted network and allow DeerFlow traffic. Owners publish directly to the public LLMWiki without an approval queue for ordinary bases; the system-owned conversation-deposit base is always published, shared by all users, excluded from My Knowledge, immutable through user/admin detail operations, and recognized by both its reserved name and `system` owner. An existing same-name remote mapping is adopted in place as the system deposit so its mapping id and document links stay valid. Its user-facing knowledge-base description and generated document metadata use the `cmzs` brand; list/detail reads synchronize stale remote descriptions, while conversation-deposit search results, source previews and embedded-detail JSON rewrite legacy brand text to `cmzs`. Deposit requests must reference a thread accessible to the current account. Personal knowledge-base lists are scoped to the current DeerFlow account even for admins, while frontend LLMWiki query caches also include the active account id so switching accounts cannot reuse another account's list/detail data. Admins remain the only role that sees the WeKnora connection-config tab. Knowledge cards open a dedicated WeKnora-style detail route: Wiki-enabled bases default to Wiki, then documents and graph; standard bases hide Wiki and default to documents. Writable ordinary knowledge-base details expose a native multi-file upload action that submits files sequentially and reports aggregate success/failure; read-only public bases and the system deposit base do not. It supports file/URL/manual-Markdown imports, Wiki page browsing, document preview, real parsed chunks and Wiki graph nodes/edges. New chats use a compact WeKnora-style searchable/grouped multi-select inside the composer, while agent create/edit pages bind spaces through an inline panel. A single proxy service identity means DeerFlow users are isolated logically while their WeKnora entities share that identity's workspace. The read-only `llmwiki_search` tool reauthorizes the current DeerFlow user on every call and renders credential-free source previews; source previews prefer cleaned Wiki/LLMWiki text when a retrieved chunk maps to a Wiki page, with raw chunks as the fallback. The detail page first creates a signed iframe session through the normal DeerFlow API auth path, then points the iframe directly at the proxied WeKnora `/platform/...` route using the signed `deerflow_weknora_embed` cookie; global auth/CSRF bypass is limited to those signed iframe proxy requests, and HTTPS cookie attributes honor `X-Forwarded-Proto` behind reverse proxies. The iframe proxy keeps WeKnora's root-relative `/platform`, `/assets`, `/locales`, and `/api/v1/*` paths, but is registered last through `llmwiki.proxy_router` so it cannot intercept first-party Gateway APIs such as `POST /api/v1/auth/login/username`. See [docs/LLMWIKI_WEKNORA_INTEGRATION_ZH.md](docs/LLMWIKI_WEKNORA_INTEGRATION_ZH.md).
- **Enterprise Research workbench (temporarily disabled)**: the independent `/api/enterprise-research` router, its report executor, and its ORM table registration are intentionally inactive. The implementation remains in the repository, but startup does not create or alter `enterprise_research_tasks` or `enterprise_research_report_jobs`; this keeps the active Deep Research workflow independent of the paused workbench and avoids MySQL schema errors.
- **Embedded WeKnora detail chrome**: the proxy hides WeKnora's application sidebar (`.main > .aside_box`) and expands the detail outlet to the full iframe width, while preserving the Wiki page's own index/navigation panel.
- **Embedded WeKnora knowledge-base switching**: the original breadcrumb picker works inside the iframe, but lists only the signed-in user's own and published knowledge bases. Every switch is reauthorized through DeerFlow and receives a fresh short-lived iframe session, so it never exposes the upstream service-admin knowledge-base list.
- **Prefixed WeKnora deployment**: the frontend may be published under a separate prefix such as `/magentweb`, while embedded WeKnora detail traffic is served by the Gateway under `/deerflow`. The Gateway rewrites HTML root assets, WeKnora API URLs, redirects, and Vite dynamic dependency maps so page assets stay under `/deerflow`; the prefixed proxy routes must be registered before the unprefixed fallback. The signed embed cookie covers `/deerflow`, and auth/CSRF recognition includes all supported static directories.
- **File-write safety**: `str_replace` serializes read-modify-write per `(sandbox.id, path)` so isolated sandboxes keep concurrency even when virtual paths match
- **Tools**: `bash`, `ls`, `read_file`, `write_file`, `str_replace` (`bash` is disabled by default when using `LocalSandboxProvider`; use `AioSandboxProvider` for isolated shell access)
@@ -170,7 +153,8 @@ FastAPI application providing REST endpoints for frontend integration:
| `GET /api/threads/{id}/uploads/list` | List uploaded files |
| `DELETE /api/threads/{id}` | Delete DeerFlow-managed local thread data after LangGraph thread deletion; unexpected failures are logged server-side and return a generic 500 detail |
| `GET /api/threads/{id}/artifacts/{path}` | Serve generated artifacts |
| `GET/POST /api/deep-research/*` | Durable Deep Research sessions and jobs. `quick`/`basic` run the compact report flow; `detailed` plans subtopics and writes independent sections; `deep` recursively investigates bounded follow-up branches. All modes reuse DeerFlow's configured material provider, LLM factory, hidden-thread artifacts, and SSE event log. Optional report illustrations use a separately configured OpenAI-compatible image endpoint and are served only through the owning session's authenticated artifact route. |
| `GET/POST /api/deep-research/*` | Durable Deep Research sessions and jobs. `quick`/`basic` run the compact report flow; `detailed` plans subtopics and writes independent sections; `deep` recursively investigates bounded follow-up branches. All modes reuse DeerFlow's configured material provider, LLM factory, hidden-thread artifacts, and SSE event log. Deep Research session/job/event/source/message writes serialise their rows in the writer transaction, rather than doing a post-commit ORM refresh or readback; this prevents MySQL read/write-splitting replica lag from failing “start writing”, dispatcher claim, report persistence, or follow-up history. Whole-document artifact/report rewrites likewise create their reversible version snapshots from writer-local ORM data and never refresh after commit, so replica lag cannot restore an otherwise successful rewrite. A report is written to the session and confirmed before its Job becomes `completed`; an unconfirmed write is retried and never exposed as a false completion. Retrying an already-created queued job re-nudges the dispatcher, recovering an interrupted HTTP response without creating a second report. If progress-event persistence, durable replay, or live fan-out is briefly unavailable, the worker/stream logs the degradation and continues or reconnects rather than failing the whole job. Whole-report rewrite accepts an optional `reportOutline` Markdown template; the durable job persists it and applies it as the chapter structure constraint. Optional report illustrations use a separately configured OpenAI-compatible image endpoint and are served only through the owning session's authenticated artifact route. |
| `/api/enterprise-research/*` | Temporarily disabled. Its router and persistence initialization are not registered, so these endpoints return 404 and no enterprise-research tables are created or migrated during startup. |
| `POST /api/ai-writing/sample/extract` | Extract text from Word/PDF/Markdown/TXT samples for AI-writing imitation mode |
| `POST /cop/saveSuperiorTask` | TaskCOP compatibility API (DeerFlow Bearer/session authentication required): create a situation-overview task with a sequential four-digit id (`0001`, `0002`, ...) and return the legacy `{state, msg, data}` envelope |
| `POST /api/taskcop/import/tasks` | TaskCOP compatibility API (authenticated): import tasks from a JSON array or `tasks`/`records` envelope; every row's supplied `id`/`taskId` is authoritative and a duplicate id completely replaces the stored task |
@@ -199,6 +183,11 @@ the right-side `report.md` sandbox can write smoothly, while bounded,
persisted `report_chunk` events remain reconnect checkpoints. The durable log
is still authoritative: a browser reconnect, slow consumer, or another worker
falls back to `?after=<seq>` replay rather than starting a second model call.
Reasoning exposed by providers either through structured reasoning fields or
inline `<think>...</think>` text is separated from report deltas, checkpoints,
and the persisted Markdown projection. It may be forwarded as an ephemeral
live `report_thinking` frame for the message-list thought trace, but is never
persisted or written to the sandbox; only answer Markdown reaches the sandbox.
The short non-streaming control-plane calls used to plan queries, sections, and
reviews are independently capped at 45 seconds (maximum 120 seconds). If query
or section planning times out, the runner emits a visible recoverable warning
@@ -245,17 +234,43 @@ per-user concurrency slot.
Saved `draft` sessions do not consume a concurrency slot; the two-job limit
only applies to sessions with a `running` or `awaiting_input` durable job.
Whole-report rewrite builds its requirement plan locally from the selected
style and instruction, publishes that plan immediately, and then opens only
one model stream for the actual Markdown writer. This avoids making users wait
for a separate model-thinking/planning pass before sandbox output begins.
`POST /api/deep-research/sessions/{id}/report-variants` is the structural
regeneration boundary: it copies the completed report and selected evidence
into a new session (including both saved `[来源:id]` and legacy streamed
`[[source:id]]` citation markers remapped to the cloned ids). The following
durable writer is explicitly queued with `operation=generate`: it does not send
the inherited report to the model as text to edit, and instead writes a fresh
report from the complete selected evidence set plus the newly confirmed
outline. Model reasoning remains a live SSE event before the first Markdown
token. New output uses display-ready `[来源:id]` citations; validation repairs
only unique near-complete `drs_src_*` ids (for example a provider-truncated
suffix) and still rejects genuinely unknown evidence ids.
The report-only model stream requests an 8192-token completion budget so a
normal multi-section report and its reference list do not stop at the common
4096-token provider default; malformed trailing citation syntax is rejected
rather than committed as a partial document.
The original report therefore remains independently available in history even
if the new report is cancelled or fails.
#### Optional report illustrations
DeerFlow's stock configuration includes image recognition but no image-generation
provider. Deep Research therefore keeps illustrations disabled until an operator
fills `config.yaml -> deep_research.image_generation` with an
OpenAI-compatible image endpoint, API key, and model. When configured, the
capabilities endpoint enables the page's “生成报告配图” switch. A selected job
requests at most four base64 image results, writes them as protected
`research-image-*.png|jpg|webp` artifacts, injects `deep-research://` links into
the Markdown report, and emits image SSE events. A failed/unconfigured image
provider only emits a recoverable warning; the text report still completes.
image endpoint, API key, and model. `provider: openai_compatible` uses the
standard `/images/generations` protocol and requires `b64_json`; `provider:
dashscope_native` supports Qwen Image 3.0 and Wan 2.7 on DashScope / Token Plan
with the native multimodal-generation protocol. Native provider image URLs are
downloaded immediately, so both providers produce protected local artifacts.
When configured, the capabilities endpoint enables the page's “生成报告配图” switch.
A selected job writes at most four `research-image-*.png|jpg|webp` artifacts,
injects `deep-research://` links into the Markdown report, and emits image SSE
events. A failed/unconfigured image provider only emits a recoverable warning;
the text report still completes.
Completed reports also expose `GET /api/deep-research/sessions/{id}/messages`
and `POST /api/deep-research/sessions/{id}/chat`. Follow-up answers are grounded
@@ -632,6 +647,12 @@ the rest of the visual tokens; the optional `overrides` object only exposes
must use the `custom-theme-` prefix. Disabled or removed custom themes cannot
remain the system default and automatically fall back to `light`.
Compact top-bar shortcuts use the same appearance endpoint. In addition to
login and theme query parameters, each shortcut may include ordered keyless
`path` parameters. The frontend resolves each selected login field and appends
it as an encoded path segment (including inside a `#/…` hash route); password
remains an ordinary login query parameter with the key `password`.
---
## Technology Stack
@@ -774,12 +795,12 @@ part of a Wiki reference card opens one right-side drawer (rather than a second
detail dialog), renders the complete page as Markdown, and reads the remote
WeKnora Wiki page directly when the local index is disabled. Even if a Wiki
citation carries a legacy document id, the drawer does not mix in the uploaded
original-file preview. A single page can
be downloaded as Markdown or Word from the drawer. **Open in Wiki** navigates
to the exact DeerFlow knowledge-base mapping and Wiki slug. Wiki search-result
cards use the same drawer interaction. Exact Wiki links now open a lightweight
native reader at `knowledge/{mapping_id}/wiki/view?wikiSlug=...`; it fetches
only the selected page and never creates a WeKnora iframe session. The former
original-file preview. A single page can be downloaded as Markdown or Word from
the drawer. **Open in Wiki** navigates to the exact DeerFlow knowledge-base
mapping and Wiki slug. Wiki search-result cards use the same drawer interaction.
Exact Wiki links now open a lightweight native reader at
`knowledge/{mapping_id}/wiki/view?wikiSlug=...`; it fetches only the selected
page and never creates a WeKnora iframe session. The former
`knowledge/{mapping_id}?tab=wiki&wikiSlug=...` links redirect to the reader
before waiting for the LLMWiki runtime check. Drawer navigation stays in the
same SPA and passes the already-loaded page as route state for immediate paint.
@@ -802,3 +823,27 @@ expose mappings that are both internally published and explicitly marked
`external_search_enabled`; the system `对话沉淀` mapping is always denied.
See `docs/LLMWIKI_DEERFLOW_LOCAL_VECTOR_INDEX_ZH.md` for configuration,
security boundaries, rollout, and API contracts.
## Document rewrite fallback
Whole-document rewrite commits the artifact/report before it attempts to store
the optional reversible before-image. If that version-snapshot write fails, the
rewritten content remains available and Gateway logs the exception; it never
restores the original solely because undo history is unavailable. The durable
result marks `versionSnapshotFailed`, allowing the sandbox to enable its normal
manual-save action without showing the user an error notification.
The optional rewrite-history hydration endpoints scope results to the
authenticated job owner. They deliberately return an empty list rather than
performing a second thread-owner check, so a legacy guest/SSO display alias
cannot make an otherwise usable sandbox report fail with a history-only 404.
Artifact and rewrite path resolution uses the persisted thread creator's
directory when metadata exists, after the route's normal ownership check. This
prevents a missing request context from incorrectly looking under
`users/default` and returning 404 for an existing sandbox file.
The final rewrite commit uses a deliberately short sibling temporary filename,
which keeps long Windows artifact paths below the path-length limit. It also
recreates a missing `outputs` parent before retrying, so transient
sandbox-directory recreation cannot discard a completed rewrite.