2947 lines
393 KiB
Markdown
2947 lines
393 KiB
Markdown
# CLAUDE.md
|
||
|
||
## Cold agent construction (2026-10-30)
|
||
|
||
`runtime/runs/worker.py` dispatches opted-in synchronous factories through
|
||
`build_agent_with_timing`, backed by `utils/background_init.py`'s
|
||
`run_background_init` and one shared `FileIOPool(1, 1, "agent-init")`.
|
||
`make_lead_agent._deerflow_background_build = True` opts the built-in Lead Agent
|
||
into background construction; unmarked custom factories retain their caller-loop
|
||
contract. Serial admission prevents concurrent SDK cold imports and keeps waiting
|
||
builds out of authentication's executor. The synchronous timing helper remains
|
||
available. Model execution, checkpoint/store attachment and streaming stay in the
|
||
original run task; no graph is shared or cached across runs/users.
|
||
|
||
Every async Lead Agent entry point uses this builder, including non-streaming
|
||
and streaming open-chat endpoints and Workflow agent nodes. Do not call the
|
||
synchronous factory directly on their event loop: it can block SDK import locks
|
||
or deadlock while another background build awaits MCP discovery on that loop.
|
||
Async tool consumers (AI writing skill agents and `task_tool`) use
|
||
`get_available_tools_async`; deep-research capabilities and job execution also
|
||
construct their material providers through `run_background_init`. Keep the
|
||
common helper in `utils` with a lazy MCP import to avoid runtime/agent/tools
|
||
import cycles. Synchronous client/provider APIs remain available.
|
||
|
||
Copy the caller's ContextVars into construction and restore the successfully built
|
||
context before streaming, including a deferred-tool registry reset to `None`.
|
||
The `mcp_initialization_loop` context manager forwards lazy MCP discovery from the
|
||
worker to the owner event loop; exit it before copying back context. MCP cache
|
||
invalidation and legacy synchronous callers retain their existing semantics.
|
||
Cancellation drains the actual worker before releasing capacity, never starts the
|
||
graph afterwards, and preserves terminal status/SSE cleanup. Check `abort_event`
|
||
after construction as well as honoring task cancellation.
|
||
|
||
Regression: `packages/harness/tests/test_agent_build_responsiveness.py`,
|
||
`test_agent_build_timing.py`, `test_mcp_background_build.py` and the run-finalization
|
||
tests; `tests/test_async_agent_entrypoints.py` and
|
||
`tests/test_async_tool_initialization.py` exercise overlapping HTTP/workflow/tool
|
||
initializers. The HTTP/workflow cases run in subprocesses so an event-loop
|
||
deadlock fails with a bounded timeout. Cold profiling found ~4.7 s in model SDK imports/client setup, compared with
|
||
~14 ms in skill loading; moving only disk reads would not address this stall.
|
||
|
||
Local live cold acceptance (2026-10-30, official MiniMax-M2.7, tools enabled):
|
||
the first build took 4.28 s while a second request received stream metadata in
|
||
402 ms. During the conservative interior of the build window, 43 health/auth
|
||
probes completed successfully; across all three conversations, health/auth
|
||
P95 latency was 28/73 ms, with no event-loop-stall warning. Every answer ended
|
||
normally and all synthetic threads were removed. Initialization/model latency
|
||
still contributes to first-answer time; this change prevents blocking the server
|
||
loop, rather than promising immediate model output.
|
||
A separate live read-only tool check uploaded a random marker, allowed only
|
||
`read_file`, and confirmed the tool result and final answer reproduced that marker
|
||
without placing it in the prompt; the stream ended normally and the thread was removed.
|
||
|
||
Worker compatibility: no HTTP/SSE, task payload, auth or database contract changes.
|
||
Running workers without reload retain old construction behavior; the new factory
|
||
marker is harmless to old executors. `worker_pool/revision.py` hashes only the
|
||
adapter directory and does not identify this harness patch. This change does not
|
||
alter the controller's existing snapshot/ImageID-based drain-and-replace policy
|
||
or `on_disconnect=cancel` behavior: restarting a production controller can still
|
||
rotate idle workers or disconnect proxied streams. Do not infer uninterrupted
|
||
worker operation from these process-local initialization changes. The live local
|
||
acceptance instance has no enabled worker controller and does not touch remote
|
||
workers. Existing worker routing/admission/storage/discovery/diagnostics/launcher
|
||
compatibility regressions: 68 passed.
|
||
|
||
Final combined regression (2026-10-30): all 58 newly added responsiveness,
|
||
cancellation, MCP-entrypoint and provider-lifecycle cases plus the 68 existing
|
||
worker compatibility cases passed together (126 passed, no skips).
|
||
The live recheck uploaded and converted 20 DOCX files while another conversation
|
||
produced 43 visible text increments; all 172 concurrent authenticated probes
|
||
succeeded (P95 105 ms, max 436 ms). A fresh-process cold build took 10.92 s while
|
||
617 heartbeat samples continued (P95 scheduling delay 25 ms, max 146 ms).
|
||
The existing backend and child workers were not restarted for this recheck;
|
||
later entrypoint changes were verified through isolated HTTP/workflow tests.
|
||
The full versioned backend/harness run completed with 3140 passed, 39 failed,
|
||
4 skipped and 1 deselected. Against baseline `47cfed02`, 30 failures also failed
|
||
on HEAD and the other 9 passed in isolated reruns on both versions; no current-only
|
||
stable failure remained. The deselected legacy
|
||
`test_cancelled_history_closes_iterator` hangs on both versions: its fake saver
|
||
implements `alist`, while `limit=1` takes `aget_tuple`, so its unbounded entry
|
||
event wait never completes. This is not a claim that the repository's entire
|
||
test suite is green. Changed Python files introduced no Ruff diagnostics.
|
||
|
||
## Upload responsiveness (2026-10-30)
|
||
|
||
`app/gateway/routers/uploads.py` runs each staged upload transaction in a dedicated
|
||
`FileIOPool(2, 2, "uploads")`, including 256 KiB spool copies, signature validation,
|
||
optional synchronous conversion, sandbox acquisition/sync, commit, rollback and
|
||
cleanup. Resolve the thread owner asynchronously before dispatch and propagate
|
||
contextvars into workers. Waiting batches consume no worker from authentication,
|
||
artifacts or event streaming. Per-event-loop directory locks serialize same-thread
|
||
batches and listing/deletion through rollback/cleanup; keep them held until a
|
||
cancelled worker stops. Listing/deletion use the artifact file pool off the loop.
|
||
Cancellation signals a cooperative event, checks between file chunks and around
|
||
commit/sync, and waits for real completion before multipart handles may close.
|
||
The response contract and upload-policy enforcement remain unchanged.
|
||
`sandbox/sandbox_provider.py` serializes initial singleton construction with
|
||
reset/replacement/shutdown. Concurrent cold uploads must not create competing
|
||
providers with independent sandbox reconciliation and idle-cleanup threads.
|
||
Regression: `packages/harness/tests/test_sandbox_provider_concurrency.py`.
|
||
|
||
`file_conversion.py` also isolates its asynchronous conversion/extraction callers
|
||
in a separate two-thread pool. PDF/Office parsing runs in a lazily initialized,
|
||
two-process pool using `spawn` on Windows and Linux, so Python parser CPU work
|
||
does not contend for the gateway GIL. Pass only file paths and converter choices;
|
||
bound returned text inside the child and write sidecars in the owning parent
|
||
worker. The global parser semaphore bounds sync/async callers together. A broken
|
||
process pool fails the current conversion and is rebuilt on the next call.
|
||
`FileIOPool.run_until_complete` provides non-abandoning cancellation for workers
|
||
that own temporary input files; cancellation cannot close/delete an input still
|
||
being parsed by a child process.
|
||
Regression: `tests/test_upload_responsiveness.py`,
|
||
`tests/test_conversion_responsiveness.py`, `tests/test_uploads_policy_enforcement.py`.
|
||
|
||
Local acceptance (Windows, Python 3.12, synthetic files via ASGI; no delay injected):
|
||
20 x 900 KiB TXT uploads reduced first concurrent probe delay from 664/682 ms to
|
||
3/4 ms. With real MarkItDown conversion, 20 x 800-paragraph DOCX uploads reduced
|
||
maximum probe scheduling lag from 286/285 ms to 41/14 ms; all 20 sidecars matched
|
||
the source text. The first spawned reader took 13.7 s to initialize/convert in
|
||
this environment, while concurrent probe lag stayed below 16 ms. These are local
|
||
responsiveness measurements, not production throughput guarantees. Upload,
|
||
policy, conversion, large-file and runtime-contention regressions: 87 passed.
|
||
Full-suite collection is separately blocked by the pre-existing missing
|
||
`deerflow.models.global_prompt` import in `tests/test_global_prompt.py`.
|
||
|
||
## Thread metadata query performance and ORM deadlines (2026-10-22)
|
||
|
||
`ThreadMetaRepository._query_conditions` is shared by search/count: common
|
||
SQLite scalar predicates and PG/MySQL string predicates plus system-thread
|
||
exclusion run before SQL pagination/counting. Preserve AUTO owner resolution and
|
||
the nonempty taskId owner bypass while applying agent_id/actionId together.
|
||
`json_compat.JsonPredicate` binds values, preserves local missing/null and numeric
|
||
equality on SQLite, and uses binary MySQL id equality and Unicode-safe blank
|
||
seat detection. Nested/unusual-key filters and PG/MySQL numeric filters remain exact
|
||
residual filters; openGauss retains its original path. Never drop an unsupported
|
||
predicate or replace this with upstream's unadapted SQLite/PG-only compiler.
|
||
|
||
PG's omitted pool_recycle defaults to 300s (other backends remain 1800s);
|
||
database.command_timeout is finite positive or null, default 30s. The shared
|
||
pooled-engine kwargs are used on initial creation and auto-create retry. Only
|
||
asyncpg receives command_timeout; this does not configure psycopg checkpointers.
|
||
Regression entries: packages/harness/tests/test_thread_metadata_sql_filters.py,
|
||
test_database_pool_deadlines.py and the existing test_thread_operation_timing.py.
|
||
Master integration (2026-10-23) retains the maintained task/agent/action index
|
||
columns as additional SQL narrowing before exact JSON predicates. Keep dedicated
|
||
task deep-link card synchronization on every existing metadata mutation. Hydrated
|
||
history still uses slim_display to bound oversized tool payloads without editing
|
||
checkpoint contents; frontend retrieval-progress scopes remain isolated per turn.
|
||
Thread POST defaults to metadata-only creation; only an explicit
|
||
initialize_checkpoint=true writes an empty checkpoint. Never enqueue an empty
|
||
checkpoint in the background: it could become the head after the real first run.
|
||
Authorized metadata-only threads have empty state/history. Metadata GET defaults
|
||
to the SQL row (running maps to busy); include_state=true and checkpoint-only
|
||
legacy rows retain state reads. Owner decorators run before unscoped row reads.
|
||
History explicitly uses root checkpoint_ns and closes alist before serialization;
|
||
only the latest head carries full channel values.
|
||
|
||
The frontend uses BackgroundHistory as the SDK's external history adapter. Its
|
||
post-stream mutate immediately captures streamed state with no checkpoint id;
|
||
explicit end triggers background hydration. EOF/error still requires existing
|
||
run confirmation/rejoin. Every async write is fenced by thread/run/epoch/request;
|
||
new turns, navigation and unmount invalidate prior requests even if abort is
|
||
ignored. Reject history missing the newest streamed turn, bound each request to
|
||
15 seconds and retry completion sync once. Failure keeps the streamed snapshot.
|
||
Normal explicit-end callbacks use the completed streamed snapshot before the
|
||
next send can change page refs; background hydration never repeats them. Recovery
|
||
may have missed final SSE frames, so its callback waits for guarded hydration or
|
||
the retained snapshot fallback. New submits use checkpoint:null to follow the
|
||
latest server head.
|
||
|
||
Auth's SQLiteUserRepository is a historical name: its injected session factory
|
||
is the shared business ORM. With database.backend=mysql, login/auth/users and SQL
|
||
business repositories use MySQL, independently of checkpoint configuration.
|
||
LangGraph Store still follows checkpointer config and startup awaits runtime
|
||
setup/migrations; do not claim total service/startup isolation from PG or migrate
|
||
existing stores implicitly. See docs/THREAD_LATENCY_OPTIMIZATION_ZH.md for the audit.
|
||
Additional regression entries: test_thread_lazy_checkpoint.py,
|
||
tests/test_account_authentication.py and frontend pnpm test:thread-history.
|
||
|
||
## Multi-user contention controls (2026-10-22)
|
||
|
||
- `utils/file_io.py` separates bounded artifact IO (4 workers / 8 admitted) and
|
||
event disk IO (2 / 4) from the default executor used by authentication.
|
||
Contextvars propagate; cancellation cannot release admission until disk work
|
||
actually stops. Heavy artifact listing, ZIP previews, sniffing and writes use it.
|
||
- `runtime/runs/manager.py` sweeps completed live records after 300 seconds,
|
||
200 candidates per pass. It requires the actual task to be done and verifies
|
||
status/error/metadata/kwargs in the backing RunStore, updating a final snapshot
|
||
when necessary. Failed persistence retains the record for later retry. It never
|
||
deletes RunStore rows, checkpoints, events or stream buffers. `watch_completion`
|
||
starts grace after finalization, and lifespan closes the sweeper. The live
|
||
superseded fence survives a newer record's retirement; finalization never adds
|
||
a business-database history read to its completion fence.
|
||
- Run routes authorize the thread before unscoped persisted lookups, and verify
|
||
that the loaded run belongs to that exact thread. Preserve private-owner 404s,
|
||
taskId/system-thread sharing and existing action/agent partitions. Completed
|
||
rejoin without a live bridge buffer emits end immediately; Last-Event-ID replay
|
||
during the existing replay window remains untouched.
|
||
- `run_events.backend=memory` is a compatibility alias for
|
||
`SpoolingRunEventStore`, stored under `runtime_home()/run-events/events.sqlite3`.
|
||
Per-thread sequence assignment and each batch commit are transactional; paged
|
||
queries do not retain completed events in RAM. No automatic event/message
|
||
deletion or truncation is introduced. The pure `MemoryRunEventStore` remains
|
||
available for unit tests. Explicit db/JSONL formats are unchanged; JSONL disk
|
||
operations now use the event executor and striped per-thread locks. Keep the
|
||
spool and its WAL on the persistent local runtime volume. Do not reclassify it
|
||
as a PG checkpoint or silently change explicit db configuration.
|
||
- `runtime/fair_queue.py` provides bounded round-robin admission by user for the
|
||
subagent scheduler (3 total / 2 per user) and background roundtables (4 / 2).
|
||
Queued jobs consume no worker; subagent deadlines include queue time. Per-user
|
||
pending capacity also prevents one user from exhausting the entire backlog.
|
||
Actual coroutine completion, including cancellation before its first step,
|
||
acknowledges slot release. Parent context/identity propagation stays intact.
|
||
`RuntimeContentionConfig` defines optional budgets; limits apply per process
|
||
and require restart. The older roundtable per-user environment override remains.
|
||
- Both internal roundtable creation helpers only create metadata. Init does not
|
||
append a synthetic seat message via loopback; `_leader_run` injects authoritative
|
||
IDs on each real input, deriving them from the thread mapping for older clients
|
||
without seat_meta. Background first-leader injection remains. Never schedule a
|
||
late empty checkpoint write: it could overwrite real history.
|
||
|
||
Regression entries: `test_runtime_contention.py`, `test_completed_run_retention.py`,
|
||
`test_thread_lazy_checkpoint.py`, `test_run_completion_timeout.py`,
|
||
`tests/test_position_roundtable_sessions.py`, roundtable role/model/seat tests,
|
||
artifact/auth/model concurrency tests and frontend `pnpm test:thread-history`.
|
||
See `docs/MULTI_USER_CONTENTION_AUDIT_ZH.md` for measured scope and remaining
|
||
model/SQL/checkpointer contention. Process-local run control still requires the
|
||
current single-worker deployment; these queues do not enable multi-worker routing.
|
||
|
||
## Ordinary knowledge smart retrieval dual surface (2026-10-16)
|
||
|
||
`SmartReasoningRetrievalMiddleware` treats knowledge retrieval as an ordinary
|
||
model-selected information tool inside the reasoning loop. A direct answer or
|
||
skill-only turn is never followed by a forced search. Once the model decides it
|
||
needs selected-library evidence, the first retrieval action is
|
||
`ordinary_knowledge_search`; after reading those raw hits, the model decides
|
||
whether Wiki search/read, regex verification, document reads, or another raw
|
||
query is needed for a real evidence gap. Each model iteration executes at most
|
||
one smart-retrieval call, and any content emitted beside that call is removed so
|
||
the final answer can only follow the completed tool loop. The middleware
|
||
deduplicates current-turn raw queries, regexes, Wiki
|
||
patterns, `page_ref+cursor` reads and document pages; a new continuation cursor
|
||
is a distinct action, while a repeated/non-advancing cursor terminates that
|
||
page. If deduplication removes an empty duplicate-only response, the middleware
|
||
makes one tool-free synthesis retry over existing evidence instead of returning
|
||
a generic non-answer. The user only asks the question; no retrieval recipe is
|
||
required in the message. Original document chunks win when Wiki and raw
|
||
evidence conflict, while stable general knowledge may explain background if it
|
||
is clearly distinguished from library facts and inference. Smart-mode answers
|
||
do not require `[1]`/`[2]` markers or a numbered reference list. Classic
|
||
retrieval does not mount this behavior.
|
||
|
||
Default WeKnora clients are cached per running event loop because they own an
|
||
async authentication lock and a cached directory task; never restore a
|
||
process-wide client cache. Smart-tool mapping authorization reads are routed to
|
||
the SQL engine's owning loop before provider I/O. These two boundaries prevent
|
||
healthy selected knowledge bases from being reported as `PROVIDER_UNAVAILABLE`
|
||
when Gateway and LangGraph work run on different loops.
|
||
|
||
Wiki creation resolves its KnowledgeQA model only from WeKnora's database model
|
||
records. It prefers an active explicit default and calls WeKnora's model debug
|
||
endpoint before binding that model to `summary_model_id` and
|
||
`wiki_config.synthesis_model_id`; `active` alone is not proof that the upstream
|
||
provider still serves the model. A candidate returning `ok=false` may be skipped,
|
||
but authentication, timeout, server, and response-contract failures remain service
|
||
errors. Existing documents that already reached Wiki dead letters require an
|
||
explicit reprocess after both model fields are repaired. Directory and focus
|
||
instructions remain persisted in the knowledge base's database `wiki_config`.
|
||
|
||
`llmwiki_reasoning_tools._wiki_search_impl` applies the safe POSIX expression
|
||
to normalized Wiki pages, but obtains bounded candidates from both the local
|
||
database keyword index and the newest-page snapshot. A completed local mirror
|
||
with no regex match probes the live WeKnora provider, which prevents newly
|
||
generated Wiki pages from being hidden by a stale mirror when periodic sync is
|
||
off. A 0/0 mirror is never considered complete. Provider `content`,
|
||
`content_md`, and `markdown` bodies are normalized for both search and page
|
||
read. Keep the live-provider zero-match fallback when changing mirror policy.
|
||
|
||
## Gongwen bounded delivery (2026-10-07)
|
||
|
||
Gongwen seed v3 ships `scripts/delivery.py` through the restricted
|
||
`run_gongwen.py deliver` / `verify-file` commands. Deliver stages a new DOCX,
|
||
runs draft or repair with one output verification, injects only the explicit
|
||
elements JSON, then checks package/XML/body/element presence without treating
|
||
the red header or imprint as body text. It publishes without replacing an
|
||
existing output and returns ready/review_required/failed JSON plus the review.
|
||
No renderer or external model is invoked; visual pagination and factual review
|
||
are not claimed. Do not add automatic repair/check loops or ad-hoc Python to
|
||
the workflow/SOUL prompts. P1/P2 warnings are not delivery blockers.
|
||
|
||
The vendored optimize command reuses its input check and returns full output
|
||
review details; failed verification exits nonzero. Draft propagates its resolved
|
||
document type into optimization. Gateway generate/repair reuse this invocation's
|
||
review, with one check fallback for older custom skill versions. Do not reuse
|
||
arbitrary client-supplied review files as trusted verification.
|
||
Regression coverage: `tests/test_gongwen_delivery.py` and existing gongwen tests.
|
||
|
||
## Retrieved skill files (2026-10-07)
|
||
|
||
`citation_display.retrieved_files_enabled` (default true) controls retrieved-file
|
||
cards globally. It is read and patched through `/api/system-settings/citation-display`;
|
||
updates require `_require_admin` and preserve other citation settings.
|
||
|
||
Chat file cards reuse the existing artifact and knowledge original-file readers.
|
||
`GET /api/llmwiki/sources/resolve` accepts either a local mapping id or a WeKnora
|
||
provider id and returns only an authorized local mapping id. A provider lookup
|
||
must be followed by the existing `get_authorized(..., write=False)` check before
|
||
returning any identifier; missing and unreadable sources both return 404.
|
||
The frontend uses this resolved id for both preview and original-file download.
|
||
|
||
## Qwen3.8 Flash Next stream integrity (2026-10-07)
|
||
|
||
`deerflow.models.chat_stream_state.ChatStreamState` belongs to one provider invocation.
|
||
`PatchedChatOpenAI` enables it only for the normalized `qwen3.8-flash-next` id;
|
||
the model factory also checks the configured name for deployments using a generic
|
||
provider id. The local-only `strict_stream_integrity` override is never sent upstream.
|
||
Keep unrelated Qwen versions and other models on their existing stream path by default.
|
||
|
||
Only explicit `choice.message` string snapshots are converted into unseen suffixes.
|
||
True `delta` text wins and is appended verbatim, including repeated content/reasoning;
|
||
structured provider payloads stay opaque. Repair a null terminal delta for this model.
|
||
EOF without a first-choice `finish_reason`, or a contradictory snapshot, raises the
|
||
content-free `IncompleteChatStreamError`. The existing LLM middleware retries within
|
||
its normal cap before any failed result reaches the checkpoint or tool executor.
|
||
Cancellation propagates without retry; a declared finish reason (including `length`)
|
||
retains provider semantics. Do not guess completion from answer punctuation.
|
||
|
||
Non-stream exchange capture is opened and closed in `_generate` / `_agenerate` so
|
||
executor-based conversion never resets a ContextVar token from another context.
|
||
Reuse the safe response dump for conversion and preserve typed `parsed` / `refusal`
|
||
metadata without serializing arbitrary parsed objects a second time.
|
||
|
||
Regression entry: `packages/harness/tests/test_patched_openai_text_integrity.py`.
|
||
Coverage includes sync/async provider replay, model/alias scoping, failed-attempt
|
||
checkpoint isolation, same-thread follow-ups, tool safety, cancellation and 20
|
||
overlapping streams that reuse the same provider completion id. These fixtures
|
||
exercise code-level compatibility gaps; they do not establish the deployed server's
|
||
actual trigger without its response evidence.
|
||
??
|
||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||
|
||
## Project Overview
|
||
|
||
### Personal follow-up question preference
|
||
|
||
`GET/PUT /api/user-preferences` exposes `followup_suggestions_enabled`, defaulting to `false` for both new accounts and older JSON rows without the key. Partial updates merge into the current account's `user_preferences.preferences` JSON alongside `hide_admin_prompt_prefix`; no database migration is needed. Frontend recommendation generation and display must both require this explicit opt-in. Regression coverage: `tests/test_user_preferences.py` (default-off behavior, persisted opt-in/opt-out, account isolation and partial-update preservation).
|
||
|
||
### Ordinary Q&A map visualization
|
||
|
||
The ordinary lead-agent prompt includes `_ORDINARY_QA_VISUALIZATION_SECTION` only for normal, non-bootstrap, non-writing, non-notebook, non-scheduled runs and defines the highest-priority output-carrier gate. Only an explicit current-request ask for HTML, a webpage/page, dashboard, large-screen view, or `.html` file permits reading `page-report-generation`, calling `page_report_template`, or writing HTML. “Draw a flowchart/chart/map/mind map” and “make a visualization” never imply HTML: ordinary flowcharts must be inline Mermaid, ordinary charts keep the established message-inline path, and requested maps emit strict JSON in a `map-json` fence (markers use `name`+`coordinate` plus optional `color`; lines are included only when the user explicitly asks to connect places, must use `coordinates` with at least two `[lng,lat]` points, not `from`/`to`/`coords`, and default to `arrow`/`flow`). The frontend recovers those common aliases, assigns distinct marker colors, draws requested lines as looping SVG arrows, and drops unusable lines so markers still render. A mind map is emitted only for an explicit current-request ask for a mind map/knowledge map (or `analysis-mindmap`); ordinary analyses, summaries, outlines, flowcharts, and generic JSON must never auto-upgrade into one. Its strict `mindmap-json` contract is a bounded rooted tree (`id`, `parentId`, `label`, semantic `kind`, optional detail/source refs), targeting 12–24 nodes with a hard cap of 36, at most four layers including the center topic, at most six top-level dimensions and three children beneath every later node. Models must consolidate related facts into `detail`/`sourceRefs`, not make one node per report bullet, datum, paragraph, or reference. An unfinished dedicated `mindmap-json` fence (and compatible historical `mindmap` fence) remains an ordinary streaming code block; only the completed, valid object lazy-renders as a compact left-to-right message card. Clicking a parent node, or its +/− control at the outgoing connection point, expands/collapses its child tree and the canvas uses an explicit visible edge stroke. The full dialog keeps its close and view tools in the top-right and a layout-derived all-map preview in the bottom-right; the message preview also permits pan, wheel zoom, and zoom/reset controls. The message card's icon-only download control opens PNG/SVG/JSON choices. Generic JSON never invokes this renderer. Ambiguity defaults to inline output. Once the HTML gate passes, delivery is mandatory as a sandbox-generated `/mnt/user-data/outputs/*.html` artifact opened with `present_files`; a full inline `html` fence is forbidden as a substitute. The shared frontend `MarkdownContent` keeps its exact legacy branch when no map-shaped fence exists; dedicated `map-json` fences and complete generic `json` fences that contain map fields and pass the full map schema are validated and lazy-rendered from `components/workspace/visualizations/`. Map/loading/error cards use the full message-content width so their edges align with the step panel. Non-map and incomplete generic JSON retain the ordinary code-block path.
|
||
|
||
HTML artifacts are single-page by default. Omitted `preferred_page_count` is normalized to `1` during direct validation. A count greater than one is allowed only when the current user directly asks for multiple pages or gives a page count; content length and report chapter count never imply multi-page intent. Explicit multi-page intent without a count defaults to a three-page lightweight portal. It is one artifact HTML, not a folder of linked pages: `<main class="pages">` contains logical `.page` sections with unique `id`/`data-page`, and `href="#page-..."` navigation works in the sandboxed Blob preview through the injected anchor handler. Its validator requires the local Bootstrap/Alpine assets, `.pages`, multiple `data-page` sections and real anchor navigation: every actual `<a href="#page-...">` and every script-set `location.hash` must match a unique logical-page id, and each page must be reachable; comments or `#overview`/`#stations` data-page values cannot spoof this check. `MultiPageHtmlMiddleware` repeats that structural inspection at the `present_files` boundary, so a model that skips validate or writes custom tabs cannot expose a nonconforming portal. A user-uploaded background goes only on the requested logical `.page` in this mode (home page when unspecified), not on every page. When a template is explicitly requested, prepare strictly excludes profiles from the wrong page-count mode, persists `requested_page_count` and `page_mode`, and validate reloads them. Validation rejects multiple detected page views (`data-page`, tabpanels, `.report-page`, or `.view`) in single-page mode, or a single view when multi-page was explicitly requested.
|
||
|
||
Ordinary HTML generation is from scratch by default: do not read `page-template-library`, call catalog/prepare, or select a bundled layout unless the current user explicitly asks to use a template. The agent writes the final path directly under `/mnt/user-data/outputs/`, which lets frontend live preview open on the first `write_file`; it then validates and presents the same file without a bash/cp copy. Single-page HTML means a scrollable responsive document, not a fixed single-screen canvas. Direct validate derives `layout_mode` and page count from `page_brief` / `preferred_page_count`; it rejects `html/body overflow:hidden`, fixed 1920×1080 viewport scaling, undersized G6 containers, and long LR diagrams. Explicit template requests may use the shared `light-document` / `dark-document` profile backed by `report-document-responsive-template.html`; prepare creates its editable `working_copy` directly in outputs and persists the same constraints.
|
||
|
||
For an explicit multi-page HTML request without an explicit template request, `MultiPageHtmlMiddleware` rejects `page_report_template` catalog/prepare calls; the model must make one complete first `write_file`, then one validate. A failed validation returns patch-only repair guidance: retain the artifact and use the smallest `str_replace` edits for all listed CSS/navigation/JSON/DOM errors before one revalidation; do not recreate the page. `html_postprocess.maybe_fix_generated_html` also injects a marker-guarded dark Bootstrap compatibility layer into dark HTML during every write or replacement. It sets Bootstrap body/emphasis/secondary tokens and explicit readable foregrounds for `.card`, `.data-table` cells, `.modal-*`, and common form controls. Validation mirrors this by rejecting dark `.card`, `.data-table`, and modal markup without an explicit matching foreground color, so a preview cannot silently render black text on dark panels.
|
||
|
||
### Generated HTML artifact visualization runtimes
|
||
|
||
The same normal lead-agent prompt includes `_HTML_ARTIFACT_VISUALIZATION_SECTION`, but its contents are executable only after the explicit HTML-intent gate above passes; visualization content alone never implies this intent. New generated HTML explicitly loads browser distributions from Gateway `/api/public/html-runtime/`: `bootstrap.min.css`, `alpine.min.js`, `bootstrap.bundle.min.js`, `g6.min.js`, `echarts.min.js`, and then `deerflow-runtime.js`; it can also load the packaged country/region map catalog: `china-provinces`, `us-states`, `japan-prefectures`, and `taiwan-counties`. Bootstrap is the responsive layout/modal layer and Alpine is the small state/list-rendering layer; jQuery, React, Vue, Babel and public CDNs are not part of this artifact contract. The files are package resources under `packages/harness/deerflow/html_runtime/assets/` and ship with the harness wheel, so HTML execution does not depend on frontend Vite/node_modules assets or a public CDN. `system_settings.json -> html_map.engine` (`GET|PUT /api/system-settings/html-map`, admin write) chooses `echarts` (default, local GeoJSON `data-df-geo`) or `leaflet` (地图网页, `data-df-map`). The map tile URL saved in the settings dialog (`system_settings.json -> html_runtime.map_tile_url`, `GET|PUT /api/system-settings/html-runtime`) is injected into `deerflow-runtime.js` as `mapTileUrl` and is used only by the Leaflet engine. Both initializers stay in the runtime so already generated pages keep rendering after the switch. The next ordinary Q&A prompt and `page_report_template` validation follow the selected engine. The runtime base URL saved in the same settings block is the browser-reachable absolute address written into generated source. An empty value falls back to an older `config.yaml` entry and then `http://localhost:8001/api/public/html-runtime`. Flowcharts use paired `data-df-flow` JSON blocks and AntV G6, not Mermaid; charts use `data-df-chart`; maps use ECharts Canvas-backed `data-df-geo` with the local GeoJSON catalog. `data-df-chart` must be strict JSON: no JavaScript callbacks or function-valued colors, with literal colors/data-item styles and string tooltip formatters instead. Dark HTML must set `data-df-theme="dark"` and explicitly give ECharts global/title/legend/axis/visible-series labels readable light text, including `nameTextStyle` for named axes. Map `description` values are plain text because the runtime escapes markup; click-to-filter or page-switch behavior listens to `deerflow:geo-map-click` and uses a valid `#page-...` anchor. `deerflow-runtime.js` waits until a hidden logical page has a real layout before creating ECharts and tracks later container changes with `ResizeObserver`, so map/chart canvases never depend on a browser-window resize for their first visible render. It initializes all three contracts, opens a detail popover for described regions/markers, and exposes catalog URLs through its public dynamic response. Map large screens use a centered map stage with narrow side rails and bottom summaries, rather than cards layered across a full-page map background. Frontend runtime replacement is bypassed whenever this explicit Gateway runtime is present. This remains separate from ordinary message rendering; ordinary `map-json` keeps its existing frontend tile renderer and `VITE_MAP_TILE_URL` configuration, and ordinary `flow-json`/`chart-json` must not be reintroduced.
|
||
|
||
User-uploaded raster images can be used as generated-HTML backgrounds through signed, unauthenticated `GET /api/public/html-images/{token}` URLs. `UploadsMiddleware` mints one URL per uploaded `.jpg/.jpeg/.png/.webp/.gif/.bmp` and injects it into `<uploaded_files>` as `Public URL`; the HTML contract requires that exact absolute URL in `background-image`, never a sandbox path, artifact API path, relative path, or base64 data URI. Single-page artifacts place it on `html/body`; the multi-page portal places it only on the requested logical page. Tokens are HMAC-signed, do not expire (the URL is written into durable HTML), and only grant read access to one file under the thread `uploads/` or `outputs/` directory. Engine: `deerflow.html_runtime.public_images`; router: `app/gateway/routers/html_images.py`; tests: `tests/test_html_images.py`.
|
||
|
||
The G6 runtime must not mutate author layout by auto-switching long LR graphs to TB or by increasing the target element height after render. Layout quality is decided before delivery: keep a flow to at most eight nodes where possible; otherwise split it or declare TB with at least 600px. Each flow gets its own document row and a parent that can grow naturally, preventing canvas overlap with later sections.
|
||
|
||
Markdown-to-HTML conversion must inventory source Mermaid/flowcharts, charts, and maps before editing. Direct `page_report_template validate` derives `required_visualizations` from its `required_features` and `page_brief`; an explicit-template prepare also persists a machine-readable `visualization_contract` in the per-output brief, which validate reloads and merges with its arguments. Required flowcharts fail validation unless the HTML contains `g6.min.js`, `deerflow-runtime.js`, a `.df-flow` container, and `data-df-flow`; embedded Mermaid and inline `new G6.Graph(...)` are errors. Required charts use ECharts + `data-df-chart`. Required maps follow `html_map.engine`: ECharts uses `data-df-geo`, an `.df-geo-map` container, and one of the four local GeoJSON catalog entries; 地图网页 uses Leaflet `leaflet.css`/`leaflet.js`, `data-df-map`, and a `.df-map` container. The unselected contract, handwritten map initialization, and (for ECharts) map tile URLs are validation errors. A failed report must be repaired and revalidated before `present_files`.
|
||
|
||
G6 nodes use light semantic rounded cards with matching borders, dark high-contrast text, centered type-prefixed labels, subtle shadows, and non-animated initial rendering (avoids a faded first frame). More than seven nodes automatically switch an `LR` request to layered `TB` layout and expand a short container to 720px, preventing long workflows from shrinking into illegible shapes; drag/zoom remain enabled.
|
||
|
||
### 问答记忆运行时总开关
|
||
|
||
通用/个人问答记忆由 `system_settings.json -> memory.enabled` 控制，默认 `false`，并且是该通用记忆域的权威总开关。关闭时 `lead_agent` 不挂载通用 `MemoryMiddleware`，工具注册不暴露 memory/hindsight 工具，prompt 不读取通用记忆，因此也不会触发 V1 的延迟提取模型调用。普通问答项目的项目级跨会话记忆是独立安全域，只受项目自身 `enabled` 控制，不得被这个通用总开关阻断。`config.yaml -> memory` 仅保留 provider/version/限额等参数，不能自行启用通用记忆。管理员通过 `GET/PUT /api/system-settings/memory` 和「设置 → 问答记忆」修改通用记忆开关；下一次创建的问答立即按新值构建。
|
||
|
||
### 项目空间（固定问答模式）
|
||
|
||
- `app/gateway/routers/analysis_projects.py`：用户私有项目 CRUD 和项目下新建/查询对话；更新请求不允许 `mode`，对话归属不通过可编辑 metadata 保存。
|
||
- 项目线程 metadata 的 `analysis_project_id` / `analysis_project_mode` / `analysis_project_name` 仅用于最近对话的类型、项目名与路由展示；创建、列表修复和每次运行都由服务端覆盖，授权与真实归属仍以 `analysis_project_threads` 绑定为准。
|
||
- `deerflow.persistence.analysis_projects`：`analysis_projects` / `analysis_project_threads` / additive `analysis_project_configs`，经 models 注册和 Gateway deps 初始化自动建表。配置表保存项目默认智能体、策略 Lens、报告结构快照及每条对话所选智能体，避免修改旧表破坏已有部署。
|
||
- `app/gateway/analysis_projects.py::prepare_project_run` 在共享 `start_run` 前置读取绑定、校验所有者及知识库、覆盖运行模式；运行知识范围是项目自身普通/助手知识库与该线程已绑定智能体普通知识库的去重并集，不能只取智能体配置而丢掉项目知识库；`authz.py` 同时保护直接读取线程。不要将 `analysis_project` 加入公共 context 白名单。
|
||
- `deerflow.agents.analysis_project` 提供三个模式的系统约束和固定方案模板；方案复用 deep_research_report 的 fixed structure，thread_harvester 在方案项目中额外采集服务端检索的引用。普通写作仍使用原有 adaptive 配置。
|
||
- 普通问答项目的跨会话记忆是独立安全域，权威 scope 为 `(owner_user_id, project_id)`，只信任 `analysis_project_threads` 的服务端绑定。Gateway 会清除客户端伪造的 `project_memory_scope` / `project_memory`，再由 `prepare_project_memory` 注入；`services.inject_validated_analysis_project_context` 同时禁用通用 `MemoryMiddleware` 与 memory/hindsight 工具。harness 只消费转义、有界、明确标为非指令数据的 `<project_memory>`，不得反向导入 `app.*`。
|
||
- 项目记忆只受项目自身 `enabled` 和 store readiness 门控；`system_settings.json -> memory.enabled` 只管理通用/个人记忆，即使它关闭，`prepare_project_memory`、成功回调入队、executor 与 dispatcher 仍须为已启用的 QA 项目工作。项目暂停会取消并围栏该项目 queued/running job。项目记忆异常必须降级为无记忆问答，不能污染普通运行配置。
|
||
- 项目记忆 job 的内部 `causal_generation` 保存在 `payload_json`（不得为此新增数据库列；兼容已由 `create_all` 建好的表），状态只按最新因果代次判断；一个全量重建的所有 job 共用 generation，任一失败使该批次失败。checkpoint 枚举完成前必须保持未 seal 的 `rebuilding`，避免先完成的部分 job 提前转 ready。dispatcher 通过 `claim_stalled_rebuilds` 恢复超过宽限期且没有活动 job 的 `rebuilding/recovering`，认领和重放期间召回必须 fail-closed。
|
||
- 持久化位于 `deerflow.persistence.analysis_project_memory`：`states` 保存项目设置和确定性快照，`items` 保存带来源、logical key、版本和可选 embedding 的原子记忆，`jobs` 保存脱敏后的幂等抽取任务。三个模型由统一 `Base.metadata.create_all` 自动建表，不需要新增 Alembic revision。
|
||
- 成功 run 只把最近完整 user/assistant exchange 入队，后台 executor 抽取并可调用共享 Wiki encoder。召回为 lexical + vector + importance/confidence/recency 混合排序；只有 embedding model/fingerprint 与 query encoder 完全匹配时使用向量，否则降级关键词。召回要排除当前 thread 的原子条目，并受 item/episode/token budget 限制；抽取任务中的 checkpoint 文本、项目名和背景在落库前必须脱敏。
|
||
- 对话编辑、单轮删除、清空、整线程删除以及直接 state 消息替换，必须先调用 `invalidate_project_memory_before_thread_mutation` 持久化失效并取消 lease，再修改 checkpoint；前置失效失败时拒绝修改。后续 rebuild 失败时保持 `rebuilding`，recall fail-closed。`start_rebuild` 删除旧的自动条目但保留手工或 pinned 条目；同 logical key 按 checkpoint `source_order` 收敛，不能由并发 worker 的完成顺序决定。dispatcher 只能按真实空槽 claim，过期任务达到 `max_attempts` 后必须终止。
|
||
- 管理 API：`GET/PATCH /api/analysis-projects/{id}/memory[/settings]`、`POST .../memory/rebuild`、`DELETE .../memory` 和 `.../memory/items*`；全部先走 `owned_project` 且只接受 `mode=qa`。前端 `ProjectMemoryPanel` 展示状态、快照、来源，并提供手工维护、固定、清空和重建。专项回归为 `tests/test_analysis_project_memory*.py` 加 `tests/test_analysis_projects.py`，必须同时覆盖内存与 SQLite、通用记忆关闭时 A 会话仍可沉淀到 B 会话召回、跨项目/跨用户隔离、项目暂停、来源失效、纠错顺序、租约与关键词降级。
|
||
- 方案生成首份报告对用户保持两步语义：「正常问答理解需求 → 结束问答并生成方案」。第一步应自然交流并先做语义级意图理解：复述真实目标、面向对象、可观察的成功结果与非目标/边界，推断必须标为暂定；后续消息增量更新整体理解，最新明确要求覆盖较早的冲突表述。仅在关键条件缺失时少量追问；信息足够后输出自然段式的简短「意图理解」，以及唯一的 `solution-intent-json` HTML 注释快照和末尾就绪标记，不得呈现为表单式访谈或额外确认阶段。快照的 `goal` / `audience` / `success_outcome` / `scope` / `constraints` / `deliverable` / `acceptance_criteria` 必须是有界非空字符串，`open_questions` 必须是有界字符串数组且只有为空时可就绪。`prepare_project_run` 删除客户端伪造的 `solution_requirements_confirmed` / `solution_intent`，只解析检查点中最新用户轮次后的最新助手回复；只有可读摘要、合法快照、末尾就绪标记和当前用户确认标记同时成立时才确认，并把归一化快照作为服务端权威 `solution_intent` 注入运行配置。`services.inject_validated_analysis_project_context` 负责绕过公共白名单安全注入这些服务端字段，确认后的系统提示把转义后的快照交给最终生成。已有 `report.md` 仅用于后续编辑的兼容解锁；`deep_research_report` 在首份方案未确认时必须拒绝执行。不要把该闸门退化为仅靠前端按钮或提示词的约定。
|
||
- 方案阶段的关键追问必须调用现有 `ask_clarification`：普通方案需求第一步至少给出 1 个最关键的可点击决策问题，一个调用只问一个决策点，提供 2–4 个稳定选项（含「其他/我补充」），总量限制 1–3 个，由 `ClarificationMiddleware` 继续输出既有 `{question, clarification_type, context, options, allow_custom, allow_multiple}` 载荷供前端点击。独立问题可同轮多次调用；不要用普通 Markdown 问句替代工具卡，也不要在用户回答前只输出快照/就绪标记。另有服务器校验的强制生成令牌 `<!-- solution-requirements:force -->`：只认本轮最后一条 user/human 消息，仍清除客户端伪造的 `solution_force_generate` / `solution_requirements_confirmed` / `solution_intent`。强制生成可复用 checkpoint 中最近一个合法但未就绪的快照（允许 `open_questions` 非空；没有则为 `None`），prompt 必须结合完整对话推断缺项、显式标注暂定假设并禁止再次澄清。确认/强制生成首报轮以 `WritingModeReportMiddleware(require_pipeline=True)` 做运行时兜底：纯文本“已生成”或澄清调用会被确定性改写为 `deep_research_report`；任何完成提醒/循环成功判定都必须同时看到固定成功工具结果和状态中真实登记的 `/mnt/user-data/outputs/report.md` artifact，被意图门禁拒绝的 ToolMessage 绝不能触发“已完成”提醒。当前用户轮次里只要管线返回结果却没有 artifact，就输出确定性的失败/重试提示并结束，禁止自动重试到循环上限；下一次用户点击「重新生成方案」才开始新调用。
|
||
- 方案项目最终落盘的 `report.md` 禁止 `[1]` / `【1】` 等数字引用标记。`deep_research_report_tool` 在 solution 模式下同时注入无编号引用要求并对最终 Markdown 做确定性清理：正文删除数字引用标记，参考来源改为无编号列表，代码围栏和行内代码保持原样；普通写作与深度研究不走该分支。后续方案修改的系统提示也必须维持这项约束。
|
||
- 前端 `frontend-web/src/pages/AnalysisProjectsPage.tsx` 和 `pages/analysis-projects/ProjectConversation.tsx` 嵌入现有 WorkspaceSidebar，复用消息流、输入框、研究进度和产物画布；策略项目默认打开真实消息流，用户问答、reasoning 与技能/工具执行步骤必须在运行中可见，结构化看板仍可一键切换。`StrategyBoard.tsx` 只解析对用户可见的「意图理解 / 多视角策略分析与辩论 / 完整策略方案」Markdown，不读取 reasoning/tool payload。完整策略方案以可导出的文档结构呈现。方案设置复用 `WorkflowConfigDialog` 管理报告结构。
|
||
- 回归测试：`tests/test_analysis_projects.py`（项目 CRUD、权限、模式锁定、运行配置、模板及记忆管理 API）和 `tests/test_analysis_project_memory*.py`（持久化、运行时、双后端可靠性与真实跨会话效果）。
|
||
|
||
DeerFlow is a LangGraph-based AI super agent system with a full-stack architecture. The backend provides a "super agent" with sandbox execution, persistent memory, subagent delegation, and extensible tool integration - all operating in per-thread isolated environments.
|
||
|
||
**Architecture**:
|
||
- **Gateway API** (port 8001): REST API plus embedded LangGraph-compatible agent runtime
|
||
- **Frontend** (port 3000): Next.js web interface
|
||
- **Nginx** (port 2026): Unified reverse proxy entry point
|
||
- **Provisioner** (port 8002, optional in Docker dev): Started only when sandbox is configured for provisioner/Kubernetes mode
|
||
|
||
**Runtime**:
|
||
- `make dev`, Docker dev, and production all run the agent runtime in Gateway via `RunManager` + `run_agent()` + `StreamBridge` (`packages/harness/deerflow/runtime/`). Nginx exposes that runtime at `/api/langgraph/*` and rewrites it to Gateway's native `/api/*` routers.
|
||
|
||
**Project Structure**:
|
||
```
|
||
deer-flow/
|
||
??? Makefile # Root commands (check, install, dev, stop)
|
||
??? config.yaml # Main application configuration
|
||
??? extensions_config.json # MCP servers and skills configuration
|
||
??? backend/ # Backend application (this directory)
|
||
? ??? Makefile # Backend-only commands (dev, gateway, lint)
|
||
? ??? langgraph.json # LangGraph Studio graph configuration
|
||
? ??? packages/
|
||
? ? ??? harness/ # deerflow-harness package (import: deerflow.*)
|
||
? ? ??? pyproject.toml
|
||
? ? ??? deerflow/
|
||
? ? ??? agents/ # LangGraph agent system
|
||
? ? ? ??? lead_agent/ # Main agent (factory + system prompt)
|
||
? ? ? ??? middlewares/ # 10 middleware components
|
||
? ? ? ??? memory/ # Memory V2: provider/manager/builtin/hindsight
|
||
? ? ? ??? thread_state.py # ThreadState schema
|
||
? ? ??? sandbox/ # Sandbox execution system
|
||
? ? ? ??? local/ # Local filesystem provider
|
||
? ? ? ??? sandbox.py # Abstract Sandbox interface
|
||
? ? ? ??? tools.py # bash, ls, read/write/str_replace
|
||
? ? ? ??? middleware.py # Sandbox lifecycle management
|
||
? ? ??? subagents/ # Subagent delegation system
|
||
? ? ? ??? builtins/ # general-purpose, bash agents
|
||
? ? ? ??? executor.py # Background execution engine
|
||
? ? ? ??? registry.py # Agent registry
|
||
? ? ??? tools/builtins/ # Built-in tools (present_files, ask_clarification, view_image)
|
||
? ? ??? mcp/ # MCP integration (tools, cache, client)
|
||
? ? ??? models/ # Model factory with thinking/vision support
|
||
? ? ??? skills/ # Skills discovery, loading, parsing
|
||
? ? ??? config/ # Configuration system (app, model, sandbox, tool, etc.)
|
||
? ? ??? community/ # Community tools (tavily, jina_ai, firecrawl, image_search, aio_sandbox)
|
||
? ? ??? reflection/ # Dynamic module loading (resolve_variable, resolve_class)
|
||
? ? ??? utils/ # Utilities (network, readability)
|
||
? ? ??? client.py # Embedded Python client (DeerFlowClient)
|
||
? ??? app/ # Application layer (import: app.*)
|
||
? ? ??? gateway/ # FastAPI Gateway API
|
||
? ? ? ??? app.py # FastAPI application
|
||
? ? ? ??? routers/ # FastAPI route modules (models, mcp, memory, skills, uploads, threads, artifacts, agents, suggestions, channels)
|
||
? ? ??? channels/ # IM platform integrations
|
||
? ??? tests/ # Test suite
|
||
? ??? docs/ # Documentation
|
||
??? frontend/ # Next.js frontend application
|
||
??? skills/ # Agent skills directory
|
||
??? public/ # Public skills (committed)
|
||
??? custom/ # Custom skills (gitignored)
|
||
```
|
||
|
||
## Important Development Guidelines
|
||
|
||
### Documentation Update Policy
|
||
**CRITICAL: Always update README.md and CLAUDE.md after every code change**
|
||
|
||
When making code changes, you MUST update the relevant documentation:
|
||
- Update `README.md` for user-facing changes (features, setup, usage instructions)
|
||
- Update `CLAUDE.md` for development changes (architecture, commands, workflows, internal systems)
|
||
- Keep documentation synchronized with the codebase at all times
|
||
- Ensure accuracy and timeliness of all documentation
|
||
|
||
### Workflow Studio stream and canvas compatibility
|
||
|
||
- `PatchedChatOpenAI` accepts provider response fields `reasoning_content` and `reasoning`, normalizing both into `AIMessage.additional_kwargs.reasoning_content`. For OpenAI-compatible models that declare `supports_thinking: true` without an explicit `when_thinking_enabled`, `create_chat_model` sends `extra_body.enable_thinking=true` whenever the frontend enables thinking. `app/gateway/workflow_agent_runner.py` is the boundary between LangGraph token chunks and durable workflow events. Provider reasoning in `AIMessage.additional_kwargs.reasoning_content`/`reasoning` is emitted to the UI as balanced inline `<think>` blocks; only visible answer content becomes the node's `text` output for downstream nodes. It must also emit every original `[AIMessage|ToolMessage, metadata]` tuple as `node.message.data.messages` with no whitelist, preview or length crop. Tool results and `additional_kwargs.clarification` are needed by the Studio's DeerFlow-compatible message UI and must survive durable SSE replay.
|
||
- An agent `ask_clarification` ToolMessage pauses the workflow at that agent node (`awaiting_input`), not as a terminal run. The pause records the source `nodeRunId`/`toolCallId`; resuming turns the submitted reply into the next user message on the same `wf-{run}-{node}` LangGraph thread. Keep the raw ToolMessage authoritative; do not invent a second card payload.
|
||
- `app/gateway/workflow_proposal_planner.py` converts the workflow controller's selected catalog roles into per-agent `mission`/`deliverable`/`scope`/`handoff` contracts. Only a selected visible `agentId` may contribute a contract; every omitted or malformed field falls back to a deterministic role-aware contract. Preserve this prompt-level worker boundary when adding strategies: researchers must not silently become final writers, and sequential roles must receive their upstream handoff binding plus an explicit next handoff.
|
||
- `app/gateway/routers/workflows_coze_compat.py` must return both `creator.self: true` and editable VCS metadata for an owned standalone workflow. Coze Playground derives its preview/read-only mode from those fields, so omitting `creator.self` disables node dragging even when the standalone page passes `readonly={false}`.
|
||
|
||
## Commands
|
||
|
||
**Root directory** (for full application):
|
||
```bash
|
||
make check # Check system requirements
|
||
make install # Install all dependencies (frontend + backend)
|
||
make dev # Start all services (Gateway + Frontend + Nginx), with config.yaml preflight
|
||
make start # Start production services locally
|
||
make stop # Stop all services
|
||
```
|
||
|
||
**Backend directory** (for backend development only):
|
||
```bash
|
||
make install # Install backend dependencies
|
||
make dev # Run Gateway API with reload (port 8001)
|
||
make gateway # Run Gateway API only (port 8001)
|
||
make test # Run all backend tests
|
||
make lint # Lint with ruff
|
||
make format # Format code with ruff
|
||
```
|
||
|
||
Regression tests related to Docker/provisioner behavior:
|
||
- `tests/test_docker_sandbox_mode_detection.py` (mode detection from `config.yaml`)
|
||
- `tests/test_provisioner_kubeconfig.py` (kubeconfig file/directory handling)
|
||
|
||
Boundary check (harness ? app import firewall):
|
||
- `tests/test_harness_boundary.py` ? ensures `packages/harness/deerflow/` never imports from `app.*`
|
||
|
||
CI runs these regression tests for every pull request via [.github/workflows/backend-unit-tests.yml](../.github/workflows/backend-unit-tests.yml).
|
||
|
||
## Architecture
|
||
|
||
### Position roundtable recovery persistence
|
||
|
||
`/api/position-roundtable` uses the shared `/api/intent/*` LangGraph flow as
|
||
the live Step-1 execution source and stores an additional SQL recovery copy in
|
||
`position_roundtable_sessions`/`position_roundtable_nodes`. The recovery copy
|
||
contains complete intent/node/summary/action-plan conversations, unsent
|
||
composer drafts, workflow snapshots, and bounded full text for generated
|
||
Markdown/JSON artifacts. This lets history and downstream closing agents keep
|
||
working when container-local checkpoint or sandbox volumes are replaced.
|
||
Revision `20260723_05` adds the conversation columns. Session deletion
|
||
explicitly removes child nodes so SQLite deployments do not depend on FK
|
||
cascade pragmas. The router is mounted canonically below
|
||
`/api/position-roundtable` and, for intranet gateway allowlists, below the
|
||
schema-hidden compatibility prefix `/api/multi-agent/position-roundtable`.
|
||
Clients may retry the compatibility route only for a framework/proxy-level
|
||
`404 Not Found`; domain-level missing-session responses must remain real 404s.
|
||
|
||
The position-roundtable state machine is multi-worker safe: node commands
|
||
(`complete`, `bind`, conversation save, reject) and session commands
|
||
(`activate`, archive, delete) lock the parent session and affected nodes in one
|
||
transaction, then record a unique command id for retry replay. The task and
|
||
intent snapshot writes use the same parent-row lock; they are permitted only
|
||
while `status=intent_pending`, so a late “confirm intent” write cannot change a
|
||
chain that another user has already activated. Activation itself rechecks the
|
||
stored confirmed intent after taking the lock. `tests/test_position_roundtable_
|
||
commands.py` and `tests/test_position_roundtable_concurrency.py` cover the
|
||
state transitions; `tests/test_roundtable_postgres_concurrency.py` is the real
|
||
PostgreSQL acceptance suite (set `DEERFLOW_POSTGRES_CONCURRENCY_TEST_URL`).
|
||
|
||
Step-2 roundtable rooms share one deployment-wide cap,
|
||
``system_settings.json -> roundtable_room_limit`` (default enabled, 3 rooms).
|
||
``GET/PUT /api/system-settings/roundtable-room-limit`` is the admin switch.
|
||
Occupancy is summed across **every account**: durable foreground leases keyed
|
||
by a hidden per-live-execution ``roomSessionId``, in-flight LangGraph runs on
|
||
threads stamped with the corresponding ``roundtable_room_key``, and executing
|
||
background jobs (``queued`` / ``running`` / ``cancel_requested`` with a live
|
||
worker lease or a recent update). Different pages running the same saved draft
|
||
have different session ids and therefore take different rooms; all seats and
|
||
Step 3 calls within one live execution share its key. Run-manager inspection
|
||
is reconciliation-only: when a durable lease/job already owns the same
|
||
draft+user/task scope, stale or overlapping run keys in that scope are not
|
||
counted again. Merely opening the page,
|
||
viewing saved history, paused ``awaiting_input`` jobs, stale historical rows,
|
||
and Step-1 intent threads do not occupy a slot. Foreground-to-background
|
||
handoff replaces the foreground lease with the durable job occupancy. There is
|
||
no per-user pool. Admission is serialized by a database gate row, and the cap
|
||
is enforced at acquire, init, every model-producing run, and background-job
|
||
start. Full pools return HTTP 409; an enabled gate with missing storage or a
|
||
missing session id fails closed. The switch applies on the next admission
|
||
without a restart. Position and eight-department roundtables are not counted.
|
||
|
||
Roundtable backend jobs use a database unique active-dedupe key plus a leased
|
||
dispatcher, never process-local start deduplication. Cancellation is two
|
||
phase (`cancel_requested` then `cancelled`): the lease owner’s watcher normally
|
||
finalizes it, while the dispatcher reaps a cancellation whose lease has
|
||
expired after a worker crash. A worker that loses lease stops its local model
|
||
run immediately. Do not reintroduce direct `executor.cancel()` from the HTTP
|
||
cancel route, because it can cancel the watcher before the database state is
|
||
finalized. The boolean defaults on these new rows must use SQLAlchemy
|
||
`false()` rather than numeric `0`, so fresh PostgreSQL schemas can be created.
|
||
|
||
Position roundtable selects the separately seeded built-in agent
|
||
`position-roundtable-intent` by passing `intent_agent_id` to the shared
|
||
`/api/intent/init` and `/api/intent/stream` endpoints. Its request lifecycle,
|
||
files, and SSE protocol match `roundtable-intent`, but its cap is five
|
||
**user-assistance rounds**. A single round may emit several independent
|
||
`ask_clarification` tool calls; those calls are separate cards, not one card
|
||
containing several numbered questions. Count that as one round by the
|
||
tool-calling AI message, never by card count. Topic-only requests must request
|
||
all material missing decisions as separate cards rather than being treated as a
|
||
resolved intent; after the fifth set is answered it must emit completed final
|
||
JSON without unresolved placeholders. The required arrays are
|
||
`coreGoals`, `riskWarnings`, `keyPoints`, and `strategicSignificance`. Do not
|
||
treat a fresh dedicated-agent `ask_clarification` as stale: it must win over
|
||
a same-run premature `[INTENT_READY]` and leave the UI waiting for the user.
|
||
Conversely, the prior turn’s historical clarification must not block the user
|
||
reply from completing. Do not
|
||
change `roundtable-intent`'s `{objective,constraints,assumptions}` contract for
|
||
this feature. The position router accepts the new schema for activation and
|
||
injects all four sections into downstream position-node and closing-agent
|
||
context; legacy stored summaries are mapped read-only for compatibility.
|
||
`/api/intent/init` must create the empty Step-1 thread through
|
||
`multi_agent._create_thread_direct`, exactly like multi-agent initialization;
|
||
do not restore the old HTTP loopback to `/api/threads`, because deployed
|
||
Gateway ports/proxies may make the automatic page initialization fail while
|
||
later manual turns still work. If checkpoint creation fails or the parallel seat
|
||
task is cancelled, the helper compensates by deleting any partial checkpoint and
|
||
metadata row (the cancellation branch runs under `asyncio.shield` so a second
|
||
cancellation cannot abort it half-way). Streaming stays on the existing loopback
|
||
path.
|
||
|
||
`ThreadMetaStore.create` never issues a post-commit `session.refresh()`. Every
|
||
`ThreadMetaRow` column is assigned before commit and the session factory uses
|
||
`expire_on_commit=False`, so the returned dict is built from the in-memory row.
|
||
The removed read-back ran on a *different* pooled connection than the INSERT,
|
||
which a MySQL read/write-splitting endpoint can route to a lagging replica — the
|
||
row comes back empty and SQLAlchemy raises `Could not refresh instance`, surfacing
|
||
as `500 "Failed to create thread"`. Concurrency makes that far more likely, so a
|
||
single-user deployment never sees it. No caller uses the return value, and all
|
||
seven call sites (`multi_agent._create_thread_direct`, `services.start_run`,
|
||
`routers/threads.create_thread`, `roundtable_inprocess_gateway`,
|
||
`scheduler/service`, `thread_shares`, `authz`) benefit. Keep new `create`
|
||
implementations read-back-free; `tests/test_position_roundtable_sessions.py::
|
||
test_thread_meta_create_never_reads_back_after_commit` guards this.
|
||
|
||
### Harness / App Split
|
||
|
||
The backend is split into two layers with a strict dependency direction:
|
||
|
||
- **Harness** (`packages/harness/deerflow/`): Publishable agent framework package (`deerflow-harness`). Import prefix: `deerflow.*`. Contains agent orchestration, tools, sandbox, models, MCP, skills, config ? everything needed to build and run agents.
|
||
- **App** (`app/`): Unpublished application code. Import prefix: `app.*`. Contains the FastAPI Gateway API and IM channel integrations (Feishu, Slack, Telegram, DingTalk).
|
||
|
||
**Dependency rule**: App imports deerflow, but deerflow never imports app. This boundary is enforced by `tests/test_harness_boundary.py` which runs in CI.
|
||
|
||
### Report Collaboration (AgentScope, in progress)
|
||
|
||
Independent of Workflow Studio and the existing roundtable. Plan: repo-root `docs/agentscope-多智能体报告协作工作台-后端实施计划.md`（`RC-BE-000`～`018` 已落地，下一票 `RC-BE-019`）。Spike baseline: `docs/AGENTSCOPE_BASELINE.md`.
|
||
|
||
- **Config**: `config.yaml → report_collaboration` (`deerflow.config.report_collaboration_config.ReportCollaborationConfig`). Default **off** (`enabled` / `worker_enabled`).
|
||
- **Contracts**: `app/report_collaboration/contracts/` — snake_case Pydantic models matching `frontend-web/src/report-collaboration/api/types.ts`. SSE envelope uses camelCase (`eventId` / `sessionId` / `runId`). `node_run_id` is always `${node_id}-attempt{attempt}`.
|
||
- **Persistence**: `deerflow.persistence.report_collaboration` (`report_collaboration_*` tables, migrations `20260905_01` / `20260906_01` / `20260907_01`). SQL + memory stores; one active run per session via `active_dedupe_key`; command/message/run/version idempotency keys refuse cross-session reuse; event `seq` taken from the run row. Session delete removes children explicitly.
|
||
- **Gateway**: `/api/report-collaboration` (`app/gateway/routers/report_collaboration.py`). `enabled=false` → 503. Pre-run intent dialogue uses the separately seeded built-in agent `report-collaboration-intent` through the shared real `/api/intent/init` + `/api/intent/stream` LangGraph thread path. The browser consumes upstream `messages-tuple` frames incrementally (text, provider reasoning and tool steps) and renders them in the normal `MessageList`; this path must never be replaced with `PlannerAgent.ainvoke()` or another wait-for-completion wrapper. `POST /sessions/{id}/intent-turns` receives only the completed streamed turn for durable mirroring; it does not invoke an intent model. An `INTENT_READY` terminal summary is converted to the report requirement and automatically fulfilled through `PlanProposalService`, with no first-plan confirmation click. The old pre-run `/messages` path returns `USE_STREAMING_INTENT`. `POST /sessions/{id}/plan-requests` remains for regeneration/compatibility. Workflow design after intent confirmation is LLM-generated, assembled against the safe agent catalog and validated (acyclic DAG, artifact types, allowlisted tools). Selecting a plan and starting a run remain independent commands. `POST /runs` freezes role models, report-structure template and run budget. Rewrite / apply / restore are live (`reporting/`). `GET /runs/{id}/stream` is a read-only durable SSE tail (`execution/live_hub.py`): persist then publish, replay + `after_seq` / `Last-Event-ID`, heartbeat comments, token-delta coalesce before seq assignment. Disconnect does **not** cancel the run. Lifespan mounts `ReportCollaborationDispatcher` only when `enabled` **and** `worker_enabled` are both true; default both false.
|
||
- **Runtime**: AgentScope adapters live in `app/report_collaboration/agentscope_runtime/` (not the publishable harness). `ConversationReplica` maps AgentScope 2.0.5 events 1:1 onto collaboration SSE types; TeamSay is `HINT_BLOCK`. `DeerFlowChatModelAdapter` wraps `create_chat_model` (no second Provider config); role models are frozen onto node `agent_snapshot` at `POST /runs` and missing/incapable models fail with 422 before the run is created. Fallback is recorded on the snapshot plus a durable `heartbeat` audit event. Nodes complete only after a validated artifact (`app/report_collaboration/execution/completion.py`). `TaskLedger` (`execution/ledger.py`) owns ready waves, attempts and terminal node status; AgentScope / coordinator suggestions never pick the next node. A writer-bearing run is `completed` only after `ReportFinalizer` publishes a formal version. `WaveExecutor` + `QualityRoleKernel` (`execution/quality_roles.py`) remain for fixtures: Verifier marks stale/contested claims, Analyst/Writer only cite supported IDs, Reviewer `revise` replans only the target write node plus review. Production worker injects `AgentScopeMemberKernel` (`agentscope_runtime/member_kernel.py`) with DeerFlow tools and per-run budget. `QualityGate` (`execution/quality_gate.py`) is the claim-level gate: conflicts/missing units, unverified facts, coverage gaps, and unmapped review targets fail before `validated`; `ArtifactBoard.supersede_previous` hides old attempts from downstream. `ReportCollaborationLiveHub` (`execution/live_hub.py`) fans out persisted envelopes to SSE viewers; `PublishingReportCollaborationStore` sanitizes secrets/privacy/hidden reasoning without truncating visible tool results, and writes redaction + event audits. Dispatcher (`execution/dispatcher.py`) atomically claims queued/recovering/expired-lease runs, heartbeats via `renew_lease`, two-phase cancels (`cancel_requested` → `finalize_cancel`), and recovers orphans (`execution/recovery.py`): keep completed work, mark leftover in-flight `interrupted`/`superseded`, open a new `planned` attempt. Late artifacts after cancel/reclaim are `superseded` and must not change terminal run status. Tests drive `dispatch_once()`; do not rely on the background loop. Per-run budgets (`security/budget.py`) fail-closed; per-user active-run caps are enforced at `create_run`.
|
||
- **Quality eval**: `app/report_collaboration/eval/` (`RC-BE-018`) — 20 annotated tasks (policy/market/enterprise/event/comparison ×4) with required angles, key facts, forbidden fabrications, templates and review rules. Deterministic scorer (no LLM judge) gates citation/angle/template/directed-mod; gold reports must pass, canned failures (including a roundtable-style essay) must fail and stay reproducible. Recommended runtime knobs live in `eval/thresholds.py` and match current `ReportCollaborationConfig` defaults (`worker_enabled` remains false). Tests: `tests/test_report_collaboration_eval.py`.
|
||
- **Runtime intervention**: `app/report_collaboration/interventions/` owns run-time natural-language commands. `RuntimeIntentRouterAgent` uses explicit rules first and a tool-free coordinator-model fallback only for ambiguity; its Pydantic decision cannot dispatch or mutate anything. `ImpactAnalyzer` validates server-side node/agent ownership, resolves angle-specific lineage, and enforces confidence/cost/parallel-branch confirmation (`intent_confirmation_threshold`, default `0.8`). `RuntimeCommandService` mirrors the human message, emits frozen command events, CAS-claims accepted commands, and applies them only through `TaskLedger` safe points. Rework persists command overlays, supersedes affected attempts/artifacts, creates monotonic attempts, and prevents late output from entering downstream. Waiting-node answers resume only those nodes. `rerun_node` without a legal target does not expand to the whole graph; requirement patches apply only after rework succeeds; stuck `executing` commands are reclaimed on takeover or lease timeout; `command.cancelled` is a durable SSE event. In-run `POST /sessions/{id}/messages` returns 409 (`RUN_ACTIVE`). Formal rewrite candidates and version apply/restore live in `reporting/` (`RC-BE-015`).
|
||
- **Dependency**: optional extra `report-collaboration` → `agentscope==2.0.5`. Do not add AgentScope to the harness `pyproject.toml`.
|
||
|
||
Tests: `tests/test_report_collaboration_*.py`.
|
||
|
||
### Workflow Studio
|
||
|
||
Coze Studio is canvas-only; ChatDev is reference-only. Plan + status per phase: `docs/WORKFLOW_STUDIO_BACKEND_DEV_ZH.md`; decisions: `docs/adr/0001-workflow-studio-runtime.md`; contracts: `docs/WORKFLOW_SCHEMA_V1_ZH.md` + `docs/WORKFLOW_SSE_V1_ZH.md` + `docs/WORKFLOW_OPENAPI_DRAFT_V1.yaml`. Master switch + all limits: `config.yaml` → `workflows` (`deerflow.config.workflow_config.WorkflowConfig`). `workflows.agent_recursion_limit` defaults to 250: LangGraph counts each model→tool research exchange as about two super-steps, so workflow agent/skill nodes must not fall back to a chat-sized 60-step limit. In the local Studio proxy, keep Rsbuild `server.compress: false`; dev gzip buffers small SSE frames and makes live planning/run output arrive only at completion despite the backend's `no-transform` / `X-Accel-Buffering: no` headers. The composer’s optional `modelName` must be validated against `AppConfig.models`: it selects the controller model and is copied only into newly generated candidate agent configs (or the projected direct-agent task), never injected into an already-confirmed proposal/run.
|
||
|
||
**Execution kernel is a custom DAG scheduler, not a compiled LangGraph.** LangGraph stays the kernel *inside* an `agent`/`skill` node. The reason is single-source-of-truth recovery: a re-leased or resumed run replays completed node outputs from `workflow_node_runs` plus the `workflow_run_events` log, so a second (checkpoint) copy of workflow-level state would have no defensible tie-breaker. See the ADR's phase-2 amendment.
|
||
|
||
**Harness** (`packages/harness/deerflow/workflows/`, never imports `app.*`):
|
||
- `schemas` / `events` / `errors` / `ports` / `validator` / `samples` — frozen v1 contracts.
|
||
- `expressions.py` — `{{ inputs.x }}` / `{{ nodes.n.data.y }}` templates and conditions resolved by path walking + a small comparator set. **No `eval`.** Roots are restricted to `inputs` / `nodes` / `run` / `loop` / `env`.
|
||
- `runtime/engine.py` — the scheduler. Every edge is `pending` → `active` (source chose this port) or `pruned`; a node is ready when its non-back incoming edges are all resolved and at least one is `active`, and is skipped when all are `pruned` (skips propagate). That one rule covers fan-out, condition branching, and merge joins with no topological pre-pass. The `loop` node's `continue` back edge is the only permitted cycle; re-entry clears the body's results and invalidates its node runs.
|
||
- `runtime/{sink,sql_runner,code_runner,context}.py`, `nodes/*` (15 executors, including deterministic `evidence_normalizer` and durable `deep_research_write`), `security/{http_policy,sql_policy,secret_box}.py`.
|
||
|
||
**Persistence**: `workflow_definitions` / `workflow_versions` (migration `20260828_01`), `workflow_runs` / `workflow_node_runs` / `workflow_run_events` / `workflow_run_artifacts` (`20260828_02`), `workflow_data_sources` (`20260828_03`), and short-lived-but-durable `workflow_planning_sessions` / `workflow_planning_proposals` (`20260831_01`). Stores under `deerflow.persistence.workflow_{runs,events,data_sources,planning}` with SQL + in-memory implementations. Because planning proposal rows have a scalar FK rather than an ORM relationship, `SqlWorkflowPlanningStore.create_session` must flush the parent session before adding its children; otherwise SQLite can insert the child first and return a 500. A planning proposal is distinct from a run: its graph can be edited with optimistic `graph_revision` checks, but it becomes executable only once selected and atomically bound to one run.
|
||
|
||
**App layer**: `workflow_dispatcher.py` (lease scan → atomic claim; an expired lease is how a dead worker's run is reclaimed, so every worker can run the loop), `workflow_executor.py` (one lease's lifecycle: replay → engine under a run timeout with heartbeat + cancel watcher → exactly one terminal transition; also runs subworkflows inline, depth ≤ 3), `workflow_retention.py` (hourly sweep of `workflows.retention.event_retention_days` / `run_retention_days`), `workflow_live_hub.py` (in-process SSE fan-out), `workflow_agent_runner.py` (agent/skill nodes → lead agent; `disable_tools=True` means an empty bound-tool set for control-plane callers), `workflow_deep_research_adapter.py` (Evidence Pack → selected Deep Research sources/job/event projection; no internal HTTP call), and `workflow_proposal_planner.py` (calls the dedicated `workflow-planner` controller over visible-agent metadata, then constructs safe candidate graphs server-side). `workflow-planner` is a separate Workflow Studio control-plane agent; never reuse or alias `roundtable-coordinator` for workflow planning, and never let it dispatch a run, call research tools, or return arbitrary graph JSON. It always runs with tools and model thinking disabled.
|
||
|
||
**Gateway routes** (register static prefixes before `workflows.router`'s `/{workflow_id}` catch-alls): `/api/workflows` CRUD/draft/validate/publish/copy, `/api/workflows/node-types` + `/resources/*`, `/api/workflows/{id}/planning-sessions{/stream}` + `/api/workflows/planning-sessions/{session_id}{,/proposals/{proposal_id},/proposals/{proposal_id}/graph}`, `/api/workflows/{id}/runs` + `/api/workflows/runs/{run_id}{,/events,/stream,/cancel,/resume,/retry,/feedback,/artifacts}`, `/api/workflows/data-sources` (+ `/{id}/introspect`, `/{id}/validate-query`), `/api/workflows/embed/{tickets,verify}`, `/api/workflow_api/*` Coze compat. The planning `/stream` endpoint sends product-safe real controller milestones (`catalog`, `controller`, `selection`, `assembly`, `validation`, `complete`) then a final `result` session; never stream agent catalog descriptions or raw planner JSON. Formal runs accept the selected `planningSessionId` and `proposalId`; creating the same pair again returns the already-bound run rather than launching another execution.
|
||
|
||
**Guarantees worth not breaking**: events are persisted *then* published, and the DB is authoritative for `seq` ordering (live frames at or below the replay cursor are dropped and re-read on the next poll, so a reconnect sees no gap; `Last-Event-ID` wins over `?after=`); a run reaching a terminal status always gets a terminal event, synthesised if necessary, or SSE clients hang; `resumeToken` appears only in the `run.awaiting_input` event, never in the run view; `POST /retry` on a `failed` run copies completed node outputs onto a new run marked `retry_of_run_id` so the engine resumes from the failed node; `POST /feedback` never mutates its source run—on a terminal acyclic run it may target only a completed `agent`/`skill`, reuses completed nodes outside that node's downstream closure, records those `reusedNodeIds` alongside the feedback summary, and puts the bounded feedback only in `RunContext.env.nodeFeedback[target]` before creating a revision run; data-source secrets are write-only (encrypted via `WORKFLOW_SECRET_KEY`, decrypted only inside the executor, reads expose `masked_target`); the `http` node denies private/loopback/link-local targets by default (SSRF), `sql_read` allows one read-only statement with bound parameters, and the `code` node is **off by default** because its runner is a hardened subprocess rather than a container.
|
||
|
||
HTTP interface resources persist a non-secret base_url plus an allowed_methods allowlist (migration
|
||
20260829_01); request headers stay only in the encrypted secret blob. The resource catalog may return
|
||
the address and methods to the owning user, but must never return headers or any decrypted secret. The
|
||
executor receives HTTP metadata only through its data-source resolver.
|
||
|
||
The Workflow Studio agent/skill resource catalog is a read-only display projection over the existing
|
||
agent and skill ownership stores. It returns stable `agentId` / `skillId`, an agent's cached skill names,
|
||
and a `createdBy` display label. Resolve human labels from the immutable owner id at request time and
|
||
fall back to that id if lookup fails; built-in records must return `系统内置`. Do not create a parallel
|
||
creator table or accept creator data from the browser.
|
||
|
||
`evidence_normalizer` is a deterministic data node, not an LLM: it consumes only configured/upstream
|
||
node results and emits a bounded `evidencePack` of claims with node provenance, role labels and gaps.
|
||
The planner inserts it between parallel research roles and the synthesis Agent. It must not treat
|
||
source text as verified fact or fetch any external resource, and its explicit `sources[].nodeId` values
|
||
must stay upstream-only under `validate_workflow_graph`.
|
||
|
||
`deep_research_write` is a durable report node, not an Agent prompt shortcut. It accepts only an
|
||
Evidence Pack, derives a session identity from the workflow run/node plus the topic, evidence snapshot and writing config, writes
|
||
the pack as selected `other` source rows with upstream-node provenance, and queues/reuses the existing
|
||
Deep Research `regenerate` job. Its live `report_delta` and persisted `report_chunk` frames are
|
||
deduplicated before becoming workflow `node.output.delta` events; only report prose is forwarded (never
|
||
planning/reasoning deltas). Workflow cancellation requests cancellation on that same job. Do not make it
|
||
self-call `/api/deep-research/*`, use Deep Research `multi_agent` inside this node, or treat Evidence
|
||
Pack claims as verified facts. The parallel-research candidate grants the node an explicit 1800-second
|
||
timeout; deployments that override `workflows.node_timeout_seconds` must keep it at least that high.
|
||
|
||
Tests: `tests/test_workflow_schemas.py`, `tests/test_workflow_definitions.py`, `tests/test_workflow_engine.py`, `tests/test_workflow_runs.py`.
|
||
|
||
**Import conventions**:
|
||
```python
|
||
# Harness internal
|
||
from deerflow.agents import make_lead_agent
|
||
from deerflow.models import create_chat_model
|
||
|
||
# App internal
|
||
from app.gateway.app import app
|
||
from app.channels.service import start_channel_service
|
||
|
||
# App ? Harness (allowed)
|
||
from deerflow.config import get_app_config
|
||
|
||
# Harness ? App (FORBIDDEN ? enforced by test_harness_boundary.py)
|
||
# from app.gateway.routers.uploads import ... # ? will fail CI
|
||
```
|
||
|
||
### Agent System
|
||
|
||
**Gallery chat link.** An agent's `config.yaml` may store `chat_link` (`url`, `open_mode` of `blank`/`self`, and light-app-shaped `route_params`). The gallery 「对话」 button opens that resolved URL; an empty url keeps the in-app chat. The list endpoint reads the mirrored `agent_extras.chat_link` column (nullable JSON, added by the startup column sweep) instead of parsing every config file. Create/update and directory backfill write the mirror; a skills-only `set_extras` leaves an existing link in place.
|
||
|
||
**Lead Agent** (`packages/harness/deerflow/agents/lead_agent/agent.py`):
|
||
- Entry point: `make_lead_agent(config: RunnableConfig)` registered in `langgraph.json`
|
||
- Dynamic model selection via `create_chat_model()` with thinking/vision support
|
||
- Tools loaded via `get_available_tools()` - combines sandbox, built-in, MCP, community, and subagent tools
|
||
- System prompt generated by `apply_prompt_template()` with skills, memory, and subagent instructions
|
||
|
||
**agentfx built-in analyst + report convert skills.** `agentfx-analyst` (「任务研判报告助手」) is seeded from `app/gateway/routers/_agentfx_seed_assets/` before `_sync_legacy_agents`. After mandatory `web_search`, the agent writes each tool JSON to `/mnt/user-data/workspace/hits/qN.json`, writes task metadata as JSON data to the fixed `/mnt/user-data/workspace/task-context.json`, and runs **one constant command** — `task-report-build` (group `all`) — to fill the 14-category `task-reports.json` for the dashboard slots; untrusted task text never enters the shell command. The four per-page skills `task-report-{enemy,our,env,judge}` remain for re-running a single dashboard page. All five are thin CLIs over the shared stdlib engine `skills/public/task-report-lib/convert_lib.py` (no SKILL.md on the lib, so it never enters the skill list). The engine emits the exact category-specific shapes consumed by `sq-report.ts`, marks evidence-derived content with `[检索证据]`, rejects cross-task partial merges, and atomically writes Unicode JSON. The model must not `write_file` that JSON. Because the sandbox maps `/mnt/user-data/...` onto a host dir, `discover_user_data_root` + `resolve_sandbox_path` map virtual paths to the current thread's literal `user-data` root; hits, task context and output are canonically confined there, including symlink and Windows drive checks. `resolve_hit_files` may try confined mapped/relative/basename candidates and reports every miss so the agent never has to write a debug script. Assemble of the 14 categories is the same function (`assemble_records`); the model must not copy or merge files. If the model is interrupted after convert, `task-reports.json` is already in outputs and the frontend import card also scans `GET /api/threads/{id}/artifacts` so入库 still works without `present_files`. Import remains `task-report-import`. Existing agent/skill directories are never overwritten. Tests: `tests/test_task_report_convert.py`, `tests/test_agentfx_seed.py`.
|
||
|
||
**Forced-research built-in agent.** `forced-research-responder` (「强制检索输出助手」) is seeded from `app/gateway/routers/_forced_research_seed_assets/` before `_sync_legacy_agents`. `ForcedResearchMiddleware` is mounted only for this stable id. On every visible user turn it requires a successful information-gathering tool result before the model may finish; `tool_choice=required` is backed by an `after_model -> model` retry for gateways that ignore tool choice, with an eight-attempt failure cap that permits only an honest failure response. Skill discovery, reading `SKILL.md`, and writing a helper script do not satisfy the gate; web/knowledge/MCP results, non-skill uploaded-file reads, or successful Python skill-script execution do. Tool filtering removes `present_files`, `skill_manage`, task/schedule/memory/clarification surfaces. The middleware permits `write_file`/`str_replace` only for `/mnt/user-data/workspace/*.py` and permits `bash` only for `.py` execution without redirection/document paths, so skills can run scripts but the agent cannot create Markdown/document artifacts or write `outputs`. Tests: `tests/test_forced_research_agent.py`.
|
||
|
||
**Elasticsearch offline operations built-in agent.** `elasticsearch-offline-ops` (「Elasticsearch 离线运维助手」) and its same-name public skill are seeded from `app/gateway/routers/_elasticsearch_offline_ops_seed_assets/` before `_sync_legacy_agents`. Seeding is missing-directory-only so administrator edits survive restarts. The agent intentionally has `file:read`, `file:write`, and `bash` but no web group: air-gapped guidance comes from the shipped version-parameterized deployment and troubleshooting references. Its SOUL requires the exact ES version/OS/architecture/topology before final commands, keeps Elasticsearch security/TLS enabled, separates the DeerFlow sandbox from the target ES hosts, and makes read-only evidence collection precede changes. `scripts/collect_diagnostics.sh` gathers Linux/service/log and authenticated cluster evidence without mutations or `--insecure`; credentials come from operator-owned netrc/header files rather than command arguments or an unrestricted curl config. `scripts/analyze_diagnostics.py` is a stdlib-only heuristic that emits matched evidence and a next read-only check, or `insufficient_evidence`; it never claims an automatic root cause. Tests: `tests/test_elasticsearch_offline_ops_seed.py`, `tests/test_seed_assets.py`.
|
||
|
||
**Strategy-selection functional agents.** `app/gateway/routers/_strategy_selection_seed.py` seeds nine stable folder-backed agents from `_strategy_selection_seed_assets/` before `_sync_legacy_agents`: intent, governance/diplomacy/economy/security/public-opinion/law-policy angles, judge, and solution editor. Seeding is missing-directory-only and must never overwrite administrator changes. The global `strategy_agent_roles` registry classifies them independently of projects: intent/judge/solution are fixed, unique roles that may be configured but never added, disabled, reordered, or deleted; angle roles are a shared, ordered catalog that administrators may add, edit, disable (soft delete), restore, and reorder through `/api/strategy-agents`. Public `GET /api/strategy-agents` returns enabled identity/description metadata only; `include_disabled=true` or `include_config=true` is admin-only. Admin writes are `POST /api/strategy-agents/angles`, `PUT /api/strategy-agents/{id}`, and `DELETE /api/strategy-agents/{id}`; DELETE retains the folder-backed agent and role row to preserve historical references. Dynamic angles remain built-in folder-backed agents under the reserved `strategy-selection-angle-` id namespace, with `web` + `file:read`; a blank SOUL on create receives a safe role template. All strategy skill writes validate that names exist, are active, and are globally visible; generic agent CRUD cannot create or hard-delete reserved strategy IDs, and parallel execution rejects disabled roles. Angle configs reuse `policy-knowledge`; intent reuses `intent-clarification-policy`; the judge has no retrieval skill; the solution editor gets only `file:write`. Keep the SOUL contracts neutral and evidence-aware, with explicit de-escalation/civilian-protection framing and hard bans on operational combat guidance and targeted political manipulation. Tests: `tests/test_strategy_selection_seed.py`, `tests/test_strategy_agent_management.py`.
|
||
|
||
Only `strategy-selection-intent` has a clarification reply surface in the strategy workbench. `parallel_agents` must exclude `ask_clarification` for every other `strategy-selection-*` role and inject a direct-delivery guard: angle, judge, and solution stages use conservative assumptions, lower confidence/score where needed, list gaps under data limitations, and still finish in one pass. The judge brief repeats this rule at runtime because existing folder-backed agent SOUL files may be administrator-customized and are intentionally never overwritten. If a legacy backend nevertheless emits a judge clarification, the frontend treats it as a bounded repair attempt rather than asking the user to repeat the same manual run. Malformed or incomplete evaluation JSON receives the same bounded repair path; prior attempts are replaced in the live panel, never concatenated into duplicate final content.
|
||
|
||
`POST /api/parallel-agents/run/stream` must remain observable during silent skill calls. Each running seat emits an `agent_progress` heartbeat every five seconds with the latest translated tool/reasoning/body phase and monotonic elapsed seconds; an upstream iterator that produces no frame for 180 seconds is closed and that seat emits `agent_error`, while the other seats continue. Set the terminal flag before queuing `agent_done` / clarification so a late heartbeat cannot reopen a completed card. The strategy workbench renders the intent Q&A history, readable skill execution steps and live heartbeat on the affected agent card; judge and synthesis traces use the same step translation. Tests: `tests/test_roundtable_model_fallback.py` and frontend `src/strategy-selection/lib.test.ts`.
|
||
|
||
**ThreadState** (`packages/harness/deerflow/agents/thread_state.py`):
|
||
- Extends `AgentState` with: `sandbox`, `thread_data`, `title`, `artifacts`, `todos`, `uploaded_files`, `viewed_images`
|
||
- Uses custom reducers: `merge_artifacts` (deduplicate), `merge_viewed_images` (merge/clear)
|
||
|
||
**Runtime Configuration** (via `config.configurable`):
|
||
- `thinking_enabled` - Enable model's extended thinking
|
||
- `model_name` - Select specific LLM model
|
||
- `is_plan_mode` - Enable TodoList middleware
|
||
- `subagent_enabled` - Enable task delegation tool
|
||
|
||
### Middleware Chain
|
||
|
||
Lead-agent middlewares are assembled in strict append order across `packages/harness/deerflow/agents/middlewares/tool_error_handling_middleware.py` (`build_lead_runtime_middlewares`) and `packages/harness/deerflow/agents/lead_agent/agent.py` (`_build_middlewares`):
|
||
|
||
1. **ThreadDataMiddleware** - Creates per-thread directories under the creator's stable isolation scope (`backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/{workspace,uploads,outputs}`); resolves persisted threads through `resolve_path_user_id(thread_id)` (falls back to `get_effective_user_id()` / `"default"` when metadata is unavailable); Web UI thread deletion now follows LangGraph thread removal with Gateway cleanup of the local thread directory
|
||
2. **UploadsMiddleware** - Tracks and injects newly uploaded files into conversation
|
||
3. **SandboxMiddleware** - Acquires sandbox, stores `sandbox_id` in state
|
||
4. **DanglingToolCallMiddleware** - Injects placeholder ToolMessages for AIMessage tool_calls that lack responses (e.g., due to user interruption), including raw provider tool-call payloads preserved only in `additional_kwargs["tool_calls"]`
|
||
5. **LLMErrorHandlingMiddleware** - Normalizes provider/model invocation failures into recoverable assistant-facing errors before later middleware/tool stages run
|
||
6. **GuardrailMiddleware** - Pre-tool-call authorization via pluggable `GuardrailProvider` protocol (optional, if `guardrails.enabled` in config). Evaluates each tool call and returns error ToolMessage on deny. Three provider options: built-in `AllowlistProvider` (zero deps), OAP policy providers (e.g. `aport-agent-guardrails`), or custom providers. See [docs/GUARDRAILS.md](docs/GUARDRAILS.md) for setup, usage, and how to implement a provider.
|
||
7. **SandboxAuditMiddleware** - Audits sandboxed shell/file operations for security logging before tool execution continues
|
||
8. **ToolErrorHandlingMiddleware** - Converts tool exceptions into error `ToolMessage`s so the run can continue instead of aborting
|
||
9. **SummarizationMiddleware** (`DeerFlowSummarizationMiddleware`) - Context reduction when approaching token limits (optional, if enabled). **Non-destructive**: overrides the stock `before_model` (which persists a `RemoveMessage(REMOVE_ALL_MESSAGES)`) to a no-op and instead compresses **transiently** in `wrap_model_call` ? the model request is replaced with `[summary, *recent]` while the full original conversation stays in state (so the frontend keeps rendering the user's real Q&A). The `summary` message is `hide_from_ui` and never persisted; summaries are cached **per-thread in a process-wide, thread-safe LRU** (`_GLOBAL_SUMMARY_CACHE`, bounded to 512 threads, sticky boundary). The cache is module-level — **not** on the middleware instance — because `make_lead_agent` rebuilds a fresh middleware on every run; an instance cache would be cold each turn and force a blocking summary LLM call on every turn of a long conversation. Keyed by `thread_id` (resolved like `ThreadDataMiddleware`), so a summary computed on one turn is reused on the next, paying for an LLM call only on the first threshold crossing or a genuine re-compaction. **Threshold-first gating**: `_plan_compression` returns passthrough (no compression at all) whenever the *full* current conversation is still under the trigger — the cache/summary substitution only engages once the real context crosses the configured trigger. **Stickiness headroom** (`_apply_stickiness_headroom`, `_KEEP_HEADROOM_FRACTION=0.75`): after the `keep` policy picks a cutoff, the kept suffix is forced below ~75% of *every* trigger before summarizing. Without it a `keep` in *messages* (e.g. 20) can itself exceed a *token* trigger (e.g. 60000) when recent messages are large (tool outputs / pasted docs), leaving the "compressed" context still over the trigger so it re-summarizes on every follow-up question; the headroom makes the sticky boundary actually stick (re-compaction only after genuinely new content). When the genuine (blocking) summarize path runs, it emits a `{"type":"context_compacting","message":"正在压缩上下文…"}` event on LangGraph's `custom` stream channel (the cheap reuse path stays silent); the chat frontend's `onCustomEvent` renders it as a transient toast. Tests: `tests/test_summarization_middleware.py`
|
||
10. **TodoListMiddleware** - Task tracking with `write_todos` tool (optional, if plan_mode)
|
||
11. **TokenUsageMiddleware** - Records token usage metrics when token tracking is enabled (optional)
|
||
12. **TitleMiddleware** - Two-phase auto-title (never shows "untitled"): `before_model` sets an instant question-derived *provisional* title the moment the first user message arrives (no LLM, zero latency), then `after_model` asynchronously generates the final LLM title and replaces it (tracked via the `title_provisional` state flag). Normalizes structured message content before prompting the title model
|
||
13. **MemoryMiddleware** - Memory V2: `wrap_model_call` injects Hindsight recall transiently; `after_agent` auto-retains the turn to Hindsight
|
||
14. **ViewImageMiddleware** - Injects base64 image data before LLM call (conditional on vision support)
|
||
15. **DeferredToolFilterMiddleware** - Hides deferred tool schemas from the bound model until tool search is enabled (optional)
|
||
16. **SubagentLimitMiddleware** - Truncates excess `task` tool calls from model response to enforce `MAX_CONCURRENT_SUBAGENTS` limit (optional, if `subagent_enabled`)
|
||
17. **LoopDetectionMiddleware** - Detects repeated tool-call loops; hard-stop responses clear both structured `tool_calls` and raw provider tool-call metadata before forcing a final text answer
|
||
18. **ContextWindowGuardMiddleware** - Final transient request-size guard; enforces the selected model's `max_input_tokens` after every context/tool injection
|
||
19. **ClarificationMiddleware** - Intercepts `ask_clarification` tool calls, interrupts via `Command(goto=END)` (must be last)
|
||
8a. **ToolMetricsMiddleware** - Appended immediately after `ToolErrorHandlingMiddleware` in `_build_runtime_middlewares`, so it sits *deepest* in the `wrap_tool_call` chain. Records one `tool_call_metrics` row per tool/skill/MCP invocation ? timing duration and bucketing failures into a coarse `error_category` (rate_limited / ip_blocked / auth_error / timeout / network_error / not_found / empty_result / tool_error). It catches both raised exceptions (before #8 converts them) and *self-handled* errors returned as error `ToolMessage`s (search tools that swallow HTTP 429/403). It also captures the targeted Agent Skill into `tool_call_metrics.skill_name`: the lead agent invokes a skill by `read_file`-ing its `/mnt/skills/<category>/<name>/SKILL.md` path (per the `<skill_system>` prompt block ? there is no dedicated "use skill" tool), so `skill_name` is recovered by scanning every tool call's string arguments for the `skills/{public,custom}/{name}/` path shape (`_SKILL_PATH_RE`), plus the explicit `name` argument of `skill_view`/`skill_manage`/`skill_archive`. `skill_view`-specific success detection handles its swallowed failures (`Skill 'x' not found.` returned as a plain string rather than raised). Regression coverage: `tests/test_tool_metrics_middleware.py`. Writes are best-effort fire-and-forget. **Per-run skill merging**: `SqlToolMetricsStore.record` upserts skill rows by `(run_id, skill_name)` instead of inserting one row per call ? a skill touched by several tool calls / retries inside one run collapses to a single row. Any success in the run wins (a skill that failed then finally succeeded is recorded as one `success`); a skill that only ever failed yields one `error` row whose `error_message` accumulates every attempt's message; durations are summed. Non-skill tool calls, and skill calls with no `run_id`, keep one row each. The read?merge?write is guarded by a per-event-loop `asyncio.Lock`. Surfaced by the admin `GET /api/admin/tool-metrics` router (which has a `kind=skill` filter and a per-skill `/summary` breakdown).
|
||
9. **PromptPrefixMiddleware** - Injects the admin-configured prompt prefix into the model request only. The prefix is tagged onto the first human message's `additional_kwargs['prompt_prefix']` by the Gateway (`app.gateway.services`) and prepended to content here at call time, so the persisted/displayed message stays prefix-free
|
||
10. **SummarizationMiddleware** (`DeerFlowSummarizationMiddleware`) - Context reduction when approaching token limits (optional, if enabled). **Non-destructive**: overrides the stock `before_model` (which persists a `RemoveMessage(REMOVE_ALL_MESSAGES)`) to a no-op and instead compresses **transiently** in `wrap_model_call` ? the model request is replaced with `[summary, *recent]` while the full original conversation stays in state (so the frontend keeps rendering the user's real Q&A). The `summary` message is `hide_from_ui` and never persisted; summaries are cached **per-thread in a process-wide, thread-safe LRU** (`_GLOBAL_SUMMARY_CACHE`, bounded to 512 threads, sticky boundary). The cache is module-level — **not** on the middleware instance — because `make_lead_agent` rebuilds a fresh middleware on every run; an instance cache would be cold each turn and force a blocking summary LLM call on every turn of a long conversation. Keyed by `thread_id` (resolved like `ThreadDataMiddleware`), so a summary computed on one turn is reused on the next, paying for an LLM call only on the first threshold crossing or a genuine re-compaction. **Threshold-first gating**: `_plan_compression` returns passthrough (no compression at all) whenever the *full* current conversation is still under the trigger — the cache/summary substitution only engages once the real context crosses the configured trigger. **Stickiness headroom** (`_apply_stickiness_headroom`, `_KEEP_HEADROOM_FRACTION=0.75`): after the `keep` policy picks a cutoff, the kept suffix is forced below ~75% of *every* trigger before summarizing. Without it a `keep` in *messages* (e.g. 20) can itself exceed a *token* trigger (e.g. 60000) when recent messages are large (tool outputs / pasted docs), leaving the "compressed" context still over the trigger so it re-summarizes on every follow-up question; the headroom makes the sticky boundary actually stick (re-compaction only after genuinely new content). When the genuine (blocking) summarize path runs, it emits a `{"type":"context_compacting","message":"正在压缩上下文…"}` event on LangGraph's `custom` stream channel (the cheap reuse path stays silent); the chat frontend's `onCustomEvent` renders it as a transient toast. Tests: `tests/test_summarization_middleware.py`
|
||
11. **TodoListMiddleware** - Task tracking with `write_todos` tool (optional, if plan_mode)
|
||
12. **TokenUsageMiddleware** - Records token usage metrics when token tracking is enabled (optional)
|
||
13. **TitleMiddleware** - Two-phase auto-title (never shows "untitled"): `before_model` sets an instant question-derived *provisional* title the moment the first user message arrives (no LLM, zero latency), then `after_model` asynchronously generates the final LLM title and replaces it (tracked via the `title_provisional` state flag). Normalizes structured message content before prompting the title model
|
||
14. **MemoryMiddleware** - Queues conversations for async memory update (filters to user + final AI responses)
|
||
15. **ViewImageMiddleware** - Injects base64 image data before LLM call (conditional on vision support)
|
||
16. **DeferredToolFilterMiddleware** - Hides deferred tool schemas from the bound model until tool search is enabled (optional)
|
||
17. **SubagentLimitMiddleware** - Truncates excess `task` tool calls from model response to enforce `MAX_CONCURRENT_SUBAGENTS` limit (optional, if `subagent_enabled`)
|
||
18. **LoopDetectionMiddleware** - Detects repeated tool-call loops; hard-stop responses clear both structured `tool_calls` and raw provider tool-call metadata before forcing a final text answer
|
||
19. **ContextWindowGuardMiddleware** - Final transient request-size guard; enforces the selected model's `max_input_tokens` after every context/tool injection
|
||
20. **ClarificationMiddleware** - Intercepts `ask_clarification` tool calls, interrupts via `Command(goto=END)` (must be last)
|
||
|
||
### Configuration System
|
||
|
||
**Main Configuration** (`config.yaml`):
|
||
|
||
Setup: Copy `config.example.yaml` to `config.yaml` in the **project root** directory.
|
||
|
||
**Config Versioning**: `config.example.yaml` has a `config_version` field. On startup, `AppConfig.from_file()` compares user version vs example version and emits a warning if outdated. Missing `config_version` = version 0. Run `make config-upgrade` to auto-merge missing fields. When changing the config schema, bump `config_version` in `config.example.yaml`.
|
||
|
||
**Config Caching**: `get_app_config()` caches the parsed config, but automatically reloads it when the resolved config path changes or the file's mtime increases. This keeps Gateway and LangGraph reads aligned with `config.yaml` edits without requiring a manual process restart.
|
||
|
||
Configuration priority:
|
||
1. Explicit `config_path` argument
|
||
2. `DEER_FLOW_CONFIG_PATH` environment variable
|
||
3. `config.yaml` in current directory (backend/)
|
||
4. `config.yaml` in parent directory (project root - **recommended location**)
|
||
|
||
Config values starting with `$` are resolved as environment variables (e.g., `$OPENAI_API_KEY`).
|
||
`ModelConfig` also declares `use_responses_api` and `output_version` so OpenAI `/v1/responses` can be enabled explicitly while still using `langchain_openai:ChatOpenAI`.
|
||
`write_file_transport` defaults to `native` and remains available for backward
|
||
compatibility. For a gateway that streams ordinary text but buffers long tool
|
||
arguments, administrators should use Settings → File streaming compatibility.
|
||
That runtime switch applies `buffered_gateway` to every compatible configured
|
||
model from the next question without restarting the Gateway. The model still
|
||
selects the original `write_file`, narrowed to
|
||
path/description/append for that request. The adapter then starts a tool-free
|
||
ordinary-content body request inside the same logical ChatModel stream and
|
||
projects it onto canonical incremental `write_file` chunks. LangGraph and
|
||
ToolNode receive one original writer call; `begin_write_file` is neither exposed
|
||
nor executed in this mode. Only a normal terminal frame after at least one body
|
||
byte appends the boolean commit marker. Provider reasoning and a leading inline
|
||
`<think>` stay outside the file; empty bodies, truncation and unexpected tool
|
||
calls abort before sandbox I/O. Ordinary answers remain a single provider
|
||
request. Turning the switch off restores each model's configured/native
|
||
transport. Native models must remain byte-for-byte unchanged while the switch
|
||
is off. `artifact_renderer` and `text_bridge` exist for older configurations
|
||
and are not the recommended buffered-gateway transport.
|
||
|
||
**Extensions Configuration** (`extensions_config.json`):
|
||
|
||
MCP servers and skills are configured together in `extensions_config.json` in project root:
|
||
|
||
Configuration priority:
|
||
1. Explicit `config_path` argument
|
||
2. `DEER_FLOW_EXTENSIONS_CONFIG_PATH` environment variable
|
||
3. `extensions_config.json` in current directory (backend/)
|
||
4. `extensions_config.json` in parent directory (project root - **recommended location**)
|
||
|
||
### Gateway API (`app/gateway/`)
|
||
|
||
FastAPI application on port 8001 with health checks at `GET|HEAD /health`, `/api/health`, and `/api/langgraph/health`. These exact paths are always public in `auth_middleware.PUBLIC_HEALTH_PATHS`, including when authentication is enabled; reverse-proxy prefixes and trailing slashes are normalized before the whitelist check. Set `GATEWAY_ENABLE_DOCS=false` to disable `/docs`, `/redoc`, and `/openapi.json` in production (default: enabled). Regression coverage: `tests/test_health_auth_bypass.py`.
|
||
|
||
**Routers**:
|
||
|
||
| Router | Endpoints |
|
||
|--------|-----------|
|
||
| **Models** (`/api/models`) | `GET /` - list models; `GET /{name}` - model details |
|
||
| **MCP** (`/api/mcp`) | `GET /config` - get config; `PUT /config` - update config (saves to extensions_config.json) |
|
||
| **Skills** (`/api/skills`) | `GET /` - list skills; `GET /{name}` - details; `PUT /{name}` - update enabled; `POST /install` - install from .skill archive (accepts standard optional frontmatter like `version`, `author`, `compatibility`) |
|
||
| **Memory V2** (`/api/memory/v2`) | `GET /` - builtin data (USER.md/MEMORY.md); `POST/PUT/DELETE /entries` - add/replace/remove entry; `GET/PATCH/DELETE /config` - effective config + per-user overrides; `POST /search` - Hindsight recall/reflect proxy |
|
||
| **Agents** (`/api/agents`) | `GET /` - list built-in, owned, and published agents; `POST /` - create user-owned agent; `GET/PUT/DELETE /{id}` - read/update/delete by stable agent id; `GET /check` - validate non-empty display name (duplicates allowed) |
|
||
| **Skills** (`/api/skills`) | `GET /` - list skills (optional `search` fuzzy-name + `tag_id` filters; each item carries `tags`); `GET /{name}` - details; `PUT /{name}` - update enabled; `POST /install` - install from .skill archive (accepts standard optional frontmatter like `version`, `author`, `compatibility`) |
|
||
| **Skills** (`/api/skills`) | `GET /` - list skills (optional `search` fuzzy-name + `tag_id` filters; each item carries `tags` + owner-authored `detail`); `GET /{name}` - details; `PUT /{name}` - update enabled; `PUT /{name}/detail` - set the long-form `detail` text (custom skill: owner or admin; built-in/public: admin only ? materializes a `skills` row on demand); `POST /install` - install from .skill archive (accepts standard optional frontmatter like `version`, `author`, `compatibility`) |
|
||
| **Skills** (`/api/skills`) | `GET /` - list skills (optional `search` fuzzy-name + `tag_id` filters; each item carries `tags` + owner-authored `detail`); `GET /{name}` - details; `PUT /{name}` - update enabled; `PUT /{name}/detail` - set the long-form `detail` text (custom skill: owner or admin; built-in/public: admin only ? materializes a `skills` row on demand); `GET /{name}/content` - raw SKILL.md source, restricted to an admin or the publisher (custom-skill owner); other users get 403 and see only description + detail; `POST /install` - install from .skill archive (accepts standard optional frontmatter like `version`, `author`, `compatibility`) |
|
||
| **Skills** (`/api/skills`) | `GET /` - list skills (optional `search` fuzzy-name + repeatable `tag_ids` multi-select filter, OR-matched; each item carries `tags` + owner-authored `detail`); `GET /{name}` - details; `PUT /{name}` - update enabled; `PUT /{name}/detail` - set the long-form `detail` text (custom skill: owner or admin; built-in/public: admin only ? materializes a `skills` row on demand); `GET /{name}/content` - raw SKILL.md source, restricted to an admin or the publisher (custom-skill owner); other users get 403 and see only description + detail; `POST /install` - install from .skill archive (accepts standard optional frontmatter like `version`, `author`, `compatibility`) |
|
||
| **Memory** (`/api/memory`) | `GET /` - memory data; `POST /reload` - force reload; `GET /config` - config; `GET /status` - config + data |
|
||
| **Agents** (`/api/agents`) | `GET /` - list built-in, owned, and published agents (optional `search` + repeatable `tag_ids` multi-select filter, OR-matched; each item carries `tags` and `featured_order`); `POST /` - create user-owned agent; `GET/PUT/DELETE /{id}` - read/update/delete by stable agent id; `GET /check` - validate non-empty display name (duplicates allowed); `GET /selector` - homepage agent picker (single admin-curated list, visibility-filtered, sorted by `featured_order` asc); `PUT /{id}/featured` - **admin** sets/clears the homepage `featured_order` (must be `published=true` to feature; passing `order: null` clears unconditionally); `PUT /featured/order` - **admin** bulk-reorders the homepage list (body `{agent_ids: [...]}` ? indices become 0..N-1) |
|
||
| **Tags** (`/api/tags`) | `GET /` - list tags (optional `tag_type` filter; open to all); `POST /` - create tag (admin); `PUT /{id}` / `DELETE /{id}` - update/delete tag (admin); `GET /assignments/{target_type}/{target_id}` - tags on a resource; `PUT /assignments/{target_type}/{target_id}` - replace a resource's tags (admin). Tags are typed `agent`/`skill`/`scheduled_task`/`recommended_question`/`light_app` and may only tag resources of their own type. |
|
||
| **Upload policy** (`/api/upload-policy`) | Database-backed global singleton. `GET /` is authenticated; `PUT /` and `POST /reset` are admin-only atomic replacements with revision/audit metadata. Startup seeds the real database row with 20 files, 100 MiB total, documents/images 10 MiB each, spreadsheets 5 MiB, and large-file budgets 12k/32k chars. The validated row is also published to the harness runtime registry without introducing an `app.*` import. |
|
||
| **Uploads** (`/api/threads/{id}/uploads`) | `POST /` reads the latest database policy, enforces category extensions/sizes/total/count, validates signatures/OOXML containers/text encoding, then commits the whole staged batch or rolls it back; optional document conversion uses `<original-name>.<ext>.md`. `GET /limits` returns the full effective policy plus compatibility fields; `GET /list` lists; `DELETE /{filename}` deletes. Uploaded raster images also receive a signed public HTML URL (`/api/public/html-images/{token}`) so generated pages can use them as `background-image`. |
|
||
| **Task Deep-link Threads** (`/api/task-deeplink-threads`) | Dedicated shared history index for the `3qfx` / `rwfx` / `xdfx` single-agent task workspaces. `GET /?task_id=&go_path=&limit=` performs an indexed `(task_id, go_path)` lookup in `task_deeplink_threads` and joins `threads_meta` by `thread_id` for the current title/status/timestamps; LangGraph checkpoints remain the source of truth for messages. Thread metadata carrying `taskRecordId + taskGoPath` is dual-written atomically by both SQL and memory thread stores and deleted with the thread. `3qfx/rwfx` use the task id as `taskRecordId`; `xdfx` uses the original action id while retaining the resolved real task id in `metadata.taskId`. `thread_type=task_deeplink` keeps these rows out of general recent chats; explicit `taskId` searches still opt out of that exclusion for compatibility. Alembic `20261019_01` is schema-only and intentionally does **not** backfill legacy task threads. Frontend hook: `core/threads/use-task-deeplink-threads.ts`. Tests: `packages/harness/tests/test_task_deeplink_threads.py`, `packages/harness/tests/test_thread_meta_task_lookup.py`. |
|
||
| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; unexpected failures are logged server-side and return a generic 500 detail |
|
||
| **Threads search** (`POST /threads/search`, LangGraph-compat) | Conversation listing. **System threads are excluded by default**: rows whose `metadata.system` is truthy (scheduler runs, roundtable workers) or whose `thread_type` is a known system type (`task-analysis` seat runs included; older seat rows are also matched by `task_analysis_seat`) never crowd the user's list; a caller whose `metadata` filter explicitly references `thread_type`/`system` (e.g. studio notebooks) opts out of the exclusion. New `query` field = case-insensitive title (`display_name`) substring search, used by the chats page's server-side pagination. `POST /threads/count` takes the same request body (limit/offset ignored) and returns `{total}` under identical filtering/exclusion semantics ? powers the chats page's ??/??? display without fetching rows (`ThreadMetaStore.count`). `ThreadMetaStore.search` gained `exclude_system`/`query` params; JSON-blob filters batch-scan until the requested `limit`/`offset` window is full (so pagination over a heavily-filtered set never under-fills), with a `thread_id` tie-breaker on the `updated_at DESC` ordering for stable OFFSET paging. Tests: `tests/test_thread_meta_search.py` |
|
||
| **Artifacts** (`/api/threads/{id}/artifacts`, `/api/artifact-library`, `/api/strategy-project-artifacts/{id}/download`) | `GET /{path}` streams thread artifacts; active content types (`text/html`, `application/xhtml+xml`, `image/svg+xml`) are always forced as download attachments to reduce XSS risk; `?download=true` still forces download for other file types. `GET /api/threads/{id}/artifact-preview?path=` returns bounded text, using streaming statistics plus deterministic head/middle/tail samples above the database threshold. `GET /api/artifact-library` combines every current-user normal thread's authoritative `outputs/` files with completed strategy-selection syntheses projected from the owner-scoped `parallel_agent_tasks.messages` snapshot. Strategy items use `source_type=strategy`, carry `strategy_project_id` / `strategy_name` / `strategy_stale`, and explicitly distinguish `strategy_source_kind=strategy_selection` from an `analysis_project_mode=strategy` thread (`analysis_project`); virtual items also expose `download_url`, so clients never infer routing from sentinel thread/path values. Search/source filters cover both. `workflowStale=true` keeps the last completed historical synthesis visible and downloadable. The dedicated strategy download route revalidates current-user ownership and snapshot completion and serves Markdown without pretending the virtual item is a thread artifact. Uploads and image/audio/video media remain excluded. Library metadata is persisted in `artifact_library_items`; list/search/filter/count/pagination read only this user-scoped SQL index. A generation-fenced `artifact_library_sync_states` lease schedules low-concurrency filesystem/task reconciliation in the background on first/stale reads or `refresh=true`, keeps the last successful snapshot visible during refresh/failure, and reports `sync_status` / `syncing` / `indexed_at` / `sync_error` to the client. Files themselves remain authoritative on disk and are never copied into the database. |
|
||
| **Word export** (`POST /api/writing/export/docx`) | Shared Markdown → DOCX generator used by conversation, artifact, scheduled-task, Deep Research, and AI-writing download surfaces. `app/gateway/word_export.py` builds OOXML with no Python conversion dependency: A4 portrait, 3.7/3.5/2.8/2.6 cm margins, real multilevel heading numbering (`一、` / `(一)` / `1.`), two-character first-line indent on heading levels 1–3, formal body/table styles, page numbers beginning at 1 with odd pages left-aligned and even pages right-aligned, and embedded `app/gateway/assets/fonts/方正小标宋简体.ttf`. Before numbering it removes complete manual heading prefixes (including `2.1` / `2.1.1`) and drops Markdown horizontal rules from Word output. The ordinary-Q&A Lead Agent can emit the matching Markdown heading forms and avoid unordered bullet lists when the admin enables `prompt_prefix.ordinary_qa_markdown_format_enabled` in `system_settings.json`; the switch defaults off, while writing setup, writing mode, notebook, scheduled runs, and SOUL-driven custom agents keep their independent prompts. The font loader validates its internal family, SHA-256, and OpenType embedding permission (`OS/2.fsType`) before export. Tests: `tests/test_word_export.py`, `tests/test_es_query_routing_prompt.py`. |
|
||
| **Deep Research** (`/api/deep-research`) | Durable, user-owned research sessions/jobs with replayable SSE events and hidden-thread artifacts. `basic`/`quick` use the compact runner; `detailed` creates bounded subtopics, searches/writes sections concurrently, and assembles a report; `deep` recursively searches follow-up questions under a deterministic breadth/depth query budget. `multi_agent` is a LangGraph 1.x editor/researcher/writer/reviewer graph: its initial plan emits `awaiting_input`, the recorder stores the interrupt and releases the lease, and `POST /jobs/{id}/resume` rebuilds the graph from the persisted approved/revised plan. A bounded `custom_outline` is frozen into the session; headings/list entries deterministically supply the detailed and multi-agent plan, and the shared writer receives it only as a guarded content constraint. The compact writer validates generated Markdown against that frozen selection: `fixed` requires every non-placeholder heading with the same level and order, while `adaptive` retries only when no core template heading survives. That correction shares the single existing retry budget (never more than two writer calls), and its deterministic fallback is outline-shaped as well. When `custom_outline` is empty (the workbench default — the user did not pick a report structure), chat collection first calls `POST /sessions/{id}/intent-outline` after the collector turn ends: the chosen model plans a Markdown skeleton from the topic plus collected materials, the outline is frozen into the session snapshot and shown in the「开始撰写」form for editing. If that step is skipped or fails, `basic`/`quick` still call `generate_intent_outline` at write time, emit `plan_created` (`source: intent`), then write through the same shared writer; generation failure is a recoverable `intent_outline_fallback` warning and does not abort the report. Completed reports support owner-scoped follow-up (`GET /sessions/{id}/messages`, `POST /sessions/{id}/chat`): context is only the current report, selected sources and recent messages; source-id citations are persisted, while `allow_new_research=true` is explicitly rejected. Owners may change that evidence set only after the job stops via `PATCH /sessions/{id}/sources/{source_id}`; this applies to future follow-ups rather than rewriting a completed report. Optional report illustrations are gated by `config.yaml -> deep_research.image_generation`: the Harness `OpenAICompatibleImageProvider` requires base64 output, the shared runner step writes only bounded `research-image-01..04.(png|jpg|webp)` artifacts and Markdown uses private `deep-research://` links. `GET /sessions/{id}/artifacts/{name}` checks session ownership before serving an image; an unavailable provider yields a warning but never fails the text report. Runners live in `packages/harness/deerflow/agents/deep_research/runners/`, may never import `app.*`, and use only the `AdapterBundle`. |
|
||
| **Gateway auth** | Owner-scoped Deep Research routes await the platform's asynchronous `get_current_user` resolver before they read or write persistence, ensuring each session and job operation receives a concrete user ID. Deep Research artifacts are always resolved from a whitelisted filename via `/mnt/user-data/outputs/<name>` before mapping to a host path, preserving the sandbox guard on native Windows and Linux hosts. |
|
||
| **Suggestions** (`/api/threads/{id}/suggestions`) | `POST /` - generate follow-up questions; rich list/block model content is normalized before JSON parsing |
|
||
| **Thread Runs** (`/api/threads/{id}/runs`) | `POST /` - create background run; `POST /stream` - create + SSE stream; `POST /wait` - create + block; `GET /` - list runs; `GET /{rid}` - run details; `POST /{rid}/cancel` - cancel; `GET /{rid}/join` - join SSE; `GET /{rid}/messages` - paginated messages `{data, has_more}`; `GET /{rid}/events` - full event stream; `GET /../run-messages` - cross-run event-store messages with feedback. The separate `GET /api/threads/{id}/messages` contract remains the chronological latest-checkpoint transcript; `GET /../token-usage` - aggregate tokens |
|
||
| **Answer feedback** (`/api/threads/{id}/messages/{mid}/feedback`) | Message-scoped `PUT`/`DELETE`, `GET /feedback-context`, and batched `POST /api/threads/{id}/feedback-contexts`; persists excellent/poor ratings, six stable poor-reason codes and optional detail idempotently per user/thread/message/agent. Legacy run-scoped feedback routes remain compatible. Admin-only `GET /api/agents/{agent_id}/feedback-score` uses the project-wide `system_role == "admin"` rule (and preserves auth-disabled mode) and returns a Bayesian user-feedback answer-quality score, raw task-success rate, weighted response-quality dimensions and reason counts; it is explicitly not a complete agent-capability evaluation. |
|
||
| **Runs** (`/api/runs`) | `POST /stream` - stateless run + SSE; `POST /wait` - stateless run + block; `GET /{rid}/messages` - paginated messages by run_id `{data, has_more}` (cursor: `after_seq`/`before_seq`); `GET /{rid}/feedback` - list feedback by run_id |
|
||
| **Open Chat** (`/api/open/chat`) | **Unauthenticated** open Q&A entry for external systems (the `/api/open/` prefix is whitelisted in `auth_middleware._PUBLIC_PATH_PREFIXES` + CSRF-exempt in `csrf_middleware.should_check_csrf`). `POST /chat` `{message \| messages[], model_name?, agent_name?, agent_id?, thinking_enabled=false, strip_think=true, thread_id?, recursion_limit=100}` → resolves the agent by **Chinese display name** (`agent_name`, matched against each `.deer-flow/agents/*/config.yaml` `name` via `_resolve_agent_id`; explicit `agent_id` wins if given), builds the lead agent via `make_lead_agent` (so the chosen **agent's** SOUL/tools/skills all apply), `ainvoke`s it once (stateless — no checkpointer, no thread persistence; runs under the `"default"` user bucket), and returns **only the final** `{answer, model_name, agent_name, agent_id, thinking_enabled, thread_id}` (no intermediate tool-call/process). **`POST /chat/stream`** is the **SSE streaming variant** (same request body) for **long tasks** — the blocking `/chat` produces no downstream bytes for the whole run, so a front nginx `proxy_read_timeout` (default 60s) cuts it off with a **504**; the stream keeps the connection warm (a `: keepalive` SSE comment every `_HEARTBEAT_SECONDS=15`s via a background producer-task + queue, plus `event: delta` AI-text increments and `event: progress`), and ends with **`event: result`** carrying the same final `{answer, …}` JSON (or `event: error`). Clients only need to read `event: result`; intermediate events are ignorable. Both endpoints share `_build_run_context` (config/agent/thread). `GET /agents` lists `{name, id}` so callers know what `agent_name` to send. **Thinking off is reliable**: when `thinking_enabled=false` the route also sets `thinking_force_disabled=true` so the lead agent calls `create_chat_model(force_disable_thinking=True)`, which injects **three** OpenAI-compatible "thinking off" shapes into `extra_body` — top-level `enable_thinking=false` (**SiliconFlow / DashScope hybrid models** like `deepseek-ai/DeepSeek-V4-*` on `api.siliconflow.cn` — the knob the nested `chat_template_kwargs` shape does NOT reach), `chat_template_kwargs.enable_thinking=false` (vLLM/Qwen native), and `thinking.type="disabled"` (Zhipu GLM) — even for models that only declare `supports_thinking: true`. For the **web-UI path** (which sends `thinking_enabled=false` but NOT `force_disable`), SiliconFlow models need `when_thinking_disabled: {extra_body: {enable_thinking: false}}` in their `config.yaml` entry (added to `deepseek-ai/DeepSeek-V4-Flash`) for thinking-off to take effect. Sets `is_scheduled_run=true` internally to skip the auto-title LLM call. **Generated-file return (`include_files=true`, default on)**: an agent that produces its report as a file (writes to `/mnt/user-data/outputs/` + `present_files`) returns only a short summary in `answer`; to recover the real deliverable both endpoints now also return `files: [{name, virtual_path, encoding(text|base64), content}]` — `_collect_output_files` scans the run's `state['thread_data']['outputs_path']` (the open thread is stateless so artifacts can't be fetched by thread, but the endpoint runs server-side and the sandbox already wrote the files to disk), text decoded as utf-8 else base64, capped by `_MAX_OUTPUT_FILES=20` / `_MAX_OUTPUT_FILE_BYTES=5MB`. `/chat` puts `files` on the response; `/chat/stream` puts it in the `event: result` payload. A run with no answer **and** no files now 502s ("…文本答案或文件"). Router `app/gateway/routers/open_chat.py`; standalone caller `test_open_chat_api.py` (repo root, streaming by default); tests `tests/test_open_chat.py`. |
|
||
| **Open 3qfx Ask** (`/api/open/3qfx/ask`) | **Unauthenticated** open entry that, given just a **任务id**, directly kicks off the 3qfx（三情分析）「第一次问答」as a **real persisted conversation** (sibling of `/api/open/chat`; same `/api/open/` auth-whitelist + CSRF-exempt). `POST /ask` `{task_id, message?, model_name?, thinking_enabled=false}` → (1) resolves the `3Q` agent from `app.state.business_mapping_store.get_mapping("3Q")` (the **same** agent the frontend deep-link `goPath=3qfx` uses; 503 if unconfigured); (2) **fetches the task detail server-side** (consumer `cop-task-detail`) and builds the opening text **identical to the frontend** `buildOpeningText` 3qfx branch — `任务名称：…\n任务内容：…\n任务类型：…\n任务id（task_id）为{task_id}`（`taskType` 1/2/3 → 打击/塑势/防御，无法识别则省略） (an explicit `message` overrides and **skips** the fetch); (3) creates a fresh thread tagged `metadata.{taskId, agent_id}` (the exact filter `useTaskThreads` uses, and `metadata.taskId` makes it task-deeplink-shared / creator-path-resolved) and **starts a background persisted run** via `start_run(..., enforce_access=False)` (writes the checkpointer, runs under the `"default"` bucket); (4) **returns immediately** `{thread_id, run_id, agent_id, status, opening_text}` — the caller just needs "started", then watches the run in the 3qfx 任务工作区 conversation list. `start_run` gained a keyword-only `enforce_access` (default `True`); the open entry passes `False` so the admin-configured agent isn't rejected by the per-(missing)-login visibility check. **Consumer fetch config** lives in `config.yaml → task_deeplink`: `cop_task_detail_url` (full URL, appends `?taskId=`), `cop_task_detail_authorization` (default `Bearer admin`), `cop_task_detail_test` + `mock.cop_task_detail` (offline fake when the upstream is unreachable — on in this deployment). Router `app/gateway/routers/open_3qfx.py`; tests `tests/test_open_3qfx.py`. |
|
||
| **Thread Shares** (`/api/thread-shares`) | Per-conversation sharing with two modes (`thread_shares.mode`). **`import`**: `POST /` `{thread_id, mode}` - create (or reuse) a revocable share code; `GET /{code}/preview` + `POST /{code}/import` - copy that conversation into the caller's account (fresh `thread_id` via `app/gateway/thread_copy.py` + new `threads_meta` row + `user-data` copy; dedup on `metadata.shared_origin`). **`view`**: a public, read-only static page ? see Public Shares below. Both modes: `GET /` - list my codes; `DELETE /{code}` - revoke. |
|
||
| **Public Shares** (`/api/public/shares`) | **Unauthenticated** (the `/api/public/` prefix is whitelisted in `auth_middleware._PUBLIC_PATH_PREFIXES`). `GET /{code}` - read-only conversation snapshot (title + serialized messages + artifacts) for a `view`-mode share; messages are read straight from the latest checkpoint. `GET /{code}/files/{path}` - serves a file referenced by that conversation (images, Markdown, text, PDF, ?) so the shared page's links are openable; path resolution is confined to the owner's thread directory. Active content (`text/html`, `application/xhtml+xml`, `image/svg+xml`) is forced to download (`_public_share_disposition`); everything else is served inline. Missing/revoked/non-`view` codes all return 404. |
|
||
| **Roundtable Drafts** (`/api/roundtable-drafts`) | **Strictly per-user** persistence for the roundtable-planning page (replaces the old browser `localStorage`). `GET /` - list drafts (lightweight meta — incl. `chain_title`, the business-chain name used this session, cheaply regex-extracted from the in-memory `step2` blob in `_draft_to_meta`/`_extract_chain_title`; null = no chain / 自由模式; surfaced on the 第一步「分析记录」history card + sidebar list instead of the old「第 X 步」); `POST /` - create; `GET/PUT/DELETE /{id}` - full read / partial update / delete (all owner-scoped). A draft = one full session (Step 1 intent + Step 2 roundtable), step snapshots stored as JSON in `PortableLongText`. Recommendation history bound to a draft: `GET /{id}/recommendations` - list newest-first; `POST /{id}/recommendations` - append one run (objective/status/rationale/picks/candidates). taskId 深链会商记录 are **not** here — they live in the separate `/api/roundtable-task-drafts` store (see below) so personal history never mixes with task records. Backed by `deerflow.persistence.roundtable_drafts` (tables `roundtable_drafts`, `roundtable_recommend_history`); store wired in `deps.py` as `app.state.roundtable_draft_store`. Tests: `tests/test_roundtable_task_drafts.py`. |
|
||
| **Roundtable Task Drafts** (`/api/roundtable-task-drafts`) | **Task-scoped, unpartitioned** store for taskId-deeplink roundtable 聊天记录 (无界嵌入抽屉). Keyed **only by `task_id`** (one task → many records), **never partitioned by user** — any authenticated caller may read/write/delete a task's records (`created_by` is audit-only). Physically separate from `roundtable_drafts` so the two never mix. `GET /by-task/{task_id}` - all drafts for a task, newest-first (history dropdown), enriched with the latest background-job status per draft via `RoundtableJobRepository.status_map_by_drafts` (each meta also carries `chain_title`, same `_extract_chain_title` regex as the personal store); `GET /by-task/{task_id}/latest` - newest draft (回显最新一条); `POST /` - create (`task_id` required); `GET/PUT/DELETE /{id}` - full read / partial update / delete (all user-agnostic). Backed by `deerflow.persistence.roundtable_task_drafts` (table `roundtable_task_drafts`, auto-created by `create_all`; migration `20260618_01`); store wired in `deps.py` as `app.state.roundtable_task_draft_store`. Existing `task_id`-bearing `roundtable_drafts` rows are migrated over by `scripts/migrate_roundtable_task_drafts.py`. Tests: `tests/test_roundtable_task_drafts.py`, `tests/test_roundtable_drafts_by_task.py`. |
|
||
| **Roundtable Diagnostics** (`/api/roundtable-diagnostics`) | 会商全流程诊断日志 (报错 + 关键事件). One row per event in `roundtable_diagnostics`: `{scope (background = 后台挂起 / foreground = 前台服务端观测 / client = 前端上报), stage (step1_intent/step2_init/step2_leader/step2_seat/step2_recommend/step2_orchestration/step3_summary/step3_report/step3_dashboard/step3_structure/upstream/job), level (warning/error — **`info` 级运行日志不再收集**：`record_diagnostic` 在写库前丢弃 `level=="info"`，只留报错；begin/consensus/gather/job_started 等纯进度事件即使仍调 `.diag(level="info")` 也成 no-op，故诊断页 = 报错日志页), event, message, detail (JSON), job_id/draft_id/task_id/user_id/agent_id/agent_name/cycle, created_at}`. **Reads are admin-only**: `GET /` - filter (scope/stage/level/event/job_id/draft_id/task_id/user_id/q/since/until; `scope` accepts a comma-separated set so 前端「前台调用」= `foreground,client`) + paginate (newest-first), each item enriched with `userName` (user_id → 邮箱 `@`-prefix via `get_local_provider().get_user`, best-effort batched cache); `GET /facets` - distinct values for filter dropdowns; `POST /admin/cleanup` - retention purge. **`POST /report` is NOT admin-gated** — any authenticated user's frontend posts the errors *the user actually hit* (HTTP 502 before any SSE frame / network drop / SSE error frame) so they're collected too; stored as `scope="client"`. Backed by `deerflow.persistence.roundtable_diagnostics` (table `roundtable_diagnostics`, auto-created by `create_all`; migration `20260619_01`); store wired in `deps.py` as `app.state.roundtable_diagnostic_store` + a process-level default store (`register_default_store`) so the harness-side engine/gateway can write across the app/harness boundary. **Writers** (all best-effort — diagnostics never break a run): background path uses `DiagnosticsRecorder` (bound to job ctx) threaded through `RoundtableJobExecutor` → `JobProgressWriter.diag(...)` (engine transitions/step3 failures/empty-delivery) + `InProcessRoundtableGateway` (per-turn timeout/report·summary·dashboard truncation+missing); foreground path uses `app/gateway/roundtable_diag.py::record_foreground_diag` in `multi_agent.py`/`intent.py`/`recommend.py` error frames. **Bug-fix shipped alongside**: `InProcessRoundtableGateway._run_agent_turn` guards each agent run with an **inactivity (idle) timeout** (`_await_run_with_idle_guard`, `ROUNDTABLE_SEAT_IDLE_TIMEOUT_SECONDS` default 1200s = 20min, tuned high for unstable intranet models) — **not** a total-duration cap, so a slow-but-alive run that keeps streaming tokens runs as long as it needs; only a run that emits **zero StreamBridge events for `idle_timeout` seconds straight** (a true freeze: 内网模型连接池挂死 / sqlite 锁在挂载卷上挂死) is cancelled + raises (顶层把 job 标 error，而非永久冻结) + records a `seat_turn_timeout` diagnostic. Activity is observed via `app.state.stream_bridge` (the worker publishes every chunk there). Optional absolute cap `ROUNDTABLE_SEAT_MAX_TOTAL_SECONDS` (default 0 = off). Frontend: admin page `pages/RoundtableDiagnosticsPage.tsx` (route `/page/strategy/admin/roundtable-diagnostics`), opened from the **会商页右上角设置按钮** (admin only — `handleSettingsClick` in `roundtable-planning/pages/RoundtablePlanningPage.tsx`); admin read client `strategy-components/api/roundtable-diagnostics.ts`. **Client error reporter** `roundtable-planning/api/diagnostics-report.ts` (`reportRoundtableError` fire-and-forget + `keepalive`, `setRoundtableErrorContext` for draft/task attribution set by the page, `stageForRoundtableAgent`) is called at every client-observable failure: `multi-agent.ts` (`streamMultiAgent` — covers Step2 leader/seat AND Step3 report/summary/dashboard/structure — reports HTTP error + SSE `{status:"error"}` frame + worker `event:error` frame [模型调用没成功] + fetch-reject [网络不可达] + mid-stream drop [连接重置/超时], deduped via a `reported` flag; `initMultiAgent` HTTP/error-frame/incomplete/mid-stream), `intent.ts` (Step1 HTTP+error-frame), `recommend.ts` (Step2 recommend HTTP+error-frame), `roundtable-jobs.ts` (job api/stream). The admin page's first filter is a **后台挂起 / 前台调用** toggle (frontend = `foreground,client`); the stage dropdown options adapt to it (background → step2+step3+job; frontend → step1+step2+step3) and each row shows the reporting user's name + 👤. Tests: `tests/test_roundtable_diagnostics.py`, `tests/test_roundtable_inprocess_timeout.py`. **Seat skill-recognition (弱模型补偿)**: a seat = one custom agent run whose configured skills (`agent config.yaml → skills`) are injected **only** via the system prompt's `<skill_system>` block (model must `read_file` the SKILL.md itself) — weak models (deepseek-chat 等) ignore it and never use their skills. `app/gateway/roundtable_seat_skills.py::append_seat_skill_directive(agent_id, task, *, chain_enabled=False)` appends an explicit Chinese directive (each configured-and-enabled skill's name + description + SKILL.md container path + "先 read_file 再按其流程取信息/执行") to the **seat's per-turn task (human turn)** — far stronger pull on weak models than the system prompt. Wired into **both** seat paths: foreground `multi_agent._special_run` (only `role in {seat, seat_parallel}`; Step3 singletons report/summary/dashboard/ingest excluded) + background `InProcessRoundtableGateway.run_seat`. No-op (returns task unchanged) when the seat has no configured skills. **2026-06-20: default-off + two opt-in switches (OR semantics)** — was global-on, now injects only when **either** the seat agent's own switch (`AgentConfig.seat_skill_directive`, edited in the agent edit dialog `agent-card.tsx` + `agent-detail-panel.tsx`) **or** the **per-seat** switch on that agent **within the selected business chain** is on. The per-seat chain switch lives **in the chain's `seats` JSON** (`ChainSeatSchema.seat_skill_directive`, no DB column — rides the existing JSON like the per-seat `model` override), edited via a small ✨ icon → dropdown (Switch + description) on each agent card in `BusinessChainEditorPage`. It is threaded **per-seat** to each seat run: foreground `multi_agent` per-call field `seat_skill_directive` (orchestration looks it up per agentId from a `{agentId: bool}` session map) → `_special_run` → `chain_enabled`; background as the job field `seat_skill_directives: dict[str,bool]` (StartJobRequest → StartParams → deps factory → `InProcessRoundtableGateway._seat_skill_directives`) → `run_seat` looks up `agent_id` → `chain_enabled`. `ROUNDTABLE_SEAT_SKILL_DIRECTIVE` is only a **master kill-switch** (default allow; set `0/false` to hard-disable regardless of the switches — never force-enables). Tests: `tests/test_roundtable_seat_skills.py`, `tests/test_agents_backend.py`. **2026-06-20 Step2 提速三件套**：(1) **每席位推理深度覆盖** —— 业务链条 `seats` JSON 里逐席位可配 `seat_mode`（flash/thinking/pro/ultra，覆盖链条级 `seatMode`；编辑器席位卡右侧 ⏱Gauge 图标下拉，null=跟随链条默认）。前台经 `useStep2Orchestration` 三处 dispatch 的 `seatModeToRequestFields(seatModes[id] ?? selectedMode)` 透传；后台经 job `seat_modes: dict[str,str]` → `StartParams` → `InProcessRoundtableGateway._seat_modes`，`run_seat` 用 `_SEAT_MODE_PARAMS`（与前端 `seatModeToRunParams` 逐字对齐）解析覆盖作业级 seat_*（未命中→跟随作业级→policy 默认）。无 DB 列（ride `seats` JSON，但 `ChainSeatSchema.seat_mode` 须显式声明否则 Pydantic 剥掉）。Tests: `tests/test_roundtable_seat_modes.py`。(2) **recommend 批内并行** —— 后端 `engine._run_recommend` 由 for-await 串行改 `asyncio.gather`（对齐前端 2026-06-13 已并行；总控一轮派多席位=判定相互独立→并发；批内用派活前 deliveries 快照、批后统一并入，语义不变）。(3) **研讨前并行『取数』两段式**（业务链条 `gather_first` 列，逐条 opt-in，migration `20260620_02`，editor「研讨前并行取数」开关）—— `gather_first=True` 时 `run_orchestration` 在分发前先 `engine._run_gather_phase`（全席位 `asyncio.gather` 跑 `_GATHER_TASK`「只取数不研讨」），产出 `gathered: dict` 经 `_seat_message(gathered=...)` 作为「已收集资料」块注入 recommend/chain/dag 各研讨轮席位任务（与「前序交付」块分开），把最慢的技能延迟前置并行、给（尤其 chain 串行的）研讨段提速；研讨阶段不限制再调技能。前台对称实现：`useStep2Orchestration.runGatherPhase` 在 `runActiveOrchestration` 分发前跑一次（`gatherPhaseDoneRef` 闸门，resetInitGate 复位），产出存 `gatheredDeliveriesRef` 经 `peerDeliveries`/初始 `priorDeliveries` 注入研讨席位。链条级 `gatherFirst`(camel)↔`gather_first`(snake) 经 `api/chains.ts` 映射；job 经 `StartJobRequest.gather_first`→`StartParams`→`run_orchestration(gather_first=…)`。Tests: `tests/test_roundtable_gather_phase.py`。(4) **削减重复广播**（前端 only，多智能体 foreground 路径）—— `suppress_pre_broadcast`（仅 leader：抑制「总控本轮输入→各席位 thread」预广播 + 「总控给X派活」→兄弟席位的 post 广播，二者一个 flag）此前只在 DAG 派活轮开；现扩到 recommend（`safetyCounter > 1`，即第 2 轮起的续轮/综合）+ chain 逐席位（`i > 0`，第 2 个席位起）+ chain 收口轮（恒 true）。**保留每种模式的首轮预广播**（它为席位打底任务背景），只 suppress 重复的续轮广播（N 次读写巨大席位 thread 的纯浪费 + 弱模型模仿派活）；席位的前序交付另经「席位交付广播」`broadcastSeatDeliveryToOthers` 送达，不受影响。后台引擎路径无此广播（`_seat_message` 直接拼上下文），故 #4 仅前端。**2026-06-20 深链接业务链抽取规范**：深链接会商（taskId+goPath，businessCode 如 rwfx→6BF）下，结构化抽取(flow-json)与方案总结报告都按所选业务链**完整层级**强制校正。结构化抽取实为 `roundtable-structure` 智能体（`flow_extract.py` 的 `_system_prompt()=load_agent_soul(STRUCTURE_AGENT_ID)`，抽取→校验两段式 `model.astream`）；其 SOUL 全局一份，旧实现把某业务层级写死进 SOUL→所有业务都按「目的→行为体」抽（bug）。修法：**单份领域知识、两角色用法**的 `deerflow.agents.roundtable_orchestrator.extraction_spec`（放 harness 层，app+harness 都可 import；`build_structure_directive`/`build_summary_directive` 按 businessCode 取；rwfx=6BF 写实「任务→目的→行为体→预期行为→驱动因素→叙事，预期行为↔驱动因素 1:1、余 1:N、label 不含步骤/阶段名、优先报告回退席位」，其余业务通用规范=强约束完整性+命名+取数且明令不套固定模板；business_code 空→""零副作用）。注入：(1) 结构化抽取 `flow_extract._hierarchy_hint`+`_validate_messages` 用 `build_structure_directive` 权威覆盖默认层级（前端 `streamFlowExtract` 传 `business_code`，源 `ingestBusinessCode`）；(2) 方案总结报告 `build_summary_directive`——前台 `multi_agent._special_run`(role==summary 且 business_code) 追加、后台 `engine.build_summary_prompt(business_code=…)`（StartParams/StartJobRequest 透传）。**仅深链接生效**。种子 `roundtable-structure/SOUL.md` 层级顺序改 spec-driven、连接示例去 rwfx 写死（存量 live SOUL 不被 seed 覆盖，但注入规范权威可纠正）。Tests: `tests/test_roundtable_extraction_spec.py`。 |
|
||
| **Roundtable Draft Shares** (`/api/roundtable-draft-shares`) | Per-draft sharing for the roundtable-planning page, mirroring Thread Shares but the shared resource is a `roundtable_drafts` row. Two modes (`roundtable_draft_shares.mode`). **`import`**: `POST /` `{draft_id, mode}` - create (or reuse) a revocable code; `GET /{code}/preview` + `POST /{code}/import` - copy that draft (step1/2/3) into the caller's account under a deterministic per-(code,recipient) id (`imp_<sha1>` ? re-import is a no-op). **`view`**: a public, read-only result page ? see Public Roundtable Shares. Both: `GET /` - list my codes; `DELETE /{code}` - revoke. Backed by `deerflow.persistence.roundtable_draft_shares` (table `roundtable_draft_shares`, migration `20260606_01`); store wired in `deps.py` as `app.state.roundtable_draft_share_store`. Tests: `tests/test_roundtable_draft_shares.py`. |
|
||
| **Public Roundtable Shares** (`/api/public/roundtable-shares`) | **Unauthenticated** (`/api/public/` whitelisted). `GET /{code}` - read-only snapshot for a `view`-mode draft share: returns the Step 3??????report HTML (`step3.html`, a self-contained document) + title/summary/`has_report`. The frontend renders it in a sandboxed `iframe` (no `allow-scripts`) at `#/roundtable-share/<code>`. Missing/revoked/non-`view` codes all return 404. |
|
||
| **Scheduled Tasks** (`/api/scheduled-tasks`) | Per-user cron-like agent tasks. `GET/POST /` - list/create owned tasks; `PATCH/DELETE /{id}` - update/delete (owner); `POST /{id}/{pause,resume,trigger}`; `GET /{id}/runs`, `GET /runs/{rid}`, run-file preview/edit/download; `GET /runs/{rid}/messages` - agent messages for the run's underlying agent run (used to inspect a stuck/running run; authorized via `_load_run`); `GET /runs/{rid}/html` - generated HTML document for an `html_page` run; `DELETE /runs/{rid}` - owner-scoped, with an admin fallback so admins can delete any user's run; publish/subscribe under `/{id}/{publish,unpublish,subscribe}` and `GET /published`. **Admin (cross-user):** `GET /admin/all` - every task; `PATCH/DELETE /admin/{id}` - edit/delete any task; `POST /admin/{id}/trigger`; `GET /admin/{id}/runs`. Admin routes require `system_role == "admin"`. |
|
||
| **HTML Page Favorites** (`/api/html-page-favorites`) | Bookmarked generated HTML pages, owner-scoped. `GET /` - list; `POST /` `{page_id, title?, description?, tags?}` - bookmark a `scheduled_html_pages` row (extracts a reusable style/layout summary via GLM5 at favorite time); a page can only be favorited once per user (`DuplicateFavoriteError` ? HTTP 409). `GET/DELETE /{id}` - read/delete (the create-task reference picker deletes favorites here). A favorite can be selected as a style reference when creating a new `html_page` task. |
|
||
| **Public Scheduled Tasks** (`/api/public/scheduled-tasks`) | **Unauthenticated** mirror for embedding. Adds `GET /runs/{rid}/html` and `GET /{task_id}/latest-html` for the `html_page` preview; the run result keeps `result.html` (only file paths are redacted). |
|
||
| **Light Apps** (`/api/light-apps`) | Light-app registry (应用管理 / 应用列表), **global/shared** (no user scoping; `created_by`/`updated_by` audit-only). Two kinds: `window_open` (launched in a new tab from the 应用列表 card grid) and `iframe` (embedded under a parent sidebar menu via `mount_location` and dynamically injected into the sidebar). Both carry `route_params` — a list of `{key, paramType: login|theme, source, customValue, themeValue}` descriptors the **frontend** resolves at launch time (login params read from `localStorage.userInfo`; theme params use a literal value) and appends to `url` as query string. `GET /` - list (filters `app_type`/`mount_location`/`enabled_only`/`search` / repeatable `tag_ids` OR-match; each item carries `tags`); `GET /{app_id}` - read one (both open to any authenticated user); `POST /`, `PUT /{app_id}`, `DELETE /{app_id}` - **admin only**; iframe apps must carry a `mount_location` (400 otherwise); the unique `name` collides as HTTP 409. Tags reuse `/api/tags` with `tag_type=light_app` (not a 岗位 subscription type). Backed by `deerflow.persistence.light_apps` (table `light_apps`, auto-created by `create_all`); store wired in `deps.py` as `app.state.light_app_store`. Tests: `tests/test_light_apps.py`, `tests/test_light_apps_tags.py`. |
|
||
| **Menu Overrides** (`/api/menu-overrides`) | Sidebar-menu overrides (菜单管理). `menu_overrides` is the administrator-maintained shared **default**; `user_menu_override_configs` stores one complete personal snapshot per user. `GET /` returns `overrides` (effective), `default_overrides`, `user_overrides`, `has_user_override`, and `editable_scope`. `PUT /` is scope-aware: administrators replace the sparse default overlay, while ordinary users replace only their own complete snapshot; an empty personal payload (or `DELETE /`) restores defaults. Personal rows win for known `node_id`s and default rows absent from the snapshot flow through, so newly-added menus follow administrator configuration without overwriting existing personal choices. The static tree (icons/routes/`adminOnly`) remains frontend-owned; iframe light-app nodes are supported; re-parenting is validated against cycles and the three-level cap. Both tables are auto-created by `create_all`. Frontend: `core/page-layout/menu-overrides.ts`, `use-sidebar-menu.ts`, `strategy-components/api/menu-overrides.ts`, and `pages/MenuManagementPage.tsx`; the settings entry is available to every user and labels the administrator view as default management. Tests: `tests/test_menu_overrides.py`, `frontend-web/src/core/page-layout/menu-overrides.test.ts`. |
|
||
| **Compact Menu Overrides** (`/api/compact-menu-overrides`) | Compact/sidebar-summary menu overrides (简介模式菜单). `compact_menu_overrides` remains the administrator default for independent type `A` / `B` layouts; `user_compact_menu_override_configs` stores a complete snapshot per `(user_id, menu_type)`. Scope-aware `GET`/`PUT`/`DELETE` use the same effective/default/personal contract as the full menu: existing snapshot nodes stay personal and unseen new default nodes flow through. Type B default rows retain the historical `B::` storage prefix; public node ids never contain it. Frontend editing is fixed to the deployment's `runtime-config.js` `VITE_COMPACT_MENU_TYPE`. Tests: `tests/test_compact_menu_overrides.py`, `frontend-web/src/core/page-layout/compact-menu-overrides.test.ts`. |
|
||
| **Sensitive Words** (`/api/sensitive-words`) | Sensitive-word / desensitize map (敏感词管理 / 脱敏词), **global/shared** (no user scoping; `updated_by` audit-only). Each row maps an original `word` → `replacement`, with `enabled` (停用 parks a word without deleting) + `sort_order`. The **frontend** ships a seed map in code (`src/lib/desensitize.ts` `SEED_DESENSITIZE_MAP`) used as the default on a fresh deployment; once an admin saves, this table is the source of truth and the frontend loads it at startup (`DesensitizeWordsLoader` → `replaceDesensitizeMap`, **enabled** rows only). `GET /` - list all words (open to any authenticated user — every page desensitizes); `PUT /` - **admin OR the `lqq` account** (`_require_manager`: `system_role == "admin"` or email `@`-prefix `== "lqq"`), atomically **replaces the whole set** (`{words: [...]}`, de-duped by `word`, last wins). Backed by `deerflow.persistence.sensitive_words` (table `sensitive_words`, PK `word` `String(191)`, auto-created by `create_all`); store wired in `deps.py` as `app.state.sensitive_word_store`. Frontend: client `strategy-components/api/sensitive-words.ts`, page `pages/SensitiveWordsPage.tsx` (route `/page/strategy/admin/sensitive-words`, entry in the bottom-left settings dropdown — **gated on username `lqq`, not admin role**, present in both sidebar modes; 「导入种子词」 merges missing seed entries). Tests: `tests/test_sensitive_words.py`. |
|
||
| **Sentiment Agent Proxy** (`/api/sentiment-agent`) | BFF for the frontend 舆情分析 virtual agent. `POST /stream` accepts the AG-UI body plus `apiUrl`/`authUsername`/`authPassword` from `runtime-config.js`, forwards to that URL with TLS verify off, and byte-pipes upstream SSE. Connection/HTTP failures return the full error in `detail`; mid-stream failures emit AG-UI `RUN_ERROR`. Distinct from `skill-packages/sentiment-analysis.skill`. Tests: `tests/test_sentiment_agent_proxy.py`. |
|
||
| **Dashboard Sessions** (`/api/dashboard-sessions`) | 「大屏绘制智能体」独立页面会话，**按外部 `task_id` 存取，不按 user 分权**（open read + shared writes；`user_id` audit-only）。一个 task 一行（upsert）：`transcript`（聊天记录 JSON）+ `report_json`（最新结构化数据，可解析 = 「有效结果」）。`GET /by-task/{task_id}` - 取该 task 会话（无则 404）；`PUT /by-task/{task_id}` - upsert（`{transcript?, report_json?}`，只覆盖传入项）。Backed by `deerflow.persistence.dashboard_sessions`（table `dashboard_sessions`, auto-created by `create_all`，无迁移）；store wired in `deps.py` as `app.state.dashboard_session_store`。前端独立页 `roundtable-planning/pages/DashboardAgentPage.tsx`（路由 `/page/strategy/dashboard-agent?taskId=`，侧边栏「问答管理 → 大屏绘制」）让智能体分析任务报告 JSON → report-json → 渲染 `DashboardApp`。Tests: `tests/test_dashboard_sessions.py`。 |
|
||
| **Task Reports** (`/api/task-reports`) | 任务级研判情报报告详情，**按 `task_id` 共享、不按 user 分权**（写入者只留 `updated_by` 审计）。`GET /by-task/{task_id}` 始终返回原 Consumer 兼容外壳 `{state:"200", msg:"操作成功！", data:[...]}`（未保存为 `data: []`）；`POST /by-task/{task_id}` 首次保存，已有报告返回 409；`PUT /by-task/{task_id}` 原子替换完整 `data` 数组。`POST /import/by-task/{task_id}` 面向 TaskCOP 导入页，接收 `sq-report-mock.json` 兼容的 `data`/`records` 数组并完整覆盖目标任务；路径中的选定任务 id 优先于文件内的 `taskId`，避免误写。每一项严格是 `{id, taskId, createTime, updateTime, sessionId, contentJson, categoryType}`；`contentJson` 以 JSON **字符串**原样落在 `records_json`，避免破坏大屏按类目解析的输入格式。保存或导入非空 `data` 后，router 会通过 `app.state.taskcop_task_store.mark_analysis_completed` 将同 id 的 TaskCOP 任务升级至状态 `25`，使旧任务列表显示并筛选为“已完成”；普通保存时不存在匹配任务的报告仍可独立保存，导入则要求目标任务存在。Backed by `deerflow.persistence.task_reports`（table `task_reports`, auto-created by `create_all`）；store wired in `deps.py` as `app.state.task_report_store`。前端客户端 `strategy-components/api/task-reports.ts`，`core/auth/dashboard-input.ts` 与 rwfx 3q 完成检查均直接读此接口，不再依赖 Consumer 或本地 sq-report mock。Tests: `tests/test_task_reports.py`。 |
|
||
| **Task Buttons** (`/api/task-buttons`) | Task-workspace configurable jump buttons (按钮管理), **global/shared** (no user scoping; `updated_by` audit-only). Each row = one jump button in a task deep-link workspace's bottom 快捷跳转 bar: `{id, business (rwfx/3qfx/xdfx), label, link_type (url/business/purpose/host), target, append_task_id, id_param (editable query-param name, e.g. id/task_id), id_kind (task/action — only xdfx differs, its deep-link id is an 行动id), open_mode (blank/self), login_params, append_auth, auth_token_param, auth_name_param, enabled, sort_order}`. **Two independent, coexisting login-append mechanisms for `url` buttons** — both resolved frontend-side: (1) `login_params` (JSON column `login_params_json`, `PortableJSON`, migration `20260625_01`) = a list of `{key, source, custom_value}` resolved at click time (`source` = a `userInfo` field or `other`→literal `custom_value`) and appended to the target URL — mirrors the light-app login params; empty = nothing appended. (2) `append_auth` + `auth_token_param` + `auth_name_param` (columns, migration `20260625_02`, default off) — appends `?{auth_token_param}={token1}&{auth_name_param}={username}` so the target system can identify the current user; gated on **token login** — the frontend only stashes `token1` (the `?authToken=`) + the raw `getTokenInfo` username (`login.token1`/`login.username`) when login went through `/login/token`, so a non-token-login session appends neither. The **frontend** holds code-seed defaults (rwfx's 4 historical buttons built from the live `task_deeplink` config); only when this table is empty do the workspace + admin page fall back to them, else this table is the sole source of truth. `GET /` - list all (open to any authenticated user — the workspace renders them); `PUT /` - **admin only** (`_require_admin`), atomically **replaces the whole set** (`{buttons: [...]}`, de-duped by `id`, last wins). Backed by `deerflow.persistence.task_buttons` (table `task_buttons`, auto-created by `create_all`); store wired in `deps.py` as `app.state.task_button_store`. Frontend: helpers `core/auth/task-buttons.ts`, client `strategy-components/api/task-buttons.ts`, admin page `pages/TaskButtonsPage.tsx` (route `/page/strategy/admin/task-buttons`, entry「按钮管理」in the bottom-left settings dropdown, admin-only, both sidebar modes; per-business tabs + 「导入默认」). Tests: `tests/test_task_buttons.py`. |
|
||
| **Fixed Questions** (`/api/fixed-questions`) | Administrator-managed exact-match fixed Q&A (问题配置), **global/shared**. Each row is `{id, question, answer, enabled, tokens_per_second, sort_order}`; stream speed is configurable per row from `1–1000 token/s` and defaults to `100`. `GET /` and whole-set atomic `PUT /` are both admin-only. At the start of an external chat run, `services._find_fixed_answer` checks the latest eligible plain-text human message with exact, case/space/punctuation-sensitive equality. Enabled matches switch the run to `make_fixed_answer_agent`: a local `BaseChatModel` uses a lossless standard-library-only word/CJK/punctuation splitter for rate-limited chunks. It must never initialize a model tokenizer such as `tiktoken.get_encoding`, because an empty tokenizer cache can trigger a public vocabulary download and break offline deployments. The local graph keeps the existing LangGraph SSE `messages`/`values` protocol, run journal, checkpoint history, deterministic local title, and refresh behavior intact while no provider/model request or model-concurrency slot is used. File/multimodal/context-prefixed turns, `canvas_agent`, `ai_writing`, writing mode, resume commands, and internal orchestration/background child runs bypass matching. Storage failures fail open to the normal model path. Backed by `deerflow.persistence.fixed_questions` (table `fixed_questions`, SHA-256 indexed lookup plus exact original-string collision check, migrations `20260821_01` + `20260821_02`), wired as `app.state.fixed_question_store`. Frontend: `strategy-components/api/fixed-questions.ts`, admin page `pages/FixedQuestionsPage.tsx`, route `/page/strategy/admin/fixed-questions`, entries in both full-layout and compact-workspace settings menus. Tests: `tests/test_fixed_questions.py`. |
|
||
| **Knowledge** (`/api/knowledge`) | Product knowledge base, **global/shared** (no user isolation; writes record `created_by`/`updated_by` for audit). `POST /ingest/thread/{thread_id}` - sediment one thread into a note (reads messages from the checkpointer, `threads:read` owner-checked); `POST /ingest/threads/batch` - batch-sediment the caller's recent threads (`skip_existing` via `.manifest.json` dedup). **Defaults to `background=true`**: the slow per-thread extraction loop is deferred to a FastAPI background task and the endpoint returns `{queued:true, processed:N}` immediately (each thread's transcript is truncated to `max_input_chars` before the LLM call). Running the loop inline for a large batch outlives the nginx gateway timeout (504), which the frontend's `parseResponse` surfaces as a generic "Request failed"; `background=false` keeps the blocking path (full per-thread `created/skipped/failed` tally) for small batches/scripts. Shared loop: `_run_batch_ingest`. **Optional model selection**: both ingest endpoints accept `model_name` (overrides `knowledge.extract_model_name` for that capture; threaded through `KnowledgeService.capture_thread` ? `extract_thread_llm_multi`). **Model-error surfacing**: when the chosen model produces nothing usable (timeout / API error / non-JSON) capture still **falls back to rule-based extraction and lands a draft** (`llm_fallback:true`); the single-thread (non-background) response and the batch response (`llm_fallback` count) carry this so the UI tells the user "???????". The "??????" chat button stays background/fire-and-forget (passes `model_name`, no per-save error toast); the batch-import dialog is synchronous and shows the fallback count. `POST /ingest/file` - **import an uploaded file** (multipart: `file` + optional `title`/`tags`/`status`/`model_name`) into wiki note(s): the file is converted to Markdown (md/txt read directly, pdf/word/ppt/excel via `convert_file_to_markdown`), distilled by the same wiki-capture LLM pipeline as thread capture (`source_type="file"`), and falls back to a verbatim **draft** note when the model returns nothing usable (`llm_fallback:true`); `POST /ingest/search` - sediment search/tool results as a reference note; `POST /notes` - manual note; `GET /notes` (keyword/tag/source_type/status + pagination); `GET/PUT/DELETE /notes/{id}` (DELETE is soft = archive, `?hard=true` removes the vault file); `POST /search` - `mode=keyword\|vector\|hybrid` (vector?keyword fallback when no embedding endpoint); `GET /notes/{id}/versions` - edit history; `GET /duplicates` + `POST /merge` - dedup; `POST /reindex` - rebuild vector index; `GET /graph` - product graph data (notes + entities, graph-blacklist filtered); `GET/POST /graph/blacklist` + `DELETE /graph/blacklist/{id}` - graph node blacklist (labels = entity names / note titles, case-insensitive match; blacklisted nodes and their edges are dropped from `/graph` and the wiki-export artifacts; table `knowledge_graph_blacklist`, auto-created); `POST /export/graph` + `GET /export/files/{graph.json\|graph.html\|graph.graphml\|cypher.txt}` - obsidian-wiki `wiki-export` artifacts (self-contained `graph.html` for iframe); `POST /resolve` `{targets:[?]}` + `GET /notes/{id}/wikilinks` - resolve `[[wikilink]]` targets to a note id / vault page / missing (powers clickable wikilinks); `GET /vault-page?path=` - read a raw vault meta/hub Markdown page (no DB note). `GET/POST /folders` + `PUT/DELETE /folders/{id}` - user-managed directories (notes carry a `folder` path; `GET /notes?folder=` filters a dir + sub-dirs); `GET/POST /extract-templates` + `GET/PUT/DELETE /extract-templates/{id}` - editable extraction-prompt templates (thread/document ingest accept `template_id`; capture/PUT-note accept `folder`). Backed by the `deerflow.knowledge` module (see below). |
|
||
|
||
Proxied through nginx: `/api/langgraph/*` ? LangGraph, all other `/api/*` ? Gateway.
|
||
|
||
### TaskCOP Situation-Overview Compatibility
|
||
|
||
The standalone legacy Vue page calls the Gateway URL in its runtime `b1ConsumerUrl` configuration directly (no Vite `/api` reverse proxy). The Gateway CORS middleware merges standard local Vite origins (`localhost`/`127.0.0.1` on ports 5173, 5174, 3000, and 8080) even when `GATEWAY_CORS_ORIGINS` was set for another frontend. This avoids a restart regression when the existing Vite server is on 5173 but the launcher sets 5174. Deployments must set `GATEWAY_CORS_ORIGINS` to any additional exact frontend origins that are allowed to call the Gateway.
|
||
|
||
`app/gateway/routers/taskcop_tasks.py` is a temporary compatibility layer for the separately deployed legacy Vue page at `/situation-overview/action/sentiment` and its task import page at `/situation-overview/action/task/import`. It preserves the old Consumer paths and response envelopes: `POST /cop/saveSuperiorTask` creates a row with a sequential four-digit string id (`0001`, `0002`, ...) and returns `{state: 201, msg, data}`; `POST /api/taskcop/import/tasks` accepts a JSON array or `tasks`/`records`/legacy list envelope, uses each supplied `id` or `taskId` as the primary key, and completely replaces duplicate IDs; `DELETE /cop/tasks/{task_id}` removes that task and its shared report document; `GET /taskAnalyseSearch/cop-task-three-list` (all), `GET /taskAnalyseSearch/cop-my-tasks` (creator-scoped), and `GET /taskAnalyseSearch/cop-task-list` return `{state: 200, data: {total, records}}`; `GET /taskAnalyseSearch/cop-task-detail?taskId=` returns one task. Existing UUID-style task rows are retained, and do not affect the new numeric sequence. An empty TaskCOP store remains empty—there are no built-in task or report seed records. It supports the page's existing `content`, `taskStatus`, `taskDirection`, `startTime`, `endTime`, `pageNum`, and `pageSize` query fields. Tasks are stored in `taskcop_tasks` through `deerflow.persistence.taskcop_tasks`, registered by `persistence.models`, and initialized as `app.state.taskcop_task_store` in `deps.py`. A non-empty save/update/import through `/api/task-reports/*` calls `mark_analysis_completed(task_id)`: it changes only pre-completion task states to `25`, so the list renders and filters the task as “已完成” without downgrading later workflow states. `trendSucc` is derived from the same status for the legacy sentiment list. Zero-padded task ids stay strings in the report API, so `0001` is never converted to `1`. The Vue router calls DeerFlow's `/api/v1/auth/login/username` directly with an incoming `username`/`userId`/`yUserId` parameter; it does not depend on the retired Consumer login service, and sends the returned DeerFlow JWT to every TaskCOP call. The TaskCOP routes are no longer public or CSRF-exempt. The Gateway fallback CORS allow-list includes TaskCOP Vite origins `http://localhost:8080` and `http://127.0.0.1:8080`; deployments must use explicit `GATEWAY_CORS_ORIGINS`. Tests: `tests/test_taskcop_tasks.py` and `tests/test_taskcop_cors.py`.
|
||
|
||
### AI Writing Intent Router (对话驱动写作台意图路由)
|
||
|
||
Backs the frontend「AI 写作·对话版」page's unified bottom composer. `POST /api/ai-writing/sessions/{session_id}/intent` (owner-checked; in `routers/ai_writing.py`) reads the session's graph checkpoint via the shared `app.state.checkpointer` — the **pending interrupt is the authoritative pause point** (the frontend's claimed `pause_point` is only a fallback when the checkpoint can't be read) — then routes the user's free text through `deerflow/agents/ai_writing/intent_router.py::parse_user_intent`: an LLM call (session's `model_name` + `fallback_chain` model fault-tolerance) that returns `{intent, pause_point, action, payload, need_research, research_query, answer, confidence, clarify}`. Safety boundaries in `normalize_intent_result`: **per-pause action whitelist strictly aligned to graph.py's real resume actions** (`ALLOWED_ACTIONS`), no pause → no action, high-cost actions (`finalize` / `force_finalize` / query-less覆盖式 `re_search`) require confidence ≥ 0.85 else downgrade to `clarify`, payload key-whitelist cleaning (material-id filtering, review-issue index bounds, section-decision back-fill with the majority decision). QA intents return `answer` directly from the context snapshot (`build_context_snapshot` — stage-tailored: materials with ids / outline / draft body capped / review issues with indices). Model failures never 5xx — they degrade to clarify. Prompts in `agents/ai_writing/prompts/intent_router_prompts.py`; tests `tests/test_ai_writing_intent.py`.
|
||
|
||
### AI Writing Built-in Functional Agents (????????)
|
||
|
||
The AI-writing graph (`packages/harness/deerflow/agents/ai_writing/`) has four stage "brains" ? ?????? / ????? / ?? / ?? ? that can each run in one of two modes, switched per-stage by `config.yaml ? ai_writing.use_builtin_agents.{researcher,outliner,writer,editor}` (default **false**, mtime hot-reloaded, new sessions pick up changes immediately):
|
||
|
||
- **Legacy (flag off)**: hard-coded prompts in `prompts/*.py`, direct LLM call (unchanged behavior).
|
||
- **Builtin-agent (flag on)**: the node loads the corresponding functional agent from `.deer-flow/agents/ai-writing-<stage>/` ? `SOUL.md` becomes the system prompt and `config.yaml`'s `model` overrides the stage model (priority: `agent.model > node explicit fallback (keyword_model/rank_model) > state.model_name`). Admins edit SOUL/model from the agent management page. Runner: `nodes/_agent_runner.py` (`run_writing_agent` / `stream_writing_agent` / `parse_marker_json`). Output contracts (marker + ```json``` blocks, intent.py-style): researcher `[KEYWORDS_READY]` / `[MATERIALS_RANKED]`, outliner `[OUTLINE_READY]`, editor (per-dimension) `[REVIEW_READY]`; writer emits raw section Markdown (keeps per-section streaming) with the legacy `[[SECTION_BLOCKED]]` sentinel in strict mode. **Orchestration stays in the nodes** (search fan-out, per-section loop, 3-dimension review loop, all pause points) ? the agent only does single structured generations, and a missing/empty SOUL silently falls back to the legacy path (`WritingAgentUnavailable`).
|
||
|
||
Seeding mirrors the roundtable pattern: verified `config.yaml` + `SOUL.md` ship in `app/gateway/routers/_ai_writing_seed_assets/<id>/`; `_ai_writing_seed.py::ensure_ai_writing_functional_agents()` copies them to `.deer-flow/agents/` when missing (never overwrites local edits), called from the startup lifespan (before `_sync_legacy_agents`, so the agents appear in `/api/agents` on first boot) and self-heals from `GET /api/ai-writing/sessions` + `GET /api/ai-writing/article-types`. Tests: `tests/test_ai_writing_seed.py`, `tests/test_ai_writing_agent_runner.py`.
|
||
|
||
**Sample imitation mode (样文仿写).** The initial writing form supports `writing_mode_type=imitate`: the user can paste sample text or upload Word/PDF/Markdown/TXT. `POST /api/ai-writing/sample/extract` converts supported document files to Markdown via `deerflow.utils.file_conversion` and returns capped plain text to the frontend. The LangGraph entry point now starts at `sample_analyzer` before `intent_parser`; non-imitation sessions return immediately, while imitation sessions analyze up to 12k chars into `sample_style_profile` (`SampleStyleProfile` in `state.py`) and emit `sample_profile_ready`. `writer_outline` and `writer_draft` call `render_sample_style_section(state)` so both outline planning and section drafting receive the same style profile, imitation strength (`light|medium|strong`), optional structure-preservation preference, and guardrails: learn structure/tone/paragraph rhythm/sentence style, but do not copy sample sentences, proprietary facts, people, organizations, cases, or data. Frontend state mirrors this with `analyzing_sample`, uploaded/pasted sample controls, history/timeline restoration of `sampleStyleProfile`, and the setup-chat direct-start path preserves the same fields. Tests live with the knowledge-base AI-writing tests (`tests/test_ai_writing_knowledge_base.py`).
|
||
|
||
**Skill-driven retrieval (检索类型).** `material_source` has three values: `general` (通用检索, default), `knowledge_base` (知识库检索), and `notebook` (我的空间); legacy `skill` is normalized to `general` by the researcher node. **`knowledge_base` 直连内网 ES 接口（= 旧版「内网直连检索」）** — it bypasses skill orchestration entirely and calls `IntranetSearchProvider` over `ai_writing.intranet_search_url` (concurrent over the generated keywords; `intranet_knowledge_base`/`intranet_search_timeout` apply). **An empty/unconfigured `intranet_search_url` does NOT disable it** — the empty string is passed straight to `IntranetSearchProvider`, which falls back to its built-in default endpoint (`intranet_stub._DEFAULT_ENDPOINT`), exactly like the legacy `get_search_provider(search_provider=intranet)` path. So 内网直连检索 works out of the box without configuring an address (configure `intranet_search_url` only to point at a different endpoint). This is the "弱模型识别不了技能" escape hatch — the old no-skill retrieval path restored as an explicit option (weak models like deepseek-chat often fail to invoke configured skills). Tests: `tests/test_ai_writing_knowledge_base.py`. General retrieval is orchestrated from the **skills configured on the `ai-writing-researcher` agent** (`config.yaml → skills`, editable from the agent management page): the built-in `knowledge-base-search` skill (知识库检索) runs the native keyword fan-out web/intranet search (the old default path), while any other configured skill searches with a `description`-targeted query; skills run in configured order with early-stop at `max_materials`, and an empty/unresolvable skills config falls back to the built-in knowledge search so retrieval always works. The skill itself is seeded from `_ai_writing_seed_assets/skills/knowledge-base-search/SKILL.md` into `skills/public/` (never overwrites), and a pre-existing researcher `config.yaml` lacking the `skills` key is patched in place with the default (explicit `skills: []` is respected). **PAUSE-1 input append flow**: when the user types natural language into the material-confirm input (`re_search`/legacy `enable_skill_search` + `user_query`), the node matches the most relevant configured skill via `SkillPicker(candidates=…)` (skipped when only one skill is configured), searches with the user's words, and **appends** the new materials to the existing package (URL-dedup, original order preserved, no LLM re-rank/truncation, 100-item hard cap). **Skill-failure fault tolerance**: a configured skill that times out / raises / has all its keyword searches fail (vs. legitimately returning 0 results) no longer just silently moves on — the researcher node emits a `skill_search_step` of type `skill_error` (carrying the error `reason`), and if the configured skills collected **no** new materials at all it runs a `fallback` pass over other available skills (`_fallback_candidate_skills`: the built-in `knowledge-base-search` native path first, then other enabled catalog skills not yet tried) instead of erroring out. The frontend `SkillStepsCard` renders `skill_error` (rose) and `fallback` (amber) so the timeline shows "技能报错 → 改用其他技能搜集". Tests: `tests/test_ai_writing_material_source.py`.
|
||
|
||
The intranet client uses `trust_env=False` and defaults `intranet_verify_ssl=false`, so internal HTTPS endpoints with self-signed/internal-CA certificates do not fail on certificate verification; set `ai_writing.intranet_verify_ssl: true` only for environments that require strict certificate checks.
|
||
|
||
**Skill-call timeout** is hardcoded **300s (5 min)** — `researcher.py` `_SKILL_SEARCH_TIMEOUT_SECONDS` (per-skill wrapper) and `_skill_agent.py` `_REAL_AGENT_TIMEOUT_SECONDS` (whole real-agent skill run). The intranet ES fallback (`IntranetSearchProvider`) gets a longer **600s (10 min)** default (`intranet_stub.py`; overridable via `ai_writing.intranet_search_timeout`) — that interface is the slow one, so our own fallback timeout is generous to avoid being blamed for "检不出".
|
||
|
||
**Parallel intranet fallback (技能 0 条 → 直接用并行预跑的内网结果).** When `ai_writing.intranet_search_url` is configured, the researcher node launches the intranet ES search (`IntranetSearchProvider` over the generated keywords) as a **concurrent `asyncio.Task` the moment the skill loop starts** (not sequentially after all skills fail). If the configured + fallback skills collect **nothing**, the node awaits that already-running task and uses its results (`_collect`) — no extra wait. If the skills **did** get materials, the task is cancelled (`_cancel_intranet_task`, suppresses CancelledError) so the intranet call isn't wasted. URL empty → no task, behavior unchanged. Tests: `tests/test_ai_writing_intranet_parallel.py`.
|
||
|
||
**Partial-result harvest + query simplification (检不出/超时的容错).** When `researcher_skill_agent: true` runs a non-native skill via `run_skill_via_real_agent` (the deployment's real path — the skill, e.g. `knowledge-search-v2`, is invoked by the real agent through `bash python3 scripts/search.py "<query>"`), one writing run issues many queries and **some time out** (the skill script prints `{"success":false,"error":"请求超时…","count":0}`) while others succeed (`{"success":true,"count":30,"results":[…]}`). Two fixes so successful data is never dropped: (1) `_drive()` captures every **ToolMessage** stdout and `_harvest_tool_results()` parses the skill's JSON **by reusing the lead-agent 参考文献 mode's proven extractor** (`deerflow.runtime.references`: `_parse_json_like` + `_first_list` + `_normalize_reference_item`) — the same logic that already extracts skill JSON into citations 100% reliably on ChatPage/AgentChatPage, with its huge field-name union (title/m_title/name, content/page_content/content_preview/snippet/…, recUuid/uuid/docId/…) and per-skill display-config support — plus skips `success:false` outputs — so even if the model's final `{materials:[]}` summary gives up (confused by the timeouts), the successful queries' real results are still collected; the function merges model materials + harvested results (dedup by url/recUuid/title, capped at `max_results`). **The harvest+merge (`_finalize`) runs on EVERY exit path — normal end, overall timeout, AND `await task` exceptions (e.g. `GraphRecursionError`)** — so when the run is cut off before the model writes its summary (observed: 10 queries × ~2 graph steps > the old `recursion_limit`, so the agent errored out with the real 30+ results already in `tool_outputs`), those results are still salvaged instead of returning `[]`. `recursion_limit` was also raised from `_REAL_AGENT_MAX_TURNS` to `_REAL_AGENT_MAX_TURNS*2+4` (it counts graph super-steps ≈ 2 per model→tool round, not model turns) so the model normally has room to finish its summary. **Early-stop on `max_results`:** `_drive` re-harvests after every tool output and, once the deduped harvested count reaches `max_results` (= `per_skill` = `max_materials`), **breaks the nested `astream`** and emits a 「已获取 N 条素材，达到上限，停止本技能检索」 step — without it a weak model keeps issuing new queries forever (observed: 100+ step-bar entries, materials far over the limit). Because the skill now returns exactly `max_materials`, the researcher loop's own `collect_limit` early-stop fires right after, so later configured skills don't run either. (2) The model is instructed (in `_MATERIAL_OUTPUT_CONTRACT`) that a 0-result/timeout query must be **retried with a simpler, core query** (drop topic suffixes like 竞选/诉讼/政策, keep the core entity: 「特朗普竞选」→「特朗普」) and that any data already retrieved must always be kept. The same deterministic simplifier `search/base.py::simplify_query` is wired into `IntranetSearchProvider.search`: an empty first attempt retries once with the simplified query. **(3) Root cause of the instability — bash output middle-truncation.** The skill prints its results as JSON to stdout, and the `bash` tool **middle-truncates** any output over `sandbox.bash_output_max_chars` (default **20000**, inserting a `... [middle truncated …] ...` marker). A `count:30` skill response (~30 records × ~600 chars) exceeds 20000 → the JSON is cut in the middle → **unparseable as a whole** by both the model and `_try_load_json` → all materials lost (while a small `count:9` response parses fine — hence the flapping). Fix: when whole-JSON parse yields no items, `_harvest_tool_results` falls back to `_scan_balanced_json_objects` — a brace-depth/string-aware scanner that recovers **every individually-complete `{…}` record object** from the truncated text (the records on each side of the truncation marker are still valid JSON), so ~16+16 records survive a 20000-char cut, far above `max_materials`. This is why the lead-agent chat pages (`/page/workspace/chats/new`, agent chat) call the same skill with 100% success — they only need the model to summarize prose from whatever it sees, never to parse a structured list, so truncation doesn't hurt them; ai-writing needs the structured records, so it must salvage them itself. Tests: `tests/test_ai_writing_skill_real_agent.py`, `tests/test_ai_writing_query_simplify.py`.
|
||
|
||
**Agentic skill execution (`ai_writing.researcher_skill_agent`).** When this flag is on (the deployment `config.yaml` enables it), the directed-skill branch runs `nodes/_skill_agent.py::run_skill_via_real_agent`: it reuses `SubagentExecutor` with the full tool stack, sandbox middleware, and a one-skill whitelist so the skill can `read_file` its `SKILL.md`, call `bash`, MCP tools, or whatever the skill instructions require. The nested run now passes `agent_id=ai-writing-researcher` in both `config.configurable` and runtime `context`, matching normal AgentChat skill visibility/whitelist behavior. It also appends a human-turn hard directive listing the exact SKILL.md path and skill directory, requiring weak models to first `read_file` the SKILL.md and then `cd <skill-dir> && python/python3 scripts/*.py ...` when the skill is script-backed; the model is explicitly forbidden from returning `materials` before executing the skill's required tools. If the real-agent path fails or yields no materials, `_execute_skill` still falls back to the legacy description→query search so materials are never empty. The built-in `knowledge-base-search` native concurrent path is unchanged. Tests: `tests/test_ai_writing_skill_real_agent.py`.
|
||
|
||
**Targeted user revision (局部修改 — 只改命中章节).** When the user submits a free-text 修改意见 on the draft-confirm card (`action == "user_revise"`), `writer_draft_node` no longer always rewrites the whole article section-by-section. Before the full-rewrite loop it tries `_targeted_user_revision` (`nodes/writer_draft.py`): `_split_draft_sections` parses the previous `current_draft.full_markdown` into `(article_title, [(section_title, body)], references_block)` (level-1 `#` title, level-2 `##` sections, trailing `## 参考文献` pulled out and kept verbatim), then `_detect_revision_scope` runs a small `llm_json` classifier (prompts `DRAFT_REVISE_SCOPE_{SYSTEM,USER}`) returning **per-section targets** `[{index, new_title, rewrite_body}]`: it maps the note to the section indices it targets ("第一段/开头/引言"→0, "结尾/结论"→last, by title/topic) **and** distinguishes a **章节标题改名** (`new_title` set, `rewrite_body=false` → rename only, body untouched) from a **正文修改** (`rewrite_body=true`). Title-only renames apply directly to the assembled `## {new_title}` heading **and are written back into `current_outline.sections[i].section_title`** (so the renamed title persists through confirm/finalize/history — this fixes "改章节标题不生效"). Body rewrites run `_stream_revise_section` (prompt `DRAFT_REVISE_SECTION_USER`, streamed as `draft_chunk`). **Only hit sections are touched; unhit sections are preserved byte-for-byte and are NOT re-emitted to the stream** — the frontend clears the right canvas to its initial placeholder on entering `revising` and shows only the live rewrite of the changed section(s), so the user sees「清空 → 流式重写 → 终稿」rather than the whole article rebuilding. The classifier returns `whole_article=true` (整体性意见 like 「整体语气更正式」/「全文都改」, **structure changes** 加章/删章/调整顺序, or a **文章总标题** change) → `_detect_revision_scope` yields `None` → `_targeted_user_revision` returns `None` → the node **falls back to the whole-article rewrite** (which still runs `_revise_outline_with_notes` for title/structure changes). Also falls back when there is no prior draft, an empty note, or scope-detection errors. Editor-review revisions (`accept_review`) keep the holistic rewrite. Reassembly preserves the article title + structure and re-appends the original references block; word count is computed on the body (excluding references). Frontend: `isRevising` in `AIWritingState` (`types/ai-writing.ts`) is set in `useAIWritingStream.ts`'s `intervention_submitted` (`ns === 'revising'`) and cleared on `draft_ready`; `AIWritingDraftPanel.tsx` shows the initial `DraftWaitingPlaceholder` while `isRevising && streamingDraftSections.length === 0`, then the streaming preview once chunks arrive. Tests: `tests/test_ai_writing_targeted_revision.py`.
|
||
|
||
**History draft editing (历史回看可编辑成稿).** The right-side draft in 历史回看 is no longer read-only — the user can edit it like the live 写作完成态 (Tiptap + 目录 + 格式工具栏 + AI 润色) and **persist** the change back to the session. `PUT /api/ai-writing/sessions/{id}/draft` `{draft_markdown, draft_title?}` (owner-checked → 404 otherwise; `draft_title` omitted ⇒ keep原标题) updates the row via the existing `AIWritingSessionRepository.update`. Frontend: `AIWritingDraftPanel` now passes `readOnly={false}` + an `onSaveDraft` handler (only in history mode) → `TextArtifactRenderer` → `WritingToolbar` 「保存」button (loading + toast); the handler is `AIWritingHistoryContext.saveHistoryDraft(markdown, title?)` → `updateAIWritingDraft` (`api/ai-writing-sessions.ts`), which write-backs the returned summary to `historySession`. Edits to the **live** completed draft stay ephemeral (export-only) as before — only history mode shows 保存. Tests: `tests/test_ai_writing_draft_update.py`.
|
||
|
||
**Background suspend (后台挂起).** A writing session can be handed off to the server to auto-drive the remaining flow to finalize, surviving page leave / refresh / account switch. `POST /api/ai-writing/sessions/{id}/background` (owner-checked, idempotent) starts `AIWritingJobExecutor` (`app/gateway/ai_writing_job_executor.py`), wired in `deps.py` as `app.state.ai_writing_job_executor`. The executor **drives the same `ai_writing` graph in-process** — it builds `make_ai_writing_graph()`, attaches the shared `app.state.checkpointer` (the very saver the runtime worker uses, so it continues the real thread state the frontend already wrote), sets the owning user via `set_current_user`, and loops `aget_state` → `ainvoke(Command(resume=…))` until `status==done`. **Auto-action per interrupt**: material/outline confirm → `confirm`; section_help → every blocked section `loose` (continue with general knowledge, never re-loop to research); draft_confirm → `to_editor` until reviewed, then `finalize` (on pass, or when `revision_count >= max_revisions`); review_confirm → `accept_review` (revise & re-review until pass or budget exhausted, then forced finalize). To avoid racing a still-running runtime worker (suspend mid-node), the driver **waits for an actionable state** (`_wait_for_actionable`): it only acts when the graph is at an interrupt / done, or when the checkpoint stops advancing (no worker → abandoned/restart). No new table — the graph checkpoint + `ai_writing_sessions` already hold all state; `status="background"` marks 挂起中 (surfaced in the history dropdown), nodes write `draft`/`review_result`/`status=done` as usual, so the user reopens the finished article from history. Per-user concurrency cap `AI_WRITING_BACKGROUND_SLOTS_PER_USER` (default 2, same SQLite-lock rationale as the roundtable executor). Startup `reconcile` (in `deps.py`) relaunches drivers for sessions left `background` after a restart (`AIWritingSessionRepository.list_by_status`). **Progress flowchart popup**: `GET /api/ai-writing/sessions/{id}/progress` (owner-checked) reads the graph checkpoint via `executor.get_progress` and derives a canonical pipeline `stage` key (`intent/research/material_confirm/outline/outline_confirm/draft/draft_confirm/review/done`, from the pending interrupt's `pause_point` or `snapshot.next` node) + `note` (素材不足求助 / 编辑打回) + review progress + `isRunning`. Frontend: 「后台挂起」 button + auto-suspend on route unmount / `beforeunload` (`AIWritingPanel.tsx`, `suspendAIWritingToBackground[Keepalive]` in `api/ai-writing-sessions.ts`); 挂起中 badge in `AIWritingHistoryDropdown.tsx`, and **clicking a 挂起中 history record opens a flowchart dialog** (`AIWritingFlowDialog.tsx` polls `getAIWritingProgress` every 2.5s, renders the fixed vertical pipeline `AIWritingFlowChart.tsx` lit by the live `stage`) so the user sees how far the background run has gotten without reopening/taking over the session. **Transcript consistency**: the rich history timeline (`AIWritingHistoryView` → `AIWritingTimeline`) replays `transcript = {progressEvents, completedInterventions}`, normally assembled+saved client-side. A background run has no client, so the executor rebuilds it at the end (`_save_transcript`/`_build_transcript_from_checkpoint`) **entirely from the final checkpoint** — both `progressEvents` and `completedInterventions` are derived from the same set of `resumed` events (every pause-resume emits exactly one), so they are always 1:1 aligned. For each `resumed`: classify its message → `(pausePoint, action)` (`_classify_resumed`, matched against `graph.py`'s resume messages), insert an `await_user` (with the pause prompt) before it, and build a **rich** intervention card enriched from final state (material_confirm→`materials`, outline_confirm→`outline`, draft_confirm→`draftTitle`/`draftWordCount`/`reviewResult`, review_confirm→`reviewResult`); `materials_ready` is inserted before the first material-confirm. (Earlier code merged the frontend's pre-suspend `completedInterventions` with auto ones and aligned by index — but the frontend's last save can predate its most recent decision, so the counts drifted and cards rendered off-by-one with empty data; deriving everything from the checkpoint root-fixes both.) Raw (snake_case) events **and** intervention data are normalized on load by the **exported** `normalizeProgressDict` / `normalize{Material,Outline,ReviewResult}` (idempotent on already-camelCase live transcripts — one normalize path for both sources), so reopening a background-completed session shows the same cards as a frontend-driven one. Tests: `tests/test_ai_writing_background.py`.
|
||
|
||
### Admin Leaderboard Snapshots (`/api/admin/users/leaderboard*`)
|
||
|
||
The admin ????? is served from **precomputed per-day snapshots** in `admin_leaderboard_daily_stats` (keyed by `(stat_date, settings_hash)`), not by aggregating raw rows on every request. A background scheduler inside the Gateway (`_leaderboard_snapshot_scheduler_loop` in `app/gateway/app.py`, singleton file-locked) ticks every `ADMIN_LEADERBOARD_BACKGROUND_INTERVAL_SECONDS` (default 300s): it prewarms recent days, then builds any `queued`/`error`/stale-`running` day. `GET /leaderboard` merges the per-day snapshots for the requested range. **Today** is rebuilt whenever its snapshot is older than `ADMIN_LEADERBOARD_TODAY_TTL_SECONDS` (default 43200s = half a day, so the leaderboard refreshes ~twice daily); a past day, once `ready`, is treated as fresh forever.
|
||
|
||
**Recent-only admin audit policy.** The server enforces a rolling **30-calendar-day** window for administrator leaderboard, user analytics, user Q&A/thread audit, LLM/tool metric list/summary/export, and the user-list Q&A counter; URL parameters cannot widen it. The Q&A fallback that reads checkpoints is additionally capped, while indexed run summaries remain the primary audit source. Gateway startup also launches the singleton `admin_audit_retention` sweep: it deletes audit metric rows, terminal `runs` summaries (which contain compact question/answer audit fields and were also last updated before the cutoff), and leaderboard snapshots older than **90 days**. Cleanup is deliberately low-priority: after the default 5-minute startup delay, it runs every 15 minutes using 100-row committed transactions, round-robins each store (at most four batches per store), yields between commits, and stops starting new batches after an 8-second sweep budget; operators may tune these bounds with `ADMIN_AUDIT_RETENTION_*` environment variables. It deliberately does **not** delete conversation threads, checkpoints, or message history; those are user-content records governed by their own lifecycle.
|
||
|
||
Two on-demand/automatic refresh paths layer on top:
|
||
- **Manual "????"** ? `POST /leaderboard/refresh-today` force-rebuilds *today's* snapshot **synchronously** (bounded by the same semaphore/timeout the scheduler uses), then returns the merged range. Lets an admin pull the latest real-time records into the stats without waiting for the TTL. Frontend: a "????" button on `AdminUserLeaderboardPage` ? `useRefreshTodayLeaderboard` (`core/admin/hooks.ts`), which primes + invalidates the `["admin-activity","leaderboard"]` query.
|
||
- **Daily finalize of the previous day** ? once per day, on/after `ADMIN_LEADERBOARD_FINALIZE_HOUR` (local Beijing, default `1` = ?? 1 ?), the scheduler tick force-recomputes *yesterday* a single time so runs/tool-calls that only landed near midnight are fully captured. The decision is the pure `deerflow.persistence.admin_stats.finalize.should_finalize_previous_day` (status/generated_at/now/hour) ? the rebuild stamps `generated_at` to today, which self-debounces it for the rest of the day (no extra state file). Tests: `tests/test_leaderboard_finalize.py`.
|
||
|
||
### Admin Concurrency Monitor (实时并发监控 / 系统压力)
|
||
|
||
A live admin monitor embedded **at the top of `AdminUserLeaderboardPage`** (`components/workspace/admin/concurrency-monitor.tsx`, `ConcurrencyMonitorPanel`) that answers "当前有几个并发的大模型调用、是谁、能不能直接停掉" and charts system pressure over time. Built on the existing in-memory `RunManager` (no new run plumbing).
|
||
|
||
- **Live concurrency + 真正停止** — `RunManager.list_active()` returns every run whose status is `pending`/`running` (the registry keeps finished records around until `cleanup`, so it filters by status). `len(list_active())` is the current concurrent-call count. Router `app/gateway/routers/admin_active_runs.py` (`/api/admin/active-runs`, **admin only** via the shared `admin_users._require_admin`): `GET /` returns `{count, server_time, runs:[{run_id, thread_id, status, created_at, elapsed_seconds, user_id, user_name, agent_id, agent_name, thread_title, kind, multitask_strategy}]}` — each run best-effort enriched (batched DB) with the owning user's email (`threads_meta.user_id`→`users.email`) and the agent's display name (`agents.name`); `kind` is derived from thread metadata (圆桌会商 / 系统/后台 / 任务对话 / 对话). `POST /{run_id}/cancel` calls `RunManager.cancel(run_id, action="interrupt")`, which sets the abort event **and cancels the run's asyncio task** — i.e. it truly aborts the in-flight model stream, freeing capacity. `POST /cancel-all` stops every in-flight run.
|
||
- **并发量折线图 (按分钟/小时)** — a lightweight background **sampler** (`_concurrency_sampler_loop` in `app/gateway/app.py`, singleton file-locked, started in the lifespan) records `len(list_active())` every `CONCURRENCY_SAMPLE_INTERVAL_SECONDS` (default 15s) into the new `concurrency_samples` table (`deerflow.persistence.concurrency`, `ConcurrencySampleRow{sampled_at, active_count}`, auto-created by `create_all`, **no migration**; store wired as `app.state.concurrency_sample_store`). ~Hourly it purges rows older than `CONCURRENCY_SAMPLE_RETENTION_HOURS` (default 168h). `GET /concurrency?granularity=minute|hour&hours=N` aggregates samples into per-bucket **peak + avg** (`ConcurrencySampleStore.aggregate`, bucketed in Python so it's identical across sqlite/mysql/postgres; fetch capped at 60k rows). Default lookback: minute→3h, hour→48h.
|
||
- **大模型出字速度折线图 (tokens/sec, 按分钟/小时)** — `GET /token-speed?granularity=&hours=` aggregates the persisted `llm_call_metrics` (`status="success"`) into per-bucket avg/min/max tokens/sec (deriving tokens/sec from `output_tokens÷duration_ms` when the stored value is null). A dip pinpoints the model/provider as the bottleneck for that window — "知道具体原因出现在哪里".
|
||
- **On/off + 不影响线上** — the whole feature is designed to never impact production: sampling is one tiny INSERT, retention-purged, all paths best-effort/guarded, queries bounded. A **runtime switch** lives in `system_settings.json → concurrency_monitor.enabled` (`ConcurrencyMonitorSettings`, **default false / opt-in**, mtime-hot-reloaded): the background loop always runs (just sleeps) but only samples when an admin turns it on, and it re-reads the flag **every tick** so toggling it on/off **takes effect immediately without a restart** (the on-demand live snapshot + 停止 stay available regardless; only the historical chart needs sampling on). `GET|PUT /api/admin/active-runs/settings` reads/writes it (admin only); the panel's 「后台采样」 Switch drives it. Env `CONCURRENCY_SAMPLE_ENABLED=0` is a hard master kill (never starts the loop). Frontend client/hooks: `core/admin/active-runs.ts` + `core/admin/hooks.ts` (`useActiveRuns` polls 4s while the panel is open, `useConcurrencySeries`/`useTokenSpeedSeries` poll 15s, `useCancelActiveRun`/`useCancelAllActiveRuns`, `useMonitorSettings`/`useUpdateMonitorSettings`). Tests: `tests/test_concurrency_monitor.py`.
|
||
|
||
### Scheduled Task Runtime
|
||
|
||
`ScheduledTaskService` (`deerflow/runtime/scheduler/service.py`) polls due tasks every `poll_interval_seconds` and executes each in its own throwaway thread, serialized by a shared `_agent_run_lock`.
|
||
|
||
**Stuck-run recovery**: each execution attempt is bounded by `_RUN_ATTEMPT_TIMEOUT_SECONDS` (30 min) via `asyncio.wait_for`. On timeout the run is retried up to `_MAX_RUN_RETRIES` (3) times, then marked `failed`. A watchdog (`_reconcile_stuck_runs`, run every poll) recovers runs left in `running` with no in-process owner ? e.g. orphaned by a crash or restart ? re-driving them through the same retry budget so a task can never stay perpetually "executing". The `scheduled_task_runs.retry_count` column tracks consumed retries.
|
||
|
||
**Unique task names**: task names are unique per user (`uq_scheduled_tasks_user_name`). `create_task`/`update_task`/`admin_update_task` proactively check via `_assert_name_available()` and raise `DuplicateTaskNameError` on a collision; the router translates that into HTTP `409` instead of letting the raw DB `IntegrityError` surface as a `500`.
|
||
|
||
**HTML page tasks (`task_kind == "html_page"`)**: a second task kind that renders a full HTML document each run instead of a Markdown report. The kind + its config (`html_config`: `page_prompt`, `negative_prompt`, `reference_favorite_id`, `model_name`) live inside `execution_context` ? no schema change to `scheduled_tasks`. Data gathering is delegated to the agent's own search skills (driven by `page_prompt`), so there is no explicit search-query/data-source field; the render model is user-selectable (`model_name`, default `zai-org/GLM-5-FP8`). `_run_task_attempt()` dispatches by kind; `_run_html_page_attempt()` runs the pipeline: gather real data via the lead agent (its search tools) ? optionally load a reference favorite's reusable style/layout ? render with the chosen model (thinking off) ? fault-tolerant JSON parse ? sanitize (BeautifulSoup, strips `<script>`/`<iframe>`/`on*`/`javascript:`) ? structural validate ? one GLM5 repair pass if invalid ? **completeness gate**: if the HTML is still not a full valid document (missing doctype/html/head/body, empty body, or residual scripts) after the repair pass, raise `HtmlGenerationError` so the run is marked **failed** (a page only counts as successful when it is a complete, previewable HTML document) ? best-effort Playwright screenshot (no-op when Playwright absent) ? archive `index.html` ? persist a `scheduled_html_pages` row ? return `result` with `format: "html"` and the full HTML inline. Pure helpers live in `deerflow/runtime/scheduler/html_page.py`. The frontend previews the HTML in a sandboxed `iframe` (no `allow-scripts`) on the run-detail and public pages; a user can bookmark a page (style summary extracted via GLM5) and later pick it as a style reference. New tables `scheduled_html_pages` + `html_page_favorites` (`deerflow/persistence/html_pages/`, wired as `app.state.html_page_store`) are created by `Base.metadata.create_all` with no migration. Tests: `tests/test_html_page_tasks.py`.
|
||
|
||
**Unified output switches (`require_html` + `require_markdown`).** The task form no longer splits "???? / HTML ??" into separate kinds. `execution_context` now carries two **independent** booleans that may both be on: `require_html` (???? HTML) and `require_markdown` (???? Markdown). `_task_requires_html()` returns true for `require_html` **or** legacy `task_kind == "html_page"` (so old tasks need no data migration), and `_run_task_attempt()` dispatches to `_run_html_page_attempt` when it is true, else the regular agent attempt. When **both** switches are on, `_gather_html_data()` returns `(text, files)` ? the gather agent run is additionally told to save a `.md` file ? and `_run_html_page_attempt` validates it (`MarkdownRequirementError` on failure) and surfaces it alongside the generated page, so the run delivers both artifacts. The single main `prompt` doubles as the HTML page requirement (`html_config.page_prompt`); there is no separate page-prompt field. Tests: `tests/test_scheduled_task_require_html.py` + the markdown-coexistence cases in `tests/test_html_page_tasks.py`.
|
||
|
||
**Template-fill mode (JSON → 固定 HTML 模板填充).** A third execution mode, distinct from the free-form `require_html` render pipeline: convert a user-supplied JSON into a **fixed** HTML template's data structure and fill it. Triggered when `execution_context.template_config.template_id` is set (chosen in the task form after selecting the built-in **「模板数据转换器」** agent `template-json-builder`). `_task_template_id()` reads it and `_run_task_attempt()` dispatches to `_run_template_fill_attempt` **before** the `require_html` check. Pipeline: take the source (the task `prompt`, pasted or uploaded .json) — which may be **pure JSON, plain text, or text-with-embedded-JSON** (`template_fill.extract_embedded_json` pre-extracts any embedded `{…}`/`[…]` with a string-aware balanced scan and surfaces it as a labeled block while keeping the full original text; plain text → no block, the model reads it semantically) → load the selected agent's `SOUL.md`+model and run a **direct** `_invoke_conversion_model` call (system=SOUL, human=conversion prompt carrying the template's compact **schema skeleton** + the recognized-JSON block + source) → `extract_json_object` → `template_fill.validate_data` (top-level keys + container types) → **on failure feed the structure errors + `field_coverage_gaps` back and retry up to `_MAX_TEMPLATE_FILL_RETRIES`=2** → `template_fill.fill_template` replaces the template's `const DATA={…}` line (and refreshes `MANIFEST.系统.更新时间`) → archive `index.html` + persist a `scheduled_html_pages` row → return `format:"html"` (reusing the existing HTML preview at `/runs/{id}/html`). Any terminal failure raises `TemplateFillError` → run marked failed (no retry, message shown). **Pure engine** `deerflow/runtime/scheduler/template_fill.py` (harness, no `app.*`): the template registry (`TEMPLATES`: id=`no1` name=`no.1` 台湾滨海防卫大屏, and id=`no2` name=`no.2` 台湾飞行部队平台 with **`convertible=False`** — a 「只展示」 template that renders its existing HTML **as-is**, no model/validation/fill, until conversion is wired in later; `_run_template_fill_attempt` short-circuits to `read_template_text` for it, and `list_templates`/`is_convertible` expose the flag so the form hides the JSON input), `list_templates`/`reference_data`/`schema_skeleton_json`/`validate_data`/`field_coverage_gaps`/`fill_template` — **the template's own embedded `DATA` is the schema source of truth** (parsed at load, 1-sample-per-array skeleton fed to the LLM, top-level structure as the validation basis); template HTML files live under `runtime/scheduler/templates/<id>/`, add a `TemplateSpec` to register a no.2. The agent is seeded like the AI-writing ones: `app/gateway/routers/_template_builder_seed.py` + `_template_builder_seed_assets/template-json-builder/{config.yaml,SOUL.md}`, called from the lifespan before `_sync_legacy_agents` (which upserts it into the `agents` table as built-in/published so it shows in `/api/agents` → the task form's 执行智能体 dropdown). Template list API: `GET /api/scheduled-tasks/html-templates` → `{templates:[{id,name,description,convertible}]}` (open). **Debug-render API** `POST /api/scheduled-tasks/template-fill/render` `{template_id, text?|data?}` → `{ok, display_only?, errors, gaps, html}` (open): no model call — `extract_embedded_json(text)` (or `data`) → `validate_data` → `fill_template`, returning the filled HTML **plus** the validation errors/field-gaps (fills even when invalid so you can see the effect). Powers the chat-page debug button below; non-convertible templates return their raw HTML. **Templates as `require_html` 参考收藏页面 (use the REAL template, all sub-pages)**: the 参考收藏页面 list in the task form also offers the templates (value `tpl:<id>`, e.g. `tpl:no1`). These templates are **JS-driven multi-page SPAs** (sub-pages/tabs are rendered by script from `DATA`), so a free-form model render would only ever produce the main page and **drop every sub-page**. Therefore, when `_run_html_page_attempt` sees a `tpl:<id>` reference (`template_fill.reference_template_id` parses it; distinguishes a template marker from a real favorite id), it short-circuits **before** the free-form pipeline to `_run_html_template_reference`, which renders the **actual template file** (all sub-pages intact): convertible templates gather real data (`_gather_html_data`) → convert to the template `DATA` (`_convert_to_template_data`, shared with template-fill mode) → **`merge_with_reference`** (per top-level key = per sub-page: any sub-page/field with no extracted content falls back to the template's **original** data, so sub-pages never render blank) → `fill_template`; non-convertible templates (no.2) output the raw template as-is; if the task prompt **and** description are both empty, or gather/conversion yields nothing, it uses the template's original data outright. **数据抽取开关** `html_config.template_extract` (default **off**, shown in the form only when a `tpl:` reference is picked): it **only** controls whether the HTML uses **extracted** data vs the template's **default** data — it does **not** affect anything else. Off → HTML fills the template's default data (no convert); on → gather+convert+merge. **Markdown is independent**: when `require_markdown` is also on, the gather agent run still runs and the `.md` is produced/validated/surfaced **regardless** of the extract switch (so turning extraction off never blocks the Markdown). Display-only templates (no.2) never extract; they still honor `require_markdown`. `_run_html_template_reference` runs gather once when `do_extract` (extract on + has prompt/desc + convertible) **or** `require_markdown`, and threads any `.md` through `_finalize_template_html(extra_files=…)`. `merge_with_reference` is also applied in template-fill mode (`_run_template_fill_attempt`) so empty sub-pages there fall back too. Archive/persist/result are shared via `_finalize_template_html`. (The earlier `style_reference` CSS-extraction approach was removed — it lost the sub-pages.) Frontend: when the agent dropdown = `template-json-builder` the form hides the HTML/Markdown switches and shows a 「HTML模板」 select (`useHtmlTemplates`) + a .json upload (reads file text into the prompt box). Tests: `tests/test_template_fill.py`, `tests/test_template_fill_attempt.py`, `tests/test_template_builder_seed.py`.
|
||
|
||
### Per-user Prompts & UI Preferences
|
||
|
||
**Custom prompts can be scoped to one agent.** `user_custom_prompts` gains a nullable `agent_id` column (migration `20260603_02`): `NULL` = global (the chip shows in every chat), otherwise the prompt only surfaces when that agent is selected. The visibility filter is applied in the frontend `CustomPromptBar` (it receives the current agent id from `ChatPage`'s selector or `AgentChatPage`'s `assistantId`); the backend (`/api/user-prompts` CRUD) just stores/returns/clears `agent_id` (update accepts an explicit `null` to clear). Tests: `tests/test_user_prompts_agent_id.py`.
|
||
|
||
**Per-user UI preferences** (`deerflow/persistence/user_preferences/`, wired as `app.state.user_preferences_store`; migration `20260603_03`, also auto-created by `create_all`): one JSON-blob row per user so new toggles need no schema change. Router `/api/user-preferences` (`GET`/`PUT`, account-scoped). Current key: `hide_admin_prompt_prefix` ? when true the chat input hides the admin-configured prompt-prefix switch and never applies the prefix (the user uses only their own prompts; enforced in `ChatPage` for both `shouldApplyPrefix` and `promptPrefixProp`). Toggled from the "???????" dialog. Tests: `tests/test_user_preferences.py`.
|
||
|
||
**HTML skills (tools the render step uses).** Two offline skills under `skills/public/` shape the generated HTML and are toggled in the skills settings page (enabled by default): `html-page-builder` (a self-contained inline-CSS design system ? colors/cards/tables/KPI/responsive) and `html-charts` (inline-SVG charts that render in the no-`allow-scripts` preview iframe; optional local ECharts documented in its `assets/README.md`). A skill opts into HTML injection via `metadata.kind: html_page` in its SKILL.md frontmatter; `_collect_html_skill_instructions()` reads the bodies of **enabled** such skills and `build_generation_prompt(skill_instructions=?)` injects them, so the (raw-model) render step honors the user's skill selection. The data-gathering step and the render/repair/style passes all use the task's selected `model_name` (no silent fallback to the default model ? and there is **no hardcoded GLM-5**; an unset selection falls back to config `models[0]`). To stop auxiliary LLM calls from leaking onto the default model, **`TitleMiddleware` is skipped when `is_scheduled_run` is set** (`_build_middlewares` in `lead_agent/agent.py`) ? scheduled/HTML runs use throwaway threads whose titles are never shown, and `config.title.model_name` is usually unset (? would call `models[0]`, not the chosen model). The completeness gate (`validate_html`) also requires a non-empty `<title>` and forbids external stylesheets, on top of the doctype/html/head/body + non-empty-body + no-script checks.
|
||
|
||
### Knowledge Base (`packages/harness/deerflow/knowledge/`)
|
||
|
||
Self-contained product knowledge base. **All knowledge capability lives in this one module** (harness layer, `deerflow.*` only ? no `app.*` imports). Wired as `app.state.knowledge_service` in `deps.py` via `make_knowledge_service(sf, config.knowledge)` (returns `None` on the memory backend or when `knowledge.enabled=false` ? router 503). Surfaced by `app/gateway/routers/knowledge.py` (`/api/knowledge`). Config: `config.yaml` ? `knowledge` (`deerflow/config/knowledge_config.py`).
|
||
|
||
**Two stores, kept in sync on every write:**
|
||
- **DB mirror** ? tables `knowledge_notes` / `knowledge_sources` / `knowledge_tags` / `knowledge_note_tags` (`knowledge/models.py`, registered in `persistence/models/__init__.py`, auto-created by `Base.metadata.create_all`; portable types `PortableLongText`/`BeijingDateTime`). Powers listing/filter/search/audit. Global/shared ? no `user_id` column; `created_by`/`updated_by` are audit-only (resolved via `get_effective_user_id()`).
|
||
- **Markdown vault** ? obsidian-wiki compatible, default `{$DEER_FLOW_HOME}/data/knowledge/obsidian-vault`. `ObsidianWikiAdapter` (`obsidian_adapter.py`) writes wiki-capture frontmatter+body, and after **every** write syncs `index.md` / `log.md` (CAPTURE/INGEST/UPDATE/ARCHIVE/EXPORT) / `hot.md` / `.manifest.json` (source-hash dedup for `skip_existing`). `markdown_vault.py` handles frontmatter build/parse, slugging, wikilink extraction and **path-traversal-proof** resolution.
|
||
|
||
**Module layout:** `models.py` (ORM: notes/sources/tags/note_tags + embeddings/versions/entities/relations) ? `repository.py` (DB only, no LLM) ? `schemas.py` (NoteDraft/SourceDraft DTOs) ? `skill_templates.py` ? `markdown_vault.py` ? `manifest.py` ? `extractor.py` (rule-based fallback) ? `llm_extractor.py` (**default**: distills declarative knowledge per wiki-capture, structured JSON incl. entities/relations; falls back to `extractor.py`) ? `sources.py` ? `exporter.py` (`graph.json`/`graph.graphml`/`cypher.txt` + self-contained vanilla-JS `graph.html`; entity nodes + relation edges. **`graph.html` is theme-aware** ? it reads `?theme=light|dark` from its own URL since the embedding iframe can't inherit the app's CSS; `KnowledgeGraphPage.tsx` appends the active day/night mode and re-skins on toggle. The layout cools down and **freezes** so dense graphs stay clickable, node radius scales with degree, and hovering/clicking a node focuses it + its direct neighbours while dimming the rest + a detail panel ? taming heavy node/edge clutter) ? `chunking.py` + `embeddings.py` (OpenAI-compatible embedding client, keyword fallback) + `search.py` (keyword/vector/hybrid) ? `sensitive.py` (secret/PII redaction) ? `dedup.py` (Jaccard + hash dedup) ? `auto_ingest.py` (background queue/worker) ? `obsidian_adapter.py` ? `service.py` (`KnowledgeService` orchestrator ? the only thing the router touches; capture entry points `capture_thread` / `capture_search` / `capture_file` / `create_manual_note`, all converging on `_persist_draft`). Tests: `tests/test_knowledge.py` (20 cases). Frontend: `frontend-web/src/core/knowledge/*` + `pages/KnowledgeBasePage.tsx` / `KnowledgeGraphPage.tsx` + chat `knowledge-save-trigger.tsx` / `knowledge-rag-toggle.tsx`; spec at `frontend-web/docs/product-knowledge-base-dev.md`.
|
||
|
||
**Wikilinks & multi-note capture** (`knowledge.{related_notes_enabled,related_notes_limit,multi_note_enabled}`, all default-on; `project_name` defaults to `zncm`):
|
||
- **Navigable `[[wikilinks]]`** ? `service.resolve_wikilink(s)` maps a `[[target]]` to a knowledge note (by id ? `vault_path` ? title) or a vault meta page (`read_vault_page`, path-traversal-proof), else `missing`. `markdown_vault.normalize_wikilink_target` canonicalizes targets (drops `|alias`, slashes, `.md`). The frontend `MarkdownRenderer` renders non-numeric `[[...]]` as clickable links from the resolver map (numeric `[[123]]` stay citation badges); note links navigate, vault links open a `VaultPageDialog`, missing links are greyed.
|
||
- **Real related notes only (no structural meta-links)** ? `extractor.related_links()` now returns `[]`: notes no longer auto-link `index` / `_meta/taxonomy` / a project hub (they were noise in every `## Related`). On capture, `service._find_related_notes` cross-links **only by shared *distinctive* tags** (project/`knowledge-base` defaults excluded; free-text title/summary matching was removed because CJK tokenization linked unrelated notes via generic words like ??/????) and `_inject_related_notes` writes real `[[Title]]` links into `## Related` (plus note?note `related` graph edges); no distinctive tags ? no related notes. The renderers omit the `## Related` section entirely when there are no related notes and no sources. The vault's own `index.md`/`_meta/taxonomy.md` still exist (obsidian scaffold) and remain resolvable for manually-written links. The router enriches `GET /notes/{id}` with `created_by_name`/`updated_by_name` (auth-provider email lookup) so the UI shows a person, not a UUID. The frontend `MarkdownRenderer` renders wiki-capture provenance markers `^[inferred]`/`^[ambiguous]` as small "??"/"??" badges.
|
||
- **LLM-extraction outcomes** ? capture distinguishes three LLM outcomes: (1) **success** ? declarative note(s); (2) **deliberate rejection** ? the model returns `should_ingest:false` with no notes (greeting / meta chatter) ? `extract_thread_llm_multi` raises `LLMRejectedIngest` and `capture_thread` raises `EmptyThreadError` (skip ? nothing persisted, so worthless chats never enter the KB); (3) **genuine failure** ? API error / non-JSON ? returns `[]`, capture falls back to the rule-based verbatim extractor and lands the note as a **`draft`** (not `approved`), logs a WARNING, and returns `llm_fallback:true` (surfaced by the router). `llm_extractor` logs an actionable WARNING/INFO for each non-success path so extraction issues are diagnosable.
|
||
- **Multi-note split** ? `llm_extractor.extract_thread_llm_multi` lets one conversation distill into 1?4 cross-linked notes (`notes:[?]` JSON array; legacy single-object output still accepted). `capture_thread` persists each, sibling-cross-links them by title, and returns `{note, created, notes:[?]}`. Gated by `multi_note_enabled` (false ? first note only). Tests: `tests/test_knowledge_wikilinks.py`.
|
||
|
||
**Phases 2?4 (all config-gated, defaults safe):**
|
||
- **Phase 2 ? vector RAG**: `knowledge.embedding.{enabled,base_url,api_key,model,...}` enables an OpenAI-compatible embedding endpoint; notes are chunked + embedded on write (`knowledge_embeddings`), `POST /search` supports `mode=keyword|vector|hybrid` (vector degrades to keyword when unconfigured). `KnowledgeRagMiddleware` (lead-agent chain) injects a transient `<knowledge-context>` recall before the model call, gated **per-conversation** by the `knowledge_rag_enabled` runtime flag (frontend chat toggle) falling back to global `knowledge.rag_enabled`. `POST /reindex` rebuilds the index.
|
||
- **Phase 3 ? governance**: `sensitive.py` redacts secrets/PII before persistence (`sensitive_redaction_enabled`); every update snapshots `knowledge_note_versions` (`GET /notes/{id}/versions`); `GET /duplicates` + `POST /merge` for dedup; background `AutoIngestQueue` (wired in `deps.py`, started/stopped in the gateway lifespan) auto-sediments finished conversations **off the chat path** when `auto_ingest_enabled` ? enqueued from `KnowledgeRagMiddleware.after_agent`, value-judged by min-chars + `skip_existing`, landing as `draft`.
|
||
- **Phase 4 ? graph**: LLM extraction also yields entities/relations ? `knowledge_entities`/`knowledge_relations`; `exporter.build_graph` adds entity nodes + note?entity/entity?entity edges (reflected in `graph.html`/`graph.json`); `GET /graph` returns product graph data.
|
||
|
||
**User directories + editable extraction templates + display polish (knowledge UX overhaul):**
|
||
- **Custom directories (`knowledge_notes.folder` + `knowledge_folders`)** ? notes carry a free-form, slash-joined `folder` path (e.g. `投研/行业`) independent of the wiki-capture `category` and of `vault_path`. `knowledge_folders` is the manageable directory list (supports empty folders + rename/move that re-points descendant folders **and** filed notes). Router CRUD under `/api/knowledge/folders` (`GET`/`POST`/`PUT/{id}`/`DELETE/{id}?reassign_to=`); `GET /notes?folder=` filters a directory + its sub-folders (`folder=""` ? unfiled). Capture (`capture_thread`/`capture_file`/`create_manual_note`) and `PUT /notes/{id}` accept `folder` (the note PUT uses `model_fields_set` so an absent key leaves the folder untouched, a present `null` clears it). Frontend `buildKnowledgeTree(notes, folderPaths)` builds the left tree from `folder` (no longer from `vault_path`); folder management dialog + `FolderSelect` in the create/import dialogs and the note detail header.
|
||
- **Editable extraction prompts (`knowledge_extract_templates`)** ? the hard-coded distillation prompts are seeded as editable templates on startup (`KnowledgeService.seed_default_templates`, idempotent — `默认 · 对话提炼` / `默认 · 文档提炼`, each `is_default` for its scope). `scope ∈ {thread, document, both}`; capture resolves an explicit `template_id` → the scope default → the built-in constant, and passes it to `llm_extractor.extract_thread_llm_multi` / `extract_document_llm_multi` as `system_prompt` (used verbatim). Router CRUD under `/api/knowledge/extract-templates` (409 on duplicate name). Frontend: extraction-template management dialog + `TemplateSelect` in the batch/file import dialogs. Editing a template is the supported way to fix "提炼太精简" (loosen the prompt to keep more detail) and to add alternate extraction logics.
|
||
- **Single source list** ? `extractor._related_section_md` no longer emits a `### Sources` sub-section into the body; the note detail shows sources once via the structured `来源` block (`note.sources`). `## Related` stays pure wikilinks so the frontend extracts it cleanly into the related-docs module.
|
||
- **Chinese section headings (display-only)** ? the frontend `localizeHeadings` (wikiUtils) maps `Context/Finding / Decision/Reasoning/Implications/Related/Sources` to 中文 at render time (applied to the body fed to both `extractToc` and the viewer, so anchors stay consistent). No data migration — historical English-heading notes render in Chinese too.
|
||
- **Historical-data cleanup** ? `scripts/cleanup_knowledge_related.py` (one-off, idempotent, `--dry-run`/`--note-id`) strips legacy `## Related` structural links (vault scaffold: `index`/`_meta/taxonomy`/`projects/…`, any slash-bearing target) and the old `### Sources` sub-section from existing notes (DB `content_md` + vault files, via `KnowledgeService.update_note`).
|
||
- Migration `20260616_02` + idempotent `scripts/sql/column_additions_mysql.sql` cover `knowledge_notes.folder` (+ index); the two new tables are auto-created by `create_all` / `_ensure_orm_columns_sync` on startup. Tests: `tests/test_knowledge.py`.
|
||
|
||
### Custom Agent Persistence
|
||
|
||
Custom agent metadata lives in the `agents` ORM table (`deerflow.persistence.agents`). `agents.id` is the stable runtime id and the filesystem directory name under `.deer-flow/agents/{id}/`; `agents.name` is display-only, can contain Chinese characters, and is not unique. `user_id IS NULL` marks built-in agents, user-owned agents are visible only to their owner unless `published = true`, and update/delete operations require ownership. The runtime accepts `agent_id` in context/configurable, and non-default `assistant_id` values are treated as agent ids after visibility checks.
|
||
|
||
Strategy-selection fan-out (`routers/parallel_agents.py`) must keep hidden reasoning separate per agent: provider reasoning blocks/fields and inline `<think>...</think>` text emit `reasoning_delta`, while visible answer text emits `delta` and only visible text may enter `agent_done.content`. Never concatenate `reasoning_delta` into a selectable strategy or generated deliverable. Inspect the final AI message with the shared `is_llm_error_message` detector and require substantive visible text before constructing a success result. A middleware fallback, empty/thinking-only response, or punctuation-only provider degeneration is a failed seat run: raise so the stream emits `agent_error`; never wrap it in `agent_done`. Regression coverage: `tests/test_roundtable_model_fallback.py`.
|
||
|
||
**Listing cache + pagination** ? to keep `GET /api/agents` fast, the gallery's per-card extras (skills + tool_groups) are denormalized into two cache tables instead of reading every agent's `config.yaml` on each request: `agent_skills` (relational, one row per (agent, skill) with `position`) and `agent_extras` (one row per agent: `tool_groups` JSON + `model` + `skills_synced_at`). `config.yaml` remains authoritative for the agent runtime; these tables are a read cache, write-through on create/update and backfilled from disk by `_sync_legacy_agents` (mtime-vs-`skills_synced_at` drift check). The list endpoint enriches via `AgentStore.get_extras_for([ids])` (two batched `IN` queries). Both tables are auto-created by `Base.metadata.create_all` (no migration needed). `GET /api/agents` supports **opt-in server-side pagination**: `scope` (`mine` = owned + favorited / `square` = built-in/published public), `page`, `page_size`, returning `{agents, total, page, page_size}`; omitting `scope`/`page` preserves the legacy "return everything" behavior. Name `search` matches the raw OR desensitized name (the frontend's static term?abbrev map is mirrored server-side in `deerflow/config/agent_name_desensitize.py`); `tag_ids` is an SQL OR-filter; `owner` keeps only agents published by that user id (the card's ??? ? built-in agents never match), powering the gallery's click-???-to-filter. `POST /api/agents/private-square/list` takes the same pagination params (incl. `owner`). Repository entry point: `AgentStore.list_paginated(...)`. Tests: `tests/test_agent_listing_cache.py::test_list_paginated_owner_filter_keeps_one_publisher`.
|
||
|
||
**Homepage selector (agent picker on the new-chat page)** ? single admin-curated list backed by `agents.featured_order` (nullable BIGINT). NULL = not featured. Only `published = true` agents can be featured (router enforces on writes); any published agent is eligible regardless of ownership. Cleared with `order: null`. The list rendered to end users is `WHERE featured_order IS NOT NULL` + visibility-filter, sorted by `featured_order ASC`. Admins reorder by sending the desired ids to `PUT /api/agents/featured/order` (assigns indices 0..N-1). BIGINT is used rather than INT so the management UI can seed initial values from `Date.now()` (overflows INT on MySQL).
|
||
|
||
### Publish Squares (????)
|
||
|
||
Admin-configurable, **access-control-free** categories that published agents/skills are filed under (distinct from the password-gated ????). The square list lives in `system_settings.json` ? `publish_squares` (`PublishSquaresSettings`: ordered `squares: [{id,name,description}]` + `default_square_id`); helpers `list_publish_squares` / `resolve_default_square_id` / `normalize_square_id` always yield a valid square (built-in "????" fallback when none configured). Admin CRUD: `GET/PUT /api/system-settings/publish-squares` (GET readable by any user ? the gallery sub-tabs + publish picker need it; PUT admin-only, auto-slugs missing ids and re-anchors a dangling default).
|
||
|
||
Each agent/skill carries a single `square_id` column (`AgentRow` / `SkillRow`, `String(64)` NOT NULL default `''`, migration `20260611_01`, idempotent SQL in `column_additions_mysql.sql`). One column = at most one square, so an item can never be "published twice". `''` = unassigned ? resolved to the default square at read/filter time. Writes normalize via `normalize_square_id`: agents on create/update (`square_id` field), skills on publish (`PUT /api/skills/custom/{name}/published` accepts optional `square_id`). Listing filters: `GET /api/agents?square=` and `GET /api/skills?square=` keep only that square's items; the **default** square additionally sweeps in legacy `''` rows (`square_is_default`). Tests: `tests/test_publish_squares.py`. Frontend: `core/system-settings` (`usePublishSquares`/`useUpdatePublishSquares`), admin `PublishSquaresSettingsPage` (embedded in the appearance/settings hub), square picker in the agent edit dialog + per-skill card, and a ??-tab square sub-filter in both galleries.
|
||
|
||
### Reference-mode Entry Switch
|
||
|
||
The 参考文献/RAG citation-mode composer entry is system-controlled through
|
||
`system_settings.json` → `citation_display.reference_mode_enabled` (default
|
||
`false`). `GET /api/system-settings/citation-display` remains readable so the
|
||
frontend can hide or show the entry consistently; `PUT` is admin-only and
|
||
partially updates `reference_mode_enabled` and/or `cleanup_words`. The normal
|
||
workspace input box, iframe chat toolbar, send context, right reference panel,
|
||
and appearance settings all use this flag, while already locked citation
|
||
threads still render the generic reference panel. A non-empty selected
|
||
LLMWiki/assistant knowledge-space set instead carries an explicit Wiki reference
|
||
contract: the frontend filters every batch to those selected server-authored ids
|
||
and renders a flat Wiki table of contents plus inline Wiki reader, never the
|
||
generic reference-card component. Projects enable this only when their project
|
||
or bound-agent knowledge ids are actually non-empty. Tests:
|
||
`tests/test_prompt_prefix.py` and
|
||
`frontend-web/src/pages/iframe-chat-mode-buttons.test.ts`.
|
||
|
||
### Agent Module Tag Config (zzJC / zzXD 按类型聚合)
|
||
|
||
Backs the zzJC / zzXD agent-chat left sidebar (「按类型聚合智能体 + 聊天记录」). An admin picks which agent **tags** belong to a module. The module's sidebar then shows only agents under those tags (empty/unconfigured = show all visible agents). Stored in `system_settings.json` → `agent_modules` (`AgentModuleSettings`: `modules: {module_key → [tag_id, ...]}`); pure helpers `get_module_tag_ids` / `set_module_tag_ids` (de-dupe + strip). Endpoints (in `app/gateway/routers/system_settings.py`): `GET /api/system-settings/agent-modules/{key}` (readable by any user — the sidebar needs it to filter), `PUT /api/system-settings/agent-modules/{key}` (full replacement; admin only). Module keys are frontend-defined (`AGENT_MODULES` in `components/workspace/agents/agent-module-sidebar.tsx`). Tests: `tests/test_agent_modules.py`. Frontend: `core/system-settings` (`useAgentModuleConfig`/`useUpdateAgentModuleConfig`) and `AgentModuleSettingsDialog`.
|
||
|
||
### wx common-agent category
|
||
|
||
The fixed-deep-blue wx full-chat route (`#/page/workspace/wx/chats/new`) hides the global header and page sidebar while retaining its workspace common-agent panel. When its optional category is unset, the panel preserves the original recommendation logic (pinned → mine → square, deduplicated). A saved category selects one Agent Gallery category: `mine`, `sentiment`, or `square:<square_id>` (for example the configured `默认广场`). It is stored in `system_settings.json` → `wx_common_agents` (`WxCommonAgentsSettings`), validated against the effective publish-square list, and exposed by `GET`/`PUT /api/system-settings/wx-common-agents`. The `wx` account and administrators can update or clear it; all users can read it. The frontend uses `useWxCommonAgents` and displays the settings gear only to wx. Tests: `tests/test_wx_common_agents.py`.
|
||
|
||
### Business-mapping authorization
|
||
|
||
`/api/business-mapping` remains globally readable. Seeded rows include the main codes (3Q / 6BF / 7BF / 8BF / AFX) plus independent slots `3q-0…3q-6`, `6-1…6-4`, and `7-1…7-4` (empty agent/chain until configured on the 配置管理 page). `3q-1…3q-6` seed as `is_custom=true` / `mapping_type=自定义类型` so `goPath=3q-1` resolves as a custom deep link; the configuration page hides those six rows and defaults the type filter to 自定义类型. Updates to 3Q, 6BF, 7BF, and those independent slots require `system_role == "admin"`; the 8BF public business-chain mapping is intentionally different and can be updated only by the trusted authenticated account whose email prefix is `lqq`. Do not rely on a username supplied in an external login URL. The frontend follows the same email-prefix rule when deciding whether to render the 8BF configuration button, while the router (`app/gateway/routers/business_mapping.py`) remains the enforcement point. Tests: `tests/test_business_mapping.py`.
|
||
|
||
### Position-collaboration foundation
|
||
|
||
### Roundtable business-chain enable/disable
|
||
|
||
`roundtable_chains.enabled` (BOOLEAN NOT NULL DEFAULT 1, migration `20260819_02`)
|
||
is the per-chain 启停 switch. New and existing chains default to **enabled**.
|
||
Owner or admin can flip it via `PUT /api/roundtable-chains/{id}` `{enabled}`.
|
||
Disabled chains stay visible in the management list (with an 启用/停用 filter)
|
||
and still show `updated_at`; the frontend hides them from the picker / public
|
||
entry rail so they cannot be selected into a roundtable. Tests:
|
||
`tests/test_roundtable_chain_stages.py` (`test_create_defaults_enabled_then_toggle`).
|
||
|
||
### Roundtable business-chain human validation
|
||
|
||
The existing `/api/roundtable-chains` `seats` JSON also accepts the optional
|
||
boolean `human_validation` field on each seat (`ChainSeatSchema`). It is a
|
||
frontend Step 2 checkpoint, not a server-side approval task: after the marked
|
||
seat completes (or, in a DAG, after its stage completes), the client pauses and
|
||
uses the existing resume button or intervention composer to continue. No new
|
||
database column, migration, or validation card is required.
|
||
|
||
The position-collaboration workspace is a separate frontend route and must not
|
||
use the original roundtable coordinator or jobs. Its shared configuration
|
||
integration point is the existing `/api/roundtable-chains` store: each `seats` JSON item
|
||
may carry an optional `position_id` (validated by `ChainSeatSchema`, no schema
|
||
migration). The original roundtable only reads its established seat fields and
|
||
therefore remains unchanged. The separate `/api/position-roundtable` router
|
||
(also mounted at `/api/multi-agent/position-roundtable` for intranet gateway
|
||
compatibility)
|
||
persists task/intent/frozen-chain snapshots in `position_roundtable_sessions`
|
||
and direct-agent node indexes in `position_roundtable_nodes` (migration
|
||
`20260721_01`). Message history remains in the established LangGraph thread
|
||
store; a node completion unlocks its next chain stage and a later upstream
|
||
revision marks downstream records `stale` for refresh.
|
||
`POST /api/position-roundtable/sessions/{id}/nodes/{node_key}/reject` uses the
|
||
same cascade: its reason is intentionally optional, the selected node becomes
|
||
`rejected`, and downstream material becomes `stale` (or `locked` when no
|
||
material exists) with `invalidated_by` set. Clients must replace their complete
|
||
session snapshot from this response so the flow, delivery list, and active
|
||
node state stay consistent. A rejected/stale node returns to `done` only after
|
||
a fresh Markdown delivery is recorded.
|
||
For direct-agent handoff, the router resolves artifact paths only inside the
|
||
completed upstream node's own thread, then includes bounded UTF-8 excerpts for
|
||
text outputs; binary artifacts stay as metadata-only references. Because each
|
||
seat runs on an isolated LangGraph thread, a downstream `read_file` on the
|
||
upstream artifact path would fail with "File not found"; the handoff therefore
|
||
also **mirrors** every upstream `outputs/` deliverable into the current seat's
|
||
own `workspace/upstream/` directory (`_mirror_upstream_artifacts`, bytes from
|
||
the upstream sandbox file with the SQL recovery copy as fallback, idempotent
|
||
per turn via size comparison, capped at 8 files / 2 MB each / 8 MB total) and
|
||
rewrites each handoff artifact ref's `path` to the mirror (`upstream_path`
|
||
keeps the original; `mirrored_to_sandbox: true`), with a matching note in the
|
||
hidden instructions telling the agent it may `read_file` the full text. The
|
||
mirror deliberately stays outside `outputs/` so it never appears in the seat's
|
||
own delivery manifest.
|
||
`POST /sessions/{id}/archive` turns the session into a read-only historical
|
||
record: task/intent updates, node binding, hidden-context construction, and
|
||
turn completion are rejected, while read endpoints remain available.
|
||
Built-in summary and action-plan completion also preserves a visible report
|
||
answer when the model misses its final `write_file` call: the router writes
|
||
that exact answer as the required Markdown artifact in the owning thread's
|
||
`outputs/` directory, so users receive both chat text and a downloadable file.
|
||
**Per-turn outputs write cap (弱模型防重复产物)**: ordinary position-node chats
|
||
and the summary builtin send run-context `position_artifact_cap: 1` (whitelisted
|
||
in `services.merge_run_context_overrides`). `PositionArtifactCapMiddleware`
|
||
then keeps at most one successful `/mnt/user-data/outputs/` `write_file` **and**
|
||
one `present_files` per user turn — truncates excess tool calls in the same
|
||
model response (and syncs raw provider `additional_kwargs.tool_calls`), rejects
|
||
further writes/presents after an `OK` / `Successfully presented files` or an
|
||
earlier sibling in the same batch, ends the turn when only excess calls remain,
|
||
and reminds the model to present once (or wrap up if already presented).
|
||
**Action-plan does not send the flag** (it must write report `.md` +
|
||
`action-plan-subtasks.json`). Tests: `tests/test_position_artifact_cap.py`.
|
||
|
||
### 岗位 (Position) Subscription System
|
||
|
||
An admin-managed aggregation layer that decides, per user, which agents / skills
|
||
/ scheduled tasks / recommended questions they are subscribed to or can see.
|
||
**A user has at most one position** (`users.position_id`, 1:1); unassigned users
|
||
inherit the single `is_default` position. Note: the admin-configured prompts
|
||
shown to users ARE the **recommended questions** (`recommended_questions` table)
|
||
— there is no separate prompt-template store; the per-user `user_custom_prompts`
|
||
(personal 常用提示词 bar) is untouched by this system.
|
||
|
||
**Persistence** (`deerflow/persistence/positions/`, wired as
|
||
`app.state.position_store`; tables auto-created by `create_all`, migration
|
||
`20260616_01`):
|
||
- `positions` (`PositionRow`): `id`, `name` (unique), `description`,
|
||
`is_default` (only one at a time — the store clears the flag on others).
|
||
- `position_tag_bindings` (`PositionTagBindingRow`): one row per
|
||
`(position, resource_type, tag)`. **No binding for a resource_type = "use the
|
||
default set".** `POSITION_RESOURCE_TYPES = (agent, skill, scheduled_task,
|
||
recommended_question)` — the same set drives the tag types (`TAG_TYPES` was
|
||
extended with `recommended_question`).
|
||
|
||
**Sync engine** (`deerflow/services/position_sync.py`, wired as
|
||
`app.state.position_sync`). `sync_user(user)` **materializes** three resource
|
||
types into the existing "我的" tables, idempotently:
|
||
- agents → `agent_favorites` rows with `origin='position'`
|
||
- **skills → `skill_favorites` rows with `origin='position'`** (table
|
||
`SkillFavoriteRow`, migration `20260616_04`, auto-created by `create_all`)
|
||
- scheduled tasks → `scheduled_task_subscriptions` rows with `origin='position'`
|
||
|
||
Skills behave **exactly like agents**: a position only *adds* its tagged skills
|
||
to the member's 我的技能 — the browse-all 广场 is never restricted. (This replaced
|
||
the earlier query-time visibility filter, which wrongly shrank the square to the
|
||
bound skill.) `PositionSyncService.resolve_visible(user, type)` returns
|
||
`set | None` (None = unrestricted) and is now used **only** for the one remaining
|
||
query-time type, `recommended_question`.
|
||
|
||
Sync only ever adds/removes `origin='position'` rows, so a user's own
|
||
favorites/subscriptions are never clobbered. Default sets (no binding for a
|
||
type): agents = built-ins (`user_id IS NULL`); skills = empty (square stays
|
||
full, nothing added to 我的); scheduled tasks = empty; questions = **untagged
|
||
only** (默认岗位 / 无标签绑定 只看到未打标的题目；打标的题目只在绑定了对应标签的
|
||
岗位展示 — the router interprets `resolve_visible → None` as "show only untagged",
|
||
not "show everything").
|
||
`origin`/`position_id` columns were added to `agent_favorites` and
|
||
`scheduled_task_subscriptions` — see migration `20260616_01` +
|
||
`scripts/sql/column_additions_mysql.sql`. The running app also auto-adds these
|
||
via `_ensure_orm_columns_sync` on startup.
|
||
|
||
**Triggers**: assigning/removing members, changing tag bindings (re-syncs all
|
||
members), becoming the default position (`sync_default_members`), and login
|
||
(`auth.py::_sync_user_position`, best-effort so a brand-new user inheriting the
|
||
default gets their grants materialized).
|
||
|
||
**API** (`app/gateway/routers/positions.py`, `/api/positions`, **admin only**):
|
||
position CRUD; `GET/PUT /{id}/tag-bindings`; `GET/POST/DELETE /{id}/members`
|
||
(batch assign/unassign → triggers sync); `POST /{id}/resync`;
|
||
`GET /users` (all users + their position); `PUT /users/{user_id}` (set one
|
||
user's position). **`GET /me` is the one non-admin route** — it returns the
|
||
caller's own effective position (explicit assignment, else the default position
|
||
unassigned users inherit, else `null`); powers the top-bar 岗位 label
|
||
(`strategy-components/Header.tsx` user cluster, via `useMyPosition`). Related
|
||
changes: `GET /api/recommended-questions` is now
|
||
filtered by the caller's position (`?all=1` bypasses for admin management) and
|
||
admin can tag questions via `/api/tags/assignments/recommended_question/{id}`.
|
||
**Default-position rule**: tags are bulk-attached to **all** questions first,
|
||
then the visibility filter applies — a position binding recommended_question 标签
|
||
sees only those tagged questions; a 默认岗位 / 无绑定 (`resolve_visible → None`)
|
||
sees only the **untagged** questions (打标的题目不再泄漏到默认岗位). `?all=1`
|
||
(admin management) skips this filter and returns every question with its tags.
|
||
The list also accepts a repeatable `tag_ids`
|
||
OR-filter — so the 推荐问题配置 page can display per-question 标签 and search by
|
||
them (frontend `RecommendedQuestionsPage` + `TagSearchBar`). Tests:
|
||
`tests/test_recommended_questions_tags.py`.
|
||
`GET /api/skills` no longer restricts by position; each skill carries a
|
||
`favorited` flag (true = in the caller's 我的技能, e.g. position-granted) so the
|
||
skills page's 我的 tab includes owned ∪ position-granted skills while the 广场
|
||
stays complete. Frontend: `core/positions`
|
||
+ admin `PositionManagementPage` (侧边栏「综合管理 → 岗位管理」 + 右下角设置菜单).
|
||
Recommended questions stay rendered **above** the chat input (their original
|
||
position), just filtered by the caller's position. Tests:
|
||
`tests/test_positions.py`.
|
||
|
||
### Portable Column Types (`deerflow/persistence/types.py`)
|
||
|
||
Two `TypeDecorator`s keep ORM columns correct across the `sqlite` / `postgres` / `mysql` backends. **All ORM timestamp/JSON columns must use these, not the raw SQLAlchemy types.**
|
||
|
||
- **`PortableJSON`** ? native `JSON` on SQLite/Postgres; `TEXT` (with manual `json.dumps`/`loads`) on MySQL, where older servers lack JSON support.
|
||
- **`PortableLongText`** ? a large text column: maps to MySQL `LONGTEXT` (4 GB) and plain `TEXT` on SQLite/Postgres (already unbounded). Use for big JSON/markdown blobs that would overflow MySQL's 64 KB `TEXT` cap ? e.g. `ai_writing_sessions.transcript`.
|
||
- **`BeijingDateTime`** ? stores timestamps as Beijing (UTC+8) wall-clock time. The app records time with `datetime.now(UTC)`, but MySQL `DATETIME` / SQLite have no timezone and persist whatever wall clock the driver serializes ? so a raw `DateTime` column would store UTC and read 8 hours behind Beijing. `BeijingDateTime` converts UTC?Beijing on bind and re-attaches `+08:00` on result, so the database holds Beijing time while Python still gets timezone-aware datetimes (instant comparisons stay correct, including in WHERE clauses). PostgreSQL `timestamptz` stores the instant unambiguously, so values pass through unchanged there. Uses a fixed UTC+8 offset (China has no DST), so no `tzdata` dependency.
|
||
|
||
### Schema Migrations / Manual Column-Add SQL (`scripts/sql/`)
|
||
|
||
Gateway startup runs Alembic through `app.gateway.app._run_db_migrations` after ORM table creation. The migration graph must always have unique revision ids and exactly one head; `tests/test_migration_graph.py` enforces both. Historical duplicate revision `20260821_01` is repaired by the `20260821_03` / `20260821_04` fixed-question chain and merge revision `20260825_01`. The duplicate Wiki revisions `20260916_01` / `20260920_01` are normalized by merge revision `20260916_03` and final reconciliation revision `20260920_02`; the latter must inspect the actual schema and idempotently ensure Wiki edges plus retrieval/incremental-sync columns so old databases stamped by either duplicate variant converge safely. Startup DDL has a bounded timeout and failures are logged without preventing the app from serving.
|
||
|
||
**Convention (required):** whenever code adds a field to an existing table (a new `mapped_column` on an ORM model), you must, in the same change:
|
||
1. Keep the matching alembic version file under `persistence/migrations/versions/` (guarded with `_has_table`/`_has_column`), and
|
||
2. Append the idempotent `ADD COLUMN` statement to **`scripts/sql/column_additions_mysql.sql`**, with a definition matching the ORM column exactly (type, length, nullability).
|
||
|
||
Operators upgrading an existing DB may run the whole file (`USE <db>;` then execute) as a recovery path; it is idempotent (existing columns are skipped via `information_schema.columns`). Index-only changes go in the sibling `scripts/sql/leaderboard_indexes_mysql.sql` (same idempotent pattern) and stay out of startup migrations because online index creation on large metric tables can delay service boot.
|
||
|
||
Leaderboard scheduled-run exclusion uses correlated `NOT EXISTS` against the
|
||
indexed `scheduled_task_runs.agent_run_id`; do not restore `NOT IN`. Legacy
|
||
scheduler detection for tool metrics is also a correlated existence check and
|
||
must not outer-join the complete `runs` table into metric aggregations. Keep the
|
||
created-at + dimension compound indexes in ORM metadata and the idempotent
|
||
MySQL index script synchronized.
|
||
|
||
### Sandbox System (`packages/harness/deerflow/sandbox/`)
|
||
|
||
**Runtime performance:** `AppConfig.runtime_performance` owns four independent
|
||
flags. `skill_view_reuse_enabled` and `skill_view_cache_enabled` default to true
|
||
when the keys are absent; set either to false and restart to restore a full
|
||
skill-view refresh on every tool call. `model_timing_nonblocking_enabled` and
|
||
`asyncio_thread_pool_enabled` stay false unless set. If application configuration
|
||
cannot be loaded, every switch stays off. `sandbox/skill_view_scope.py` wraps synchronous sandbox tools with a
|
||
per-invocation `ContextVar` snapshot, reset in `finally`. Reuse covers provider
|
||
initialization, path authorization from the discovered `enabled_roots`, command
|
||
translation and output masking; failed refreshes must not be retried by error
|
||
masking. A new tool call must validate again, including after skill offlining.
|
||
The optional `skill_view.py` cache validates fresh extensions bytes plus known
|
||
canonical/view file and directory metadata without the refresh lock on warm hits
|
||
of `refresh_skill_sandbox_view` itself. Sandbox tools do not use that full check
|
||
on the hot path: a bash command that does not name `/mnt/skills` does not stat
|
||
the catalogue; a command or read that names a skill reconciles only that skill
|
||
and any support library it declares. Skill-management file saves, deletions,
|
||
online/offline changes, skill deletion, and skill-address apply/rollback sync
|
||
that skill into the view immediately and publish the new extensions hash, so the
|
||
next command stays on the narrow path. Agent `skill_manage` create, edit, patch,
|
||
and delete do the same for the sandbox view and, when the enabled-skills prompt
|
||
cache is already warm, replace only that skill in the cache. A cold prompt cache
|
||
still reloads the full catalogue. Creating a skill stats `custom/<name>/SKILL.md`
|
||
instead of parsing every skill. If the model omits `content`, create reads a
|
||
`write_file` path under the thread workspace instead of failing the turn.
|
||
`write_file` and `remove_file` sync that skill's view and leave the prompt cache unchanged.
|
||
Creating an agent calls `skill_list` / `search_skills`; when this cache is warm those
|
||
tools reuse it instead of parsing every SKILL.md again, and `skill_view` reads only
|
||
the named skill. A cold cache still loads the catalogue, then checks the canonical
|
||
directory before a full walk.
|
||
A repeated-tool warning is appended to the assistant tool-call message so the
|
||
tool result still follows it; a user message between them makes the next model
|
||
request fail. A missing view or an extensions hash the
|
||
sync did not publish still falls back to the full rebuild. Direct edits outside
|
||
those writes stay stale until the next full rebuild. Offline skills are removed
|
||
from the view at disable time, which is what keeps a hidden shell path from
|
||
running them. Diagnostic samples are process-local and are appended only
|
||
after the sandbox tool call returns: each row stores the command text and the
|
||
skill names invoked by that call. The admin page is `/page/workspace/admin/skill-timing`
|
||
(`GET /api/admin/skill-view-diagnostics`). Recording never runs while the model
|
||
is still streaming, and a failed write cannot change the tool result.
|
||
Overlapping warm validations with the same cache entry and fresh extensions hash
|
||
are process-local single-flight; the flight is removed before followers wake, so a
|
||
later call validates again and a different extensions hash never joins. It is not
|
||
a TTL: mutations, support-library changes and target tampering invalidate the
|
||
snapshot; only rebuilds take the cross-process lock and publish the marker and
|
||
metadata manifest. Keep fixed mount paths, fail closed on invalid availability,
|
||
and never follow a target symlink while repairing or clearing the capability.
|
||
`SKILL_VIEW_CACHE` provides lock-wait/refresh/total timing and entry counts, not
|
||
skill contents. The full metadata check is O(number of known entries), not O(1);
|
||
the tool hot path above does not run it.
|
||
Model timing callback opt-in uses `run_inline=True` only for lightweight timing
|
||
and bounded enqueue; slow logging handlers run on a dedicated daemon, never on
|
||
the event loop or default tool executor. Queue overflow is best-effort diagnostic
|
||
loss with a `LLM_TIMING_LOG_DROPPED` summary. Other callbacks remain independent.
|
||
Regressions live in `packages/harness/tests/test_runtime_performance_config.py`,
|
||
`test_skill_view_tool_reuse.py`, `test_skill_view_cache.py`, and
|
||
`test_model_timing_nonblocking.py`. The first three flags do not increase worker
|
||
or thread counts. The fourth enables Gateway default-executor tuning in
|
||
`app/gateway/runtime_executor.py`, called from lifespan after config/logging setup
|
||
and before `langgraph_runtime`. Resolve nonempty `DEER_FLOW_ASYNCIO_THREAD_POOL_SIZE`
|
||
first, `DEERFLOW_IO_THREADS` second, and YAML `asyncio_thread_pool_size` last
|
||
(default 100, integer range 1..1024). Disabled means do not touch the executor,
|
||
even if an environment override is present. An invalid selected environment
|
||
value fails startup rather than silently reverting to a small pool. Log the
|
||
effective value/source/pid through `GATEWAY_ASYNCIO_EXECUTOR`; the AppConfig log
|
||
shows only the YAML value, not the final environment override. Uvicorn's owning
|
||
event loop cleans up the executor after runtime teardown: do not prematurely
|
||
shutdown it in lifespan, and never schedule its shutdown on itself with to_thread.
|
||
This does not modify AnyIO's separate pool, subagent event loops/executors,
|
||
Gateway worker count or upstream model quotas. Tests:
|
||
`packages/harness/tests/test_gateway_runtime_executor.py`. Restart after thread
|
||
pool changes; model timing instances also need restart for consistent behavior.
|
||
|
||
**Interface**: Abstract `Sandbox` with `execute_command`, `read_file`, `write_file`, `list_dir`
|
||
**Provider Pattern**: `SandboxProvider` with `acquire`, `get`, `release` lifecycle
|
||
**Implementations**:
|
||
- `LocalSandboxProvider` - Singleton local filesystem execution with path mappings
|
||
- `AioSandboxProvider` (`packages/harness/deerflow/community/`) - Docker-based isolation
|
||
|
||
**Virtual Path System**:
|
||
- Agent sees: `/mnt/user-data/{workspace,uploads,outputs}`, `/mnt/skills`
|
||
- Physical: `backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/...`, `deer-flow/skills/`
|
||
- Translation: `replace_virtual_path()` / `replace_virtual_paths_in_command()`
|
||
- Detection: `is_local_sandbox()` checks `sandbox_id == "local"`
|
||
- Configured mount portability: `AppConfig._resolve_relative_mount_paths` resolves each relative `sandbox.mounts[*].host_path` against the loaded `config.yaml` directory before validation/provider construction, never against the process cwd. Keep repository-owned knowledge mounts relative (for example `knowledge/knowledge_base`); absolute paths remain unchanged, and sandbox providers always receive an absolute path.
|
||
- **`{user_id}` for path resolution = `resolve_path_user_id(thread_id)`, using the persisted creator for every tracked thread.** The route's regular owner check still authorizes access first; creator resolution only selects the stable physical directory (`threads_meta.user_id`) and avoids a lost request context falling back to `users/default`. This also preserves cross-account task-deeplink conversations (rwfx/3qfx/xdfx) and roundtable system threads. Positive creator mappings are cached process-wide and DB/memory/missing-row failures fall back to the current login. It is wired at the thread-path consumers including `ThreadDataMiddleware.before_agent`, `present_file_tool`, uploads, artifact serving, and embedded `client.get_artifact`. **Rewrite-history recovery is metadata-only and job-owner scoped** (`list_by_session(..., user_id=current_user)`), so its GET/list endpoints intentionally skip redundant thread-owner checks and return an empty history instead of a history-only 404; the start/commit endpoints retain strict thread write authorization. Tests: `tests/test_thread_paths_owner.py`, `tests/test_thread_paths_roundtable.py`.
|
||
|
||
- **Durable document-rewrite commit:** `document_rewrite_pipeline.atomic_write_text` writes through a deliberately short sibling temporary filename so long Windows artifact paths stay below the path-length limit, and guarantees the target `outputs` parent exists. If the sandbox recreates that directory in the narrow commit window, it repairs the directory and retries; the original file is otherwise only changed through `os.replace`.
|
||
|
||
- **Deep Research structural regeneration:** `POST /api/deep-research/sessions/{id}/report-variants` creates a distinct completed session from the source report and its selected evidence. It rebases both saved `[来源：source_id]` and legacy streamed `[[source:source_id]]` markers to cloned ids, and removes inherited markers with no selected clone. The frontend uses this endpoint only for “选择结构再生成”, then queues the durable writer with `operation=generate`; generation receives the confirmed outline and complete selected evidence set but never receives the inherited report as text to edit. The new report's completed card can subsequently start a separate in-place `AI 全文改写`. Requirement plans are deterministic and operation-specific, model reasoning is forwarded as live SSE before the first Markdown token, and generation prompts emit display-ready `[来源：id]` citations. Citation validation repairs only a unique near-complete `drs_src_*` prefix (provider-truncated suffix) and still rejects genuinely unknown ids. The report-only stream requests 8192 output tokens (except Codex Responses models, which reject token-limit kwargs) and rejects a malformed trailing citation instead of committing a provider-truncated report. Never restore the former second planning-model call unless its latency and streaming contract are redesigned together.
|
||
|
||
**Sandbox Tools** (in `packages/harness/deerflow/sandbox/tools.py`):
|
||
- `bash` - Execute commands with path translation and error handling
|
||
- `ls` - Directory listing (tree format, max 2 levels)
|
||
- `read_file` - Read file contents with optional line range
|
||
- `write_file` - Write/append to files, creates directories
|
||
- `str_replace` - Substring replacement (single or all occurrences); same-path serialization is scoped to `(sandbox.id, path)` so isolated sandboxes do not contend on identical virtual paths inside one process
|
||
|
||
### Subagent System (`packages/harness/deerflow/subagents/`)
|
||
|
||
**Built-in Agents**: `general-purpose` (all tools except `task`) and `bash` (command specialist)
|
||
**Execution**: Dual thread pool - `_scheduler_pool` (3 workers) + `_execution_pool` (3 workers)
|
||
**Concurrency**: `MAX_CONCURRENT_SUBAGENTS = 3` enforced by `SubagentLimitMiddleware` (truncates excess tool calls in `after_model`), 15-minute timeout
|
||
**Flow**: `task()` tool ? `SubagentExecutor` ? background thread ? poll 5s ? SSE events ? result
|
||
**Events**: `task_started`, `task_running`, `task_completed`/`task_failed`/`task_timed_out`
|
||
|
||
### Tool System (`packages/harness/deerflow/tools/`)
|
||
|
||
`get_available_tools(groups, include_mcp, model_name, subagent_enabled)` assembles:
|
||
1. **Config-defined tools** - Enabled entries are resolved from `config.yaml` via `resolve_variable()`; entries with `enabled: false` are omitted before import and are not registered with the agent
|
||
2. **MCP tools** - From enabled MCP servers (lazy initialized, cached with mtime invalidation)
|
||
3. **Built-in tools**:
|
||
- `present_files` - Make output files visible to user (only `/mnt/user-data/outputs`)
|
||
- `ask_clarification` - Request clarification (intercepted by ClarificationMiddleware ? interrupts)
|
||
- `view_image` - Read image as base64 (added only if model supports vision)
|
||
4. **Subagent tool** (if enabled):
|
||
- `task` - Delegate to subagent (description, prompt, subagent_type, max_turns)
|
||
|
||
**Community tools** (`packages/harness/deerflow/community/`):
|
||
- `tavily/` - Web search (5 results default) and web fetch (4KB limit)
|
||
- `jina_ai/` - Web fetch via Jina reader API with readability extraction
|
||
- `firecrawl/` - Web scraping via Firecrawl API
|
||
- `configurable_search/` - Config-driven `POST` JSON `web_search` replacement. It deep-copies the configured payload and overwrites only `query`, supports HTTP/HTTPS plus configurable `verify_ssl`, timeout and result limit, normalizes nullable `recUuid`/`content`/`title`/`url` fields, and rewrites a result URL through `result_url_template` only when `recUuid` is non-empty. The required response envelope is `{"results": [...]}`. Tests use `httpx.MockTransport` and never require the deployment endpoint.
|
||
|
||
**ACP agent tools**:
|
||
- `invoke_acp_agent` - Invokes external ACP-compatible agents from `config.yaml`
|
||
- ACP launchers must be real ACP adapters. The standard `codex` CLI is not ACP-compatible by itself; configure a wrapper such as `npx -y @zed-industries/codex-acp` or an installed `codex-acp` binary
|
||
- Missing ACP executables now return an actionable error message instead of a raw `[Errno 2]`
|
||
- Each ACP agent uses a per-thread workspace at `{base_dir}/users/{user_id}/threads/{thread_id}/acp-workspace/`. The workspace is accessible to the lead agent via the virtual path `/mnt/acp-workspace/` (read-only). In docker sandbox mode, the directory is volume-mounted into the container at `/mnt/acp-workspace` (read-only); in local sandbox mode, path translation is handled by `tools.py`
|
||
- `image_search/` - Image search via DuckDuckGo
|
||
|
||
### MCP System (`packages/harness/deerflow/mcp/`)
|
||
|
||
- Uses `langchain-mcp-adapters` `MultiServerMCPClient` for multi-server management
|
||
- **Lazy initialization**: Tools loaded on first use via `get_cached_mcp_tools()`
|
||
- **Cache invalidation**: Detects config file changes via mtime comparison
|
||
- **Transports**: stdio (command-based), SSE, HTTP
|
||
- **OAuth (HTTP/SSE)**: Supports token endpoint flows (`client_credentials`, `refresh_token`) with automatic token refresh + Authorization header injection
|
||
- **Runtime updates**: Gateway API saves to extensions_config.json; LangGraph detects via mtime
|
||
|
||
### Skills System (`packages/harness/deerflow/skills/`)
|
||
|
||
- **Cognitive situation-map runtime parity**: `cognitive-situation-map` has a
|
||
source runtime under `skills/public/cognitive-situation-map/assets/runtime/`
|
||
and a Gateway-served copy under
|
||
`packages/harness/deerflow/html_runtime/assets/cognitive-situation-map/`.
|
||
Keep their JavaScript, schema, and map atlas assets in sync. `template.aspect`
|
||
locks a new page to `16:9` (1920×1080, the default when unspecified) or
|
||
`32:9` (2560×720, explicit ultra-wide only); pages without the field keep
|
||
the older viewport-adaptive layout. Built-in regions
|
||
include Taiwan Strait, South China Sea, the Philippines-centred surrounding
|
||
region (with its southern Sulu/Celebes approaches), First Island Chain, and
|
||
the U.S.--Pacific connection; every region is a four-quadrant atlas for the
|
||
terrain / maritime / tactical / briefing variants. `scripts/render.mjs`
|
||
permits a direct input `scenario.json` as the sole item in a fresh output
|
||
directory; all other non-empty output directories need a new path or explicit
|
||
overwrite authorization.
|
||
|
||
- **Location**: `deer-flow/skills/{public,custom}/`
|
||
- **Format**: Directory with `SKILL.md` (YAML frontmatter: name, description, license, allowed-tools). `name` and `description` are the minimal metadata contract; the frontend install dialog renders the shared `MINIMAL_SKILL_MD_EXAMPLE` containing only those two keys so optional fields are not mistaken for requirements. Keep `frontend-web/src/core/skill-example.test.ts` aligned with backend validation.
|
||
- **Loading**: `load_skills()` recursively scans `skills/{public,custom}` for `SKILL.md`, parses metadata, and reads enabled state from extensions_config.json. **A directory containing `SKILL.md` is a self-contained leaf** — discovery does not descend into its subdirectories, so a skill *package* that bundles its own `examples/`/`references/` sub-skills (each with a SKILL.md) surfaces as **one** skill, not a swarm of phantom entries.
|
||
- **Built-in seed source**: Product-owned defaults live under `app/gateway/routers/_*_seed_assets/skills/` and must be versioned even though runtime `skills/` trees are ignored. Seeders copy only when a target directory is absent and never overwrite administrator changes. Internal support packages such as `task-report-lib` deliberately omit `SKILL.md` and declare dependents in `.deerflow-skill-support.json`; keep `tests/test_seed_assets.py` and `tests/test_task_report_convert.py` synchronized with the shipped inventory and contracts.
|
||
- **Online/offline lifecycle**: reuse `Skill.enabled`; do not add a parallel lifecycle flag and do not substitute `published` or curator `state`. `PUT /api/skills/{name}` is a global switch: administrators may toggle every skill, while an ordinary user may toggle only a non-built-in `custom` skill whose persisted `owner_user_id` matches that user. It patches only `skills.<name>.enabled` in the raw `extensions_config.json`, preserving MCP placeholders and unrelated/unknown fields, then reloads configuration and refreshes the lead prompt cache. Writes use atomic replacement on ordinary files and an `fsync` in-place fallback for writable Docker single-file bind mounts; supported launchers instead keep the runtime file under a writable persisted config directory. Offline skills remain on disk and in the full catalogue (`load_skills(enabled_only=False)`), and existing agent/workflow references are deliberately retained so bringing the skill online restores them. The per-skill incremental view sync must treat a missing destination as stale even when its target-cache entry is also absent, so re-enabling recopies the latest canonical package (including `SKILL.md` and nested helpers edited while offline) after the published copy was removed. Every model-facing path must use the enabled catalogue: prompt injection, `search_skills`, `skill_list`, `skill_view`, subagents, AI-writing/deep-research/roundtable discovery, and Workflow run-start resource validation. Direct sandbox access is enforced by a copied online-only view under writable `DEER_FLOW_HOME`; `/mnt/skills` in LocalSandbox and AIO/remote sandboxes must point only to that view, never to the canonical tree. AIO creation/reuse attests the view marker and fails closed when an older provisioner ignores the requested mount; explicitly declared internal support libraries (such as `task-report-lib`) enter the view only while a dependent skill is online. Path-sensitive `DEER_FLOW_*` variables and `PYTHONPATH` are removed from LocalSandbox bash while non-path settings such as `DEER_FLOW_GATEWAY_PORT` remain. LocalSandbox host bash still shares the backend UID and is not an OS isolation boundary; production requiring hostile-code isolation must use an attested AIO sandbox. Frontend controls are in the administrator 技能管理 list and, for owners, 「技能 → 技能列表 → 我的」; offline rows and retained agent references remain visible with an 已下线 badge and can be removed, while new selectors exclude them.
|
||
- **Discovery↔delete consistency**: a skill is keyed for discovery by its frontmatter `name`, but its on-disk directory name may differ (hand-dropped / mis-extracted package). `LocalSkillStorage._resolve_skill_dir(name, category)` resolves the *discovered* directory (canonical `<category>/<name>` fast-path, else scan-by-frontmatter-name); `custom_skill_exists`/`read_custom_skill`/`delete_custom_skill` all go through it, so **anything that gets listed can also be read and deleted** regardless of directory name (the old code assumed `custom/<name>/` and 404'd "Custom skill 'x' not found." on any non-canonical layout). Delete is path-guarded to never escape the `custom/` root. Tests: `tests/test_skill_storage_delete.py`.
|
||
- **Injection**: Enabled skills listed in agent system prompt with container paths
|
||
- **Configurable `es_query` routing prompt**: `config.yaml → skills.es_query_routing`
|
||
contains an `enabled` switch and editable `prompt`. When enabled,
|
||
`lead_agent.prompt._build_es_query_routing_section` injects the rule into the
|
||
default lead-agent system prompt. Custom-agent-only prompts receive it only
|
||
when their explicit skill allowlist contains `es_query`; disabling the switch
|
||
or leaving the prompt blank emits no block. Immediately before an ordinary Q&A
|
||
graph is created, `lead_agent.agent._log_es_query_routing_registration` checks
|
||
the exact configured block against the final system-prompt string and logs
|
||
`enabled`, allowlist eligibility, `registered`, character count, and a short
|
||
SHA-256 fingerprint (never the prompt body). Tests:
|
||
`tests/test_es_query_routing_prompt.py`.
|
||
- **Installation**: `POST /api/skills/install` extracts .skill ZIP archive to custom/ directory. Direct uploads use the ownership-aware `POST /api/skills/validate?target_category=...` preflight plus `POST /api/skills/install-upload`: a same-owner custom conflict may be uploaded again with `overwrite=true`, while a different owner's conflict is non-overwritable and returns that owner's username (falling back to user id). The install endpoint repeats the ownership check, including for admins targeting `custom`, so callers cannot bypass the preflight. Public-skill overwrite remains admin-only.
|
||
- **Installable sentiment analysis package**: `skill-packages/sentiment-analysis.skill` is deliberately separate from the frontend virtual-agent chat page. The chat page goes through Gateway `POST /api/sentiment-agent/stream` (runtime-config URL/Basic Auth in the body, TLS verify off, raw AG-UI SSE passthrough). The skill package's self-contained `config/sentiment-agent.json` owns the external AG-UI address for agent-skill callers; Basic Auth is resolved from `SENTIMENT_AGENT_AUTH_USERNAME` / `SENTIMENT_AGENT_AUTH_PASSWORD` at runtime (not committed into the package). `scripts/sentiment_stream.py` uses only the Python standard library and converts upstream SSE to stable NDJSON events (`thinking_delta`, `answer_delta`, `done`), with a Markdown convenience mode for normal agent calls. Installation/calling guide: `skill-packages/sentiment-analysis/docs/调用与安装说明.md`.
|
||
- **Chinese name (`name_zh`)**: a Chinese display name stored in the **`skills` DB table** (`SkillRow.name_zh`, `String(128)`, migration `20260609_01`) ? never written to the on-disk SKILL.md, so it works for built-in skills too. Threaded through `SkillStore` create/ensure_legacy/update/update_admin exactly like `detail`; surfaced on `SkillResponse.name_zh` from the ownership record in `_skill_to_response`. Authorization mirrors `detail`: custom skill ? owner or admin; built-in/public ? admin only (row materialized on demand via `ensure_legacy`). Endpoints (in `app/gateway/routers/skills.py`): `PUT /api/skills/{name}/name-zh` (set/clear), `POST /api/skills/{name}/name-zh/generate` (LLM generates one), `POST /api/skills/name-zh/generate-missing` (????: **admin-only** bulk-generate for every skill that lacks one, bounded-concurrency `asyncio.gather`). **Optional model selection**: both generate endpoints accept an optional JSON body `{model_name}` ? an explicit name wins, else the curator model setting, else the main model (`_resolve_skill_name_zh_model_name`). **Model-error surfacing**: `_generate_skill_name_zh` raises `SkillNameZhModelError` when the model itself fails (build/timeout/API error) ? distinct from the model returning an unusable name (? `None`). The single endpoint maps that to `502 ?????,?????????:?`; the bulk endpoint fails fast on a model-build error (502) and otherwise carries a representative `model_error` string in `SkillNameZhBulkResponse` so the UI can say "N ?????????". The frontend (`skill-settings-page.tsx` list + `skill-detail-dialog.tsx` editor, plus the agent skill pickers/badges in `NewAgentPage.tsx` + `agent-card.tsx`) shows `name_zh || name`, offers per-skill "AI ??" plus an **admin-only** header "???????" button, and a compact model dropdown (`components/workspace/model-name-select.tsx`, default = configured model) beside each action.
|
||
|
||
#### Skill Address Management (技能地址管理 — URL/IP 批量替换)
|
||
|
||
Admin-only maintenance tool to find every `http(s)://` URL and IPv4 (`ip[:port]`)
|
||
address in the **running** skills (`skills/{public,custom}`, all `.md` + `.py`,
|
||
recursing into `scripts/`/`references/`/`templates/`) and rewrite them in bulk —
|
||
e.g. swapping a UAT endpoint for production across all skills at once. The same
|
||
address commonly lives in both a skill's `SKILL.md` and its `scripts/*.py`, so
|
||
results are **aggregated by unique address** with every occurrence (`file:line` +
|
||
snippet) and replaced together.
|
||
|
||
- **Pure engine** — `deerflow/skills/address_scan.py` (harness, no `app.*`):
|
||
`find_spans` / `scan_occurrences` (URL regex with trailing-punctuation strip;
|
||
IPv4 with 0–255 octet check + optional port; an IP inside a URL is *not*
|
||
separately surfaced), `apply_replacements` (**substring-safe**: re-detects exact
|
||
token spans and only rewrites a span whose text equals a requested `from`, so
|
||
`1.2.3.4` is never rewritten inside `11.2.3.40`), `infer_kind`,
|
||
`is_valid_replacement`. Tested in `tests/test_skill_addresses.py`.
|
||
- **Router** — `app/gateway/routers/skill_addresses.py` (`/api/skill-addresses`,
|
||
**admin only**). **Availability at scale (150+ skills):** every filesystem
|
||
walk/read/write runs via `asyncio.to_thread` so the event loop is never blocked.
|
||
Routes: `GET /scan` (aggregated, one-shot) + `GET /scan/stream` (per-skill NDJSON
|
||
progress `{type:start|progress|result}`); `POST /preview` (dry-run diff +
|
||
validation warnings); `POST /apply` + `POST /apply/stream` (per-file NDJSON
|
||
progress); `GET /backups`; `POST /rollback/{backup_id}`. **Apply safety:**
|
||
optimistic lock via `expected_total` (409 if files changed since the scan),
|
||
path-guard (`is_relative_to` the skills root), auto disk backup to
|
||
`{skills_root}/.address-edits/{timestamp}/` (the `.`-prefixed dir is excluded
|
||
from scans), then `refresh_skills_system_prompt_cache_async()` so edits take
|
||
effect immediately. Registered in `app.py` after `sensitive_words.router`.
|
||
**Per-edit history:** each apply also writes a sibling manifest
|
||
`{skills_root}/.address-edits/{timestamp}.manifest.json` (best-effort) capturing
|
||
`{created_at, replaced_total, replacements:[{from,to,kind}], changed_files}`; the
|
||
sibling-not-child placement keeps it out of both the restore set and the file
|
||
count. `GET /backups` returns each record enriched with `replaced_total` +
|
||
`replacements` (from the manifest; falls back to dir mtime when absent), and
|
||
`POST /rollback/{backup_id}` restores a single record — so the UI shows a full
|
||
modification history and rolls back any one entry independently.
|
||
- **Frontend** — the panel is the reusable `SkillAddressesPanel` (exported from
|
||
`pages/SkillAddressesPage.tsx`; its default export keeps the standalone route
|
||
`/page/strategy/admin/skill-addresses` as a direct-link fallback). It is
|
||
**embedded as a「技能地址管理」tab inside the admin「技能管理」page**
|
||
`pages/AdminSkillsPage.tsx` (Tabs: 技能列表 + 技能地址管理; reached from the
|
||
「技能管理」item in the「设置和更多」dropdown `workspace-nav-menu.tsx` →
|
||
`/page/workspace/admin/skills`) — it is no longer a separate entry in the
|
||
`components/page-sidebar.tsx` settings dropdown. The panel:
|
||
auto-scan on open with a **progress bar** (扫描逐技能 / 应用逐文件 X/N), URL/IP
|
||
type filter, search, a「片段批量套用」helper (e.g. all `uat.4.cn` → `prod.4.cn`),
|
||
per-address「替换为」input with expandable occurrences, inline 预览 panel, 应用,
|
||
**multi-keyword search** (space-separated, all-must-hit), and a **修改记录**
|
||
(modification history) panel — each apply is one record (time / which addresses
|
||
changed from→to / file & replacement counts) with a per-record 回退此次
|
||
(single-record rollback, with inline confirm). Client + NDJSON stream parsing in
|
||
`strategy-components/api/skill-addresses.ts`. Scope is the running skills only;
|
||
the `deploy/minimal/skills/` copy is not touched (sync separately).
|
||
|
||
#### Skill Compression (技能压缩)
|
||
|
||
When the skill catalogue grows large, injecting every enabled skill full-text
|
||
into the lead agent's system prompt bloats it and wastes tokens. **Skill
|
||
compression** is an admin-gated mode that keeps only *常驻(免压缩)* skills in the
|
||
prompt body and surfaces the rest on demand via keyword retrieval.
|
||
|
||
- **Master switch** — `system_settings.json → skill_compression`
|
||
(`SkillCompressionSettings`: `enabled` default **false**, `search_top_n`
|
||
default 5, `keep_index_in_prompt` default true). Read/written by
|
||
`GET`/`PUT /api/system-settings/skill-compression` (GET open to any user — the
|
||
skill detail page reads it; PUT admin-only, refreshes the prompt cache).
|
||
- **Per-skill 常驻 flag** — stored in the **DB**: `SkillRow.always_on`
|
||
(`Boolean`, migration `20260616_03`, idempotent SQL in
|
||
`column_additions_mysql.sql`), **not** in `extensions_config.json` (the
|
||
source-tree config file is read-only / unsafe to rewrite in some deployments,
|
||
and built-in skills carry no file entry). Threaded through `SkillStore`
|
||
create/ensure_legacy/update/update_admin + `list_always_on_names()` exactly
|
||
like `name_zh`/`detail`; surfaced on `SkillResponse.always_on` from the
|
||
ownership record in `_skill_to_response`. Toggled via
|
||
`PUT /api/skills/{name}/always-on` (custom skill → owner or admin; built-in /
|
||
public → admin only, materializes a row on demand via `ensure_legacy`). The
|
||
prompt builder stamps `Skill.always_on` from `list_always_on_names()` via a
|
||
sync DB bridge (`_apply_always_on` in `lead_agent/prompt.py`).
|
||
- **Prompt split** — `get_skills_prompt_section` (`lead_agent/prompt.py`) reads
|
||
the compression settings (skipped for restricted custom agents — they already
|
||
have a curated allowlist). When **off**, behaviour is unchanged (all enabled
|
||
skills full-text). When **on**, `always_on` skills render as full `<skill>`
|
||
blocks; the rest are dropped from the body, replaced by a `search_skills`
|
||
usage note plus (when `keep_index_in_prompt`) a name-only `<compressed_skills>`
|
||
index. The `_get_cached_skills_prompt_section` cache key includes the
|
||
`always_on` flag + compression flags so changes invalidate correctly.
|
||
- **Retrieval tool** — `search_skills` (`tools/builtins/skill_tools.py`,
|
||
`search_skills_impl`) keyword-scores `name + description` over enabled,
|
||
visible, non-archived skills (same visibility filter as `skill_list`) and
|
||
returns top-N `{name, description, category, location}`. It is registered in
|
||
`get_available_tools` **only when compression is enabled** (offline,
|
||
deterministic, zero external deps). The agent then `read_file`s the returned
|
||
`location`.
|
||
|
||
Tests: `tests/test_skill_compression.py`.
|
||
|
||
#### Skill Curator (`packages/harness/deerflow/skills/curator.py`)
|
||
|
||
Background skill lifecycle manager. The Gateway `lifespan` starts a resident
|
||
scheduler task (`_curator_scheduler_loop` in `app/gateway/app.py`) that ? after a
|
||
5-minute startup delay ? calls `maybe_run_curator()` every hour. A cross-process
|
||
file lock (`.curator_scheduler.lock`) elects a single leader worker so the
|
||
scheduler runs once per deployment. Per run:
|
||
|
||
- **Master switch** ? `skill_evolution.enabled` (default **`false`**) gates the
|
||
whole feature; `curator.enabled` (default **`false`**) gates the curator
|
||
specifically. Both must be `true` for the curator to act. `should_run_now()`
|
||
and `_curator_thread_target()` both enforce this ? so manual `POST
|
||
/api/curator/run` also respects the master switch. Config is
|
||
`config.yaml.skill_evolution` deep-merged with per-user
|
||
`users/{user_id}/skill_evolution.json`.
|
||
- **Per-user run lock** ? `.curator_state` carries `running` / `started_at`;
|
||
`_try_acquire_run_lock()` prevents the same user's curator from running
|
||
concurrently (scheduler vs. manual trigger, or double manual clicks). A lock
|
||
whose `started_at` is older than `_CURATOR_STALE_LOCK_HOURS` (6h) is treated as
|
||
process-crash residue and can be taken over.
|
||
- **Concurrency cap** ? `maybe_run_curator()` collects all due users and hands
|
||
them to `_dispatch_curator_runs()`, which runs them through a
|
||
`ThreadPoolExecutor` capped at `_CURATOR_MAX_CONCURRENCY` (5) ? avoids a
|
||
thundering herd of LLM calls when many users come due at once (e.g. first
|
||
deploy).
|
||
- **Failure back-off** ? `_release_run_lock()` always stamps `last_attempt_at`.
|
||
When a first run keeps crashing (`last_run_at` never set), `should_run_now()`
|
||
retries on `_CURATOR_RETRY_HOURS` (6h) instead of every hourly tick.
|
||
- **`should_run_now()`** is the real gate: the hourly tick only checks "is the
|
||
per-user `interval_hours` (default 168h) elapsed". Timestamp parsing tolerates
|
||
naive (tz-less) strings ? a corrupt `last_run_at` triggers a re-run rather than
|
||
a permanent deadlock.
|
||
- **Published-skill filter** ? `SkillStore.list_published_names()` is a batch
|
||
query; the curator fetches the published-name set once per run instead of one
|
||
DB round-trip per candidate skill.
|
||
- **LLM review timeout** ? `_run_llm_review()` runs `agent.invoke()` in a worker
|
||
thread bounded by `curator.review_timeout_seconds` (default 600s). The model's
|
||
per-request `request_timeout` can't bound a multi-turn React agent loop, so on
|
||
timeout the run is abandoned (a warning is written to `curator_log`, visible in
|
||
the Web UI) and the curator finishes cleanly + releases the run lock. The
|
||
worker context is copied via `contextvars.copy_context()` so tool calls still
|
||
resolve the correct `user_id`.
|
||
- **Review model** ? `_run_llm_review()` uses `curator.model_name` (a
|
||
`config.yaml` `models[]` entry name) if set, else the primary model; it is
|
||
editable from the Web UI evolution panel. The skill security scanner
|
||
(`scan_skill_content` in `security_scanner.py`) reuses the same
|
||
`skill_evolution.curator.model_name` setting ? review and moderation share one
|
||
model. There is no separate moderation-model config field.
|
||
|
||
### Model Factory (`packages/harness/deerflow/models/factory.py`)
|
||
|
||
- `create_chat_model(name, thinking_enabled)` instantiates LLM from config via reflection
|
||
- Supports `thinking_enabled` flag with per-model `when_thinking_enabled` overrides
|
||
- `force_disable_thinking=True` (kwarg) hard-disables thinking for models that explicitly declare `supports_thinking: true` even when the model config declares no `when_thinking_*`/`thinking` shape — many internal vLLM/Qwen models only set that flag, so `thinking_enabled=False` alone injects nothing and they keep reasoning on. The flag injects the OpenAI-compatible disable shapes (`extra_body.enable_thinking=false`, `extra_body.chat_template_kwargs.enable_thinking=false`, and `extra_body.thinking.type="disabled"`) only for classes that accept `extra_body`. Models with `supports_thinking: false`, especially generic OpenAI-compatible proxies such as LiteLLM, receive no vendor-specific thinking fields because those gateways may reject unknown parameters. Mirrors `TitleMiddleware._force_disable_thinking`. The lead agent reads the runtime flag `thinking_force_disabled` and forwards it (used by `/api/open/chat`)
|
||
- Supports vLLM-style thinking toggles via `when_thinking_enabled.extra_body.chat_template_kwargs.enable_thinking` for Qwen reasoning models, while normalizing legacy `thinking` configs for backward compatibility
|
||
- Supports `supports_vision` flag for image understanding models
|
||
- Config values starting with `$` resolved as environment variables
|
||
- Missing provider modules surface actionable install hints from reflection resolvers (for example `uv add langchain-google-genai`)
|
||
- Generic `langchain_openai:ChatOpenAI` entries are transparently wrapped by
|
||
`PatchedChatOpenAI`. Besides preserving reasoning/tool-call metadata, the
|
||
patch recognizes compatible gateways that encode failures inside HTTP 200
|
||
payloads (`base_resp.status_code != 0` with null/empty `choices`) in both
|
||
streaming and non-streaming paths. It raises the stable
|
||
`OpenAICompatibleUpstreamError`; `LLMErrorHandlingMiddleware` retries
|
||
transient concurrency/inference failures, while genuine quota/balance
|
||
failures are classified as non-retriable and surfaced immediately. Embedded
|
||
vendor failures therefore cannot become empty successful assistant turns.
|
||
The same adapter repairs only bounded, information-preserving compatibility
|
||
deviations before LangChain conversion: a streamed full `message`,
|
||
choice-root tool calls, tool calls beside an empty `delta`, singleton tool
|
||
call objects, structured `function.arguments`, and missing/string indexes.
|
||
Tool argument assembly is invocation-local (never model-instance shared),
|
||
assigns a stable canonical index to staggered id-bearing parallel calls, and
|
||
distinguishes confirmed cumulative snapshots from standard OpenAI deltas;
|
||
once a call is identified as delta mode, brace fragments and all later bytes
|
||
pass through unchanged. Gemini `thought_signature` / `thoughtSignature`
|
||
(including `extra_content.google` spellings) is extracted before the SDK
|
||
drops extension fields, stored as dedicated id/index metadata, and restored
|
||
at the same wire location on the next request. Keep the concurrent-stream,
|
||
nested-brace, signature round-trip, and real-SDK structured-argument tests in
|
||
`packages/harness/tests/test_patched_openai_tool_args.py` when changing this
|
||
boundary.
|
||
- Model observability is content-free. `ModelTimingCallback` logs connection,
|
||
first chunk, completion/failure/cancellation latency against a sanitized
|
||
upstream origin and run id. `PatchedChatOpenAI` emits one
|
||
`OPENAI_TOOL_STREAM_SUMMARY` for tool/repair/error streams with shape counts
|
||
and first-tool-argument latency. `LLMErrorHandlingMiddleware` logs only stable
|
||
retry fields (`attempt`, `reason`, error type, status and provider code); raw
|
||
provider details remain available only for in-process classification and must
|
||
never enter exception strings, logs, or stack traces. Prompts, response text,
|
||
tool arguments, file contents, credentials and raw exception messages are
|
||
forbidden in these records. `GeneratorExit`/`CancelledError` is an INFO-level
|
||
`status=cancelled`, not a failed-call alarm.
|
||
Diagnostic handlers must not replace a successful result, the original model
|
||
exception or cancellation when they fail. Safe upstream exception strings
|
||
retain private classification detail only in memory: context recovery must
|
||
still recognize numeric vendor codes with context-length messages, and auth
|
||
classification must check structured codes as well as human descriptions.
|
||
- LangGraph run-creation streams must expose both `Content-Location` (canonical
|
||
run resource) and `Location` (the `/threads/{thread}/runs/{run}/stream`
|
||
reconnect path relative to the SDK's `/api` base). The CORS middleware must
|
||
continue exposing both headers. The frontend records each consumed SSE event
|
||
id before yielding it to React. Cursor ownership must match exactly one SDK
|
||
message accumulator: same-reader recovery supplies that cursor; a newly
|
||
mounted hook or a real thread switch must reconstruct from `-1`, not inherit
|
||
another reader's cursor. Shared API clients never rewrite that sentinel.
|
||
`useThreadStream` counts message-tuple
|
||
changes as business activity, probes a known active run only after 90 seconds
|
||
without a business frame, and stops only the local reader before rejoining.
|
||
Clean-EOF, transport-error, conflict and watchdog triggers share one
|
||
single-flight reconciliation so they cannot double-join or double-finalize.
|
||
- The default `MemoryStreamBridge` and `RunManager` are process-local. Gateway
|
||
streaming must run with one uvicorn worker: `Location`, `/join`, cancellation,
|
||
and `Last-Event-ID` recovery are separate HTTP requests and a different worker
|
||
cannot see the originating worker's live task or buffered events. A durable
|
||
checkpointer/RunStore stores history but is not a shared live stream/control
|
||
plane. `scripts/start_backend-online.sh` and
|
||
`offline-backend-20260512/start_backend.sh` must keep `GATEWAY_WORKERS=1` as
|
||
their safe default and reject larger values unless an operator explicitly accepts
|
||
the unsafe override. Multi-worker support requires shared implementations for
|
||
both event transport and live-run ownership; sticky routing is not a fix.
|
||
|
||
### Roundtable Model Fault Tolerance (会商模型自动容错 / 换下一个模型)
|
||
|
||
会商第二步（leader/seat）+ 第三步（summary/report/dashboard + 结构化抽取）单轮执行时，**某模型出问题就自动切到下一个模型继续**，并告知用户「『X』模型异常 → 已切换『Y』」。前台实时（`multi_agent.py`）与后台挂起（`engine.py` + `InProcessRoundtableGateway`）两条路径都覆盖。
|
||
|
||
- **可检测的模型失败**：`LLMErrorHandlingMiddleware` 在同一模型重试 3 次后**不抛异常**，而是返回兜底 `AIMessage` —— 现在给它打 `additional_kwargs[LLM_ERROR_MARKER="deerflow_llm_error"]=reason`（model_dump 保留，经 checkpointer 序列化仍在）。`is_llm_error_message(msg)` 据标记（回退按已知兜底文案正文）识别，返回 reason。
|
||
- **候选模型链**：保留真正模型故障后的自动换模，但“仅有思考 / 暂时无正文 / 空交付”不再判作模型故障，绝不据此切换用户选择的模型。`app/gateway/roundtable_model_fallback.py::fallback_model_chain(current)` 按 `config.yaml models[]` 顺序给出 `[当前模型, …其余]`（去重保序）。`reason_text(reason)` 翻译成简短中文。总开关 env `ROUNDTABLE_MODEL_FALLBACK`（默认开，`0/false/off` 关）；`ROUNDTABLE_MODEL_FALLBACK_MAX`（0=试完全部）。
|
||
- **触发换模型的情形**：① 单轮 run 抛异常（start_run 失败 / idle 卡死 → `timeout`，其余 `exception`）；② 兜底错误文案（`is_llm_error_message`）。交付无效、仅有思考、空串以及普通总控轮既无正文又无派活**均不触发换模**；唯一例外是无工具的最终收口请求已经完整结束仍无正文，此时结果客观不可交付，才继续自动换下一个模型。席位仍只认剥掉 `<think>` 后的可见正文（`deerflow.agents.roundtable_orchestrator.delivery`：`visible_seat_text` / `classify_seat_delivery`）：前台 `_special_run` 对无效交付 yield `status:"invalid_delivery"` 且不广播；`GET /last-ai-message` 回 `{delivered, content, length, invalid_reason}`（思考-only 标 `thinking_only`，总控席位清单标 ⚠️）。后台引擎只把可见正文写入 `deliveries`，无效席位继续走既有未交付处理，不切换模型。Step3 单例（report/summary/dashboard/ingest/结构化）的产物是落盘文件，同样只对真模型错误换模型。
|
||
- **Step2 总控工具与可配置收口轮数**：普通总控使用真正白名单，只允许 `agent_orchestration` / `ask_clarification`；`write_file`、`present_files`、技能目录与文件/命令工具均不可见。`system_settings.json -> roundtable_room_limit.max_cycles`（默认 8，范围 2..100；管理员在设置弹窗「会商并发与收口」中修改）控制 recommend 模式到第几轮切换为显式 `finalize_only`：空工具白名单、注入已有有效席位全文、要求直接输出最终总结（失败席位如实标缺口），不能表现为“已停止”。一次会商启动时锁定当时值，运行中修改从下一次研讨生效；前台实时会商与后台作业读取同一设置。同一页的「会商文件流式兼容」(`write_file_compatibility_enabled`，默认关) 只作用于会商总控、席位、第三步和后台作业：全局文件流式兼容开着时会商一并使用；全局关着而本开关开着时，只有这些会商 run 改走 `buffered_gateway`。普通对话不读这个开关。chain / dag 的最终总结仍在既定席位完成后走同一纯正文路径；LangGraph 单轮 recursion limit 仍为 1000。
|
||
- **后台**：`InProcessRoundtableGateway._run_turn_with_fallback(...)` 包住 `_run_agent_turn`，逐候选模型试；换模型落 `model_switch` 诊断、全失败落 `model_fallback_exhausted`。`run_leader/run_seat/run_report/run_summary/run_dashboard_data` 全走它（report/summary/dashboard 的产物自愈重试保留，并记住已切到的可用模型用于后续续写轮）。各 Turn/Result 带回 `switches: list[ModelSwitch]`，引擎 `_emit_switches` → `JobProgressWriter.model_switch(...)` 往研讨时间线插一条 `role="system"` 系统提示气泡。
|
||
- **前台**：`_special_run`/`_leader_run` stream 收尾后仅检测明确的模型错误标记；命中才 yield `{status:"model_switch", role, agent_name, failed_model, next_model, reason}` 帧并用下一个模型重新流式，思考-only/空正文不进入该分支；终态帧带 `model_used`。结构化抽取 `flow_extract.py` 的 round1 `model.astream` 也包了同款换模型循环（emit `event:"model_switch"` 帧）。
|
||
- **Step2 单席位隔离**：`engine._run_recommend/_run_chain/_run_dag` 的每个席位执行包 try/except —— 一个席位彻底失败（落 `seat_failed` 诊断 + 置空交付）不拖垮整批/整链/整个作业。
|
||
- **前端**：`multi-agent.ts` 解析 `model_switch` 帧 → `onModelSwitch` 回调（清空失败尝试的半截气泡 + 累计的 tool_call delta，phase 退回 connected）；`useStep2Orchestration` 弹 toast + 时间线插「系统提示」气泡；`useStep3Summary/Report/Dashboard` + `flow-extract.ts` 重置对应气泡并在步骤卡留可见提示；后台路径经 `job-live.ts::toStep2Dialogue` 把 `role="system"` 渲染成 amber「系统提示」气泡。
|
||
- Tests: `tests/test_roundtable_model_fallback.py`。
|
||
|
||
### vLLM Provider (`packages/harness/deerflow/models/vllm_provider.py`)
|
||
|
||
- `VllmChatModel` subclasses `langchain_openai:ChatOpenAI` for vLLM OpenAI-compatible endpoints
|
||
- Thinking text is accepted from either `reasoning` (newer vLLM parsers) or `reasoning_content` (older vLLM / DeepSeek-style). The UI still reads canonical `additional_kwargs.reasoning_content`. Follow-up turns echo only the wire field that actually arrived, so other ChatOpenAI models are unchanged
|
||
- Shared helpers live in `deerflow/models/reasoning_fields.py` and are also used by `PatchedChatOpenAI` (the default `langchain_openai:ChatOpenAI` patch)
|
||
- Designed for configs that enable thinking through `extra_body.chat_template_kwargs.enable_thinking` on vLLM Qwen reasoning models, while accepting the older `thinking` alias
|
||
- Per-model switch: set only that entry's `use: deerflow.models.vllm_provider:VllmChatModel`. Do not change SiliconFlow / DeepSeek entries that already use `langchain_openai:ChatOpenAI`
|
||
|
||
### IM Channels System (`app/channels/`)
|
||
|
||
Bridges external messaging platforms (Feishu, Slack, Telegram, DingTalk) to the DeerFlow agent via the LangGraph Server.
|
||
|
||
|
||
**Architecture**: Channels communicate with Gateway through the `langgraph-sdk` HTTP client (same as the frontend), ensuring threads are created and managed server-side. The internal SDK client injects process-local internal auth plus a matching CSRF cookie/header pair so Gateway accepts state-changing thread/run requests from channel workers without relying on browser session cookies.
|
||
|
||
**Components**:
|
||
- `message_bus.py` - Async pub/sub hub (`InboundMessage` ? queue ? dispatcher; `OutboundMessage` ? callbacks ? channels)
|
||
- `store.py` - JSON-file persistence mapping `channel_name:chat_id[:topic_id]` ? `thread_id` (keys are `channel:chat` for root conversations and `channel:chat:topic` for threaded conversations)
|
||
- `manager.py` - Core dispatcher: creates threads via `client.threads.create()`, routes commands, keeps Slack/Telegram on `client.runs.wait()`, and uses `client.runs.stream(["messages-tuple", "values"])` for Feishu incremental outbound updates
|
||
- `base.py` - Abstract `Channel` base class (start/stop/send lifecycle)
|
||
- `service.py` - Manages lifecycle of all configured channels from `config.yaml`
|
||
- `slack.py` / `feishu.py` / `telegram.py` / `dingtalk.py` - Platform-specific implementations (`feishu.py` tracks the running card `message_id` in memory and patches the same card in place; `dingtalk.py` optionally uses AI Card streaming for in-place updates when `card_template_id` is configured)
|
||
|
||
**Message Flow**:
|
||
1. External platform -> Channel impl -> `MessageBus.publish_inbound()`
|
||
2. `ChannelManager._dispatch_loop()` consumes from queue
|
||
3. For chat: look up/create thread through Gateway's LangGraph-compatible API
|
||
4. Feishu chat: `runs.stream()` ? accumulate AI text ? publish multiple outbound updates (`is_final=False`) ? publish final outbound (`is_final=True`)
|
||
5. Slack/Telegram chat: `runs.wait()` ? extract final response ? publish outbound
|
||
6. Feishu channel sends one running reply card up front, then patches the same card for each outbound update (card JSON sets `config.update_multi=true` for Feishu's patch API requirement)
|
||
7. DingTalk AI Card mode (when `card_template_id` configured): `runs.stream()` ? create card with initial text ? stream updates via `PUT /v1.0/card/streaming` ? finalize on `is_final=True`. Falls back to `sampleMarkdown` if card creation or streaming fails
|
||
8. For commands (`/new`, `/status`, `/models`, `/memory`, `/help`): handle locally or query Gateway API
|
||
9. Outbound ? channel callbacks ? platform reply
|
||
|
||
**Configuration** (`config.yaml` -> `channels`):
|
||
- `langgraph_url` - LangGraph-compatible Gateway API base URL (default: `http://localhost:8001/api`)
|
||
- `gateway_url` - Gateway API URL for auxiliary commands (default: `http://localhost:8001`)
|
||
- In Docker Compose, IM channels run inside the `gateway` container, so `localhost` points back to that container. Use `http://gateway:8001/api` for `langgraph_url` and `http://gateway:8001` for `gateway_url`, or set `DEER_FLOW_CHANNELS_LANGGRAPH_URL` / `DEER_FLOW_CHANNELS_GATEWAY_URL`.
|
||
- Per-channel configs: `feishu` (app_id, app_secret), `slack` (bot_token, app_token), `telegram` (bot_token), `dingtalk` (client_id, client_secret, optional `card_template_id` for AI Card streaming)
|
||
|
||
|
||
### Memory System ? Memory V2 (`packages/harness/deerflow/agents/memory/`)
|
||
|
||
Pluggable memory architecture (Hermes-style). See `docs/MEMORY_V2_DESIGN_ZH.md`
|
||
and `docs/HINDSIGHT_SETUP_ZH.md` for the full design.
|
||
|
||
**Components**:
|
||
- `provider.py` - `MemoryProvider` abstract base (12 lifecycle/hook methods)
|
||
- `manager.py` - `MemoryManager` orchestrates builtin + ?1 external provider; `build_memory_manager()` is the construction entry point
|
||
- `providers/builtin.py` - `BuiltinFileProvider`: local USER.md / MEMORY.md, ? delimiter, char quotas, file locking, atomic writes, frozen snapshot
|
||
- `providers/hindsight.py` - `HindsightProvider`: HTTP client to a self-hosted Hindsight Docker instance, dual-bank routing
|
||
- `security.py` - injection/exfiltration scan + invisible-unicode block for memory content
|
||
- `message_processing.py` - `filter_messages_for_memory` (last-turn extraction for Hindsight retain)
|
||
|
||
**Two stores (binary classification)**:
|
||
- `USER.md` at `{base_dir}/users/{user_id}/USER.md` ? user profile, shared across agents
|
||
- `MEMORY.md` at `{base_dir}/users/{user_id}/agents/{agent_id}/MEMORY.md` ? agent-private working notes
|
||
- `user_id` resolved via `get_effective_user_id()`; `"default"` in no-auth mode
|
||
|
||
**Write paths**:
|
||
- LLM actively calls the `memory` tool (`tools/builtins/memory_tool.py`) ? writes USER.md/MEMORY.md, mirrors to Hindsight
|
||
- LLM calls `hindsight_recall`/`reflect`/`retain` tools (`tools/builtins/hindsight_tools.py`) ? only in `tools`/`hybrid` memory_mode
|
||
- Hindsight `auto_retain` ? every turn auto-stored to the work bank (via `MemoryMiddleware.after_agent`)
|
||
|
||
**Injection**:
|
||
- builtin frozen snapshot ? injected into the system prompt via `lead_agent/prompt.py` `_get_memory_context`
|
||
- Hindsight dynamic recall ? injected per model call via `MemoryMiddleware.wrap_model_call` (transient ? not persisted to state, so it never leaks to the frontend)
|
||
|
||
**Hindsight dual-bank isolation** (B-scheme):
|
||
- global user bank `deerflow-user-{user_id}` ? cross-agent
|
||
- agent work bank `deerflow-work-{user_id}-{agent_id}` ? per-agent
|
||
- `hindsight-client` is an optional dependency; when missing, `HindsightProvider.is_available()` returns False and the system falls back to builtin-only
|
||
|
||
**Migration**: `PYTHONPATH=. python scripts/migrate_memory_to_v2.py [--all|--user-id X] [--dry-run]` converts legacy `memory.json` into USER.md/MEMORY.md.
|
||
|
||
**Configuration** (`config.yaml` ? `memory`, nested): `enabled`, `provider` (builtin/hindsight),
|
||
`builtin.{memory_char_limit,user_char_limit,...}`, `hindsight.{mode,api_url,api_key,bank_id_template,work_bank_id_template,memory_mode,...}`,
|
||
`injection.{enabled,max_tokens,include_builtin,include_hindsight,context_tag}`, `security.{scan_content,block_invisible_unicode}`.
|
||
Per-user runtime overrides (whitelisted fields) live at `{base_dir}/users/{user_id}/memory_config.json` ? see `get_effective_memory_config()`.
|
||
|
||
**Gateway API** ? see the Memory router below (`/api/memory/v2*`).
|
||
|
||
### Reflection System (`packages/harness/deerflow/reflection/`)
|
||
|
||
- `resolve_variable(path)` - Import module and return variable (e.g., `module.path:variable_name`)
|
||
- `resolve_class(path, base_class)` - Import and validate class against base class
|
||
|
||
### Config Schema
|
||
|
||
**`config.yaml`** key sections:
|
||
- `models[]` - LLM configs with `use` class path, `supports_thinking`, `supports_vision`, provider-specific fields
|
||
- vLLM reasoning models should use `deerflow.models.vllm_provider:VllmChatModel`; for Qwen-style parsers prefer `when_thinking_enabled.extra_body.chat_template_kwargs.enable_thinking`, and DeerFlow will also normalize the older `thinking` alias
|
||
- `tools[]` - Tool configs with `use` variable path and `group`
|
||
- `tool_groups[]` - Logical groupings for tools
|
||
- `sandbox.use` - Sandbox provider class path
|
||
- `skills.path` / `skills.container_path` - Host and container paths to skills directory
|
||
- `title` - Auto-title generation (enabled, max_words, max_chars, prompt_template)
|
||
- `summarization` - Context summarization (enabled, trigger conditions, keep policy)
|
||
- `subagents.enabled` - Master switch for subagent delegation
|
||
- `memory` - Memory system (enabled, storage_path, debounce_seconds, model_name, max_facts, fact_confidence_threshold, injection_enabled, max_injection_tokens)
|
||
- `auth_login` - Custom login modes layered on the base auth (`deerflow/config/login_config.py`):
|
||
- `username_login.{require_password, password}` - Shared password gate for `POST /api/v1/auth/login/username`; blank configured secret falls back to `123ewq`. An account's own password/passphrase takes precedence. **Account login, registration and SSO policy (2026-10-05):** `system_settings.json -> authentication` stores `password_login_enabled=false`, `registration_enabled=false`, `single_session_enabled=false`, `sso_enabled=true` defaults; administrator GET/PUT `/api/system-settings/authentication` edits them immediately, and anonymous `/api/v1/auth/options` exposes the switches plus `needs_setup`. At least one login method must remain enabled; registration needs password login. Missing personal credentials use the existing shared passphrase (default `123ewq`), so disabling SSO does not require an administrator to first set a personal password. `/login/local` accepts username or email; `/register` accepts `{username,password}` (or historical `{email,password}`), always creates a regular user, and returns the full bearer login response. Names containing `admin` never create or promote administrators; persisted roles remain authoritative. Passwords and URL passphrases share the same versioned hash in `users.password_hash` and `users.login_password`; legacy plaintext passphrases remain verifiable and take precedence over an old independent password hash until reset. User `/change-password` and administrator `/users/{id}/login-password` (alias `/reset-password`) update both columns and atomically increment `token_version`; stale profile writes cannot overwrite credentials or revive sessions. Password removal is rejected. `/me.has_password` denotes personal credentials; `uses_default_password` marks shared-passphrase fallback, and `requires_current_password=true` requires the current shared passphrase even for the first password change. Self changes/resets return a replacement bearer and Cookie; administrator list items return `has_login_password`, never hashes or newly written plaintext (existing legacy plaintext remains admin-only for old link-copy compatibility). New link copies use only the reset input still in frontend memory. SSO policy also protects token/OAuth and the parallel panel `/api/parallel-agents/auth/token` compatibility entry, which now shares standard password/rate-limit/approval checks. Public first-admin initialization returns a full login response and inserts an `auth_initialization` singleton claim plus the first administrator in one transaction; the ORM automatically creates that table plus `auth_usernames`, whose unique username claim prevents concurrent registrations across email domains without rewriting historical users. Username SSO rechecks the final account credential after a concurrent registration or reset. When password login is enabled, username and parallel compatibility login require a personal or shared/default passphrase regardless of the old require_password flag. All verified issuing paths explicitly set JWT auth_method=password/passphrase/sso; its default remains None for legacy/identity-only sessions. Gateway and LangGraph share session_policy_error, rejecting unmarked sessions under enabled password login and SSO sessions when SSO is disabled. DEER_FLOW_AUTH_DISABLED cannot bypass an enabled password-login or single-session policy; disabling password login preserves legacy compatibility behavior. CLI `auth.reset_admin` rotates both credential columns. Single-session login (one online location per account) is the separate single_session_enabled=false policy, not external SSO. Every successful login route, including the parallel alias, registration and initialization, calls _prepare_login_session: the SQL repository atomically increments token_version and returns that transaction's current user before minting the JWT. Ordinary competing logins advance versions independently so only the latest survives, while credential/approval changes after verification cause 409 rather than authenticating a stale password. Failed logins never advance the version; when disabled, logins keep existing sessions. Password changes still always rotate the version. Tests: `tests/test_account_authentication.py`, `tests/test_username_login_domain.py`; contracts: `docs/AUTH_LOGIN.md`.
|
||
- `token_login.{enabled, token_info_url, service_authorization, timeout_seconds, verify_ssl, ca_cert_path, mock_tokens}` - Token-exchange login (`POST /api/v1/auth/login/token`). The gateway POSTs to `token_info_url` with the caller-supplied token placed directly in the `Authorization: Bearer <token>` header (no cookie; the static `service_authorization` field is deprecated/unused, kept only for backward compat), reads `data.username` from a `{"state":"200",...}` body, then signs the matching local user in (auto-registering if absent). `token_info_url` is HTTPS ? `verify_ssl: false` skips cert verification for self-signed/internal UAT hosts, or set `ca_cert_path` to a PEM bundle (takes precedence, keeps verification on). **`mock_tokens` (offline test mapping `{token: username}`)**: when an incoming token matches a key, `_fetch_username_from_token` returns the mapped username and **skips the `token_info_url` call entirely** — for environments where the upstream `getTokenInfo` host is unreachable (e.g. external networks). Empty in prod (no effect); seeded `{"123ewq": "admin"}` in this deployment so `/login/任意?authToken=123ewq` signs in as admin. **`UsernameLoginResponse` now also returns `username`** (the raw resolved login name, pre email-normalization) on both `/login/token` and `/login/username`, so the frontend can append a faithful `?username=` on jump links (按钮管理 追加登录态) instead of the lossy email local-part. See [docs/AUTH_LOGIN.md](docs/AUTH_LOGIN.md) for the full request/response contract.
|
||
|
||
Both endpoints are whitelisted in `auth_middleware._PUBLIC_EXACT_PATHS`. The shared `_get_or_create_user_by_username()` helper (in `routers/auth.py`) backs both `/login/username` and `/login/token`.
|
||
|
||
- `log_system` - Admin-only external log-system entry (`deerflow/config/log_system_config.py`): `{enabled, url, label}`. Surfaced read-only by `GET /api/system-settings/log-system` (admin only); the frontend renders it as a "????" item in the bottom-right settings menu (`workspace-nav-menu.tsx`) that opens `url` in a new tab. Disabled by default.
|
||
|
||
**`extensions_config.json`**:
|
||
- `mcpServers` - Map of server name ? config (enabled, type, command, args, env, url, headers, oauth, description)
|
||
- `skills` - Map of skill name ? state (enabled). 常驻(免压缩)/always_on lives in the `skills` DB table, not here.
|
||
|
||
Both can be modified at runtime via Gateway API endpoints or `DeerFlowClient` methods.
|
||
|
||
### Conditional WeKnora LLMWiki
|
||
|
||
**Selected knowledge-space policy:** if either `llmwiki_knowledge_base_ids` or
|
||
`assistant_knowledge_base_ids` is non-empty, Gateway sets
|
||
`knowledge_space_strict`. This is server policy, not a caller preference:
|
||
after ordinary context merging, Gateway overwrites both final
|
||
`configurable` and `context` with the authorized knowledge ids,
|
||
`knowledge_space_strict=true`, and the merged exclusion list, so a request
|
||
cannot pre-seed `false`, forged ids, or `excluded_tools=[]` to reopen external
|
||
retrieval. The lead agent must fail closed: no Web/browser
|
||
tools, MCP, external retrieval skills, ACP, deep-research pipeline, or
|
||
subagents. Only the pre-retrieved selected-space context and the read-only
|
||
`llmwiki_search` compatibility tool may retrieve knowledge; that tool supports
|
||
both ordinary and assistant Wiki stores and cannot accept IDs from the model.
|
||
Ordinary Wiki retrieval runs vector and local lexical channels concurrently,
|
||
uses RRF, reports raw vector threshold diagnostics, and continues serving a
|
||
coherent last-successful snapshot after a later sync is cancelled/failed.
|
||
`config.yaml` defaults the Wiki encoder to SiliconFlow `BAAI/bge-m3`; keep
|
||
`SILICONFLOW_API_KEY` outside Git. The frontend derives four starter questions
|
||
from the selected space names and hides generic recommendations while these
|
||
questions are present. Design and remaining graph/profile phases are in
|
||
`docs/KNOWLEDGE_SPACE_RETRIEVAL_AND_WIKI_EXPANSION_ZH.md`.
|
||
|
||
**Knowledge-base research bookshelf:** `/page/workspace/knowledge/:id/research`
|
||
is ordinary Lead Agent Q&A constrained to one existing knowledge base; it does
|
||
not invoke or reuse the legacy Deep Research workflow. Any account with read
|
||
access may create a private session/draft, while only the existing KB writer
|
||
may publish or overwrite its Markdown Wiki article. Published snapshots follow
|
||
the KB's read permission, but the session/thread/draft and the continuation
|
||
link remain creator-only (administrators do not inherit another user's private
|
||
chat). Every run resolves the immutable thread binding from
|
||
`knowledge_research_sessions` (registered on `deerflow.persistence.models` so
|
||
startup `create_all` builds the table; `get_by_thread` treats a missing table
|
||
as no binding so ordinary `/runs/stream` is not aborted), reauthorizes KB read access, accepts exactly one
|
||
text-only human turn, forces classic single-KB retrieval and disables tools,
|
||
memory, prompt prefixes, planning and subagents; fixed-answer shortcuts are
|
||
bypassed. Saving manually upserts a stable
|
||
`knowledge-research/{publication_id}` Wiki slug under the reserved root
|
||
「深度研究」 without enabling automatic Wiki generation, uses a saved hash for
|
||
optimistic overwrite checks and idempotent retries, and keeps the editable
|
||
publication title separate from the Q&A title. REST endpoints live in
|
||
`app/gateway/routers/llmwiki.py`, runtime enforcement in
|
||
`app/gateway/knowledge_research.py`, persistence/migration in
|
||
`deerflow.persistence.knowledge_research` / `20261022_01`, and regression
|
||
coverage in `tests/test_knowledge_research.py`.
|
||
|
||
**Embedding ownership:** WeKnora's per-knowledge-base `embedding_model_id`
|
||
indexes uploaded source documents; `llmwiki.local_wiki_index.embedding` indexes
|
||
the generated Wiki-page mirror inside DeerFlow. Do not couple these two model
|
||
choices. `GET /api/llmwiki/embedding-models` returns a strict credential-free
|
||
projection of WeKnora Embedding models, and KB creation must validate and pass
|
||
the selected active model id. When the caller omits a choice, retain WeKnora's
|
||
active-default behavior. Existing KBs cannot change this binding in place, and
|
||
cross-KB WeKnora searches should use model-compatible knowledge bases.
|
||
|
||
Wiki-enabled knowledge bases accept bounded user-owned content/extraction
|
||
instructions plus two independent optional generation inputs:
|
||
`wiki_config.directory_paths` (a one-to-three-level reference structure) and
|
||
`wiki_config.focus_instructions` (a strict extraction scope). The raw values
|
||
and normalized reference tree live in `llmwiki_knowledge_bases`; the effective
|
||
prompt tells WeKnora to use the directory names and hierarchy when they fit the
|
||
source, while allowing it to add, merge, rename, or omit nodes according to the
|
||
actual evidence. Directory guidance is never an allowlist and never a fact
|
||
source. `WeKnoraClient` merges edits into the remote `wiki_config` so synthesis
|
||
models and provider runtime knobs are retained.
|
||
|
||
Creation does not pre-create the reference skeleton. Mutation-settle and
|
||
periodic reconciliation run only for the strict focus scope: all page decisions
|
||
complete before any deletion, focus-outside generated pages are removed, and
|
||
the directory layout inferred by WeKnora is preserved. An incomplete or invalid
|
||
classifier result makes no destructive change and is retried. Directory-only
|
||
configuration therefore needs no post-generation reconciliation. The current
|
||
`directory_paths` field and the legacy `wiki_parent_directories` field have the
|
||
same prompt-only reference semantics; neither may trigger folder creation,
|
||
movement, deletion, or periodic directory reconciliation. Empty inputs keep
|
||
WeKnora's automatic behavior. Schema version 3 identifies this reference-only
|
||
prompt semantics. The background coordinator recompiles managed schema-2 rows
|
||
from their database-owned raw inputs and advances them only after the WeKnora
|
||
update succeeds; failures remain on schema 2 for retry.
|
||
Regression coverage:
|
||
`tests/test_wiki_directory_constraints.py` and
|
||
`tests/test_llmwiki_parent_directory_routing.py`.
|
||
|
||
Ordinary knowledge-base cards expose an owner/admin-only name/description/Wiki-instructions edit dialog in `WeKnoraKnowledgeWorkspace.tsx`, reusing `useLlmWikiActions().update` and `PATCH /api/llmwiki/knowledge-bases/{id}`. The route trims names, rejects blank or reserved conversation-deposit names, writes to WeKnora before updating local metadata, and retains existing write authorization. Default test questions are derived from the authoritative remote display name. The editor omits `test_questions` unless the user actually changes that field, and `knowledge_profile.generated_test_questions_name` recognizes only the exact legacy four-question template as automatic; genuinely edited lists remain custom. `_mapping_view` must resolve both questions and `test_questions_customized` from a profile row containing the current remote name, so a rename immediately replaces old-name defaults without a migration. A successful remote list is authoritative for every list scope, including `personal`: absent remote IDs are hidden without deleting mappings/references. Upstream failures remain errors. The grid also filters cached `remote_status=missing` rows and no longer renders stale-record cleanup cards. Regression coverage: `tests/test_llmwiki_knowledge_management.py`, `tests/test_llmwiki_weknora.py`.
|
||
|
||
`llmwiki.weknora.api_base_url` is the provider switch: non-empty enables WeKnora, empty preserves the legacy LLMWiki UI. WeKnora is independently deployed and owns PostgreSQL/Redis/object storage/vector storage/parsers/models; DeerFlow config contains only `api_base_url` and optional `web_base_url`. DeerFlow deliberately sends no WeKnora credential, so the WeKnora endpoint must be restricted to a trusted network and allow DeerFlow traffic itself or through its reverse proxy. There is intentionally no `default_embedding_model_id`: `WeKnoraClient` queries `/api/v1/models` and uses WeKnora's active Embedding model when creating a KB; Wiki creation additionally resolves an active KnowledgeQA model inside WeKnora for synthesis. The native BFF is `app/gateway/routers/llmwiki.py`; ownership/publication mappings live in `deerflow.persistence.llmwiki`; the browser only sees DeerFlow mapping IDs. Knowledge-base owners publish immediately (`private` -> `published`) without a review queue, and can unpublish ordinary bases. The conversation-deposit KB is system-owned, always published, shared by all DeerFlow users, excluded from My Knowledge, and immutable through user/admin detail operations; recognition requires both the reserved name and `system` owner. Existing same-name remote mappings are adopted in place as the system deposit, preserving mapping ids and document links. Its user-facing description, generated Markdown source and tags use `cmzs` branding; list/detail reads synchronize stale remote descriptions, and search results, source previews and embedded-detail JSON recursively replace legacy brand text in historical conversation-deposit payloads. Deposit requests must reference a thread accessible to the current account. Personal knowledge-base list/search views are scoped to the current DeerFlow account even for admins; frontend LLMWiki caches include the live account id, and admin write power does not make other users' private KBs appear as owned. Only `system_role === "admin"` sees the WeKnora connection-config tab. Knowledge cards navigate to a dedicated WeKnora-style detail route. Wiki-enabled bases render and default to `Wiki -> documents -> graph`; standard bases omit Wiki and render `documents -> graph`. The detail page supports sequential multi-file upload with an aggregate result for writable ordinary bases, hides it for public read-only/system-deposit bases, and uses a generic loading label while the embed session starts. It also supports URL and published manual-Markdown imports, browses real Wiki pages/content, proxies authenticated original-file previews, pages through `/chunks/{knowledge_id}`, and renders the Wiki page graph from `/knowledgebase/{kb_id}/wiki/graph`; document details use a right sheet rather than the removed KB modal. The credential-free proxy uses one WeKnora service identity, so per-user separation is enforced by DeerFlow mappings while all remote KBs physically share that identity's WeKnora workspace. WeKnora iframe detail pages first call `/api/llmwiki/knowledge-bases/{id}/weknora/session` through the normal DeerFlow API auth path, then load `/platform/knowledge-bases/{weknora_id}` with a signed `deerflow_weknora_embed` cookie for `/platform`, `/assets`, `/locales`, root assets, and API calls. Injected fetch/XHR rewriting sends all iframe `/api/v1/*`, including colliding `/api/v1/auth/*`, through `/api/llmwiki/weknora-embed/api/v1/*`; Gateway auth/CSRF bypass only applies to those signed iframe proxy requests, the proxy still revalidates the mapped KB, and cookie security honors `X-Forwarded-Proto` for HTTPS reverse proxies. `llmwiki_search` is conditionally registered in `tools.py`, is read-only, and rechecks the current DeerFlow user, agent `llmwiki_knowledge_base_ids`, and thread-context selection on every call. Source previews prefer the cleaned Wiki/LLMWiki page text when a retrieved chunk maps to a Wiki page, with raw split chunks as the fallback. Agent bindings use an inline WeKnora-style searchable/grouped multi-select in the standard agent create/detail UI; new chats use the same interaction pattern as a compact composer dropdown, and selections are written to thread metadata/context and restored on reopen. The DeerFlow frontend is native React and the WeKnora frontend is not modified or embedded. Full deployment and behavior notes: `docs/LLMWIKI_WEKNORA_INTEGRATION_ZH.md`. Tests: `tests/test_llmwiki_weknora.py`.
|
||
|
||
The embedded detail proxy hides WeKnora's outer `.main > .aside_box` application sidebar and expands the route outlet, but deliberately keeps the Wiki page's internal directory/index sidebar. Do not add an HTML `sandbox` attribute to this trusted, permission-filtered detail iframe: a sandboxed ancestor causes Chromium's built-in Blob PDF viewer to render the browser-blocked page. The Gateway follows upstream redirects only for `/knowledge/{id}/preview` and `/knowledge-bases/{id}/files`, keeping PDF bytes and protected Wiki images inside the signed proxy instead of sending the browser to WeKnora object storage.
|
||
Native/private and public Wiki readers must rewrite protected Markdown image
|
||
sources to their mapping-scoped DeerFlow `/files?file_path=` endpoints and keep
|
||
real HTTP URLs on rendered image elements. The embedded guard also rewrites
|
||
protected fetch/XHR values and hydrates late `img[data-protected-src]`
|
||
placeholders with the signed iframe HTTP proxy URL. Observe both
|
||
`data-protected-src` and `src`, because WeKnora may replace the URL after first
|
||
paint. Never create/revoke a temporary object URL here: its lightbox reuses the
|
||
thumbnail `src` after click. Never expose the service token or relax the
|
||
current-KB authorization to make images work.
|
||
|
||
**Embedded native knowledge-base switching:** The original WeKnora breadcrumb picker remains available inside the detail iframe. A signed 15-minute embed cookie carries only the current DeerFlow user's visible mappings (their own plus published bases); the proxied `/api/v1/knowledge-bases` response is filtered to that allowlist. Choosing another entry performs a full iframe navigation, rechecks the DeerFlow mapping and read/write permission, then reissues the cookie for the selected base. Never expose the upstream service-admin list directly or permit an arbitrary remote knowledge-base ID.
|
||
|
||
The independent enterprise-research workbench is **temporarily disabled**. Its source remains under `app/gateway/routers/enterprise_research.py` and `deerflow.persistence.enterprise_research`, but Gateway does not mount `/api/enterprise-research`, initialize its stores/executor, or register its ORM rows in `Base.metadata`. This ensures startup does not create or alter `enterprise_research_tasks` / `enterprise_research_report_jobs` on MySQL. It remains separate from `/api/deep-research`; do not re-enable it incidentally while changing the active Deep Research workflow. Regression coverage for its preserved implementation remains in `tests/test_enterprise_research_tasks.py`.
|
||
|
||
### Embedded Client (`packages/harness/deerflow/client.py`)
|
||
|
||
`DeerFlowClient` provides direct in-process access to all DeerFlow capabilities without HTTP services. All return types align with the Gateway API response schemas, so consumer code works identically in HTTP and embedded modes.
|
||
|
||
**Architecture**: Imports the same `deerflow` modules that Gateway API uses. Shares the same config files and data directories. No FastAPI dependency.
|
||
|
||
**Agent Conversation**:
|
||
- `chat(message, thread_id)` ? synchronous, accumulates streaming deltas per message-id and returns the final AI text
|
||
- `stream(message, thread_id)` ? subscribes to LangGraph `stream_mode=["values", "messages", "custom"]` and yields `StreamEvent`:
|
||
- `"values"` ? full state snapshot (title, messages, artifacts); AI text already delivered via `messages` mode is **not** re-synthesized here to avoid duplicate deliveries
|
||
- `"messages-tuple"` ? per-chunk update: for AI text this is a **delta** (concat per `id` to rebuild the full message); tool calls and tool results are emitted once each
|
||
- `"custom"` ? forwarded from `StreamWriter`
|
||
- `"end"` ? stream finished (carries cumulative `usage` counted once per message id)
|
||
- Agent created lazily via `create_agent()` + `_build_middlewares()`, same as `make_lead_agent`
|
||
- Supports `checkpointer` parameter for state persistence across turns
|
||
- `reset_agent()` forces agent recreation (e.g. after memory or skill changes)
|
||
- See [docs/STREAMING.md](docs/STREAMING.md) for the full design: why Gateway and DeerFlowClient are parallel paths, LangGraph's `stream_mode` semantics, the per-id dedup invariants, and regression testing strategy
|
||
|
||
**Gateway Equivalent Methods** (replaces Gateway API):
|
||
|
||
| Category | Methods | Return format |
|
||
|----------|---------|---------------|
|
||
| Models | `list_models()`, `get_model(name)` | `{"models": [...]}`, `{name, display_name, ...}` |
|
||
| MCP | `get_mcp_config()`, `update_mcp_config(servers)` | `{"mcp_servers": {...}}` |
|
||
| Skills | `list_skills()`, `get_skill(name)`, `update_skill(name, enabled)`, `install_skill(path)` | `{"skills": [...]}` |
|
||
| Memory | `get_memory()`, `reload_memory()`, `get_memory_config()`, `get_memory_status()` | dict |
|
||
| Uploads | `upload_files(thread_id, files)`, `list_uploads(thread_id)`, `delete_upload(thread_id, filename)` | `{"success": true, "files": [...]}`, `{"files": [...], "count": N}` |
|
||
| Artifacts | `get_artifact(thread_id, path)` ? `(bytes, mime_type)` | tuple |
|
||
|
||
**Key difference from Gateway**: Upload accepts local `Path` objects instead of HTTP `UploadFile`, rejects directory paths before copying, and reuses a single worker when document conversion must run inside an active event loop. Artifact returns `(bytes, mime_type)` instead of HTTP Response. The new Gateway-only thread cleanup route deletes `.deer-flow/threads/{thread_id}` after LangGraph thread deletion; there is no matching `DeerFlowClient` method yet. `update_mcp_config()` and `update_skill()` automatically invalidate the cached agent.
|
||
|
||
**Tests**: `tests/test_client.py` (77 unit tests including `TestGatewayConformance`), `tests/test_client_live.py` (live integration tests, requires config.yaml)
|
||
|
||
**Gateway Conformance Tests** (`TestGatewayConformance`): Validate that every dict-returning client method conforms to the corresponding Gateway Pydantic response model. Each test parses the client output through the Gateway model ? if Gateway adds a required field that the client doesn't provide, Pydantic raises `ValidationError` and CI catches the drift. Covers: `ModelsListResponse`, `ModelResponse`, `SkillsListResponse`, `SkillResponse`, `SkillInstallResponse`, `McpConfigResponse`, `UploadResponse`, `MemoryConfigResponse`, `MemoryStatusResponse`.
|
||
|
||
## Development Workflow
|
||
|
||
### Test-Driven Development (TDD) ? MANDATORY
|
||
|
||
**Every new feature or bug fix MUST be accompanied by unit tests. No exceptions.**
|
||
|
||
- Write tests in `backend/tests/` following the existing naming convention `test_<feature>.py`
|
||
- Run the full suite before and after your change: `make test`
|
||
- Tests must pass before a feature is considered complete
|
||
- For lightweight config/utility modules, prefer pure unit tests with no external dependencies
|
||
- If a module causes circular import issues in tests, add a `sys.modules` mock in `tests/conftest.py` (see existing example for `deerflow.subagents.executor`)
|
||
|
||
```bash
|
||
# Run all tests
|
||
make test
|
||
|
||
# Run a specific test file
|
||
PYTHONPATH=. uv run pytest tests/test_<feature>.py -v
|
||
```
|
||
|
||
### Running the Full Application
|
||
|
||
From the **project root** directory:
|
||
```bash
|
||
make dev
|
||
```
|
||
|
||
This starts all services and makes the application available at `http://localhost:2026`.
|
||
|
||
**All startup modes:**
|
||
|
||
| | **Local Foreground** | **Local Daemon** | **Docker Dev** | **Docker Prod** |
|
||
|---|---|---|---|---|
|
||
| **Dev** | `./scripts/serve.sh --dev`<br/>`make dev` | `./scripts/serve.sh --dev --daemon`<br/>`make dev-daemon` | `./scripts/docker.sh start`<br/>`make docker-start` | ? |
|
||
| **Prod** | `./scripts/serve.sh --prod`<br/>`make start` | `./scripts/serve.sh --prod --daemon`<br/>`make start-daemon` | ? | `./scripts/deploy.sh`<br/>`make up` |
|
||
|
||
| Action | Local | Docker Dev | Docker Prod |
|
||
|---|---|---|---|
|
||
| **Stop** | `./scripts/serve.sh --stop`<br/>`make stop` | `./scripts/docker.sh stop`<br/>`make docker-stop` | `./scripts/deploy.sh down`<br/>`make down` |
|
||
| **Restart** | `./scripts/serve.sh --restart [flags]` | `./scripts/docker.sh restart` | ? |
|
||
|
||
**Nginx routing**:
|
||
- `/api/langgraph/*` ? Gateway embedded runtime (8001), rewritten to `/api/*`
|
||
- `/api/*` (other) ? Gateway API (8001)
|
||
- `/` (non-API) ? Frontend (3000)
|
||
|
||
### Running Backend Services Separately
|
||
|
||
From the **backend** directory:
|
||
|
||
```bash
|
||
# Gateway API
|
||
make gateway
|
||
```
|
||
|
||
Direct access (without nginx):
|
||
- Gateway: `http://localhost:8001`
|
||
|
||
### Browser CORS
|
||
|
||
The Gateway always permits local Vite origins on ports 5173, 5174, 3000, and
|
||
8080 for both `localhost` and `127.0.0.1` with credentials. A deployed
|
||
frontend on another origin must be listed exactly in `GATEWAY_CORS_ORIGINS`
|
||
(for example `http://47.88.25.99:7010`); explicit origins are merged with the
|
||
local development origins. Regression: `tests/test_taskcop_cors.py`.
|
||
|
||
### Frontend Configuration
|
||
|
||
The frontend uses environment variables to connect to backend services:
|
||
- `NEXT_PUBLIC_LANGGRAPH_BASE_URL` - Defaults to `/api/langgraph` (through nginx)
|
||
- `NEXT_PUBLIC_BACKEND_BASE_URL` - Defaults to empty string (through nginx)
|
||
|
||
When using `make dev` from root, the frontend automatically connects through nginx.
|
||
|
||
## Key Features
|
||
|
||
### Administrator-managed custom themes
|
||
|
||
`system_settings.json` → `appearance` now stores `custom_themes` alongside
|
||
the built-in default theme. A row is `CustomThemePreset` (`id`, `name`,
|
||
six-digit `primary`, optional primary-button `text_color`, `mode`, `overrides`, `enabled`,
|
||
`sort_order`). `overrides` is intentionally a short interaction-only list:
|
||
`button_hover`, `navigation_gradient_start`, `navigation_gradient_end`,
|
||
`selected_background`, `border`, `muted`, `muted_text`, `destructive`,
|
||
`brand_title`, and `primary_foreground`. The navigation gradient endpoints
|
||
default from `primary` and are only stored when an administrator overrides them.
|
||
All values must be six-digit hex strings; the frontend derives every remaining
|
||
semantic CSS/TDesign token so admins do not maintain a full raw token sheet.
|
||
Ids must be `custom-theme-*`, names and ids are unique, and the router limits
|
||
the list to 12. `GET/PUT /api/system-settings/appearance` is the single
|
||
read/write surface: reads are available to all users, writes require admin.
|
||
The router re-orders rows deterministically and only permits an enabled custom
|
||
theme as `default_theme`; removing or disabling the default safely falls back
|
||
to `light`. Regression coverage: `tests/test_appearance_theme.py`.
|
||
|
||
Compact top-bar shortcuts live in the same `appearance` setting. Their route
|
||
parameters support `login`, `theme`, and keyless `path`. A `path` parameter has
|
||
no query-key input; the operator selects its value source from the right-hand
|
||
dropdown and the frontend appends the resolved, URL-encoded value to the
|
||
external URL path (or Hash route) in configured order. Password stays an
|
||
ordinary `login` query parameter with the key `password`, so an external login
|
||
link can become `#/login/<user>?password=…`. Regression coverage:
|
||
`tests/test_appearance_theme.py`.
|
||
|
||
### File Upload
|
||
|
||
Chat upload configuration is a database-backed global policy, not a static
|
||
`config.yaml` size tuple. `deps.py` creates and seeds the singleton row, the
|
||
Gateway loads it for every request, and `deerflow.uploads.policy` exposes only a
|
||
validated runtime copy to the harness. The administrator page under settings
|
||
edits or resets the same row. The built-in defaults are 20 files / 100 MiB total;
|
||
documents and images are 10 MiB per file, and spreadsheets are 5 MiB.
|
||
|
||
- Endpoint: `POST /api/threads/{thread_id}/uploads`; policy endpoints:
|
||
`GET/PUT /api/upload-policy`, `POST /api/upload-policy/reset`
|
||
- Supported policy formats: TXT/Markdown, PDF, DOC/DOCX, PPT/PPTX, JPG/JPEG/PNG,
|
||
and XLS/XLSX/CSV; the server restricts edits to parsers it can validate safely
|
||
- The whole batch is streamed to a private staging directory, checked for count,
|
||
total size, per-category size, extension, signature/container and encoding,
|
||
then atomically committed; validation or sandbox-sync failure restores old files
|
||
- Optional PDF/PPT/Excel/Word conversion always runs off the event loop with
|
||
bounded concurrency and a fixed extracted-text ceiling; companions are named
|
||
`report.docx.md`, with legacy `report.md` lookup retained for old uploads
|
||
- Files stay in thread-isolated directories; `UploadsMiddleware` injects bounded
|
||
JSON metadata plus small content samples, never unbounded bodies or repeated
|
||
converted companions
|
||
- Range-free large local reads return streaming full-file statistics and
|
||
deterministic head/middle/tail samples. Explicit line reads, `grep`, artifact
|
||
previews, image dimensions/derivatives and final model context all have hard caps
|
||
- Uploaded raster images also receive a signed public HTML URL (`/api/public/html-images/{token}`) so generated pages can use them as `background-image`
|
||
|
||
See [docs/FILE_UPLOAD.md](docs/FILE_UPLOAD.md) for details.
|
||
|
||
### Plan Mode
|
||
|
||
TodoList middleware for complex multi-step tasks:
|
||
- Controlled via runtime config: `config.configurable.is_plan_mode = True`
|
||
- Provides `write_todos` tool for task tracking
|
||
- One task in_progress at a time, real-time updates
|
||
|
||
See [docs/plan_mode_usage.md](docs/plan_mode_usage.md) for details.
|
||
|
||
### Context Summarization
|
||
|
||
Automatic conversation summarization when approaching token limits:
|
||
- **Administrator recent-turn window:** `system_settings.json` →
|
||
`conversation_context.recent_turn_window_enabled` is a deployment-wide,
|
||
default-off alternative to summarization. Administrators manage it in
|
||
「设置 → 上下文管理」 through
|
||
`GET/PUT /api/system-settings/conversation-context`. The same block stores
|
||
`max_conversation_rounds` (default 20, range 1–50), which `ChatPage` uses as
|
||
the maximum number of real user turns in one ordinary chat; missing values in
|
||
older settings files retain the 20-turn default. When enabled, each lead
|
||
agent model request retains only the current turn and the two preceding real
|
||
user turns, together with their assistant/tool continuations and any hidden
|
||
context attached to the retained boundary. The operation is request-only:
|
||
checkpoints and the transcript shown in the UI remain complete. This mode
|
||
takes priority over the per-chat 「上下文压缩」 toggle and never invokes the
|
||
summary model.
|
||
- **On/off switch is per-request, not the config file.** The chat input box has a
|
||
「上下文压缩」 toggle (below 参考文献, **default off**) → frontend sends
|
||
`context.summarization_enabled` → whitelisted in `_CONTEXT_CONFIGURABLE_KEYS`
|
||
(`app/gateway/services.py`) → `_build_middlewares` reads it from the run config
|
||
and calls `_create_summarization_middleware(force_enabled=summarization_enabled)`.
|
||
`force_enabled=True` mounts the middleware even when `config.summarization.enabled`
|
||
is false; when the flag is absent (channels / scheduled runs) it falls back to
|
||
the config flag. `config.summarization.enabled` now defaults to **false**.
|
||
- `config.yaml` `summarization` still supplies the **tuning**: trigger types
|
||
(tokens, messages, or fraction of max input) and the keep policy.
|
||
- A compressed request must retain the latest non-empty `HumanMessage`, even
|
||
when it has already been folded into the hidden summary. This keeps strict
|
||
OpenAI-compatible gateways from rejecting a tool-continuation call with
|
||
`No user query found in message`.
|
||
- LangChain keeps the agent prompt in `ModelRequest.system_message`, separate
|
||
from `ModelRequest.messages`. Compression must merge its hidden summary into
|
||
that dedicated slot and remove every `SystemMessage` from the ordinary
|
||
message list. Strict Qwen-style adapters accept only one system message at
|
||
index zero and otherwise return HTTP 400 (`system message must be at the
|
||
beginning`). Keep this normalization adapter-independent; the payload-level
|
||
OpenAI-compatible normalizer is a second line of defence and also merges
|
||
adjacent leading system messages.
|
||
- Keeps recent messages while summarizing older ones.
|
||
- **The model input hard guard is independent and always on when configured.**
|
||
Set `models[].max_input_tokens` (or `profile.max_input_tokens`) to the safe
|
||
input ceiling; `max_tokens` is output-only. `ContextWindowGuardMiddleware`
|
||
is mounted at the innermost end of both lead-agent and subagent model-call
|
||
chains, after memory/RAG/image/custom injections. It counts the complete
|
||
request (system prompt, messages/reasoning, tool schemas and response format),
|
||
then transiently removes old complete turns and truncates oversized bodies as
|
||
a last resort without mutating checkpoint state. Provider context-length
|
||
errors are non-retriable so a tokenizer mismatch cannot trigger repeated
|
||
submission of the same oversized request.
|
||
|
||
Tests: `tests/test_prompt_prefix.py` (system-setting API/defaults) and
|
||
`tests/test_summarization_multiturn.py` (summarization plus transient recent-turn
|
||
window behavior).
|
||
See [docs/summarization.md](docs/summarization.md) for details.
|
||
|
||
### Vision Support / Image Recognition (??)
|
||
|
||
Two paths, selected by `config.yaml` ? `vision`:
|
||
|
||
**Dedicated vision model (delegated recognition)** ? set `vision.model_name` to a model from `models[]`. The `view_image` tool sends the image to that model and returns its **text description** as the tool result, so a text-only main chat model can still "see" images. `view_image` accepts an optional `question` argument for targeted queries; the default prompt is `vision.prompt` (overridable), and `vision.max_tokens` caps the recognition output. Resolution/gating lives in `deerflow/models/vision.py` (`resolve_vision_model_name`, `vision_enabled`, `is_delegated_vision`, `recognize_image`).
|
||
|
||
**Native vision (fallback)** ? when `vision.model_name` is unset, behavior falls back to the main chat model, which only works if it has `supports_vision: true`:
|
||
- `ViewImageMiddleware` injects base64 image data into the conversation before the LLM call
|
||
- `view_image` stores the image in `state.viewed_images` for that middleware
|
||
|
||
Both the `view_image` tool and `ViewImageMiddleware` are registered whenever `vision_enabled(model_name)` is true (dedicated model configured **or** main model supports vision). Config schema: `deerflow/config/vision_config.py` (`VisionConfig`); tests: `tests/test_vision.py`.
|
||
|
||
### Upload symlink-safety (cross-platform)
|
||
|
||
`deerflow/uploads/manager.py:open_upload_file_no_symlink` writes uploads without following a pre-existing destination symlink. `O_NOFOLLOW` is applied where available (POSIX) but is **optional** ? Windows lacks `os.O_NOFOLLOW`/`O_NONBLOCK`, so requiring it previously made every Windows upload fail with `UnsafeUploadPathError`. Pre-existing symlinks are still rejected via `lstat`, and the opened fd is re-checked with `fstat`. `O_BINARY` is added on Windows. Regression: `tests/test_uploads_windows_safe.py`.
|
||
|
||
## Code Style
|
||
|
||
- Uses `ruff` for linting and formatting
|
||
- Line length: 240 characters
|
||
- Python 3.12+ with type hints
|
||
- Double quotes, space indentation
|
||
|
||
## Documentation
|
||
|
||
### WeKnora iframe behind `/deerflow`
|
||
|
||
The LLMWiki WeKnora detail iframe is publicly served through the Gateway at
|
||
`/deerflow`, independently of the frontend deployment prefix (for example
|
||
`/magentweb`). Register `llmwiki.proxy_router` with the `/deerflow` prefix
|
||
before its unprefixed fallback router, otherwise the unprefixed catch-all
|
||
captures `/deerflow/*`. HTML root-relative assets, API strings, and Vite's
|
||
`__vite__mapDeps` table are rewritten for `/deerflow`; generic JavaScript
|
||
regular-expression literals must not be rewritten. Auth and CSRF bypass only
|
||
applies when the short-lived signed WeKnora embed cookie is valid. Keep static
|
||
path recognition in `app/gateway/weknora_embed.py` synchronized with the
|
||
proxy's static fallback paths in `app/gateway/routers/llmwiki.py`.
|
||
|
||
Static assets use a separate signed-cookie-only path: never fetch an upstream
|
||
admin session or query per-KB ACLs for public JS/CSS/fonts. HTML and data APIs
|
||
still revalidate mapping permissions. `app/gateway/weknora_static.py` coalesces
|
||
concurrent reads and bounds its LRU by bytes and entries; cache keys include
|
||
the upstream web URL, public prefix, asset path and query. Bundle rewriting,
|
||
gzip and ETag generation run in a worker thread. Fingerprinted `/assets/`
|
||
files receive immutable browser caching; `/config.js` and upstream private,
|
||
no-store, no-cache or identity-varying responses must not enter the shared
|
||
cache. Explicit browser cache bypass refreshes the cached representation.
|
||
|
||
The injected iframe guard observes root theme changes separately and ignores
|
||
its own writes. Process only changed content branches, coalesced per batch;
|
||
ordinary loaded pages must have no recurring DOM-maintenance timer. Source
|
||
detail deep links retain a bounded nine-second drawer fallback. Preserve the
|
||
inner Wiki directory, protected images and theme repair. Regression tests:
|
||
`tests/test_weknora_static.py`, `tests/test_weknora_embed_script.py`.
|
||
|
||
### Position-roundtable role directory
|
||
|
||
`position_roles` is the global shared directory for position-roundtable work
|
||
views. It is deliberately separate from organisation/RBAC `positions`.
|
||
`app.gateway.deps.langgraph_runtime()` seeds the four stable built-in ids from
|
||
`app/gateway/position_role_defaults.py` via `ensure_seed()` (missing rows only;
|
||
never overwrite operator edits). The API is `GET/PUT /api/position-roles`;
|
||
writes are admin-only. Exactly one enabled `intent` role is required and owns
|
||
the initial intent workspace plus the summary/action-plan closing agents. Do
|
||
not delete a built-in id because historical chain nodes persist those ids;
|
||
custom role ids may be added and removed.
|
||
|
||
### Position-roundtable task-scoped session history
|
||
|
||
`position_roundtable_sessions.external_task_id` is optional. When populated from the frontend route `taskId`, new rows set `task_scoped = true`: a task can have multiple sessions and all users opening that task see the same sessions, nodes and persisted conversation snapshots. When it is NULL/empty, keep the original `user_id`-scoped personal workspace behavior. `20260808_01` adds `task_scoped` with default `false` (so all historical records remain personal) plus the task/update-time index; it must not backfill or make `external_task_id` non-null.
|
||
|
||
### Roundtable artifact submission status
|
||
|
||
`roundtable_artifact_submissions` keeps task-level submission state for a
|
||
generated deliverable and is deliberately separate from task details and any
|
||
external task-system delivery. The API is
|
||
`GET /api/roundtable/tasks/{taskId}/artifacts?submitted=true` and
|
||
`PUT /api/roundtable/tasks/{taskId}/artifacts/submission`. All callers use the
|
||
normal Gateway Bearer authentication; records are shared by task while
|
||
`created_by`/`submitted_by`/`cancelled_by` are audit-only. The server derives a
|
||
SHA-256 idempotency key from `taskId + sourceKey + path + revision`: submitting
|
||
the same revision is an upsert, while cancellation only flips `submitted` and
|
||
sets `cancelled_at` (it never deletes history). Store only the sandbox file
|
||
reference (`threadId`, path, source/node/session fields, name/MIME and revision)
|
||
and timestamps; do not persist Markdown content or call a formal delivery
|
||
interface. Previews continue to read the DeerFlow artifact by `threadId + path`.
|
||
|
||
See `docs/` directory for detailed documentation:
|
||
- [../README.md](../README.md) - Offline Docker wheelhouse preparation and startup workflow. The pinned versions in `../offline-runtime-requirements.txt` and `../start_backend.sh` must stay synchronized. Startup checks `croniter`, `hindsight-client`, the aiomysql compatibility bundle, and Redis, installing missing or incompatible packages from wheelhouse only. The bundle installs `aiomysql==0.3.2`, `PyMySQL==1.2.0`, and `SQLAlchemy==2.0.51` together; the startup probe also verifies that SQLAlchemy's adapted `ping(reconnect=False)` signature is present, preventing the old-adapter missing-`reconnect` failure. The download target defaults to CPython 3.12 on Linux x86_64 and verifies dependency closure using wheelhouse only.
|
||
- [CONFIGURATION.md](docs/CONFIGURATION.md) - Configuration options
|
||
- [ARCHITECTURE.md](docs/ARCHITECTURE.md) - Architecture details
|
||
- [API.md](docs/API.md) - API reference
|
||
- [SETUP.md](docs/SETUP.md) - Setup guide
|
||
- [FILE_UPLOAD.md](docs/FILE_UPLOAD.md) - File upload feature
|
||
- [PATH_EXAMPLES.md](docs/PATH_EXAMPLES.md) - Path types and usage
|
||
- [summarization.md](docs/summarization.md) - Context summarization
|
||
- [plan_mode_usage.md](docs/plan_mode_usage.md) - Plan mode with TodoList
|
||
- [BACKEND_RUN_STOP_ZH.md](docs/BACKEND_RUN_STOP_ZH.md) - Windows-first backend start/stop and troubleshooting guide
|
||
|
||
### Deep Research retrieval
|
||
|
||
`deerflow.agents.deep_research.adapters.material_provider.DeerFlowMaterialProvider`
|
||
must remain on the Q&A retrieval path. Enabled `deep-search`, `web-research`,
|
||
and knowledge-base skills run through the existing `SubagentExecutor` skill
|
||
runtime only when their required Q&A tool is registered; then the provider may
|
||
use configured `web_search`. A disabled placeholder endpoint is not a valid
|
||
search channel: use the maintained keyless DuckDuckGo provider and a relevant
|
||
real-web fallback as read-only sources, keep source provenance, and never
|
||
enable or call a placeholder just to make the Deep Research UI look configured. Regression coverage is in
|
||
`tests/test_deep_research_material_provider.py`.
|
||
When a queued Deep Research job is cancelled, the Gateway must also clear the
|
||
owning session's `active_job_id` and mark it `cancelled`; otherwise the stale
|
||
session incorrectly consumes that user's concurrency limit.
|
||
|
||
Report prose is the one Deep Research LLM path that must use
|
||
`DeerFlowCompletionBackend.stream_complete()` rather than `complete()`: it
|
||
drives `model.astream()`, sends native `report_delta` frames through
|
||
`ReportChunkEmitter`, and persists bounded `report_chunk` checkpoints. The
|
||
app-layer `DeepResearchLiveHub` fans out those live frames inside one Gateway
|
||
worker only; the database event log still owns `?after=<seq>` reconnect replay
|
||
and multi-worker recovery. Inject the hub callback into `PersistingEventSink`
|
||
and never import `app.*` from `deerflow.*`. Keep structured plan/review calls
|
||
on `complete()`. Before the search fan-out, basic mode emits one
|
||
`queries_planned` event and detailed/multi-agent modes emit one per subtopic;
|
||
they are durable user-visible search terms, not chain-of-thought. On completion, retain
|
||
the terminal `lastJobId` in `usage_snapshot` so a refreshed conversation page
|
||
can replay its trace from the event log. Detailed mode uses stable targets (`introduction`,
|
||
`section:N`, `conclusion`) because section writers are concurrent;
|
||
multi-agent revisions must emit `report_reset` before their next report
|
||
version. Do not persist raw model reasoning or one persisted event per provider
|
||
token. Models that expose thinking inside ordinary `<think>...</think>` text
|
||
must be split statefully before a report delta, checkpoint, or Markdown
|
||
projection is created; an optional live-only `report_thinking` frame can drive
|
||
the message-list thought trace, while the sandbox receives answer Markdown
|
||
only. The post-report digest (`summarize_report`) must likewise forward
|
||
provider reasoning through live-only `summary_thinking` frames so the
|
||
chat timeline can render thought while waiting for the first `summary_delta`.
|
||
The Vite page starts Word downloads through
|
||
`frontend-web/src/core/scheduled-tasks/markdown-docx.ts`, but the Gateway owns
|
||
generation through `POST /api/writing/export/docx`. Keep the standard-library
|
||
OOXML builder and its embedded-font validation; do not add a Python document-
|
||
conversion dependency merely to export text.
|
||
All non-streaming Deep Research control-plane calls (`complete()` for query /
|
||
section / plan / review JSON) run through `asyncio.wait_for` using the frozen
|
||
`DeepResearchConfig.control_plane_timeout_seconds` (45 seconds by default,
|
||
clamped to 120). Do not apply that total-duration cap to `stream_complete()`:
|
||
reports may legitimately be long but must continue emitting visible deltas. A
|
||
basic query-planning timeout/failure emits `query_planning_fallback` and searches
|
||
the original topic; detailed and multi-agent plan creation similarly fall back
|
||
to their original topic/section, rather than leaving a durable job stuck in
|
||
`planning`. Tests: `tests/test_deep_research_llm_adapter.py` and
|
||
`tests/test_deep_research_runners.py`.
|
||
The per-user Deep Research concurrency guard counts only session statuses
|
||
`running` and `awaiting_input`. A `draft` has no durable job and must never
|
||
consume a slot; otherwise saved drafts make `POST /sessions/{id}/jobs` return
|
||
429 permanently. Regression: `test_saved_draft_sessions_do_not_consume_research_concurrency`.
|
||
|
||
`POST /api/deep-research/sessions/{id}/jobs` is retry-safe under MySQL
|
||
read/write splitting. Deep Research session/job/event/source/message writes
|
||
must serialise their changed row while the write transaction is still open
|
||
(`flush` for inserts, select-after-update **before** commit), never through a
|
||
post-commit `session.refresh()` or a fresh pooled read. Replica lag otherwise
|
||
turns a committed job into a misleading 500, can make dispatcher `claim_job()`
|
||
lose its snapshot, or can fail report/follow-up persistence after the model has
|
||
already completed. Always nudge the dispatcher for both a newly created job and
|
||
an idempotently reused queued job; otherwise a retry after such an interrupted
|
||
response returns the old job but leaves it waiting for the periodic scan. The
|
||
SSE replay generator must catch temporary event/job-store reads, emit a
|
||
recoverable warning, and keep the connection open so the next poll can repair
|
||
the cursor instead of surfacing an ASGI `ExceptionGroup`. Regressions:
|
||
`test_deep_research_writes_never_depend_on_post_commit_refresh`,
|
||
`test_job_create_never_reads_back_after_commit`,
|
||
`test_retry_of_existing_queued_job_nudges_dispatcher`, and
|
||
`test_stream_stays_open_when_durable_replay_has_a_transient_failure`.
|
||
|
||
At ordinary report completion, persist and confirm the session's
|
||
`report_markdown` projection **before** marking the Job `completed`. Retry a
|
||
short transient projection failure; if it cannot be confirmed, let the job
|
||
become failed rather than reporting a report that vanishes on refresh.
|
||
`test_report_is_persisted_before_job_is_marked_completed` and
|
||
`test_report_projection_failure_never_marks_job_completed` cover the ordering.
|
||
|
||
The chat-collection frontend must keep report-writing progress in the existing
|
||
workspace `MessageList` / `MessageGroup` tool-step renderer: inject only
|
||
virtual `deep_research_progress` tool calls for the durable phases, keep the
|
||
completed phase visible after the job ends, and append the digest as a normal
|
||
assistant message. Do not recreate a second deep-research-only conversation
|
||
timeline. On `summarizing`, insert that same normal assistant message with its
|
||
streaming marker and update it for every SSE `summary_delta`; when the report
|
||
completes, replace its content in place with the final completion/digest
|
||
message rather than waiting to append a summary. Completed chat-collection
|
||
sessions also inject a normal `present_files` card into the transcript, named
|
||
from the report Markdown H1 rather than its internal `report.md` path.
|
||
Replaying a completed history session must not auto-open its
|
||
sandbox; opening that card selects its virtual write-file report instead, and
|
||
its file-card download action delegates to the Deep Research Markdown/Word
|
||
export dialog. The default in-job concurrency and per-user active-job cap are
|
||
both three. Before the user confirms the report configuration, derive its
|
||
displayed material count from unique collector tool-result rows because the
|
||
durable Deep Research source store is populated when the writing job harvests
|
||
that thread. The Deep Research composer model picker must use the installed
|
||
TDesign `Select` backed by `/api/models`; its explicit choice is sent as
|
||
`model_name` on collector turns and applied to `fast_model`, `smart_model`, and
|
||
`strategic_model` when creating a session or starting a report job. Disable it
|
||
while collection or report writing is active so one durable run cannot switch
|
||
models mid-stream.
|
||
|
||
Report structures (`/api/report-structures`) may store an optional
|
||
`retrieval_directions` list (检索方向) and an optional `retrieval_skills`
|
||
list (检索来源 / 指定技能). Empty / unset keeps today's collector
|
||
query-planning, default tools, and runner sub-query generation. A non-empty
|
||
directions list is frozen into `DeepResearchConfig.retrieval_directions`: the
|
||
chat collector's first user message gets a `<retrieval-directions>` block that
|
||
asks it to **split concrete retrievable terms along those facets** (never search
|
||
the direction labels themselves), and legacy `basic`/`detailed`/`multi_agent`/
|
||
`deep` runners pass the same constraint into query-planning instead of using
|
||
`课题 + 方向` as the search string. A non-empty skills list is frozen
|
||
into `DeepResearchConfig.retrieval_skills`: the chat collector's first user
|
||
message gets a `<retrieval-skills>` block, the collector run context receives
|
||
`extra_skills` so `skill_view`/`skill_list` can see those names, and legacy
|
||
material collection runs those skills instead of the default Q&A research-skill
|
||
trio. Helpers live in `deerflow.agents.deep_research.retrieval`.
|
||
|
||
Completed-report Q&A has two endpoints: retain `POST /sessions/{id}/chat` as the
|
||
legacy JSON response for compatibility, and use `POST /sessions/{id}/chat/stream`
|
||
for report-grounded Q&A SSE. The workbench must send a strong scoped edit
|
||
instruction (for example “修改第一段” / “重新生成一下第一段” / “重写第二节”) to
|
||
the separate `POST /sessions/{id}/rewrite/stream` report-section writing pipeline.
|
||
That endpoint requires a deterministic current-report range (otherwise 422), so
|
||
its output is always a replacement candidate rather than a Q&A answer. The
|
||
candidate is streamed in the ordinary message row
|
||
while the sandbox preserves the current report. When generation ends, inject a
|
||
confirmation card. Only the owner’s explicit
|
||
`POST /sessions/{id}/messages/{messageId}/rewrite-proposal` action `apply` may
|
||
persist the assembled `report_markdown`; `dismiss` preserves the original. The
|
||
proposal is bound to a hash of the report it was generated from, so stale
|
||
candidates fail with 409 rather than overwrite a later edit. Ambiguous or
|
||
ordinary evidence questions must remain non-mutating, and cancellation or a
|
||
stream error must never save a partial report. The rewrite prompt receives only
|
||
the selected range plus the selected evidence set, never a new web search.
|
||
|
||
### DeerFlow-local LLMWiki Wiki index
|
||
|
||
`llmwiki.local_wiki_index` is an opt-in local-vector subsystem. WeKnora is the
|
||
Wiki generation/sync source and the Wiki-only fallback while an entire mapping
|
||
has not completed vectorization. The readiness gate requires `state=idle`, a
|
||
successful full scan, matching embedding fingerprint, no failed pages,
|
||
`ready_page_count == local_page_count`, and vectors for a non-empty library.
|
||
Ready mappings read DeerFlow's `llmwiki_wiki_pages` /
|
||
`llmwiki_wiki_vectors` / `llmwiki_wiki_sync_states` tables; incomplete mappings
|
||
in the Wiki-page retrieval service and `llmwiki_search` tool call only WeKnora
|
||
`/knowledgebase/{id}/wiki/pages?query=...` plus Wiki-page detail and never use
|
||
raw chunks as a silent fallback. The pre-answer RAG orchestrator separately
|
||
runs the explicit, authorized `/knowledge-search` raw-chunk channel described
|
||
under **Knowledge retrieval diagnostics** below. Core code lives under
|
||
`deerflow.integrations.weknora.local_index` and
|
||
`deerflow.persistence.llmwiki_index`; preserve the harness → app import
|
||
firewall. Gateway composition injects the store/search/sync services through
|
||
`local_index.runtime`. `enabled` turns on local vector search and `auto_sync`
|
||
defaults to the lease-protected background scheduler. The scheduler performs
|
||
an immediate incremental scan at startup, then rescans every configured
|
||
interval so new WeKnora libraries and asynchronously generated Wiki pages are
|
||
picked up without a manual vectorization click. Unchanged content must reuse
|
||
existing vectors.
|
||
|
||
Vectors are normalized little-endian float32 BLOBs. A page mirror and all of
|
||
its vectors are replaced in one transaction only after strict batched
|
||
embedding succeeds. Never delete the old generation before a remote embedding
|
||
call. The local Wiki-vector service must not silently substitute WeKnora raw
|
||
chunks when embedding fails; the pre-answer orchestrator's separately labelled,
|
||
authorized raw-chunk channel is independent rather than that fallback. NumPy
|
||
matrices are per-mapping LRU snapshots invalidated by
|
||
`index_revision`; large decode/matrix work runs through `asyncio.to_thread`.
|
||
`embedding.dimensions` is a local response-size invariant and fingerprint
|
||
input, not an outbound API field: fixed-dimension BGE/OpenAI-compatible
|
||
endpoints can reject a `dimensions` request parameter with HTTP 400. When the
|
||
value is omitted, the embedding client probes and freezes the first response's
|
||
dimension before sync lease/rebuild decisions.
|
||
Embedding errors must retain the upstream status and bounded response body in
|
||
logs, without exposing API keys. SQL vector writes are split into statements
|
||
of at most `database_batch_size` rows (default 50) inside the same atomic page
|
||
replacement transaction.
|
||
During a fingerprint rebuild the mapping uses the Wiki-page API until every
|
||
page is successfully rebuilt. `WikiRetrievalService` partitions mixed requests
|
||
per mapping, so ready libraries stay local while incomplete libraries use that
|
||
Wiki-only fallback.
|
||
|
||
Library-level overview requests (for example, “此知识库都讲了什么” or “这个知识空间有哪些内容”) are
|
||
recognized once by `local_index.overview` and shared by local vector and remote Wiki-page retrieval. A completed
|
||
local index returns representative page bodies without embedding the broad query or applying `min_similarity`; an
|
||
incomplete index asks the remote page API for its directory rather than attempting a wording match that can return
|
||
zero. In both paths directory, index, and summary pages are ineligible, and a page summary must never substitute
|
||
for its body. `AssistantKnowledgeRepository` implements the same user-facing overview semantics independently for
|
||
assistant knowledge bases.
|
||
|
||
**Knowledge retrieval diagnostics.** `app.gateway.llmwiki_rag.build_llmwiki_rag_context` is the single
|
||
pre-answer retrieval path for both normal Q&A and `POST /api/llmwiki/retrieval-debug`; do not build a second
|
||
debug-only ranking implementation. The debug endpoint accepts explicit ordinary and assistant knowledge-base ids
|
||
and returns the bounded `knowledge_retrieval_trace`: selected/authorized spaces, actual retrieval mode, channel
|
||
candidate counts, vector threshold diagnostics, warnings/fallback, zero-result reason, timings, and the exact
|
||
bounded evidence made available to the model. A normal run stores the same trace on the latest `HumanMessage`,
|
||
publishes it in the initial SSE `metadata` event, and exposes durable pre-retrieval plus model-driven
|
||
`llmwiki_search` tool traces through `GET /api/threads/{thread_id}/reference-batches` under `retrieval_traces`.
|
||
Keep trace results bounded to the same eight sources and 1400 characters per source as the answer prompt; never
|
||
expose provider credentials or unrelated library contents. If a client omits selection keys on a later turn,
|
||
recover the persisted thread selection; an explicitly present empty list must continue to clear the scope.
|
||
|
||
Ordinary pre-retrieval searches both generated Wiki page bodies/sections and authorized WeKnora raw document
|
||
chunks. The raw channel must be called only with remote ids derived from already-authorized DeerFlow mappings,
|
||
must reject every returned row whose remote id is not in that authorized map, and must allowlist/bound source
|
||
fields before adding them to prompts or traces. Run raw-chunk search concurrently with Wiki retrieval; one channel
|
||
failing marks the trace partial but must not discard successful evidence from the other. A ready local Wiki snapshot
|
||
uses vector + lexical channels with RRF; incomplete mappings use lexical Wiki recall and the remote Wiki-page API;
|
||
Wiki and raw result lists are fused without comparing provider score scales. Raw provider scores are channel scores,
|
||
not cosine similarity. Assistant knowledge runs vector and keyword/raw-chunk recall together and fuses both ranked
|
||
lists with weighted RRF; exact literal identifiers such as device or incident codes take priority, so imported raw
|
||
chunks remain searchable even when compatible Wiki vectors exist. This is not guaranteed pure-vector retrieval on
|
||
every turn. The same pre-answer orchestrator performs bounded graph expansion for ordinary Wiki links and imported
|
||
assistant entity relations whenever the selected repository exposes `get_adjacent_pages`. It uses up to four directly
|
||
recalled Wiki pages as
|
||
seeds (two per authorized mapping), traverses one hop by default and at most two hops for explicit multi-hop
|
||
wording or low-confidence vector diagnostics, inspects at most 24 adjacency rows, and accepts at most six pages
|
||
(three per mapping). Every returned page is rechecked against the authorized local mapping ids, must contain a
|
||
real eligible Wiki body, and carries a local-only slug/title path plus source/chunk provenance. Decayed walk scores
|
||
rank graph candidates; fixed graph slots merge them with Wiki/raw evidence without comparing unrelated score
|
||
scales. Trace fields include graph seeds/candidates/accepted/used, requested/executed hops, paths, depth reason and
|
||
`timings_ms.graph_expansion`; capability is dynamic. Graph failure marks the trace partial and must never discard
|
||
successful Wiki, raw-chunk, or assistant evidence. Assistant relation traversal resolves both endpoints through the
|
||
entities' `wiki_page_id`, and requires the relation, both entities, and both pages to be active and inside the same
|
||
selected assistant base; it must never follow cross-base relations.
|
||
|
||
The frontend must keep Wiki citations visibly distinct from raw-document
|
||
citations while hiding provider-brand wording from user-facing labels, status
|
||
text, and errors. Any click on a Wiki source card opens the shared right-side
|
||
source drawer, which renders the complete Markdown page. When the local index
|
||
is disabled, the drawer reads the remote WeKnora Wiki-page endpoint directly
|
||
and must not surface the local-index-disabled response. A Wiki target must not
|
||
also load or render the legacy original-document preview, even when its citation
|
||
still carries a document id. The drawer supports single-page Markdown and Word
|
||
downloads, and its **Open in Wiki** link carries both the DeerFlow knowledge-base
|
||
mapping id and the exact `wikiSlug`. Wiki search-result cards use this same
|
||
drawer instead of navigating immediately. Exact-page navigation uses the native
|
||
`knowledge/{mapping_id}/wiki/view?wikiSlug=...` reader, not the WeKnora detail
|
||
iframe. Keep legacy `knowledge/{mapping_id}?tab=wiki&wikiSlug=...` links fast by
|
||
redirecting them in `LlmWikiKnowledgePage` before its runtime-loading gate.
|
||
Drawer-to-reader navigation stays in the current SPA and passes the fetched
|
||
Wiki page in router state so the article can render before the background query
|
||
refresh completes. The native reader must retain a searchable/collapsible left
|
||
Wiki directory, reconstruct hierarchy from `parent_slug`, `wiki_path`,
|
||
`category_path`, and slug fallback, expand/highlight the active page, and keep
|
||
directory navigation on the same lightweight route rather than mounting the
|
||
WeKnora iframe.
|
||
|
||
Generated HashRouter URLs must retain `window.location.pathname` (for example
|
||
the `/web/` reverse-proxy prefix); never emit root-absolute `/#/...` citation,
|
||
reader, or public-embed links. Protected Wiki image rendering must use real HTTP
|
||
URLs, never `URL.createObjectURL`. Public readers use the published file proxy;
|
||
authenticated readers obtain a short-lived signed capability from
|
||
`/files/access-url` and load `/api/public/llmwiki/files/{token}`. The signed
|
||
payload is restricted to `kind=wiki_file`, mapping id, remote id, validated
|
||
protected path, and expiry, and responses provide an inline UTF-8 filename for
|
||
new-tab viewing and browser Save As. The proxied WeKnora detail iframe must
|
||
likewise leave its scoped HTTP file-proxy URL on every protected image and
|
||
restore it if upstream rendering replaces `src`; its click preview reads that
|
||
same DOM value, so a fetched-and-revoked object URL is not acceptable.
|
||
|
||
Third-party pages embed published Wiki content through the chrome-free route
|
||
`/#/embed/knowledge/{mapping_id}` (optional exact-page `wikiSlug`). It defaults
|
||
to a recognized Wiki index page (title `索引`/`index`, page type `index`, or an
|
||
`index`/`home`/`readme` slug) and promotes that page to the first public-directory
|
||
entry. When upstream pages exist without an explicit index shell, the frontend
|
||
injects a virtual `__index__` page and builds its catalogue from the page list;
|
||
an all-empty dynamic index response must also fall back to that page list. It retains
|
||
directory navigation and supports the shared public-embed `theme` / `bg` /
|
||
`text` appearance inputs. WeKnora dynamically renders index page-type counts,
|
||
nested category headings, page summaries, and `data-slug` links even though the
|
||
stored index Markdown only contains its introduction; the native public reader
|
||
reads the same ordered groups through the published-only
|
||
`/api/public/llmwiki/knowledge-bases/{id}/wiki/index` proxy and renders them with
|
||
`buildWikiIndexCatalogMarkdown`. The Gateway paginates every WeKnora index type
|
||
(`summary`, `entity`, `concept`, `synthesis`, `comparison`) to completion while
|
||
preserving upstream group/item order. Index Markdown
|
||
may also contain WeKnora shell-only placeholder links;
|
||
`resolveWikiIndexPageLinks` matches their visible labels to
|
||
the loaded page title, slug, or alias and rewrites them to exact embedded
|
||
`wikiSlug` routes. Preserve WeKnora's `wiki-content-link` interaction: primary
|
||
brand color, medium weight, dashed underline becoming solid on hover, focus
|
||
outline, and in-reader SPA navigation rather than a page reload. `GET
|
||
/api/public/llmwiki/knowledge-bases` is the anonymous discovery endpoint. It
|
||
validates published ordinary mappings against WeKnora, drops stale mappings,
|
||
and returns safe comprehensive metadata from the WeKnora list/detail/page APIs:
|
||
document/chunk/processing/share counts, non-secret configuration and timestamps,
|
||
Wiki page type/status counts and explicit/generated index entry, plus aggregate
|
||
local-vector state/counts/timestamps. Never expose remote WeKnora ids,
|
||
tenant/creator identity, storage credentials, API credentials, raw vectors, or
|
||
per-page indexing errors. All anonymous reader calls use that
|
||
public namespace and the `/knowledge-bases/{id}` Wiki-page children. Those
|
||
endpoints must resolve with a synthetic anonymous user and `is_admin=False`,
|
||
explicitly require `publication_status=published`, and hard-deny the system
|
||
conversation-deposit mapping. Never point this page at the authenticated
|
||
`/api/llmwiki` reads or weaken the published-only gate merely because the
|
||
mapping id is hard to guess.
|
||
|
||
Owners manually request a metadata-first incremental sync/vectorization through
|
||
`POST /api/llmwiki/knowledge-bases/{id}/sync`; the older
|
||
`POST /api/llmwiki/knowledge-bases/{id}/vectorize` route is a compatibility
|
||
alias. Status is available through `GET
|
||
/api/llmwiki/knowledge-bases/{id}/wiki-index-status`. That response includes
|
||
aggregate `progress` and per-page `pages` details; current-scan progress is
|
||
derived from `active_sync_id` / `last_seen_sync_id`, while each page exposes its
|
||
index state, vector count, timestamps, and bounded error. `last_sync` is the
|
||
persisted, lease-fenced terminal result and includes the sync id/mode/status and
|
||
added/updated/skipped/deleted/failed counters. Frequent progress polling must
|
||
pass `include_pages=false` so it reads only persisted aggregate state; the Wiki
|
||
management view keeps the default detailed response when it needs per-page
|
||
rows. The Wiki management
|
||
page must show the progress bar and a status badge beside every article. It
|
||
also exports all remote processed Wiki pages via `GET
|
||
/api/llmwiki/knowledge-bases/{id}/wiki/export.xlsx`. Directory export must keep
|
||
the full path, current directory, parent path, and dynamic per-level columns;
|
||
long Markdown is split into Excel-safe continuation columns instead of being
|
||
truncated. Workbook creation runs in `asyncio.to_thread`.
|
||
|
||
Internal search authorization always starts from DeerFlow mapping ids and is
|
||
rechecked by the Gateway. External endpoints use the isolated
|
||
`/api/external/llmwiki/wiki` namespace and `X-API-Key`; require published +
|
||
external-enabled + index-enabled on every request and response, and hard-deny
|
||
the `system` / `对话沉淀` mapping. Those API-key endpoints never return remote
|
||
WeKnora ids, provenance refs, credentials, or vector bytes.
|
||
|
||
`POST /api/knowledge/vector-search` is a deliberately separate anonymous API.
|
||
It has no token/API-key requirement or rate limit, applies route-scoped
|
||
`Access-Control-Allow-Origin: *`, searches every published local Wiki index
|
||
regardless of `external_search_enabled`, and still hard-denies the `system` /
|
||
`对话沉淀` mapping before and after search. Its contract is section-level (one
|
||
result per matched vector, so Wiki-page fields and URLs may repeat), default
|
||
`top_k=10`, with optional `include_page_content`. Unlike the protected legacy
|
||
external namespace, this endpoint intentionally returns DeerFlow/WeKnora ids
|
||
and raw source/chunk provenance, but never credentials or vector bytes. Wiki
|
||
links use `llmwiki.local_wiki_index.frontend_base_url` (development default
|
||
`http://127.0.0.1:5174`) and point to the native
|
||
`/#/page/workspace/knowledge/{mapping_id}/wiki/view?wikiSlug=...` route.
|
||
Regression tests:
|
||
`tests/test_llmwiki_local_index_repository.py`,
|
||
`tests/test_llmwiki_wiki_sync.py`, `tests/test_llmwiki_vector_search.py`,
|
||
`tests/test_external_llmwiki_search.py`,
|
||
`tests/test_public_knowledge_vector_search.py`, and
|
||
`tests/test_llmwiki_wiki_retrieval.py`.
|
||
|
||
The sandbox whole-report rewrite may carry an optional `reportOutline` Markdown
|
||
template from the shared report-structure library. Persist the original template
|
||
in the durable input snapshot and append it as an explicit chapter-structure
|
||
constraint before requirements planning and prose generation; a template must
|
||
never be used as permission to invent facts absent from the current report.
|
||
|
||
`DocumentRewriteVersionRepository.create` backs the reversible before-image for
|
||
both artifact and whole-report rewrites. It must flush and serialize the new row
|
||
on the writer before commit, and must never call `session.refresh()` after
|
||
commit: a read/write-splitting MySQL endpoint can route that refresh to a
|
||
lagging replica, falsely report a version-save failure, and trigger the caller's
|
||
intentional restore of the rewritten document.
|
||
|
||
Version snapshots are an undo enhancement, not a condition for preserving an
|
||
already committed rewrite. If snapshot creation fails after the artifact/report
|
||
write, log the exception, complete the rewrite with
|
||
`versionSnapshotFailed: true`, and do not emit a user-facing failure or restore
|
||
the original. The sandbox uses that signal only to enable its ordinary manual
|
||
save fallback.
|
||
|
||
### Offline local-sandbox Office runtime
|
||
|
||
The DeerFlow-only offline deployment uses `LocalSandboxProvider`, so document
|
||
dependencies belong in the backend image rather than a separate sandbox image.
|
||
`deploy/linux-docker-offline/Dockerfile.backend` installs python-docx,
|
||
LibreOffice Writer/Calc/Impress, Pandoc, Poppler, and CJK fonts in addition to
|
||
the MarkItDown Office extras. `file_conversion.py` normalizes legacy
|
||
DOC/XLS/PPT and ODT/ODS/ODP through a per-conversion LibreOffice user profile
|
||
before MarkItDown extraction, avoiding concurrent profile locks. The image
|
||
build and `OFFLINE_INSTALL.sh --office-check` both run
|
||
`scripts/office_runtime_selfcheck.py`; that smoke test creates and re-parses
|
||
minimal DOCX/XLSX/PPTX files and does not need the application database.
|
||
|
||
## Skill/assistant knowledge invariants
|
||
|
||
Skill distillation is implemented in `deerflow.skill_knowledge` and
|
||
`deerflow.persistence.skill_knowledge`; native assistant bases are in
|
||
`deerflow.assistant_knowledge` and
|
||
`deerflow.persistence.assistant_knowledge`. Gateway composition, remote adapters,
|
||
the lease dispatcher, and APIs remain in `app.gateway` so the harness-to-app
|
||
import firewall is preserved.
|
||
|
||
Never execute a skill or read/upload/summarize its code and script files.
|
||
Knowledge fingerprints exclude those files and Markdown code fences. All
|
||
resyncs must stage and validate the knowledge digest before activation; the old
|
||
snapshot remains authoritative on failure. A binding may have only one active
|
||
item (`active_dedupe_key` plus atomic lease claims). Skill distillation targets
|
||
only writable WeKnora Wiki knowledge bases in this product version; reject direct
|
||
assistant-base targets, non-Wiki/FAQ/conversation/readonly targets, and direct
|
||
Neo4j writes. The create/list/resync/retry/detach flow is available to ordinary
|
||
users for their own visible skills, writable knowledge bases, jobs and bindings;
|
||
administrator-only auto-sync/review APIs remain privileged. Detach/delete operates on source contributions, not canonical rows,
|
||
and shared entities/relations are retired only when their final contribution is
|
||
gone.
|
||
|
||
Assistant knowledge has one global full-library base (`知识梳理总库`) and any
|
||
number of admin-created custom bases. Administrators always retain its management
|
||
surface and own all mutations. Day-to-day ordinary-user exposure is controlled by
|
||
`system_settings.json -> knowledge_base_product_version`: `v1` (the default) hides
|
||
assistant navigation, selectors and cross-links so ordinary users see only ordinary
|
||
knowledge; `v2` releases those surfaces again. The switch must not delete persisted
|
||
assistant bindings, and administrators remain able to configure assistant knowledge in
|
||
either mode. This is a recoverable product-surface switch rather than an assistant read
|
||
API authorization boundary; hidden assistant selections are omitted from new day-to-day
|
||
chat runs while their persisted bindings remain intact. Assistant bases accept local documents independently
|
||
of WeKnora via `/api/assistant-knowledge/imports/files`, as well as WeKnora export
|
||
packages or imports triggered from a WeKnora mapping. Local documents use the
|
||
Gateway `assistant_file_ingest` pipeline: persisted original files, chunked LLM Wiki
|
||
extraction, cached body embeddings, and import-job status/retry. Do not add direct skill
|
||
imports. Importing into a
|
||
custom assistant base must also mirror the package into the global base for
|
||
cross-source entity alignment. Admins may rename any assistant base and delete
|
||
custom bases. Deletion removes the custom base's Wiki pages, vectors, entities,
|
||
relations, import audit rows, and package files; the global base is never
|
||
deletable because it owns full-library aggregation.
|
||
|
||
Assistant-knowledge reads keep the landing and article paths size-bounded.
|
||
`list_bases()` is metadata/statistics-only and must never call the lazy global
|
||
alignment pass: imports align their snapshot before commit, while content reads
|
||
use the last committed snapshot without waiting for the global import/alignment
|
||
locks; search and export retain `_ensure_base_aligned` for legacy data. Directory,
|
||
article, source and import-job reads resolve their base with `include_stats=False`
|
||
so loading/polling does not recount every content table. The Wiki directory calls
|
||
`GET /api/assistant-knowledge/wiki-pages?include_content=false`; repository
|
||
`load_only` projection must exclude both `content_markdown` and `search_text`
|
||
(response-only deletion is insufficient), and the frontend loads only the
|
||
active slug through `GET /wiki-pages/{slug}`. The explicit index sorts into the
|
||
first directory batch; that batch unlocks the reader immediately while later
|
||
500-row batches append in the background. A URL-provided slug must fetch in
|
||
parallel with the directory. An existing base with zero sources and zero Wiki
|
||
pages is a valid, reachable state: keep the Wiki management surface mounted and
|
||
show upload/package-import actions instead of requiring a document or redirecting
|
||
away. The ordinary Wiki empty reader must likewise link writable users to the
|
||
manual Wiki editor. Do not enable global pages, sources, or import-job queries
|
||
while the landing route has no `base_id`.
|
||
|
||
The assistant-base switcher is a hot metadata path. `list_bases()` must load all
|
||
four counters with one grouped statement rather than four serial counts per
|
||
base, and a metadata-only rename must not rescan content tables. Assistant
|
||
vector search validates base ids without loading counters, checks vector
|
||
fingerprint/dimensions before encoding, scans only vector blobs plus minimal
|
||
page identity in matrix batches, and hydrates bodies only for final top results.
|
||
Chinese lexical fallback must extract bounded meaningful subphrases rather than
|
||
treating the entire no-space question as one SQL token.
|
||
|
||
The shared WeKnora runtime client retains its authenticated session and caches
|
||
the upstream knowledge-base directory for 15 seconds, coalescing concurrent
|
||
scope requests. Knowledge-base and document mutations invalidate that cache;
|
||
generation checks prevent an older in-flight list response from repopulating it.
|
||
|
||
Local Wiki synchronization is metadata-first. A WeKnora page-list item that
|
||
contains the body is authoritative for hashing and must not trigger a detail
|
||
GET. For legacy list shapes without a body, only matching explicit `version` or
|
||
`updated_at` markers may skip the detail GET; missing markers must fail safe to
|
||
the detail API. A content-identical page still refreshes category/parent/path,
|
||
source/chunk references, page version and remote timestamps under the same
|
||
mapping lease, without replacing vectors. Missing pages are retired only after
|
||
a complete paginated scan and a final owner/sync-id lease fence. A bodyless
|
||
metadata-only scan is not a complete graph snapshot: if the provider graph call
|
||
also fails, preserve the prior graph rather than replacing it with partial
|
||
edges.
|
||
|
||
Both Gateway document APIs and successful content mutations through the
|
||
embedded WeKnora proxy schedule one per-mapping debounced retry sequence with
|
||
bounded 1/8/30/90/300-second backoff. Never cancel an in-flight sync to debounce
|
||
a newer mutation; wake the coordinator, update the local generation, and let
|
||
the database lease deduplicate workers. While WeKnora generates Wiki content,
|
||
poll both the knowledge-base state and per-document pending subtasks, then retain a post-processing scan and
|
||
one short quiet confirmation. Before retiring missing pages or replacing graph
|
||
edges, the sync service must re-read processing state; processing or an
|
||
unverifiable state produces a retryable partial result and preserves the prior
|
||
page/vector/graph snapshot. The periodic scheduler reads the remote
|
||
knowledge-base directory once per tick, skips unchanged/processing libraries
|
||
(including forced reconciliation), and retains a 10-minute fallback.
|
||
|
||
### Wiki package and local encoder invariants (2026-09)
|
||
|
||
Ordinary knowledge detail landing is globally configured by `ordinary_knowledge_landing` (`wiki` default / `weknora`), with admin-only PUT `/api/system-settings/ordinary-knowledge-landing`. Explicit document/graph/article links bypass this default. Wiki reading and the embedded WeKnora view provide reciprocal navigation; `?view=weknora` explicitly selects the iframe and must not loop back to the default Wiki route.
|
||
|
||
- `knowledge_transfer.sync_to_global` is called after successful explicit local
|
||
Wiki vectorization; it skips conversation/system mappings. Concurrent triggers
|
||
for one mapping are coalesced inside a Gateway process around the complete
|
||
check/create/import sequence. Cross-worker safety lives at the repository
|
||
transaction boundary: every assistant import takes the target base row
|
||
`FOR UPDATE` before graph alignment or dependent snapshot writes, preserving
|
||
one lock order across workers. Retry only recognized transient lock-wait,
|
||
deadlock and serialization-contention errors; let the failed transaction roll
|
||
back fully, then replay the whole package with bounded backoff. Cancellation
|
||
and non-contention failures must propagate unchanged. Do not enable a new
|
||
watcher or automatic remote-write scheduler as a side effect.
|
||
- `wiki_packages` exposes file-backed assistant exports and restores into empty
|
||
ordinary Wiki libraries. Validate vectors before any remote mutation; recreate
|
||
folder IDs from category paths using provider APIs (a raw category_path is
|
||
discarded by WeKnora without a matching folder_id). Preserve page metadata and
|
||
rebuild backlinks after restoring all pages.
|
||
- `PackageArchive` is a replayable, disk-backed JSONL staging index. Always close
|
||
it in finally. Runtime imports must not use the legacy in-memory read_package.
|
||
- Preserve normalized float32 base64 bytes exactly. Same dimension alone is not
|
||
model compatibility. Assistant vector retrieval is fingerprint-gated cosine
|
||
search, and any keyword fallback must be labeled as such.
|
||
Preserve a valid zero section_index. Export/search vectors belonging to each
|
||
current source snapshot (`source.last_import_job_id`), not merely the page's
|
||
current revision: one aligned page may intentionally contain current vectors
|
||
from several sources.
|
||
- The `local` embedding provider uses FastEmbed with official BAAI/bge-m3 ONNX,
|
||
1024 dimensions, bounded CPU batches and a per-client async lock. Run inference
|
||
off the event loop. Existing page vectors are reusable; zero-vector nonempty
|
||
pages need backfill, not a forced rebuild of every page.
|
||
FastEmbed is a standard backend dependency. Encoder configuration, directory
|
||
upload and warmup live only in the administrator global settings dialog.
|
||
When the effective provider is local, its `system_settings.json` path override wins over config.yaml; uploaded
|
||
files are streamed into immutable `.deer-flow/models/bge-m3-revisions/<revision>` directories, never the source tree.
|
||
Encoder warmup is explicitly started from the settings UI/API and must never block application startup
|
||
or turn a model-load failure into a DeerFlow service failure. Local provider
|
||
is offline-only: require local_model_path or DEERFLOW_BGE_M3_MODEL_PATH and
|
||
never let the application download model weights. Keep
|
||
`local_process_isolation=true` by default: inference runs in the bundled JSONL
|
||
worker subprocess, and shutdown must terminate that worker without waiting on
|
||
model teardown.
|
||
- The administrator encoder override is an explicit persisted mode, not an
|
||
inference from whether `model_path` happens to be non-empty. `configured`
|
||
uses the current `config.yaml` provider (including OpenAI-compatible);
|
||
saving/uploading a local directory sets `local` plus a new model revision,
|
||
while `/api/llmwiki/local-encoder/use-configured` switches back. Every switch
|
||
hot-rebuilds embedding/cache/search/sync/retrieval as one graph and updates
|
||
the harness runtime registration. Status must remain readable for a remote
|
||
provider. The scheduler stays alive as a lightweight config watcher but only
|
||
scans when effective `enabled && auto_sync`; keep the default/portable release
|
||
configs at `auto_sync=true` because asynchronous Wiki generation can finish
|
||
after the event-triggered first scan. A changed local model revision must
|
||
change the embedding fingerprint so vectors from replaced weights are never
|
||
reused. Uploaded weights live in immutable revision directories. A replaced
|
||
embedding generation gets a bounded grace period for in-flight work, then
|
||
rejects stragglers, drains real thread work, and releases its model process.
|
||
- Every successful mapping snapshot is mirrored to the global assistant base,
|
||
including empty snapshots so deleted source pages are retired. Re-import is
|
||
source-replacement semantics: absent Wiki/chunk/entity/relation pages must not
|
||
remain visible merely because they existed in an older source snapshot.
|
||
- Provider graph nodes bind to their original wiki_slug. They must not create
|
||
duplicate synthetic entity/relation articles. Store original Wiki metadata
|
||
separately from assistant workflow status. Count Wiki pages separately from
|
||
raw document chunks.
|
||
- Assistant graph retrieval resolves each active relation through both active
|
||
endpoint entities' `wiki_page_id` values and their active Wiki pages. Every
|
||
relation, entity, seed page, and neighbor page must match the requested
|
||
`base_id`; never traverse a corrupt cross-base edge. Pre-retrieval uses the
|
||
same bounded one/two-hop walk as ordinary Wiki, labels graph evidence as
|
||
assistant evidence, and lets a directly recalled non-seed page remain a
|
||
traversal frontier without duplicating it in the answer context.
|
||
- The global base aligns entities by normalized display name and aliases across
|
||
source records, and aligns content Wiki pages by normalized title. Do not use
|
||
source slug as the global canonical identity. Merge unique knowledge blocks
|
||
while retaining source contributions/origins. Uploaded packages must rebind
|
||
the temporary upload source to the stable WeKnora knowledge-base id in the
|
||
package, so re-upload is source-replacement rather than append. Retrieval
|
||
candidates exclude summary/index/directory pages; keyword matching, vector
|
||
snippets, citations and prompts must use knowledge body/sections, never page
|
||
summary as a substitute.
|
||
- Migration `20260903_01` adds nullable page metadata and merges the pre-existing
|
||
knowledge/workflow Alembic heads. Regression coverage is in
|
||
`tests/test_wiki_package_roundtrip.py`, `test_llmwiki_wiki_sync.py` and
|
||
`test_assistant_knowledge_repository.py`.
|
||
|
||
## Startup schema failure invariant
|
||
|
||
Gateway availability must not depend on successful startup DDL. ORM
|
||
`Base.metadata.create_all` (including database auto-create retries) logs schema
|
||
failures and continues with the existing engine/schema; do not restore a bare
|
||
`raise` that aborts FastAPI lifespan. Startup seed reads/writes must be
|
||
best-effort and use a process-local default where one exists. LangGraph
|
||
Checkpointer and Store own separate `setup()` tables, so `deps.langgraph_runtime`
|
||
must fall back to `InMemorySaver` / `InMemoryStore` if entering either persistent
|
||
context fails. The fallback keeps the API process available but is intentionally
|
||
non-durable; operators must still repair the logged database error and restart.
|
||
|
||
Regression coverage lives in `tests/test_ensure_orm_columns_sync.py` and must
|
||
exercise both ORM `create_all` failure and runtime-backend `setup()` failure.
|
||
|
||
## openGauss / GaussDB persistence invariant
|
||
|
||
`database.backend=gauss` is an application-ORM backend, parallel to `mysql`.
|
||
It normalizes `database.gauss_url` to `opengauss+asyncpg://` and requires the
|
||
`gauss` optional dependency (`asyncpg` plus `opengauss-sqlalchemy`). Never route
|
||
a Gauss URL through SQLAlchemy's generic PostgreSQL dialect: the wire protocol
|
||
may connect, but dialect initialization and schema DDL can fail against the
|
||
Gauss server/version behavior. Keep the LangGraph checkpointer on the separately
|
||
configured SQLite backend; the PostgreSQL checkpointer schema is not part of the
|
||
Gauss compatibility contract. Treat both `postgresql` and `opengauss` dialect
|
||
names as timezone-aware PostgreSQL-family engines in portable types and
|
||
dialect-specific session settings.
|
||
|
||
### openGauss Dolphin / MySQL-protocol mode
|
||
|
||
The offline deployment may select `DATABASE_TYPE=gauss_mysql`. Its renderer
|
||
keeps `database.backend=mysql` and sets `DEERFLOW_DATABASE_PRODUCT=gauss_mysql`;
|
||
this explicit product marker is the switch because SQLAlchemy cannot safely
|
||
distinguish Dolphin from MySQL before dialect initialization. Only that mode
|
||
installs `deerflow.persistence.gauss_mysql_compat` for the configured async
|
||
driver. An unset marker, or `mysql`, must retain the stock MySQL dialect and
|
||
schema-maintenance behavior unchanged.
|
||
|
||
Dolphin table existence and additive schema checks use `current_schema()` plus
|
||
the native `pg_catalog` through `gauss_schema.ensure_gauss_schema_sync`.
|
||
Never run the generic MySQL `SHOW CREATE TABLE` reflection sweep, MySQL JSON
|
||
projection backfill, MySQL session-timeout variables, or application-side
|
||
`CREATE DATABASE` retry in this mode: these paths can misread the schema or
|
||
abort the openGauss transaction with error 322. Database/user/schema bootstrap
|
||
remains the offline deployment layer's responsibility. The startup DDL block
|
||
is still best-effort under the global startup-schema invariant and always
|
||
releases its host-local cross-worker lock.
|
||
|
||
`PortableJSON` writes strict JSON for both products. Its legacy decoder is
|
||
available only on a dialect class carrying the Dolphin compatibility marker;
|
||
ordinary MySQL must continue to reject malformed/non-JSON values rather than
|
||
silently applying the Dolphin fallbacks.
|
||
|
||
## 2026-09-14 我的项目与知识库验收修复
|
||
|
||
- 简洁模式的「我的项目」列表、策略工作台和普通问答项目恢复工作区功能导航；窄屏提供展开菜单按钮。
|
||
- 普通问答先创建具名项目，再在项目内创建多个对话。列表读取 `/api/analysis-projects` 的 `qa` 项目，不再读取、标记或自动归入全局普通聊天。
|
||
- 项目设置更新保留既有知识库绑定。原「项目空间」与官方入口继续可用；我的项目下问答路由为 `qa/projects/groups/*`。
|
||
- 策略生成卡片统一先显示思考/执行过程，再显示正文；评判、方案及展开视图沿用相同顺序。
|
||
- 完成评判但尚未勾选策略的项目可以从方案列表进入，在工作台选择后生成；已保存方案在任务修改后仍可打开。旧列表请求不再覆盖新请求状态。
|
||
- 知识库管理页读取上游最新目录，避免远端删除后命中旧缓存；检索调用仍共享短期目录缓存。
|
||
- 管理员，或管理员已开放 `v2` 展示后的普通用户，访问尚未初始化的助手知识库时显示明确空态及刷新按钮。
|
||
|
||
验证：后端相关回归 273 项、前端策略/项目回归 31 项通过；真实 BAAI/bge-m3 编码、指定库向量检索与草稿隔离通过；本地隔离服务的问答项目创建/分组、真实问答、方案整理、助手库创建/编辑/文件入库/向量检索/详情均通过。完整验收记录见仓库 `docs/ji-knowledge-projects-acceptance.md`。不在源码或文档中保存接口密钥。
|
||
|
||
2026-09-14 日常使用复查：补修项目历史入口与移动侧栏、默认智能体同步、项目加载重试、助手知识检索过期响应隔离及无结果指引；更新 ScrollArea 的 React 19 兼容修复。后端再次 273 项通过，前端 33 项通过。范围及限制见仓库 docs/daily-usage-audit-20260914.md。
|
||
|
||
### 知识库扩展与统计检索（2026-09-20）
|
||
|
||
- `llmwiki_knowledge_bases.test_questions` / `retrieval_settings` 保存每库测试问题与检索策略；`NULL` 测试问题表示使用 `knowledge_profile.default_test_questions`，显式空数组表示禁用。列表/详情统一通过 `_mapping_view` 返回解析后的配置。
|
||
- `deerflow.integrations.weknora.local_index.advanced` 是问答与调试页共享的权威实现。实体扩展只允许一跳、数量有上限，来源为别名/Wiki 链接/Markdown 链接；扩展查询各自运行现有 Wiki-only 混合检索，再进行 RRF 融合。不得把扩展改成无限图遍历，也不得绕过当前用户的 mapping-id 授权。
|
||
- 时间区间与计数在后端确定性计算，模型不得自行数检索结果。优先按结构化 `source_refs` 的稳定来源标识去重；没有可识别来源记录时返回 `unit=wiki_page`。日期优先读取来源字段/元数据和 Wiki frontmatter，远端创建/更新时间只是带 `date_source` 的后备。区间统计排除无日期记录并返回 `undated_excluded_count`。
|
||
- `POST /api/llmwiki/knowledge-bases/{id}/retrieval-debug` 与 `llmwiki_search` 使用同一个 `advanced_search`。调试响应必须保留 query plan、statistics、references、diagnostics 和 request id，供 `/page/workspace/knowledge/{id}/retrieval-debug` 及浏览器 Network 面板排错。
|
||
- 全局与单库 retrieval-debug 页面、入口和 API 均仅管理员可用。前端隐藏不是权限边界；两个 POST 路由必须在参数解析后的任何检索、索引运行时重建或证据读取之前完成管理员校验。
|
||
- 回归测试：`tests/test_llmwiki_advanced_retrieval.py`；同时运行 `test_llmwiki_wiki_retrieval.py`、`test_llmwiki_vector_search.py`、`test_llmwiki_local_index_repository.py` 和 `test_llmwiki_weknora.py`。
|
||
|
||
### 知识库标题权重与引用切片（2026-09-22）
|
||
|
||
- 检索排序必须把原始用户查询的标题/别名命中作为高置信信号，不能让实体扩展词
|
||
冒充原始查询。统一实现位于 `deerflow.integrations.weknora.local_index.ranking`；
|
||
普通 Wiki 的向量、关键词、远端 Wiki、原文 chunk 以及助手知识库融合均保留
|
||
`title_match_score`，标题信号以有界乘数并入 RRF/相关性分数，不能作为绝对主排序键。
|
||
普通 Wiki 的全检索链标题加成由
|
||
`llmwiki.local_wiki_index.title_boost_enabled/weight` 控制；关闭或设为 0 必须恢复
|
||
原排序。
|
||
- 引用详情必须同时展示本次命中的切片和文档/Wiki 正文。`/api/llmwiki/sources/context`
|
||
返回 `focus_chunk_id` 及邻近 `chunks`；前端 `WeKnoraSourceDrawer` 使用精确焦点切片
|
||
作为详情顶部的「本次检索命中切片」，不能只重复文档级摘要。
|
||
- 对话页的后续问题保留生成结果，但只允许在消息列表底部渲染。`MessageList` 上报底部
|
||
状态，`ChatPage`、`AgentChatPage` 与分析项目对话通过 `InputBox` 的
|
||
`followupsDisplayAllowed` 控制；上滑隐藏，回到底部恢复。
|
||
### OpenAI-compatible stream regression coverage
|
||
|
||
- `PatchedChatOpenAI` must emit partial tool arguments immediately, before a complete JSON document exists.
|
||
`test_patched_openai_reasoning_streaming.py` replays actual SDK SSE through a gated local transport and
|
||
the Gateway message serializer, covering standard deltas, cumulative snapshots, empty tool headers,
|
||
and repeated complete tool names. Per-invocation/index name state must reset on a new call id; registered
|
||
complete names permit safe deduplication before arguments start, while legitimate name fragments remain intact.
|
||
- Empty `error: {}` is not an upstream error. Empty `delta` beside a `message.reasoning*` extension must not
|
||
discard reasoning; a valid delta field takes priority. The existing per-stream summary records only counts,
|
||
timing and allowlisted request flags (`incoming_reasoning_chunks`, `emitted_reasoning_chunks`,
|
||
`first_reasoning_ms`, `request_enable_thinking`, `repeated_tool_names`), not prompt/answer/argument bytes.
|
||
- Model-facing sandbox `write_file` executes once its full arguments are validated. Partial argument display
|
||
is a frontend virtual artifact, not incremental persistent writes. Do not confuse local replay with a
|
||
production upstream/20-user load test.
|
||
- `buffered_gateway` regression coverage must preserve the native external contract: the provider selects the
|
||
original metadata-only `write_file`, first confirmed body characters from the second ordinary-text request
|
||
appear immediately as canonical incremental `write_file` argument chunks, and LangGraph sees exactly one
|
||
original writer call. It must never publish or execute `begin_write_file`. Ordinary answers stay on one
|
||
provider request; renderer text never appears as assistant content; reasoning stays in the reasoning channel;
|
||
and empty/truncated output cannot reach file I/O. The isolated provider `content` channel is the typed
|
||
transaction boundary; do not add another model-authored JSON wrapper or delimiter. The older
|
||
`artifact_renderer` handshake retains its existing internal-control invariants. Unconfigured models remain
|
||
byte-for-byte on the native path.
|
||
|
||
### Thread operation timing
|
||
|
||
`app.gateway.operation_timing.OperationTiming` adds content-free phase logs and `Server-Timing` headers
|
||
to thread create/search, including existing-thread idempotent returns and HTTP error paths. Preserve direct
|
||
internal Python callers by keeping the injected Response optional. Checkpoint creation remains synchronous
|
||
from the endpoint's perspective; do not silently defer it and break immediate state reads. Phase time includes
|
||
saver lock/pool waits, not just SQL execution; handler time excludes pre-handler queueing. `run_agent` measures
|
||
the synchronous agent factory with `AGENT_BUILD_TIMING` without changing task ContextVars or MCP lifecycle.
|
||
Database archive/restore SQL is a separate DBA-operated handoff, not source code or an application migration.
|
||
Never wire database archiving into application startup or package it as part of this performance change.
|
||
|
||
### Uploaded-file context and bounded reads
|
||
|
||
- `deerflow.uploads.inspection` performs streaming whole-file statistics and
|
||
deterministic head/middle/tail sampling. It reads limits from the harness
|
||
runtime upload-policy registry and never imports `app.*`.
|
||
- `read_file` uses provider-level bounded range reads; both LocalSandbox and
|
||
AioSandbox implement them without fetching the complete file. A range-free
|
||
large local-file read returns the structured inspection payload.
|
||
- `UploadsMiddleware` keeps bounded file metadata/content samples in agent
|
||
state and prefixes a JSON-encoded `<uploaded_files>` manifest to the latest
|
||
real HumanMessage only in the transient model request. It never writes the
|
||
manifest into checkpoint history, and rebuilds legacy state from the upload
|
||
directory, so provider retries, tool loops and later follow-ups retain file
|
||
evidence. Generated Markdown companions are excluded as duplicate uploads.
|
||
- Only convertible Office/PDF originals may resolve Markdown sidecars. Plain
|
||
TXT/Markdown/CSV is sampled directly; same-stem user Markdown must never be
|
||
mistaken for the companion of a text or image upload.
|
||
- Converted companions use collision-free `<original-name>.<ext>.md` names and
|
||
extracted Markdown has a fixed character ceiling. Conversion always runs off
|
||
the event loop with bounded worker concurrency.
|
||
- `grep` scans configured upload sizes with total-byte, time, result and line
|
||
bounds. `ContextWindowGuardMiddleware` infers common provider window fields
|
||
when available. It never guesses a smaller window for an unprofiled model;
|
||
after a real provider context-length rejection it retries with progressively
|
||
bounded requests, prioritising the upload manifest and file-reading tools.
|
||
|
||
### Q&A tool and terminal-stream liveness (2026-09-28)
|
||
|
||
- `sandbox/skill_view.py` bounds warm-validation followers to 30 seconds. A cold catalogue rebuild may wait up to 120 seconds, and overlapping callers in the same process share that one rebuild (including when the metadata cache is off or the extensions file cannot be read) instead of each taking the process lock. The same thread may re-enter the view lock; it must not wait on itself. Timeout fails closed; never publish a stale view, remove a live leader, or bypass skill availability checks. Existing cache/reuse semantics remain intact.
|
||
- LocalSandbox commands close stdin, create an independent process group on POSIX, collect output in temporary files, and terminate the tree on timeout with bounded reaping. `SandboxConfig.command_timeout_seconds` is an optional finite positive number, default 600. Captured output is read with a 1 MiB per-stream memory budget; running commands have a sampled 16 MiB capture guard. AIO lock acquisition is bounded independently of remote request execution.
|
||
- Local text reads reject non-regular files after nonblocking open; range readers stop collecting an oversized current line at the output budget. Preserve the existing next-physical-line continuation convention.
|
||
- `runtime/runs/worker.py` gives terminal checkpoint inspection/status persistence individual 10-second budgets and final metadata cleanup one shared 10-second budget. Always publish the SSE terminator from the cleanup finally. These budgets must never detach executing graphs or rollback writes. Check cancellation and whether a newer run exists before further title/thread-status work; SQL already issued may not be reversible.
|
||
- Frontend terminal recovery checks send epoch/run id, clears the SDK reconnect marker before stopping the local reader, and then applies the checkpoint. Pending submit alone does not identify a new question. Ordinary SDK requests time out in 30 seconds while SSE retains its unlimited total duration.
|
||
- Regression and operational limits: [问答卡住排查与修复](../../docs/问答卡住排查与修复-20260928.md). This checkout contains no `config.example.yaml`; old config files remain valid without the new optional field.
|
||
|
||
### Concurrent skill refresh publication
|
||
|
||
`sandbox/skill_view.py` coalesces overlapping cold/invalidated refreshes by storage/runtime key and availability-config hash. Followers share one final tree validation, then recheck the current cache identity, config hash and capability marker before returning. They never reuse a completed flight, join a different permission generation, cancel a live leader after their 30-second timeout, or return the old capability after a failed refresh. The optimization does not cache filesystem metadata on a TTL or remove the remaining per-call scan cost. Regression: `packages/harness/tests/test_skill_view_refresh_coalescing.py`.
|
||
|
||
TodoMiddleware's internal HumanMessages carry `hide_from_ui=true` and `lc_source=todo`. The frontend's shared `isHiddenFromUIMessage` also recognizes the exact historical names `todo_reminder` and `todo_completion_reminder`; question navigation uses the same predicate. Never filter a genuine user's question just because its text quotes `<system_reminder>`. Regression: `test_todo_reminder_visibility.py` and frontend `todo-reminder-visibility.test.ts`.
|
||
|
||
Measured Windows test scope and remaining limitations are recorded in [本地问答与技能并发验收](../../docs/本地问答与技能并发验收-20260928.md). Frontend compact/agent onboarding mounts TDesign Guide only while a valid step is active: its idle resize listener otherwise dereferences an absent step. Completion/skip persistence and interrupted-tour behavior remain unchanged.
|
||
|
||
### Force-situation consumer skill authentication (2026-09-30)
|
||
|
||
The seven consumer-facing custom skill scripts (`fs-collect`, `fs-task-detail-info`,
|
||
`fs-timeline-info`, `fs-insight-info`, `fs-current-judgment-info`, `fs-target-info`,
|
||
and `fs-connection-info`) reserve line 1 for `# API_URL=...` and line 2 for
|
||
`# AUTHORIZATION=Bearer admin`. `monitor_api_common.monitor_authorization_from_second_line`
|
||
loads that exact second-line value; an explicit CLI `--authorization` remains the highest-priority
|
||
runtime override. Keep certificate verification disabled and proxies bypassed for these intranet
|
||
calls. The active sandbox skill mirror must be synchronized with the canonical custom skills.
|
||
Gateway detail writes emitted by `fs-collect` and `fs-detail-submit` log a token-masked request and
|
||
the corresponding response so an interrupted chain can be distinguished from a failed persistence
|
||
call without exposing the gateway bearer token.
|
||
|
||
### Task-detail custom skill password authentication (2026-10-26)
|
||
|
||
`skills/custom/task-context` and `skills/custom/task-detail-update` authenticate detail reads/writes
|
||
with a short-lived token minted by `POST /api/v1/auth/login/local`, using the runtime-only
|
||
`TASK_DETAIL_AGENT_USERNAME` / `TASK_DETAIL_AGENT_PASSWORD` pair and `TASK_DETAIL_GATEWAY_URL`.
|
||
They must never use general `DEERFLOW_AUTH_TOKEN` or `TOOLS_TOKEN`, nor emit the password or JWT in
|
||
an input, trace, or model-visible output. `/api/parallel-agents/auth/token` is only a legacy fallback:
|
||
it follows `sso_enabled` and returns 403 when external SSO is disabled. Password-login failures other
|
||
than the policy 403 must fail closed; do not fall back to identity-only token minting. Keep the
|
||
canonical custom directories and `.deer-flow/.sandbox-online-skills` mirror synchronized through the
|
||
custom-skill reconciliation flow. `scripts/setup_biz_detail_assistant.py` logs in through the same
|
||
formal endpoint and accepts `DEERFLOW_SETUP_BASE`, `DEERFLOW_SETUP_USERNAME`,
|
||
`DEERFLOW_SETUP_PASSWORD` for deployment overrides; it may fall back to the legacy username route
|
||
only for the formal login policy 403, never for a credential or rate-limit failure.
|
||
|
||
`biz-detail-assistant` must retain the native `ask_clarification` tool. Do not special-case it by
|
||
adding that tool to `requested_excluded_tools`, and do not mount `DetailConfirmMiddleware` to forge
|
||
confirmation cards from a plain model response: those cards are not genuine model tool calls and break
|
||
the standard pause/resume interaction. The final `ClarificationMiddleware` is the single authority for
|
||
all missing-information, scope, approach and risk-confirmation questions.
|