deerflow-code/offline-backend-20260512/backend/CLAUDE.md

CLAUDE.md

?? This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

DeerFlow is a LangGraph-based AI super agent system with a full-stack architecture. The backend provides a "super agent" with sandbox execution, persistent memory, subagent delegation, and extensible tool integration - all operating in per-thread isolated environments.

Architecture:

• Gateway API (port 8001): REST API plus embedded LangGraph-compatible agent runtime
• Frontend (port 3000): Next.js web interface
• Nginx (port 2026): Unified reverse proxy entry point
• Provisioner (port 8002, optional in Docker dev): Started only when sandbox is configured for provisioner/Kubernetes mode

Runtime:

• make dev, Docker dev, and production all run the agent runtime in Gateway via RunManager + run_agent() + StreamBridge (packages/harness/deerflow/runtime/). Nginx exposes that runtime at /api/langgraph/* and rewrites it to Gateway's native /api/* routers.

Project Structure:

deer-flow/
??? Makefile                    # Root commands (check, install, dev, stop)
??? config.yaml                 # Main application configuration
??? extensions_config.json      # MCP servers and skills configuration
??? backend/                    # Backend application (this directory)
?   ??? Makefile               # Backend-only commands (dev, gateway, lint)
?   ??? langgraph.json         # LangGraph Studio graph configuration
?   ??? packages/
?   ?   ??? harness/           # deerflow-harness package (import: deerflow.*)
?   ?       ??? pyproject.toml
?   ?       ??? deerflow/
?   ?           ??? agents/            # LangGraph agent system
?   ?           ?   ??? lead_agent/    # Main agent (factory + system prompt)
?   ?           ?   ??? middlewares/   # 10 middleware components
?   ?           ?   ??? memory/        # Memory V2: provider/manager/builtin/hindsight
?   ?           ?   ??? thread_state.py # ThreadState schema
?   ?           ??? sandbox/           # Sandbox execution system
?   ?           ?   ??? local/         # Local filesystem provider
?   ?           ?   ??? sandbox.py     # Abstract Sandbox interface
?   ?           ?   ??? tools.py       # bash, ls, read/write/str_replace
?   ?           ?   ??? middleware.py  # Sandbox lifecycle management
?   ?           ??? subagents/         # Subagent delegation system
?   ?           ?   ??? builtins/      # general-purpose, bash agents
?   ?           ?   ??? executor.py    # Background execution engine
?   ?           ?   ??? registry.py    # Agent registry
?   ?           ??? tools/builtins/    # Built-in tools (present_files, ask_clarification, view_image)
?   ?           ??? mcp/               # MCP integration (tools, cache, client)
?   ?           ??? models/            # Model factory with thinking/vision support
?   ?           ??? skills/            # Skills discovery, loading, parsing
?   ?           ??? config/            # Configuration system (app, model, sandbox, tool, etc.)
?   ?           ??? community/         # Community tools (tavily, jina_ai, firecrawl, image_search, aio_sandbox)
?   ?           ??? reflection/        # Dynamic module loading (resolve_variable, resolve_class)
?   ?           ??? utils/             # Utilities (network, readability)
?   ?           ??? client.py          # Embedded Python client (DeerFlowClient)
?   ??? app/                   # Application layer (import: app.*)
?   ?   ??? gateway/           # FastAPI Gateway API
?   ?   ?   ??? app.py         # FastAPI application
?   ?   ?   ??? routers/       # FastAPI route modules (models, mcp, memory, skills, uploads, threads, artifacts, agents, suggestions, channels)
?   ?   ??? channels/          # IM platform integrations
?   ??? tests/                 # Test suite
?   ??? docs/                  # Documentation
??? frontend/                   # Next.js frontend application
??? skills/                     # Agent skills directory
    ??? public/                # Public skills (committed)
    ??? custom/                # Custom skills (gitignored)

Important Development Guidelines

Documentation Update Policy

CRITICAL: Always update README.md and CLAUDE.md after every code change

When making code changes, you MUST update the relevant documentation:

• Update README.md for user-facing changes (features, setup, usage instructions)
• Update CLAUDE.md for development changes (architecture, commands, workflows, internal systems)
• Keep documentation synchronized with the codebase at all times
• Ensure accuracy and timeliness of all documentation

Commands

Root directory (for full application):

make check      # Check system requirements
make install    # Install all dependencies (frontend + backend)
make dev        # Start all services (Gateway + Frontend + Nginx), with config.yaml preflight
make start      # Start production services locally
make stop       # Stop all services

Backend directory (for backend development only):

make install    # Install backend dependencies
make dev        # Run Gateway API with reload (port 8001)
make gateway    # Run Gateway API only (port 8001)
make test       # Run all backend tests
make lint       # Lint with ruff
make format     # Format code with ruff

Regression tests related to Docker/provisioner behavior:

• tests/test_docker_sandbox_mode_detection.py (mode detection from config.yaml)
• tests/test_provisioner_kubeconfig.py (kubeconfig file/directory handling)

Boundary check (harness ? app import firewall):

• tests/test_harness_boundary.py ? ensures packages/harness/deerflow/ never imports from app.*

CI runs these regression tests for every pull request via .github/workflows/backend-unit-tests.yml.

Architecture

Harness / App Split

The backend is split into two layers with a strict dependency direction:

• Harness (packages/harness/deerflow/): Publishable agent framework package (deerflow-harness). Import prefix: deerflow.*. Contains agent orchestration, tools, sandbox, models, MCP, skills, config ? everything needed to build and run agents.
• App (app/): Unpublished application code. Import prefix: app.*. Contains the FastAPI Gateway API and IM channel integrations (Feishu, Slack, Telegram, DingTalk).

Dependency rule: App imports deerflow, but deerflow never imports app. This boundary is enforced by tests/test_harness_boundary.py which runs in CI.

Import conventions:

# Harness internal
from deerflow.agents import make_lead_agent
from deerflow.models import create_chat_model

# App internal
from app.gateway.app import app
from app.channels.service import start_channel_service

# App ? Harness (allowed)
from deerflow.config import get_app_config

# Harness ? App (FORBIDDEN ? enforced by test_harness_boundary.py)
# from app.gateway.routers.uploads import ...  # ? will fail CI

Agent System

Lead Agent (packages/harness/deerflow/agents/lead_agent/agent.py):

• Entry point: make_lead_agent(config: RunnableConfig) registered in langgraph.json
• Dynamic model selection via create_chat_model() with thinking/vision support
• Tools loaded via get_available_tools() - combines sandbox, built-in, MCP, community, and subagent tools
• System prompt generated by apply_prompt_template() with skills, memory, and subagent instructions

ThreadState (packages/harness/deerflow/agents/thread_state.py):

• Extends AgentState with: sandbox, thread_data, title, artifacts, todos, uploaded_files, viewed_images
• Uses custom reducers: merge_artifacts (deduplicate), merge_viewed_images (merge/clear)

Runtime Configuration (via config.configurable):

• thinking_enabled - Enable model's extended thinking
• model_name - Select specific LLM model
• is_plan_mode - Enable TodoList middleware
• subagent_enabled - Enable task delegation tool

Middleware Chain

Lead-agent middlewares are assembled in strict append order across packages/harness/deerflow/agents/middlewares/tool_error_handling_middleware.py (build_lead_runtime_middlewares) and packages/harness/deerflow/agents/lead_agent/agent.py (_build_middlewares):

1. ThreadDataMiddleware - Creates per-thread directories under the user's isolation scope (backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/{workspace,uploads,outputs}); resolves user_id via get_effective_user_id() (falls back to "default" in no-auth mode); Web UI thread deletion now follows LangGraph thread removal with Gateway cleanup of the local thread directory
2. UploadsMiddleware - Tracks and injects newly uploaded files into conversation
3. SandboxMiddleware - Acquires sandbox, stores sandbox_id in state
4. DanglingToolCallMiddleware - Injects placeholder ToolMessages for AIMessage tool_calls that lack responses (e.g., due to user interruption), including raw provider tool-call payloads preserved only in additional_kwargs["tool_calls"]
5. LLMErrorHandlingMiddleware - Normalizes provider/model invocation failures into recoverable assistant-facing errors before later middleware/tool stages run
6. GuardrailMiddleware - Pre-tool-call authorization via pluggable GuardrailProvider protocol (optional, if guardrails.enabled in config). Evaluates each tool call and returns error ToolMessage on deny. Three provider options: built-in AllowlistProvider (zero deps), OAP policy providers (e.g. aport-agent-guardrails), or custom providers. See docs/GUARDRAILS.md for setup, usage, and how to implement a provider.
7. SandboxAuditMiddleware - Audits sandboxed shell/file operations for security logging before tool execution continues
8. ToolErrorHandlingMiddleware - Converts tool exceptions into error ToolMessages so the run can continue instead of aborting
9. SummarizationMiddleware (DeerFlowSummarizationMiddleware) - Context reduction when approaching token limits (optional, if enabled). Non-destructive: overrides the stock before_model (which persists a RemoveMessage(REMOVE_ALL_MESSAGES)) to a no-op and instead compresses transiently in wrap_model_call ? the model request is replaced with [summary, *recent] while the full original conversation stays in state (so the frontend keeps rendering the user's real Q&A). The summary message is hide_from_ui and never persisted; summaries are cached per-thread in a process-wide, thread-safe LRU (_GLOBAL_SUMMARY_CACHE, bounded to 512 threads, sticky boundary). The cache is module-level — not on the middleware instance — because make_lead_agent rebuilds a fresh middleware on every run; an instance cache would be cold each turn and force a blocking summary LLM call on every turn of a long conversation. Keyed by thread_id (resolved like ThreadDataMiddleware), so a summary computed on one turn is reused on the next, paying for an LLM call only on the first threshold crossing or a genuine re-compaction. Threshold-first gating: _plan_compression returns passthrough (no compression at all) whenever the full current conversation is still under the trigger — the cache/summary substitution only engages once the real context crosses the configured trigger. Stickiness headroom (_apply_stickiness_headroom, _KEEP_HEADROOM_FRACTION=0.75): after the keep policy picks a cutoff, the kept suffix is forced below ~75% of every trigger before summarizing. Without it a keep in messages (e.g. 20) can itself exceed a token trigger (e.g. 60000) when recent messages are large (tool outputs / pasted docs), leaving the "compressed" context still over the trigger so it re-summarizes on every follow-up question; the headroom makes the sticky boundary actually stick (re-compaction only after genuinely new content). When the genuine (blocking) summarize path runs, it emits a {"type":"context_compacting","message":"正在压缩上下文…"} event on LangGraph's custom stream channel (the cheap reuse path stays silent); the chat frontend's onCustomEvent renders it as a transient toast. Tests: tests/test_summarization_middleware.py
10. TodoListMiddleware - Task tracking with write_todos tool (optional, if plan_mode)
11. TokenUsageMiddleware - Records token usage metrics when token tracking is enabled (optional)
12. TitleMiddleware - Two-phase auto-title (never shows "untitled"): before_model sets an instant question-derived provisional title the moment the first user message arrives (no LLM, zero latency), then after_model asynchronously generates the final LLM title and replaces it (tracked via the title_provisional state flag). Normalizes structured message content before prompting the title model
13. MemoryMiddleware - Memory V2: wrap_model_call injects Hindsight recall transiently; after_agent auto-retains the turn to Hindsight
14. ViewImageMiddleware - Injects base64 image data before LLM call (conditional on vision support)
15. DeferredToolFilterMiddleware - Hides deferred tool schemas from the bound model until tool search is enabled (optional)
16. SubagentLimitMiddleware - Truncates excess task tool calls from model response to enforce MAX_CONCURRENT_SUBAGENTS limit (optional, if subagent_enabled)
17. LoopDetectionMiddleware - Detects repeated tool-call loops; hard-stop responses clear both structured tool_calls and raw provider tool-call metadata before forcing a final text answer
18. ClarificationMiddleware - Intercepts ask_clarification tool calls, interrupts via Command(goto=END) (must be last) 8a. ToolMetricsMiddleware - Appended immediately after ToolErrorHandlingMiddleware in _build_runtime_middlewares, so it sits deepest in the wrap_tool_call chain. Records one tool_call_metrics row per tool/skill/MCP invocation ? timing duration and bucketing failures into a coarse error_category (rate_limited / ip_blocked / auth_error / timeout / network_error / not_found / empty_result / tool_error). It catches both raised exceptions (before #8 converts them) and self-handled errors returned as error ToolMessages (search tools that swallow HTTP 429/403). It also captures the targeted Agent Skill into tool_call_metrics.skill_name: the lead agent invokes a skill by read_file-ing its /mnt/skills/<category>/<name>/SKILL.md path (per the <skill_system> prompt block ? there is no dedicated "use skill" tool), so skill_name is recovered by scanning every tool call's string arguments for the skills/{public,custom}/{name}/ path shape (_SKILL_PATH_RE), plus the explicit name argument of skill_view/skill_manage/skill_archive. skill_view-specific success detection handles its swallowed failures (Skill 'x' not found. returned as a plain string rather than raised). Regression coverage: tests/test_tool_metrics_middleware.py. Writes are best-effort fire-and-forget. Per-run skill merging: SqlToolMetricsStore.record upserts skill rows by (run_id, skill_name) instead of inserting one row per call ? a skill touched by several tool calls / retries inside one run collapses to a single row. Any success in the run wins (a skill that failed then finally succeeded is recorded as one success); a skill that only ever failed yields one error row whose error_message accumulates every attempt's message; durations are summed. Non-skill tool calls, and skill calls with no run_id, keep one row each. The read?merge?write is guarded by a per-event-loop asyncio.Lock. Surfaced by the admin GET /api/admin/tool-metrics router (which has a kind=skill filter and a per-skill /summary breakdown).
19. PromptPrefixMiddleware - Injects the admin-configured prompt prefix into the model request only. The prefix is tagged onto the first human message's additional_kwargs['prompt_prefix'] by the Gateway (app.gateway.services) and prepended to content here at call time, so the persisted/displayed message stays prefix-free
20. SummarizationMiddleware (DeerFlowSummarizationMiddleware) - Context reduction when approaching token limits (optional, if enabled). Non-destructive: overrides the stock before_model (which persists a RemoveMessage(REMOVE_ALL_MESSAGES)) to a no-op and instead compresses transiently in wrap_model_call ? the model request is replaced with [summary, *recent] while the full original conversation stays in state (so the frontend keeps rendering the user's real Q&A). The summary message is hide_from_ui and never persisted; summaries are cached per-thread in a process-wide, thread-safe LRU (_GLOBAL_SUMMARY_CACHE, bounded to 512 threads, sticky boundary). The cache is module-level — not on the middleware instance — because make_lead_agent rebuilds a fresh middleware on every run; an instance cache would be cold each turn and force a blocking summary LLM call on every turn of a long conversation. Keyed by thread_id (resolved like ThreadDataMiddleware), so a summary computed on one turn is reused on the next, paying for an LLM call only on the first threshold crossing or a genuine re-compaction. Threshold-first gating: _plan_compression returns passthrough (no compression at all) whenever the full current conversation is still under the trigger — the cache/summary substitution only engages once the real context crosses the configured trigger. Stickiness headroom (_apply_stickiness_headroom, _KEEP_HEADROOM_FRACTION=0.75): after the keep policy picks a cutoff, the kept suffix is forced below ~75% of every trigger before summarizing. Without it a keep in messages (e.g. 20) can itself exceed a token trigger (e.g. 60000) when recent messages are large (tool outputs / pasted docs), leaving the "compressed" context still over the trigger so it re-summarizes on every follow-up question; the headroom makes the sticky boundary actually stick (re-compaction only after genuinely new content). When the genuine (blocking) summarize path runs, it emits a {"type":"context_compacting","message":"正在压缩上下文…"} event on LangGraph's custom stream channel (the cheap reuse path stays silent); the chat frontend's onCustomEvent renders it as a transient toast. Tests: tests/test_summarization_middleware.py
21. TodoListMiddleware - Task tracking with write_todos tool (optional, if plan_mode)
22. TokenUsageMiddleware - Records token usage metrics when token tracking is enabled (optional)
23. TitleMiddleware - Two-phase auto-title (never shows "untitled"): before_model sets an instant question-derived provisional title the moment the first user message arrives (no LLM, zero latency), then after_model asynchronously generates the final LLM title and replaces it (tracked via the title_provisional state flag). Normalizes structured message content before prompting the title model
24. MemoryMiddleware - Queues conversations for async memory update (filters to user + final AI responses)
25. ViewImageMiddleware - Injects base64 image data before LLM call (conditional on vision support)
26. DeferredToolFilterMiddleware - Hides deferred tool schemas from the bound model until tool search is enabled (optional)
27. SubagentLimitMiddleware - Truncates excess task tool calls from model response to enforce MAX_CONCURRENT_SUBAGENTS limit (optional, if subagent_enabled)
28. LoopDetectionMiddleware - Detects repeated tool-call loops; hard-stop responses clear both structured tool_calls and raw provider tool-call metadata before forcing a final text answer
29. ClarificationMiddleware - Intercepts ask_clarification tool calls, interrupts via Command(goto=END) (must be last)

Configuration System

Main Configuration (config.yaml):

Setup: Copy config.example.yaml to config.yaml in the project root directory.

Config Versioning: config.example.yaml has a config_version field. On startup, AppConfig.from_file() compares user version vs example version and emits a warning if outdated. Missing config_version = version 0. Run make config-upgrade to auto-merge missing fields. When changing the config schema, bump config_version in config.example.yaml.

Config Caching: get_app_config() caches the parsed config, but automatically reloads it when the resolved config path changes or the file's mtime increases. This keeps Gateway and LangGraph reads aligned with config.yaml edits without requiring a manual process restart.

Configuration priority:

1. Explicit config_path argument
2. DEER_FLOW_CONFIG_PATH environment variable
3. config.yaml in current directory (backend/)
4. config.yaml in parent directory (project root - recommended location)

Config values starting with $ are resolved as environment variables (e.g., $OPENAI_API_KEY). ModelConfig also declares use_responses_api and output_version so OpenAI /v1/responses can be enabled explicitly while still using langchain_openai:ChatOpenAI.

Extensions Configuration (extensions_config.json):

MCP servers and skills are configured together in extensions_config.json in project root:

Configuration priority:

1. Explicit config_path argument
2. DEER_FLOW_EXTENSIONS_CONFIG_PATH environment variable
3. extensions_config.json in current directory (backend/)
4. extensions_config.json in parent directory (project root - recommended location)

Gateway API (app/gateway/)

FastAPI application on port 8001 with health check at GET /health. Set GATEWAY_ENABLE_DOCS=false to disable /docs, /redoc, and /openapi.json in production (default: enabled).

Routers:

Router	Endpoints
Models (/api/models)	GET / - list models; GET /{name} - model details
MCP (/api/mcp)	GET /config - get config; PUT /config - update config (saves to extensions_config.json)
Skills (/api/skills)	GET / - list skills; GET /{name} - details; PUT /{name} - update enabled; POST /install - install from .skill archive (accepts standard optional frontmatter like version, author, compatibility)
Memory V2 (/api/memory/v2)	GET / - builtin data (USER.md/MEMORY.md); POST/PUT/DELETE /entries - add/replace/remove entry; GET/PATCH/DELETE /config - effective config + per-user overrides; POST /search - Hindsight recall/reflect proxy
Agents (/api/agents)	GET / - list built-in, owned, and published agents; POST / - create user-owned agent; GET/PUT/DELETE /{id} - read/update/delete by stable agent id; GET /check - validate non-empty display name (duplicates allowed)
Skills (/api/skills)	GET / - list skills (optional search fuzzy-name + tag_id filters; each item carries tags); GET /{name} - details; PUT /{name} - update enabled; POST /install - install from .skill archive (accepts standard optional frontmatter like version, author, compatibility)
Skills (/api/skills)	GET / - list skills (optional search fuzzy-name + tag_id filters; each item carries tags + owner-authored detail); GET /{name} - details; PUT /{name} - update enabled; PUT /{name}/detail - set the long-form detail text (custom skill: owner or admin; built-in/public: admin only ? materializes a skills row on demand); POST /install - install from .skill archive (accepts standard optional frontmatter like version, author, compatibility)
Skills (/api/skills)	GET / - list skills (optional search fuzzy-name + tag_id filters; each item carries tags + owner-authored detail); GET /{name} - details; PUT /{name} - update enabled; PUT /{name}/detail - set the long-form detail text (custom skill: owner or admin; built-in/public: admin only ? materializes a skills row on demand); GET /{name}/content - raw SKILL.md source, restricted to an admin or the publisher (custom-skill owner); other users get 403 and see only description + detail; POST /install - install from .skill archive (accepts standard optional frontmatter like version, author, compatibility)
Skills (/api/skills)	GET / - list skills (optional search fuzzy-name + repeatable tag_ids multi-select filter, OR-matched; each item carries tags + owner-authored detail); GET /{name} - details; PUT /{name} - update enabled; PUT /{name}/detail - set the long-form detail text (custom skill: owner or admin; built-in/public: admin only ? materializes a skills row on demand); GET /{name}/content - raw SKILL.md source, restricted to an admin or the publisher (custom-skill owner); other users get 403 and see only description + detail; POST /install - install from .skill archive (accepts standard optional frontmatter like version, author, compatibility)
Memory (/api/memory)	GET / - memory data; POST /reload - force reload; GET /config - config; GET /status - config + data
Agents (/api/agents)	GET / - list built-in, owned, and published agents (optional search + repeatable tag_ids multi-select filter, OR-matched; each item carries tags and featured_order); POST / - create user-owned agent; GET/PUT/DELETE /{id} - read/update/delete by stable agent id; GET /check - validate non-empty display name (duplicates allowed); GET /selector - homepage agent picker (single admin-curated list, visibility-filtered, sorted by featured_order asc); PUT /{id}/featured - admin sets/clears the homepage featured_order (must be published=true to feature; passing order: null clears unconditionally); PUT /featured/order - admin bulk-reorders the homepage list (body {agent_ids: [...]} ? indices become 0..N-1)
Tags (/api/tags)	GET / - list tags (optional tag_type filter; open to all); POST / - create tag (admin); PUT /{id} / DELETE /{id} - update/delete tag (admin); GET /assignments/{target_type}/{target_id} - tags on a resource; PUT /assignments/{target_type}/{target_id} - replace a resource's tags (admin). Tags are typed agent/skill/scheduled_task and may only tag resources of their own type.
Uploads (/api/threads/{id}/uploads)	POST / - upload files (auto-converts PDF/PPT/Excel/Word); GET /list - list; DELETE /{filename} - delete
Threads (/api/threads/{id})	DELETE / - remove DeerFlow-managed local thread data after LangGraph thread deletion; unexpected failures are logged server-side and return a generic 500 detail
Threads search (POST /threads/search, LangGraph-compat)	Conversation listing. System threads are excluded by default: rows whose metadata.system is truthy (scheduler runs, roundtable workers) never crowd the user's list; a caller whose metadata filter explicitly references thread_type/system (e.g. studio notebooks) opts out of the exclusion. New query field = case-insensitive title (display_name) substring search, used by the chats page's server-side pagination. POST /threads/count takes the same request body (limit/offset ignored) and returns {total} under identical filtering/exclusion semantics ? powers the chats page's ??/??? display without fetching rows (ThreadMetaStore.count). ThreadMetaStore.search gained exclude_system/query params; JSON-blob filters batch-scan until the requested limit/offset window is full (so pagination over a heavily-filtered set never under-fills), with a thread_id tie-breaker on the updated_at DESC ordering for stable OFFSET paging. Tests: tests/test_thread_meta_search.py
Artifacts (/api/threads/{id}/artifacts)	GET /{path} - serve artifacts; active content types (text/html, application/xhtml+xml, image/svg+xml) are always forced as download attachments to reduce XSS risk; ?download=true still forces download for other file types
Suggestions (/api/threads/{id}/suggestions)	POST / - generate follow-up questions; rich list/block model content is normalized before JSON parsing
Thread Runs (/api/threads/{id}/runs)	POST / - create background run; POST /stream - create + SSE stream; POST /wait - create + block; GET / - list runs; GET /{rid} - run details; POST /{rid}/cancel - cancel; GET /{rid}/join - join SSE; GET /{rid}/messages - paginated messages {data, has_more}; GET /{rid}/events - full event stream; GET /../messages - thread messages with feedback; GET /../token-usage - aggregate tokens
Feedback (/api/threads/{id}/runs/{rid}/feedback)	PUT / - upsert feedback; DELETE / - delete user feedback; POST / - create feedback; GET / - list feedback; GET /stats - aggregate stats; DELETE /{fid} - delete specific
Runs (/api/runs)	POST /stream - stateless run + SSE; POST /wait - stateless run + block; GET /{rid}/messages - paginated messages by run_id {data, has_more} (cursor: after_seq/before_seq); GET /{rid}/feedback - list feedback by run_id
Open Chat (/api/open/chat)	Unauthenticated open Q&A entry for external systems (the /api/open/ prefix is whitelisted in auth_middleware._PUBLIC_PATH_PREFIXES + CSRF-exempt in csrf_middleware.should_check_csrf). POST /chat {message | messages[], model_name?, agent_name?, agent_id?, thinking_enabled=false, strip_think=true, thread_id?, recursion_limit=100} → resolves the agent by Chinese display name (agent_name, matched against each .deer-flow/agents/*/config.yaml name via _resolve_agent_id; explicit agent_id wins if given), builds the lead agent via make_lead_agent (so the chosen agent's SOUL/tools/skills all apply), ainvokes it once (stateless — no checkpointer, no thread persistence; runs under the "default" user bucket), and returns only the final {answer, model_name, agent_name, agent_id, thinking_enabled, thread_id} (no intermediate tool-call/process). POST /chat/stream is the SSE streaming variant (same request body) for long tasks — the blocking /chat produces no downstream bytes for the whole run, so a front nginx proxy_read_timeout (default 60s) cuts it off with a 504; the stream keeps the connection warm (a : keepalive SSE comment every _HEARTBEAT_SECONDS=15s via a background producer-task + queue, plus event: delta AI-text increments and event: progress), and ends with event: result carrying the same final {answer, …} JSON (or event: error). Clients only need to read event: result; intermediate events are ignorable. Both endpoints share _build_run_context (config/agent/thread). GET /agents lists {name, id} so callers know what agent_name to send. Thinking off is reliable: when thinking_enabled=false the route also sets thinking_force_disabled=true so the lead agent calls create_chat_model(force_disable_thinking=True), which injects three OpenAI-compatible "thinking off" shapes into extra_body — top-level enable_thinking=false (SiliconFlow / DashScope hybrid models like deepseek-ai/DeepSeek-V4-* on api.siliconflow.cn — the knob the nested chat_template_kwargs shape does NOT reach), chat_template_kwargs.enable_thinking=false (vLLM/Qwen native), and thinking.type="disabled" (Zhipu GLM) — even for models that only declare supports_thinking: true. For the web-UI path (which sends thinking_enabled=false but NOT force_disable), SiliconFlow models need when_thinking_disabled: {extra_body: {enable_thinking: false}} in their config.yaml entry (added to deepseek-ai/DeepSeek-V4-Flash) for thinking-off to take effect. Sets is_scheduled_run=true internally to skip the auto-title LLM call. Generated-file return (include_files=true, default on): an agent that produces its report as a file (writes to /mnt/user-data/outputs/ + present_files) returns only a short summary in answer; to recover the real deliverable both endpoints now also return `files: [{name, virtual_path, encoding(text
Thread Shares (/api/thread-shares)	Per-conversation sharing with two modes (thread_shares.mode). import: POST / {thread_id, mode} - create (or reuse) a revocable share code; GET /{code}/preview + POST /{code}/import - copy that conversation into the caller's account (fresh thread_id via app/gateway/thread_copy.py + new threads_meta row + user-data copy; dedup on metadata.shared_origin). view: a public, read-only static page ? see Public Shares below. Both modes: GET / - list my codes; DELETE /{code} - revoke.
Public Shares (/api/public/shares)	Unauthenticated (the /api/public/ prefix is whitelisted in auth_middleware._PUBLIC_PATH_PREFIXES). GET /{code} - read-only conversation snapshot (title + serialized messages + artifacts) for a view-mode share; messages are read straight from the latest checkpoint. GET /{code}/files/{path} - serves a file referenced by that conversation (images, Markdown, text, PDF, ?) so the shared page's links are openable; path resolution is confined to the owner's thread directory. Active content (text/html, application/xhtml+xml, image/svg+xml) is forced to download (_public_share_disposition); everything else is served inline. Missing/revoked/non-view codes all return 404.
Roundtable Drafts (/api/roundtable-drafts)	Strictly per-user persistence for the roundtable-planning page (replaces the old browser localStorage). GET / - list drafts (lightweight meta — incl. chain_title, the business-chain name used this session, cheaply regex-extracted from the in-memory step2 blob in _draft_to_meta/_extract_chain_title; null = no chain / 自由模式; surfaced on the 第一步「分析记录」history card + sidebar list instead of the old「第 X 步」); POST / - create; GET/PUT/DELETE /{id} - full read / partial update / delete (all owner-scoped). A draft = one full session (Step 1 intent + Step 2 roundtable), step snapshots stored as JSON in PortableLongText. Recommendation history bound to a draft: GET /{id}/recommendations - list newest-first; POST /{id}/recommendations - append one run (objective/status/rationale/picks/candidates). taskId 深链会商记录 are not here — they live in the separate /api/roundtable-task-drafts store (see below) so personal history never mixes with task records. Backed by deerflow.persistence.roundtable_drafts (tables roundtable_drafts, roundtable_recommend_history); store wired in deps.py as app.state.roundtable_draft_store. Tests: tests/test_roundtable_task_drafts.py.
Roundtable Task Drafts (/api/roundtable-task-drafts)	Task-scoped, unpartitioned store for taskId-deeplink roundtable 聊天记录 (无界嵌入抽屉). Keyed only by task_id (one task → many records), never partitioned by user — any authenticated caller may read/write/delete a task's records (created_by is audit-only). Physically separate from roundtable_drafts so the two never mix. GET /by-task/{task_id} - all drafts for a task, newest-first (history dropdown), enriched with the latest background-job status per draft via RoundtableJobRepository.status_map_by_drafts (each meta also carries chain_title, same _extract_chain_title regex as the personal store); GET /by-task/{task_id}/latest - newest draft (回显最新一条); POST / - create (task_id required); GET/PUT/DELETE /{id} - full read / partial update / delete (all user-agnostic). Backed by deerflow.persistence.roundtable_task_drafts (table roundtable_task_drafts, auto-created by create_all; migration 20260618_01); store wired in deps.py as app.state.roundtable_task_draft_store. Existing task_id-bearing roundtable_drafts rows are migrated over by scripts/migrate_roundtable_task_drafts.py. Tests: tests/test_roundtable_task_drafts.py, tests/test_roundtable_drafts_by_task.py.
Roundtable Diagnostics (/api/roundtable-diagnostics)	会商全流程诊断日志 (报错 + 关键事件). One row per event in roundtable_diagnostics: {scope (background = 后台挂起 / foreground = 前台服务端观测 / client = 前端上报), stage (step1_intent/step2_init/step2_leader/step2_seat/step2_recommend/step2_orchestration/step3_summary/step3_report/step3_dashboard/step3_structure/upstream/job), level (warning/error — **info 级运行日志不再收集**：record_diagnostic在写库前丢弃level=="info"，只留报错；begin/consensus/gather/job_started 等纯进度事件即使仍调 .diag(level="info") 也成 no-op，故诊断页 = 报错日志页), event, message, detail (JSON), job_id/draft_id/task_id/user_id/agent_id/agent_name/cycle, created_at}. Reads are admin-only: GET / - filter (scope/stage/level/event/job_id/draft_id/task_id/user_id/q/since/until; scope accepts a comma-separated set so 前端「前台调用」= foreground,client) + paginate (newest-first), each item enriched with userName (user_id → 邮箱 @-prefix via get_local_provider().get_user, best-effort batched cache); GET /facets - distinct values for filter dropdowns; POST /admin/cleanup - retention purge. POST /report is NOT admin-gated — any authenticated user's frontend posts the errors the user actually hit (HTTP 502 before any SSE frame / network drop / SSE error frame) so they're collected too; stored as scope="client". Backed by deerflow.persistence.roundtable_diagnostics (table roundtable_diagnostics, auto-created by create_all; migration 20260619_01); store wired in deps.py as app.state.roundtable_diagnostic_store + a process-level default store (register_default_store) so the harness-side engine/gateway can write across the app/harness boundary. Writers (all best-effort — diagnostics never break a run): background path uses DiagnosticsRecorder (bound to job ctx) threaded through RoundtableJobExecutor → JobProgressWriter.diag(...) (engine transitions/step3 failures/empty-delivery) + InProcessRoundtableGateway (per-turn timeout/report·summary·dashboard truncation+missing); foreground path uses app/gateway/roundtable_diag.py::record_foreground_diag in multi_agent.py/intent.py/recommend.py error frames. Bug-fix shipped alongside: InProcessRoundtableGateway._run_agent_turn guards each agent run with an inactivity (idle) timeout (_await_run_with_idle_guard, ROUNDTABLE_SEAT_IDLE_TIMEOUT_SECONDS default 1200s = 20min, tuned high for unstable intranet models) — not a total-duration cap, so a slow-but-alive run that keeps streaming tokens runs as long as it needs; only a run that emits zero StreamBridge events for idle_timeout seconds straight (a true freeze: 内网模型连接池挂死 / sqlite 锁在挂载卷上挂死) is cancelled + raises (顶层把 job 标 error，而非永久冻结) + records a seat_turn_timeout diagnostic. Activity is observed via app.state.stream_bridge (the worker publishes every chunk there). Optional absolute cap ROUNDTABLE_SEAT_MAX_TOTAL_SECONDS (default 0 = off). Frontend: admin page pages/RoundtableDiagnosticsPage.tsx (route /page/strategy/admin/roundtable-diagnostics), opened from the 会商页右上角设置按钮 (admin only — handleSettingsClick in roundtable-planning/pages/RoundtablePlanningPage.tsx); admin read client strategy-components/api/roundtable-diagnostics.ts. Client error reporter roundtable-planning/api/diagnostics-report.ts (reportRoundtableError fire-and-forget + keepalive, setRoundtableErrorContext for draft/task attribution set by the page, stageForRoundtableAgent) is called at every client-observable failure: multi-agent.ts (streamMultiAgent — covers Step2 leader/seat AND Step3 report/summary/dashboard/structure — reports HTTP error + SSE {status:"error"} frame + worker event:error frame [模型调用没成功] + fetch-reject [网络不可达] + mid-stream drop [连接重置/超时], deduped via a reported flag; initMultiAgent HTTP/error-frame/incomplete/mid-stream), intent.ts (Step1 HTTP+error-frame), recommend.ts (Step2 recommend HTTP+error-frame), roundtable-jobs.ts (job api/stream). The admin page's first filter is a 后台挂起 / 前台调用 toggle (frontend = foreground,client); the stage dropdown options adapt to it (background → step2+step3+job; frontend → step1+step2+step3) and each row shows the reporting user's name + 👤. Tests: tests/test_roundtable_diagnostics.py, tests/test_roundtable_inprocess_timeout.py. Seat skill-recognition (弱模型补偿): a seat = one custom agent run whose configured skills (agent config.yaml → skills) are injected only via the system prompt's <skill_system> block (model must read_file the SKILL.md itself) — weak models (deepseek-chat 等) ignore it and never use their skills. app/gateway/roundtable_seat_skills.py::append_seat_skill_directive(agent_id, task, *, chain_enabled=False) appends an explicit Chinese directive (each configured-and-enabled skill's name + description + SKILL.md container path + "先 read_file 再按其流程取信息/执行") to the seat's per-turn task (human turn) — far stronger pull on weak models than the system prompt. Wired into both seat paths: foreground multi_agent._special_run (only role in {seat, seat_parallel}; Step3 singletons report/summary/dashboard/ingest excluded) + background InProcessRoundtableGateway.run_seat. No-op (returns task unchanged) when the seat has no configured skills. 2026-06-20: default-off + two opt-in switches (OR semantics) — was global-on, now injects only when either the seat agent's own switch (AgentConfig.seat_skill_directive, edited in the agent edit dialog agent-card.tsx + agent-detail-panel.tsx) or the per-seat switch on that agent within the selected business chain is on. The per-seat chain switch lives in the chain's seats JSON (ChainSeatSchema.seat_skill_directive, no DB column — rides the existing JSON like the per-seat model override), edited via a small ✨ icon → dropdown (Switch + description) on each agent card in BusinessChainEditorPage. It is threaded per-seat to each seat run: foreground multi_agent per-call field seat_skill_directive (orchestration looks it up per agentId from a {agentId: bool} session map) → _special_run → chain_enabled; background as the job field seat_skill_directives: dict[str,bool] (StartJobRequest → StartParams → deps factory → InProcessRoundtableGateway._seat_skill_directives) → run_seat looks up agent_id → chain_enabled. ROUNDTABLE_SEAT_SKILL_DIRECTIVE is only a master kill-switch (default allow; set 0/false to hard-disable regardless of the switches — never force-enables). Tests: tests/test_roundtable_seat_skills.py, tests/test_agents_backend.py. 2026-06-20 Step2 提速三件套：(1) 每席位推理深度覆盖 —— 业务链条 seats JSON 里逐席位可配 seat_mode（flash/thinking/pro/ultra，覆盖链条级 seatMode；编辑器席位卡右侧 ⏱Gauge 图标下拉，null=跟随链条默认）。前台经 useStep2Orchestration 三处 dispatch 的 seatModeToRequestFields(seatModes[id] ?? selectedMode) 透传；后台经 job seat_modes: dict[str,str] → StartParams → InProcessRoundtableGateway._seat_modes，run_seat 用 _SEAT_MODE_PARAMS（与前端 seatModeToRunParams 逐字对齐）解析覆盖作业级 seat_*（未命中→跟随作业级→policy 默认）。无 DB 列（ride seats JSON，但 ChainSeatSchema.seat_mode 须显式声明否则 Pydantic 剥掉）。Tests: tests/test_roundtable_seat_modes.py。(2) recommend 批内并行 —— 后端 engine._run_recommend 由 for-await 串行改 asyncio.gather（对齐前端 2026-06-13 已并行；总控一轮派多席位=判定相互独立→并发；批内用派活前 deliveries 快照、批后统一并入，语义不变）。(3) 研讨前并行『取数』两段式（业务链条 gather_first 列，逐条 opt-in，migration 20260620_02，editor「研讨前并行取数」开关）—— gather_first=True 时 run_orchestration 在分发前先 engine._run_gather_phase（全席位 asyncio.gather 跑 _GATHER_TASK「只取数不研讨」），产出 gathered: dict 经 _seat_message(gathered=...) 作为「已收集资料」块注入 recommend/chain/dag 各研讨轮席位任务（与「前序交付」块分开），把最慢的技能延迟前置并行、给（尤其 chain 串行的）研讨段提速；研讨阶段不限制再调技能。前台对称实现：useStep2Orchestration.runGatherPhase 在 runActiveOrchestration 分发前跑一次（gatherPhaseDoneRef 闸门，resetInitGate 复位），产出存 gatheredDeliveriesRef 经 peerDeliveries/初始 priorDeliveries 注入研讨席位。链条级 gatherFirst(camel)↔gather_first(snake) 经 api/chains.ts 映射；job 经 StartJobRequest.gather_first→StartParams→run_orchestration(gather_first=…)。Tests: tests/test_roundtable_gather_phase.py。(4) 削减重复广播（前端 only，多智能体 foreground 路径）—— suppress_pre_broadcast（仅 leader：抑制「总控本轮输入→各席位 thread」预广播 + 「总控给X派活」→兄弟席位的 post 广播，二者一个 flag）此前只在 DAG 派活轮开；现扩到 recommend（safetyCounter > 1，即第 2 轮起的续轮/综合）+ chain 逐席位（i > 0，第 2 个席位起）+ chain 收口轮（恒 true）。保留每种模式的首轮预广播（它为席位打底任务背景），只 suppress 重复的续轮广播（N 次读写巨大席位 thread 的纯浪费 + 弱模型模仿派活）；席位的前序交付另经「席位交付广播」broadcastSeatDeliveryToOthers 送达，不受影响。后台引擎路径无此广播（_seat_message 直接拼上下文），故 #4 仅前端。2026-06-20 深链接业务链抽取规范：深链接会商（taskId+goPath，businessCode 如 rwfx→6BF）下，结构化抽取(flow-json)与方案总结报告都按所选业务链完整层级强制校正。结构化抽取实为 roundtable-structure 智能体（flow_extract.py 的 _system_prompt()=load_agent_soul(STRUCTURE_AGENT_ID)，抽取→校验两段式 model.astream）；其 SOUL 全局一份，旧实现把某业务层级写死进 SOUL→所有业务都按「目的→行为体」抽（bug）。修法：单份领域知识、两角色用法的 deerflow.agents.roundtable_orchestrator.extraction_spec（放 harness 层，app+harness 都可 import；build_structure_directive/build_summary_directive 按 businessCode 取；rwfx=6BF 写实「任务→目的→行为体→预期行为→驱动因素→叙事，预期行为↔驱动因素 1:1、余 1:N、label 不含步骤/阶段名、优先报告回退席位」，其余业务通用规范=强约束完整性+命名+取数且明令不套固定模板；business_code 空→""零副作用）。注入：(1) 结构化抽取 flow_extract._hierarchy_hint+_validate_messages 用 build_structure_directive 权威覆盖默认层级（前端 streamFlowExtract 传 business_code，源 ingestBusinessCode）；(2) 方案总结报告 build_summary_directive——前台 multi_agent._special_run(role==summary 且 business_code) 追加、后台 engine.build_summary_prompt(business_code=…)（StartParams/StartJobRequest 透传）。仅深链接生效。种子 roundtable-structure/SOUL.md 层级顺序改 spec-driven、连接示例去 rwfx 写死（存量 live SOUL 不被 seed 覆盖，但注入规范权威可纠正）。Tests: tests/test_roundtable_extraction_spec.py。
Roundtable Draft Shares (/api/roundtable-draft-shares)	Per-draft sharing for the roundtable-planning page, mirroring Thread Shares but the shared resource is a roundtable_drafts row. Two modes (roundtable_draft_shares.mode). import: POST / {draft_id, mode} - create (or reuse) a revocable code; GET /{code}/preview + POST /{code}/import - copy that draft (step1/2/3) into the caller's account under a deterministic per-(code,recipient) id (imp_<sha1> ? re-import is a no-op). view: a public, read-only result page ? see Public Roundtable Shares. Both: GET / - list my codes; DELETE /{code} - revoke. Backed by deerflow.persistence.roundtable_draft_shares (table roundtable_draft_shares, migration 20260606_01); store wired in deps.py as app.state.roundtable_draft_share_store. Tests: tests/test_roundtable_draft_shares.py.
Public Roundtable Shares (/api/public/roundtable-shares)	Unauthenticated (/api/public/ whitelisted). GET /{code} - read-only snapshot for a view-mode draft share: returns the Step 3??????report HTML (step3.html, a self-contained document) + title/summary/has_report. The frontend renders it in a sandboxed iframe (no allow-scripts) at #/roundtable-share/<code>. Missing/revoked/non-view codes all return 404.
Scheduled Tasks (/api/scheduled-tasks)	Per-user cron-like agent tasks. GET/POST / - list/create owned tasks; PATCH/DELETE /{id} - update/delete (owner); POST /{id}/{pause,resume,trigger}; GET /{id}/runs, GET /runs/{rid}, run-file preview/edit/download; GET /runs/{rid}/messages - agent messages for the run's underlying agent run (used to inspect a stuck/running run; authorized via _load_run); GET /runs/{rid}/html - generated HTML document for an html_page run; DELETE /runs/{rid} - owner-scoped, with an admin fallback so admins can delete any user's run; publish/subscribe under /{id}/{publish,unpublish,subscribe} and GET /published. Admin (cross-user): GET /admin/all - every task; PATCH/DELETE /admin/{id} - edit/delete any task; POST /admin/{id}/trigger; GET /admin/{id}/runs. Admin routes require system_role == "admin".
HTML Page Favorites (/api/html-page-favorites)	Bookmarked generated HTML pages, owner-scoped. GET / - list; POST / {page_id, title?, description?, tags?} - bookmark a scheduled_html_pages row (extracts a reusable style/layout summary via GLM5 at favorite time); a page can only be favorited once per user (DuplicateFavoriteError ? HTTP 409). GET/DELETE /{id} - read/delete (the create-task reference picker deletes favorites here). A favorite can be selected as a style reference when creating a new html_page task.
Public Scheduled Tasks (/api/public/scheduled-tasks)	Unauthenticated mirror for embedding. Adds GET /runs/{rid}/html and GET /{task_id}/latest-html for the html_page preview; the run result keeps result.html (only file paths are redacted).
Light Apps (/api/light-apps)	Light-app registry (????? / ????), global/shared (no user scoping; created_by/updated_by audit-only). Two kinds: window_open (launched in a new tab from the ???? card grid) and iframe (embedded under a parent sidebar menu via mount_location and dynamically injected into the sidebar). Both carry route_params ? a list of {key, paramType: login|theme, source, customValue, themeValue} descriptors the frontend resolves at launch time (login params read from localStorage.userInfo; theme params use a literal value) and appends to url as query string. GET / - list (filters app_type/mount_location/enabled_only/search); GET /{app_id} - read one (both open to any authenticated user); POST /, PUT /{app_id}, DELETE /{app_id} - admin only; iframe apps must carry a mount_location (400 otherwise); the unique name collides as HTTP 409. Backed by deerflow.persistence.light_apps (table light_apps, auto-created by create_all); store wired in deps.py as app.state.light_app_store. Tests: tests/test_light_apps.py.
Menu Overrides (/api/menu-overrides)	Sidebar-menu overrides (菜单管理), global/shared (no user scoping; updated_by audit-only). The static menu tree (icons/routes/adminOnly) lives in the frontend code (core/page-layout/sidebar-menu.ts); iframe light-apps are injected at render time. This router persists a per-node overlay the frontend merges onto that tree: parent_id (re-parent, moves the whole subtree; NULL = top level), sort_order (reorder among siblings), label (rename; NULL keeps code label), disabled (停用 — hides the node and its subtree). A node with no row renders exactly as code defines it, so new code menus / newly-registered iframe apps appear automatically and are immediately configurable. GET / - list all overrides (open to any authenticated user); PUT / - admin only, atomically replaces the whole set ({overrides: [...]}); rejects re-parenting that would exceed 3 levels or form a cycle (_validate_depth, HTTP 400). Backed by deerflow.persistence.menu_overrides (table menu_overrides, auto-created by create_all); store wired in deps.py as app.state.menu_override_store. Frontend: merge logic core/page-layout/menu-overrides.ts (applyMenuOverrides in use-sidebar-menu.ts), client strategy-components/api/menu-overrides.ts, admin page pages/MenuManagementPage.tsx (@dnd-kit drag-reorder/re-parent + 「移动到」parent selector + ↑↓ arrows + 改名/停用), entry in the bottom-left settings dropdown (components/page-sidebar.tsx, admin-only). Tests: tests/test_menu_overrides.py, frontend-web/src/core/page-layout/menu-overrides.test.ts.
Sensitive Words (/api/sensitive-words)	Sensitive-word / desensitize map (敏感词管理 / 脱敏词), global/shared (no user scoping; updated_by audit-only). Each row maps an original word → replacement, with enabled (停用 parks a word without deleting) + sort_order. The frontend ships a seed map in code (src/lib/desensitize.ts SEED_DESENSITIZE_MAP) used as the default on a fresh deployment; once an admin saves, this table is the source of truth and the frontend loads it at startup (DesensitizeWordsLoader → replaceDesensitizeMap, enabled rows only). GET / - list all words (open to any authenticated user — every page desensitizes); PUT / - admin OR the lqq account (_require_manager: system_role == "admin" or email @-prefix == "lqq"), atomically replaces the whole set ({words: [...]}, de-duped by word, last wins). Backed by deerflow.persistence.sensitive_words (table sensitive_words, PK word String(191), auto-created by create_all); store wired in deps.py as app.state.sensitive_word_store. Frontend: client strategy-components/api/sensitive-words.ts, page pages/SensitiveWordsPage.tsx (route /page/strategy/admin/sensitive-words, entry in the bottom-left settings dropdown — gated on username lqq, not admin role, present in both sidebar modes; 「导入种子词」 merges missing seed entries). Tests: tests/test_sensitive_words.py.
Dashboard Sessions (/api/dashboard-sessions)	「大屏绘制智能体」独立页面会话，按外部 task_id 存取，不按 user 分权（open read + shared writes；user_id audit-only）。一个 task 一行（upsert）：transcript（聊天记录 JSON）+ report_json（最新结构化数据，可解析 = 「有效结果」）。GET /by-task/{task_id} - 取该 task 会话（无则 404）；PUT /by-task/{task_id} - upsert（{transcript?, report_json?}，只覆盖传入项）。Backed by deerflow.persistence.dashboard_sessions（table dashboard_sessions, auto-created by create_all，无迁移）；store wired in deps.py as app.state.dashboard_session_store。前端独立页 roundtable-planning/pages/DashboardAgentPage.tsx（路由 /page/strategy/dashboard-agent?taskId=，侧边栏「问答管理 → 大屏绘制」）让智能体分析登录拉取的业务 JSON → report-json → 渲染 DashboardApp；输入 seam core/auth/dashboard-input.ts（JSON 接口未就绪时回退 copTaskDetail 占位）。Tests: tests/test_dashboard_sessions.py。
Task Buttons (/api/task-buttons)	Task-workspace configurable jump buttons (按钮管理), global/shared (no user scoping; updated_by audit-only). Each row = one jump button in a task deep-link workspace's bottom 快捷跳转 bar: {id, business (rwfx/3qfx/xdfx), label, link_type (url/business), target, append_task_id, id_param (editable query-param name, e.g. id/task_id), id_kind (task/action — only xdfx differs, its deep-link id is an 行动id), enabled, sort_order}. The frontend holds code-seed defaults (rwfx's 4 historical buttons built from the live task_deeplink config); only when this table is empty do the workspace + admin page fall back to them, else this table is the sole source of truth. GET / - list all (open to any authenticated user — the workspace renders them); PUT / - admin only (_require_admin), atomically replaces the whole set ({buttons: [...]}, de-duped by id, last wins). Backed by deerflow.persistence.task_buttons (table task_buttons, auto-created by create_all); store wired in deps.py as app.state.task_button_store. Frontend: helpers core/auth/task-buttons.ts, client strategy-components/api/task-buttons.ts, admin page pages/TaskButtonsPage.tsx (route /page/strategy/admin/task-buttons, entry「按钮管理」in the bottom-left settings dropdown, admin-only, both sidebar modes; per-business tabs + 「导入默认」). Tests: tests/test_task_buttons.py.
Knowledge (/api/knowledge)	Product knowledge base, global/shared (no user isolation; writes record created_by/updated_by for audit). POST /ingest/thread/{thread_id} - sediment one thread into a note (reads messages from the checkpointer, threads:read owner-checked); POST /ingest/threads/batch - batch-sediment the caller's recent threads (skip_existing via .manifest.json dedup). Defaults to background=true: the slow per-thread extraction loop is deferred to a FastAPI background task and the endpoint returns {queued:true, processed:N} immediately (each thread's transcript is truncated to max_input_chars before the LLM call). Running the loop inline for a large batch outlives the nginx gateway timeout (504), which the frontend's parseResponse surfaces as a generic "Request failed"; background=false keeps the blocking path (full per-thread created/skipped/failed tally) for small batches/scripts. Shared loop: _run_batch_ingest. Optional model selection: both ingest endpoints accept model_name (overrides knowledge.extract_model_name for that capture; threaded through KnowledgeService.capture_thread ? extract_thread_llm_multi). Model-error surfacing: when the chosen model produces nothing usable (timeout / API error / non-JSON) capture still falls back to rule-based extraction and lands a draft (llm_fallback:true); the single-thread (non-background) response and the batch response (llm_fallback count) carry this so the UI tells the user "???????". The "??????" chat button stays background/fire-and-forget (passes model_name, no per-save error toast); the batch-import dialog is synchronous and shows the fallback count. POST /ingest/file - import an uploaded file (multipart: file + optional title/tags/status/model_name) into wiki note(s): the file is converted to Markdown (md/txt read directly, pdf/word/ppt/excel via convert_file_to_markdown), distilled by the same wiki-capture LLM pipeline as thread capture (source_type="file"), and falls back to a verbatim draft note when the model returns nothing usable (llm_fallback:true); POST /ingest/search - sediment search/tool results as a reference note; POST /notes - manual note; GET /notes (keyword/tag/source_type/status + pagination); GET/PUT/DELETE /notes/{id} (DELETE is soft = archive, ?hard=true removes the vault file); POST /search - mode=keyword|vector|hybrid (vector?keyword fallback when no embedding endpoint); GET /notes/{id}/versions - edit history; GET /duplicates + POST /merge - dedup; POST /reindex - rebuild vector index; GET /graph - product graph data (notes + entities, graph-blacklist filtered); GET/POST /graph/blacklist + DELETE /graph/blacklist/{id} - graph node blacklist (labels = entity names / note titles, case-insensitive match; blacklisted nodes and their edges are dropped from /graph and the wiki-export artifacts; table knowledge_graph_blacklist, auto-created); POST /export/graph + GET /export/files/{graph.json|graph.html|graph.graphml|cypher.txt} - obsidian-wiki wiki-export artifacts (self-contained graph.html for iframe); POST /resolve {targets:[?]} + GET /notes/{id}/wikilinks - resolve [[wikilink]] targets to a note id / vault page / missing (powers clickable wikilinks); GET /vault-page?path= - read a raw vault meta/hub Markdown page (no DB note). GET/POST /folders + PUT/DELETE /folders/{id} - user-managed directories (notes carry a folder path; GET /notes?folder= filters a dir + sub-dirs); GET/POST /extract-templates + GET/PUT/DELETE /extract-templates/{id} - editable extraction-prompt templates (thread/document ingest accept template_id; capture/PUT-note accept folder). Backed by the deerflow.knowledge module (see below).

Proxied through nginx: /api/langgraph/* ? LangGraph, all other /api/* ? Gateway.

AI Writing Built-in Functional Agents (????????)

The AI-writing graph (packages/harness/deerflow/agents/ai_writing/) has four stage "brains" ? ?????? / ????? / ?? / ?? ? that can each run in one of two modes, switched per-stage by config.yaml ? ai_writing.use_builtin_agents.{researcher,outliner,writer,editor} (default false, mtime hot-reloaded, new sessions pick up changes immediately):

• Legacy (flag off): hard-coded prompts in prompts/*.py, direct LLM call (unchanged behavior).
• Builtin-agent (flag on): the node loads the corresponding functional agent from .deer-flow/agents/ai-writing-<stage>/ ? SOUL.md becomes the system prompt and config.yaml's model overrides the stage model (priority: agent.model > node explicit fallback (keyword_model/rank_model) > state.model_name). Admins edit SOUL/model from the agent management page. Runner: nodes/_agent_runner.py (run_writing_agent / stream_writing_agent / parse_marker_json). Output contracts (marker + json blocks, intent.py-style): researcher [KEYWORDS_READY] / [MATERIALS_RANKED], outliner [OUTLINE_READY], editor (per-dimension) [REVIEW_READY]; writer emits raw section Markdown (keeps per-section streaming) with the legacy [[SECTION_BLOCKED]] sentinel in strict mode. Orchestration stays in the nodes (search fan-out, per-section loop, 3-dimension review loop, all pause points) ? the agent only does single structured generations, and a missing/empty SOUL silently falls back to the legacy path (WritingAgentUnavailable).

Seeding mirrors the roundtable pattern: verified config.yaml + SOUL.md ship in app/gateway/routers/_ai_writing_seed_assets/<id>/; _ai_writing_seed.py::ensure_ai_writing_functional_agents() copies them to .deer-flow/agents/ when missing (never overwrites local edits), called from the startup lifespan (before _sync_legacy_agents, so the agents appear in /api/agents on first boot) and self-heals from GET /api/ai-writing/sessions + GET /api/ai-writing/article-types. Tests: tests/test_ai_writing_seed.py, tests/test_ai_writing_agent_runner.py.

Skill-driven retrieval (检索类型). material_source has exactly two values: general (通用检索, default) and notebook (我的空间); legacy knowledge_base/skill are normalized to general by the researcher node. General retrieval is orchestrated from the skills configured on the ai-writing-researcher agent (config.yaml → skills, editable from the agent management page): the built-in knowledge-base-search skill (知识库检索) runs the native keyword fan-out web/intranet search (the old default path), while any other configured skill searches with a description-targeted query; skills run in configured order with early-stop at max_materials, and an empty/unresolvable skills config falls back to the built-in knowledge search so retrieval always works. The skill itself is seeded from _ai_writing_seed_assets/skills/knowledge-base-search/SKILL.md into skills/public/ (never overwrites), and a pre-existing researcher config.yaml lacking the skills key is patched in place with the default (explicit skills: [] is respected). PAUSE-1 input append flow: when the user types natural language into the material-confirm input (re_search/legacy enable_skill_search + user_query), the node matches the most relevant configured skill via SkillPicker(candidates=…) (skipped when only one skill is configured), searches with the user's words, and appends the new materials to the existing package (URL-dedup, original order preserved, no LLM re-rank/truncation, 100-item hard cap). Skill-failure fault tolerance: a configured skill that times out / raises / has all its keyword searches fail (vs. legitimately returning 0 results) no longer just silently moves on — the researcher node emits a skill_search_step of type skill_error (carrying the error reason), and if the configured skills collected no new materials at all it runs a fallback pass over other available skills (_fallback_candidate_skills: the built-in knowledge-base-search native path first, then other enabled catalog skills not yet tried) instead of erroring out. The frontend SkillStepsCard renders skill_error (rose) and fallback (amber) so the timeline shows "技能报错 → 改用其他技能搜集". Tests: tests/test_ai_writing_material_source.py.

Skill-call timeout is AI_WRITING_SKILL_SEARCH_TIMEOUT_SECONDS (env, default 120s; researcher.py _SKILL_SEARCH_TIMEOUT_SECONDS); the DuckDuckGo provider's internal cap was raised 30s→120s so it doesn't undercut it.

Agentic skill execution (ai_writing.researcher_skill_agent, default false). By default a configured non-native skill is not run agent-style — the node just turns its description into a query for the generic search provider. With this flag on, the directed-skill branch instead runs nodes/_skill_agent.py::run_skill_search_agent: it injects the skill's SKILL.md (read from skill.skill_file) alongside the researcher SOUL as the system prompt, binds a search tool to the researcher model, and runs a bounded (_MAX_TURNS=4) tool-call loop so the model drives the retrieval per the skill's instructions; the SearchResults the tool returns are harvested directly as materials (no fuzzy text extraction). Requires a tool-calling model; if the model can't tool-call (returns no tool_calls) or the agent yields nothing, _execute_skill falls back to the legacy description→query search so materials are never empty. The built-in knowledge-base-search native concurrent path is unchanged. Tests: tests/test_ai_writing_skill_agent.py.

Background suspend (后台挂起). A writing session can be handed off to the server to auto-drive the remaining flow to finalize, surviving page leave / refresh / account switch. POST /api/ai-writing/sessions/{id}/background (owner-checked, idempotent) starts AIWritingJobExecutor (app/gateway/ai_writing_job_executor.py), wired in deps.py as app.state.ai_writing_job_executor. The executor drives the same ai_writing graph in-process — it builds make_ai_writing_graph(), attaches the shared app.state.checkpointer (the very saver the runtime worker uses, so it continues the real thread state the frontend already wrote), sets the owning user via set_current_user, and loops aget_state → ainvoke(Command(resume=…)) until status==done. Auto-action per interrupt: material/outline confirm → confirm; section_help → every blocked section loose (continue with general knowledge, never re-loop to research); draft_confirm → to_editor until reviewed, then finalize (on pass, or when revision_count >= max_revisions); review_confirm → accept_review (revise & re-review until pass or budget exhausted, then forced finalize). To avoid racing a still-running runtime worker (suspend mid-node), the driver waits for an actionable state (_wait_for_actionable): it only acts when the graph is at an interrupt / done, or when the checkpoint stops advancing (no worker → abandoned/restart). No new table — the graph checkpoint + ai_writing_sessions already hold all state; status="background" marks 挂起中 (surfaced in the history dropdown), nodes write draft/review_result/status=done as usual, so the user reopens the finished article from history. Per-user concurrency cap AI_WRITING_BACKGROUND_SLOTS_PER_USER (default 2, same SQLite-lock rationale as the roundtable executor). Startup reconcile (in deps.py) relaunches drivers for sessions left background after a restart (AIWritingSessionRepository.list_by_status). Progress flowchart popup: GET /api/ai-writing/sessions/{id}/progress (owner-checked) reads the graph checkpoint via executor.get_progress and derives a canonical pipeline stage key (intent/research/material_confirm/outline/outline_confirm/draft/draft_confirm/review/done, from the pending interrupt's pause_point or snapshot.next node) + note (素材不足求助 / 编辑打回) + review progress + isRunning. Frontend: 「后台挂起」 button + auto-suspend on route unmount / beforeunload (AIWritingPanel.tsx, suspendAIWritingToBackground[Keepalive] in api/ai-writing-sessions.ts); 挂起中 badge in AIWritingHistoryDropdown.tsx, and clicking a 挂起中 history record opens a flowchart dialog (AIWritingFlowDialog.tsx polls getAIWritingProgress every 2.5s, renders the fixed vertical pipeline AIWritingFlowChart.tsx lit by the live stage) so the user sees how far the background run has gotten without reopening/taking over the session. Transcript consistency: the rich history timeline (AIWritingHistoryView → AIWritingTimeline) replays transcript = {progressEvents, completedInterventions}, normally assembled+saved client-side. A background run has no client, so the executor rebuilds it at the end (_save_transcript/_build_transcript_from_checkpoint) entirely from the final checkpoint — both progressEvents and completedInterventions are derived from the same set of resumed events (every pause-resume emits exactly one), so they are always 1:1 aligned. For each resumed: classify its message → (pausePoint, action) (_classify_resumed, matched against graph.py's resume messages), insert an await_user (with the pause prompt) before it, and build a rich intervention card enriched from final state (material_confirm→materials, outline_confirm→outline, draft_confirm→draftTitle/draftWordCount/reviewResult, review_confirm→reviewResult); materials_ready is inserted before the first material-confirm. (Earlier code merged the frontend's pre-suspend completedInterventions with auto ones and aligned by index — but the frontend's last save can predate its most recent decision, so the counts drifted and cards rendered off-by-one with empty data; deriving everything from the checkpoint root-fixes both.) Raw (snake_case) events and intervention data are normalized on load by the exported normalizeProgressDict / normalize{Material,Outline,ReviewResult} (idempotent on already-camelCase live transcripts — one normalize path for both sources), so reopening a background-completed session shows the same cards as a frontend-driven one. Tests: tests/test_ai_writing_background.py.

Admin Leaderboard Snapshots (/api/admin/users/leaderboard*)

The admin ????? is served from precomputed per-day snapshots in admin_leaderboard_daily_stats (keyed by (stat_date, settings_hash)), not by aggregating raw rows on every request. A background scheduler inside the Gateway (_leaderboard_snapshot_scheduler_loop in app/gateway/app.py, singleton file-locked) ticks every ADMIN_LEADERBOARD_BACKGROUND_INTERVAL_SECONDS (default 300s): it prewarms recent days, then builds any queued/error/stale-running day. GET /leaderboard merges the per-day snapshots for the requested range. Today is rebuilt whenever its snapshot is older than ADMIN_LEADERBOARD_TODAY_TTL_SECONDS (default 43200s = half a day, so the leaderboard refreshes ~twice daily); a past day, once ready, is treated as fresh forever.

Two on-demand/automatic refresh paths layer on top:

• Manual "????" ? POST /leaderboard/refresh-today force-rebuilds today's snapshot synchronously (bounded by the same semaphore/timeout the scheduler uses), then returns the merged range. Lets an admin pull the latest real-time records into the stats without waiting for the TTL. Frontend: a "????" button on AdminUserLeaderboardPage ? useRefreshTodayLeaderboard (core/admin/hooks.ts), which primes + invalidates the ["admin-activity","leaderboard"] query.
• Daily finalize of the previous day ? once per day, on/after ADMIN_LEADERBOARD_FINALIZE_HOUR (local Beijing, default 1 = ?? 1 ?), the scheduler tick force-recomputes yesterday a single time so runs/tool-calls that only landed near midnight are fully captured. The decision is the pure deerflow.persistence.admin_stats.finalize.should_finalize_previous_day (status/generated_at/now/hour) ? the rebuild stamps generated_at to today, which self-debounces it for the rest of the day (no extra state file). Tests: tests/test_leaderboard_finalize.py.

Scheduled Task Runtime

ScheduledTaskService (deerflow/runtime/scheduler/service.py) polls due tasks every poll_interval_seconds and executes each in its own throwaway thread, serialized by a shared _agent_run_lock.

Stuck-run recovery: each execution attempt is bounded by _RUN_ATTEMPT_TIMEOUT_SECONDS (30 min) via asyncio.wait_for. On timeout the run is retried up to _MAX_RUN_RETRIES (3) times, then marked failed. A watchdog (_reconcile_stuck_runs, run every poll) recovers runs left in running with no in-process owner ? e.g. orphaned by a crash or restart ? re-driving them through the same retry budget so a task can never stay perpetually "executing". The scheduled_task_runs.retry_count column tracks consumed retries.

Unique task names: task names are unique per user (uq_scheduled_tasks_user_name). create_task/update_task/admin_update_task proactively check via _assert_name_available() and raise DuplicateTaskNameError on a collision; the router translates that into HTTP 409 instead of letting the raw DB IntegrityError surface as a 500.

HTML page tasks (task_kind == "html_page"): a second task kind that renders a full HTML document each run instead of a Markdown report. The kind + its config (html_config: page_prompt, negative_prompt, reference_favorite_id, model_name) live inside execution_context ? no schema change to scheduled_tasks. Data gathering is delegated to the agent's own search skills (driven by page_prompt), so there is no explicit search-query/data-source field; the render model is user-selectable (model_name, default zai-org/GLM-5-FP8). _run_task_attempt() dispatches by kind; _run_html_page_attempt() runs the pipeline: gather real data via the lead agent (its search tools) ? optionally load a reference favorite's reusable style/layout ? render with the chosen model (thinking off) ? fault-tolerant JSON parse ? sanitize (BeautifulSoup, strips <script>/<iframe>/on*/javascript:) ? structural validate ? one GLM5 repair pass if invalid ? completeness gate: if the HTML is still not a full valid document (missing doctype/html/head/body, empty body, or residual scripts) after the repair pass, raise HtmlGenerationError so the run is marked failed (a page only counts as successful when it is a complete, previewable HTML document) ? best-effort Playwright screenshot (no-op when Playwright absent) ? archive index.html ? persist a scheduled_html_pages row ? return result with format: "html" and the full HTML inline. Pure helpers live in deerflow/runtime/scheduler/html_page.py. The frontend previews the HTML in a sandboxed iframe (no allow-scripts) on the run-detail and public pages; a user can bookmark a page (style summary extracted via GLM5) and later pick it as a style reference. New tables scheduled_html_pages + html_page_favorites (deerflow/persistence/html_pages/, wired as app.state.html_page_store) are created by Base.metadata.create_all with no migration. Tests: tests/test_html_page_tasks.py.

Unified output switches (require_html + require_markdown). The task form no longer splits "???? / HTML ??" into separate kinds. execution_context now carries two independent booleans that may both be on: require_html (???? HTML) and require_markdown (???? Markdown). _task_requires_html() returns true for require_html or legacy task_kind == "html_page" (so old tasks need no data migration), and _run_task_attempt() dispatches to _run_html_page_attempt when it is true, else the regular agent attempt. When both switches are on, _gather_html_data() returns (text, files) ? the gather agent run is additionally told to save a .md file ? and _run_html_page_attempt validates it (MarkdownRequirementError on failure) and surfaces it alongside the generated page, so the run delivers both artifacts. The single main prompt doubles as the HTML page requirement (html_config.page_prompt); there is no separate page-prompt field. Tests: tests/test_scheduled_task_require_html.py + the markdown-coexistence cases in tests/test_html_page_tasks.py.

Template-fill mode (JSON → 固定 HTML 模板填充). A third execution mode, distinct from the free-form require_html render pipeline: convert a user-supplied JSON into a fixed HTML template's data structure and fill it. Triggered when execution_context.template_config.template_id is set (chosen in the task form after selecting the built-in 「模板数据转换器」 agent template-json-builder). _task_template_id() reads it and _run_task_attempt() dispatches to _run_template_fill_attempt before the require_html check. Pipeline: take the source (the task prompt, pasted or uploaded .json) — which may be pure JSON, plain text, or text-with-embedded-JSON (template_fill.extract_embedded_json pre-extracts any embedded {…}/[…] with a string-aware balanced scan and surfaces it as a labeled block while keeping the full original text; plain text → no block, the model reads it semantically) → load the selected agent's SOUL.md+model and run a direct _invoke_conversion_model call (system=SOUL, human=conversion prompt carrying the template's compact schema skeleton + the recognized-JSON block + source) → extract_json_object → template_fill.validate_data (top-level keys + container types) → on failure feed the structure errors + field_coverage_gaps back and retry up to _MAX_TEMPLATE_FILL_RETRIES=2 → template_fill.fill_template replaces the template's const DATA={…} line (and refreshes MANIFEST.系统.更新时间) → archive index.html + persist a scheduled_html_pages row → return format:"html" (reusing the existing HTML preview at /runs/{id}/html). Any terminal failure raises TemplateFillError → run marked failed (no retry, message shown). Pure engine deerflow/runtime/scheduler/template_fill.py (harness, no app.*): the template registry (TEMPLATES: id=no1 name=no.1 台湾滨海防卫大屏, and id=no2 name=no.2 台湾飞行部队平台 with convertible=False — a 「只展示」 template that renders its existing HTML as-is, no model/validation/fill, until conversion is wired in later; _run_template_fill_attempt short-circuits to read_template_text for it, and list_templates/is_convertible expose the flag so the form hides the JSON input), list_templates/reference_data/schema_skeleton_json/validate_data/field_coverage_gaps/fill_template — the template's own embedded DATA is the schema source of truth (parsed at load, 1-sample-per-array skeleton fed to the LLM, top-level structure as the validation basis); template HTML files live under runtime/scheduler/templates/<id>/, add a TemplateSpec to register a no.2. The agent is seeded like the AI-writing ones: app/gateway/routers/_template_builder_seed.py + _template_builder_seed_assets/template-json-builder/{config.yaml,SOUL.md}, called from the lifespan before _sync_legacy_agents (which upserts it into the agents table as built-in/published so it shows in /api/agents → the task form's 执行智能体 dropdown). Template list API: GET /api/scheduled-tasks/html-templates → {templates:[{id,name,description,convertible}]} (open). Debug-render API POST /api/scheduled-tasks/template-fill/render {template_id, text?|data?} → {ok, display_only?, errors, gaps, html} (open): no model call — extract_embedded_json(text) (or data) → validate_data → fill_template, returning the filled HTML plus the validation errors/field-gaps (fills even when invalid so you can see the effect). Powers the chat-page debug button below; non-convertible templates return their raw HTML. Templates as require_html 参考收藏页面 (use the REAL template, all sub-pages): the 参考收藏页面 list in the task form also offers the templates (value tpl:<id>, e.g. tpl:no1). These templates are JS-driven multi-page SPAs (sub-pages/tabs are rendered by script from DATA), so a free-form model render would only ever produce the main page and drop every sub-page. Therefore, when _run_html_page_attempt sees a tpl:<id> reference (template_fill.reference_template_id parses it; distinguishes a template marker from a real favorite id), it short-circuits before the free-form pipeline to _run_html_template_reference, which renders the actual template file (all sub-pages intact): convertible templates gather real data (_gather_html_data) → convert to the template DATA (_convert_to_template_data, shared with template-fill mode) → merge_with_reference (per top-level key = per sub-page: any sub-page/field with no extracted content falls back to the template's original data, so sub-pages never render blank) → fill_template; non-convertible templates (no.2) output the raw template as-is; if the task prompt and description are both empty, or gather/conversion yields nothing, it uses the template's original data outright. 数据抽取开关 html_config.template_extract (default off, shown in the form only when a tpl: reference is picked): it only controls whether the HTML uses extracted data vs the template's default data — it does not affect anything else. Off → HTML fills the template's default data (no convert); on → gather+convert+merge. Markdown is independent: when require_markdown is also on, the gather agent run still runs and the .md is produced/validated/surfaced regardless of the extract switch (so turning extraction off never blocks the Markdown). Display-only templates (no.2) never extract; they still honor require_markdown. _run_html_template_reference runs gather once when do_extract (extract on + has prompt/desc + convertible) or require_markdown, and threads any .md through _finalize_template_html(extra_files=…). merge_with_reference is also applied in template-fill mode (_run_template_fill_attempt) so empty sub-pages there fall back too. Archive/persist/result are shared via _finalize_template_html. (The earlier style_reference CSS-extraction approach was removed — it lost the sub-pages.) Frontend: when the agent dropdown = template-json-builder the form hides the HTML/Markdown switches and shows a 「HTML模板」 select (useHtmlTemplates) + a .json upload (reads file text into the prompt box). Tests: tests/test_template_fill.py, tests/test_template_fill_attempt.py, tests/test_template_builder_seed.py.

Per-user Prompts & UI Preferences

Custom prompts can be scoped to one agent. user_custom_prompts gains a nullable agent_id column (migration 20260603_02): NULL = global (the chip shows in every chat), otherwise the prompt only surfaces when that agent is selected. The visibility filter is applied in the frontend CustomPromptBar (it receives the current agent id from ChatPage's selector or AgentChatPage's assistantId); the backend (/api/user-prompts CRUD) just stores/returns/clears agent_id (update accepts an explicit null to clear). Tests: tests/test_user_prompts_agent_id.py.

Per-user UI preferences (deerflow/persistence/user_preferences/, wired as app.state.user_preferences_store; migration 20260603_03, also auto-created by create_all): one JSON-blob row per user so new toggles need no schema change. Router /api/user-preferences (GET/PUT, account-scoped). Current key: hide_admin_prompt_prefix ? when true the chat input hides the admin-configured prompt-prefix switch and never applies the prefix (the user uses only their own prompts; enforced in ChatPage for both shouldApplyPrefix and promptPrefixProp). Toggled from the "???????" dialog. Tests: tests/test_user_preferences.py.

HTML skills (tools the render step uses). Two offline skills under skills/public/ shape the generated HTML and are toggled in the skills settings page (enabled by default): html-page-builder (a self-contained inline-CSS design system ? colors/cards/tables/KPI/responsive) and html-charts (inline-SVG charts that render in the no-allow-scripts preview iframe; optional local ECharts documented in its assets/README.md). A skill opts into HTML injection via metadata.kind: html_page in its SKILL.md frontmatter; _collect_html_skill_instructions() reads the bodies of enabled such skills and build_generation_prompt(skill_instructions=?) injects them, so the (raw-model) render step honors the user's skill selection. The data-gathering step and the render/repair/style passes all use the task's selected model_name (no silent fallback to the default model ? and there is no hardcoded GLM-5; an unset selection falls back to config models[0]). To stop auxiliary LLM calls from leaking onto the default model, TitleMiddleware is skipped when is_scheduled_run is set (_build_middlewares in lead_agent/agent.py) ? scheduled/HTML runs use throwaway threads whose titles are never shown, and config.title.model_name is usually unset (? would call models[0], not the chosen model). The completeness gate (validate_html) also requires a non-empty <title> and forbids external stylesheets, on top of the doctype/html/head/body + non-empty-body + no-script checks.

Knowledge Base (packages/harness/deerflow/knowledge/)

Self-contained product knowledge base. All knowledge capability lives in this one module (harness layer, deerflow.* only ? no app.* imports). Wired as app.state.knowledge_service in deps.py via make_knowledge_service(sf, config.knowledge) (returns None on the memory backend or when knowledge.enabled=false ? router 503). Surfaced by app/gateway/routers/knowledge.py (/api/knowledge). Config: config.yaml ? knowledge (deerflow/config/knowledge_config.py).

Two stores, kept in sync on every write:

• DB mirror ? tables knowledge_notes / knowledge_sources / knowledge_tags / knowledge_note_tags (knowledge/models.py, registered in persistence/models/__init__.py, auto-created by Base.metadata.create_all; portable types PortableLongText/BeijingDateTime). Powers listing/filter/search/audit. Global/shared ? no user_id column; created_by/updated_by are audit-only (resolved via get_effective_user_id()).
• Markdown vault ? obsidian-wiki compatible, default {$DEER_FLOW_HOME}/data/knowledge/obsidian-vault. ObsidianWikiAdapter (obsidian_adapter.py) writes wiki-capture frontmatter+body, and after every write syncs index.md / log.md (CAPTURE/INGEST/UPDATE/ARCHIVE/EXPORT) / hot.md / .manifest.json (source-hash dedup for skip_existing). markdown_vault.py handles frontmatter build/parse, slugging, wikilink extraction and path-traversal-proof resolution.

Module layout: models.py (ORM: notes/sources/tags/note_tags + embeddings/versions/entities/relations) ? repository.py (DB only, no LLM) ? schemas.py (NoteDraft/SourceDraft DTOs) ? skill_templates.py ? markdown_vault.py ? manifest.py ? extractor.py (rule-based fallback) ? llm_extractor.py (default: distills declarative knowledge per wiki-capture, structured JSON incl. entities/relations; falls back to extractor.py) ? sources.py ? exporter.py (graph.json/graph.graphml/cypher.txt + self-contained vanilla-JS graph.html; entity nodes + relation edges. graph.html is theme-aware ? it reads ?theme=light|dark from its own URL since the embedding iframe can't inherit the app's CSS; KnowledgeGraphPage.tsx appends the active day/night mode and re-skins on toggle. The layout cools down and freezes so dense graphs stay clickable, node radius scales with degree, and hovering/clicking a node focuses it + its direct neighbours while dimming the rest + a detail panel ? taming heavy node/edge clutter) ? chunking.py + embeddings.py (OpenAI-compatible embedding client, keyword fallback) + search.py (keyword/vector/hybrid) ? sensitive.py (secret/PII redaction) ? dedup.py (Jaccard + hash dedup) ? auto_ingest.py (background queue/worker) ? obsidian_adapter.py ? service.py (KnowledgeService orchestrator ? the only thing the router touches; capture entry points capture_thread / capture_search / capture_file / create_manual_note, all converging on _persist_draft). Tests: tests/test_knowledge.py (20 cases). Frontend: frontend-web/src/core/knowledge/* + pages/KnowledgeBasePage.tsx / KnowledgeGraphPage.tsx + chat knowledge-save-trigger.tsx / knowledge-rag-toggle.tsx; spec at frontend-web/docs/product-knowledge-base-dev.md.

Wikilinks & multi-note capture (knowledge.{related_notes_enabled,related_notes_limit,multi_note_enabled}, all default-on; project_name defaults to zncm):

• Navigable [[wikilinks]] ? service.resolve_wikilink(s) maps a [[target]] to a knowledge note (by id ? vault_path ? title) or a vault meta page (read_vault_page, path-traversal-proof), else missing. markdown_vault.normalize_wikilink_target canonicalizes targets (drops |alias, slashes, .md). The frontend MarkdownRenderer renders non-numeric [[...]] as clickable links from the resolver map (numeric [[123]] stay citation badges); note links navigate, vault links open a VaultPageDialog, missing links are greyed.
• Real related notes only (no structural meta-links) ? extractor.related_links() now returns []: notes no longer auto-link index / _meta/taxonomy / a project hub (they were noise in every ## Related). On capture, service._find_related_notes cross-links only by shared distinctive tags (project/knowledge-base defaults excluded; free-text title/summary matching was removed because CJK tokenization linked unrelated notes via generic words like ??/????) and _inject_related_notes writes real [[Title]] links into ## Related (plus note?note related graph edges); no distinctive tags ? no related notes. The renderers omit the ## Related section entirely when there are no related notes and no sources. The vault's own index.md/_meta/taxonomy.md still exist (obsidian scaffold) and remain resolvable for manually-written links. The router enriches GET /notes/{id} with created_by_name/updated_by_name (auth-provider email lookup) so the UI shows a person, not a UUID. The frontend MarkdownRenderer renders wiki-capture provenance markers ^[inferred]/^[ambiguous] as small "??"/"??" badges.
• LLM-extraction outcomes ? capture distinguishes three LLM outcomes: (1) success ? declarative note(s); (2) deliberate rejection ? the model returns should_ingest:false with no notes (greeting / meta chatter) ? extract_thread_llm_multi raises LLMRejectedIngest and capture_thread raises EmptyThreadError (skip ? nothing persisted, so worthless chats never enter the KB); (3) genuine failure ? API error / non-JSON ? returns [], capture falls back to the rule-based verbatim extractor and lands the note as a draft (not approved), logs a WARNING, and returns llm_fallback:true (surfaced by the router). llm_extractor logs an actionable WARNING/INFO for each non-success path so extraction issues are diagnosable.
• Multi-note split ? llm_extractor.extract_thread_llm_multi lets one conversation distill into 1?4 cross-linked notes (notes:[?] JSON array; legacy single-object output still accepted). capture_thread persists each, sibling-cross-links them by title, and returns {note, created, notes:[?]}. Gated by multi_note_enabled (false ? first note only). Tests: tests/test_knowledge_wikilinks.py.

Phases 2?4 (all config-gated, defaults safe):

• Phase 2 ? vector RAG: knowledge.embedding.{enabled,base_url,api_key,model,...} enables an OpenAI-compatible embedding endpoint; notes are chunked + embedded on write (knowledge_embeddings), POST /search supports mode=keyword|vector|hybrid (vector degrades to keyword when unconfigured). KnowledgeRagMiddleware (lead-agent chain) injects a transient <knowledge-context> recall before the model call, gated per-conversation by the knowledge_rag_enabled runtime flag (frontend chat toggle) falling back to global knowledge.rag_enabled. POST /reindex rebuilds the index.
• Phase 3 ? governance: sensitive.py redacts secrets/PII before persistence (sensitive_redaction_enabled); every update snapshots knowledge_note_versions (GET /notes/{id}/versions); GET /duplicates + POST /merge for dedup; background AutoIngestQueue (wired in deps.py, started/stopped in the gateway lifespan) auto-sediments finished conversations off the chat path when auto_ingest_enabled ? enqueued from KnowledgeRagMiddleware.after_agent, value-judged by min-chars + skip_existing, landing as draft.
• Phase 4 ? graph: LLM extraction also yields entities/relations ? knowledge_entities/knowledge_relations; exporter.build_graph adds entity nodes + note?entity/entity?entity edges (reflected in graph.html/graph.json); GET /graph returns product graph data.

User directories + editable extraction templates + display polish (knowledge UX overhaul):

• Custom directories (knowledge_notes.folder + knowledge_folders) ? notes carry a free-form, slash-joined folder path (e.g. 投研/行业) independent of the wiki-capture category and of vault_path. knowledge_folders is the manageable directory list (supports empty folders + rename/move that re-points descendant folders and filed notes). Router CRUD under /api/knowledge/folders (GET/POST/PUT/{id}/DELETE/{id}?reassign_to=); GET /notes?folder= filters a directory + its sub-folders (folder="" ? unfiled). Capture (capture_thread/capture_file/create_manual_note) and PUT /notes/{id} accept folder (the note PUT uses model_fields_set so an absent key leaves the folder untouched, a present null clears it). Frontend buildKnowledgeTree(notes, folderPaths) builds the left tree from folder (no longer from vault_path); folder management dialog + FolderSelect in the create/import dialogs and the note detail header.
• Editable extraction prompts (knowledge_extract_templates) ? the hard-coded distillation prompts are seeded as editable templates on startup (KnowledgeService.seed_default_templates, idempotent — 默认 · 对话提炼 / 默认 · 文档提炼, each is_default for its scope). scope ∈ {thread, document, both}; capture resolves an explicit template_id → the scope default → the built-in constant, and passes it to llm_extractor.extract_thread_llm_multi / extract_document_llm_multi as system_prompt (used verbatim). Router CRUD under /api/knowledge/extract-templates (409 on duplicate name). Frontend: extraction-template management dialog + TemplateSelect in the batch/file import dialogs. Editing a template is the supported way to fix "提炼太精简" (loosen the prompt to keep more detail) and to add alternate extraction logics.
• Single source list ? extractor._related_section_md no longer emits a ### Sources sub-section into the body; the note detail shows sources once via the structured 来源 block (note.sources). ## Related stays pure wikilinks so the frontend extracts it cleanly into the related-docs module.
• Chinese section headings (display-only) ? the frontend localizeHeadings (wikiUtils) maps Context/Finding / Decision/Reasoning/Implications/Related/Sources to 中文 at render time (applied to the body fed to both extractToc and the viewer, so anchors stay consistent). No data migration — historical English-heading notes render in Chinese too.
• Historical-data cleanup ? scripts/cleanup_knowledge_related.py (one-off, idempotent, --dry-run/--note-id) strips legacy ## Related structural links (vault scaffold: index/_meta/taxonomy/projects/…, any slash-bearing target) and the old ### Sources sub-section from existing notes (DB content_md + vault files, via KnowledgeService.update_note).
• Migration 20260616_02 + idempotent scripts/sql/column_additions_mysql.sql cover knowledge_notes.folder (+ index); the two new tables are auto-created by create_all / _ensure_orm_columns_sync on startup. Tests: tests/test_knowledge.py.

Custom Agent Persistence

Custom agent metadata lives in the agents ORM table (deerflow.persistence.agents). agents.id is the stable runtime id and the filesystem directory name under .deer-flow/agents/{id}/; agents.name is display-only, can contain Chinese characters, and is not unique. user_id IS NULL marks built-in agents, user-owned agents are visible only to their owner unless published = true, and update/delete operations require ownership. The runtime accepts agent_id in context/configurable, and non-default assistant_id values are treated as agent ids after visibility checks.

Listing cache + pagination ? to keep GET /api/agents fast, the gallery's per-card extras (skills + tool_groups) are denormalized into two cache tables instead of reading every agent's config.yaml on each request: agent_skills (relational, one row per (agent, skill) with position) and agent_extras (one row per agent: tool_groups JSON + model + skills_synced_at). config.yaml remains authoritative for the agent runtime; these tables are a read cache, write-through on create/update and backfilled from disk by _sync_legacy_agents (mtime-vs-skills_synced_at drift check). The list endpoint enriches via AgentStore.get_extras_for([ids]) (two batched IN queries). Both tables are auto-created by Base.metadata.create_all (no migration needed). GET /api/agents supports opt-in server-side pagination: scope (mine = owned + favorited / square = built-in/published public), page, page_size, returning {agents, total, page, page_size}; omitting scope/page preserves the legacy "return everything" behavior. Name search matches the raw OR desensitized name (the frontend's static term?abbrev map is mirrored server-side in deerflow/config/agent_name_desensitize.py); tag_ids is an SQL OR-filter; owner keeps only agents published by that user id (the card's ??? ? built-in agents never match), powering the gallery's click-???-to-filter. POST /api/agents/private-square/list takes the same pagination params (incl. owner). Repository entry point: AgentStore.list_paginated(...). Tests: tests/test_agent_listing_cache.py::test_list_paginated_owner_filter_keeps_one_publisher.

Homepage selector (agent picker on the new-chat page) ? single admin-curated list backed by agents.featured_order (nullable BIGINT). NULL = not featured. Only published = true agents can be featured (router enforces on writes); any published agent is eligible regardless of ownership. Cleared with order: null. The list rendered to end users is WHERE featured_order IS NOT NULL + visibility-filter, sorted by featured_order ASC. Admins reorder by sending the desired ids to PUT /api/agents/featured/order (assigns indices 0..N-1). BIGINT is used rather than INT so the management UI can seed initial values from Date.now() (overflows INT on MySQL).

Publish Squares (????)

Admin-configurable, access-control-free categories that published agents/skills are filed under (distinct from the password-gated ????). The square list lives in system_settings.json ? publish_squares (PublishSquaresSettings: ordered squares: [{id,name,description}] + default_square_id); helpers list_publish_squares / resolve_default_square_id / normalize_square_id always yield a valid square (built-in "????" fallback when none configured). Admin CRUD: GET/PUT /api/system-settings/publish-squares (GET readable by any user ? the gallery sub-tabs + publish picker need it; PUT admin-only, auto-slugs missing ids and re-anchors a dangling default).

Each agent/skill carries a single square_id column (AgentRow / SkillRow, String(64) NOT NULL default '', migration 20260611_01, idempotent SQL in column_additions_mysql.sql). One column = at most one square, so an item can never be "published twice". '' = unassigned ? resolved to the default square at read/filter time. Writes normalize via normalize_square_id: agents on create/update (square_id field), skills on publish (PUT /api/skills/custom/{name}/published accepts optional square_id). Listing filters: GET /api/agents?square= and GET /api/skills?square= keep only that square's items; the default square additionally sweeps in legacy '' rows (square_is_default). Tests: tests/test_publish_squares.py. Frontend: core/system-settings (usePublishSquares/useUpdatePublishSquares), admin PublishSquaresSettingsPage (embedded in the appearance/settings hub), square picker in the agent edit dialog + per-skill card, and a ??-tab square sub-filter in both galleries.

Agent Module Tag Config (zzJC / zzXD 按类型聚合)

Backs the zzJC / zzXD agent-chat left sidebar (「按类型聚合智能体 + 聊天记录」). An admin picks which agent tags belong to a module; the module's left sidebar then shows only agents under those tags (empty/unconfigured = show all visible agents). Stored in system_settings.json → agent_modules (AgentModuleSettings: modules: {module_key → [tag_id, ...]}); pure helpers get_module_tag_ids / set_module_tag_ids (de-dupe + strip). Endpoints (in app/gateway/routers/system_settings.py): GET /api/system-settings/agent-modules/{key} (readable by any user — the sidebar needs it to filter), PUT /api/system-settings/agent-modules/{key} (admin only, full replacement of the tag-id list). Module keys are frontend-defined (AGENT_MODULES in components/workspace/agents/agent-module-sidebar.tsx, keyed by the module's landing agent id: 6bf→zzjc, 10d69…→zzxd). Tests: tests/test_agent_modules.py. Frontend: core/system-settings (useAgentModuleConfig/useUpdateAgentModuleConfig), admin AgentModuleSettingsDialog (gear button in the agent-chat header).

岗位 (Position) Subscription System

An admin-managed aggregation layer that decides, per user, which agents / skills / scheduled tasks / recommended questions they are subscribed to or can see. A user has at most one position (users.position_id, 1:1); unassigned users inherit the single is_default position. Note: the admin-configured prompts shown to users ARE the recommended questions (recommended_questions table) — there is no separate prompt-template store; the per-user user_custom_prompts (personal 常用提示词 bar) is untouched by this system.

Persistence (deerflow/persistence/positions/, wired as app.state.position_store; tables auto-created by create_all, migration 20260616_01):

• positions (PositionRow): id, name (unique), description, is_default (only one at a time — the store clears the flag on others).
• position_tag_bindings (PositionTagBindingRow): one row per (position, resource_type, tag). No binding for a resource_type = "use the default set". POSITION_RESOURCE_TYPES = (agent, skill, scheduled_task, recommended_question) — the same set drives the tag types (TAG_TYPES was extended with recommended_question).

Sync engine (deerflow/services/position_sync.py, wired as app.state.position_sync). sync_user(user) materializes three resource types into the existing "我的" tables, idempotently:

• agents → agent_favorites rows with origin='position'
• skills → skill_favorites rows with origin='position' (table SkillFavoriteRow, migration 20260616_04, auto-created by create_all)
• scheduled tasks → scheduled_task_subscriptions rows with origin='position'

Skills behave exactly like agents: a position only adds its tagged skills to the member's 我的技能 — the browse-all 广场 is never restricted. (This replaced the earlier query-time visibility filter, which wrongly shrank the square to the bound skill.) PositionSyncService.resolve_visible(user, type) returns set | None (None = unrestricted) and is now used only for the one remaining query-time type, recommended_question.

Sync only ever adds/removes origin='position' rows, so a user's own favorites/subscriptions are never clobbered. Default sets (no binding for a type): agents = built-ins (user_id IS NULL); skills = empty (square stays full, nothing added to 我的); scheduled tasks = empty; questions = untagged only (默认岗位 / 无标签绑定 只看到未打标的题目；打标的题目只在绑定了对应标签的 岗位展示 — the router interprets resolve_visible → None as "show only untagged", not "show everything"). origin/position_id columns were added to agent_favorites and scheduled_task_subscriptions — see migration 20260616_01 + scripts/sql/column_additions_mysql.sql. The running app also auto-adds these via _ensure_orm_columns_sync on startup.

Triggers: assigning/removing members, changing tag bindings (re-syncs all members), becoming the default position (sync_default_members), and login (auth.py::_sync_user_position, best-effort so a brand-new user inheriting the default gets their grants materialized).

API (app/gateway/routers/positions.py, /api/positions, admin only): position CRUD; GET/PUT /{id}/tag-bindings; GET/POST/DELETE /{id}/members (batch assign/unassign → triggers sync); POST /{id}/resync; GET /users (all users + their position); PUT /users/{user_id} (set one user's position). GET /me is the one non-admin route — it returns the caller's own effective position (explicit assignment, else the default position unassigned users inherit, else null); powers the top-bar 岗位 label (strategy-components/Header.tsx user cluster, via useMyPosition). Related changes: GET /api/recommended-questions is now filtered by the caller's position (?all=1 bypasses for admin management) and admin can tag questions via /api/tags/assignments/recommended_question/{id}. Default-position rule: tags are bulk-attached to all questions first, then the visibility filter applies — a position binding recommended_question 标签 sees only those tagged questions; a 默认岗位 / 无绑定 (resolve_visible → None) sees only the untagged questions (打标的题目不再泄漏到默认岗位). ?all=1 (admin management) skips this filter and returns every question with its tags. The list also accepts a repeatable tag_ids OR-filter — so the 推荐问题配置 page can display per-question 标签 and search by them (frontend RecommendedQuestionsPage + TagSearchBar). Tests: tests/test_recommended_questions_tags.py. GET /api/skills no longer restricts by position; each skill carries a favorited flag (true = in the caller's 我的技能, e.g. position-granted) so the skills page's 我的 tab includes owned ∪ position-granted skills while the 广场 stays complete. Frontend: core/positions

• admin PositionManagementPage (侧边栏「综合管理 → 岗位管理」 + 右下角设置菜单). Recommended questions stay rendered above the chat input (their original position), just filtered by the caller's position. Tests: tests/test_positions.py.

Portable Column Types (deerflow/persistence/types.py)

Two TypeDecorators keep ORM columns correct across the sqlite / postgres / mysql backends. All ORM timestamp/JSON columns must use these, not the raw SQLAlchemy types.

• PortableJSON ? native JSON on SQLite/Postgres; TEXT (with manual json.dumps/loads) on MySQL, where older servers lack JSON support.
• PortableLongText ? a large text column: maps to MySQL LONGTEXT (4 GB) and plain TEXT on SQLite/Postgres (already unbounded). Use for big JSON/markdown blobs that would overflow MySQL's 64 KB TEXT cap ? e.g. ai_writing_sessions.transcript.
• BeijingDateTime ? stores timestamps as Beijing (UTC+8) wall-clock time. The app records time with datetime.now(UTC), but MySQL DATETIME / SQLite have no timezone and persist whatever wall clock the driver serializes ? so a raw DateTime column would store UTC and read 8 hours behind Beijing. BeijingDateTime converts UTC?Beijing on bind and re-attaches +08:00 on result, so the database holds Beijing time while Python still gets timezone-aware datetimes (instant comparisons stay correct, including in WHERE clauses). PostgreSQL timestamptz stores the instant unambiguously, so values pass through unchanged there. Uses a fixed UTC+8 offset (China has no DST), so no tzdata dependency.

Schema Migrations / Manual Column-Add SQL (scripts/sql/)

Alembic auto-migration is disabled ? _run_alembic_upgrade(url) is commented out in persistence/engine.py. On startup only Base.metadata.create_all runs, which creates new tables but never patches existing ones (no new columns, no new indexes). So a column added to an ORM model appears automatically on a brand-new deployment, but is missing on any existing MySQL database ? and code reading/writing that column will error there until the column is added by hand.

Convention (required): whenever code adds a field to an existing table (a new mapped_column on an ORM model), you must, in the same change:

1. Keep the matching alembic version file under persistence/migrations/versions/ (guarded with _has_table/_has_column), and
2. Append the idempotent ADD COLUMN statement to scripts/sql/column_additions_mysql.sql, with a definition matching the ORM column exactly (type, length, nullability).

Operators upgrading an existing DB run the whole file (USE <db>; then execute) ? it is idempotent (existing columns are skipped via information_schema.columns), so it is safe to re-run. Index-only changes go in the sibling scripts/sql/leaderboard_indexes_mysql.sql (same idempotent pattern).

Sandbox System (packages/harness/deerflow/sandbox/)

Interface: Abstract Sandbox with execute_command, read_file, write_file, list_dir Provider Pattern: SandboxProvider with acquire, get, release lifecycle Implementations:

• LocalSandboxProvider - Singleton local filesystem execution with path mappings
• AioSandboxProvider (packages/harness/deerflow/community/) - Docker-based isolation

Virtual Path System:

• Agent sees: /mnt/user-data/{workspace,uploads,outputs}, /mnt/skills
• Physical: backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/..., deer-flow/skills/
• Translation: replace_virtual_path() / replace_virtual_paths_in_command()
• Detection: is_local_sandbox() checks sandbox_id == "local"
• {user_id} for path resolution = resolve_path_user_id(thread_id), NOT the current login. Task-deeplink conversations (rwfx/3qfx/xdfx, threads_meta.metadata.taskId) are shared across all users (see the taskId escape branches in thread_meta/sql.py), but generated files/uploads live under a per-user dir. Resolving by current login meant account B opening a shared task conversation 404'd on the md/artifacts account A generated ("切换账号后生成的文件找不到"). deerflow/runtime/thread_paths.py (resolve_path_user_id sync / aresolve_path_user_id async) resolves a task-shared thread to its creator (threads_meta.user_id), so every account reads/writes the same physical directory; personal threads resolve to the current login unchanged (no behavior change). Wired at every thread-path seam: write source ThreadDataMiddleware.before_agent (drives all sandbox writes via thread_data paths), present_file_tool, uploads/manager.get_uploads_dir + uploads_middleware + uploads router, artifact serving (path_utils.{a,}resolve_thread_virtual_path → artifacts/skills routers), embedded client.get_artifact. Positive (taskId→creator) results are cached process-wide (immutable mapping); any failure / memory-mode store / missing row falls back to the current login. Caveat: files a non-creator wrote into a shared thread before this fix stay orphaned under that writer's dir (no data loss); all new writes consolidate under the creator. Extended to 圆桌会商 system threads (metadata.thread_type == "roundtable" — Step 2 seats/coordinator + Step 3 report/summary/dashboard/structure, created by multi_agent.py with _ROUNDTABLE_THREAD_METADATA): their Step2/Step3 sandbox artifacts (md/流程图/报告) must stay viewable when a历史草稿 is reopened or a taskId会商 draft is opened by another account, so thread_paths._is_creator_shared_thread(metadata) treats both taskId and thread_type=="roundtable" as creator-shared, and ThreadMetaRepository.check_access (the artifacts router's owner_check) bypasses the owner filter for both — roundtable thread ids are only surfaced inside the owner's draft, so this does not let unrelated accounts discover others' 会商 files. Tests: tests/test_thread_paths_owner.py, tests/test_thread_paths_roundtable.py.

Sandbox Tools (in packages/harness/deerflow/sandbox/tools.py):

• bash - Execute commands with path translation and error handling
• ls - Directory listing (tree format, max 2 levels)
• read_file - Read file contents with optional line range
• write_file - Write/append to files, creates directories
• str_replace - Substring replacement (single or all occurrences); same-path serialization is scoped to (sandbox.id, path) so isolated sandboxes do not contend on identical virtual paths inside one process

Subagent System (packages/harness/deerflow/subagents/)

Built-in Agents: general-purpose (all tools except task) and bash (command specialist) Execution: Dual thread pool - _scheduler_pool (3 workers) + _execution_pool (3 workers) Concurrency: MAX_CONCURRENT_SUBAGENTS = 3 enforced by SubagentLimitMiddleware (truncates excess tool calls in after_model), 15-minute timeout Flow: task() tool ? SubagentExecutor ? background thread ? poll 5s ? SSE events ? result Events: task_started, task_running, task_completed/task_failed/task_timed_out

Tool System (packages/harness/deerflow/tools/)

get_available_tools(groups, include_mcp, model_name, subagent_enabled) assembles:

1. Config-defined tools - Resolved from config.yaml via resolve_variable()
2. MCP tools - From enabled MCP servers (lazy initialized, cached with mtime invalidation)
3. Built-in tools:
   • present_files - Make output files visible to user (only /mnt/user-data/outputs)
   • ask_clarification - Request clarification (intercepted by ClarificationMiddleware ? interrupts)
   • view_image - Read image as base64 (added only if model supports vision)
4. Subagent tool (if enabled):
   • task - Delegate to subagent (description, prompt, subagent_type, max_turns)

Community tools (packages/harness/deerflow/community/):

• tavily/ - Web search (5 results default) and web fetch (4KB limit)
• jina_ai/ - Web fetch via Jina reader API with readability extraction
• firecrawl/ - Web scraping via Firecrawl API

ACP agent tools:

• invoke_acp_agent - Invokes external ACP-compatible agents from config.yaml
• ACP launchers must be real ACP adapters. The standard codex CLI is not ACP-compatible by itself; configure a wrapper such as npx -y @zed-industries/codex-acp or an installed codex-acp binary
• Missing ACP executables now return an actionable error message instead of a raw [Errno 2]
• Each ACP agent uses a per-thread workspace at {base_dir}/users/{user_id}/threads/{thread_id}/acp-workspace/. The workspace is accessible to the lead agent via the virtual path /mnt/acp-workspace/ (read-only). In docker sandbox mode, the directory is volume-mounted into the container at /mnt/acp-workspace (read-only); in local sandbox mode, path translation is handled by tools.py
• image_search/ - Image search via DuckDuckGo

MCP System (packages/harness/deerflow/mcp/)

• Uses langchain-mcp-adapters MultiServerMCPClient for multi-server management
• Lazy initialization: Tools loaded on first use via get_cached_mcp_tools()
• Cache invalidation: Detects config file changes via mtime comparison
• Transports: stdio (command-based), SSE, HTTP
• OAuth (HTTP/SSE): Supports token endpoint flows (client_credentials, refresh_token) with automatic token refresh + Authorization header injection
• Runtime updates: Gateway API saves to extensions_config.json; LangGraph detects via mtime

Skills System (packages/harness/deerflow/skills/)

• Location: deer-flow/skills/{public,custom}/
• Format: Directory with SKILL.md (YAML frontmatter: name, description, license, allowed-tools)
• Loading: load_skills() recursively scans skills/{public,custom} for SKILL.md, parses metadata, and reads enabled state from extensions_config.json. A directory containing SKILL.md is a self-contained leaf — discovery does not descend into its subdirectories, so a skill package that bundles its own examples//references/ sub-skills (each with a SKILL.md) surfaces as one skill, not a swarm of phantom entries.
• Discovery↔delete consistency: a skill is keyed for discovery by its frontmatter name, but its on-disk directory name may differ (hand-dropped / mis-extracted package). LocalSkillStorage._resolve_skill_dir(name, category) resolves the discovered directory (canonical <category>/<name> fast-path, else scan-by-frontmatter-name); custom_skill_exists/read_custom_skill/delete_custom_skill all go through it, so anything that gets listed can also be read and deleted regardless of directory name (the old code assumed custom/<name>/ and 404'd "Custom skill 'x' not found." on any non-canonical layout). Delete is path-guarded to never escape the custom/ root. Tests: tests/test_skill_storage_delete.py.
• Injection: Enabled skills listed in agent system prompt with container paths
• Installation: POST /api/skills/install extracts .skill ZIP archive to custom/ directory
• Chinese name (name_zh): a Chinese display name stored in the skills DB table (SkillRow.name_zh, String(128), migration 20260609_01) ? never written to the on-disk SKILL.md, so it works for built-in skills too. Threaded through SkillStore create/ensure_legacy/update/update_admin exactly like detail; surfaced on SkillResponse.name_zh from the ownership record in _skill_to_response. Authorization mirrors detail: custom skill ? owner or admin; built-in/public ? admin only (row materialized on demand via ensure_legacy). Endpoints (in app/gateway/routers/skills.py): PUT /api/skills/{name}/name-zh (set/clear), POST /api/skills/{name}/name-zh/generate (LLM generates one), POST /api/skills/name-zh/generate-missing (????: admin-only bulk-generate for every skill that lacks one, bounded-concurrency asyncio.gather). Optional model selection: both generate endpoints accept an optional JSON body {model_name} ? an explicit name wins, else the curator model setting, else the main model (_resolve_skill_name_zh_model_name). Model-error surfacing: _generate_skill_name_zh raises SkillNameZhModelError when the model itself fails (build/timeout/API error) ? distinct from the model returning an unusable name (? None). The single endpoint maps that to 502 ?????,?????????:?; the bulk endpoint fails fast on a model-build error (502) and otherwise carries a representative model_error string in SkillNameZhBulkResponse so the UI can say "N ?????????". The frontend (skill-settings-page.tsx list + skill-detail-dialog.tsx editor, plus the agent skill pickers/badges in NewAgentPage.tsx + agent-card.tsx) shows name_zh || name, offers per-skill "AI ??" plus an admin-only header "???????" button, and a compact model dropdown (components/workspace/model-name-select.tsx, default = configured model) beside each action.

Skill Address Management (技能地址管理 — URL/IP 批量替换)

Admin-only maintenance tool to find every http(s):// URL and IPv4 (ip[:port]) address in the running skills (skills/{public,custom}, all .md + .py, recursing into scripts//references//templates/) and rewrite them in bulk — e.g. swapping a UAT endpoint for production across all skills at once. The same address commonly lives in both a skill's SKILL.md and its scripts/*.py, so results are aggregated by unique address with every occurrence (file:line + snippet) and replaced together.

• Pure engine — deerflow/skills/address_scan.py (harness, no app.*): find_spans / scan_occurrences (URL regex with trailing-punctuation strip; IPv4 with 0–255 octet check + optional port; an IP inside a URL is not separately surfaced), apply_replacements (substring-safe: re-detects exact token spans and only rewrites a span whose text equals a requested from, so 1.2.3.4 is never rewritten inside 11.2.3.40), infer_kind, is_valid_replacement. Tested in tests/test_skill_addresses.py.
• Router — app/gateway/routers/skill_addresses.py (/api/skill-addresses, admin only). Availability at scale (150+ skills): every filesystem walk/read/write runs via asyncio.to_thread so the event loop is never blocked. Routes: GET /scan (aggregated, one-shot) + GET /scan/stream (per-skill NDJSON progress {type:start|progress|result}); POST /preview (dry-run diff + validation warnings); POST /apply + POST /apply/stream (per-file NDJSON progress); GET /backups; POST /rollback/{backup_id}. Apply safety: optimistic lock via expected_total (409 if files changed since the scan), path-guard (is_relative_to the skills root), auto disk backup to {skills_root}/.address-edits/{timestamp}/ (the .-prefixed dir is excluded from scans), then refresh_skills_system_prompt_cache_async() so edits take effect immediately. Registered in app.py after sensitive_words.router. Per-edit history: each apply also writes a sibling manifest {skills_root}/.address-edits/{timestamp}.manifest.json (best-effort) capturing {created_at, replaced_total, replacements:[{from,to,kind}], changed_files}; the sibling-not-child placement keeps it out of both the restore set and the file count. GET /backups returns each record enriched with replaced_total + replacements (from the manifest; falls back to dir mtime when absent), and POST /rollback/{backup_id} restores a single record — so the UI shows a full modification history and rolls back any one entry independently.
• Frontend — the panel is the reusable SkillAddressesPanel (exported from pages/SkillAddressesPage.tsx; its default export keeps the standalone route /page/strategy/admin/skill-addresses as a direct-link fallback). It is embedded as a「技能地址管理」tab inside the admin「技能管理」page pages/AdminSkillsPage.tsx (Tabs: 技能列表 + 技能地址管理; reached from the 「技能管理」item in the「设置和更多」dropdown workspace-nav-menu.tsx → /page/workspace/admin/skills) — it is no longer a separate entry in the components/page-sidebar.tsx settings dropdown. The panel: auto-scan on open with a progress bar (扫描逐技能 / 应用逐文件 X/N), URL/IP type filter, search, a「片段批量套用」helper (e.g. all uat.4.cn → prod.4.cn), per-address「替换为」input with expandable occurrences, inline 预览 panel, 应用, multi-keyword search (space-separated, all-must-hit), and a 修改记录 (modification history) panel — each apply is one record (time / which addresses changed from→to / file & replacement counts) with a per-record 回退此次 (single-record rollback, with inline confirm). Client + NDJSON stream parsing in strategy-components/api/skill-addresses.ts. Scope is the running skills only; the deploy/minimal/skills/ copy is not touched (sync separately).

Skill Compression (技能压缩)

When the skill catalogue grows large, injecting every enabled skill full-text into the lead agent's system prompt bloats it and wastes tokens. Skill compression is an admin-gated mode that keeps only 常驻(免压缩) skills in the prompt body and surfaces the rest on demand via keyword retrieval.

• Master switch — system_settings.json → skill_compression (SkillCompressionSettings: enabled default false, search_top_n default 5, keep_index_in_prompt default true). Read/written by GET/PUT /api/system-settings/skill-compression (GET open to any user — the skill detail page reads it; PUT admin-only, refreshes the prompt cache).
• Per-skill 常驻 flag — stored in the DB: SkillRow.always_on (Boolean, migration 20260616_03, idempotent SQL in column_additions_mysql.sql), not in extensions_config.json (the source-tree config file is read-only / unsafe to rewrite in some deployments, and built-in skills carry no file entry). Threaded through SkillStore create/ensure_legacy/update/update_admin + list_always_on_names() exactly like name_zh/detail; surfaced on SkillResponse.always_on from the ownership record in _skill_to_response. Toggled via PUT /api/skills/{name}/always-on (custom skill → owner or admin; built-in / public → admin only, materializes a row on demand via ensure_legacy). The prompt builder stamps Skill.always_on from list_always_on_names() via a sync DB bridge (_apply_always_on in lead_agent/prompt.py).
• Prompt split — get_skills_prompt_section (lead_agent/prompt.py) reads the compression settings (skipped for restricted custom agents — they already have a curated allowlist). When off, behaviour is unchanged (all enabled skills full-text). When on, always_on skills render as full <skill> blocks; the rest are dropped from the body, replaced by a search_skills usage note plus (when keep_index_in_prompt) a name-only <compressed_skills> index. The _get_cached_skills_prompt_section cache key includes the always_on flag + compression flags so changes invalidate correctly.
• Retrieval tool — search_skills (tools/builtins/skill_tools.py, search_skills_impl) keyword-scores name + description over enabled, visible, non-archived skills (same visibility filter as skill_list) and returns top-N {name, description, category, location}. It is registered in get_available_tools only when compression is enabled (offline, deterministic, zero external deps). The agent then read_files the returned location.

Tests: tests/test_skill_compression.py.

Skill Curator (packages/harness/deerflow/skills/curator.py)

Background skill lifecycle manager. The Gateway lifespan starts a resident scheduler task (_curator_scheduler_loop in app/gateway/app.py) that ? after a 5-minute startup delay ? calls maybe_run_curator() every hour. A cross-process file lock (.curator_scheduler.lock) elects a single leader worker so the scheduler runs once per deployment. Per run:

• Master switch ? skill_evolution.enabled (default false) gates the whole feature; curator.enabled (default false) gates the curator specifically. Both must be true for the curator to act. should_run_now() and _curator_thread_target() both enforce this ? so manual POST /api/curator/run also respects the master switch. Config is config.yaml.skill_evolution deep-merged with per-user users/{user_id}/skill_evolution.json.
• Per-user run lock ? .curator_state carries running / started_at; _try_acquire_run_lock() prevents the same user's curator from running concurrently (scheduler vs. manual trigger, or double manual clicks). A lock whose started_at is older than _CURATOR_STALE_LOCK_HOURS (6h) is treated as process-crash residue and can be taken over.
• Concurrency cap ? maybe_run_curator() collects all due users and hands them to _dispatch_curator_runs(), which runs them through a ThreadPoolExecutor capped at _CURATOR_MAX_CONCURRENCY (5) ? avoids a thundering herd of LLM calls when many users come due at once (e.g. first deploy).
• Failure back-off ? _release_run_lock() always stamps last_attempt_at. When a first run keeps crashing (last_run_at never set), should_run_now() retries on _CURATOR_RETRY_HOURS (6h) instead of every hourly tick.
• should_run_now() is the real gate: the hourly tick only checks "is the per-user interval_hours (default 168h) elapsed". Timestamp parsing tolerates naive (tz-less) strings ? a corrupt last_run_at triggers a re-run rather than a permanent deadlock.
• Published-skill filter ? SkillStore.list_published_names() is a batch query; the curator fetches the published-name set once per run instead of one DB round-trip per candidate skill.
• LLM review timeout ? _run_llm_review() runs agent.invoke() in a worker thread bounded by curator.review_timeout_seconds (default 600s). The model's per-request request_timeout can't bound a multi-turn React agent loop, so on timeout the run is abandoned (a warning is written to curator_log, visible in the Web UI) and the curator finishes cleanly + releases the run lock. The worker context is copied via contextvars.copy_context() so tool calls still resolve the correct user_id.
• Review model ? _run_llm_review() uses curator.model_name (a config.yaml models[] entry name) if set, else the primary model; it is editable from the Web UI evolution panel. The skill security scanner (scan_skill_content in security_scanner.py) reuses the same skill_evolution.curator.model_name setting ? review and moderation share one model. There is no separate moderation-model config field.

Model Factory (packages/harness/deerflow/models/factory.py)

• create_chat_model(name, thinking_enabled) instantiates LLM from config via reflection
• Supports thinking_enabled flag with per-model when_thinking_enabled overrides
• force_disable_thinking=True (kwarg) hard-disables thinking even when the model config declares no when_thinking_*/thinking shape — many internal vLLM/Qwen models only set supports_thinking: true, so thinking_enabled=False alone injects nothing and they keep reasoning on. The flag injects both OpenAI-compatible disable shapes (extra_body.chat_template_kwargs.enable_thinking=false for vLLM/Qwen + extra_body.thinking.type="disabled" for Zhipu GLM), only for classes that accept extra_body. Mirrors TitleMiddleware._force_disable_thinking. The lead agent reads the runtime flag thinking_force_disabled and forwards it (used by /api/open/chat)
• Supports vLLM-style thinking toggles via when_thinking_enabled.extra_body.chat_template_kwargs.enable_thinking for Qwen reasoning models, while normalizing legacy thinking configs for backward compatibility
• Supports supports_vision flag for image understanding models
• Config values starting with $ resolved as environment variables
• Missing provider modules surface actionable install hints from reflection resolvers (for example uv add langchain-google-genai)

Roundtable Model Fault Tolerance (会商模型自动容错 / 换下一个模型)

会商第二步（leader/seat）+ 第三步（summary/report/dashboard + 结构化抽取）单轮执行时，某模型出问题就自动切到下一个模型继续，并告知用户「『X』模型异常 → 已切换『Y』」。前台实时（multi_agent.py）与后台挂起（engine.py + InProcessRoundtableGateway）两条路径都覆盖。

• 可检测的模型失败：LLMErrorHandlingMiddleware 在同一模型重试 3 次后不抛异常，而是返回兜底 AIMessage —— 现在给它打 additional_kwargs[LLM_ERROR_MARKER="deerflow_llm_error"]=reason（model_dump 保留，经 checkpointer 序列化仍在）。is_llm_error_message(msg) 据标记（回退按已知兜底文案正文）识别，返回 reason。
• 候选模型链：app/gateway/roundtable_model_fallback.py::fallback_model_chain(current) 按 config.yaml models[] 顺序给出 [当前模型, …其余]（去重保序）。reason_text(reason) 翻译成简短中文。总开关 env ROUNDTABLE_MODEL_FALLBACK（默认开，0/false/off 关 → 单模型旧行为）；ROUNDTABLE_MODEL_FALLBACK_MAX（0=试完全部）。
• 触发换模型的三种情形：① 单轮 run 抛异常（start_run 失败 / idle 卡死 → timeout，其余 exception）；② 兜底错误文案（is_llm_error_message）；③ 交付为空（仅 seat/leader；leader 还含「既无正文又无派活」）。Step3 单例（report/summary/dashboard/ingest/结构化）的产物是落盘文件，不把「正文为空」当失败，只对真模型错误换模型。
• 后台：InProcessRoundtableGateway._run_turn_with_fallback(...) 包住 _run_agent_turn，逐候选模型试；换模型落 model_switch 诊断、全失败落 model_fallback_exhausted。run_leader/run_seat/run_report/run_summary/run_dashboard_data 全走它（report/summary/dashboard 的产物自愈重试保留，并记住已切到的可用模型用于后续续写轮）。各 Turn/Result 带回 switches: list[ModelSwitch]，引擎 _emit_switches → JobProgressWriter.model_switch(...) 往研讨时间线插一条 role="system" 系统提示气泡。
• 前台：_special_run/_leader_run stream 收尾后检测失败 → yield {status:"model_switch", role, agent_name, failed_model, next_model, reason} 帧并用下一个模型重新流式；终态帧带 model_used。结构化抽取 flow_extract.py 的 round1 model.astream 也包了同款换模型循环（emit event:"model_switch" 帧）。
• Step2 单席位隔离：engine._run_recommend/_run_chain/_run_dag 的每个席位执行包 try/except —— 一个席位彻底失败（落 seat_failed 诊断 + 置空交付）不拖垮整批/整链/整个作业。
• 前端：multi-agent.ts 解析 model_switch 帧 → onModelSwitch 回调（清空失败尝试的半截气泡 + 累计的 tool_call delta，phase 退回 connected）；useStep2Orchestration 弹 toast + 时间线插「系统提示」气泡；useStep3Summary/Report/Dashboard + flow-extract.ts 重置对应气泡并在步骤卡留可见提示；后台路径经 job-live.ts::toStep2Dialogue 把 role="system" 渲染成 amber「系统提示」气泡。
• Tests: tests/test_roundtable_model_fallback.py。

vLLM Provider (packages/harness/deerflow/models/vllm_provider.py)

• VllmChatModel subclasses langchain_openai:ChatOpenAI for vLLM 0.19.0 OpenAI-compatible endpoints
• Preserves vLLM's non-standard assistant reasoning field on full responses, streaming deltas, and follow-up tool-call turns
• Designed for configs that enable thinking through extra_body.chat_template_kwargs.enable_thinking on vLLM 0.19.0 Qwen reasoning models, while accepting the older thinking alias

IM Channels System (app/channels/)

Bridges external messaging platforms (Feishu, Slack, Telegram, DingTalk) to the DeerFlow agent via the LangGraph Server.

Architecture: Channels communicate with Gateway through the langgraph-sdk HTTP client (same as the frontend), ensuring threads are created and managed server-side. The internal SDK client injects process-local internal auth plus a matching CSRF cookie/header pair so Gateway accepts state-changing thread/run requests from channel workers without relying on browser session cookies.

Components:

• message_bus.py - Async pub/sub hub (InboundMessage ? queue ? dispatcher; OutboundMessage ? callbacks ? channels)
• store.py - JSON-file persistence mapping channel_name:chat_id[:topic_id] ? thread_id (keys are channel:chat for root conversations and channel:chat:topic for threaded conversations)
• manager.py - Core dispatcher: creates threads via client.threads.create(), routes commands, keeps Slack/Telegram on client.runs.wait(), and uses client.runs.stream(["messages-tuple", "values"]) for Feishu incremental outbound updates
• base.py - Abstract Channel base class (start/stop/send lifecycle)
• service.py - Manages lifecycle of all configured channels from config.yaml
• slack.py / feishu.py / telegram.py / dingtalk.py - Platform-specific implementations (feishu.py tracks the running card message_id in memory and patches the same card in place; dingtalk.py optionally uses AI Card streaming for in-place updates when card_template_id is configured)

Message Flow:

1. External platform -> Channel impl -> MessageBus.publish_inbound()
2. ChannelManager._dispatch_loop() consumes from queue
3. For chat: look up/create thread through Gateway's LangGraph-compatible API
4. Feishu chat: runs.stream() ? accumulate AI text ? publish multiple outbound updates (is_final=False) ? publish final outbound (is_final=True)
5. Slack/Telegram chat: runs.wait() ? extract final response ? publish outbound
6. Feishu channel sends one running reply card up front, then patches the same card for each outbound update (card JSON sets config.update_multi=true for Feishu's patch API requirement)
7. DingTalk AI Card mode (when card_template_id configured): runs.stream() ? create card with initial text ? stream updates via PUT /v1.0/card/streaming ? finalize on is_final=True. Falls back to sampleMarkdown if card creation or streaming fails
8. For commands (/new, /status, /models, /memory, /help): handle locally or query Gateway API
9. Outbound ? channel callbacks ? platform reply

Configuration (config.yaml -> channels):

• langgraph_url - LangGraph-compatible Gateway API base URL (default: http://localhost:8001/api)
• gateway_url - Gateway API URL for auxiliary commands (default: http://localhost:8001)
• In Docker Compose, IM channels run inside the gateway container, so localhost points back to that container. Use http://gateway:8001/api for langgraph_url and http://gateway:8001 for gateway_url, or set DEER_FLOW_CHANNELS_LANGGRAPH_URL / DEER_FLOW_CHANNELS_GATEWAY_URL.
• Per-channel configs: feishu (app_id, app_secret), slack (bot_token, app_token), telegram (bot_token), dingtalk (client_id, client_secret, optional card_template_id for AI Card streaming)

Memory System ? Memory V2 (packages/harness/deerflow/agents/memory/)

Pluggable memory architecture (Hermes-style). See docs/MEMORY_V2_DESIGN_ZH.md and docs/HINDSIGHT_SETUP_ZH.md for the full design.

Components:

• provider.py - MemoryProvider abstract base (12 lifecycle/hook methods)
• manager.py - MemoryManager orchestrates builtin + ?1 external provider; build_memory_manager() is the construction entry point
• providers/builtin.py - BuiltinFileProvider: local USER.md / MEMORY.md, ? delimiter, char quotas, file locking, atomic writes, frozen snapshot
• providers/hindsight.py - HindsightProvider: HTTP client to a self-hosted Hindsight Docker instance, dual-bank routing
• security.py - injection/exfiltration scan + invisible-unicode block for memory content
• message_processing.py - filter_messages_for_memory (last-turn extraction for Hindsight retain)

Two stores (binary classification):

• USER.md at {base_dir}/users/{user_id}/USER.md ? user profile, shared across agents
• MEMORY.md at {base_dir}/users/{user_id}/agents/{agent_id}/MEMORY.md ? agent-private working notes
• user_id resolved via get_effective_user_id(); "default" in no-auth mode

Write paths:

• LLM actively calls the memory tool (tools/builtins/memory_tool.py) ? writes USER.md/MEMORY.md, mirrors to Hindsight
• LLM calls hindsight_recall/reflect/retain tools (tools/builtins/hindsight_tools.py) ? only in tools/hybrid memory_mode
• Hindsight auto_retain ? every turn auto-stored to the work bank (via MemoryMiddleware.after_agent)

Injection:

• builtin frozen snapshot ? injected into the system prompt via lead_agent/prompt.py _get_memory_context
• Hindsight dynamic recall ? injected per model call via MemoryMiddleware.wrap_model_call (transient ? not persisted to state, so it never leaks to the frontend)

Hindsight dual-bank isolation (B-scheme):

• global user bank deerflow-user-{user_id} ? cross-agent
• agent work bank deerflow-work-{user_id}-{agent_id} ? per-agent
• hindsight-client is an optional dependency; when missing, HindsightProvider.is_available() returns False and the system falls back to builtin-only

Migration: PYTHONPATH=. python scripts/migrate_memory_to_v2.py [--all|--user-id X] [--dry-run] converts legacy memory.json into USER.md/MEMORY.md.

Configuration (config.yaml ? memory, nested): enabled, provider (builtin/hindsight), builtin.{memory_char_limit,user_char_limit,...}, hindsight.{mode,api_url,api_key,bank_id_template,work_bank_id_template,memory_mode,...}, injection.{enabled,max_tokens,include_builtin,include_hindsight,context_tag}, security.{scan_content,block_invisible_unicode}. Per-user runtime overrides (whitelisted fields) live at {base_dir}/users/{user_id}/memory_config.json ? see get_effective_memory_config().

Gateway API ? see the Memory router below (/api/memory/v2*).

Reflection System (packages/harness/deerflow/reflection/)

• resolve_variable(path) - Import module and return variable (e.g., module.path:variable_name)
• resolve_class(path, base_class) - Import and validate class against base class

Config Schema

config.yaml key sections:

• models[] - LLM configs with use class path, supports_thinking, supports_vision, provider-specific fields
• vLLM reasoning models should use deerflow.models.vllm_provider:VllmChatModel; for Qwen-style parsers prefer when_thinking_enabled.extra_body.chat_template_kwargs.enable_thinking, and DeerFlow will also normalize the older thinking alias
• tools[] - Tool configs with use variable path and group
• tool_groups[] - Logical groupings for tools
• sandbox.use - Sandbox provider class path
• skills.path / skills.container_path - Host and container paths to skills directory
• title - Auto-title generation (enabled, max_words, max_chars, prompt_template)
• summarization - Context summarization (enabled, trigger conditions, keep policy)
• subagents.enabled - Master switch for subagent delegation
• memory - Memory system (enabled, storage_path, debounce_seconds, model_name, max_facts, fact_confidence_threshold, injection_enabled, max_injection_tokens)
• auth_login - Custom login modes layered on the base auth (deerflow/config/login_config.py):
  • username_login.{require_password, password} - Password gate for POST /api/v1/auth/login/username. When require_password=true, the (otherwise passwordless) username login additionally requires ?password=<shared-secret>; empty password falls back to 123ewq. Gate only ? it does not change which account logs in.
  • token_login.{enabled, token_info_url, service_authorization, timeout_seconds, verify_ssl, ca_cert_path} - Token-exchange login (POST /api/v1/auth/login/token). The gateway POSTs to token_info_url with the caller-supplied token placed directly in the Authorization: Bearer <token> header (no cookie; the static service_authorization field is deprecated/unused, kept only for backward compat), reads data.username from a {"state":"200",...} body, then signs the matching local user in (auto-registering if absent). token_info_url is HTTPS ? verify_ssl: false skips cert verification for self-signed/internal UAT hosts, or set ca_cert_path to a PEM bundle (takes precedence, keeps verification on). See docs/AUTH_LOGIN.md for the full request/response contract.

Both endpoints are whitelisted in auth_middleware._PUBLIC_EXACT_PATHS. The shared _get_or_create_user_by_username() helper (in routers/auth.py) backs both /login/username and /login/token.

• log_system - Admin-only external log-system entry (deerflow/config/log_system_config.py): {enabled, url, label}. Surfaced read-only by GET /api/system-settings/log-system (admin only); the frontend renders it as a "????" item in the bottom-right settings menu (workspace-nav-menu.tsx) that opens url in a new tab. Disabled by default.

extensions_config.json:

• mcpServers - Map of server name ? config (enabled, type, command, args, env, url, headers, oauth, description)
• skills - Map of skill name ? state (enabled). 常驻(免压缩)/always_on lives in the skills DB table, not here.

Both can be modified at runtime via Gateway API endpoints or DeerFlowClient methods.

Embedded Client (packages/harness/deerflow/client.py)

DeerFlowClient provides direct in-process access to all DeerFlow capabilities without HTTP services. All return types align with the Gateway API response schemas, so consumer code works identically in HTTP and embedded modes.

Architecture: Imports the same deerflow modules that Gateway API uses. Shares the same config files and data directories. No FastAPI dependency.

Agent Conversation:

• chat(message, thread_id) ? synchronous, accumulates streaming deltas per message-id and returns the final AI text
• stream(message, thread_id) ? subscribes to LangGraph stream_mode=["values", "messages", "custom"] and yields StreamEvent:
  • "values" ? full state snapshot (title, messages, artifacts); AI text already delivered via messages mode is not re-synthesized here to avoid duplicate deliveries
  • "messages-tuple" ? per-chunk update: for AI text this is a delta (concat per id to rebuild the full message); tool calls and tool results are emitted once each
  • "custom" ? forwarded from StreamWriter
  • "end" ? stream finished (carries cumulative usage counted once per message id)
• Agent created lazily via create_agent() + _build_middlewares(), same as make_lead_agent
• Supports checkpointer parameter for state persistence across turns
• reset_agent() forces agent recreation (e.g. after memory or skill changes)
• See docs/STREAMING.md for the full design: why Gateway and DeerFlowClient are parallel paths, LangGraph's stream_mode semantics, the per-id dedup invariants, and regression testing strategy

Gateway Equivalent Methods (replaces Gateway API):

Category	Methods	Return format
Models	list_models(), get_model(name)	{"models": [...]}, {name, display_name, ...}
MCP	get_mcp_config(), update_mcp_config(servers)	{"mcp_servers": {...}}
Skills	list_skills(), get_skill(name), update_skill(name, enabled), install_skill(path)	{"skills": [...]}
Memory	get_memory(), reload_memory(), get_memory_config(), get_memory_status()	dict
Uploads	upload_files(thread_id, files), list_uploads(thread_id), delete_upload(thread_id, filename)	{"success": true, "files": [...]}, {"files": [...], "count": N}
Artifacts	get_artifact(thread_id, path) ? (bytes, mime_type)	tuple

Key difference from Gateway: Upload accepts local Path objects instead of HTTP UploadFile, rejects directory paths before copying, and reuses a single worker when document conversion must run inside an active event loop. Artifact returns (bytes, mime_type) instead of HTTP Response. The new Gateway-only thread cleanup route deletes .deer-flow/threads/{thread_id} after LangGraph thread deletion; there is no matching DeerFlowClient method yet. update_mcp_config() and update_skill() automatically invalidate the cached agent.

Tests: tests/test_client.py (77 unit tests including TestGatewayConformance), tests/test_client_live.py (live integration tests, requires config.yaml)

Gateway Conformance Tests (TestGatewayConformance): Validate that every dict-returning client method conforms to the corresponding Gateway Pydantic response model. Each test parses the client output through the Gateway model ? if Gateway adds a required field that the client doesn't provide, Pydantic raises ValidationError and CI catches the drift. Covers: ModelsListResponse, ModelResponse, SkillsListResponse, SkillResponse, SkillInstallResponse, McpConfigResponse, UploadResponse, MemoryConfigResponse, MemoryStatusResponse.

Development Workflow

Test-Driven Development (TDD) ? MANDATORY

Every new feature or bug fix MUST be accompanied by unit tests. No exceptions.

• Write tests in backend/tests/ following the existing naming convention test_<feature>.py
• Run the full suite before and after your change: make test
• Tests must pass before a feature is considered complete
• For lightweight config/utility modules, prefer pure unit tests with no external dependencies
• If a module causes circular import issues in tests, add a sys.modules mock in tests/conftest.py (see existing example for deerflow.subagents.executor)

# Run all tests
make test

# Run a specific test file
PYTHONPATH=. uv run pytest tests/test_<feature>.py -v

Running the Full Application

From the project root directory:

make dev

This starts all services and makes the application available at http://localhost:2026.

All startup modes:

	Local Foreground	Local Daemon	Docker Dev	Docker Prod
Dev	./scripts/serve.sh --dev make dev	./scripts/serve.sh --dev --daemon make dev-daemon	./scripts/docker.sh start make docker-start	?
Prod	./scripts/serve.sh --prod make start	./scripts/serve.sh --prod --daemon make start-daemon	?	./scripts/deploy.sh make up

Action	Local	Docker Dev	Docker Prod
Stop	./scripts/serve.sh --stop make stop	./scripts/docker.sh stop make docker-stop	./scripts/deploy.sh down make down
Restart	./scripts/serve.sh --restart [flags]	./scripts/docker.sh restart	?

Nginx routing:

• /api/langgraph/* ? Gateway embedded runtime (8001), rewritten to /api/*
• /api/* (other) ? Gateway API (8001)
• / (non-API) ? Frontend (3000)

Running Backend Services Separately

From the backend directory:

# Gateway API
make gateway

Direct access (without nginx):

• Gateway: http://localhost:8001

Frontend Configuration

The frontend uses environment variables to connect to backend services:

• NEXT_PUBLIC_LANGGRAPH_BASE_URL - Defaults to /api/langgraph (through nginx)
• NEXT_PUBLIC_BACKEND_BASE_URL - Defaults to empty string (through nginx)

When using make dev from root, the frontend automatically connects through nginx.

Key Features

File Upload

Multi-file upload with automatic document conversion:

• Endpoint: POST /api/threads/{thread_id}/uploads
• Supports: PDF, PPT, Excel, Word documents (converted via markitdown)
• Rejects directory inputs before copying so uploads stay all-or-nothing
• Reuses one conversion worker per request when called from an active event loop
• Files stored in thread-isolated directories
• Agent receives uploaded file list via UploadsMiddleware

See docs/FILE_UPLOAD.md for details.

Plan Mode

TodoList middleware for complex multi-step tasks:

• Controlled via runtime config: config.configurable.is_plan_mode = True
• Provides write_todos tool for task tracking
• One task in_progress at a time, real-time updates

See docs/plan_mode_usage.md for details.

Context Summarization

Automatic conversation summarization when approaching token limits:

• On/off switch is per-request, not the config file. The chat input box has a 「上下文压缩」 toggle (below 参考文献, default off) → frontend sends context.summarization_enabled → whitelisted in _CONTEXT_CONFIGURABLE_KEYS (app/gateway/services.py) → _build_middlewares reads it from the run config and calls _create_summarization_middleware(force_enabled=summarization_enabled). force_enabled=True mounts the middleware even when config.summarization.enabled is false; when the flag is absent (channels / scheduled runs) it falls back to the config flag. config.summarization.enabled now defaults to false.
• config.yaml summarization still supplies the tuning: trigger types (tokens, messages, or fraction of max input) and the keep policy.
• Keeps recent messages while summarizing older ones.

Tests: tests/test_summarization_toggle.py (gating + whitelist), tests/test_summarization_middleware.py (cache behavior). See docs/summarization.md for details.

Vision Support / Image Recognition (??)

Two paths, selected by config.yaml ? vision:

Dedicated vision model (delegated recognition) ? set vision.model_name to a model from models[]. The view_image tool sends the image to that model and returns its text description as the tool result, so a text-only main chat model can still "see" images. view_image accepts an optional question argument for targeted queries; the default prompt is vision.prompt (overridable), and vision.max_tokens caps the recognition output. Resolution/gating lives in deerflow/models/vision.py (resolve_vision_model_name, vision_enabled, is_delegated_vision, recognize_image).

Native vision (fallback) ? when vision.model_name is unset, behavior falls back to the main chat model, which only works if it has supports_vision: true:

• ViewImageMiddleware injects base64 image data into the conversation before the LLM call
• view_image stores the image in state.viewed_images for that middleware

Both the view_image tool and ViewImageMiddleware are registered whenever vision_enabled(model_name) is true (dedicated model configured or main model supports vision). Config schema: deerflow/config/vision_config.py (VisionConfig); tests: tests/test_vision.py.

Upload symlink-safety (cross-platform)

deerflow/uploads/manager.py:open_upload_file_no_symlink writes uploads without following a pre-existing destination symlink. O_NOFOLLOW is applied where available (POSIX) but is optional ? Windows lacks os.O_NOFOLLOW/O_NONBLOCK, so requiring it previously made every Windows upload fail with UnsafeUploadPathError. Pre-existing symlinks are still rejected via lstat, and the opened fd is re-checked with fstat. O_BINARY is added on Windows. Regression: tests/test_uploads_windows_safe.py.

Code Style

• Uses ruff for linting and formatting
• Line length: 240 characters
• Python 3.12+ with type hints
• Double quotes, space indentation

Documentation

See docs/ directory for detailed documentation:

• CONFIGURATION.md - Configuration options
• ARCHITECTURE.md - Architecture details
• API.md - API reference
• SETUP.md - Setup guide
• FILE_UPLOAD.md - File upload feature
• PATH_EXAMPLES.md - Path types and usage
• summarization.md - Context summarization
• plan_mode_usage.md - Plan mode with TodoList
• BACKEND_RUN_STOP_ZH.md - Windows-first backend start/stop and troubleshooting guide