sync update from LAN repo (source master @ ca48c4ec)
This commit is contained in:
1 parent
8bc8b0e082
commit
ea9dcff9c0
873 files changed
+107866
-14011
No files matched your search
@@ -1,5 +1,232 @@
|
||||
# CLAUDE.md
|
||||
|
||||
## Cold agent construction (2026-10-30)
|
||||
|
||||
`runtime/runs/worker.py` dispatches opted-in synchronous factories through
|
||||
`build_agent_with_timing`, backed by `utils/background_init.py`'s
|
||||
`run_background_init` and one shared `FileIOPool(1, 1, "agent-init")`.
|
||||
`make_lead_agent._deerflow_background_build = True` opts the built-in Lead Agent
|
||||
into background construction; unmarked custom factories retain their caller-loop
|
||||
contract. Serial admission prevents concurrent SDK cold imports and keeps waiting
|
||||
builds out of authentication's executor. The synchronous timing helper remains
|
||||
available. Model execution, checkpoint/store attachment and streaming stay in the
|
||||
original run task; no graph is shared or cached across runs/users.
|
||||
|
||||
Every async Lead Agent entry point uses this builder, including non-streaming
|
||||
and streaming open-chat endpoints and Workflow agent nodes. Do not call the
|
||||
synchronous factory directly on their event loop: it can block SDK import locks
|
||||
or deadlock while another background build awaits MCP discovery on that loop.
|
||||
Async tool consumers (AI writing skill agents and `task_tool`) use
|
||||
`get_available_tools_async`; deep-research capabilities and job execution also
|
||||
construct their material providers through `run_background_init`. Keep the
|
||||
common helper in `utils` with a lazy MCP import to avoid runtime/agent/tools
|
||||
import cycles. Synchronous client/provider APIs remain available.
|
||||
|
||||
Copy the caller's ContextVars into construction and restore the successfully built
|
||||
context before streaming, including a deferred-tool registry reset to `None`.
|
||||
The `mcp_initialization_loop` context manager forwards lazy MCP discovery from the
|
||||
worker to the owner event loop; exit it before copying back context. MCP cache
|
||||
invalidation and legacy synchronous callers retain their existing semantics.
|
||||
Cancellation drains the actual worker before releasing capacity, never starts the
|
||||
graph afterwards, and preserves terminal status/SSE cleanup. Check `abort_event`
|
||||
after construction as well as honoring task cancellation.
|
||||
|
||||
Regression: `packages/harness/tests/test_agent_build_responsiveness.py`,
|
||||
`test_agent_build_timing.py`, `test_mcp_background_build.py` and the run-finalization
|
||||
tests; `tests/test_async_agent_entrypoints.py` and
|
||||
`tests/test_async_tool_initialization.py` exercise overlapping HTTP/workflow/tool
|
||||
initializers. The HTTP/workflow cases run in subprocesses so an event-loop
|
||||
deadlock fails with a bounded timeout. Cold profiling found ~4.7 s in model SDK imports/client setup, compared with
|
||||
~14 ms in skill loading; moving only disk reads would not address this stall.
|
||||
|
||||
Local live cold acceptance (2026-10-30, official MiniMax-M2.7, tools enabled):
|
||||
the first build took 4.28 s while a second request received stream metadata in
|
||||
402 ms. During the conservative interior of the build window, 43 health/auth
|
||||
probes completed successfully; across all three conversations, health/auth
|
||||
P95 latency was 28/73 ms, with no event-loop-stall warning. Every answer ended
|
||||
normally and all synthetic threads were removed. Initialization/model latency
|
||||
still contributes to first-answer time; this change prevents blocking the server
|
||||
loop, rather than promising immediate model output.
|
||||
A separate live read-only tool check uploaded a random marker, allowed only
|
||||
`read_file`, and confirmed the tool result and final answer reproduced that marker
|
||||
without placing it in the prompt; the stream ended normally and the thread was removed.
|
||||
|
||||
Worker compatibility: no HTTP/SSE, task payload, auth or database contract changes.
|
||||
Running workers without reload retain old construction behavior; the new factory
|
||||
marker is harmless to old executors. `worker_pool/revision.py` hashes only the
|
||||
adapter directory and does not identify this harness patch. This change does not
|
||||
alter the controller's existing snapshot/ImageID-based drain-and-replace policy
|
||||
or `on_disconnect=cancel` behavior: restarting a production controller can still
|
||||
rotate idle workers or disconnect proxied streams. Do not infer uninterrupted
|
||||
worker operation from these process-local initialization changes. The live local
|
||||
acceptance instance has no enabled worker controller and does not touch remote
|
||||
workers. Existing worker routing/admission/storage/discovery/diagnostics/launcher
|
||||
compatibility regressions: 68 passed.
|
||||
|
||||
Final combined regression (2026-10-30): all 58 newly added responsiveness,
|
||||
cancellation, MCP-entrypoint and provider-lifecycle cases plus the 68 existing
|
||||
worker compatibility cases passed together (126 passed, no skips).
|
||||
The live recheck uploaded and converted 20 DOCX files while another conversation
|
||||
produced 43 visible text increments; all 172 concurrent authenticated probes
|
||||
succeeded (P95 105 ms, max 436 ms). A fresh-process cold build took 10.92 s while
|
||||
617 heartbeat samples continued (P95 scheduling delay 25 ms, max 146 ms).
|
||||
The existing backend and child workers were not restarted for this recheck;
|
||||
later entrypoint changes were verified through isolated HTTP/workflow tests.
|
||||
The full versioned backend/harness run completed with 3140 passed, 39 failed,
|
||||
4 skipped and 1 deselected. Against baseline `47cfed02`, 30 failures also failed
|
||||
on HEAD and the other 9 passed in isolated reruns on both versions; no current-only
|
||||
stable failure remained. The deselected legacy
|
||||
`test_cancelled_history_closes_iterator` hangs on both versions: its fake saver
|
||||
implements `alist`, while `limit=1` takes `aget_tuple`, so its unbounded entry
|
||||
event wait never completes. This is not a claim that the repository's entire
|
||||
test suite is green. Changed Python files introduced no Ruff diagnostics.
|
||||
|
||||
## Upload responsiveness (2026-10-30)
|
||||
|
||||
`app/gateway/routers/uploads.py` runs each staged upload transaction in a dedicated
|
||||
`FileIOPool(2, 2, "uploads")`, including 256 KiB spool copies, signature validation,
|
||||
optional synchronous conversion, sandbox acquisition/sync, commit, rollback and
|
||||
cleanup. Resolve the thread owner asynchronously before dispatch and propagate
|
||||
contextvars into workers. Waiting batches consume no worker from authentication,
|
||||
artifacts or event streaming. Per-event-loop directory locks serialize same-thread
|
||||
batches and listing/deletion through rollback/cleanup; keep them held until a
|
||||
cancelled worker stops. Listing/deletion use the artifact file pool off the loop.
|
||||
Cancellation signals a cooperative event, checks between file chunks and around
|
||||
commit/sync, and waits for real completion before multipart handles may close.
|
||||
The response contract and upload-policy enforcement remain unchanged.
|
||||
`sandbox/sandbox_provider.py` serializes initial singleton construction with
|
||||
reset/replacement/shutdown. Concurrent cold uploads must not create competing
|
||||
providers with independent sandbox reconciliation and idle-cleanup threads.
|
||||
Regression: `packages/harness/tests/test_sandbox_provider_concurrency.py`.
|
||||
|
||||
`file_conversion.py` also isolates its asynchronous conversion/extraction callers
|
||||
in a separate two-thread pool. PDF/Office parsing runs in a lazily initialized,
|
||||
two-process pool using `spawn` on Windows and Linux, so Python parser CPU work
|
||||
does not contend for the gateway GIL. Pass only file paths and converter choices;
|
||||
bound returned text inside the child and write sidecars in the owning parent
|
||||
worker. The global parser semaphore bounds sync/async callers together. A broken
|
||||
process pool fails the current conversion and is rebuilt on the next call.
|
||||
`FileIOPool.run_until_complete` provides non-abandoning cancellation for workers
|
||||
that own temporary input files; cancellation cannot close/delete an input still
|
||||
being parsed by a child process.
|
||||
Regression: `tests/test_upload_responsiveness.py`,
|
||||
`tests/test_conversion_responsiveness.py`, `tests/test_uploads_policy_enforcement.py`.
|
||||
|
||||
Local acceptance (Windows, Python 3.12, synthetic files via ASGI; no delay injected):
|
||||
20 x 900 KiB TXT uploads reduced first concurrent probe delay from 664/682 ms to
|
||||
3/4 ms. With real MarkItDown conversion, 20 x 800-paragraph DOCX uploads reduced
|
||||
maximum probe scheduling lag from 286/285 ms to 41/14 ms; all 20 sidecars matched
|
||||
the source text. The first spawned reader took 13.7 s to initialize/convert in
|
||||
this environment, while concurrent probe lag stayed below 16 ms. These are local
|
||||
responsiveness measurements, not production throughput guarantees. Upload,
|
||||
policy, conversion, large-file and runtime-contention regressions: 87 passed.
|
||||
Full-suite collection is separately blocked by the pre-existing missing
|
||||
`deerflow.models.global_prompt` import in `tests/test_global_prompt.py`.
|
||||
|
||||
## Thread metadata query performance and ORM deadlines (2026-10-22)
|
||||
|
||||
`ThreadMetaRepository._query_conditions` is shared by search/count: common
|
||||
SQLite scalar predicates and PG/MySQL string predicates plus system-thread
|
||||
exclusion run before SQL pagination/counting. Preserve AUTO owner resolution and
|
||||
the nonempty taskId owner bypass while applying agent_id/actionId together.
|
||||
`json_compat.JsonPredicate` binds values, preserves local missing/null and numeric
|
||||
equality on SQLite, and uses binary MySQL id equality and Unicode-safe blank
|
||||
seat detection. Nested/unusual-key filters and PG/MySQL numeric filters remain exact
|
||||
residual filters; openGauss retains its original path. Never drop an unsupported
|
||||
predicate or replace this with upstream's unadapted SQLite/PG-only compiler.
|
||||
|
||||
PG's omitted pool_recycle defaults to 300s (other backends remain 1800s);
|
||||
database.command_timeout is finite positive or null, default 30s. The shared
|
||||
pooled-engine kwargs are used on initial creation and auto-create retry. Only
|
||||
asyncpg receives command_timeout; this does not configure psycopg checkpointers.
|
||||
Regression entries: packages/harness/tests/test_thread_metadata_sql_filters.py,
|
||||
test_database_pool_deadlines.py and the existing test_thread_operation_timing.py.
|
||||
Master integration (2026-10-23) retains the maintained task/agent/action index
|
||||
columns as additional SQL narrowing before exact JSON predicates. Keep dedicated
|
||||
task deep-link card synchronization on every existing metadata mutation. Hydrated
|
||||
history still uses slim_display to bound oversized tool payloads without editing
|
||||
checkpoint contents; frontend retrieval-progress scopes remain isolated per turn.
|
||||
Thread POST defaults to metadata-only creation; only an explicit
|
||||
initialize_checkpoint=true writes an empty checkpoint. Never enqueue an empty
|
||||
checkpoint in the background: it could become the head after the real first run.
|
||||
Authorized metadata-only threads have empty state/history. Metadata GET defaults
|
||||
to the SQL row (running maps to busy); include_state=true and checkpoint-only
|
||||
legacy rows retain state reads. Owner decorators run before unscoped row reads.
|
||||
History explicitly uses root checkpoint_ns and closes alist before serialization;
|
||||
only the latest head carries full channel values.
|
||||
|
||||
The frontend uses BackgroundHistory as the SDK's external history adapter. Its
|
||||
post-stream mutate immediately captures streamed state with no checkpoint id;
|
||||
explicit end triggers background hydration. EOF/error still requires existing
|
||||
run confirmation/rejoin. Every async write is fenced by thread/run/epoch/request;
|
||||
new turns, navigation and unmount invalidate prior requests even if abort is
|
||||
ignored. Reject history missing the newest streamed turn, bound each request to
|
||||
15 seconds and retry completion sync once. Failure keeps the streamed snapshot.
|
||||
Normal explicit-end callbacks use the completed streamed snapshot before the
|
||||
next send can change page refs; background hydration never repeats them. Recovery
|
||||
may have missed final SSE frames, so its callback waits for guarded hydration or
|
||||
the retained snapshot fallback. New submits use checkpoint:null to follow the
|
||||
latest server head.
|
||||
|
||||
Auth's SQLiteUserRepository is a historical name: its injected session factory
|
||||
is the shared business ORM. With database.backend=mysql, login/auth/users and SQL
|
||||
business repositories use MySQL, independently of checkpoint configuration.
|
||||
LangGraph Store still follows checkpointer config and startup awaits runtime
|
||||
setup/migrations; do not claim total service/startup isolation from PG or migrate
|
||||
existing stores implicitly. See docs/THREAD_LATENCY_OPTIMIZATION_ZH.md for the audit.
|
||||
Additional regression entries: test_thread_lazy_checkpoint.py,
|
||||
tests/test_account_authentication.py and frontend pnpm test:thread-history.
|
||||
|
||||
## Multi-user contention controls (2026-10-22)
|
||||
|
||||
- `utils/file_io.py` separates bounded artifact IO (4 workers / 8 admitted) and
|
||||
event disk IO (2 / 4) from the default executor used by authentication.
|
||||
Contextvars propagate; cancellation cannot release admission until disk work
|
||||
actually stops. Heavy artifact listing, ZIP previews, sniffing and writes use it.
|
||||
- `runtime/runs/manager.py` sweeps completed live records after 300 seconds,
|
||||
200 candidates per pass. It requires the actual task to be done and verifies
|
||||
status/error/metadata/kwargs in the backing RunStore, updating a final snapshot
|
||||
when necessary. Failed persistence retains the record for later retry. It never
|
||||
deletes RunStore rows, checkpoints, events or stream buffers. `watch_completion`
|
||||
starts grace after finalization, and lifespan closes the sweeper. The live
|
||||
superseded fence survives a newer record's retirement; finalization never adds
|
||||
a business-database history read to its completion fence.
|
||||
- Run routes authorize the thread before unscoped persisted lookups, and verify
|
||||
that the loaded run belongs to that exact thread. Preserve private-owner 404s,
|
||||
taskId/system-thread sharing and existing action/agent partitions. Completed
|
||||
rejoin without a live bridge buffer emits end immediately; Last-Event-ID replay
|
||||
during the existing replay window remains untouched.
|
||||
- `run_events.backend=memory` is a compatibility alias for
|
||||
`SpoolingRunEventStore`, stored under `runtime_home()/run-events/events.sqlite3`.
|
||||
Per-thread sequence assignment and each batch commit are transactional; paged
|
||||
queries do not retain completed events in RAM. No automatic event/message
|
||||
deletion or truncation is introduced. The pure `MemoryRunEventStore` remains
|
||||
available for unit tests. Explicit db/JSONL formats are unchanged; JSONL disk
|
||||
operations now use the event executor and striped per-thread locks. Keep the
|
||||
spool and its WAL on the persistent local runtime volume. Do not reclassify it
|
||||
as a PG checkpoint or silently change explicit db configuration.
|
||||
- `runtime/fair_queue.py` provides bounded round-robin admission by user for the
|
||||
subagent scheduler (3 total / 2 per user) and background roundtables (4 / 2).
|
||||
Queued jobs consume no worker; subagent deadlines include queue time. Per-user
|
||||
pending capacity also prevents one user from exhausting the entire backlog.
|
||||
Actual coroutine completion, including cancellation before its first step,
|
||||
acknowledges slot release. Parent context/identity propagation stays intact.
|
||||
`RuntimeContentionConfig` defines optional budgets; limits apply per process
|
||||
and require restart. The older roundtable per-user environment override remains.
|
||||
- Both internal roundtable creation helpers only create metadata. Init does not
|
||||
append a synthetic seat message via loopback; `_leader_run` injects authoritative
|
||||
IDs on each real input, deriving them from the thread mapping for older clients
|
||||
without seat_meta. Background first-leader injection remains. Never schedule a
|
||||
late empty checkpoint write: it could overwrite real history.
|
||||
|
||||
Regression entries: `test_runtime_contention.py`, `test_completed_run_retention.py`,
|
||||
`test_thread_lazy_checkpoint.py`, `test_run_completion_timeout.py`,
|
||||
`tests/test_position_roundtable_sessions.py`, roundtable role/model/seat tests,
|
||||
artifact/auth/model concurrency tests and frontend `pnpm test:thread-history`.
|
||||
See `docs/MULTI_USER_CONTENTION_AUDIT_ZH.md` for measured scope and remaining
|
||||
model/SQL/checkpointer contention. Process-local run control still requires the
|
||||
current single-worker deployment; these queues do not enable multi-worker routing.
|
||||
|
||||
## Ordinary knowledge smart retrieval dual surface (2026-10-16)
|
||||
|
||||
`SmartReasoningRetrievalMiddleware` treats knowledge retrieval as an ordinary
|
||||
@@ -120,7 +347,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
||||
|
||||
### Ordinary Q&A map visualization
|
||||
|
||||
The ordinary lead-agent prompt includes `_ORDINARY_QA_VISUALIZATION_SECTION` only for normal, non-bootstrap, non-writing, non-notebook, non-scheduled runs and defines the highest-priority output-carrier gate. Only an explicit current-request ask for HTML, a webpage/page, dashboard, large-screen view, or `.html` file permits reading `page-report-generation`, calling `page_report_template`, or writing HTML. “Draw a flowchart/chart/map/mind map” and “make a visualization” never imply HTML: ordinary flowcharts must be inline Mermaid, ordinary charts keep the established message-inline path, and requested maps emit strict JSON in a `map-json` fence. A mind map is emitted only for an explicit current-request ask for a mind map/knowledge map (or `analysis-mindmap`); ordinary analyses, summaries, outlines, flowcharts, and generic JSON must never auto-upgrade into one. Its strict `mindmap-json` contract is a bounded rooted tree (`id`, `parentId`, `label`, semantic `kind`, optional detail/source refs), targeting 12–24 nodes with a hard cap of 36, at most four layers including the center topic, at most six top-level dimensions and three children beneath every later node. Models must consolidate related facts into `detail`/`sourceRefs`, not make one node per report bullet, datum, paragraph, or reference. An unfinished dedicated `mindmap-json` fence (and compatible historical `mindmap` fence) remains an ordinary streaming code block; only the completed, valid object lazy-renders as a compact left-to-right message card. Clicking a parent node, or its +/− control at the outgoing connection point, expands/collapses its child tree and the canvas uses an explicit visible edge stroke. The full dialog keeps its close and view tools in the top-right and a layout-derived all-map preview in the bottom-right; the message preview also permits pan, wheel zoom, and zoom/reset controls. The message card's icon-only download control opens PNG/SVG/JSON choices. Generic JSON never invokes this renderer. Ambiguity defaults to inline output. Once the HTML gate passes, delivery is mandatory as a sandbox-generated `/mnt/user-data/outputs/*.html` artifact opened with `present_files`; a full inline `html` fence is forbidden as a substitute. The shared frontend `MarkdownContent` keeps its exact legacy branch when no map-shaped fence exists; dedicated `map-json` fences and complete generic `json` fences that contain map fields and pass the full map schema are validated and lazy-rendered from `components/workspace/visualizations/`. Map/loading/error cards use the full message-content width so their edges align with the step panel. Non-map and incomplete generic JSON retain the ordinary code-block path.
|
||||
The ordinary lead-agent prompt includes `_ORDINARY_QA_VISUALIZATION_SECTION` only for normal, non-bootstrap, non-writing, non-notebook, non-scheduled runs and defines the highest-priority output-carrier gate. Only an explicit current-request ask for HTML, a webpage/page, dashboard, large-screen view, or `.html` file permits reading `page-report-generation`, calling `page_report_template`, or writing HTML. “Draw a flowchart/chart/map/mind map” and “make a visualization” never imply HTML: ordinary flowcharts must be inline Mermaid, ordinary charts keep the established message-inline path, and requested maps emit strict JSON in a `map-json` fence (markers use `name`+`coordinate` plus optional `color`; lines are included only when the user explicitly asks to connect places, must use `coordinates` with at least two `[lng,lat]` points, not `from`/`to`/`coords`, and default to `arrow`/`flow`). The frontend recovers those common aliases, assigns distinct marker colors, draws requested lines as looping SVG arrows, and drops unusable lines so markers still render. A mind map is emitted only for an explicit current-request ask for a mind map/knowledge map (or `analysis-mindmap`); ordinary analyses, summaries, outlines, flowcharts, and generic JSON must never auto-upgrade into one. Its strict `mindmap-json` contract is a bounded rooted tree (`id`, `parentId`, `label`, semantic `kind`, optional detail/source refs), targeting 12–24 nodes with a hard cap of 36, at most four layers including the center topic, at most six top-level dimensions and three children beneath every later node. Models must consolidate related facts into `detail`/`sourceRefs`, not make one node per report bullet, datum, paragraph, or reference. An unfinished dedicated `mindmap-json` fence (and compatible historical `mindmap` fence) remains an ordinary streaming code block; only the completed, valid object lazy-renders as a compact left-to-right message card. Clicking a parent node, or its +/− control at the outgoing connection point, expands/collapses its child tree and the canvas uses an explicit visible edge stroke. The full dialog keeps its close and view tools in the top-right and a layout-derived all-map preview in the bottom-right; the message preview also permits pan, wheel zoom, and zoom/reset controls. The message card's icon-only download control opens PNG/SVG/JSON choices. Generic JSON never invokes this renderer. Ambiguity defaults to inline output. Once the HTML gate passes, delivery is mandatory as a sandbox-generated `/mnt/user-data/outputs/*.html` artifact opened with `present_files`; a full inline `html` fence is forbidden as a substitute. The shared frontend `MarkdownContent` keeps its exact legacy branch when no map-shaped fence exists; dedicated `map-json` fences and complete generic `json` fences that contain map fields and pass the full map schema are validated and lazy-rendered from `components/workspace/visualizations/`. Map/loading/error cards use the full message-content width so their edges align with the step panel. Non-map and incomplete generic JSON retain the ordinary code-block path.
|
||||
|
||||
HTML artifacts are single-page by default. Omitted `preferred_page_count` is normalized to `1` during direct validation. A count greater than one is allowed only when the current user directly asks for multiple pages or gives a page count; content length and report chapter count never imply multi-page intent. Explicit multi-page intent without a count defaults to a three-page lightweight portal. It is one artifact HTML, not a folder of linked pages: `<main class="pages">` contains logical `.page` sections with unique `id`/`data-page`, and `href="#page-..."` navigation works in the sandboxed Blob preview through the injected anchor handler. Its validator requires the local Bootstrap/Alpine assets, `.pages`, multiple `data-page` sections and real anchor navigation: every actual `<a href="#page-...">` and every script-set `location.hash` must match a unique logical-page id, and each page must be reachable; comments or `#overview`/`#stations` data-page values cannot spoof this check. `MultiPageHtmlMiddleware` repeats that structural inspection at the `present_files` boundary, so a model that skips validate or writes custom tabs cannot expose a nonconforming portal. A user-uploaded background goes only on the requested logical `.page` in this mode (home page when unspecified), not on every page. When a template is explicitly requested, prepare strictly excludes profiles from the wrong page-count mode, persists `requested_page_count` and `page_mode`, and validate reloads them. Validation rejects multiple detected page views (`data-page`, tabpanels, `.report-page`, or `.view`) in single-page mode, or a single view when multi-page was explicitly requested.
|
||||
|
||||
@@ -130,7 +357,7 @@ For an explicit multi-page HTML request without an explicit template request, `M
|
||||
|
||||
### Generated HTML artifact visualization runtimes
|
||||
|
||||
The same normal lead-agent prompt includes `_HTML_ARTIFACT_VISUALIZATION_SECTION`, but its contents are executable only after the explicit HTML-intent gate above passes; visualization content alone never implies this intent. New generated HTML explicitly loads browser distributions from Gateway `/api/public/html-runtime/`: `bootstrap.min.css`, `alpine.min.js`, `bootstrap.bundle.min.js`, `g6.min.js`, `echarts.min.js`, and then `deerflow-runtime.js`; it can also load the packaged country/region map catalog: `china-provinces`, `us-states`, `japan-prefectures`, and `taiwan-counties`. Bootstrap is the responsive layout/modal layer and Alpine is the small state/list-rendering layer; jQuery, React, Vue, Babel and public CDNs are not part of this artifact contract. The files are package resources under `packages/harness/deerflow/html_runtime/assets/` and ship with the harness wheel, so HTML execution does not depend on frontend Vite/node_modules assets or a public CDN. `system_settings.json -> html_map.engine` (`GET|PUT /api/system-settings/html-map`, admin write) chooses `echarts` (default, local GeoJSON `data-df-geo`) or `leaflet` (地图网页, `data-df-map`). `config.yaml -> html_runtime.map_tile_url` is injected into `deerflow-runtime.js` as `mapTileUrl` and is used only by the Leaflet engine. Both initializers stay in the runtime so already generated pages keep rendering after the switch. The next ordinary Q&A prompt and `page_report_template` validation follow the selected engine. `config.yaml -> html_runtime.base_url` is the browser-reachable absolute runtime base written into generated source (localhost default; deployments must replace it with the real Gateway origin). Flowcharts use paired `data-df-flow` JSON blocks and AntV G6, not Mermaid; charts use `data-df-chart`; maps use ECharts Canvas-backed `data-df-geo` with the local GeoJSON catalog. `data-df-chart` must be strict JSON: no JavaScript callbacks or function-valued colors, with literal colors/data-item styles and string tooltip formatters instead. Dark HTML must set `data-df-theme="dark"` and explicitly give ECharts global/title/legend/axis/visible-series labels readable light text, including `nameTextStyle` for named axes. Map `description` values are plain text because the runtime escapes markup; click-to-filter or page-switch behavior listens to `deerflow:geo-map-click` and uses a valid `#page-...` anchor. `deerflow-runtime.js` waits until a hidden logical page has a real layout before creating ECharts and tracks later container changes with `ResizeObserver`, so map/chart canvases never depend on a browser-window resize for their first visible render. It initializes all three contracts, opens a detail popover for described regions/markers, and exposes catalog URLs through its public dynamic response. Map large screens use a centered map stage with narrow side rails and bottom summaries, rather than cards layered across a full-page map background. Frontend runtime replacement is bypassed whenever this explicit Gateway runtime is present. This remains separate from ordinary message rendering; ordinary `map-json` keeps its existing frontend tile renderer and `VITE_MAP_TILE_URL` configuration, and ordinary `flow-json`/`chart-json` must not be reintroduced.
|
||||
The same normal lead-agent prompt includes `_HTML_ARTIFACT_VISUALIZATION_SECTION`, but its contents are executable only after the explicit HTML-intent gate above passes; visualization content alone never implies this intent. New generated HTML explicitly loads browser distributions from Gateway `/api/public/html-runtime/`: `bootstrap.min.css`, `alpine.min.js`, `bootstrap.bundle.min.js`, `g6.min.js`, `echarts.min.js`, and then `deerflow-runtime.js`; it can also load the packaged country/region map catalog: `china-provinces`, `us-states`, `japan-prefectures`, and `taiwan-counties`. Bootstrap is the responsive layout/modal layer and Alpine is the small state/list-rendering layer; jQuery, React, Vue, Babel and public CDNs are not part of this artifact contract. The files are package resources under `packages/harness/deerflow/html_runtime/assets/` and ship with the harness wheel, so HTML execution does not depend on frontend Vite/node_modules assets or a public CDN. `system_settings.json -> html_map.engine` (`GET|PUT /api/system-settings/html-map`, admin write) chooses `echarts` (default, local GeoJSON `data-df-geo`) or `leaflet` (地图网页, `data-df-map`). The map tile URL saved in the settings dialog (`system_settings.json -> html_runtime.map_tile_url`, `GET|PUT /api/system-settings/html-runtime`) is injected into `deerflow-runtime.js` as `mapTileUrl` and is used only by the Leaflet engine. Both initializers stay in the runtime so already generated pages keep rendering after the switch. The next ordinary Q&A prompt and `page_report_template` validation follow the selected engine. The runtime base URL saved in the same settings block is the browser-reachable absolute address written into generated source. An empty value falls back to an older `config.yaml` entry and then `http://localhost:8001/api/public/html-runtime`. Flowcharts use paired `data-df-flow` JSON blocks and AntV G6, not Mermaid; charts use `data-df-chart`; maps use ECharts Canvas-backed `data-df-geo` with the local GeoJSON catalog. `data-df-chart` must be strict JSON: no JavaScript callbacks or function-valued colors, with literal colors/data-item styles and string tooltip formatters instead. Dark HTML must set `data-df-theme="dark"` and explicitly give ECharts global/title/legend/axis/visible-series labels readable light text, including `nameTextStyle` for named axes. Map `description` values are plain text because the runtime escapes markup; click-to-filter or page-switch behavior listens to `deerflow:geo-map-click` and uses a valid `#page-...` anchor. `deerflow-runtime.js` waits until a hidden logical page has a real layout before creating ECharts and tracks later container changes with `ResizeObserver`, so map/chart canvases never depend on a browser-window resize for their first visible render. It initializes all three contracts, opens a detail popover for described regions/markers, and exposes catalog URLs through its public dynamic response. Map large screens use a centered map stage with narrow side rails and bottom summaries, rather than cards layered across a full-page map background. Frontend runtime replacement is bypassed whenever this explicit Gateway runtime is present. This remains separate from ordinary message rendering; ordinary `map-json` keeps its existing frontend tile renderer and `VITE_MAP_TILE_URL` configuration, and ordinary `flow-json`/`chart-json` must not be reintroduced.
|
||||
|
||||
User-uploaded raster images can be used as generated-HTML backgrounds through signed, unauthenticated `GET /api/public/html-images/{token}` URLs. `UploadsMiddleware` mints one URL per uploaded `.jpg/.jpeg/.png/.webp/.gif/.bmp` and injects it into `<uploaded_files>` as `Public URL`; the HTML contract requires that exact absolute URL in `background-image`, never a sandbox path, artifact API path, relative path, or base64 data URI. Single-page artifacts place it on `html/body`; the multi-page portal places it only on the requested logical page. Tokens are HMAC-signed, do not expire (the URL is written into durable HTML), and only grant read access to one file under the thread `uploads/` or `outputs/` directory. Engine: `deerflow.html_runtime.public_images`; router: `app/gateway/routers/html_images.py`; tests: `tests/test_html_images.py`.
|
||||
|
||||
@@ -303,6 +530,28 @@ commands.py` and `tests/test_position_roundtable_concurrency.py` cover the
|
||||
state transitions; `tests/test_roundtable_postgres_concurrency.py` is the real
|
||||
PostgreSQL acceptance suite (set `DEERFLOW_POSTGRES_CONCURRENCY_TEST_URL`).
|
||||
|
||||
Step-2 roundtable rooms share one deployment-wide cap,
|
||||
``system_settings.json -> roundtable_room_limit`` (default enabled, 3 rooms).
|
||||
``GET/PUT /api/system-settings/roundtable-room-limit`` is the admin switch.
|
||||
Occupancy is summed across **every account**: durable foreground leases keyed
|
||||
by a hidden per-live-execution ``roomSessionId``, in-flight LangGraph runs on
|
||||
threads stamped with the corresponding ``roundtable_room_key``, and executing
|
||||
background jobs (``queued`` / ``running`` / ``cancel_requested`` with a live
|
||||
worker lease or a recent update). Different pages running the same saved draft
|
||||
have different session ids and therefore take different rooms; all seats and
|
||||
Step 3 calls within one live execution share its key. Run-manager inspection
|
||||
is reconciliation-only: when a durable lease/job already owns the same
|
||||
draft+user/task scope, stale or overlapping run keys in that scope are not
|
||||
counted again. Merely opening the page,
|
||||
viewing saved history, paused ``awaiting_input`` jobs, stale historical rows,
|
||||
and Step-1 intent threads do not occupy a slot. Foreground-to-background
|
||||
handoff replaces the foreground lease with the durable job occupancy. There is
|
||||
no per-user pool. Admission is serialized by a database gate row, and the cap
|
||||
is enforced at acquire, init, every model-producing run, and background-job
|
||||
start. Full pools return HTTP 409; an enabled gate with missing storage or a
|
||||
missing session id fails closed. The switch applies on the next admission
|
||||
without a restart. Position and eight-department roundtables are not counted.
|
||||
|
||||
Roundtable backend jobs use a database unique active-dedupe key plus a leased
|
||||
dispatcher, never process-local start deduplication. Cancellation is two
|
||||
phase (`cancel_requested` then `cancelled`): the lease owner’s watcher normally
|
||||
@@ -449,6 +698,8 @@ from deerflow.config import get_app_config
|
||||
|
||||
### Agent System
|
||||
|
||||
**Gallery chat link.** An agent's `config.yaml` may store `chat_link` (`url`, `open_mode` of `blank`/`self`, and light-app-shaped `route_params`). The gallery 「对话」 button opens that resolved URL; an empty url keeps the in-app chat. The list endpoint reads the mirrored `agent_extras.chat_link` column (nullable JSON, added by the startup column sweep) instead of parsing every config file. Create/update and directory backfill write the mirror; a skills-only `set_extras` leaves an existing link in place.
|
||||
|
||||
**Lead Agent** (`packages/harness/deerflow/agents/lead_agent/agent.py`):
|
||||
- Entry point: `make_lead_agent(config: RunnableConfig)` registered in `langgraph.json`
|
||||
- Dynamic model selection via `create_chat_model()` with thinking/vision support
|
||||
@@ -463,6 +714,8 @@ from deerflow.config import get_app_config
|
||||
|
||||
**Strategy-selection functional agents.** `app/gateway/routers/_strategy_selection_seed.py` seeds nine stable folder-backed agents from `_strategy_selection_seed_assets/` before `_sync_legacy_agents`: intent, governance/diplomacy/economy/security/public-opinion/law-policy angles, judge, and solution editor. Seeding is missing-directory-only and must never overwrite administrator changes. The global `strategy_agent_roles` registry classifies them independently of projects: intent/judge/solution are fixed, unique roles that may be configured but never added, disabled, reordered, or deleted; angle roles are a shared, ordered catalog that administrators may add, edit, disable (soft delete), restore, and reorder through `/api/strategy-agents`. Public `GET /api/strategy-agents` returns enabled identity/description metadata only; `include_disabled=true` or `include_config=true` is admin-only. Admin writes are `POST /api/strategy-agents/angles`, `PUT /api/strategy-agents/{id}`, and `DELETE /api/strategy-agents/{id}`; DELETE retains the folder-backed agent and role row to preserve historical references. Dynamic angles remain built-in folder-backed agents under the reserved `strategy-selection-angle-` id namespace, with `web` + `file:read`; a blank SOUL on create receives a safe role template. All strategy skill writes validate that names exist, are active, and are globally visible; generic agent CRUD cannot create or hard-delete reserved strategy IDs, and parallel execution rejects disabled roles. Angle configs reuse `policy-knowledge`; intent reuses `intent-clarification-policy`; the judge has no retrieval skill; the solution editor gets only `file:write`. Keep the SOUL contracts neutral and evidence-aware, with explicit de-escalation/civilian-protection framing and hard bans on operational combat guidance and targeted political manipulation. Tests: `tests/test_strategy_selection_seed.py`, `tests/test_strategy_agent_management.py`.
|
||||
|
||||
Only `strategy-selection-intent` has a clarification reply surface in the strategy workbench. `parallel_agents` must exclude `ask_clarification` for every other `strategy-selection-*` role and inject a direct-delivery guard: angle, judge, and solution stages use conservative assumptions, lower confidence/score where needed, list gaps under data limitations, and still finish in one pass. The judge brief repeats this rule at runtime because existing folder-backed agent SOUL files may be administrator-customized and are intentionally never overwritten. If a legacy backend nevertheless emits a judge clarification, the frontend treats it as a bounded repair attempt rather than asking the user to repeat the same manual run. Malformed or incomplete evaluation JSON receives the same bounded repair path; prior attempts are replaced in the live panel, never concatenated into duplicate final content.
|
||||
|
||||
`POST /api/parallel-agents/run/stream` must remain observable during silent skill calls. Each running seat emits an `agent_progress` heartbeat every five seconds with the latest translated tool/reasoning/body phase and monotonic elapsed seconds; an upstream iterator that produces no frame for 180 seconds is closed and that seat emits `agent_error`, while the other seats continue. Set the terminal flag before queuing `agent_done` / clarification so a late heartbeat cannot reopen a completed card. The strategy workbench renders the intent Q&A history, readable skill execution steps and live heartbeat on the affected agent card; judge and synthesis traces use the same step translation. Tests: `tests/test_roundtable_model_fallback.py` and frontend `src/strategy-selection/lib.test.ts`.
|
||||
|
||||
**ThreadState** (`packages/harness/deerflow/agents/thread_state.py`):
|
||||
@@ -593,7 +846,7 @@ FastAPI application on port 8001 with health checks at `GET|HEAD /health`, `/api
|
||||
| **Answer feedback** (`/api/threads/{id}/messages/{mid}/feedback`) | Message-scoped `PUT`/`DELETE`, `GET /feedback-context`, and batched `POST /api/threads/{id}/feedback-contexts`; persists excellent/poor ratings, six stable poor-reason codes and optional detail idempotently per user/thread/message/agent. Legacy run-scoped feedback routes remain compatible. Admin-only `GET /api/agents/{agent_id}/feedback-score` uses the project-wide `system_role == "admin"` rule (and preserves auth-disabled mode) and returns a Bayesian user-feedback answer-quality score, raw task-success rate, weighted response-quality dimensions and reason counts; it is explicitly not a complete agent-capability evaluation. |
|
||||
| **Runs** (`/api/runs`) | `POST /stream` - stateless run + SSE; `POST /wait` - stateless run + block; `GET /{rid}/messages` - paginated messages by run_id `{data, has_more}` (cursor: `after_seq`/`before_seq`); `GET /{rid}/feedback` - list feedback by run_id |
|
||||
| **Open Chat** (`/api/open/chat`) | **Unauthenticated** open Q&A entry for external systems (the `/api/open/` prefix is whitelisted in `auth_middleware._PUBLIC_PATH_PREFIXES` + CSRF-exempt in `csrf_middleware.should_check_csrf`). `POST /chat` `{message \| messages[], model_name?, agent_name?, agent_id?, thinking_enabled=false, strip_think=true, thread_id?, recursion_limit=100}` → resolves the agent by **Chinese display name** (`agent_name`, matched against each `.deer-flow/agents/*/config.yaml` `name` via `_resolve_agent_id`; explicit `agent_id` wins if given), builds the lead agent via `make_lead_agent` (so the chosen **agent's** SOUL/tools/skills all apply), `ainvoke`s it once (stateless — no checkpointer, no thread persistence; runs under the `"default"` user bucket), and returns **only the final** `{answer, model_name, agent_name, agent_id, thinking_enabled, thread_id}` (no intermediate tool-call/process). **`POST /chat/stream`** is the **SSE streaming variant** (same request body) for **long tasks** — the blocking `/chat` produces no downstream bytes for the whole run, so a front nginx `proxy_read_timeout` (default 60s) cuts it off with a **504**; the stream keeps the connection warm (a `: keepalive` SSE comment every `_HEARTBEAT_SECONDS=15`s via a background producer-task + queue, plus `event: delta` AI-text increments and `event: progress`), and ends with **`event: result`** carrying the same final `{answer, …}` JSON (or `event: error`). Clients only need to read `event: result`; intermediate events are ignorable. Both endpoints share `_build_run_context` (config/agent/thread). `GET /agents` lists `{name, id}` so callers know what `agent_name` to send. **Thinking off is reliable**: when `thinking_enabled=false` the route also sets `thinking_force_disabled=true` so the lead agent calls `create_chat_model(force_disable_thinking=True)`, which injects **three** OpenAI-compatible "thinking off" shapes into `extra_body` — top-level `enable_thinking=false` (**SiliconFlow / DashScope hybrid models** like `deepseek-ai/DeepSeek-V4-*` on `api.siliconflow.cn` — the knob the nested `chat_template_kwargs` shape does NOT reach), `chat_template_kwargs.enable_thinking=false` (vLLM/Qwen native), and `thinking.type="disabled"` (Zhipu GLM) — even for models that only declare `supports_thinking: true`. For the **web-UI path** (which sends `thinking_enabled=false` but NOT `force_disable`), SiliconFlow models need `when_thinking_disabled: {extra_body: {enable_thinking: false}}` in their `config.yaml` entry (added to `deepseek-ai/DeepSeek-V4-Flash`) for thinking-off to take effect. Sets `is_scheduled_run=true` internally to skip the auto-title LLM call. **Generated-file return (`include_files=true`, default on)**: an agent that produces its report as a file (writes to `/mnt/user-data/outputs/` + `present_files`) returns only a short summary in `answer`; to recover the real deliverable both endpoints now also return `files: [{name, virtual_path, encoding(text|base64), content}]` — `_collect_output_files` scans the run's `state['thread_data']['outputs_path']` (the open thread is stateless so artifacts can't be fetched by thread, but the endpoint runs server-side and the sandbox already wrote the files to disk), text decoded as utf-8 else base64, capped by `_MAX_OUTPUT_FILES=20` / `_MAX_OUTPUT_FILE_BYTES=5MB`. `/chat` puts `files` on the response; `/chat/stream` puts it in the `event: result` payload. A run with no answer **and** no files now 502s ("…文本答案或文件"). Router `app/gateway/routers/open_chat.py`; standalone caller `test_open_chat_api.py` (repo root, streaming by default); tests `tests/test_open_chat.py`. |
|
||||
| **Open 3qfx Ask** (`/api/open/3qfx/ask`) | **Unauthenticated** open entry that, given just a **任务id**, directly kicks off the 3qfx(三情分析)「第一次问答」as a **real persisted conversation** (sibling of `/api/open/chat`; same `/api/open/` auth-whitelist + CSRF-exempt). `POST /ask` `{task_id, message?, model_name?, thinking_enabled=false}` → (1) resolves the `3Q` agent from `app.state.business_mapping_store.get_mapping("3Q")` (the **same** agent the frontend deep-link `goPath=3qfx` uses; 503 if unconfigured); (2) **fetches the task detail server-side** (consumer `cop-task-detail`) and builds the opening text **identical to the frontend** `buildOpeningText` 3qfx branch — `任务名称:…\n任务内容:…\n任务id(task_id)为{task_id}` (an explicit `message` overrides and **skips** the fetch); (3) creates a fresh thread tagged `metadata.{taskId, agent_id}` (the exact filter `useTaskThreads` uses, and `metadata.taskId` makes it task-deeplink-shared / creator-path-resolved) and **starts a background persisted run** via `start_run(..., enforce_access=False)` (writes the checkpointer, runs under the `"default"` bucket); (4) **returns immediately** `{thread_id, run_id, agent_id, status, opening_text}` — the caller just needs "started", then watches the run in the 3qfx 任务工作区 conversation list. `start_run` gained a keyword-only `enforce_access` (default `True`); the open entry passes `False` so the admin-configured agent isn't rejected by the per-(missing)-login visibility check. **Consumer fetch config** lives in `config.yaml → task_deeplink`: `cop_task_detail_url` (full URL, appends `?taskId=`), `cop_task_detail_authorization` (default `Bearer admin`), `cop_task_detail_test` + `mock.cop_task_detail` (offline fake when the upstream is unreachable — on in this deployment). Router `app/gateway/routers/open_3qfx.py`; tests `tests/test_open_3qfx.py`. |
|
||||
| **Open 3qfx Ask** (`/api/open/3qfx/ask`) | **Unauthenticated** open entry that, given just a **任务id**, directly kicks off the 3qfx(三情分析)「第一次问答」as a **real persisted conversation** (sibling of `/api/open/chat`; same `/api/open/` auth-whitelist + CSRF-exempt). `POST /ask` `{task_id, message?, model_name?, thinking_enabled=false}` → (1) resolves the `3Q` agent from `app.state.business_mapping_store.get_mapping("3Q")` (the **same** agent the frontend deep-link `goPath=3qfx` uses; 503 if unconfigured); (2) **fetches the task detail server-side** (consumer `cop-task-detail`) and builds the opening text **identical to the frontend** `buildOpeningText` 3qfx branch — `任务名称:…\n任务内容:…\n任务类型:…\n任务id(task_id)为{task_id}`(`taskType` 1/2/3 → 打击/塑势/防御,无法识别则省略) (an explicit `message` overrides and **skips** the fetch); (3) creates a fresh thread tagged `metadata.{taskId, agent_id}` (the exact filter `useTaskThreads` uses, and `metadata.taskId` makes it task-deeplink-shared / creator-path-resolved) and **starts a background persisted run** via `start_run(..., enforce_access=False)` (writes the checkpointer, runs under the `"default"` bucket); (4) **returns immediately** `{thread_id, run_id, agent_id, status, opening_text}` — the caller just needs "started", then watches the run in the 3qfx 任务工作区 conversation list. `start_run` gained a keyword-only `enforce_access` (default `True`); the open entry passes `False` so the admin-configured agent isn't rejected by the per-(missing)-login visibility check. **Consumer fetch config** lives in `config.yaml → task_deeplink`: `cop_task_detail_url` (full URL, appends `?taskId=`), `cop_task_detail_authorization` (default `Bearer admin`), `cop_task_detail_test` + `mock.cop_task_detail` (offline fake when the upstream is unreachable — on in this deployment). Router `app/gateway/routers/open_3qfx.py`; tests `tests/test_open_3qfx.py`. |
|
||||
| **Thread Shares** (`/api/thread-shares`) | Per-conversation sharing with two modes (`thread_shares.mode`). **`import`**: `POST /` `{thread_id, mode}` - create (or reuse) a revocable share code; `GET /{code}/preview` + `POST /{code}/import` - copy that conversation into the caller's account (fresh `thread_id` via `app/gateway/thread_copy.py` + new `threads_meta` row + `user-data` copy; dedup on `metadata.shared_origin`). **`view`**: a public, read-only static page ? see Public Shares below. Both modes: `GET /` - list my codes; `DELETE /{code}` - revoke. |
|
||||
| **Public Shares** (`/api/public/shares`) | **Unauthenticated** (the `/api/public/` prefix is whitelisted in `auth_middleware._PUBLIC_PATH_PREFIXES`). `GET /{code}` - read-only conversation snapshot (title + serialized messages + artifacts) for a `view`-mode share; messages are read straight from the latest checkpoint. `GET /{code}/files/{path}` - serves a file referenced by that conversation (images, Markdown, text, PDF, ?) so the shared page's links are openable; path resolution is confined to the owner's thread directory. Active content (`text/html`, `application/xhtml+xml`, `image/svg+xml`) is forced to download (`_public_share_disposition`); everything else is served inline. Missing/revoked/non-`view` codes all return 404. |
|
||||
| **Roundtable Drafts** (`/api/roundtable-drafts`) | **Strictly per-user** persistence for the roundtable-planning page (replaces the old browser `localStorage`). `GET /` - list drafts (lightweight meta — incl. `chain_title`, the business-chain name used this session, cheaply regex-extracted from the in-memory `step2` blob in `_draft_to_meta`/`_extract_chain_title`; null = no chain / 自由模式; surfaced on the 第一步「分析记录」history card + sidebar list instead of the old「第 X 步」); `POST /` - create; `GET/PUT/DELETE /{id}` - full read / partial update / delete (all owner-scoped). A draft = one full session (Step 1 intent + Step 2 roundtable), step snapshots stored as JSON in `PortableLongText`. Recommendation history bound to a draft: `GET /{id}/recommendations` - list newest-first; `POST /{id}/recommendations` - append one run (objective/status/rationale/picks/candidates). taskId 深链会商记录 are **not** here — they live in the separate `/api/roundtable-task-drafts` store (see below) so personal history never mixes with task records. Backed by `deerflow.persistence.roundtable_drafts` (tables `roundtable_drafts`, `roundtable_recommend_history`); store wired in `deps.py` as `app.state.roundtable_draft_store`. Tests: `tests/test_roundtable_task_drafts.py`. |
|
||||
@@ -660,6 +913,8 @@ The intranet client uses `trust_env=False` and defaults `intranet_verify_ssl=fal
|
||||
|
||||
The admin ????? is served from **precomputed per-day snapshots** in `admin_leaderboard_daily_stats` (keyed by `(stat_date, settings_hash)`), not by aggregating raw rows on every request. A background scheduler inside the Gateway (`_leaderboard_snapshot_scheduler_loop` in `app/gateway/app.py`, singleton file-locked) ticks every `ADMIN_LEADERBOARD_BACKGROUND_INTERVAL_SECONDS` (default 300s): it prewarms recent days, then builds any `queued`/`error`/stale-`running` day. `GET /leaderboard` merges the per-day snapshots for the requested range. **Today** is rebuilt whenever its snapshot is older than `ADMIN_LEADERBOARD_TODAY_TTL_SECONDS` (default 43200s = half a day, so the leaderboard refreshes ~twice daily); a past day, once `ready`, is treated as fresh forever.
|
||||
|
||||
**Recent-only admin audit policy.** The server enforces a rolling **30-calendar-day** window for administrator leaderboard, user analytics, user Q&A/thread audit, LLM/tool metric list/summary/export, and the user-list Q&A counter; URL parameters cannot widen it. The Q&A fallback that reads checkpoints is additionally capped, while indexed run summaries remain the primary audit source. Gateway startup also launches the singleton `admin_audit_retention` sweep: it deletes audit metric rows, terminal `runs` summaries (which contain compact question/answer audit fields and were also last updated before the cutoff), and leaderboard snapshots older than **90 days**. Cleanup is deliberately low-priority: after the default 5-minute startup delay, it runs every 15 minutes using 100-row committed transactions, round-robins each store (at most four batches per store), yields between commits, and stops starting new batches after an 8-second sweep budget; operators may tune these bounds with `ADMIN_AUDIT_RETENTION_*` environment variables. It deliberately does **not** delete conversation threads, checkpoints, or message history; those are user-content records governed by their own lifecycle.
|
||||
|
||||
Two on-demand/automatic refresh paths layer on top:
|
||||
- **Manual "????"** ? `POST /leaderboard/refresh-today` force-rebuilds *today's* snapshot **synchronously** (bounded by the same semaphore/timeout the scheduler uses), then returns the merged range. Lets an admin pull the latest real-time records into the stats without waiting for the TTL. Frontend: a "????" button on `AdminUserLeaderboardPage` ? `useRefreshTodayLeaderboard` (`core/admin/hooks.ts`), which primes + invalidates the `["admin-activity","leaderboard"]` query.
|
||||
- **Daily finalize of the previous day** ? once per day, on/after `ADMIN_LEADERBOARD_FINALIZE_HOUR` (local Beijing, default `1` = ?? 1 ?), the scheduler tick force-recomputes *yesterday* a single time so runs/tool-calls that only landed near midnight are fully captured. The decision is the pure `deerflow.persistence.admin_stats.finalize.should_finalize_previous_day` (status/generated_at/now/hour) ? the rebuild stamps `generated_at` to today, which self-debounces it for the rest of the day (no extra state file). Tests: `tests/test_leaderboard_finalize.py`.
|
||||
@@ -768,7 +1023,7 @@ The fixed-deep-blue wx full-chat route (`#/page/workspace/wx/chats/new`) hides t
|
||||
|
||||
### Business-mapping authorization
|
||||
|
||||
`/api/business-mapping` remains globally readable. Seeded rows include the main codes (3Q / 6BF / 7BF / 8BF / AFX) plus independent slots `3q-0…3q-6`, `6-1…6-4`, and `7-1…7-4` (empty agent/chain until configured on the 配置管理 page). Updates to 3Q, 6BF, 7BF, and those independent slots require `system_role == "admin"`; the 8BF public business-chain mapping is intentionally different and can be updated only by the trusted authenticated account whose email prefix is `lqq`. Do not rely on a username supplied in an external login URL. The frontend follows the same email-prefix rule when deciding whether to render the 8BF configuration button, while the router (`app/gateway/routers/business_mapping.py`) remains the enforcement point. Tests: `tests/test_business_mapping.py`.
|
||||
`/api/business-mapping` remains globally readable. Seeded rows include the main codes (3Q / 6BF / 7BF / 8BF / AFX) plus independent slots `3q-0…3q-6`, `6-1…6-4`, and `7-1…7-4` (empty agent/chain until configured on the 配置管理 page). `3q-1…3q-6` seed as `is_custom=true` / `mapping_type=自定义类型` so `goPath=3q-1` resolves as a custom deep link; the configuration page hides those six rows and defaults the type filter to 自定义类型. Updates to 3Q, 6BF, 7BF, and those independent slots require `system_role == "admin"`; the 8BF public business-chain mapping is intentionally different and can be updated only by the trusted authenticated account whose email prefix is `lqq`. Do not rely on a username supplied in an external login URL. The frontend follows the same email-prefix rule when deciding whether to render the 8BF configuration button, while the router (`app/gateway/routers/business_mapping.py`) remains the enforcement point. Tests: `tests/test_business_mapping.py`.
|
||||
|
||||
### Position-collaboration foundation
|
||||
|
||||
@@ -1357,10 +1612,11 @@ scheduler runs once per deployment. Per run:
|
||||
会商第二步(leader/seat)+ 第三步(summary/report/dashboard + 结构化抽取)单轮执行时,**某模型出问题就自动切到下一个模型继续**,并告知用户「『X』模型异常 → 已切换『Y』」。前台实时(`multi_agent.py`)与后台挂起(`engine.py` + `InProcessRoundtableGateway`)两条路径都覆盖。
|
||||
|
||||
- **可检测的模型失败**:`LLMErrorHandlingMiddleware` 在同一模型重试 3 次后**不抛异常**,而是返回兜底 `AIMessage` —— 现在给它打 `additional_kwargs[LLM_ERROR_MARKER="deerflow_llm_error"]=reason`(model_dump 保留,经 checkpointer 序列化仍在)。`is_llm_error_message(msg)` 据标记(回退按已知兜底文案正文)识别,返回 reason。
|
||||
- **候选模型链**:`app/gateway/roundtable_model_fallback.py::fallback_model_chain(current)` 按 `config.yaml models[]` 顺序给出 `[当前模型, …其余]`(去重保序)。`reason_text(reason)` 翻译成简短中文。总开关 env `ROUNDTABLE_MODEL_FALLBACK`(默认开,`0/false/off` 关 → 单模型旧行为);`ROUNDTABLE_MODEL_FALLBACK_MAX`(0=试完全部)。
|
||||
- **触发换模型的情形**:① 单轮 run 抛异常(start_run 失败 / idle 卡死 → `timeout`,其余 `exception`);② 兜底错误文案(`is_llm_error_message`);③ 交付无效(仅 seat/leader;leader 还含「既无正文又无派活」)。席位**只认剥掉 `<think>` 后的可见正文**——思考-only / 空串 / 断流都不算已交付(`deerflow.agents.roundtable_orchestrator.delivery`:`visible_seat_text` / `classify_seat_delivery`)。前台 `_special_run` 对此 yield `status:"invalid_delivery"` 且不广播;`GET /last-ai-message` 回 `{delivered, content, length, invalid_reason}`(思考-only 标 `thinking_only`,总控席位清单标 ⚠️)。后台引擎只把可见正文写入 `deliveries`,续轮用 `build_continue_prompt`:有无效席位时强制重派,禁止声称「已完成交付」。Step3 单例(report/summary/dashboard/ingest/结构化)的产物是落盘文件,**不**把「正文为空」当失败,只对真模型错误换模型。
|
||||
- **候选模型链**:保留真正模型故障后的自动换模,但“仅有思考 / 暂时无正文 / 空交付”不再判作模型故障,绝不据此切换用户选择的模型。`app/gateway/roundtable_model_fallback.py::fallback_model_chain(current)` 按 `config.yaml models[]` 顺序给出 `[当前模型, …其余]`(去重保序)。`reason_text(reason)` 翻译成简短中文。总开关 env `ROUNDTABLE_MODEL_FALLBACK`(默认开,`0/false/off` 关);`ROUNDTABLE_MODEL_FALLBACK_MAX`(0=试完全部)。
|
||||
- **触发换模型的情形**:① 单轮 run 抛异常(start_run 失败 / idle 卡死 → `timeout`,其余 `exception`);② 兜底错误文案(`is_llm_error_message`)。交付无效、仅有思考、空串以及普通总控轮既无正文又无派活**均不触发换模**;唯一例外是无工具的最终收口请求已经完整结束仍无正文,此时结果客观不可交付,才继续自动换下一个模型。席位仍只认剥掉 `<think>` 后的可见正文(`deerflow.agents.roundtable_orchestrator.delivery`:`visible_seat_text` / `classify_seat_delivery`):前台 `_special_run` 对无效交付 yield `status:"invalid_delivery"` 且不广播;`GET /last-ai-message` 回 `{delivered, content, length, invalid_reason}`(思考-only 标 `thinking_only`,总控席位清单标 ⚠️)。后台引擎只把可见正文写入 `deliveries`,无效席位继续走既有未交付处理,不切换模型。Step3 单例(report/summary/dashboard/ingest/结构化)的产物是落盘文件,同样只对真模型错误换模型。
|
||||
- **Step2 总控工具与可配置收口轮数**:普通总控使用真正白名单,只允许 `agent_orchestration` / `ask_clarification`;`write_file`、`present_files`、技能目录与文件/命令工具均不可见。`system_settings.json -> roundtable_room_limit.max_cycles`(默认 8,范围 2..100;管理员在设置弹窗「会商并发与收口」中修改)控制 recommend 模式到第几轮切换为显式 `finalize_only`:空工具白名单、注入已有有效席位全文、要求直接输出最终总结(失败席位如实标缺口),不能表现为“已停止”。一次会商启动时锁定当时值,运行中修改从下一次研讨生效;前台实时会商与后台作业读取同一设置。同一页的「会商文件流式兼容」(`write_file_compatibility_enabled`,默认关) 只作用于会商总控、席位、第三步和后台作业:全局文件流式兼容开着时会商一并使用;全局关着而本开关开着时,只有这些会商 run 改走 `buffered_gateway`。普通对话不读这个开关。chain / dag 的最终总结仍在既定席位完成后走同一纯正文路径;LangGraph 单轮 recursion limit 仍为 1000。
|
||||
- **后台**:`InProcessRoundtableGateway._run_turn_with_fallback(...)` 包住 `_run_agent_turn`,逐候选模型试;换模型落 `model_switch` 诊断、全失败落 `model_fallback_exhausted`。`run_leader/run_seat/run_report/run_summary/run_dashboard_data` 全走它(report/summary/dashboard 的产物自愈重试保留,并记住已切到的可用模型用于后续续写轮)。各 Turn/Result 带回 `switches: list[ModelSwitch]`,引擎 `_emit_switches` → `JobProgressWriter.model_switch(...)` 往研讨时间线插一条 `role="system"` 系统提示气泡。
|
||||
- **前台**:`_special_run`/`_leader_run` stream 收尾后检测失败 → yield `{status:"model_switch", role, agent_name, failed_model, next_model, reason}` 帧并用下一个模型重新流式;终态帧带 `model_used`。结构化抽取 `flow_extract.py` 的 round1 `model.astream` 也包了同款换模型循环(emit `event:"model_switch"` 帧)。
|
||||
- **前台**:`_special_run`/`_leader_run` stream 收尾后仅检测明确的模型错误标记;命中才 yield `{status:"model_switch", role, agent_name, failed_model, next_model, reason}` 帧并用下一个模型重新流式,思考-only/空正文不进入该分支;终态帧带 `model_used`。结构化抽取 `flow_extract.py` 的 round1 `model.astream` 也包了同款换模型循环(emit `event:"model_switch"` 帧)。
|
||||
- **Step2 单席位隔离**:`engine._run_recommend/_run_chain/_run_dag` 的每个席位执行包 try/except —— 一个席位彻底失败(落 `seat_failed` 诊断 + 置空交付)不拖垮整批/整链/整个作业。
|
||||
- **前端**:`multi-agent.ts` 解析 `model_switch` 帧 → `onModelSwitch` 回调(清空失败尝试的半截气泡 + 累计的 tool_call delta,phase 退回 connected);`useStep2Orchestration` 弹 toast + 时间线插「系统提示」气泡;`useStep3Summary/Report/Dashboard` + `flow-extract.ts` 重置对应气泡并在步骤卡留可见提示;后台路径经 `job-live.ts::toStep2Dialogue` 把 `role="system"` 渲染成 amber「系统提示」气泡。
|
||||
- Tests: `tests/test_roundtable_model_fallback.py`。
|
||||
@@ -1502,6 +1758,29 @@ from the selected space names and hides generic recommendations while these
|
||||
questions are present. Design and remaining graph/profile phases are in
|
||||
`docs/KNOWLEDGE_SPACE_RETRIEVAL_AND_WIKI_EXPANSION_ZH.md`.
|
||||
|
||||
**Knowledge-base research bookshelf:** `/page/workspace/knowledge/:id/research`
|
||||
is ordinary Lead Agent Q&A constrained to one existing knowledge base; it does
|
||||
not invoke or reuse the legacy Deep Research workflow. Any account with read
|
||||
access may create a private session/draft, while only the existing KB writer
|
||||
may publish or overwrite its Markdown Wiki article. Published snapshots follow
|
||||
the KB's read permission, but the session/thread/draft and the continuation
|
||||
link remain creator-only (administrators do not inherit another user's private
|
||||
chat). Every run resolves the immutable thread binding from
|
||||
`knowledge_research_sessions` (registered on `deerflow.persistence.models` so
|
||||
startup `create_all` builds the table; `get_by_thread` treats a missing table
|
||||
as no binding so ordinary `/runs/stream` is not aborted), reauthorizes KB read access, accepts exactly one
|
||||
text-only human turn, forces classic single-KB retrieval and disables tools,
|
||||
memory, prompt prefixes, planning and subagents; fixed-answer shortcuts are
|
||||
bypassed. Saving manually upserts a stable
|
||||
`knowledge-research/{publication_id}` Wiki slug under the reserved root
|
||||
「深度研究」 without enabling automatic Wiki generation, uses a saved hash for
|
||||
optimistic overwrite checks and idempotent retries, and keeps the editable
|
||||
publication title separate from the Q&A title. REST endpoints live in
|
||||
`app/gateway/routers/llmwiki.py`, runtime enforcement in
|
||||
`app/gateway/knowledge_research.py`, persistence/migration in
|
||||
`deerflow.persistence.knowledge_research` / `20261022_01`, and regression
|
||||
coverage in `tests/test_knowledge_research.py`.
|
||||
|
||||
**Embedding ownership:** WeKnora's per-knowledge-base `embedding_model_id`
|
||||
indexes uploaded source documents; `llmwiki.local_wiki_index.embedding` indexes
|
||||
the generated Wiki-page mirror inside DeerFlow. Do not couple these two model
|
||||
@@ -1742,7 +2021,10 @@ Automatic conversation summarization when approaching token limits:
|
||||
`conversation_context.recent_turn_window_enabled` is a deployment-wide,
|
||||
default-off alternative to summarization. Administrators manage it in
|
||||
「设置 → 上下文管理」 through
|
||||
`GET/PUT /api/system-settings/conversation-context`. When enabled, each lead
|
||||
`GET/PUT /api/system-settings/conversation-context`. The same block stores
|
||||
`max_conversation_rounds` (default 20, range 1–50), which `ChatPage` uses as
|
||||
the maximum number of real user turns in one ordinary chat; missing values in
|
||||
older settings files retain the 20-turn default. When enabled, each lead
|
||||
agent model request retains only the current turn and the two preceding real
|
||||
user turns, together with their assistant/tool continuations and any hidden
|
||||
context attached to the retained boundary. The operation is request-only:
|
||||
@@ -2641,3 +2923,24 @@ calls. The active sandbox skill mirror must be synchronized with the canonical c
|
||||
Gateway detail writes emitted by `fs-collect` and `fs-detail-submit` log a token-masked request and
|
||||
the corresponding response so an interrupted chain can be distinguished from a failed persistence
|
||||
call without exposing the gateway bearer token.
|
||||
|
||||
### Task-detail custom skill password authentication (2026-10-26)
|
||||
|
||||
`skills/custom/task-context` and `skills/custom/task-detail-update` authenticate detail reads/writes
|
||||
with a short-lived token minted by `POST /api/v1/auth/login/local`, using the runtime-only
|
||||
`TASK_DETAIL_AGENT_USERNAME` / `TASK_DETAIL_AGENT_PASSWORD` pair and `TASK_DETAIL_GATEWAY_URL`.
|
||||
They must never use general `DEERFLOW_AUTH_TOKEN` or `TOOLS_TOKEN`, nor emit the password or JWT in
|
||||
an input, trace, or model-visible output. `/api/parallel-agents/auth/token` is only a legacy fallback:
|
||||
it follows `sso_enabled` and returns 403 when external SSO is disabled. Password-login failures other
|
||||
than the policy 403 must fail closed; do not fall back to identity-only token minting. Keep the
|
||||
canonical custom directories and `.deer-flow/.sandbox-online-skills` mirror synchronized through the
|
||||
custom-skill reconciliation flow. `scripts/setup_biz_detail_assistant.py` logs in through the same
|
||||
formal endpoint and accepts `DEERFLOW_SETUP_BASE`, `DEERFLOW_SETUP_USERNAME`,
|
||||
`DEERFLOW_SETUP_PASSWORD` for deployment overrides; it may fall back to the legacy username route
|
||||
only for the formal login policy 403, never for a credential or rate-limit failure.
|
||||
|
||||
`biz-detail-assistant` must retain the native `ask_clarification` tool. Do not special-case it by
|
||||
adding that tool to `requested_excluded_tools`, and do not mount `DetailConfirmMiddleware` to forge
|
||||
confirmation cards from a plain model response: those cards are not genuine model tool calls and break
|
||||
the standard pause/resume interaction. The final `ClarificationMiddleware` is the single authority for
|
||||
all missing-information, scope, approach and risk-confirmation questions.
|
||||
Reference in new issue
Block a user