sync update from LAN repo (source dhk-1 @ 3b1ec5b4)
This commit is contained in:
1 parent
da6521b6cb
commit
264e2200d6
995 files changed
+148105
-13399
No files matched your search
@@ -1,12 +1,353 @@
|
||||
# CLAUDE.md
|
||||
|
||||
## Cold agent construction (2026-10-30)
|
||||
|
||||
`runtime/runs/worker.py` dispatches opted-in synchronous factories through
|
||||
`build_agent_with_timing`, backed by `utils/background_init.py`'s
|
||||
`run_background_init` and one shared `FileIOPool(1, 1, "agent-init")`.
|
||||
`make_lead_agent._deerflow_background_build = True` opts the built-in Lead Agent
|
||||
into background construction; unmarked custom factories retain their caller-loop
|
||||
contract. Serial admission prevents concurrent SDK cold imports and keeps waiting
|
||||
builds out of authentication's executor. The synchronous timing helper remains
|
||||
available. Model execution, checkpoint/store attachment and streaming stay in the
|
||||
original run task; no graph is shared or cached across runs/users.
|
||||
|
||||
Every async Lead Agent entry point uses this builder, including non-streaming
|
||||
and streaming open-chat endpoints and Workflow agent nodes. Do not call the
|
||||
synchronous factory directly on their event loop: it can block SDK import locks
|
||||
or deadlock while another background build awaits MCP discovery on that loop.
|
||||
Async tool consumers (AI writing skill agents and `task_tool`) use
|
||||
`get_available_tools_async`; deep-research capabilities and job execution also
|
||||
construct their material providers through `run_background_init`. Keep the
|
||||
common helper in `utils` with a lazy MCP import to avoid runtime/agent/tools
|
||||
import cycles. Synchronous client/provider APIs remain available.
|
||||
|
||||
Copy the caller's ContextVars into construction and restore the successfully built
|
||||
context before streaming, including a deferred-tool registry reset to `None`.
|
||||
The `mcp_initialization_loop` context manager forwards lazy MCP discovery from the
|
||||
worker to the owner event loop; exit it before copying back context. MCP cache
|
||||
invalidation and legacy synchronous callers retain their existing semantics.
|
||||
Cancellation drains the actual worker before releasing capacity, never starts the
|
||||
graph afterwards, and preserves terminal status/SSE cleanup. Check `abort_event`
|
||||
after construction as well as honoring task cancellation.
|
||||
|
||||
Regression: `packages/harness/tests/test_agent_build_responsiveness.py`,
|
||||
`test_agent_build_timing.py`, `test_mcp_background_build.py` and the run-finalization
|
||||
tests; `tests/test_async_agent_entrypoints.py` and
|
||||
`tests/test_async_tool_initialization.py` exercise overlapping HTTP/workflow/tool
|
||||
initializers. The HTTP/workflow cases run in subprocesses so an event-loop
|
||||
deadlock fails with a bounded timeout. Cold profiling found ~4.7 s in model SDK imports/client setup, compared with
|
||||
~14 ms in skill loading; moving only disk reads would not address this stall.
|
||||
|
||||
Local live cold acceptance (2026-10-30, official MiniMax-M2.7, tools enabled):
|
||||
the first build took 4.28 s while a second request received stream metadata in
|
||||
402 ms. During the conservative interior of the build window, 43 health/auth
|
||||
probes completed successfully; across all three conversations, health/auth
|
||||
P95 latency was 28/73 ms, with no event-loop-stall warning. Every answer ended
|
||||
normally and all synthetic threads were removed. Initialization/model latency
|
||||
still contributes to first-answer time; this change prevents blocking the server
|
||||
loop, rather than promising immediate model output.
|
||||
A separate live read-only tool check uploaded a random marker, allowed only
|
||||
`read_file`, and confirmed the tool result and final answer reproduced that marker
|
||||
without placing it in the prompt; the stream ended normally and the thread was removed.
|
||||
|
||||
Worker compatibility: no HTTP/SSE, task payload, auth or database contract changes.
|
||||
Running workers without reload retain old construction behavior; the new factory
|
||||
marker is harmless to old executors. `worker_pool/revision.py` hashes only the
|
||||
adapter directory and does not identify this harness patch. This change does not
|
||||
alter the controller's existing snapshot/ImageID-based drain-and-replace policy
|
||||
or `on_disconnect=cancel` behavior: restarting a production controller can still
|
||||
rotate idle workers or disconnect proxied streams. Do not infer uninterrupted
|
||||
worker operation from these process-local initialization changes. The live local
|
||||
acceptance instance has no enabled worker controller and does not touch remote
|
||||
workers. Existing worker routing/admission/storage/discovery/diagnostics/launcher
|
||||
compatibility regressions: 68 passed.
|
||||
|
||||
Final combined regression (2026-10-30): all 58 newly added responsiveness,
|
||||
cancellation, MCP-entrypoint and provider-lifecycle cases plus the 68 existing
|
||||
worker compatibility cases passed together (126 passed, no skips).
|
||||
The live recheck uploaded and converted 20 DOCX files while another conversation
|
||||
produced 43 visible text increments; all 172 concurrent authenticated probes
|
||||
succeeded (P95 105 ms, max 436 ms). A fresh-process cold build took 10.92 s while
|
||||
617 heartbeat samples continued (P95 scheduling delay 25 ms, max 146 ms).
|
||||
The existing backend and child workers were not restarted for this recheck;
|
||||
later entrypoint changes were verified through isolated HTTP/workflow tests.
|
||||
The full versioned backend/harness run completed with 3140 passed, 39 failed,
|
||||
4 skipped and 1 deselected. Against baseline `47cfed02`, 30 failures also failed
|
||||
on HEAD and the other 9 passed in isolated reruns on both versions; no current-only
|
||||
stable failure remained. The deselected legacy
|
||||
`test_cancelled_history_closes_iterator` hangs on both versions: its fake saver
|
||||
implements `alist`, while `limit=1` takes `aget_tuple`, so its unbounded entry
|
||||
event wait never completes. This is not a claim that the repository's entire
|
||||
test suite is green. Changed Python files introduced no Ruff diagnostics.
|
||||
|
||||
## Upload responsiveness (2026-10-30)
|
||||
|
||||
`app/gateway/routers/uploads.py` runs each staged upload transaction in a dedicated
|
||||
`FileIOPool(2, 2, "uploads")`, including 256 KiB spool copies, signature validation,
|
||||
optional synchronous conversion, sandbox acquisition/sync, commit, rollback and
|
||||
cleanup. Resolve the thread owner asynchronously before dispatch and propagate
|
||||
contextvars into workers. Waiting batches consume no worker from authentication,
|
||||
artifacts or event streaming. Per-event-loop directory locks serialize same-thread
|
||||
batches and listing/deletion through rollback/cleanup; keep them held until a
|
||||
cancelled worker stops. Listing/deletion use the artifact file pool off the loop.
|
||||
Cancellation signals a cooperative event, checks between file chunks and around
|
||||
commit/sync, and waits for real completion before multipart handles may close.
|
||||
The response contract and upload-policy enforcement remain unchanged.
|
||||
`sandbox/sandbox_provider.py` serializes initial singleton construction with
|
||||
reset/replacement/shutdown. Concurrent cold uploads must not create competing
|
||||
providers with independent sandbox reconciliation and idle-cleanup threads.
|
||||
Regression: `packages/harness/tests/test_sandbox_provider_concurrency.py`.
|
||||
|
||||
`file_conversion.py` also isolates its asynchronous conversion/extraction callers
|
||||
in a separate two-thread pool. PDF/Office parsing runs in a lazily initialized,
|
||||
two-process pool using `spawn` on Windows and Linux, so Python parser CPU work
|
||||
does not contend for the gateway GIL. Pass only file paths and converter choices;
|
||||
bound returned text inside the child and write sidecars in the owning parent
|
||||
worker. The global parser semaphore bounds sync/async callers together. A broken
|
||||
process pool fails the current conversion and is rebuilt on the next call.
|
||||
`FileIOPool.run_until_complete` provides non-abandoning cancellation for workers
|
||||
that own temporary input files; cancellation cannot close/delete an input still
|
||||
being parsed by a child process.
|
||||
Regression: `tests/test_upload_responsiveness.py`,
|
||||
`tests/test_conversion_responsiveness.py`, `tests/test_uploads_policy_enforcement.py`.
|
||||
|
||||
Local acceptance (Windows, Python 3.12, synthetic files via ASGI; no delay injected):
|
||||
20 x 900 KiB TXT uploads reduced first concurrent probe delay from 664/682 ms to
|
||||
3/4 ms. With real MarkItDown conversion, 20 x 800-paragraph DOCX uploads reduced
|
||||
maximum probe scheduling lag from 286/285 ms to 41/14 ms; all 20 sidecars matched
|
||||
the source text. The first spawned reader took 13.7 s to initialize/convert in
|
||||
this environment, while concurrent probe lag stayed below 16 ms. These are local
|
||||
responsiveness measurements, not production throughput guarantees. Upload,
|
||||
policy, conversion, large-file and runtime-contention regressions: 87 passed.
|
||||
Full-suite collection is separately blocked by the pre-existing missing
|
||||
`deerflow.models.global_prompt` import in `tests/test_global_prompt.py`.
|
||||
|
||||
## Thread metadata query performance and ORM deadlines (2026-10-22)
|
||||
|
||||
`ThreadMetaRepository._query_conditions` is shared by search/count: common
|
||||
SQLite scalar predicates and PG/MySQL string predicates plus system-thread
|
||||
exclusion run before SQL pagination/counting. Preserve AUTO owner resolution and
|
||||
the nonempty taskId owner bypass while applying agent_id/actionId together.
|
||||
`json_compat.JsonPredicate` binds values, preserves local missing/null and numeric
|
||||
equality on SQLite, and uses binary MySQL id equality and Unicode-safe blank
|
||||
seat detection. Nested/unusual-key filters and PG/MySQL numeric filters remain exact
|
||||
residual filters; openGauss retains its original path. Never drop an unsupported
|
||||
predicate or replace this with upstream's unadapted SQLite/PG-only compiler.
|
||||
|
||||
PG's omitted pool_recycle defaults to 300s (other backends remain 1800s);
|
||||
database.command_timeout is finite positive or null, default 30s. The shared
|
||||
pooled-engine kwargs are used on initial creation and auto-create retry. Only
|
||||
asyncpg receives command_timeout; this does not configure psycopg checkpointers.
|
||||
Regression entries: packages/harness/tests/test_thread_metadata_sql_filters.py,
|
||||
test_database_pool_deadlines.py and the existing test_thread_operation_timing.py.
|
||||
Master integration (2026-10-23) retains the maintained task/agent/action index
|
||||
columns as additional SQL narrowing before exact JSON predicates. Keep dedicated
|
||||
task deep-link card synchronization on every existing metadata mutation. Hydrated
|
||||
history still uses slim_display to bound oversized tool payloads without editing
|
||||
checkpoint contents; frontend retrieval-progress scopes remain isolated per turn.
|
||||
Thread POST defaults to metadata-only creation; only an explicit
|
||||
initialize_checkpoint=true writes an empty checkpoint. Never enqueue an empty
|
||||
checkpoint in the background: it could become the head after the real first run.
|
||||
Authorized metadata-only threads have empty state/history. Metadata GET defaults
|
||||
to the SQL row (running maps to busy); include_state=true and checkpoint-only
|
||||
legacy rows retain state reads. Owner decorators run before unscoped row reads.
|
||||
History explicitly uses root checkpoint_ns and closes alist before serialization;
|
||||
only the latest head carries full channel values.
|
||||
|
||||
The frontend uses BackgroundHistory as the SDK's external history adapter. Its
|
||||
post-stream mutate immediately captures streamed state with no checkpoint id;
|
||||
explicit end triggers background hydration. EOF/error still requires existing
|
||||
run confirmation/rejoin. Every async write is fenced by thread/run/epoch/request;
|
||||
new turns, navigation and unmount invalidate prior requests even if abort is
|
||||
ignored. Reject history missing the newest streamed turn, bound each request to
|
||||
15 seconds and retry completion sync once. Failure keeps the streamed snapshot.
|
||||
Normal explicit-end callbacks use the completed streamed snapshot before the
|
||||
next send can change page refs; background hydration never repeats them. Recovery
|
||||
may have missed final SSE frames, so its callback waits for guarded hydration or
|
||||
the retained snapshot fallback. New submits use checkpoint:null to follow the
|
||||
latest server head.
|
||||
|
||||
Auth's SQLiteUserRepository is a historical name: its injected session factory
|
||||
is the shared business ORM. With database.backend=mysql, login/auth/users and SQL
|
||||
business repositories use MySQL, independently of checkpoint configuration.
|
||||
LangGraph Store still follows checkpointer config and startup awaits runtime
|
||||
setup/migrations; do not claim total service/startup isolation from PG or migrate
|
||||
existing stores implicitly. See docs/THREAD_LATENCY_OPTIMIZATION_ZH.md for the audit.
|
||||
Additional regression entries: test_thread_lazy_checkpoint.py,
|
||||
tests/test_account_authentication.py and frontend pnpm test:thread-history.
|
||||
|
||||
## Multi-user contention controls (2026-10-22)
|
||||
|
||||
- `utils/file_io.py` separates bounded artifact IO (4 workers / 8 admitted) and
|
||||
event disk IO (2 / 4) from the default executor used by authentication.
|
||||
Contextvars propagate; cancellation cannot release admission until disk work
|
||||
actually stops. Heavy artifact listing, ZIP previews, sniffing and writes use it.
|
||||
- `runtime/runs/manager.py` sweeps completed live records after 300 seconds,
|
||||
200 candidates per pass. It requires the actual task to be done and verifies
|
||||
status/error/metadata/kwargs in the backing RunStore, updating a final snapshot
|
||||
when necessary. Failed persistence retains the record for later retry. It never
|
||||
deletes RunStore rows, checkpoints, events or stream buffers. `watch_completion`
|
||||
starts grace after finalization, and lifespan closes the sweeper. The live
|
||||
superseded fence survives a newer record's retirement; finalization never adds
|
||||
a business-database history read to its completion fence.
|
||||
- Run routes authorize the thread before unscoped persisted lookups, and verify
|
||||
that the loaded run belongs to that exact thread. Preserve private-owner 404s,
|
||||
taskId/system-thread sharing and existing action/agent partitions. Completed
|
||||
rejoin without a live bridge buffer emits end immediately; Last-Event-ID replay
|
||||
during the existing replay window remains untouched.
|
||||
- `run_events.backend=memory` is a compatibility alias for
|
||||
`SpoolingRunEventStore`, stored under `runtime_home()/run-events/events.sqlite3`.
|
||||
Per-thread sequence assignment and each batch commit are transactional; paged
|
||||
queries do not retain completed events in RAM. No automatic event/message
|
||||
deletion or truncation is introduced. The pure `MemoryRunEventStore` remains
|
||||
available for unit tests. Explicit db/JSONL formats are unchanged; JSONL disk
|
||||
operations now use the event executor and striped per-thread locks. Keep the
|
||||
spool and its WAL on the persistent local runtime volume. Do not reclassify it
|
||||
as a PG checkpoint or silently change explicit db configuration.
|
||||
- `runtime/fair_queue.py` provides bounded round-robin admission by user for the
|
||||
subagent scheduler (3 total / 2 per user) and background roundtables (4 / 2).
|
||||
Queued jobs consume no worker; subagent deadlines include queue time. Per-user
|
||||
pending capacity also prevents one user from exhausting the entire backlog.
|
||||
Actual coroutine completion, including cancellation before its first step,
|
||||
acknowledges slot release. Parent context/identity propagation stays intact.
|
||||
`RuntimeContentionConfig` defines optional budgets; limits apply per process
|
||||
and require restart. The older roundtable per-user environment override remains.
|
||||
- Both internal roundtable creation helpers only create metadata. Init does not
|
||||
append a synthetic seat message via loopback; `_leader_run` injects authoritative
|
||||
IDs on each real input, deriving them from the thread mapping for older clients
|
||||
without seat_meta. Background first-leader injection remains. Never schedule a
|
||||
late empty checkpoint write: it could overwrite real history.
|
||||
|
||||
Regression entries: `test_runtime_contention.py`, `test_completed_run_retention.py`,
|
||||
`test_thread_lazy_checkpoint.py`, `test_run_completion_timeout.py`,
|
||||
`tests/test_position_roundtable_sessions.py`, roundtable role/model/seat tests,
|
||||
artifact/auth/model concurrency tests and frontend `pnpm test:thread-history`.
|
||||
See `docs/MULTI_USER_CONTENTION_AUDIT_ZH.md` for measured scope and remaining
|
||||
model/SQL/checkpointer contention. Process-local run control still requires the
|
||||
current single-worker deployment; these queues do not enable multi-worker routing.
|
||||
|
||||
## Ordinary knowledge smart retrieval dual surface (2026-10-16)
|
||||
|
||||
`SmartReasoningRetrievalMiddleware` treats knowledge retrieval as an ordinary
|
||||
model-selected information tool inside the reasoning loop. A direct answer or
|
||||
skill-only turn is never followed by a forced search. Once the model decides it
|
||||
needs selected-library evidence, the first retrieval action is
|
||||
`ordinary_knowledge_search`; after reading those raw hits, the model decides
|
||||
whether Wiki search/read, regex verification, document reads, or another raw
|
||||
query is needed for a real evidence gap. Each model iteration executes at most
|
||||
one smart-retrieval call, and any content emitted beside that call is removed so
|
||||
the final answer can only follow the completed tool loop. The middleware
|
||||
deduplicates current-turn raw queries, regexes, Wiki
|
||||
patterns, `page_ref+cursor` reads and document pages; a new continuation cursor
|
||||
is a distinct action, while a repeated/non-advancing cursor terminates that
|
||||
page. If deduplication removes an empty duplicate-only response, the middleware
|
||||
makes one tool-free synthesis retry over existing evidence instead of returning
|
||||
a generic non-answer. The user only asks the question; no retrieval recipe is
|
||||
required in the message. Original document chunks win when Wiki and raw
|
||||
evidence conflict, while stable general knowledge may explain background if it
|
||||
is clearly distinguished from library facts and inference. Smart-mode answers
|
||||
do not require `[1]`/`[2]` markers or a numbered reference list. Classic
|
||||
retrieval does not mount this behavior.
|
||||
|
||||
Default WeKnora clients are cached per running event loop because they own an
|
||||
async authentication lock and a cached directory task; never restore a
|
||||
process-wide client cache. Smart-tool mapping authorization reads are routed to
|
||||
the SQL engine's owning loop before provider I/O. These two boundaries prevent
|
||||
healthy selected knowledge bases from being reported as `PROVIDER_UNAVAILABLE`
|
||||
when Gateway and LangGraph work run on different loops.
|
||||
|
||||
Wiki creation resolves its KnowledgeQA model only from WeKnora's database model
|
||||
records. It prefers an active explicit default and calls WeKnora's model debug
|
||||
endpoint before binding that model to `summary_model_id` and
|
||||
`wiki_config.synthesis_model_id`; `active` alone is not proof that the upstream
|
||||
provider still serves the model. A candidate returning `ok=false` may be skipped,
|
||||
but authentication, timeout, server, and response-contract failures remain service
|
||||
errors. Existing documents that already reached Wiki dead letters require an
|
||||
explicit reprocess after both model fields are repaired. Directory and focus
|
||||
instructions remain persisted in the knowledge base's database `wiki_config`.
|
||||
|
||||
`llmwiki_reasoning_tools._wiki_search_impl` applies the safe POSIX expression
|
||||
to normalized Wiki pages, but obtains bounded candidates from both the local
|
||||
database keyword index and the newest-page snapshot. A completed local mirror
|
||||
with no regex match probes the live WeKnora provider, which prevents newly
|
||||
generated Wiki pages from being hidden by a stale mirror when periodic sync is
|
||||
off. A 0/0 mirror is never considered complete. Provider `content`,
|
||||
`content_md`, and `markdown` bodies are normalized for both search and page
|
||||
read. Keep the live-provider zero-match fallback when changing mirror policy.
|
||||
|
||||
## Gongwen bounded delivery (2026-10-07)
|
||||
|
||||
Gongwen seed v3 ships `scripts/delivery.py` through the restricted
|
||||
`run_gongwen.py deliver` / `verify-file` commands. Deliver stages a new DOCX,
|
||||
runs draft or repair with one output verification, injects only the explicit
|
||||
elements JSON, then checks package/XML/body/element presence without treating
|
||||
the red header or imprint as body text. It publishes without replacing an
|
||||
existing output and returns ready/review_required/failed JSON plus the review.
|
||||
No renderer or external model is invoked; visual pagination and factual review
|
||||
are not claimed. Do not add automatic repair/check loops or ad-hoc Python to
|
||||
the workflow/SOUL prompts. P1/P2 warnings are not delivery blockers.
|
||||
|
||||
The vendored optimize command reuses its input check and returns full output
|
||||
review details; failed verification exits nonzero. Draft propagates its resolved
|
||||
document type into optimization. Gateway generate/repair reuse this invocation's
|
||||
review, with one check fallback for older custom skill versions. Do not reuse
|
||||
arbitrary client-supplied review files as trusted verification.
|
||||
Regression coverage: `tests/test_gongwen_delivery.py` and existing gongwen tests.
|
||||
|
||||
## Retrieved skill files (2026-10-07)
|
||||
|
||||
`citation_display.retrieved_files_enabled` (default true) controls retrieved-file
|
||||
cards globally. It is read and patched through `/api/system-settings/citation-display`;
|
||||
updates require `_require_admin` and preserve other citation settings.
|
||||
|
||||
Chat file cards reuse the existing artifact and knowledge original-file readers.
|
||||
`GET /api/llmwiki/sources/resolve` accepts either a local mapping id or a WeKnora
|
||||
provider id and returns only an authorized local mapping id. A provider lookup
|
||||
must be followed by the existing `get_authorized(..., write=False)` check before
|
||||
returning any identifier; missing and unreadable sources both return 404.
|
||||
The frontend uses this resolved id for both preview and original-file download.
|
||||
|
||||
## Qwen3.8 Flash Next stream integrity (2026-10-07)
|
||||
|
||||
`deerflow.models.chat_stream_state.ChatStreamState` belongs to one provider invocation.
|
||||
`PatchedChatOpenAI` enables it only for the normalized `qwen3.8-flash-next` id;
|
||||
the model factory also checks the configured name for deployments using a generic
|
||||
provider id. The local-only `strict_stream_integrity` override is never sent upstream.
|
||||
Keep unrelated Qwen versions and other models on their existing stream path by default.
|
||||
|
||||
Only explicit `choice.message` string snapshots are converted into unseen suffixes.
|
||||
True `delta` text wins and is appended verbatim, including repeated content/reasoning;
|
||||
structured provider payloads stay opaque. Repair a null terminal delta for this model.
|
||||
EOF without a first-choice `finish_reason`, or a contradictory snapshot, raises the
|
||||
content-free `IncompleteChatStreamError`. The existing LLM middleware retries within
|
||||
its normal cap before any failed result reaches the checkpoint or tool executor.
|
||||
Cancellation propagates without retry; a declared finish reason (including `length`)
|
||||
retains provider semantics. Do not guess completion from answer punctuation.
|
||||
|
||||
Non-stream exchange capture is opened and closed in `_generate` / `_agenerate` so
|
||||
executor-based conversion never resets a ContextVar token from another context.
|
||||
Reuse the safe response dump for conversion and preserve typed `parsed` / `refusal`
|
||||
metadata without serializing arbitrary parsed objects a second time.
|
||||
|
||||
Regression entry: `packages/harness/tests/test_patched_openai_text_integrity.py`.
|
||||
Coverage includes sync/async provider replay, model/alias scoping, failed-attempt
|
||||
checkpoint isolation, same-thread follow-ups, tool safety, cancellation and 20
|
||||
overlapping streams that reuse the same provider completion id. These fixtures
|
||||
exercise code-level compatibility gaps; they do not establish the deployed server's
|
||||
actual trigger without its response evidence.
|
||||
??
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Project Overview
|
||||
|
||||
### Personal follow-up question preference
|
||||
|
||||
`GET/PUT /api/user-preferences` exposes `followup_suggestions_enabled`, defaulting to `false` for both new accounts and older JSON rows without the key. Partial updates merge into the current account's `user_preferences.preferences` JSON alongside `hide_admin_prompt_prefix`; no database migration is needed. Frontend recommendation generation and display must both require this explicit opt-in. Regression coverage: `tests/test_user_preferences.py` (default-off behavior, persisted opt-in/opt-out, account isolation and partial-update preservation).
|
||||
|
||||
### Ordinary Q&A map visualization
|
||||
|
||||
The ordinary lead-agent prompt includes `_ORDINARY_QA_VISUALIZATION_SECTION` only for normal, non-bootstrap, non-writing, non-notebook, non-scheduled runs and defines the highest-priority output-carrier gate. Only an explicit current-request ask for HTML, a webpage/page, dashboard, large-screen view, or `.html` file permits reading `page-report-generation`, calling `page_report_template`, or writing HTML. “Draw a flowchart/chart/map/mind map” and “make a visualization” never imply HTML: ordinary flowcharts must be inline Mermaid, ordinary charts keep the established message-inline path, and requested maps emit strict JSON in a `map-json` fence. A mind map is emitted only for an explicit current-request ask for a mind map/knowledge map (or `analysis-mindmap`); ordinary analyses, summaries, outlines, flowcharts, and generic JSON must never auto-upgrade into one. Its strict `mindmap-json` contract is a bounded rooted tree (`id`, `parentId`, `label`, semantic `kind`, optional detail/source refs), targeting 12–24 nodes with a hard cap of 36, at most four layers including the center topic, at most six top-level dimensions and three children beneath every later node. Models must consolidate related facts into `detail`/`sourceRefs`, not make one node per report bullet, datum, paragraph, or reference. An unfinished dedicated `mindmap-json` fence (and compatible historical `mindmap` fence) remains an ordinary streaming code block; only the completed, valid object lazy-renders as a compact left-to-right message card. Clicking a parent node, or its +/− control at the outgoing connection point, expands/collapses its child tree and the canvas uses an explicit visible edge stroke. The full dialog keeps its close and view tools in the top-right and a layout-derived all-map preview in the bottom-right; the message preview also permits pan, wheel zoom, and zoom/reset controls. The message card's icon-only download control opens PNG/SVG/JSON choices. Generic JSON never invokes this renderer. Ambiguity defaults to inline output. Once the HTML gate passes, delivery is mandatory as a sandbox-generated `/mnt/user-data/outputs/*.html` artifact opened with `present_files`; a full inline `html` fence is forbidden as a substitute. The shared frontend `MarkdownContent` keeps its exact legacy branch when no map-shaped fence exists; dedicated `map-json` fences and complete generic `json` fences that contain map fields and pass the full map schema are validated and lazy-rendered from `components/workspace/visualizations/`. Map/loading/error cards use the full message-content width so their edges align with the step panel. Non-map and incomplete generic JSON retain the ordinary code-block path.
|
||||
The ordinary lead-agent prompt includes `_ORDINARY_QA_VISUALIZATION_SECTION` only for normal, non-bootstrap, non-writing, non-notebook, non-scheduled runs and defines the highest-priority output-carrier gate. Only an explicit current-request ask for HTML, a webpage/page, dashboard, large-screen view, or `.html` file permits reading `page-report-generation`, calling `page_report_template`, or writing HTML. “Draw a flowchart/chart/map/mind map” and “make a visualization” never imply HTML: ordinary flowcharts must be inline Mermaid, ordinary charts keep the established message-inline path, and requested maps emit strict JSON in a `map-json` fence (markers use `name`+`coordinate` plus optional `color`; lines are included only when the user explicitly asks to connect places, must use `coordinates` with at least two `[lng,lat]` points, not `from`/`to`/`coords`, and default to `arrow`/`flow`). The frontend recovers those common aliases, assigns distinct marker colors, draws requested lines as looping SVG arrows, and drops unusable lines so markers still render. A mind map is emitted only for an explicit current-request ask for a mind map/knowledge map (or `analysis-mindmap`); ordinary analyses, summaries, outlines, flowcharts, and generic JSON must never auto-upgrade into one. Its strict `mindmap-json` contract is a bounded rooted tree (`id`, `parentId`, `label`, semantic `kind`, optional detail/source refs), targeting 12–24 nodes with a hard cap of 36, at most four layers including the center topic, at most six top-level dimensions and three children beneath every later node. Models must consolidate related facts into `detail`/`sourceRefs`, not make one node per report bullet, datum, paragraph, or reference. An unfinished dedicated `mindmap-json` fence (and compatible historical `mindmap` fence) remains an ordinary streaming code block; only the completed, valid object lazy-renders as a compact left-to-right message card. Clicking a parent node, or its +/− control at the outgoing connection point, expands/collapses its child tree and the canvas uses an explicit visible edge stroke. The full dialog keeps its close and view tools in the top-right and a layout-derived all-map preview in the bottom-right; the message preview also permits pan, wheel zoom, and zoom/reset controls. The message card's icon-only download control opens PNG/SVG/JSON choices. Generic JSON never invokes this renderer. Ambiguity defaults to inline output. Once the HTML gate passes, delivery is mandatory as a sandbox-generated `/mnt/user-data/outputs/*.html` artifact opened with `present_files`; a full inline `html` fence is forbidden as a substitute. The shared frontend `MarkdownContent` keeps its exact legacy branch when no map-shaped fence exists; dedicated `map-json` fences and complete generic `json` fences that contain map fields and pass the full map schema are validated and lazy-rendered from `components/workspace/visualizations/`. Map/loading/error cards use the full message-content width so their edges align with the step panel. Non-map and incomplete generic JSON retain the ordinary code-block path.
|
||||
|
||||
HTML artifacts are single-page by default. Omitted `preferred_page_count` is normalized to `1` during direct validation. A count greater than one is allowed only when the current user directly asks for multiple pages or gives a page count; content length and report chapter count never imply multi-page intent. Explicit multi-page intent without a count defaults to a three-page lightweight portal. It is one artifact HTML, not a folder of linked pages: `<main class="pages">` contains logical `.page` sections with unique `id`/`data-page`, and `href="#page-..."` navigation works in the sandboxed Blob preview through the injected anchor handler. Its validator requires the local Bootstrap/Alpine assets, `.pages`, multiple `data-page` sections and real anchor navigation: every actual `<a href="#page-...">` and every script-set `location.hash` must match a unique logical-page id, and each page must be reachable; comments or `#overview`/`#stations` data-page values cannot spoof this check. `MultiPageHtmlMiddleware` repeats that structural inspection at the `present_files` boundary, so a model that skips validate or writes custom tabs cannot expose a nonconforming portal. A user-uploaded background goes only on the requested logical `.page` in this mode (home page when unspecified), not on every page. When a template is explicitly requested, prepare strictly excludes profiles from the wrong page-count mode, persists `requested_page_count` and `page_mode`, and validate reloads them. Validation rejects multiple detected page views (`data-page`, tabpanels, `.report-page`, or `.view`) in single-page mode, or a single view when multi-page was explicitly requested.
|
||||
|
||||
@@ -16,13 +357,13 @@ For an explicit multi-page HTML request without an explicit template request, `M
|
||||
|
||||
### Generated HTML artifact visualization runtimes
|
||||
|
||||
The same normal lead-agent prompt includes `_HTML_ARTIFACT_VISUALIZATION_SECTION`, but its contents are executable only after the explicit HTML-intent gate above passes; visualization content alone never implies this intent. New generated HTML explicitly loads browser distributions from Gateway `/api/public/html-runtime/`: `bootstrap.min.css`, `alpine.min.js`, `bootstrap.bundle.min.js`, `g6.min.js`, `echarts.min.js`, and then `deerflow-runtime.js`; it can also load the packaged country/region map catalog: `china-provinces`, `us-states`, `japan-prefectures`, and `taiwan-counties`. Bootstrap is the responsive layout/modal layer and Alpine is the small state/list-rendering layer; jQuery, React, Vue, Babel and public CDNs are not part of this artifact contract. The files are package resources under `packages/harness/deerflow/html_runtime/assets/` and ship with the harness wheel, so HTML execution does not depend on frontend Vite/node_modules assets, Leaflet, map tiles, or a public CDN. `config.yaml -> html_runtime.base_url` is the browser-reachable absolute runtime base written into generated source (localhost default; deployments must replace it with the real Gateway origin). Flowcharts use paired `data-df-flow` JSON blocks and AntV G6, not Mermaid; charts use `data-df-chart`; maps use ECharts Canvas-backed `data-df-geo` with the local GeoJSON catalog. `data-df-chart` must be strict JSON: no JavaScript callbacks or function-valued colors, with literal colors/data-item styles and string tooltip formatters instead. Dark HTML must set `data-df-theme="dark"` and explicitly give ECharts global/title/legend/axis/visible-series labels readable light text, including `nameTextStyle` for named axes. Map `description` values are plain text because the runtime escapes markup; click-to-filter or page-switch behavior listens to `deerflow:geo-map-click` and uses a valid `#page-...` anchor. `deerflow-runtime.js` waits until a hidden logical page has a real layout before creating ECharts and tracks later container changes with `ResizeObserver`, so map/chart canvases never depend on a browser-window resize for their first visible render. It initializes all three contracts, opens a detail popover for described regions/markers, and exposes catalog URLs through its public dynamic response. Map large screens use a centered map stage with narrow side rails and bottom summaries, rather than cards layered across a full-page map background. Frontend runtime replacement is bypassed whenever this explicit Gateway runtime is present. This remains separate from ordinary message rendering; ordinary `map-json` keeps its existing frontend tile renderer and `VITE_MAP_TILE_URL` configuration, and ordinary `flow-json`/`chart-json` must not be reintroduced.
|
||||
The same normal lead-agent prompt includes `_HTML_ARTIFACT_VISUALIZATION_SECTION`, but its contents are executable only after the explicit HTML-intent gate above passes; visualization content alone never implies this intent. New generated HTML explicitly loads browser distributions from Gateway `/api/public/html-runtime/`: `bootstrap.min.css`, `alpine.min.js`, `bootstrap.bundle.min.js`, `g6.min.js`, `echarts.min.js`, and then `deerflow-runtime.js`; it can also load the packaged country/region map catalog: `china-provinces`, `us-states`, `japan-prefectures`, and `taiwan-counties`. Bootstrap is the responsive layout/modal layer and Alpine is the small state/list-rendering layer; jQuery, React, Vue, Babel and public CDNs are not part of this artifact contract. The files are package resources under `packages/harness/deerflow/html_runtime/assets/` and ship with the harness wheel, so HTML execution does not depend on frontend Vite/node_modules assets or a public CDN. `system_settings.json -> html_map.engine` (`GET|PUT /api/system-settings/html-map`, admin write) chooses `echarts` (default, local GeoJSON `data-df-geo`) or `leaflet` (地图网页, `data-df-map`). The map tile URL saved in the settings dialog (`system_settings.json -> html_runtime.map_tile_url`, `GET|PUT /api/system-settings/html-runtime`) is injected into `deerflow-runtime.js` as `mapTileUrl` and is used only by the Leaflet engine. Both initializers stay in the runtime so already generated pages keep rendering after the switch. The next ordinary Q&A prompt and `page_report_template` validation follow the selected engine. The runtime base URL saved in the same settings block is the browser-reachable absolute address written into generated source. An empty value falls back to an older `config.yaml` entry and then `http://localhost:8001/api/public/html-runtime`. Flowcharts use paired `data-df-flow` JSON blocks and AntV G6, not Mermaid; charts use `data-df-chart`; maps use ECharts Canvas-backed `data-df-geo` with the local GeoJSON catalog. `data-df-chart` must be strict JSON: no JavaScript callbacks or function-valued colors, with literal colors/data-item styles and string tooltip formatters instead. Dark HTML must set `data-df-theme="dark"` and explicitly give ECharts global/title/legend/axis/visible-series labels readable light text, including `nameTextStyle` for named axes. Map `description` values are plain text because the runtime escapes markup; click-to-filter or page-switch behavior listens to `deerflow:geo-map-click` and uses a valid `#page-...` anchor. `deerflow-runtime.js` waits until a hidden logical page has a real layout before creating ECharts and tracks later container changes with `ResizeObserver`, so map/chart canvases never depend on a browser-window resize for their first visible render. It initializes all three contracts, opens a detail popover for described regions/markers, and exposes catalog URLs through its public dynamic response. Map large screens use a centered map stage with narrow side rails and bottom summaries, rather than cards layered across a full-page map background. Frontend runtime replacement is bypassed whenever this explicit Gateway runtime is present. This remains separate from ordinary message rendering; ordinary `map-json` keeps its existing frontend tile renderer and `VITE_MAP_TILE_URL` configuration, and ordinary `flow-json`/`chart-json` must not be reintroduced.
|
||||
|
||||
User-uploaded raster images can be used as generated-HTML backgrounds through signed, unauthenticated `GET /api/public/html-images/{token}` URLs. `UploadsMiddleware` mints one URL per uploaded `.jpg/.jpeg/.png/.webp/.gif/.bmp` and injects it into `<uploaded_files>` as `Public URL`; the HTML contract requires that exact absolute URL in `background-image`, never a sandbox path, artifact API path, relative path, or base64 data URI. Single-page artifacts place it on `html/body`; the multi-page portal places it only on the requested logical page. Tokens are HMAC-signed, do not expire (the URL is written into durable HTML), and only grant read access to one file under the thread `uploads/` or `outputs/` directory. Engine: `deerflow.html_runtime.public_images`; router: `app/gateway/routers/html_images.py`; tests: `tests/test_html_images.py`.
|
||||
|
||||
The G6 runtime must not mutate author layout by auto-switching long LR graphs to TB or by increasing the target element height after render. Layout quality is decided before delivery: keep a flow to at most eight nodes where possible; otherwise split it or declare TB with at least 600px. Each flow gets its own document row and a parent that can grow naturally, preventing canvas overlap with later sections.
|
||||
|
||||
Markdown-to-HTML conversion must inventory source Mermaid/flowcharts, charts, and maps before editing. Direct `page_report_template validate` derives `required_visualizations` from its `required_features` and `page_brief`; an explicit-template prepare also persists a machine-readable `visualization_contract` in the per-output brief, which validate reloads and merges with its arguments. Required flowcharts fail validation unless the HTML contains `g6.min.js`, `deerflow-runtime.js`, a `.df-flow` container, and `data-df-flow`; embedded Mermaid and inline `new G6.Graph(...)` are errors. Required charts use ECharts + `data-df-chart`; required maps use ECharts + `data-df-geo`, an `.df-geo-map` container, and one of the four local GeoJSON catalog entries. Leaflet, `data-df-map`, map tiles, and manual `echarts.init` are validation errors for generated HTML. A failed report must be repaired and revalidated before `present_files`.
|
||||
Markdown-to-HTML conversion must inventory source Mermaid/flowcharts, charts, and maps before editing. Direct `page_report_template validate` derives `required_visualizations` from its `required_features` and `page_brief`; an explicit-template prepare also persists a machine-readable `visualization_contract` in the per-output brief, which validate reloads and merges with its arguments. Required flowcharts fail validation unless the HTML contains `g6.min.js`, `deerflow-runtime.js`, a `.df-flow` container, and `data-df-flow`; embedded Mermaid and inline `new G6.Graph(...)` are errors. Required charts use ECharts + `data-df-chart`. Required maps follow `html_map.engine`: ECharts uses `data-df-geo`, an `.df-geo-map` container, and one of the four local GeoJSON catalog entries; 地图网页 uses Leaflet `leaflet.css`/`leaflet.js`, `data-df-map`, and a `.df-map` container. The unselected contract, handwritten map initialization, and (for ECharts) map tile URLs are validation errors. A failed report must be repaired and revalidated before `present_files`.
|
||||
|
||||
G6 nodes use light semantic rounded cards with matching borders, dark high-contrast text, centered type-prefixed labels, subtle shadows, and non-animated initial rendering (avoids a faded first frame). More than seven nodes automatically switch an `LR` request to layered `TB` layout and expand a short container to 720px, preventing long workflows from shrinking into illegible shapes; drag/zoom remain enabled.
|
||||
|
||||
@@ -189,6 +530,28 @@ commands.py` and `tests/test_position_roundtable_concurrency.py` cover the
|
||||
state transitions; `tests/test_roundtable_postgres_concurrency.py` is the real
|
||||
PostgreSQL acceptance suite (set `DEERFLOW_POSTGRES_CONCURRENCY_TEST_URL`).
|
||||
|
||||
Step-2 roundtable rooms share one deployment-wide cap,
|
||||
``system_settings.json -> roundtable_room_limit`` (default enabled, 3 rooms).
|
||||
``GET/PUT /api/system-settings/roundtable-room-limit`` is the admin switch.
|
||||
Occupancy is summed across **every account**: durable foreground leases keyed
|
||||
by a hidden per-live-execution ``roomSessionId``, in-flight LangGraph runs on
|
||||
threads stamped with the corresponding ``roundtable_room_key``, and executing
|
||||
background jobs (``queued`` / ``running`` / ``cancel_requested`` with a live
|
||||
worker lease or a recent update). Different pages running the same saved draft
|
||||
have different session ids and therefore take different rooms; all seats and
|
||||
Step 3 calls within one live execution share its key. Run-manager inspection
|
||||
is reconciliation-only: when a durable lease/job already owns the same
|
||||
draft+user/task scope, stale or overlapping run keys in that scope are not
|
||||
counted again. Merely opening the page,
|
||||
viewing saved history, paused ``awaiting_input`` jobs, stale historical rows,
|
||||
and Step-1 intent threads do not occupy a slot. Foreground-to-background
|
||||
handoff replaces the foreground lease with the durable job occupancy. There is
|
||||
no per-user pool. Admission is serialized by a database gate row, and the cap
|
||||
is enforced at acquire, init, every model-producing run, and background-job
|
||||
start. Full pools return HTTP 409; an enabled gate with missing storage or a
|
||||
missing session id fails closed. The switch applies on the next admission
|
||||
without a restart. Position and eight-department roundtables are not counted.
|
||||
|
||||
Roundtable backend jobs use a database unique active-dedupe key plus a leased
|
||||
dispatcher, never process-local start deduplication. Cancellation is two
|
||||
phase (`cancel_requested` then `cancelled`): the lease owner’s watcher normally
|
||||
@@ -335,6 +698,8 @@ from deerflow.config import get_app_config
|
||||
|
||||
### Agent System
|
||||
|
||||
**Gallery chat link.** An agent's `config.yaml` may store `chat_link` (`url`, `open_mode` of `blank`/`self`, and light-app-shaped `route_params`). The gallery 「对话」 button opens that resolved URL; an empty url keeps the in-app chat. The list endpoint reads the mirrored `agent_extras.chat_link` column (nullable JSON, added by the startup column sweep) instead of parsing every config file. Create/update and directory backfill write the mirror; a skills-only `set_extras` leaves an existing link in place.
|
||||
|
||||
**Lead Agent** (`packages/harness/deerflow/agents/lead_agent/agent.py`):
|
||||
- Entry point: `make_lead_agent(config: RunnableConfig)` registered in `langgraph.json`
|
||||
- Dynamic model selection via `create_chat_model()` with thinking/vision support
|
||||
@@ -345,8 +710,12 @@ from deerflow.config import get_app_config
|
||||
|
||||
**Forced-research built-in agent.** `forced-research-responder` (「强制检索输出助手」) is seeded from `app/gateway/routers/_forced_research_seed_assets/` before `_sync_legacy_agents`. `ForcedResearchMiddleware` is mounted only for this stable id. On every visible user turn it requires a successful information-gathering tool result before the model may finish; `tool_choice=required` is backed by an `after_model -> model` retry for gateways that ignore tool choice, with an eight-attempt failure cap that permits only an honest failure response. Skill discovery, reading `SKILL.md`, and writing a helper script do not satisfy the gate; web/knowledge/MCP results, non-skill uploaded-file reads, or successful Python skill-script execution do. Tool filtering removes `present_files`, `skill_manage`, task/schedule/memory/clarification surfaces. The middleware permits `write_file`/`str_replace` only for `/mnt/user-data/workspace/*.py` and permits `bash` only for `.py` execution without redirection/document paths, so skills can run scripts but the agent cannot create Markdown/document artifacts or write `outputs`. Tests: `tests/test_forced_research_agent.py`.
|
||||
|
||||
**Elasticsearch offline operations built-in agent.** `elasticsearch-offline-ops` (「Elasticsearch 离线运维助手」) and its same-name public skill are seeded from `app/gateway/routers/_elasticsearch_offline_ops_seed_assets/` before `_sync_legacy_agents`. Seeding is missing-directory-only so administrator edits survive restarts. The agent intentionally has `file:read`, `file:write`, and `bash` but no web group: air-gapped guidance comes from the shipped version-parameterized deployment and troubleshooting references. Its SOUL requires the exact ES version/OS/architecture/topology before final commands, keeps Elasticsearch security/TLS enabled, separates the DeerFlow sandbox from the target ES hosts, and makes read-only evidence collection precede changes. `scripts/collect_diagnostics.sh` gathers Linux/service/log and authenticated cluster evidence without mutations or `--insecure`; credentials come from operator-owned netrc/header files rather than command arguments or an unrestricted curl config. `scripts/analyze_diagnostics.py` is a stdlib-only heuristic that emits matched evidence and a next read-only check, or `insufficient_evidence`; it never claims an automatic root cause. Tests: `tests/test_elasticsearch_offline_ops_seed.py`, `tests/test_seed_assets.py`.
|
||||
|
||||
**Strategy-selection functional agents.** `app/gateway/routers/_strategy_selection_seed.py` seeds nine stable folder-backed agents from `_strategy_selection_seed_assets/` before `_sync_legacy_agents`: intent, governance/diplomacy/economy/security/public-opinion/law-policy angles, judge, and solution editor. Seeding is missing-directory-only and must never overwrite administrator changes. The global `strategy_agent_roles` registry classifies them independently of projects: intent/judge/solution are fixed, unique roles that may be configured but never added, disabled, reordered, or deleted; angle roles are a shared, ordered catalog that administrators may add, edit, disable (soft delete), restore, and reorder through `/api/strategy-agents`. Public `GET /api/strategy-agents` returns enabled identity/description metadata only; `include_disabled=true` or `include_config=true` is admin-only. Admin writes are `POST /api/strategy-agents/angles`, `PUT /api/strategy-agents/{id}`, and `DELETE /api/strategy-agents/{id}`; DELETE retains the folder-backed agent and role row to preserve historical references. Dynamic angles remain built-in folder-backed agents under the reserved `strategy-selection-angle-` id namespace, with `web` + `file:read`; a blank SOUL on create receives a safe role template. All strategy skill writes validate that names exist, are active, and are globally visible; generic agent CRUD cannot create or hard-delete reserved strategy IDs, and parallel execution rejects disabled roles. Angle configs reuse `policy-knowledge`; intent reuses `intent-clarification-policy`; the judge has no retrieval skill; the solution editor gets only `file:write`. Keep the SOUL contracts neutral and evidence-aware, with explicit de-escalation/civilian-protection framing and hard bans on operational combat guidance and targeted political manipulation. Tests: `tests/test_strategy_selection_seed.py`, `tests/test_strategy_agent_management.py`.
|
||||
|
||||
Only `strategy-selection-intent` has a clarification reply surface in the strategy workbench. `parallel_agents` must exclude `ask_clarification` for every other `strategy-selection-*` role and inject a direct-delivery guard: angle, judge, and solution stages use conservative assumptions, lower confidence/score where needed, list gaps under data limitations, and still finish in one pass. The judge brief repeats this rule at runtime because existing folder-backed agent SOUL files may be administrator-customized and are intentionally never overwritten. If a legacy backend nevertheless emits a judge clarification, the frontend treats it as a bounded repair attempt rather than asking the user to repeat the same manual run. Malformed or incomplete evaluation JSON receives the same bounded repair path; prior attempts are replaced in the live panel, never concatenated into duplicate final content.
|
||||
|
||||
`POST /api/parallel-agents/run/stream` must remain observable during silent skill calls. Each running seat emits an `agent_progress` heartbeat every five seconds with the latest translated tool/reasoning/body phase and monotonic elapsed seconds; an upstream iterator that produces no frame for 180 seconds is closed and that seat emits `agent_error`, while the other seats continue. Set the terminal flag before queuing `agent_done` / clarification so a late heartbeat cannot reopen a completed card. The strategy workbench renders the intent Q&A history, readable skill execution steps and live heartbeat on the affected agent card; judge and synthesis traces use the same step translation. Tests: `tests/test_roundtable_model_fallback.py` and frontend `src/strategy-selection/lib.test.ts`.
|
||||
|
||||
**ThreadState** (`packages/harness/deerflow/agents/thread_state.py`):
|
||||
@@ -470,14 +839,14 @@ FastAPI application on port 8001 with health checks at `GET|HEAD /health`, `/api
|
||||
| **Threads search** (`POST /threads/search`, LangGraph-compat) | Conversation listing. **System threads are excluded by default**: rows whose `metadata.system` is truthy (scheduler runs, roundtable workers) or whose `thread_type` is a known system type (`task-analysis` seat runs included; older seat rows are also matched by `task_analysis_seat`) never crowd the user's list; a caller whose `metadata` filter explicitly references `thread_type`/`system` (e.g. studio notebooks) opts out of the exclusion. New `query` field = case-insensitive title (`display_name`) substring search, used by the chats page's server-side pagination. `POST /threads/count` takes the same request body (limit/offset ignored) and returns `{total}` under identical filtering/exclusion semantics ? powers the chats page's ??/??? display without fetching rows (`ThreadMetaStore.count`). `ThreadMetaStore.search` gained `exclude_system`/`query` params; JSON-blob filters batch-scan until the requested `limit`/`offset` window is full (so pagination over a heavily-filtered set never under-fills), with a `thread_id` tie-breaker on the `updated_at DESC` ordering for stable OFFSET paging. Tests: `tests/test_thread_meta_search.py` |
|
||||
| **Artifacts** (`/api/threads/{id}/artifacts`, `/api/artifact-library`, `/api/strategy-project-artifacts/{id}/download`) | `GET /{path}` streams thread artifacts; active content types (`text/html`, `application/xhtml+xml`, `image/svg+xml`) are always forced as download attachments to reduce XSS risk; `?download=true` still forces download for other file types. `GET /api/threads/{id}/artifact-preview?path=` returns bounded text, using streaming statistics plus deterministic head/middle/tail samples above the database threshold. `GET /api/artifact-library` combines every current-user normal thread's authoritative `outputs/` files with completed strategy-selection syntheses projected from the owner-scoped `parallel_agent_tasks.messages` snapshot. Strategy items use `source_type=strategy`, carry `strategy_project_id` / `strategy_name` / `strategy_stale`, and explicitly distinguish `strategy_source_kind=strategy_selection` from an `analysis_project_mode=strategy` thread (`analysis_project`); virtual items also expose `download_url`, so clients never infer routing from sentinel thread/path values. Search/source filters cover both. `workflowStale=true` keeps the last completed historical synthesis visible and downloadable. The dedicated strategy download route revalidates current-user ownership and snapshot completion and serves Markdown without pretending the virtual item is a thread artifact. Uploads and image/audio/video media remain excluded. Library metadata is persisted in `artifact_library_items`; list/search/filter/count/pagination read only this user-scoped SQL index. A generation-fenced `artifact_library_sync_states` lease schedules low-concurrency filesystem/task reconciliation in the background on first/stale reads or `refresh=true`, keeps the last successful snapshot visible during refresh/failure, and reports `sync_status` / `syncing` / `indexed_at` / `sync_error` to the client. Files themselves remain authoritative on disk and are never copied into the database. |
|
||||
| **Word export** (`POST /api/writing/export/docx`) | Shared Markdown → DOCX generator used by conversation, artifact, scheduled-task, Deep Research, and AI-writing download surfaces. `app/gateway/word_export.py` builds OOXML with no Python conversion dependency: A4 portrait, 3.7/3.5/2.8/2.6 cm margins, real multilevel heading numbering (`一、` / `(一)` / `1.`), two-character first-line indent on heading levels 1–3, formal body/table styles, page numbers beginning at 1 with odd pages left-aligned and even pages right-aligned, and embedded `app/gateway/assets/fonts/方正小标宋简体.ttf`. Before numbering it removes complete manual heading prefixes (including `2.1` / `2.1.1`) and drops Markdown horizontal rules from Word output. The ordinary-Q&A Lead Agent can emit the matching Markdown heading forms and avoid unordered bullet lists when the admin enables `prompt_prefix.ordinary_qa_markdown_format_enabled` in `system_settings.json`; the switch defaults off, while writing setup, writing mode, notebook, scheduled runs, and SOUL-driven custom agents keep their independent prompts. The font loader validates its internal family, SHA-256, and OpenType embedding permission (`OS/2.fsType`) before export. Tests: `tests/test_word_export.py`, `tests/test_es_query_routing_prompt.py`. |
|
||||
| **Deep Research** (`/api/deep-research`) | Durable, user-owned research sessions/jobs with replayable SSE events and hidden-thread artifacts. `basic`/`quick` use the compact runner; `detailed` creates bounded subtopics, searches/writes sections concurrently, and assembles a report; `deep` recursively searches follow-up questions under a deterministic breadth/depth query budget. `multi_agent` is a LangGraph 1.x editor/researcher/writer/reviewer graph: its initial plan emits `awaiting_input`, the recorder stores the interrupt and releases the lease, and `POST /jobs/{id}/resume` rebuilds the graph from the persisted approved/revised plan. A bounded `custom_outline` is frozen into the session; headings/list entries deterministically supply the detailed and multi-agent plan, and the shared writer receives it only as a guarded content constraint. When `custom_outline` is empty (the workbench default — the user did not pick a report structure), chat collection first calls `POST /sessions/{id}/intent-outline` after the collector turn ends: the chosen model plans a Markdown skeleton from the topic plus collected materials, the outline is frozen into the session snapshot and shown in the「开始撰写」form for editing. If that step is skipped or fails, `basic`/`quick` still call `generate_intent_outline` at write time, emit `plan_created` (`source: intent`), then write through the same shared writer; generation failure is a recoverable `intent_outline_fallback` warning and does not abort the report. Completed reports support owner-scoped follow-up (`GET /sessions/{id}/messages`, `POST /sessions/{id}/chat`): context is only the current report, selected sources and recent messages; source-id citations are persisted, while `allow_new_research=true` is explicitly rejected. Owners may change that evidence set only after the job stops via `PATCH /sessions/{id}/sources/{source_id}`; this applies to future follow-ups rather than rewriting a completed report. Optional report illustrations are gated by `config.yaml -> deep_research.image_generation`: the Harness `OpenAICompatibleImageProvider` requires base64 output, the shared runner step writes only bounded `research-image-01..04.(png|jpg|webp)` artifacts and Markdown uses private `deep-research://` links. `GET /sessions/{id}/artifacts/{name}` checks session ownership before serving an image; an unavailable provider yields a warning but never fails the text report. Runners live in `packages/harness/deerflow/agents/deep_research/runners/`, may never import `app.*`, and use only the `AdapterBundle`. |
|
||||
| **Deep Research** (`/api/deep-research`) | Durable, user-owned research sessions/jobs with replayable SSE events and hidden-thread artifacts. `basic`/`quick` use the compact runner; `detailed` creates bounded subtopics, searches/writes sections concurrently, and assembles a report; `deep` recursively searches follow-up questions under a deterministic breadth/depth query budget. `multi_agent` is a LangGraph 1.x editor/researcher/writer/reviewer graph: its initial plan emits `awaiting_input`, the recorder stores the interrupt and releases the lease, and `POST /jobs/{id}/resume` rebuilds the graph from the persisted approved/revised plan. A bounded `custom_outline` is frozen into the session; headings/list entries deterministically supply the detailed and multi-agent plan, and the shared writer receives it only as a guarded content constraint. The compact writer validates generated Markdown against that frozen selection: `fixed` requires every non-placeholder heading with the same level and order, while `adaptive` retries only when no core template heading survives. That correction shares the single existing retry budget (never more than two writer calls), and its deterministic fallback is outline-shaped as well. When `custom_outline` is empty (the workbench default — the user did not pick a report structure), chat collection first calls `POST /sessions/{id}/intent-outline` after the collector turn ends: the chosen model plans a Markdown skeleton from the topic plus collected materials, the outline is frozen into the session snapshot and shown in the「开始撰写」form for editing. If that step is skipped or fails, `basic`/`quick` still call `generate_intent_outline` at write time, emit `plan_created` (`source: intent`), then write through the same shared writer; generation failure is a recoverable `intent_outline_fallback` warning and does not abort the report. Completed reports support owner-scoped follow-up (`GET /sessions/{id}/messages`, `POST /sessions/{id}/chat`): context is only the current report, selected sources and recent messages; source-id citations are persisted, while `allow_new_research=true` is explicitly rejected. Owners may change that evidence set only after the job stops via `PATCH /sessions/{id}/sources/{source_id}`; this applies to future follow-ups rather than rewriting a completed report. Optional report illustrations are gated by `config.yaml -> deep_research.image_generation`: the Harness `OpenAICompatibleImageProvider` requires base64 output, the shared runner step writes only bounded `research-image-01..04.(png|jpg|webp)` artifacts and Markdown uses private `deep-research://` links. `GET /sessions/{id}/artifacts/{name}` checks session ownership before serving an image; an unavailable provider yields a warning but never fails the text report. Runners live in `packages/harness/deerflow/agents/deep_research/runners/`, may never import `app.*`, and use only the `AdapterBundle`. |
|
||||
| **Gateway auth** | Owner-scoped Deep Research routes await the platform's asynchronous `get_current_user` resolver before they read or write persistence, ensuring each session and job operation receives a concrete user ID. Deep Research artifacts are always resolved from a whitelisted filename via `/mnt/user-data/outputs/<name>` before mapping to a host path, preserving the sandbox guard on native Windows and Linux hosts. |
|
||||
| **Suggestions** (`/api/threads/{id}/suggestions`) | `POST /` - generate follow-up questions; rich list/block model content is normalized before JSON parsing |
|
||||
| **Thread Runs** (`/api/threads/{id}/runs`) | `POST /` - create background run; `POST /stream` - create + SSE stream; `POST /wait` - create + block; `GET /` - list runs; `GET /{rid}` - run details; `POST /{rid}/cancel` - cancel; `GET /{rid}/join` - join SSE; `GET /{rid}/messages` - paginated messages `{data, has_more}`; `GET /{rid}/events` - full event stream; `GET /../messages` - thread messages with feedback; `GET /../token-usage` - aggregate tokens |
|
||||
| **Thread Runs** (`/api/threads/{id}/runs`) | `POST /` - create background run; `POST /stream` - create + SSE stream; `POST /wait` - create + block; `GET /` - list runs; `GET /{rid}` - run details; `POST /{rid}/cancel` - cancel; `GET /{rid}/join` - join SSE; `GET /{rid}/messages` - paginated messages `{data, has_more}`; `GET /{rid}/events` - full event stream; `GET /../run-messages` - cross-run event-store messages with feedback. The separate `GET /api/threads/{id}/messages` contract remains the chronological latest-checkpoint transcript; `GET /../token-usage` - aggregate tokens |
|
||||
| **Answer feedback** (`/api/threads/{id}/messages/{mid}/feedback`) | Message-scoped `PUT`/`DELETE`, `GET /feedback-context`, and batched `POST /api/threads/{id}/feedback-contexts`; persists excellent/poor ratings, six stable poor-reason codes and optional detail idempotently per user/thread/message/agent. Legacy run-scoped feedback routes remain compatible. Admin-only `GET /api/agents/{agent_id}/feedback-score` uses the project-wide `system_role == "admin"` rule (and preserves auth-disabled mode) and returns a Bayesian user-feedback answer-quality score, raw task-success rate, weighted response-quality dimensions and reason counts; it is explicitly not a complete agent-capability evaluation. |
|
||||
| **Runs** (`/api/runs`) | `POST /stream` - stateless run + SSE; `POST /wait` - stateless run + block; `GET /{rid}/messages` - paginated messages by run_id `{data, has_more}` (cursor: `after_seq`/`before_seq`); `GET /{rid}/feedback` - list feedback by run_id |
|
||||
| **Open Chat** (`/api/open/chat`) | **Unauthenticated** open Q&A entry for external systems (the `/api/open/` prefix is whitelisted in `auth_middleware._PUBLIC_PATH_PREFIXES` + CSRF-exempt in `csrf_middleware.should_check_csrf`). `POST /chat` `{message \| messages[], model_name?, agent_name?, agent_id?, thinking_enabled=false, strip_think=true, thread_id?, recursion_limit=100}` → resolves the agent by **Chinese display name** (`agent_name`, matched against each `.deer-flow/agents/*/config.yaml` `name` via `_resolve_agent_id`; explicit `agent_id` wins if given), builds the lead agent via `make_lead_agent` (so the chosen **agent's** SOUL/tools/skills all apply), `ainvoke`s it once (stateless — no checkpointer, no thread persistence; runs under the `"default"` user bucket), and returns **only the final** `{answer, model_name, agent_name, agent_id, thinking_enabled, thread_id}` (no intermediate tool-call/process). **`POST /chat/stream`** is the **SSE streaming variant** (same request body) for **long tasks** — the blocking `/chat` produces no downstream bytes for the whole run, so a front nginx `proxy_read_timeout` (default 60s) cuts it off with a **504**; the stream keeps the connection warm (a `: keepalive` SSE comment every `_HEARTBEAT_SECONDS=15`s via a background producer-task + queue, plus `event: delta` AI-text increments and `event: progress`), and ends with **`event: result`** carrying the same final `{answer, …}` JSON (or `event: error`). Clients only need to read `event: result`; intermediate events are ignorable. Both endpoints share `_build_run_context` (config/agent/thread). `GET /agents` lists `{name, id}` so callers know what `agent_name` to send. **Thinking off is reliable**: when `thinking_enabled=false` the route also sets `thinking_force_disabled=true` so the lead agent calls `create_chat_model(force_disable_thinking=True)`, which injects **three** OpenAI-compatible "thinking off" shapes into `extra_body` — top-level `enable_thinking=false` (**SiliconFlow / DashScope hybrid models** like `deepseek-ai/DeepSeek-V4-*` on `api.siliconflow.cn` — the knob the nested `chat_template_kwargs` shape does NOT reach), `chat_template_kwargs.enable_thinking=false` (vLLM/Qwen native), and `thinking.type="disabled"` (Zhipu GLM) — even for models that only declare `supports_thinking: true`. For the **web-UI path** (which sends `thinking_enabled=false` but NOT `force_disable`), SiliconFlow models need `when_thinking_disabled: {extra_body: {enable_thinking: false}}` in their `config.yaml` entry (added to `deepseek-ai/DeepSeek-V4-Flash`) for thinking-off to take effect. Sets `is_scheduled_run=true` internally to skip the auto-title LLM call. **Generated-file return (`include_files=true`, default on)**: an agent that produces its report as a file (writes to `/mnt/user-data/outputs/` + `present_files`) returns only a short summary in `answer`; to recover the real deliverable both endpoints now also return `files: [{name, virtual_path, encoding(text|base64), content}]` — `_collect_output_files` scans the run's `state['thread_data']['outputs_path']` (the open thread is stateless so artifacts can't be fetched by thread, but the endpoint runs server-side and the sandbox already wrote the files to disk), text decoded as utf-8 else base64, capped by `_MAX_OUTPUT_FILES=20` / `_MAX_OUTPUT_FILE_BYTES=5MB`. `/chat` puts `files` on the response; `/chat/stream` puts it in the `event: result` payload. A run with no answer **and** no files now 502s ("…文本答案或文件"). Router `app/gateway/routers/open_chat.py`; standalone caller `test_open_chat_api.py` (repo root, streaming by default); tests `tests/test_open_chat.py`. |
|
||||
| **Open 3qfx Ask** (`/api/open/3qfx/ask`) | **Unauthenticated** open entry that, given just a **任务id**, directly kicks off the 3qfx(三情分析)「第一次问答」as a **real persisted conversation** (sibling of `/api/open/chat`; same `/api/open/` auth-whitelist + CSRF-exempt). `POST /ask` `{task_id, message?, model_name?, thinking_enabled=false}` → (1) resolves the `3Q` agent from `app.state.business_mapping_store.get_mapping("3Q")` (the **same** agent the frontend deep-link `goPath=3qfx` uses; 503 if unconfigured); (2) **fetches the task detail server-side** (consumer `cop-task-detail`) and builds the opening text **identical to the frontend** `buildOpeningText` 3qfx branch — `任务名称:…\n任务内容:…\n任务id(task_id)为{task_id}` (an explicit `message` overrides and **skips** the fetch); (3) creates a fresh thread tagged `metadata.{taskId, agent_id}` (the exact filter `useTaskThreads` uses, and `metadata.taskId` makes it task-deeplink-shared / creator-path-resolved) and **starts a background persisted run** via `start_run(..., enforce_access=False)` (writes the checkpointer, runs under the `"default"` bucket); (4) **returns immediately** `{thread_id, run_id, agent_id, status, opening_text}` — the caller just needs "started", then watches the run in the 3qfx 任务工作区 conversation list. `start_run` gained a keyword-only `enforce_access` (default `True`); the open entry passes `False` so the admin-configured agent isn't rejected by the per-(missing)-login visibility check. **Consumer fetch config** lives in `config.yaml → task_deeplink`: `cop_task_detail_url` (full URL, appends `?taskId=`), `cop_task_detail_authorization` (default `Bearer admin`), `cop_task_detail_test` + `mock.cop_task_detail` (offline fake when the upstream is unreachable — on in this deployment). Router `app/gateway/routers/open_3qfx.py`; tests `tests/test_open_3qfx.py`. |
|
||||
| **Open 3qfx Ask** (`/api/open/3qfx/ask`) | **Unauthenticated** open entry that, given just a **任务id**, directly kicks off the 3qfx(三情分析)「第一次问答」as a **real persisted conversation** (sibling of `/api/open/chat`; same `/api/open/` auth-whitelist + CSRF-exempt). `POST /ask` `{task_id, message?, model_name?, thinking_enabled=false}` → (1) resolves the `3Q` agent from `app.state.business_mapping_store.get_mapping("3Q")` (the **same** agent the frontend deep-link `goPath=3qfx` uses; 503 if unconfigured); (2) **fetches the task detail server-side** (consumer `cop-task-detail`) and builds the opening text **identical to the frontend** `buildOpeningText` 3qfx branch — `任务名称:…\n任务内容:…\n任务类型:…\n任务id(task_id)为{task_id}`(`taskType` 1/2/3 → 打击/塑势/防御,无法识别则省略) (an explicit `message` overrides and **skips** the fetch); (3) creates a fresh thread tagged `metadata.{taskId, agent_id}` (the exact filter `useTaskThreads` uses, and `metadata.taskId` makes it task-deeplink-shared / creator-path-resolved) and **starts a background persisted run** via `start_run(..., enforce_access=False)` (writes the checkpointer, runs under the `"default"` bucket); (4) **returns immediately** `{thread_id, run_id, agent_id, status, opening_text}` — the caller just needs "started", then watches the run in the 3qfx 任务工作区 conversation list. `start_run` gained a keyword-only `enforce_access` (default `True`); the open entry passes `False` so the admin-configured agent isn't rejected by the per-(missing)-login visibility check. **Consumer fetch config** lives in `config.yaml → task_deeplink`: `cop_task_detail_url` (full URL, appends `?taskId=`), `cop_task_detail_authorization` (default `Bearer admin`), `cop_task_detail_test` + `mock.cop_task_detail` (offline fake when the upstream is unreachable — on in this deployment). Router `app/gateway/routers/open_3qfx.py`; tests `tests/test_open_3qfx.py`. |
|
||||
| **Thread Shares** (`/api/thread-shares`) | Per-conversation sharing with two modes (`thread_shares.mode`). **`import`**: `POST /` `{thread_id, mode}` - create (or reuse) a revocable share code; `GET /{code}/preview` + `POST /{code}/import` - copy that conversation into the caller's account (fresh `thread_id` via `app/gateway/thread_copy.py` + new `threads_meta` row + `user-data` copy; dedup on `metadata.shared_origin`). **`view`**: a public, read-only static page ? see Public Shares below. Both modes: `GET /` - list my codes; `DELETE /{code}` - revoke. |
|
||||
| **Public Shares** (`/api/public/shares`) | **Unauthenticated** (the `/api/public/` prefix is whitelisted in `auth_middleware._PUBLIC_PATH_PREFIXES`). `GET /{code}` - read-only conversation snapshot (title + serialized messages + artifacts) for a `view`-mode share; messages are read straight from the latest checkpoint. `GET /{code}/files/{path}` - serves a file referenced by that conversation (images, Markdown, text, PDF, ?) so the shared page's links are openable; path resolution is confined to the owner's thread directory. Active content (`text/html`, `application/xhtml+xml`, `image/svg+xml`) is forced to download (`_public_share_disposition`); everything else is served inline. Missing/revoked/non-`view` codes all return 404. |
|
||||
| **Roundtable Drafts** (`/api/roundtable-drafts`) | **Strictly per-user** persistence for the roundtable-planning page (replaces the old browser `localStorage`). `GET /` - list drafts (lightweight meta — incl. `chain_title`, the business-chain name used this session, cheaply regex-extracted from the in-memory `step2` blob in `_draft_to_meta`/`_extract_chain_title`; null = no chain / 自由模式; surfaced on the 第一步「分析记录」history card + sidebar list instead of the old「第 X 步」); `POST /` - create; `GET/PUT/DELETE /{id}` - full read / partial update / delete (all owner-scoped). A draft = one full session (Step 1 intent + Step 2 roundtable), step snapshots stored as JSON in `PortableLongText`. Recommendation history bound to a draft: `GET /{id}/recommendations` - list newest-first; `POST /{id}/recommendations` - append one run (objective/status/rationale/picks/candidates). taskId 深链会商记录 are **not** here — they live in the separate `/api/roundtable-task-drafts` store (see below) so personal history never mixes with task records. Backed by `deerflow.persistence.roundtable_drafts` (tables `roundtable_drafts`, `roundtable_recommend_history`); store wired in `deps.py` as `app.state.roundtable_draft_store`. Tests: `tests/test_roundtable_task_drafts.py`. |
|
||||
@@ -544,6 +913,8 @@ The intranet client uses `trust_env=False` and defaults `intranet_verify_ssl=fal
|
||||
|
||||
The admin ????? is served from **precomputed per-day snapshots** in `admin_leaderboard_daily_stats` (keyed by `(stat_date, settings_hash)`), not by aggregating raw rows on every request. A background scheduler inside the Gateway (`_leaderboard_snapshot_scheduler_loop` in `app/gateway/app.py`, singleton file-locked) ticks every `ADMIN_LEADERBOARD_BACKGROUND_INTERVAL_SECONDS` (default 300s): it prewarms recent days, then builds any `queued`/`error`/stale-`running` day. `GET /leaderboard` merges the per-day snapshots for the requested range. **Today** is rebuilt whenever its snapshot is older than `ADMIN_LEADERBOARD_TODAY_TTL_SECONDS` (default 43200s = half a day, so the leaderboard refreshes ~twice daily); a past day, once `ready`, is treated as fresh forever.
|
||||
|
||||
**Recent-only admin audit policy.** The server enforces a rolling **30-calendar-day** window for administrator leaderboard, user analytics, user Q&A/thread audit, LLM/tool metric list/summary/export, and the user-list Q&A counter; URL parameters cannot widen it. The Q&A fallback that reads checkpoints is additionally capped, while indexed run summaries remain the primary audit source. Gateway startup also launches the singleton `admin_audit_retention` sweep: it deletes audit metric rows, terminal `runs` summaries (which contain compact question/answer audit fields and were also last updated before the cutoff), and leaderboard snapshots older than **90 days**. Cleanup is deliberately low-priority: after the default 5-minute startup delay, it runs every 15 minutes using 100-row committed transactions, round-robins each store (at most four batches per store), yields between commits, and stops starting new batches after an 8-second sweep budget; operators may tune these bounds with `ADMIN_AUDIT_RETENTION_*` environment variables. It deliberately does **not** delete conversation threads, checkpoints, or message history; those are user-content records governed by their own lifecycle.
|
||||
|
||||
Two on-demand/automatic refresh paths layer on top:
|
||||
- **Manual "????"** ? `POST /leaderboard/refresh-today` force-rebuilds *today's* snapshot **synchronously** (bounded by the same semaphore/timeout the scheduler uses), then returns the merged range. Lets an admin pull the latest real-time records into the stats without waiting for the TTL. Frontend: a "????" button on `AdminUserLeaderboardPage` ? `useRefreshTodayLeaderboard` (`core/admin/hooks.ts`), which primes + invalidates the `["admin-activity","leaderboard"]` query.
|
||||
- **Daily finalize of the previous day** ? once per day, on/after `ADMIN_LEADERBOARD_FINALIZE_HOUR` (local Beijing, default `1` = ?? 1 ?), the scheduler tick force-recomputes *yesterday* a single time so runs/tool-calls that only landed near midnight are fully captured. The decision is the pure `deerflow.persistence.admin_stats.finalize.should_finalize_previous_day` (status/generated_at/now/hour) ? the rebuild stamps `generated_at` to today, which self-debounces it for the rest of the day (no extra state file). Tests: `tests/test_leaderboard_finalize.py`.
|
||||
@@ -652,7 +1023,7 @@ The fixed-deep-blue wx full-chat route (`#/page/workspace/wx/chats/new`) hides t
|
||||
|
||||
### Business-mapping authorization
|
||||
|
||||
`/api/business-mapping` remains globally readable. Seeded rows include the main codes (3Q / 6BF / 7BF / 8BF / AFX) plus independent slots `3q-0…3q-6`, `6-1…6-4`, and `7-1…7-4` (empty agent/chain until configured on the 配置管理 page). Updates to 3Q, 6BF, 7BF, and those independent slots require `system_role == "admin"`; the 8BF public business-chain mapping is intentionally different and can be updated only by the trusted authenticated account whose email prefix is `lqq`. Do not rely on a username supplied in an external login URL. The frontend follows the same email-prefix rule when deciding whether to render the 8BF configuration button, while the router (`app/gateway/routers/business_mapping.py`) remains the enforcement point. Tests: `tests/test_business_mapping.py`.
|
||||
`/api/business-mapping` remains globally readable. Seeded rows include the main codes (3Q / 6BF / 7BF / 8BF / AFX) plus independent slots `3q-0…3q-6`, `6-1…6-4`, and `7-1…7-4` (empty agent/chain until configured on the 配置管理 page). `3q-1…3q-6` seed as `is_custom=true` / `mapping_type=自定义类型` so `goPath=3q-1` resolves as a custom deep link; the configuration page hides those six rows and defaults the type filter to 自定义类型. Updates to 3Q, 6BF, 7BF, and those independent slots require `system_role == "admin"`; the 8BF public business-chain mapping is intentionally different and can be updated only by the trusted authenticated account whose email prefix is `lqq`. Do not rely on a username supplied in an external login URL. The frontend follows the same email-prefix rule when deciding whether to render the 8BF configuration button, while the router (`app/gateway/routers/business_mapping.py`) remains the enforcement point. Tests: `tests/test_business_mapping.py`.
|
||||
|
||||
### Position-collaboration foundation
|
||||
|
||||
@@ -855,7 +1226,20 @@ the catalogue; a command or read that names a skill reconciles only that skill
|
||||
and any support library it declares. Skill-management file saves, deletions,
|
||||
online/offline changes, skill deletion, and skill-address apply/rollback sync
|
||||
that skill into the view immediately and publish the new extensions hash, so the
|
||||
next command stays on the narrow path. A missing view or an extensions hash the
|
||||
next command stays on the narrow path. Agent `skill_manage` create, edit, patch,
|
||||
and delete do the same for the sandbox view and, when the enabled-skills prompt
|
||||
cache is already warm, replace only that skill in the cache. A cold prompt cache
|
||||
still reloads the full catalogue. Creating a skill stats `custom/<name>/SKILL.md`
|
||||
instead of parsing every skill. If the model omits `content`, create reads a
|
||||
`write_file` path under the thread workspace instead of failing the turn.
|
||||
`write_file` and `remove_file` sync that skill's view and leave the prompt cache unchanged.
|
||||
Creating an agent calls `skill_list` / `search_skills`; when this cache is warm those
|
||||
tools reuse it instead of parsing every SKILL.md again, and `skill_view` reads only
|
||||
the named skill. A cold cache still loads the catalogue, then checks the canonical
|
||||
directory before a full walk.
|
||||
A repeated-tool warning is appended to the assistant tool-call message so the
|
||||
tool result still follows it; a user message between them makes the next model
|
||||
request fail. A missing view or an extensions hash the
|
||||
sync did not publish still falls back to the full rebuild. Direct edits outside
|
||||
those writes stay stale until the next full rebuild. Offline skills are removed
|
||||
from the view at disable time, which is what keeps a hidden shell path from
|
||||
@@ -970,7 +1354,10 @@ pool changes; model timing instances also need restart for consistent behavior.
|
||||
source runtime under `skills/public/cognitive-situation-map/assets/runtime/`
|
||||
and a Gateway-served copy under
|
||||
`packages/harness/deerflow/html_runtime/assets/cognitive-situation-map/`.
|
||||
Keep their JavaScript, schema, and map atlas assets in sync. Built-in regions
|
||||
Keep their JavaScript, schema, and map atlas assets in sync. `template.aspect`
|
||||
locks a new page to `16:9` (1920×1080, the default when unspecified) or
|
||||
`32:9` (2560×720, explicit ultra-wide only); pages without the field keep
|
||||
the older viewport-adaptive layout. Built-in regions
|
||||
include Taiwan Strait, South China Sea, the Philippines-centred surrounding
|
||||
region (with its southern Sulu/Celebes approaches), First Island Chain, and
|
||||
the U.S.--Pacific connection; every region is a four-quadrant atlas for the
|
||||
@@ -983,7 +1370,7 @@ pool changes; model timing instances also need restart for consistent behavior.
|
||||
- **Format**: Directory with `SKILL.md` (YAML frontmatter: name, description, license, allowed-tools). `name` and `description` are the minimal metadata contract; the frontend install dialog renders the shared `MINIMAL_SKILL_MD_EXAMPLE` containing only those two keys so optional fields are not mistaken for requirements. Keep `frontend-web/src/core/skill-example.test.ts` aligned with backend validation.
|
||||
- **Loading**: `load_skills()` recursively scans `skills/{public,custom}` for `SKILL.md`, parses metadata, and reads enabled state from extensions_config.json. **A directory containing `SKILL.md` is a self-contained leaf** — discovery does not descend into its subdirectories, so a skill *package* that bundles its own `examples/`/`references/` sub-skills (each with a SKILL.md) surfaces as **one** skill, not a swarm of phantom entries.
|
||||
- **Built-in seed source**: Product-owned defaults live under `app/gateway/routers/_*_seed_assets/skills/` and must be versioned even though runtime `skills/` trees are ignored. Seeders copy only when a target directory is absent and never overwrite administrator changes. Internal support packages such as `task-report-lib` deliberately omit `SKILL.md` and declare dependents in `.deerflow-skill-support.json`; keep `tests/test_seed_assets.py` and `tests/test_task_report_convert.py` synchronized with the shipped inventory and contracts.
|
||||
- **Online/offline lifecycle**: reuse `Skill.enabled`; do not add a parallel lifecycle flag and do not substitute `published` or curator `state`. `PUT /api/skills/{name}` is a global switch: administrators may toggle every skill, while an ordinary user may toggle only a non-built-in `custom` skill whose persisted `owner_user_id` matches that user. It patches only `skills.<name>.enabled` in the raw `extensions_config.json`, preserving MCP placeholders and unrelated/unknown fields, then reloads configuration and refreshes the lead prompt cache. Writes use atomic replacement on ordinary files and an `fsync` in-place fallback for writable Docker single-file bind mounts; supported launchers instead keep the runtime file under a writable persisted config directory. Offline skills remain on disk and in the full catalogue (`load_skills(enabled_only=False)`), and existing agent/workflow references are deliberately retained so bringing the skill online restores them. Every model-facing path must use the enabled catalogue: prompt injection, `search_skills`, `skill_list`, `skill_view`, subagents, AI-writing/deep-research/roundtable discovery, and Workflow run-start resource validation. Direct sandbox access is enforced by a copied online-only view under writable `DEER_FLOW_HOME`; `/mnt/skills` in LocalSandbox and AIO/remote sandboxes must point only to that view, never to the canonical tree. AIO creation/reuse attests the view marker and fails closed when an older provisioner ignores the requested mount; explicitly declared internal support libraries (such as `task-report-lib`) enter the view only while a dependent skill is online. Path-sensitive `DEER_FLOW_*` variables and `PYTHONPATH` are removed from LocalSandbox bash while non-path settings such as `DEER_FLOW_GATEWAY_PORT` remain. LocalSandbox host bash still shares the backend UID and is not an OS isolation boundary; production requiring hostile-code isolation must use an attested AIO sandbox. Frontend controls are in the administrator 技能管理 list and, for owners, 「技能 → 技能列表 → 我的」; offline rows and retained agent references remain visible with an 已下线 badge and can be removed, while new selectors exclude them.
|
||||
- **Online/offline lifecycle**: reuse `Skill.enabled`; do not add a parallel lifecycle flag and do not substitute `published` or curator `state`. `PUT /api/skills/{name}` is a global switch: administrators may toggle every skill, while an ordinary user may toggle only a non-built-in `custom` skill whose persisted `owner_user_id` matches that user. It patches only `skills.<name>.enabled` in the raw `extensions_config.json`, preserving MCP placeholders and unrelated/unknown fields, then reloads configuration and refreshes the lead prompt cache. Writes use atomic replacement on ordinary files and an `fsync` in-place fallback for writable Docker single-file bind mounts; supported launchers instead keep the runtime file under a writable persisted config directory. Offline skills remain on disk and in the full catalogue (`load_skills(enabled_only=False)`), and existing agent/workflow references are deliberately retained so bringing the skill online restores them. The per-skill incremental view sync must treat a missing destination as stale even when its target-cache entry is also absent, so re-enabling recopies the latest canonical package (including `SKILL.md` and nested helpers edited while offline) after the published copy was removed. Every model-facing path must use the enabled catalogue: prompt injection, `search_skills`, `skill_list`, `skill_view`, subagents, AI-writing/deep-research/roundtable discovery, and Workflow run-start resource validation. Direct sandbox access is enforced by a copied online-only view under writable `DEER_FLOW_HOME`; `/mnt/skills` in LocalSandbox and AIO/remote sandboxes must point only to that view, never to the canonical tree. AIO creation/reuse attests the view marker and fails closed when an older provisioner ignores the requested mount; explicitly declared internal support libraries (such as `task-report-lib`) enter the view only while a dependent skill is online. Path-sensitive `DEER_FLOW_*` variables and `PYTHONPATH` are removed from LocalSandbox bash while non-path settings such as `DEER_FLOW_GATEWAY_PORT` remain. LocalSandbox host bash still shares the backend UID and is not an OS isolation boundary; production requiring hostile-code isolation must use an attested AIO sandbox. Frontend controls are in the administrator 技能管理 list and, for owners, 「技能 → 技能列表 → 我的」; offline rows and retained agent references remain visible with an 已下线 badge and can be removed, while new selectors exclude them.
|
||||
- **Discovery↔delete consistency**: a skill is keyed for discovery by its frontmatter `name`, but its on-disk directory name may differ (hand-dropped / mis-extracted package). `LocalSkillStorage._resolve_skill_dir(name, category)` resolves the *discovered* directory (canonical `<category>/<name>` fast-path, else scan-by-frontmatter-name); `custom_skill_exists`/`read_custom_skill`/`delete_custom_skill` all go through it, so **anything that gets listed can also be read and deleted** regardless of directory name (the old code assumed `custom/<name>/` and 404'd "Custom skill 'x' not found." on any non-canonical layout). Delete is path-guarded to never escape the `custom/` root. Tests: `tests/test_skill_storage_delete.py`.
|
||||
- **Injection**: Enabled skills listed in agent system prompt with container paths
|
||||
- **Configurable `es_query` routing prompt**: `config.yaml → skills.es_query_routing`
|
||||
@@ -1225,10 +1612,11 @@ scheduler runs once per deployment. Per run:
|
||||
会商第二步(leader/seat)+ 第三步(summary/report/dashboard + 结构化抽取)单轮执行时,**某模型出问题就自动切到下一个模型继续**,并告知用户「『X』模型异常 → 已切换『Y』」。前台实时(`multi_agent.py`)与后台挂起(`engine.py` + `InProcessRoundtableGateway`)两条路径都覆盖。
|
||||
|
||||
- **可检测的模型失败**:`LLMErrorHandlingMiddleware` 在同一模型重试 3 次后**不抛异常**,而是返回兜底 `AIMessage` —— 现在给它打 `additional_kwargs[LLM_ERROR_MARKER="deerflow_llm_error"]=reason`(model_dump 保留,经 checkpointer 序列化仍在)。`is_llm_error_message(msg)` 据标记(回退按已知兜底文案正文)识别,返回 reason。
|
||||
- **候选模型链**:`app/gateway/roundtable_model_fallback.py::fallback_model_chain(current)` 按 `config.yaml models[]` 顺序给出 `[当前模型, …其余]`(去重保序)。`reason_text(reason)` 翻译成简短中文。总开关 env `ROUNDTABLE_MODEL_FALLBACK`(默认开,`0/false/off` 关 → 单模型旧行为);`ROUNDTABLE_MODEL_FALLBACK_MAX`(0=试完全部)。
|
||||
- **触发换模型的情形**:① 单轮 run 抛异常(start_run 失败 / idle 卡死 → `timeout`,其余 `exception`);② 兜底错误文案(`is_llm_error_message`);③ 交付无效(仅 seat/leader;leader 还含「既无正文又无派活」)。席位**只认剥掉 `<think>` 后的可见正文**——思考-only / 空串 / 断流都不算已交付(`deerflow.agents.roundtable_orchestrator.delivery`:`visible_seat_text` / `classify_seat_delivery`)。前台 `_special_run` 对此 yield `status:"invalid_delivery"` 且不广播;`GET /last-ai-message` 回 `{delivered, content, length, invalid_reason}`(思考-only 标 `thinking_only`,总控席位清单标 ⚠️)。后台引擎只把可见正文写入 `deliveries`,续轮用 `build_continue_prompt`:有无效席位时强制重派,禁止声称「已完成交付」。Step3 单例(report/summary/dashboard/ingest/结构化)的产物是落盘文件,**不**把「正文为空」当失败,只对真模型错误换模型。
|
||||
- **候选模型链**:保留真正模型故障后的自动换模,但“仅有思考 / 暂时无正文 / 空交付”不再判作模型故障,绝不据此切换用户选择的模型。`app/gateway/roundtable_model_fallback.py::fallback_model_chain(current)` 按 `config.yaml models[]` 顺序给出 `[当前模型, …其余]`(去重保序)。`reason_text(reason)` 翻译成简短中文。总开关 env `ROUNDTABLE_MODEL_FALLBACK`(默认开,`0/false/off` 关);`ROUNDTABLE_MODEL_FALLBACK_MAX`(0=试完全部)。
|
||||
- **触发换模型的情形**:① 单轮 run 抛异常(start_run 失败 / idle 卡死 → `timeout`,其余 `exception`);② 兜底错误文案(`is_llm_error_message`)。交付无效、仅有思考、空串以及普通总控轮既无正文又无派活**均不触发换模**;唯一例外是无工具的最终收口请求已经完整结束仍无正文,此时结果客观不可交付,才继续自动换下一个模型。席位仍只认剥掉 `<think>` 后的可见正文(`deerflow.agents.roundtable_orchestrator.delivery`:`visible_seat_text` / `classify_seat_delivery`):前台 `_special_run` 对无效交付 yield `status:"invalid_delivery"` 且不广播;`GET /last-ai-message` 回 `{delivered, content, length, invalid_reason}`(思考-only 标 `thinking_only`,总控席位清单标 ⚠️)。后台引擎只把可见正文写入 `deliveries`,无效席位继续走既有未交付处理,不切换模型。Step3 单例(report/summary/dashboard/ingest/结构化)的产物是落盘文件,同样只对真模型错误换模型。
|
||||
- **Step2 总控工具与可配置收口轮数**:普通总控使用真正白名单,只允许 `agent_orchestration` / `ask_clarification`;`write_file`、`present_files`、技能目录与文件/命令工具均不可见。`system_settings.json -> roundtable_room_limit.max_cycles`(默认 8,范围 2..100;管理员在设置弹窗「会商并发与收口」中修改)控制 recommend 模式到第几轮切换为显式 `finalize_only`:空工具白名单、注入已有有效席位全文、要求直接输出最终总结(失败席位如实标缺口),不能表现为“已停止”。一次会商启动时锁定当时值,运行中修改从下一次研讨生效;前台实时会商与后台作业读取同一设置。同一页的「会商文件流式兼容」(`write_file_compatibility_enabled`,默认关) 只作用于会商总控、席位、第三步和后台作业:全局文件流式兼容开着时会商一并使用;全局关着而本开关开着时,只有这些会商 run 改走 `buffered_gateway`。普通对话不读这个开关。chain / dag 的最终总结仍在既定席位完成后走同一纯正文路径;LangGraph 单轮 recursion limit 仍为 1000。
|
||||
- **后台**:`InProcessRoundtableGateway._run_turn_with_fallback(...)` 包住 `_run_agent_turn`,逐候选模型试;换模型落 `model_switch` 诊断、全失败落 `model_fallback_exhausted`。`run_leader/run_seat/run_report/run_summary/run_dashboard_data` 全走它(report/summary/dashboard 的产物自愈重试保留,并记住已切到的可用模型用于后续续写轮)。各 Turn/Result 带回 `switches: list[ModelSwitch]`,引擎 `_emit_switches` → `JobProgressWriter.model_switch(...)` 往研讨时间线插一条 `role="system"` 系统提示气泡。
|
||||
- **前台**:`_special_run`/`_leader_run` stream 收尾后检测失败 → yield `{status:"model_switch", role, agent_name, failed_model, next_model, reason}` 帧并用下一个模型重新流式;终态帧带 `model_used`。结构化抽取 `flow_extract.py` 的 round1 `model.astream` 也包了同款换模型循环(emit `event:"model_switch"` 帧)。
|
||||
- **前台**:`_special_run`/`_leader_run` stream 收尾后仅检测明确的模型错误标记;命中才 yield `{status:"model_switch", role, agent_name, failed_model, next_model, reason}` 帧并用下一个模型重新流式,思考-only/空正文不进入该分支;终态帧带 `model_used`。结构化抽取 `flow_extract.py` 的 round1 `model.astream` 也包了同款换模型循环(emit `event:"model_switch"` 帧)。
|
||||
- **Step2 单席位隔离**:`engine._run_recommend/_run_chain/_run_dag` 的每个席位执行包 try/except —— 一个席位彻底失败(落 `seat_failed` 诊断 + 置空交付)不拖垮整批/整链/整个作业。
|
||||
- **前端**:`multi-agent.ts` 解析 `model_switch` 帧 → `onModelSwitch` 回调(清空失败尝试的半截气泡 + 累计的 tool_call delta,phase 退回 connected);`useStep2Orchestration` 弹 toast + 时间线插「系统提示」气泡;`useStep3Summary/Report/Dashboard` + `flow-extract.ts` 重置对应气泡并在步骤卡留可见提示;后台路径经 `job-live.ts::toStep2Dialogue` 把 `role="system"` 渲染成 amber「系统提示」气泡。
|
||||
- Tests: `tests/test_roundtable_model_fallback.py`。
|
||||
@@ -1334,7 +1722,7 @@ Per-user runtime overrides (whitelisted fields) live at `{base_dir}/users/{user_
|
||||
- `subagents.enabled` - Master switch for subagent delegation
|
||||
- `memory` - Memory system (enabled, storage_path, debounce_seconds, model_name, max_facts, fact_confidence_threshold, injection_enabled, max_injection_tokens)
|
||||
- `auth_login` - Custom login modes layered on the base auth (`deerflow/config/login_config.py`):
|
||||
- `username_login.{require_password, password}` - **Shared** password gate for `POST /api/v1/auth/login/username`. When `require_password=true`, the (otherwise passwordless) username login additionally requires `?password=<shared-secret>`; empty `password` falls back to `123ewq`. Gate only ? it does not change which account logs in. **Per-account override (账号级登录口令)**: independent of this shared gate, an admin can assign **one specific account** an individual **plaintext** password via the 白名单管理 page (`users.login_password` column, migration `20260625_04`, nullable). When the target account carries a `login_password`, `/login/username` **requires** `?password=` to match it **exactly** and that personal password **takes precedence over** the shared gate (the shared secret will not unlock that account); accounts with no personal password keep the shared-gate / passwordless behavior. The check happens before auto-register via the lookup-only helper `_lookup_user_by_username` (so a brand-new username is unaffected). Surfaced/managed admin-only: `GET /api/v1/auth/users` returns `UserListItem.login_password` (plaintext — the list is already admin-gated), and `PUT /api/v1/auth/users/{user_id}/login-password` `{password: str|null}` sets (non-empty) / clears (null·empty) it without touching `token_version`. Plaintext storage is intentional (the password travels in the login URL query anyway, so it stays re-copyable). The user logs in via the existing `#/login/<username>?password=xxxx`. Tests: `tests/test_login_personal_password.py`.
|
||||
- `username_login.{require_password, password}` - Shared password gate for `POST /api/v1/auth/login/username`; blank configured secret falls back to `123ewq`. An account's own password/passphrase takes precedence. **Account login, registration and SSO policy (2026-10-05):** `system_settings.json -> authentication` stores `password_login_enabled=false`, `registration_enabled=false`, `single_session_enabled=false`, `sso_enabled=true` defaults; administrator GET/PUT `/api/system-settings/authentication` edits them immediately, and anonymous `/api/v1/auth/options` exposes the switches plus `needs_setup`. At least one login method must remain enabled; registration needs password login. Missing personal credentials use the existing shared passphrase (default `123ewq`), so disabling SSO does not require an administrator to first set a personal password. `/login/local` accepts username or email; `/register` accepts `{username,password}` (or historical `{email,password}`), always creates a regular user, and returns the full bearer login response. Names containing `admin` never create or promote administrators; persisted roles remain authoritative. Passwords and URL passphrases share the same versioned hash in `users.password_hash` and `users.login_password`; legacy plaintext passphrases remain verifiable and take precedence over an old independent password hash until reset. User `/change-password` and administrator `/users/{id}/login-password` (alias `/reset-password`) update both columns and atomically increment `token_version`; stale profile writes cannot overwrite credentials or revive sessions. Password removal is rejected. `/me.has_password` denotes personal credentials; `uses_default_password` marks shared-passphrase fallback, and `requires_current_password=true` requires the current shared passphrase even for the first password change. Self changes/resets return a replacement bearer and Cookie; administrator list items return `has_login_password`, never hashes or newly written plaintext (existing legacy plaintext remains admin-only for old link-copy compatibility). New link copies use only the reset input still in frontend memory. SSO policy also protects token/OAuth and the parallel panel `/api/parallel-agents/auth/token` compatibility entry, which now shares standard password/rate-limit/approval checks. Public first-admin initialization returns a full login response and inserts an `auth_initialization` singleton claim plus the first administrator in one transaction; the ORM automatically creates that table plus `auth_usernames`, whose unique username claim prevents concurrent registrations across email domains without rewriting historical users. Username SSO rechecks the final account credential after a concurrent registration or reset. When password login is enabled, username and parallel compatibility login require a personal or shared/default passphrase regardless of the old require_password flag. All verified issuing paths explicitly set JWT auth_method=password/passphrase/sso; its default remains None for legacy/identity-only sessions. Gateway and LangGraph share session_policy_error, rejecting unmarked sessions under enabled password login and SSO sessions when SSO is disabled. DEER_FLOW_AUTH_DISABLED cannot bypass an enabled password-login or single-session policy; disabling password login preserves legacy compatibility behavior. CLI `auth.reset_admin` rotates both credential columns. Single-session login (one online location per account) is the separate single_session_enabled=false policy, not external SSO. Every successful login route, including the parallel alias, registration and initialization, calls _prepare_login_session: the SQL repository atomically increments token_version and returns that transaction's current user before minting the JWT. Ordinary competing logins advance versions independently so only the latest survives, while credential/approval changes after verification cause 409 rather than authenticating a stale password. Failed logins never advance the version; when disabled, logins keep existing sessions. Password changes still always rotate the version. Tests: `tests/test_account_authentication.py`, `tests/test_username_login_domain.py`; contracts: `docs/AUTH_LOGIN.md`.
|
||||
- `token_login.{enabled, token_info_url, service_authorization, timeout_seconds, verify_ssl, ca_cert_path, mock_tokens}` - Token-exchange login (`POST /api/v1/auth/login/token`). The gateway POSTs to `token_info_url` with the caller-supplied token placed directly in the `Authorization: Bearer <token>` header (no cookie; the static `service_authorization` field is deprecated/unused, kept only for backward compat), reads `data.username` from a `{"state":"200",...}` body, then signs the matching local user in (auto-registering if absent). `token_info_url` is HTTPS ? `verify_ssl: false` skips cert verification for self-signed/internal UAT hosts, or set `ca_cert_path` to a PEM bundle (takes precedence, keeps verification on). **`mock_tokens` (offline test mapping `{token: username}`)**: when an incoming token matches a key, `_fetch_username_from_token` returns the mapped username and **skips the `token_info_url` call entirely** — for environments where the upstream `getTokenInfo` host is unreachable (e.g. external networks). Empty in prod (no effect); seeded `{"123ewq": "admin"}` in this deployment so `/login/任意?authToken=123ewq` signs in as admin. **`UsernameLoginResponse` now also returns `username`** (the raw resolved login name, pre email-normalization) on both `/login/token` and `/login/username`, so the frontend can append a faithful `?username=` on jump links (按钮管理 追加登录态) instead of the lossy email local-part. See [docs/AUTH_LOGIN.md](docs/AUTH_LOGIN.md) for the full request/response contract.
|
||||
|
||||
Both endpoints are whitelisted in `auth_middleware._PUBLIC_EXACT_PATHS`. The shared `_get_or_create_user_by_username()` helper (in `routers/auth.py`) backs both `/login/username` and `/login/token`.
|
||||
@@ -1370,6 +1758,29 @@ from the selected space names and hides generic recommendations while these
|
||||
questions are present. Design and remaining graph/profile phases are in
|
||||
`docs/KNOWLEDGE_SPACE_RETRIEVAL_AND_WIKI_EXPANSION_ZH.md`.
|
||||
|
||||
**Knowledge-base research bookshelf:** `/page/workspace/knowledge/:id/research`
|
||||
is ordinary Lead Agent Q&A constrained to one existing knowledge base; it does
|
||||
not invoke or reuse the legacy Deep Research workflow. Any account with read
|
||||
access may create a private session/draft, while only the existing KB writer
|
||||
may publish or overwrite its Markdown Wiki article. Published snapshots follow
|
||||
the KB's read permission, but the session/thread/draft and the continuation
|
||||
link remain creator-only (administrators do not inherit another user's private
|
||||
chat). Every run resolves the immutable thread binding from
|
||||
`knowledge_research_sessions` (registered on `deerflow.persistence.models` so
|
||||
startup `create_all` builds the table; `get_by_thread` treats a missing table
|
||||
as no binding so ordinary `/runs/stream` is not aborted), reauthorizes KB read access, accepts exactly one
|
||||
text-only human turn, forces classic single-KB retrieval and disables tools,
|
||||
memory, prompt prefixes, planning and subagents; fixed-answer shortcuts are
|
||||
bypassed. Saving manually upserts a stable
|
||||
`knowledge-research/{publication_id}` Wiki slug under the reserved root
|
||||
「深度研究」 without enabling automatic Wiki generation, uses a saved hash for
|
||||
optimistic overwrite checks and idempotent retries, and keeps the editable
|
||||
publication title separate from the Q&A title. REST endpoints live in
|
||||
`app/gateway/routers/llmwiki.py`, runtime enforcement in
|
||||
`app/gateway/knowledge_research.py`, persistence/migration in
|
||||
`deerflow.persistence.knowledge_research` / `20261022_01`, and regression
|
||||
coverage in `tests/test_knowledge_research.py`.
|
||||
|
||||
**Embedding ownership:** WeKnora's per-knowledge-base `embedding_model_id`
|
||||
indexes uploaded source documents; `llmwiki.local_wiki_index.embedding` indexes
|
||||
the generated Wiki-page mirror inside DeerFlow. Do not couple these two model
|
||||
@@ -1379,15 +1790,33 @@ the selected active model id. When the caller omits a choice, retain WeKnora's
|
||||
active-default behavior. Existing KBs cannot change this binding in place, and
|
||||
cross-KB WeKnora searches should use model-compatible knowledge bases.
|
||||
|
||||
Wiki-enabled knowledge bases accept two bounded, user-owned prompt extensions:
|
||||
`wiki_config.content_instructions` (generated-page language, structure and
|
||||
emphasis) and `wiki_config.extraction_instructions` (domain entities/concepts
|
||||
to prioritize). The create/edit dialogs expose both values; Gateway accepts
|
||||
only those two fields and `WeKnoraClient` merges edits into the remote
|
||||
`wiki_config` so `synthesis_model_id`, extraction granularity and WeKnora-owned
|
||||
runtime knobs are never dropped. WeKnora's upload/reparse Wiki ingest is the
|
||||
actual consumer: saving instructions affects later ingests and does not itself
|
||||
rewrite existing pages.
|
||||
Wiki-enabled knowledge bases accept bounded user-owned content/extraction
|
||||
instructions plus two independent optional generation inputs:
|
||||
`wiki_config.directory_paths` (a one-to-three-level reference structure) and
|
||||
`wiki_config.focus_instructions` (a strict extraction scope). The raw values
|
||||
and normalized reference tree live in `llmwiki_knowledge_bases`; the effective
|
||||
prompt tells WeKnora to use the directory names and hierarchy when they fit the
|
||||
source, while allowing it to add, merge, rename, or omit nodes according to the
|
||||
actual evidence. Directory guidance is never an allowlist and never a fact
|
||||
source. `WeKnoraClient` merges edits into the remote `wiki_config` so synthesis
|
||||
models and provider runtime knobs are retained.
|
||||
|
||||
Creation does not pre-create the reference skeleton. Mutation-settle and
|
||||
periodic reconciliation run only for the strict focus scope: all page decisions
|
||||
complete before any deletion, focus-outside generated pages are removed, and
|
||||
the directory layout inferred by WeKnora is preserved. An incomplete or invalid
|
||||
classifier result makes no destructive change and is retried. Directory-only
|
||||
configuration therefore needs no post-generation reconciliation. The current
|
||||
`directory_paths` field and the legacy `wiki_parent_directories` field have the
|
||||
same prompt-only reference semantics; neither may trigger folder creation,
|
||||
movement, deletion, or periodic directory reconciliation. Empty inputs keep
|
||||
WeKnora's automatic behavior. Schema version 3 identifies this reference-only
|
||||
prompt semantics. The background coordinator recompiles managed schema-2 rows
|
||||
from their database-owned raw inputs and advances them only after the WeKnora
|
||||
update succeeds; failures remain on schema 2 for retry.
|
||||
Regression coverage:
|
||||
`tests/test_wiki_directory_constraints.py` and
|
||||
`tests/test_llmwiki_parent_directory_routing.py`.
|
||||
|
||||
Ordinary knowledge-base cards expose an owner/admin-only name/description/Wiki-instructions edit dialog in `WeKnoraKnowledgeWorkspace.tsx`, reusing `useLlmWikiActions().update` and `PATCH /api/llmwiki/knowledge-bases/{id}`. The route trims names, rejects blank or reserved conversation-deposit names, writes to WeKnora before updating local metadata, and retains existing write authorization. Default test questions are derived from the authoritative remote display name. The editor omits `test_questions` unless the user actually changes that field, and `knowledge_profile.generated_test_questions_name` recognizes only the exact legacy four-question template as automatic; genuinely edited lists remain custom. `_mapping_view` must resolve both questions and `test_questions_customized` from a profile row containing the current remote name, so a rename immediately replaces old-name defaults without a migration. A successful remote list is authoritative for every list scope, including `personal`: absent remote IDs are hidden without deleting mappings/references. Upstream failures remain errors. The grid also filters cached `remote_status=missing` rows and no longer renders stale-record cleanup cards. Regression coverage: `tests/test_llmwiki_knowledge_management.py`, `tests/test_llmwiki_weknora.py`.
|
||||
|
||||
@@ -1588,6 +2017,20 @@ See [docs/plan_mode_usage.md](docs/plan_mode_usage.md) for details.
|
||||
### Context Summarization
|
||||
|
||||
Automatic conversation summarization when approaching token limits:
|
||||
- **Administrator recent-turn window:** `system_settings.json` →
|
||||
`conversation_context.recent_turn_window_enabled` is a deployment-wide,
|
||||
default-off alternative to summarization. Administrators manage it in
|
||||
「设置 → 上下文管理」 through
|
||||
`GET/PUT /api/system-settings/conversation-context`. The same block stores
|
||||
`max_conversation_rounds` (default 20, range 1–50), which `ChatPage` uses as
|
||||
the maximum number of real user turns in one ordinary chat; missing values in
|
||||
older settings files retain the 20-turn default. When enabled, each lead
|
||||
agent model request retains only the current turn and the two preceding real
|
||||
user turns, together with their assistant/tool continuations and any hidden
|
||||
context attached to the retained boundary. The operation is request-only:
|
||||
checkpoints and the transcript shown in the UI remain complete. This mode
|
||||
takes priority over the per-chat 「上下文压缩」 toggle and never invokes the
|
||||
summary model.
|
||||
- **On/off switch is per-request, not the config file.** The chat input box has a
|
||||
「上下文压缩」 toggle (below 参考文献, **default off**) → frontend sends
|
||||
`context.summarization_enabled` → whitelisted in `_CONTEXT_CONFIGURABLE_KEYS`
|
||||
@@ -1622,8 +2065,9 @@ Automatic conversation summarization when approaching token limits:
|
||||
errors are non-retriable so a tokenizer mismatch cannot trigger repeated
|
||||
submission of the same oversized request.
|
||||
|
||||
Tests: `tests/test_summarization_toggle.py` (gating + whitelist),
|
||||
`tests/test_summarization_middleware.py` (cache behavior).
|
||||
Tests: `tests/test_prompt_prefix.py` (system-setting API/defaults) and
|
||||
`tests/test_summarization_multiturn.py` (summarization plus transient recent-turn
|
||||
window behavior).
|
||||
See [docs/summarization.md](docs/summarization.md) for details.
|
||||
|
||||
### Vision Support / Image Recognition (??)
|
||||
@@ -1664,6 +2108,23 @@ applies when the short-lived signed WeKnora embed cookie is valid. Keep static
|
||||
path recognition in `app/gateway/weknora_embed.py` synchronized with the
|
||||
proxy's static fallback paths in `app/gateway/routers/llmwiki.py`.
|
||||
|
||||
Static assets use a separate signed-cookie-only path: never fetch an upstream
|
||||
admin session or query per-KB ACLs for public JS/CSS/fonts. HTML and data APIs
|
||||
still revalidate mapping permissions. `app/gateway/weknora_static.py` coalesces
|
||||
concurrent reads and bounds its LRU by bytes and entries; cache keys include
|
||||
the upstream web URL, public prefix, asset path and query. Bundle rewriting,
|
||||
gzip and ETag generation run in a worker thread. Fingerprinted `/assets/`
|
||||
files receive immutable browser caching; `/config.js` and upstream private,
|
||||
no-store, no-cache or identity-varying responses must not enter the shared
|
||||
cache. Explicit browser cache bypass refreshes the cached representation.
|
||||
|
||||
The injected iframe guard observes root theme changes separately and ignores
|
||||
its own writes. Process only changed content branches, coalesced per batch;
|
||||
ordinary loaded pages must have no recurring DOM-maintenance timer. Source
|
||||
detail deep links retain a bounded nine-second drawer fallback. Preserve the
|
||||
inner Wiki directory, protected images and theme repair. Regression tests:
|
||||
`tests/test_weknora_static.py`, `tests/test_weknora_embed_script.py`.
|
||||
|
||||
### Position-roundtable role directory
|
||||
|
||||
`position_roles` is the global shared directory for position-roundtable work
|
||||
@@ -2137,7 +2598,10 @@ deletable because it owns full-library aggregation.
|
||||
Assistant-knowledge reads keep the landing and article paths size-bounded.
|
||||
`list_bases()` is metadata/statistics-only and must never call the lazy global
|
||||
alignment pass: imports align their snapshot before commit, while content reads
|
||||
and search retain `_ensure_base_aligned`. The Wiki directory calls
|
||||
use the last committed snapshot without waiting for the global import/alignment
|
||||
locks; search and export retain `_ensure_base_aligned` for legacy data. Directory,
|
||||
article, source and import-job reads resolve their base with `include_stats=False`
|
||||
so loading/polling does not recount every content table. The Wiki directory calls
|
||||
`GET /api/assistant-knowledge/wiki-pages?include_content=false`; repository
|
||||
`load_only` projection must exclude both `content_markdown` and `search_text`
|
||||
(response-only deletion is insufficient), and the frontend loads only the
|
||||
@@ -2181,8 +2645,8 @@ Both Gateway document APIs and successful content mutations through the
|
||||
embedded WeKnora proxy schedule one per-mapping debounced retry sequence with
|
||||
bounded 1/8/30/90/300-second backoff. Never cancel an in-flight sync to debounce
|
||||
a newer mutation; wake the coordinator, update the local generation, and let
|
||||
the database lease deduplicate workers. While WeKnora reports processing, poll
|
||||
only the cheap knowledge-base state and retain both a post-processing scan and
|
||||
the database lease deduplicate workers. While WeKnora generates Wiki content,
|
||||
poll both the knowledge-base state and per-document pending subtasks, then retain a post-processing scan and
|
||||
one short quiet confirmation. Before retiring missing pages or replacing graph
|
||||
edges, the sync service must re-read processing state; processing or an
|
||||
unverifiable state produces a retryable partial result and preserves the prior
|
||||
@@ -2306,6 +2770,31 @@ Gauss compatibility contract. Treat both `postgresql` and `opengauss` dialect
|
||||
names as timezone-aware PostgreSQL-family engines in portable types and
|
||||
dialect-specific session settings.
|
||||
|
||||
### openGauss Dolphin / MySQL-protocol mode
|
||||
|
||||
The offline deployment may select `DATABASE_TYPE=gauss_mysql`. Its renderer
|
||||
keeps `database.backend=mysql` and sets `DEERFLOW_DATABASE_PRODUCT=gauss_mysql`;
|
||||
this explicit product marker is the switch because SQLAlchemy cannot safely
|
||||
distinguish Dolphin from MySQL before dialect initialization. Only that mode
|
||||
installs `deerflow.persistence.gauss_mysql_compat` for the configured async
|
||||
driver. An unset marker, or `mysql`, must retain the stock MySQL dialect and
|
||||
schema-maintenance behavior unchanged.
|
||||
|
||||
Dolphin table existence and additive schema checks use `current_schema()` plus
|
||||
the native `pg_catalog` through `gauss_schema.ensure_gauss_schema_sync`.
|
||||
Never run the generic MySQL `SHOW CREATE TABLE` reflection sweep, MySQL JSON
|
||||
projection backfill, MySQL session-timeout variables, or application-side
|
||||
`CREATE DATABASE` retry in this mode: these paths can misread the schema or
|
||||
abort the openGauss transaction with error 322. Database/user/schema bootstrap
|
||||
remains the offline deployment layer's responsibility. The startup DDL block
|
||||
is still best-effort under the global startup-schema invariant and always
|
||||
releases its host-local cross-worker lock.
|
||||
|
||||
`PortableJSON` writes strict JSON for both products. Its legacy decoder is
|
||||
available only on a dialect class carrying the Dolphin compatibility marker;
|
||||
ordinary MySQL must continue to reject malformed/non-JSON values rather than
|
||||
silently applying the Dolphin fallbacks.
|
||||
|
||||
## 2026-09-14 我的项目与知识库验收修复
|
||||
|
||||
- 简洁模式的「我的项目」列表、策略工作台和普通问答项目恢复工作区功能导航;窄屏提供展开菜单按钮。
|
||||
@@ -2434,3 +2923,24 @@ calls. The active sandbox skill mirror must be synchronized with the canonical c
|
||||
Gateway detail writes emitted by `fs-collect` and `fs-detail-submit` log a token-masked request and
|
||||
the corresponding response so an interrupted chain can be distinguished from a failed persistence
|
||||
call without exposing the gateway bearer token.
|
||||
|
||||
### Task-detail custom skill password authentication (2026-10-26)
|
||||
|
||||
`skills/custom/task-context` and `skills/custom/task-detail-update` authenticate detail reads/writes
|
||||
with a short-lived token minted by `POST /api/v1/auth/login/local`, using the runtime-only
|
||||
`TASK_DETAIL_AGENT_USERNAME` / `TASK_DETAIL_AGENT_PASSWORD` pair and `TASK_DETAIL_GATEWAY_URL`.
|
||||
They must never use general `DEERFLOW_AUTH_TOKEN` or `TOOLS_TOKEN`, nor emit the password or JWT in
|
||||
an input, trace, or model-visible output. `/api/parallel-agents/auth/token` is only a legacy fallback:
|
||||
it follows `sso_enabled` and returns 403 when external SSO is disabled. Password-login failures other
|
||||
than the policy 403 must fail closed; do not fall back to identity-only token minting. Keep the
|
||||
canonical custom directories and `.deer-flow/.sandbox-online-skills` mirror synchronized through the
|
||||
custom-skill reconciliation flow. `scripts/setup_biz_detail_assistant.py` logs in through the same
|
||||
formal endpoint and accepts `DEERFLOW_SETUP_BASE`, `DEERFLOW_SETUP_USERNAME`,
|
||||
`DEERFLOW_SETUP_PASSWORD` for deployment overrides; it may fall back to the legacy username route
|
||||
only for the formal login policy 403, never for a credential or rate-limit failure.
|
||||
|
||||
`biz-detail-assistant` must retain the native `ask_clarification` tool. Do not special-case it by
|
||||
adding that tool to `requested_excluded_tools`, and do not mount `DetailConfirmMiddleware` to forge
|
||||
confirmation cards from a plain model response: those cards are not genuine model tool calls and break
|
||||
the standard pause/resume interaction. The final `ClarificationMiddleware` is the single authority for
|
||||
all missing-information, scope, approach and risk-confirmation questions.
|
||||
Reference in new issue
Block a user