wsk/deerflow-code
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

sync update from LAN repo (source master @ ca48c4ec)

Browse Source
This commit is contained in:
gozeros committed 2026-11-05 10:53:54 +08:00
1 parent 8bc8b0e082
commit ea9dcff9c0
873 files changed
+107866 -14011
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

offline-backend-20260512/.env
+1 -1
View File
@@ -18,7 +18,7 @@ SERPER_API_KEY=your-serper-api-key
# GATEWAY_CORS_ORIGINS must include the exact frontend origins, separated by commas.
AUTH_JWT_SECRET=i3_BiBK44sYHiFIkChhDs46y57ZewJKO3diQDw01anI
GATEWAY_CORS_ORIGINS=http://localhost:5173,http://127.0.0.1:5173,http://localhost:5174,http://127.0.0.1:5174,http://localhost:5183,http://127.0.0.1:5183,http://localhost:4200,http://47.88.25.99:7010
GATEWAY_ENABLE_DOCS=false
GATEWAY_ENABLE_DOCS=true
GATEWAY_WORKERS=1
# Only effective when config.yaml runtime_performance.asyncio_thread_pool_enabled=true.
DEER_FLOW_ASYNCIO_THREAD_POOL_SIZE=100
offline-backend-20260512/backend/.dockerignore
+2
View File
@@ -15,3 +15,5 @@ logs
docker-data
dist
wheelhouse
.worker-pool
**/worker-control
offline-backend-20260512/backend/CLAUDE.md
+311 -8
View File
@@ -1,5 +1,232 @@
# CLAUDE.md
## Cold agent construction (2026-10-30)
`runtime/runs/worker.py` dispatches opted-in synchronous factories through
`build_agent_with_timing`, backed by `utils/background_init.py`'s
`run_background_init` and one shared `FileIOPool(1, 1, "agent-init")`.
`make_lead_agent._deerflow_background_build = True` opts the built-in Lead Agent
into background construction; unmarked custom factories retain their caller-loop
contract. Serial admission prevents concurrent SDK cold imports and keeps waiting
builds out of authentication's executor. The synchronous timing helper remains
available. Model execution, checkpoint/store attachment and streaming stay in the
original run task; no graph is shared or cached across runs/users.
Every async Lead Agent entry point uses this builder, including non-streaming
and streaming open-chat endpoints and Workflow agent nodes. Do not call the
synchronous factory directly on their event loop: it can block SDK import locks
or deadlock while another background build awaits MCP discovery on that loop.
Async tool consumers (AI writing skill agents and `task_tool`) use
`get_available_tools_async`; deep-research capabilities and job execution also
construct their material providers through `run_background_init`. Keep the
common helper in `utils` with a lazy MCP import to avoid runtime/agent/tools
import cycles. Synchronous client/provider APIs remain available.
Copy the caller's ContextVars into construction and restore the successfully built
context before streaming, including a deferred-tool registry reset to `None`.
The `mcp_initialization_loop` context manager forwards lazy MCP discovery from the
worker to the owner event loop; exit it before copying back context. MCP cache
invalidation and legacy synchronous callers retain their existing semantics.
Cancellation drains the actual worker before releasing capacity, never starts the
graph afterwards, and preserves terminal status/SSE cleanup. Check `abort_event`
after construction as well as honoring task cancellation.
Regression: `packages/harness/tests/test_agent_build_responsiveness.py`,
`test_agent_build_timing.py`, `test_mcp_background_build.py` and the run-finalization
tests; `tests/test_async_agent_entrypoints.py` and
`tests/test_async_tool_initialization.py` exercise overlapping HTTP/workflow/tool
initializers. The HTTP/workflow cases run in subprocesses so an event-loop
deadlock fails with a bounded timeout. Cold profiling found ~4.7 s in model SDK imports/client setup, compared with
~14 ms in skill loading; moving only disk reads would not address this stall.
Local live cold acceptance (2026-10-30, official MiniMax-M2.7, tools enabled):
the first build took 4.28 s while a second request received stream metadata in
402 ms. During the conservative interior of the build window, 43 health/auth
probes completed successfully; across all three conversations, health/auth
P95 latency was 28/73 ms, with no event-loop-stall warning. Every answer ended
normally and all synthetic threads were removed. Initialization/model latency
still contributes to first-answer time; this change prevents blocking the server
loop, rather than promising immediate model output.
A separate live read-only tool check uploaded a random marker, allowed only
`read_file`, and confirmed the tool result and final answer reproduced that marker
without placing it in the prompt; the stream ended normally and the thread was removed.
Worker compatibility: no HTTP/SSE, task payload, auth or database contract changes.
Running workers without reload retain old construction behavior; the new factory
marker is harmless to old executors. `worker_pool/revision.py` hashes only the
adapter directory and does not identify this harness patch. This change does not
alter the controller's existing snapshot/ImageID-based drain-and-replace policy
or `on_disconnect=cancel` behavior: restarting a production controller can still
rotate idle workers or disconnect proxied streams. Do not infer uninterrupted
worker operation from these process-local initialization changes. The live local
acceptance instance has no enabled worker controller and does not touch remote
workers. Existing worker routing/admission/storage/discovery/diagnostics/launcher
compatibility regressions: 68 passed.
Final combined regression (2026-10-30): all 58 newly added responsiveness,
cancellation, MCP-entrypoint and provider-lifecycle cases plus the 68 existing
worker compatibility cases passed together (126 passed, no skips).
The live recheck uploaded and converted 20 DOCX files while another conversation
produced 43 visible text increments; all 172 concurrent authenticated probes
succeeded (P95 105 ms, max 436 ms). A fresh-process cold build took 10.92 s while
617 heartbeat samples continued (P95 scheduling delay 25 ms, max 146 ms).
The existing backend and child workers were not restarted for this recheck;
later entrypoint changes were verified through isolated HTTP/workflow tests.
The full versioned backend/harness run completed with 3140 passed, 39 failed,
4 skipped and 1 deselected. Against baseline `47cfed02`, 30 failures also failed
on HEAD and the other 9 passed in isolated reruns on both versions; no current-only
stable failure remained. The deselected legacy
`test_cancelled_history_closes_iterator` hangs on both versions: its fake saver
implements `alist`, while `limit=1` takes `aget_tuple`, so its unbounded entry
event wait never completes. This is not a claim that the repository's entire
test suite is green. Changed Python files introduced no Ruff diagnostics.
## Upload responsiveness (2026-10-30)
`app/gateway/routers/uploads.py` runs each staged upload transaction in a dedicated
`FileIOPool(2, 2, "uploads")`, including 256 KiB spool copies, signature validation,
optional synchronous conversion, sandbox acquisition/sync, commit, rollback and
cleanup. Resolve the thread owner asynchronously before dispatch and propagate
contextvars into workers. Waiting batches consume no worker from authentication,
artifacts or event streaming. Per-event-loop directory locks serialize same-thread
batches and listing/deletion through rollback/cleanup; keep them held until a
cancelled worker stops. Listing/deletion use the artifact file pool off the loop.
Cancellation signals a cooperative event, checks between file chunks and around
commit/sync, and waits for real completion before multipart handles may close.
The response contract and upload-policy enforcement remain unchanged.
`sandbox/sandbox_provider.py` serializes initial singleton construction with
reset/replacement/shutdown. Concurrent cold uploads must not create competing
providers with independent sandbox reconciliation and idle-cleanup threads.
Regression: `packages/harness/tests/test_sandbox_provider_concurrency.py`.
`file_conversion.py` also isolates its asynchronous conversion/extraction callers
in a separate two-thread pool. PDF/Office parsing runs in a lazily initialized,
two-process pool using `spawn` on Windows and Linux, so Python parser CPU work
does not contend for the gateway GIL. Pass only file paths and converter choices;
bound returned text inside the child and write sidecars in the owning parent
worker. The global parser semaphore bounds sync/async callers together. A broken
process pool fails the current conversion and is rebuilt on the next call.
`FileIOPool.run_until_complete` provides non-abandoning cancellation for workers
that own temporary input files; cancellation cannot close/delete an input still
being parsed by a child process.
Regression: `tests/test_upload_responsiveness.py`,
`tests/test_conversion_responsiveness.py`, `tests/test_uploads_policy_enforcement.py`.
Local acceptance (Windows, Python 3.12, synthetic files via ASGI; no delay injected):
20 x 900 KiB TXT uploads reduced first concurrent probe delay from 664/682 ms to
3/4 ms. With real MarkItDown conversion, 20 x 800-paragraph DOCX uploads reduced
maximum probe scheduling lag from 286/285 ms to 41/14 ms; all 20 sidecars matched
the source text. The first spawned reader took 13.7 s to initialize/convert in
this environment, while concurrent probe lag stayed below 16 ms. These are local
responsiveness measurements, not production throughput guarantees. Upload,
policy, conversion, large-file and runtime-contention regressions: 87 passed.
Full-suite collection is separately blocked by the pre-existing missing
`deerflow.models.global_prompt` import in `tests/test_global_prompt.py`.
## Thread metadata query performance and ORM deadlines (2026-10-22)
`ThreadMetaRepository._query_conditions` is shared by search/count: common
SQLite scalar predicates and PG/MySQL string predicates plus system-thread
exclusion run before SQL pagination/counting. Preserve AUTO owner resolution and
the nonempty taskId owner bypass while applying agent_id/actionId together.
`json_compat.JsonPredicate` binds values, preserves local missing/null and numeric
equality on SQLite, and uses binary MySQL id equality and Unicode-safe blank
seat detection. Nested/unusual-key filters and PG/MySQL numeric filters remain exact
residual filters; openGauss retains its original path. Never drop an unsupported
predicate or replace this with upstream's unadapted SQLite/PG-only compiler.
PG's omitted pool_recycle defaults to 300s (other backends remain 1800s);
database.command_timeout is finite positive or null, default 30s. The shared
pooled-engine kwargs are used on initial creation and auto-create retry. Only
asyncpg receives command_timeout; this does not configure psycopg checkpointers.
Regression entries: packages/harness/tests/test_thread_metadata_sql_filters.py,
test_database_pool_deadlines.py and the existing test_thread_operation_timing.py.
Master integration (2026-10-23) retains the maintained task/agent/action index
columns as additional SQL narrowing before exact JSON predicates. Keep dedicated
task deep-link card synchronization on every existing metadata mutation. Hydrated
history still uses slim_display to bound oversized tool payloads without editing
checkpoint contents; frontend retrieval-progress scopes remain isolated per turn.
Thread POST defaults to metadata-only creation; only an explicit
initialize_checkpoint=true writes an empty checkpoint. Never enqueue an empty
checkpoint in the background: it could become the head after the real first run.
Authorized metadata-only threads have empty state/history. Metadata GET defaults
to the SQL row (running maps to busy); include_state=true and checkpoint-only
legacy rows retain state reads. Owner decorators run before unscoped row reads.
History explicitly uses root checkpoint_ns and closes alist before serialization;
only the latest head carries full channel values.
The frontend uses BackgroundHistory as the SDK's external history adapter. Its
post-stream mutate immediately captures streamed state with no checkpoint id;
explicit end triggers background hydration. EOF/error still requires existing
run confirmation/rejoin. Every async write is fenced by thread/run/epoch/request;
new turns, navigation and unmount invalidate prior requests even if abort is
ignored. Reject history missing the newest streamed turn, bound each request to
15 seconds and retry completion sync once. Failure keeps the streamed snapshot.
Normal explicit-end callbacks use the completed streamed snapshot before the
next send can change page refs; background hydration never repeats them. Recovery
may have missed final SSE frames, so its callback waits for guarded hydration or
the retained snapshot fallback. New submits use checkpoint:null to follow the
latest server head.
Auth's SQLiteUserRepository is a historical name: its injected session factory
is the shared business ORM. With database.backend=mysql, login/auth/users and SQL
business repositories use MySQL, independently of checkpoint configuration.
LangGraph Store still follows checkpointer config and startup awaits runtime
setup/migrations; do not claim total service/startup isolation from PG or migrate
existing stores implicitly. See docs/THREAD_LATENCY_OPTIMIZATION_ZH.md for the audit.
Additional regression entries: test_thread_lazy_checkpoint.py,
tests/test_account_authentication.py and frontend pnpm test:thread-history.
## Multi-user contention controls (2026-10-22)
- `utils/file_io.py` separates bounded artifact IO (4 workers / 8 admitted) and
event disk IO (2 / 4) from the default executor used by authentication.
Contextvars propagate; cancellation cannot release admission until disk work
actually stops. Heavy artifact listing, ZIP previews, sniffing and writes use it.
- `runtime/runs/manager.py` sweeps completed live records after 300 seconds,
200 candidates per pass. It requires the actual task to be done and verifies
status/error/metadata/kwargs in the backing RunStore, updating a final snapshot
when necessary. Failed persistence retains the record for later retry. It never
deletes RunStore rows, checkpoints, events or stream buffers. `watch_completion`
starts grace after finalization, and lifespan closes the sweeper. The live
superseded fence survives a newer record's retirement; finalization never adds
a business-database history read to its completion fence.
- Run routes authorize the thread before unscoped persisted lookups, and verify
that the loaded run belongs to that exact thread. Preserve private-owner 404s,
taskId/system-thread sharing and existing action/agent partitions. Completed
rejoin without a live bridge buffer emits end immediately; Last-Event-ID replay
during the existing replay window remains untouched.
- `run_events.backend=memory` is a compatibility alias for
`SpoolingRunEventStore`, stored under `runtime_home()/run-events/events.sqlite3`.
Per-thread sequence assignment and each batch commit are transactional; paged
queries do not retain completed events in RAM. No automatic event/message
deletion or truncation is introduced. The pure `MemoryRunEventStore` remains
available for unit tests. Explicit db/JSONL formats are unchanged; JSONL disk
operations now use the event executor and striped per-thread locks. Keep the
spool and its WAL on the persistent local runtime volume. Do not reclassify it
as a PG checkpoint or silently change explicit db configuration.
- `runtime/fair_queue.py` provides bounded round-robin admission by user for the
subagent scheduler (3 total / 2 per user) and background roundtables (4 / 2).
Queued jobs consume no worker; subagent deadlines include queue time. Per-user
pending capacity also prevents one user from exhausting the entire backlog.
Actual coroutine completion, including cancellation before its first step,
acknowledges slot release. Parent context/identity propagation stays intact.
`RuntimeContentionConfig` defines optional budgets; limits apply per process
and require restart. The older roundtable per-user environment override remains.
- Both internal roundtable creation helpers only create metadata. Init does not
append a synthetic seat message via loopback; `_leader_run` injects authoritative
IDs on each real input, deriving them from the thread mapping for older clients
without seat_meta. Background first-leader injection remains. Never schedule a
late empty checkpoint write: it could overwrite real history.
Regression entries: `test_runtime_contention.py`, `test_completed_run_retention.py`,
`test_thread_lazy_checkpoint.py`, `test_run_completion_timeout.py`,
`tests/test_position_roundtable_sessions.py`, roundtable role/model/seat tests,
artifact/auth/model concurrency tests and frontend `pnpm test:thread-history`.
See `docs/MULTI_USER_CONTENTION_AUDIT_ZH.md` for measured scope and remaining
model/SQL/checkpointer contention. Process-local run control still requires the
current single-worker deployment; these queues do not enable multi-worker routing.
## Ordinary knowledge smart retrieval dual surface (2026-10-16)
`SmartReasoningRetrievalMiddleware` treats knowledge retrieval as an ordinary
@@ -120,7 +347,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
### Ordinary Q&A map visualization
The ordinary lead-agent prompt includes `_ORDINARY_QA_VISUALIZATION_SECTION` only for normal, non-bootstrap, non-writing, non-notebook, non-scheduled runs and defines the highest-priority output-carrier gate. Only an explicit current-request ask for HTML, a webpage/page, dashboard, large-screen view, or `.html` file permits reading `page-report-generation`, calling `page_report_template`, or writing HTML. “Draw a flowchart/chart/map/mind map” and “make a visualization” never imply HTML: ordinary flowcharts must be inline Mermaid, ordinary charts keep the established message-inline path, and requested maps emit strict JSON in a `map-json` fence. A mind map is emitted only for an explicit current-request ask for a mind map/knowledge map (or `analysis-mindmap`); ordinary analyses, summaries, outlines, flowcharts, and generic JSON must never auto-upgrade into one. Its strict `mindmap-json` contract is a bounded rooted tree (`id`, `parentId`, `label`, semantic `kind`, optional detail/source refs), targeting 12–24 nodes with a hard cap of 36, at most four layers including the center topic, at most six top-level dimensions and three children beneath every later node. Models must consolidate related facts into `detail`/`sourceRefs`, not make one node per report bullet, datum, paragraph, or reference. An unfinished dedicated `mindmap-json` fence (and compatible historical `mindmap` fence) remains an ordinary streaming code block; only the completed, valid object lazy-renders as a compact left-to-right message card. Clicking a parent node, or its +/− control at the outgoing connection point, expands/collapses its child tree and the canvas uses an explicit visible edge stroke. The full dialog keeps its close and view tools in the top-right and a layout-derived all-map preview in the bottom-right; the message preview also permits pan, wheel zoom, and zoom/reset controls. The message card's icon-only download control opens PNG/SVG/JSON choices. Generic JSON never invokes this renderer. Ambiguity defaults to inline output. Once the HTML gate passes, delivery is mandatory as a sandbox-generated `/mnt/user-data/outputs/*.html` artifact opened with `present_files`; a full inline `html` fence is forbidden as a substitute. The shared frontend `MarkdownContent` keeps its exact legacy branch when no map-shaped fence exists; dedicated `map-json` fences and complete generic `json` fences that contain map fields and pass the full map schema are validated and lazy-rendered from `components/workspace/visualizations/`. Map/loading/error cards use the full message-content width so their edges align with the step panel. Non-map and incomplete generic JSON retain the ordinary code-block path.
The ordinary lead-agent prompt includes `_ORDINARY_QA_VISUALIZATION_SECTION` only for normal, non-bootstrap, non-writing, non-notebook, non-scheduled runs and defines the highest-priority output-carrier gate. Only an explicit current-request ask for HTML, a webpage/page, dashboard, large-screen view, or `.html` file permits reading `page-report-generation`, calling `page_report_template`, or writing HTML. “Draw a flowchart/chart/map/mind map” and “make a visualization” never imply HTML: ordinary flowcharts must be inline Mermaid, ordinary charts keep the established message-inline path, and requested maps emit strict JSON in a `map-json` fence (markers use `name`+`coordinate` plus optional `color`; lines are included only when the user explicitly asks to connect places, must use `coordinates` with at least two `[lng,lat]` points, not `from`/`to`/`coords`, and default to `arrow`/`flow`). The frontend recovers those common aliases, assigns distinct marker colors, draws requested lines as looping SVG arrows, and drops unusable lines so markers still render. A mind map is emitted only for an explicit current-request ask for a mind map/knowledge map (or `analysis-mindmap`); ordinary analyses, summaries, outlines, flowcharts, and generic JSON must never auto-upgrade into one. Its strict `mindmap-json` contract is a bounded rooted tree (`id`, `parentId`, `label`, semantic `kind`, optional detail/source refs), targeting 12–24 nodes with a hard cap of 36, at most four layers including the center topic, at most six top-level dimensions and three children beneath every later node. Models must consolidate related facts into `detail`/`sourceRefs`, not make one node per report bullet, datum, paragraph, or reference. An unfinished dedicated `mindmap-json` fence (and compatible historical `mindmap` fence) remains an ordinary streaming code block; only the completed, valid object lazy-renders as a compact left-to-right message card. Clicking a parent node, or its +/− control at the outgoing connection point, expands/collapses its child tree and the canvas uses an explicit visible edge stroke. The full dialog keeps its close and view tools in the top-right and a layout-derived all-map preview in the bottom-right; the message preview also permits pan, wheel zoom, and zoom/reset controls. The message card's icon-only download control opens PNG/SVG/JSON choices. Generic JSON never invokes this renderer. Ambiguity defaults to inline output. Once the HTML gate passes, delivery is mandatory as a sandbox-generated `/mnt/user-data/outputs/*.html` artifact opened with `present_files`; a full inline `html` fence is forbidden as a substitute. The shared frontend `MarkdownContent` keeps its exact legacy branch when no map-shaped fence exists; dedicated `map-json` fences and complete generic `json` fences that contain map fields and pass the full map schema are validated and lazy-rendered from `components/workspace/visualizations/`. Map/loading/error cards use the full message-content width so their edges align with the step panel. Non-map and incomplete generic JSON retain the ordinary code-block path.
HTML artifacts are single-page by default. Omitted `preferred_page_count` is normalized to `1` during direct validation. A count greater than one is allowed only when the current user directly asks for multiple pages or gives a page count; content length and report chapter count never imply multi-page intent. Explicit multi-page intent without a count defaults to a three-page lightweight portal. It is one artifact HTML, not a folder of linked pages: `<main class="pages">` contains logical `.page` sections with unique `id`/`data-page`, and `href="#page-..."` navigation works in the sandboxed Blob preview through the injected anchor handler. Its validator requires the local Bootstrap/Alpine assets, `.pages`, multiple `data-page` sections and real anchor navigation: every actual `<a href="#page-...">` and every script-set `location.hash` must match a unique logical-page id, and each page must be reachable; comments or `#overview`/`#stations` data-page values cannot spoof this check. `MultiPageHtmlMiddleware` repeats that structural inspection at the `present_files` boundary, so a model that skips validate or writes custom tabs cannot expose a nonconforming portal. A user-uploaded background goes only on the requested logical `.page` in this mode (home page when unspecified), not on every page. When a template is explicitly requested, prepare strictly excludes profiles from the wrong page-count mode, persists `requested_page_count` and `page_mode`, and validate reloads them. Validation rejects multiple detected page views (`data-page`, tabpanels, `.report-page`, or `.view`) in single-page mode, or a single view when multi-page was explicitly requested.
@@ -130,7 +357,7 @@ For an explicit multi-page HTML request without an explicit template request, `M
### Generated HTML artifact visualization runtimes
The same normal lead-agent prompt includes `_HTML_ARTIFACT_VISUALIZATION_SECTION`, but its contents are executable only after the explicit HTML-intent gate above passes; visualization content alone never implies this intent. New generated HTML explicitly loads browser distributions from Gateway `/api/public/html-runtime/`: `bootstrap.min.css`, `alpine.min.js`, `bootstrap.bundle.min.js`, `g6.min.js`, `echarts.min.js`, and then `deerflow-runtime.js`; it can also load the packaged country/region map catalog: `china-provinces`, `us-states`, `japan-prefectures`, and `taiwan-counties`. Bootstrap is the responsive layout/modal layer and Alpine is the small state/list-rendering layer; jQuery, React, Vue, Babel and public CDNs are not part of this artifact contract. The files are package resources under `packages/harness/deerflow/html_runtime/assets/` and ship with the harness wheel, so HTML execution does not depend on frontend Vite/node_modules assets or a public CDN. `system_settings.json -> html_map.engine` (`GET|PUT /api/system-settings/html-map`, admin write) chooses `echarts` (default, local GeoJSON `data-df-geo`) or `leaflet` (地图网页, `data-df-map`). `config.yaml -> html_runtime.map_tile_url` is injected into `deerflow-runtime.js` as `mapTileUrl` and is used only by the Leaflet engine. Both initializers stay in the runtime so already generated pages keep rendering after the switch. The next ordinary Q&A prompt and `page_report_template` validation follow the selected engine. `config.yaml -> html_runtime.base_url` is the browser-reachable absolute runtime base written into generated source (localhost default; deployments must replace it with the real Gateway origin). Flowcharts use paired `data-df-flow` JSON blocks and AntV G6, not Mermaid; charts use `data-df-chart`; maps use ECharts Canvas-backed `data-df-geo` with the local GeoJSON catalog. `data-df-chart` must be strict JSON: no JavaScript callbacks or function-valued colors, with literal colors/data-item styles and string tooltip formatters instead. Dark HTML must set `data-df-theme="dark"` and explicitly give ECharts global/title/legend/axis/visible-series labels readable light text, including `nameTextStyle` for named axes. Map `description` values are plain text because the runtime escapes markup; click-to-filter or page-switch behavior listens to `deerflow:geo-map-click` and uses a valid `#page-...` anchor. `deerflow-runtime.js` waits until a hidden logical page has a real layout before creating ECharts and tracks later container changes with `ResizeObserver`, so map/chart canvases never depend on a browser-window resize for their first visible render. It initializes all three contracts, opens a detail popover for described regions/markers, and exposes catalog URLs through its public dynamic response. Map large screens use a centered map stage with narrow side rails and bottom summaries, rather than cards layered across a full-page map background. Frontend runtime replacement is bypassed whenever this explicit Gateway runtime is present. This remains separate from ordinary message rendering; ordinary `map-json` keeps its existing frontend tile renderer and `VITE_MAP_TILE_URL` configuration, and ordinary `flow-json`/`chart-json` must not be reintroduced.
The same normal lead-agent prompt includes `_HTML_ARTIFACT_VISUALIZATION_SECTION`, but its contents are executable only after the explicit HTML-intent gate above passes; visualization content alone never implies this intent. New generated HTML explicitly loads browser distributions from Gateway `/api/public/html-runtime/`: `bootstrap.min.css`, `alpine.min.js`, `bootstrap.bundle.min.js`, `g6.min.js`, `echarts.min.js`, and then `deerflow-runtime.js`; it can also load the packaged country/region map catalog: `china-provinces`, `us-states`, `japan-prefectures`, and `taiwan-counties`. Bootstrap is the responsive layout/modal layer and Alpine is the small state/list-rendering layer; jQuery, React, Vue, Babel and public CDNs are not part of this artifact contract. The files are package resources under `packages/harness/deerflow/html_runtime/assets/` and ship with the harness wheel, so HTML execution does not depend on frontend Vite/node_modules assets or a public CDN. `system_settings.json -> html_map.engine` (`GET|PUT /api/system-settings/html-map`, admin write) chooses `echarts` (default, local GeoJSON `data-df-geo`) or `leaflet` (地图网页, `data-df-map`). The map tile URL saved in the settings dialog (`system_settings.json -> html_runtime.map_tile_url`, `GET|PUT /api/system-settings/html-runtime`) is injected into `deerflow-runtime.js` as `mapTileUrl` and is used only by the Leaflet engine. Both initializers stay in the runtime so already generated pages keep rendering after the switch. The next ordinary Q&A prompt and `page_report_template` validation follow the selected engine. The runtime base URL saved in the same settings block is the browser-reachable absolute address written into generated source. An empty value falls back to an older `config.yaml` entry and then `http://localhost:8001/api/public/html-runtime`. Flowcharts use paired `data-df-flow` JSON blocks and AntV G6, not Mermaid; charts use `data-df-chart`; maps use ECharts Canvas-backed `data-df-geo` with the local GeoJSON catalog. `data-df-chart` must be strict JSON: no JavaScript callbacks or function-valued colors, with literal colors/data-item styles and string tooltip formatters instead. Dark HTML must set `data-df-theme="dark"` and explicitly give ECharts global/title/legend/axis/visible-series labels readable light text, including `nameTextStyle` for named axes. Map `description` values are plain text because the runtime escapes markup; click-to-filter or page-switch behavior listens to `deerflow:geo-map-click` and uses a valid `#page-...` anchor. `deerflow-runtime.js` waits until a hidden logical page has a real layout before creating ECharts and tracks later container changes with `ResizeObserver`, so map/chart canvases never depend on a browser-window resize for their first visible render. It initializes all three contracts, opens a detail popover for described regions/markers, and exposes catalog URLs through its public dynamic response. Map large screens use a centered map stage with narrow side rails and bottom summaries, rather than cards layered across a full-page map background. Frontend runtime replacement is bypassed whenever this explicit Gateway runtime is present. This remains separate from ordinary message rendering; ordinary `map-json` keeps its existing frontend tile renderer and `VITE_MAP_TILE_URL` configuration, and ordinary `flow-json`/`chart-json` must not be reintroduced.
User-uploaded raster images can be used as generated-HTML backgrounds through signed, unauthenticated `GET /api/public/html-images/{token}` URLs. `UploadsMiddleware` mints one URL per uploaded `.jpg/.jpeg/.png/.webp/.gif/.bmp` and injects it into `<uploaded_files>` as `Public URL`; the HTML contract requires that exact absolute URL in `background-image`, never a sandbox path, artifact API path, relative path, or base64 data URI. Single-page artifacts place it on `html/body`; the multi-page portal places it only on the requested logical page. Tokens are HMAC-signed, do not expire (the URL is written into durable HTML), and only grant read access to one file under the thread `uploads/` or `outputs/` directory. Engine: `deerflow.html_runtime.public_images`; router: `app/gateway/routers/html_images.py`; tests: `tests/test_html_images.py`.
@@ -303,6 +530,28 @@ commands.py` and `tests/test_position_roundtable_concurrency.py` cover the
state transitions; `tests/test_roundtable_postgres_concurrency.py` is the real
PostgreSQL acceptance suite (set `DEERFLOW_POSTGRES_CONCURRENCY_TEST_URL`).
Step-2 roundtable rooms share one deployment-wide cap,
``system_settings.json -> roundtable_room_limit`` (default enabled, 3 rooms).
``GET/PUT /api/system-settings/roundtable-room-limit`` is the admin switch.
Occupancy is summed across **every account**: durable foreground leases keyed
by a hidden per-live-execution ``roomSessionId``, in-flight LangGraph runs on
threads stamped with the corresponding ``roundtable_room_key``, and executing
background jobs (``queued`` / ``running`` / ``cancel_requested`` with a live
worker lease or a recent update). Different pages running the same saved draft
have different session ids and therefore take different rooms; all seats and
Step 3 calls within one live execution share its key. Run-manager inspection
is reconciliation-only: when a durable lease/job already owns the same
draft+user/task scope, stale or overlapping run keys in that scope are not
counted again. Merely opening the page,
viewing saved history, paused ``awaiting_input`` jobs, stale historical rows,
and Step-1 intent threads do not occupy a slot. Foreground-to-background
handoff replaces the foreground lease with the durable job occupancy. There is
no per-user pool. Admission is serialized by a database gate row, and the cap
is enforced at acquire, init, every model-producing run, and background-job
start. Full pools return HTTP 409; an enabled gate with missing storage or a
missing session id fails closed. The switch applies on the next admission
without a restart. Position and eight-department roundtables are not counted.
Roundtable backend jobs use a database unique active-dedupe key plus a leased
dispatcher, never process-local start deduplication. Cancellation is two
phase (`cancel_requested` then `cancelled`): the lease owner’s watcher normally
@@ -449,6 +698,8 @@ from deerflow.config import get_app_config
### Agent System
**Gallery chat link.** An agent's `config.yaml` may store `chat_link` (`url`, `open_mode` of `blank`/`self`, and light-app-shaped `route_params`). The gallery 「对话」 button opens that resolved URL; an empty url keeps the in-app chat. The list endpoint reads the mirrored `agent_extras.chat_link` column (nullable JSON, added by the startup column sweep) instead of parsing every config file. Create/update and directory backfill write the mirror; a skills-only `set_extras` leaves an existing link in place.
**Lead Agent** (`packages/harness/deerflow/agents/lead_agent/agent.py`):
- Entry point: `make_lead_agent(config: RunnableConfig)` registered in `langgraph.json`
- Dynamic model selection via `create_chat_model()` with thinking/vision support
@@ -463,6 +714,8 @@ from deerflow.config import get_app_config
**Strategy-selection functional agents.** `app/gateway/routers/_strategy_selection_seed.py` seeds nine stable folder-backed agents from `_strategy_selection_seed_assets/` before `_sync_legacy_agents`: intent, governance/diplomacy/economy/security/public-opinion/law-policy angles, judge, and solution editor. Seeding is missing-directory-only and must never overwrite administrator changes. The global `strategy_agent_roles` registry classifies them independently of projects: intent/judge/solution are fixed, unique roles that may be configured but never added, disabled, reordered, or deleted; angle roles are a shared, ordered catalog that administrators may add, edit, disable (soft delete), restore, and reorder through `/api/strategy-agents`. Public `GET /api/strategy-agents` returns enabled identity/description metadata only; `include_disabled=true` or `include_config=true` is admin-only. Admin writes are `POST /api/strategy-agents/angles`, `PUT /api/strategy-agents/{id}`, and `DELETE /api/strategy-agents/{id}`; DELETE retains the folder-backed agent and role row to preserve historical references. Dynamic angles remain built-in folder-backed agents under the reserved `strategy-selection-angle-` id namespace, with `web` + `file:read`; a blank SOUL on create receives a safe role template. All strategy skill writes validate that names exist, are active, and are globally visible; generic agent CRUD cannot create or hard-delete reserved strategy IDs, and parallel execution rejects disabled roles. Angle configs reuse `policy-knowledge`; intent reuses `intent-clarification-policy`; the judge has no retrieval skill; the solution editor gets only `file:write`. Keep the SOUL contracts neutral and evidence-aware, with explicit de-escalation/civilian-protection framing and hard bans on operational combat guidance and targeted political manipulation. Tests: `tests/test_strategy_selection_seed.py`, `tests/test_strategy_agent_management.py`.
Only `strategy-selection-intent` has a clarification reply surface in the strategy workbench. `parallel_agents` must exclude `ask_clarification` for every other `strategy-selection-*` role and inject a direct-delivery guard: angle, judge, and solution stages use conservative assumptions, lower confidence/score where needed, list gaps under data limitations, and still finish in one pass. The judge brief repeats this rule at runtime because existing folder-backed agent SOUL files may be administrator-customized and are intentionally never overwritten. If a legacy backend nevertheless emits a judge clarification, the frontend treats it as a bounded repair attempt rather than asking the user to repeat the same manual run. Malformed or incomplete evaluation JSON receives the same bounded repair path; prior attempts are replaced in the live panel, never concatenated into duplicate final content.
`POST /api/parallel-agents/run/stream` must remain observable during silent skill calls. Each running seat emits an `agent_progress` heartbeat every five seconds with the latest translated tool/reasoning/body phase and monotonic elapsed seconds; an upstream iterator that produces no frame for 180 seconds is closed and that seat emits `agent_error`, while the other seats continue. Set the terminal flag before queuing `agent_done` / clarification so a late heartbeat cannot reopen a completed card. The strategy workbench renders the intent Q&A history, readable skill execution steps and live heartbeat on the affected agent card; judge and synthesis traces use the same step translation. Tests: `tests/test_roundtable_model_fallback.py` and frontend `src/strategy-selection/lib.test.ts`.
**ThreadState** (`packages/harness/deerflow/agents/thread_state.py`):
@@ -593,7 +846,7 @@ FastAPI application on port 8001 with health checks at `GET|HEAD /health`, `/api
| **Answer feedback** (`/api/threads/{id}/messages/{mid}/feedback`) | Message-scoped `PUT`/`DELETE`, `GET /feedback-context`, and batched `POST /api/threads/{id}/feedback-contexts`; persists excellent/poor ratings, six stable poor-reason codes and optional detail idempotently per user/thread/message/agent. Legacy run-scoped feedback routes remain compatible. Admin-only `GET /api/agents/{agent_id}/feedback-score` uses the project-wide `system_role == "admin"` rule (and preserves auth-disabled mode) and returns a Bayesian user-feedback answer-quality score, raw task-success rate, weighted response-quality dimensions and reason counts; it is explicitly not a complete agent-capability evaluation. |
| **Runs** (`/api/runs`) | `POST /stream` - stateless run + SSE; `POST /wait` - stateless run + block; `GET /{rid}/messages` - paginated messages by run_id `{data, has_more}` (cursor: `after_seq`/`before_seq`); `GET /{rid}/feedback` - list feedback by run_id |
| **Open Chat** (`/api/open/chat`) | **Unauthenticated** open Q&A entry for external systems (the `/api/open/` prefix is whitelisted in `auth_middleware._PUBLIC_PATH_PREFIXES` + CSRF-exempt in `csrf_middleware.should_check_csrf`). `POST /chat` `{message \| messages[], model_name?, agent_name?, agent_id?, thinking_enabled=false, strip_think=true, thread_id?, recursion_limit=100}` → resolves the agent by **Chinese display name** (`agent_name`, matched against each `.deer-flow/agents/*/config.yaml` `name` via `_resolve_agent_id`; explicit `agent_id` wins if given), builds the lead agent via `make_lead_agent` (so the chosen **agent's** SOUL/tools/skills all apply), `ainvoke`s it once (stateless — no checkpointer, no thread persistence; runs under the `"default"` user bucket), and returns **only the final** `{answer, model_name, agent_name, agent_id, thinking_enabled, thread_id}` (no intermediate tool-call/process). **`POST /chat/stream`** is the **SSE streaming variant** (same request body) for **long tasks** — the blocking `/chat` produces no downstream bytes for the whole run, so a front nginx `proxy_read_timeout` (default 60s) cuts it off with a **504**; the stream keeps the connection warm (a `: keepalive` SSE comment every `_HEARTBEAT_SECONDS=15`s via a background producer-task + queue, plus `event: delta` AI-text increments and `event: progress`), and ends with **`event: result`** carrying the same final `{answer, …}` JSON (or `event: error`). Clients only need to read `event: result`; intermediate events are ignorable. Both endpoints share `_build_run_context` (config/agent/thread). `GET /agents` lists `{name, id}` so callers know what `agent_name` to send. **Thinking off is reliable**: when `thinking_enabled=false` the route also sets `thinking_force_disabled=true` so the lead agent calls `create_chat_model(force_disable_thinking=True)`, which injects **three** OpenAI-compatible "thinking off" shapes into `extra_body` — top-level `enable_thinking=false` (**SiliconFlow / DashScope hybrid models** like `deepseek-ai/DeepSeek-V4-*` on `api.siliconflow.cn` — the knob the nested `chat_template_kwargs` shape does NOT reach), `chat_template_kwargs.enable_thinking=false` (vLLM/Qwen native), and `thinking.type="disabled"` (Zhipu GLM) — even for models that only declare `supports_thinking: true`. For the **web-UI path** (which sends `thinking_enabled=false` but NOT `force_disable`), SiliconFlow models need `when_thinking_disabled: {extra_body: {enable_thinking: false}}` in their `config.yaml` entry (added to `deepseek-ai/DeepSeek-V4-Flash`) for thinking-off to take effect. Sets `is_scheduled_run=true` internally to skip the auto-title LLM call. **Generated-file return (`include_files=true`, default on)**: an agent that produces its report as a file (writes to `/mnt/user-data/outputs/` + `present_files`) returns only a short summary in `answer`; to recover the real deliverable both endpoints now also return `files: [{name, virtual_path, encoding(text|base64), content}]` — `_collect_output_files` scans the run's `state['thread_data']['outputs_path']` (the open thread is stateless so artifacts can't be fetched by thread, but the endpoint runs server-side and the sandbox already wrote the files to disk), text decoded as utf-8 else base64, capped by `_MAX_OUTPUT_FILES=20` / `_MAX_OUTPUT_FILE_BYTES=5MB`. `/chat` puts `files` on the response; `/chat/stream` puts it in the `event: result` payload. A run with no answer **and** no files now 502s ("…文本答案或文件"). Router `app/gateway/routers/open_chat.py`; standalone caller `test_open_chat_api.py` (repo root, streaming by default); tests `tests/test_open_chat.py`. |
| **Open 3qfx Ask** (`/api/open/3qfx/ask`) | **Unauthenticated** open entry that, given just a **任务id**, directly kicks off the 3qfx（三情分析）「第一次问答」as a **real persisted conversation** (sibling of `/api/open/chat`; same `/api/open/` auth-whitelist + CSRF-exempt). `POST /ask` `{task_id, message?, model_name?, thinking_enabled=false}` → (1) resolves the `3Q` agent from `app.state.business_mapping_store.get_mapping("3Q")` (the **same** agent the frontend deep-link `goPath=3qfx` uses; 503 if unconfigured); (2) **fetches the task detail server-side** (consumer `cop-task-detail`) and builds the opening text **identical to the frontend** `buildOpeningText` 3qfx branch — `任务名称：…\n任务内容：…\n任务id（task_id）为{task_id}` (an explicit `message` overrides and **skips** the fetch); (3) creates a fresh thread tagged `metadata.{taskId, agent_id}` (the exact filter `useTaskThreads` uses, and `metadata.taskId` makes it task-deeplink-shared / creator-path-resolved) and **starts a background persisted run** via `start_run(..., enforce_access=False)` (writes the checkpointer, runs under the `"default"` bucket); (4) **returns immediately** `{thread_id, run_id, agent_id, status, opening_text}` — the caller just needs "started", then watches the run in the 3qfx 任务工作区 conversation list. `start_run` gained a keyword-only `enforce_access` (default `True`); the open entry passes `False` so the admin-configured agent isn't rejected by the per-(missing)-login visibility check. **Consumer fetch config** lives in `config.yaml → task_deeplink`: `cop_task_detail_url` (full URL, appends `?taskId=`), `cop_task_detail_authorization` (default `Bearer admin`), `cop_task_detail_test` + `mock.cop_task_detail` (offline fake when the upstream is unreachable — on in this deployment). Router `app/gateway/routers/open_3qfx.py`; tests `tests/test_open_3qfx.py`. |
| **Open 3qfx Ask** (`/api/open/3qfx/ask`) | **Unauthenticated** open entry that, given just a **任务id**, directly kicks off the 3qfx（三情分析）「第一次问答」as a **real persisted conversation** (sibling of `/api/open/chat`; same `/api/open/` auth-whitelist + CSRF-exempt). `POST /ask` `{task_id, message?, model_name?, thinking_enabled=false}` → (1) resolves the `3Q` agent from `app.state.business_mapping_store.get_mapping("3Q")` (the **same** agent the frontend deep-link `goPath=3qfx` uses; 503 if unconfigured); (2) **fetches the task detail server-side** (consumer `cop-task-detail`) and builds the opening text **identical to the frontend** `buildOpeningText` 3qfx branch — `任务名称：…\n任务内容：…\n任务类型：…\n任务id（task_id）为{task_id}`（`taskType` 1/2/3 → 打击/塑势/防御，无法识别则省略） (an explicit `message` overrides and **skips** the fetch); (3) creates a fresh thread tagged `metadata.{taskId, agent_id}` (the exact filter `useTaskThreads` uses, and `metadata.taskId` makes it task-deeplink-shared / creator-path-resolved) and **starts a background persisted run** via `start_run(..., enforce_access=False)` (writes the checkpointer, runs under the `"default"` bucket); (4) **returns immediately** `{thread_id, run_id, agent_id, status, opening_text}` — the caller just needs "started", then watches the run in the 3qfx 任务工作区 conversation list. `start_run` gained a keyword-only `enforce_access` (default `True`); the open entry passes `False` so the admin-configured agent isn't rejected by the per-(missing)-login visibility check. **Consumer fetch config** lives in `config.yaml → task_deeplink`: `cop_task_detail_url` (full URL, appends `?taskId=`), `cop_task_detail_authorization` (default `Bearer admin`), `cop_task_detail_test` + `mock.cop_task_detail` (offline fake when the upstream is unreachable — on in this deployment). Router `app/gateway/routers/open_3qfx.py`; tests `tests/test_open_3qfx.py`. |
| **Thread Shares** (`/api/thread-shares`) | Per-conversation sharing with two modes (`thread_shares.mode`). **`import`**: `POST /` `{thread_id, mode}` - create (or reuse) a revocable share code; `GET /{code}/preview` + `POST /{code}/import` - copy that conversation into the caller's account (fresh `thread_id` via `app/gateway/thread_copy.py` + new `threads_meta` row + `user-data` copy; dedup on `metadata.shared_origin`). **`view`**: a public, read-only static page ? see Public Shares below. Both modes: `GET /` - list my codes; `DELETE /{code}` - revoke. |
| **Public Shares** (`/api/public/shares`) | **Unauthenticated** (the `/api/public/` prefix is whitelisted in `auth_middleware._PUBLIC_PATH_PREFIXES`). `GET /{code}` - read-only conversation snapshot (title + serialized messages + artifacts) for a `view`-mode share; messages are read straight from the latest checkpoint. `GET /{code}/files/{path}` - serves a file referenced by that conversation (images, Markdown, text, PDF, ?) so the shared page's links are openable; path resolution is confined to the owner's thread directory. Active content (`text/html`, `application/xhtml+xml`, `image/svg+xml`) is forced to download (`_public_share_disposition`); everything else is served inline. Missing/revoked/non-`view` codes all return 404. |
| **Roundtable Drafts** (`/api/roundtable-drafts`) | **Strictly per-user** persistence for the roundtable-planning page (replaces the old browser `localStorage`). `GET /` - list drafts (lightweight meta — incl. `chain_title`, the business-chain name used this session, cheaply regex-extracted from the in-memory `step2` blob in `_draft_to_meta`/`_extract_chain_title`; null = no chain / 自由模式; surfaced on the 第一步「分析记录」history card + sidebar list instead of the old「第 X 步」); `POST /` - create; `GET/PUT/DELETE /{id}` - full read / partial update / delete (all owner-scoped). A draft = one full session (Step 1 intent + Step 2 roundtable), step snapshots stored as JSON in `PortableLongText`. Recommendation history bound to a draft: `GET /{id}/recommendations` - list newest-first; `POST /{id}/recommendations` - append one run (objective/status/rationale/picks/candidates). taskId 深链会商记录 are **not** here — they live in the separate `/api/roundtable-task-drafts` store (see below) so personal history never mixes with task records. Backed by `deerflow.persistence.roundtable_drafts` (tables `roundtable_drafts`, `roundtable_recommend_history`); store wired in `deps.py` as `app.state.roundtable_draft_store`. Tests: `tests/test_roundtable_task_drafts.py`. |
@@ -660,6 +913,8 @@ The intranet client uses `trust_env=False` and defaults `intranet_verify_ssl=fal
The admin ????? is served from **precomputed per-day snapshots** in `admin_leaderboard_daily_stats` (keyed by `(stat_date, settings_hash)`), not by aggregating raw rows on every request. A background scheduler inside the Gateway (`_leaderboard_snapshot_scheduler_loop` in `app/gateway/app.py`, singleton file-locked) ticks every `ADMIN_LEADERBOARD_BACKGROUND_INTERVAL_SECONDS` (default 300s): it prewarms recent days, then builds any `queued`/`error`/stale-`running` day. `GET /leaderboard` merges the per-day snapshots for the requested range. **Today** is rebuilt whenever its snapshot is older than `ADMIN_LEADERBOARD_TODAY_TTL_SECONDS` (default 43200s = half a day, so the leaderboard refreshes ~twice daily); a past day, once `ready`, is treated as fresh forever.
**Recent-only admin audit policy.** The server enforces a rolling **30-calendar-day** window for administrator leaderboard, user analytics, user Q&A/thread audit, LLM/tool metric list/summary/export, and the user-list Q&A counter; URL parameters cannot widen it. The Q&A fallback that reads checkpoints is additionally capped, while indexed run summaries remain the primary audit source. Gateway startup also launches the singleton `admin_audit_retention` sweep: it deletes audit metric rows, terminal `runs` summaries (which contain compact question/answer audit fields and were also last updated before the cutoff), and leaderboard snapshots older than **90 days**. Cleanup is deliberately low-priority: after the default 5-minute startup delay, it runs every 15 minutes using 100-row committed transactions, round-robins each store (at most four batches per store), yields between commits, and stops starting new batches after an 8-second sweep budget; operators may tune these bounds with `ADMIN_AUDIT_RETENTION_*` environment variables. It deliberately does **not** delete conversation threads, checkpoints, or message history; those are user-content records governed by their own lifecycle.
Two on-demand/automatic refresh paths layer on top:
- **Manual "????"** ? `POST /leaderboard/refresh-today` force-rebuilds *today's* snapshot **synchronously** (bounded by the same semaphore/timeout the scheduler uses), then returns the merged range. Lets an admin pull the latest real-time records into the stats without waiting for the TTL. Frontend: a "????" button on `AdminUserLeaderboardPage` ? `useRefreshTodayLeaderboard` (`core/admin/hooks.ts`), which primes + invalidates the `["admin-activity","leaderboard"]` query.
- **Daily finalize of the previous day** ? once per day, on/after `ADMIN_LEADERBOARD_FINALIZE_HOUR` (local Beijing, default `1` = ?? 1 ?), the scheduler tick force-recomputes *yesterday* a single time so runs/tool-calls that only landed near midnight are fully captured. The decision is the pure `deerflow.persistence.admin_stats.finalize.should_finalize_previous_day` (status/generated_at/now/hour) ? the rebuild stamps `generated_at` to today, which self-debounces it for the rest of the day (no extra state file). Tests: `tests/test_leaderboard_finalize.py`.
@@ -768,7 +1023,7 @@ The fixed-deep-blue wx full-chat route (`#/page/workspace/wx/chats/new`) hides t
### Business-mapping authorization
`/api/business-mapping` remains globally readable. Seeded rows include the main codes (3Q / 6BF / 7BF / 8BF / AFX) plus independent slots `3q-0…3q-6`, `6-1…6-4`, and `7-1…7-4` (empty agent/chain until configured on the 配置管理 page). Updates to 3Q, 6BF, 7BF, and those independent slots require `system_role == "admin"`; the 8BF public business-chain mapping is intentionally different and can be updated only by the trusted authenticated account whose email prefix is `lqq`. Do not rely on a username supplied in an external login URL. The frontend follows the same email-prefix rule when deciding whether to render the 8BF configuration button, while the router (`app/gateway/routers/business_mapping.py`) remains the enforcement point. Tests: `tests/test_business_mapping.py`.
`/api/business-mapping` remains globally readable. Seeded rows include the main codes (3Q / 6BF / 7BF / 8BF / AFX) plus independent slots `3q-0…3q-6`, `6-1…6-4`, and `7-1…7-4` (empty agent/chain until configured on the 配置管理 page). `3q-1…3q-6` seed as `is_custom=true` / `mapping_type=自定义类型` so `goPath=3q-1` resolves as a custom deep link; the configuration page hides those six rows and defaults the type filter to 自定义类型. Updates to 3Q, 6BF, 7BF, and those independent slots require `system_role == "admin"`; the 8BF public business-chain mapping is intentionally different and can be updated only by the trusted authenticated account whose email prefix is `lqq`. Do not rely on a username supplied in an external login URL. The frontend follows the same email-prefix rule when deciding whether to render the 8BF configuration button, while the router (`app/gateway/routers/business_mapping.py`) remains the enforcement point. Tests: `tests/test_business_mapping.py`.
### Position-collaboration foundation
@@ -1357,10 +1612,11 @@ scheduler runs once per deployment. Per run:
会商第二步（leader/seat）+ 第三步（summary/report/dashboard + 结构化抽取）单轮执行时，**某模型出问题就自动切到下一个模型继续**，并告知用户「『X』模型异常 → 已切换『Y』」。前台实时（`multi_agent.py`）与后台挂起（`engine.py` + `InProcessRoundtableGateway`）两条路径都覆盖。
- **可检测的模型失败**：`LLMErrorHandlingMiddleware` 在同一模型重试 3 次后**不抛异常**，而是返回兜底 `AIMessage` —— 现在给它打 `additional_kwargs[LLM_ERROR_MARKER="deerflow_llm_error"]=reason`（model_dump 保留，经 checkpointer 序列化仍在）。`is_llm_error_message(msg)` 据标记（回退按已知兜底文案正文）识别，返回 reason。
- **候选模型链**：`app/gateway/roundtable_model_fallback.py::fallback_model_chain(current)` 按 `config.yaml models[]` 顺序给出 `[当前模型, …其余]`（去重保序）。`reason_text(reason)` 翻译成简短中文。总开关 env `ROUNDTABLE_MODEL_FALLBACK`（默认开，`0/false/off` 关 → 单模型旧行为）；`ROUNDTABLE_MODEL_FALLBACK_MAX`（0=试完全部）。
- **触发换模型的情形**：① 单轮 run 抛异常（start_run 失败 / idle 卡死 → `timeout`，其余 `exception`）；② 兜底错误文案（`is_llm_error_message`）；③ 交付无效（仅 seat/leader；leader 还含「既无正文又无派活」）。席位**只认剥掉 `<think>` 后的可见正文**——思考-only / 空串 / 断流都不算已交付（`deerflow.agents.roundtable_orchestrator.delivery`：`visible_seat_text` / `classify_seat_delivery`）。前台 `_special_run` 对此 yield `status:"invalid_delivery"` 且不广播；`GET /last-ai-message` 回 `{delivered, content, length, invalid_reason}`（思考-only 标 `thinking_only`，总控席位清单标 ⚠️）。后台引擎只把可见正文写入 `deliveries`，续轮用 `build_continue_prompt`：有无效席位时强制重派，禁止声称「已完成交付」。Step3 单例（report/summary/dashboard/ingest/结构化）的产物是落盘文件，**不**把「正文为空」当失败，只对真模型错误换模型。
- **候选模型链**：保留真正模型故障后的自动换模，但“仅有思考 / 暂时无正文 / 空交付”不再判作模型故障，绝不据此切换用户选择的模型。`app/gateway/roundtable_model_fallback.py::fallback_model_chain(current)` 按 `config.yaml models[]` 顺序给出 `[当前模型, …其余]`（去重保序）。`reason_text(reason)` 翻译成简短中文。总开关 env `ROUNDTABLE_MODEL_FALLBACK`（默认开，`0/false/off` 关）；`ROUNDTABLE_MODEL_FALLBACK_MAX`（0=试完全部）。
- **触发换模型的情形**：① 单轮 run 抛异常（start_run 失败 / idle 卡死 → `timeout`，其余 `exception`）；② 兜底错误文案（`is_llm_error_message`）。交付无效、仅有思考、空串以及普通总控轮既无正文又无派活**均不触发换模**；唯一例外是无工具的最终收口请求已经完整结束仍无正文，此时结果客观不可交付，才继续自动换下一个模型。席位仍只认剥掉 `<think>` 后的可见正文（`deerflow.agents.roundtable_orchestrator.delivery`：`visible_seat_text` / `classify_seat_delivery`）：前台 `_special_run` 对无效交付 yield `status:"invalid_delivery"` 且不广播；`GET /last-ai-message` 回 `{delivered, content, length, invalid_reason}`（思考-only 标 `thinking_only`，总控席位清单标 ⚠️）。后台引擎只把可见正文写入 `deliveries`，无效席位继续走既有未交付处理，不切换模型。Step3 单例（report/summary/dashboard/ingest/结构化）的产物是落盘文件，同样只对真模型错误换模型。
- **Step2 总控工具与可配置收口轮数**：普通总控使用真正白名单，只允许 `agent_orchestration` / `ask_clarification`；`write_file`、`present_files`、技能目录与文件/命令工具均不可见。`system_settings.json -> roundtable_room_limit.max_cycles`（默认 8，范围 2..100；管理员在设置弹窗「会商并发与收口」中修改）控制 recommend 模式到第几轮切换为显式 `finalize_only`：空工具白名单、注入已有有效席位全文、要求直接输出最终总结（失败席位如实标缺口），不能表现为“已停止”。一次会商启动时锁定当时值，运行中修改从下一次研讨生效；前台实时会商与后台作业读取同一设置。同一页的「会商文件流式兼容」(`write_file_compatibility_enabled`，默认关) 只作用于会商总控、席位、第三步和后台作业：全局文件流式兼容开着时会商一并使用；全局关着而本开关开着时，只有这些会商 run 改走 `buffered_gateway`。普通对话不读这个开关。chain / dag 的最终总结仍在既定席位完成后走同一纯正文路径；LangGraph 单轮 recursion limit 仍为 1000。
- **后台**：`InProcessRoundtableGateway._run_turn_with_fallback(...)` 包住 `_run_agent_turn`，逐候选模型试；换模型落 `model_switch` 诊断、全失败落 `model_fallback_exhausted`。`run_leader/run_seat/run_report/run_summary/run_dashboard_data` 全走它（report/summary/dashboard 的产物自愈重试保留，并记住已切到的可用模型用于后续续写轮）。各 Turn/Result 带回 `switches: list[ModelSwitch]`，引擎 `_emit_switches` → `JobProgressWriter.model_switch(...)` 往研讨时间线插一条 `role="system"` 系统提示气泡。
- **前台**：`_special_run`/`_leader_run` stream 收尾后检测失败 → yield `{status:"model_switch", role, agent_name, failed_model, next_model, reason}` 帧并用下一个模型重新流式；终态帧带 `model_used`。结构化抽取 `flow_extract.py` 的 round1 `model.astream` 也包了同款换模型循环（emit `event:"model_switch"` 帧）。
- **前台**：`_special_run`/`_leader_run` stream 收尾后仅检测明确的模型错误标记；命中才 yield `{status:"model_switch", role, agent_name, failed_model, next_model, reason}` 帧并用下一个模型重新流式，思考-only/空正文不进入该分支；终态帧带 `model_used`。结构化抽取 `flow_extract.py` 的 round1 `model.astream` 也包了同款换模型循环（emit `event:"model_switch"` 帧）。
- **Step2 单席位隔离**：`engine._run_recommend/_run_chain/_run_dag` 的每个席位执行包 try/except —— 一个席位彻底失败（落 `seat_failed` 诊断 + 置空交付）不拖垮整批/整链/整个作业。
- **前端**：`multi-agent.ts` 解析 `model_switch` 帧 → `onModelSwitch` 回调（清空失败尝试的半截气泡 + 累计的 tool_call delta，phase 退回 connected）；`useStep2Orchestration` 弹 toast + 时间线插「系统提示」气泡；`useStep3Summary/Report/Dashboard` + `flow-extract.ts` 重置对应气泡并在步骤卡留可见提示；后台路径经 `job-live.ts::toStep2Dialogue` 把 `role="system"` 渲染成 amber「系统提示」气泡。
- Tests: `tests/test_roundtable_model_fallback.py`。
@@ -1502,6 +1758,29 @@ from the selected space names and hides generic recommendations while these
questions are present. Design and remaining graph/profile phases are in
`docs/KNOWLEDGE_SPACE_RETRIEVAL_AND_WIKI_EXPANSION_ZH.md`.
**Knowledge-base research bookshelf:** `/page/workspace/knowledge/:id/research`
is ordinary Lead Agent Q&A constrained to one existing knowledge base; it does
not invoke or reuse the legacy Deep Research workflow. Any account with read
access may create a private session/draft, while only the existing KB writer
may publish or overwrite its Markdown Wiki article. Published snapshots follow
the KB's read permission, but the session/thread/draft and the continuation
link remain creator-only (administrators do not inherit another user's private
chat). Every run resolves the immutable thread binding from
`knowledge_research_sessions` (registered on `deerflow.persistence.models` so
startup `create_all` builds the table; `get_by_thread` treats a missing table
as no binding so ordinary `/runs/stream` is not aborted), reauthorizes KB read access, accepts exactly one
text-only human turn, forces classic single-KB retrieval and disables tools,
memory, prompt prefixes, planning and subagents; fixed-answer shortcuts are
bypassed. Saving manually upserts a stable
`knowledge-research/{publication_id}` Wiki slug under the reserved root
「深度研究」 without enabling automatic Wiki generation, uses a saved hash for
optimistic overwrite checks and idempotent retries, and keeps the editable
publication title separate from the Q&A title. REST endpoints live in
`app/gateway/routers/llmwiki.py`, runtime enforcement in
`app/gateway/knowledge_research.py`, persistence/migration in
`deerflow.persistence.knowledge_research` / `20261022_01`, and regression
coverage in `tests/test_knowledge_research.py`.
**Embedding ownership:** WeKnora's per-knowledge-base `embedding_model_id`
indexes uploaded source documents; `llmwiki.local_wiki_index.embedding` indexes
the generated Wiki-page mirror inside DeerFlow. Do not couple these two model
@@ -1742,7 +2021,10 @@ Automatic conversation summarization when approaching token limits:
`conversation_context.recent_turn_window_enabled` is a deployment-wide,
default-off alternative to summarization. Administrators manage it in
「设置 → 上下文管理」 through
`GET/PUT /api/system-settings/conversation-context`. When enabled, each lead
`GET/PUT /api/system-settings/conversation-context`. The same block stores
`max_conversation_rounds` (default 20, range 1–50), which `ChatPage` uses as
the maximum number of real user turns in one ordinary chat; missing values in
older settings files retain the 20-turn default. When enabled, each lead
agent model request retains only the current turn and the two preceding real
user turns, together with their assistant/tool continuations and any hidden
context attached to the retained boundary. The operation is request-only:
@@ -2641,3 +2923,24 @@ calls. The active sandbox skill mirror must be synchronized with the canonical c
Gateway detail writes emitted by `fs-collect` and `fs-detail-submit` log a token-masked request and
the corresponding response so an interrupted chain can be distinguished from a failed persistence
call without exposing the gateway bearer token.
### Task-detail custom skill password authentication (2026-10-26)
`skills/custom/task-context` and `skills/custom/task-detail-update` authenticate detail reads/writes
with a short-lived token minted by `POST /api/v1/auth/login/local`, using the runtime-only
`TASK_DETAIL_AGENT_USERNAME` / `TASK_DETAIL_AGENT_PASSWORD` pair and `TASK_DETAIL_GATEWAY_URL`.
They must never use general `DEERFLOW_AUTH_TOKEN` or `TOOLS_TOKEN`, nor emit the password or JWT in
an input, trace, or model-visible output. `/api/parallel-agents/auth/token` is only a legacy fallback:
it follows `sso_enabled` and returns 403 when external SSO is disabled. Password-login failures other
than the policy 403 must fail closed; do not fall back to identity-only token minting. Keep the
canonical custom directories and `.deer-flow/.sandbox-online-skills` mirror synchronized through the
custom-skill reconciliation flow. `scripts/setup_biz_detail_assistant.py` logs in through the same
formal endpoint and accepts `DEERFLOW_SETUP_BASE`, `DEERFLOW_SETUP_USERNAME`,
`DEERFLOW_SETUP_PASSWORD` for deployment overrides; it may fall back to the legacy username route
only for the formal login policy 403, never for a credential or rate-limit failure.
`biz-detail-assistant` must retain the native `ask_clarification` tool. Do not special-case it by
adding that tool to `requested_excluded_tools`, and do not mount `DetailConfirmMiddleware` to forge
confirmation cards from a plain model response: those cards are not genuine model tool calls and break
the standard pause/resume interaction. The final `ClarificationMiddleware` is the single authority for
all missing-information, scope, approach and risk-confirmation questions.
offline-backend-20260512/backend/README.md
+166 -6
View File
@@ -1,5 +1,28 @@
# DeerFlow Backend
## 首次问答初始化响应性（2026-10-30）
普通问答、开放问答接口和工作流智能体节点的模型 SDK 加载、客户端初始化及
智能体构建统一在独立初始化线程执行，
同时进入的构建请求异步排队。首次问答仍需等待初始化完成，但不会因此占住
其他请求的处理主循环，已有对话可继续输出。用户身份、工具发现状态及 MCP
连接仍按原会话隔离；初始化期间取消会等待当前构建收尾，随后结束，不再发起模型调用。
AI 写作、子任务及深度研究的工具初始化也使用同一通道，避免首次发现或刷新
MCP 工具时与其他问答互相等待。
该优化只对重新加载代码的进程生效；已运行的子 Worker 可以继续使用旧版本，
不会自动获得此优化。启用 Worker 池的生产主服务重启还涉及原有镜像轮换和
流式连接断开策略，不能把这项修复当作不停机升级保证。
## 多文件上传响应性（2026-10-30）
会话附件的写入、格式校验和沙箱同步在独立工作池执行，每个后端进程最多同时
处理 2 批上传，其余异步等待。PDF/Office 文档解析进一步放到最多 2 个独立进程，
减少解析器对聊天请求的 CPU 争用。同一会话的上传按批次串行；列表和删除操作等待
该批次完成或回滚。上传成功仍表示整批文件已处理完成，可以直接用于问答。
文件数量、大小、类型限制以及失败回滚规则不变。更新后需重启后端生效。
首次文档转换需要启动解析进程并加载依赖，后续复用进程；此时其他请求仍可响应。
沙箱管理器的首次构造也有并发保护，避免两批上传同时首次进入时创建重复实例。
## 普通知识库智能检索双检索面（2026-10-16）
智能检索全局开关默认关闭，且只保存在数据库。管理员开启后，新对话默认使用智能检索，
@@ -18,6 +41,10 @@
并在建库前调用模型调试接口做一次真实可用性检查；调试失败时不会创建一个随后把全部文档送入
Wiki 死信的知识库。模型默认项、知识库摘要模型和 Wiki 合成模型均由数据库保存，不写入业务配置文件。
知识库深度研究会话表 `knowledge_research_sessions` 由启动 `create_all` 创建。
每次 `/api/threads/{id}/runs/stream` 都会按 thread 查这张表以判断是否绑定单库研究；
表尚未建好时按普通问答处理，不会让流式接口因缺表而失败。
## 公文助手与网页报告校验优化（2026-10-07）
公文助手新增预置 `run_gongwen.py deliver` / `verify-file`，一次交付调用完成
@@ -41,6 +68,22 @@ Python 脚本与重复检查。结构检查不替代事实核实和真实分页
「当前文件正在生成，请稍等片刻，完成后就可以切换查看其他文件。」
自动切换静默跳过，生成完成、停止或离开会话后解除保护。
## 网页报告校验优化（2026-10-07）
网页报告写入和校验时自动补齐深色组件及图表文字颜色。校验返回自动修复项、
完整错误列表和 `repair_checklist`，要求集中完成局部修复后再校验，避免逐条
循环和整页重写。遗留小字号声明不再单独拦截。
## 沙箱流式展示与切换保护（2026-10-07）
普通问答和智能体问答的 `write_file` 没有有效路径时，不创建占位制品来打开
空沙箱，而在步骤条内用绿色小字流式展示正文；后续完整路径到达后恢复正常
文件预览，正常文件的自动打开与沙箱实时展示保留。
当前预览文件正在生成或全文改写时，沙箱阻止切换到其他文件。用户操作时提示：
「当前文件正在生成，请稍等片刻，完成后就可以切换查看其他文件。」
自动切换静默跳过，生成完成、停止或离开会话后解除保护。
## 技能检索文件展示（2026-10-07）
普通问答、智能体问答中，技能返回的会话文件路径、原文件链接或知识库文档标识，
@@ -77,6 +120,21 @@ OpenAI 兼容适配器对 `qwen3.8-flash-next` 自动启用按请求隔离的正
「单点登录（同账号仅一处在线）」由独立 `single_session_enabled` 控制，默认关闭；`sso_enabled` 仅表示既有外部单点/免登入口，两个功能分开。开启单处登录后，每次成功的账号密码、用户名口令、上游 token、并行面板、注册或初始化登录，都在数据库内原子递增该账号 `token_version` 后签新 JWT，旧会话立即返回 401，前端轮询 `/me` 后退出旧页面。失败登录不会踢人，关闭时允许多处会话并存；修改/重置密码仍始终撤销旧会话。并发登录通过同一行的原子递增保证最终仅最高版本会话有效；事务还比较已验证的密码/口令与权限快照，避免并发重置密码后旧凭据借新登录版本恢复。
### 任务详情技能免鉴权访问（2026-10-27）
右侧 `task-context` 与 `task-detail-update` 仅访问
`GET/PUT /api/force-situation/tasks/{taskId}/detail`，不再调用登录、领取 JWT、读取 token 环境变量
或发送 `Authorization`。两个脚本只保留 Gateway 地址配置；当前任务详情读写及版本读取路由在
Gateway 鉴权与 CSRF 中间件中直接开放，因此不依赖 `password_login_enabled`、`sso_enabled`、
账号密码、登录白名单或多实例 JWT 密钥配置。
既有定时详情技能仍会调用旧 `/api/parallel-agents/auth/token` 作为前置步骤。该接口在登录策略
拒绝时返回非 JWT 兼容值，使旧技能可以继续调用已开放的详情接口；兼容值不能访问其他受保护路由。
`biz-detail-assistant` 的所有补充信息、范围选择和写回确认都必须使用原生
`ask_clarification`；不得为该智能体排除该工具或由中间件伪造“用户协助”消息。原生澄清中间件负责
暂停本轮、输出标准卡片并在用户回复后恢复同一会话。
普通知识库默认详情页面：管理员可在知识库顶部选择“本系统 Wiki 索引”或“嵌入 WeKnora”，对应 `ordinary_knowledge_landing`（默认 `wiki`）。Wiki 库首次进入默认展示本系统索引，右上角“打开 WeKnora”切换到内嵌管理页；内嵌页可返回索引。非 Wiki 库仍进入 WeKnora 文档页。设置通过 `/api/system-settings/ordinary-knowledge-landing` 保存，仅管理员可修改；指定文章、文档、图谱的直达链接优先于默认设置。
助手知识库可独立于 WeKnora 使用：管理员进入助手知识库，选择或创建库后点击“上传文件生成 Wiki”。支持 Markdown/TXT/CSV、PDF、Word、Excel、PowerPoint（单文件 100 MB，解析正文上限 400 万字符）。文件先写入运行目录 `assistant-knowledge/files/{job_id}`，后台分段调用已配置的语言模型生成分类 Wiki、实体关系，再通过 DeerFlow 编码器对正文分段向量化，保存到助手库并汇入知识梳理总库。无需配置 WeKnora URL；配置后仍可直接上传，也保留 WeKnora 数据包导入。文件处理沿用管理员写入权限。日常界面默认采用 `knowledge_base_product_version=v1`，普通用户只展示普通知识库；管理员始终可管理助手知识库，并可在知识库顶部设置中切换到 `v2`，向普通用户开放助手知识库入口、选择器和关联跳转。该开关是可恢复的产品展示开关：隐藏时日常问答的新运行不带助手库选择，但不会删除已有助手库或历史绑定；助手知识的既有读/搜索接口契约不因该开关改变。
@@ -169,9 +227,9 @@ The single LangGraph agent (`lead_agent`) is the runtime entry point, created vi
- **Subagent delegation** for parallel task execution
- **System prompt** with skills injection, memory context, and working directory guidance
普通问答采用严格的输出载体门禁：只有本次请求明确要求 HTML、网页、页面、看板、大屏或 `.html` 文件时，才允许读取网页生成技能、调用页面模板工具或创建 HTML；“画流程图”“画图表”“画地图”“画思维导图”“做可视化”本身都不构成 HTML 意图。普通流程图直接在消息中输出 Mermaid，普通图表沿用既有消息内方式，普通地图输出 `map-json`。只有本次请求明确要求“思维导图 / 脑图 / 知识导图”或点名 `analysis-mindmap` 时，才会输出专用 `mindmap-json`；普通分析、总结、提纲和结构化 JSON 不会自动生成导图。未闭合的 `mindmap-json`（兼容历史 `mindmap` 围栏）会原样作为代码块流式显示，闭合且本地校验通过后才懒加载横向分析树预览。导图必须是一棵含唯一根节点的受限层级树：目标 12–24 个节点、硬上限 36 个，最多 4 层（含中心主题），中心主题至多 6 个一级维度、每个后续节点至多 3 个子节点；不得把报告的每个数据、段落或引用逐条转成节点。点击父节点本身，或其连线起点处的＋/−，即可展开/收起子节点，且连接线明确可见；全量弹窗的右上角含关闭与视图工具、右下角含由实际布局绘制的全图预览，消息预览同样支持拖动、滚轮缩放和视图工具；消息卡片的单个下载图标会打开 PNG、SVG、JSON 选项。明确的 HTML 请求必须进入沙箱生成 `/mnt/user-data/outputs/*.html` 并通过 `present_files` 打开右侧制品预览，禁止用聊天中的完整 HTML 代码块代替文件。前端负责本地校验和渲染，并兼容模型误写成普通 `json` 围栏的合法地图契约；只有包含地图字段且通过完整 schema 校验的 JSON 才会按地图渲染，其他 JSON 保持原代码块。地图卡片占满消息内容区，与步骤条左右边界对齐；格式错误时显示可读错误与原始数据，不中断整条消息。
普通问答采用严格的输出载体门禁：只有本次请求明确要求 HTML、网页、页面、看板、大屏或 `.html` 文件时，才允许读取网页生成技能、调用页面模板工具或创建 HTML；“画流程图”“画图表”“画地图”“画思维导图”“做可视化”本身都不构成 HTML 意图。普通流程图直接在消息中输出 Mermaid，普通图表沿用既有消息内方式，普通地图输出 `map-json`。只有本次请求明确要求“思维导图 / 脑图 / 知识导图”或点名 `analysis-mindmap` 时，才会输出专用 `mindmap-json`；普通分析、总结、提纲和结构化 JSON 不会自动生成导图。未闭合的 `mindmap-json`（兼容历史 `mindmap` 围栏）会原样作为代码块流式显示，闭合且本地校验通过后才懒加载横向分析树预览。导图必须是一棵含唯一根节点的受限层级树：目标 12–24 个节点、硬上限 36 个，最多 4 层（含中心主题），中心主题至多 6 个一级维度、每个后续节点至多 3 个子节点；不得把报告的每个数据、段落或引用逐条转成节点。点击父节点本身，或其连线起点处的＋/−，即可展开/收起子节点，且连接线明确可见；全量弹窗的右上角含关闭与视图工具、右下角含由实际布局绘制的全图预览，消息预览同样支持拖动、滚轮缩放和视图工具；消息卡片的单个下载图标会打开 PNG、SVG、JSON 选项。明确的 HTML 请求必须进入沙箱生成 `/mnt/user-data/outputs/*.html` 并通过 `present_files` 打开右侧制品预览，禁止用聊天中的完整 HTML 代码块代替文件。前端负责本地校验和渲染，并兼容模型误写成普通 `json` 围栏的合法地图契约；只有包含地图字段且通过完整 schema 校验的 JSON 才会按地图渲染，其他 JSON 保持原代码块。解析前会把常见的 `coords`/`from`/`to`/`lng`/`lat` 写法归一成契约字段；标点可按 `color` 着色（未写则自动配色）。只有用户明确要求连线时才输出 `lines`，此时连线带箭头并沿线循环流动；无法恢复的连线会被丢弃，标点仍可渲染。地图卡片占满消息内容区，与步骤条左右边界对齐；格式错误时显示可读错误与原始数据，不中断整条消息。
生成 `.html` 网页、看板或大屏时使用独立的离线绘图契约：流程图使用 `data-df-flow` + AntV G6（不使用 Mermaid），图表使用 `data-df-chart`，地图使用本地 GeoJSON + ECharts Canvas 的 `data-df-geo`。Gateway 从公开路径 `/api/public/html-runtime/` 提供浏览器版 `g6.min.js`、`echarts.min.js`、`bootstrap.min.css`、`bootstrap.bundle.min.js`、`alpine.min.js` 和统一初始化脚本 `deerflow-runtime.js`，以及中国省级、美国州级、日本都道府县、台湾地区县市四份地图资源（`china-provinces` / `us-states` / `japan-prefectures` / `taiwan-counties`）。Bootstrap 负责布局与标准弹窗，Alpine 负责轻量状态和动态列表；生成的 HTML 源码会直接引用这些后台文件，不依赖前端 React 构建产物或公网 CDN。设置里的「HTML 地图」可选 ECharts 本地 GeoJSON，或地图网页（Leaflet）；地图网页的瓦片模板是 `config.yaml -> html_runtime.map_tile_url`。运行时会等隐藏的逻辑页真实可见后再初始化 ECharts，并通过 `ResizeObserver` 跟随沙箱和页面容器尺寸变化，避免地图或图表首屏以零宽度渲染。区域/标点的 `description` 必须为纯文本，点击后由运行时显示详情浮层；需要筛选或切到明细页时监听 `deerflow:geo-map-click`，不能把链接、HTML 或 `onclick` 写进 `description`。深色 HTML 需声明 `data-df-theme="dark"`，其 ECharts 的全局、标题、图例、坐标轴和可见系列标签都必须显式使用浅色文字，避免 ECharts 默认黑字。这些文件作为 `packages/harness/deerflow/html_runtime/assets/` 的包资源随 harness wheel 一起发布，第三方许可同目录保存。`config.yaml -> html_runtime.base_url` 必须填写浏览器可访问的 Gateway Runtime 地址（本机默认 `http://localhost:8001/api/public/html-runtime`），运行时据此解析本地 GeoJSON 地址；内网部署只需把它替换为实际 Gateway 地址。用户上传的栅格图片若要作为生成 HTML 的背景图，智能体会使用同一 Gateway 源下的签名地址 `/api/public/html-images/{token}` 写入 `background-image`，不要写沙箱路径或制品接口。该能力不改变普通问答消息列表，普通问答地图仍走原有 `map-json` 前端瓦片渲染与 `frontend-web/public/runtime-config.js` 的 `VITE_MAP_TILE_URL`。地图大屏使用“中间地图舞台”而非整页地图背景：中间地图保持可见和可点击，两侧为窄信息栏、底部为摘要面板；提到的城市和行政区必须分别写入 `markers` 与高亮 `regions`。
生成 `.html` 网页、看板或大屏时使用独立的离线绘图契约：流程图使用 `data-df-flow` + AntV G6（不使用 Mermaid），图表使用 `data-df-chart`，地图使用本地 GeoJSON + ECharts Canvas 的 `data-df-geo`。Gateway 从公开路径 `/api/public/html-runtime/` 提供浏览器版 `g6.min.js`、`echarts.min.js`、`bootstrap.min.css`、`bootstrap.bundle.min.js`、`alpine.min.js` 和统一初始化脚本 `deerflow-runtime.js`，以及中国省级、美国州级、日本都道府县、台湾地区县市四份地图资源（`china-provinces` / `us-states` / `japan-prefectures` / `taiwan-counties`）。Bootstrap 负责布局与标准弹窗，Alpine 负责轻量状态和动态列表；生成的 HTML 源码会直接引用这些后台文件，不依赖前端 React 构建产物或公网 CDN。设置里的「HTML 地图」可选 ECharts 本地 GeoJSON，或地图网页（Leaflet）；地图网页的瓦片模板在设置 → 外观的「地图瓦片地址」中配置。运行时会等隐藏的逻辑页真实可见后再初始化 ECharts，并通过 `ResizeObserver` 跟随沙箱和页面容器尺寸变化，避免地图或图表首屏以零宽度渲染。区域/标点的 `description` 必须为纯文本，点击后由运行时显示详情浮层；需要筛选或切到明细页时监听 `deerflow:geo-map-click`，不能把链接、HTML 或 `onclick` 写进 `description`。深色 HTML 需声明 `data-df-theme="dark"`，其 ECharts 的全局、标题、图例、坐标轴和可见系列标签都必须显式使用浅色文字，避免 ECharts 默认黑字。这些文件作为 `packages/harness/deerflow/html_runtime/assets/` 的包资源随 harness wheel 一起发布，第三方许可同目录保存。设置 → 外观的「HTML 运行库地址」填写浏览器可访问的 Gateway Runtime 地址（未配置时默认 `http://localhost:8001/api/public/html-runtime`），运行时据此解析本地 GeoJSON 地址；内网部署在设置弹窗里改成用户能够打开的域名即可。用户上传的栅格图片若要作为生成 HTML 的背景图，智能体会使用同一 Gateway 源下的签名地址 `/api/public/html-images/{token}` 写入 `background-image`，不要写沙箱路径或制品接口。该能力不改变普通问答消息列表，普通问答地图仍走原有 `map-json` 前端瓦片渲染与 `frontend-web/public/runtime-config.js` 的 `VITE_MAP_TILE_URL`。地图大屏使用“中间地图舞台”而非整页地图背景：中间地图保持可见和可点击，两侧为窄信息栏、底部为摘要面板；提到的城市和行政区必须分别写入 `markers` 与高亮 `regions`。
从已有 Markdown 报告生成 HTML 时，必须根据材料盘点 Mermaid/流程图、图表及地图，并在 `page_report_template validate` 的 `required_features` 和 `page_brief` 中如实声明。流程图必须包含 `g6.min.js`、`deerflow-runtime.js`、`.df-flow` 与 `data-df-flow`，并拒绝 Mermaid、手写 HTML/CSS 流程节点以及页面内自行初始化 `G6.Graph`；图表和地图分别检查 ECharts 资源与 `data-df-chart` / `data-df-geo` 声明式数据块。`data-df-chart` 必须是严格 JSON，校验会拒绝函数、箭头函数和其他 JavaScript 回调；颜色使用字面量或数据项 `itemStyle`，提示文本使用字符串模板。地图只能使用上述四个内置 GeoJSON，校验会拒绝 Leaflet、`data-df-map`、瓦片和页面内自行初始化 ECharts。默认页面从零开始直接写入最终 outputs HTML，不读取或预览模板；只有用户明确要求模板时，prepare 才会把要求持久化到同名 brief，validate 会自动恢复并与调用参数合并。校验失败时不得交付，必须修复后重新校验。
@@ -292,7 +350,7 @@ FastAPI application providing REST endpoints for frontend integration:
| `GET/POST /api/agents` | List visible agents or create a user-owned agent |
| `GET/PUT/DELETE /api/agents/{id}` | Read, update, or delete an id-addressed agent |
| `GET/POST/PUT/DELETE /api/roundtable-chains` | Manage user-owned roundtable business chains; each seat may optionally carry `position_id` for the separate position-collaboration workspace, plus `human_validation` to mark a Step 2 client-side pause checkpoint after that seat/stage completes (the original roundtable has no extra validation card) |
| `GET/PUT /api/business-mapping` | Read global business mappings (seeded 3Q/6BF/7BF/8BF/AFX plus independent slots 3q-0…3q-6, 6-1…6-4, 7-1…7-4). Updates to 3Q/6BF/7BF and the independent slots require an administrator; the 8BF public business-chain mapping is maintained exclusively by the trusted `lqq` account (derived from the authenticated email prefix). |
| `GET/PUT /api/business-mapping` | Read global business mappings (seeded 3Q/6BF/7BF/8BF/AFX plus independent slots 3q-0…3q-6, 6-1…6-4, 7-1…7-4). `3q-1…3q-6` seed as custom-type rows and stay off the configuration-page list. Updates to 3Q/6BF/7BF and the independent slots require an administrator; the 8BF public business-chain mapping is maintained exclusively by the trusted `lqq` account (derived from the authenticated email prefix). |
| `GET/POST/PUT /api/position-roundtable/sessions` | Standalone position-collaboration sessions: freeze task/intent/chain snapshots, index one direct-agent node per seat, build upstream handoff context (including bounded UTF-8 excerpts of text artifacts), and make archived sessions read-only without invoking the original roundtable coordinator or jobs. Built-in summary/action-plan completion mirrors visible report text into a Markdown artifact when a model omits its final `write_file` call. `POST /sessions/{id}/nodes/{nodeKey}/reject` accepts an optional reason, marks the rejected delivery for rework, and invalidates affected downstream deliveries so they are regenerated in order. The same router is also mounted at `/api/multi-agent/position-roundtable/*` for intranet gateways that only forward the established multi-agent namespace; the compatibility mount is intentionally omitted from OpenAPI. |
| `GET /api/memory` | Retrieve memory data |
| `POST /api/memory/reload` | Force memory reload |
@@ -1579,6 +1637,99 @@ Gateway 启动时会尽力执行应用 ORM、LangGraph Checkpointer 和 Store
`Automatic ORM schema creation failed`、`in-memory fallback` 或启动填充失败信息修复
数据库权限和表结构，并重启服务恢复持久化能力。
## Thread query performance and PostgreSQL ORM deadlines
Thread search/count now push common metadata and system-thread filters into SQL,
preserving private ownership and shared taskId + agent/action filtering. Nested
filters, the openGauss dialect and PG/MySQL numeric filters retain the exact
compatibility path. MySQL string identifiers compare case and trailing spaces
exactly. These changes reduce application-side scanning; JSON indexes still need
to be selected from the deployed database's query plan.
For PostgreSQL ORM connections, `database.pool_recycle` defaults to 300 seconds
when omitted, and `database.command_timeout` defaults to 30 seconds. Set the latter
to a positive finite number or `null` to disable. Explicit pool settings remain
effective; MySQL/openGauss keep their 1800-second recycle default. The asyncpg
deadline is also applied when rebuilding the engine after database creation.
MySQL/SQLite drivers do not receive asyncpg arguments. These ORM settings do not
remove checkpoint saver lock waits or configure the separate psycopg pool.
Thread creation now defaults to metadata-only persistence: the first graph run
creates its checkpoint. `initialize_checkpoint: true` retains eager initialization
for integration clients. Authorized empty threads expose an empty state/history;
private ownership, task sharing and agent/action partitioning remain unchanged.
Thread metadata GET uses the business database; `include_state=true` opts into
checkpoint values, and checkpoint-only legacy threads retain their fallback.
The frontend now releases the composer on an explicit stream end and refreshes
final history in the background. Thread/run/generation guards reject late results
after another send or navigation, including stale replica snapshots. Sync timeout
or failure preserves the streamed answer; EOF without an end still uses recovery.
Login, identity, user management and SQL business repositories use `database`;
with `backend: mysql` they do not query the PG checkpointer. Runtime initialization,
legacy LangGraph Store migration and explicit conversation/agent-state operations
can still use PG. This change does not migrate those stores or existing data.
See [thread latency implementation and database separation](docs/THREAD_LATENCY_OPTIMIZATION_ZH.md)
for configuration precedence, MySQL-only thread creation, background history
refresh, compatibility limits and validation scope.
Artifact directory scans, skill ZIP previews, content sniffing and text writes now
use a bounded file executor separate from login's default executor. Completed runs
leave the live registry after a five-minute grace period only once their tasks
have stopped and their final snapshots have been verified in the RunStore. SQL
history, checkpoints and the existing SSE replay buffer are not deleted.
The `run_events.backend: memory` compatibility setting now uses a local SQLite
event spool at `${DEER_FLOW_HOME}/run-events/events.sqlite3` (default
`.deer-flow/run-events/events.sqlite3`). Full messages and events stay on disk;
there is no automatic history TTL or content truncation. Keep that runtime
directory on a writable persistent local volume, including its WAL files.
Explicit `db` continues to use the business database (MySQL when configured),
and `jsonl` keeps its existing files while moving disk IO off the event loop.
No existing SQL/JSONL events or conversation checkpoints are migrated.
Background subagents and roundtable turns use user round-robin queues with both
global and per-user admission limits. Queued work occupies no execution slot;
cancellation must finish the actual child before releasing its slot. Optional
`config.yaml` budgets (effective after restart):
```yaml
runtime_contention:
completed_run_retention_seconds: 300
run_sweep_seconds: 10
subagent_concurrency: 3
subagent_per_user: 2
background_concurrency: 4
background_per_user: 2
queue_limit: 256
```
The existing `ROUNDTABLE_BACKGROUND_SLOTS_PER_USER` variable still overrides the
roundtable per-user budget, capped by its global limit. Each user also has a
bounded pending queue; overload returns an explicit failure rather than growing
an unlimited backlog. A subagent deadline includes its queue wait.
Multi-agent roundtable step 2 has a separate room cap. Administrators change
it under Settings → 会商并发 (`roundtable_room_limit`, default on, 3 rooms)
without restarting. The count is the number of live roundtable executions
across every account (foreground leases/live LangGraph runs plus executing
background jobs), not the current login. Merely opening a tab or viewing chat
history does not count, but two tabs that both run the same saved draft count
as two rooms. All seats and Step 3 work in one execution share one room;
foreground-to-background handoff keeps one room. A full pool keeps the user on
step 1.
Roundtable seat/coordinator creation is metadata-only too. The authoritative seat
list is included in the first real leader input, including for older clients;
ordinary messages, broadcasts and resumable execution retain their persistence.
The master integration also retains dedicated task deep-link history cards and
their indexed queries, plus bounded display snapshots for long conversations.
See [multi-user contention audit and implementation](docs/MULTI_USER_CONTENTION_AUDIT_ZH.md)
for regression scope, remaining shared resources and production metrics. Local
test results do not establish production capacity.
## openGauss / GaussDB application database
The application ORM can use a PG-compatible openGauss/GaussDB database through
@@ -1661,12 +1812,14 @@ Beta(8,2) 先验抑制小样本极值；零样本返回 `score: null` 和
`VITE_COMPACT_MENU_TYPE` 隔离 A/B 两套菜单。
## 技能读取、模型日志与线程池并发优化
实际加载的 `config.yaml`（`DEER_FLOW_CONFIG_PATH` 指定文件优先）支持以下四个独立开关。
实际加载的 `config.yaml`（`DEER_FLOW_CONFIG_PATH` 指定文件优先）支持以下独立开关。
`skill_view_reuse_enabled` 和 `skill_view_cache_enabled` 在配置里不写时默认开启；
要恢复每次工具调用都全量刷新技能视图，把对应项写成 `false` 后重启。另外两项仍默认关闭：
要恢复每次工具调用都全量刷新技能视图，把对应项写成 `false` 后重启。
模型计时与线程池开关默认关闭，工具统计写入也默认暂停：
```yaml
runtime_performance:
tool_metrics_enabled: false
skill_view_reuse_enabled: true
skill_view_cache_enabled: true
model_timing_nonblocking_enabled: true
@@ -1674,6 +1827,13 @@ runtime_performance:
asyncio_thread_pool_size: 100
```
- `tool_metrics_enabled`：工具/技能调用统计写入开关，**缺省为 false（暂停）**，旧部署配置
没有此项时也暂停。主智能体和子智能体不再装载工具统计中间件，不创建统计写库任务、
不执行技能统计合并查询，从源头减少与登录共用业务数据库连接池的竞争。
正常工具调用、错误处理、问答消息/checkpoint 和模型调用统计继续运行；已有
`tool_call_metrics` 记录与查询/导出接口保留。暂停期间工具/技能审计明细及排行榜的
工具/技能调用计数不再新增，且不缓存或事后补录。恢复时设为 `true` 并重启后台。
部署新代码并重启后才对所有运行实例生效；这不消除历史统计查询或其他数据库操作的开销。
- `skill_view_reuse_enabled`：一次沙箱工具调用共用一份已验证技能视图，路径检查、
本地/AIO 沙箱初始化、命令转换和输出脱敏不再反复刷新。下一次工具调用重新验证，
不跨问答永久沿用启停状态。
@@ -1694,7 +1854,7 @@ runtime_performance:
开启时选中变量的无效值使启动失败。线程按需创建，不会在启动时立即创建 100 个线程。
修改后重启后台以确保全部运行实例使用一致配置；回退时将相应项设为 `false` 并重启，
无需删除技能或缓存目录。启动/配置加载日志 `RUNTIME_PERFORMANCE` 会打印四项开关及 YAML 中的上限。
无需删除技能或缓存目录。启动/配置加载日志 `RUNTIME_PERFORMANCE` 会打印开关及 YAML 中的上限。
`GATEWAY_ASYNCIO_EXECUTOR` 才是线程池最终生效值（含环境变量优先级）及来源；线程池改动必须重启生效。
技能日志 `SKILL_VIEW_CACHE` 含 `status`、`lock_wait_ms`、`refresh_ms`、`total_ms`、
源/目标条目数及启用技能数；正常命中为 DEBUG，重建/合并等待为 INFO，配合
offline-backend-20260512/backend/app/gateway/admin_audit_retention.py
+290
View File
@@ -0,0 +1,290 @@
"""Bounded retention cleanup for administrator audit and analytics data.
This deliberately removes only data written for operational inspection:
provider/tool call metrics, compact terminal run summaries, and precomputed
leaderboard snapshots. It does not touch user threads or LangGraph
checkpoints, which are user conversation history and require a separate,
explicit retention policy.
"""
from __future__ import annotations
import asyncio
import logging
from collections.abc import Awaitable, Callable
from datetime import UTC, datetime, timedelta
from time import monotonic
from typing import Any
from sqlalchemy import and_, delete, or_, select
from app.gateway.admin_query_window import ADMIN_AUDIT_RETENTION_DAYS
from deerflow.persistence.admin_stats.model import AdminLeaderboardDailyStatRow
from deerflow.persistence.types import BEIJING_TZ
logger = logging.getLogger(__name__)
_DEFAULT_BATCH_SIZE = 100
_MAX_BATCH_SIZE = 500
_MAX_SNAPSHOT_BATCH_SIZE = 200
_DEFAULT_MAX_BATCHES_PER_STORE = 4
_MAX_BATCHES_PER_STORE = 16
_DEFAULT_INTER_BATCH_PAUSE_SECONDS = 0.05
_DEFAULT_MAX_SWEEP_SECONDS = 8.0
def _bounded_int(value: int, *, default: int, maximum: int) -> int:
"""Return a safe positive integer for background maintenance work."""
try:
return max(1, min(int(value), maximum))
except (TypeError, ValueError):
return default
def _bounded_seconds(value: float, *, default: float, maximum: float) -> float:
"""Return a non-negative, finite-enough scheduler duration."""
try:
return max(0.0, min(float(value), maximum))
except (TypeError, ValueError):
return default
async def _purge_store_batch(
store: Any,
cutoff: datetime,
*,
batch_size: int,
method_name: str = "purge_before",
) -> int:
"""Delete one committed, primary-key-bounded batch from an audit store."""
if store is None:
return 0
purge = getattr(store, method_name, None)
if purge is None:
return 0
return int(await purge(cutoff, batch_size=batch_size) or 0)
async def _purge_snapshot_batch(
session_factory,
cutoff_date: str,
*,
batch_size: int,
) -> int:
"""Delete one short transaction of old leaderboard rows.
Snapshot rows use a composite primary key. Selecting those keys first is
deliberately more conservative than deleting whole historical dates: a
single date can have many configuration hashes, and deleting all of them
in one transaction could otherwise hold a write lock for too long.
"""
if session_factory is None:
return 0
size = min(batch_size, _MAX_SNAPSHOT_BATCH_SIZE)
async with session_factory() as session:
keys = list(
(
await session.execute(
select(
AdminLeaderboardDailyStatRow.stat_date,
AdminLeaderboardDailyStatRow.settings_hash,
)
.where(AdminLeaderboardDailyStatRow.stat_date < cutoff_date)
.order_by(
AdminLeaderboardDailyStatRow.stat_date.asc(),
AdminLeaderboardDailyStatRow.settings_hash.asc(),
)
.limit(size)
)
).all()
)
if not keys:
return 0
result = await session.execute(
delete(AdminLeaderboardDailyStatRow).where(
or_(
*(
and_(
AdminLeaderboardDailyStatRow.stat_date == stat_date,
AdminLeaderboardDailyStatRow.settings_hash == settings_hash,
)
for stat_date, settings_hash in keys
)
)
)
)
await session.commit()
rowcount = result.rowcount
return len(keys) if rowcount is None or rowcount < 0 else int(rowcount)
async def sweep_admin_audit_retention(
*,
session_factory,
llm_metrics_store: Any = None,
tool_metrics_store: Any = None,
run_store: Any = None,
retention_days: int = ADMIN_AUDIT_RETENTION_DAYS,
now: datetime | None = None,
batch_size: int = _DEFAULT_BATCH_SIZE,
max_batches_per_store: int = _DEFAULT_MAX_BATCHES_PER_STORE,
inter_batch_pause_seconds: float = _DEFAULT_INTER_BATCH_PAUSE_SECONDS,
max_sweep_seconds: float = _DEFAULT_MAX_SWEEP_SECONDS,
should_stop: Callable[[], bool] | None = None,
on_progress: Callable[[dict[str, Any]], Awaitable[None] | None] | None = None,
) -> dict[str, int]:
"""Run one safe retention pass and return deletion counts.
Every delete is a small independent transaction. The sources are visited
round-robin, and the scheduler checks its wall-clock budget between
committed batches while cooperatively pausing between them. A historic
backlog is therefore drained over later sweeps rather than monopolising
the database or keeping a long write lock.
"""
days = max(1, int(retention_days))
cutoff = (now or datetime.now(UTC)) - timedelta(days=days)
cutoff_date = cutoff.astimezone(BEIJING_TZ).date().isoformat()
safe_batch_size = _bounded_int(
batch_size,
default=_DEFAULT_BATCH_SIZE,
maximum=_MAX_BATCH_SIZE,
)
safe_max_batches = _bounded_int(
max_batches_per_store,
default=_DEFAULT_MAX_BATCHES_PER_STORE,
maximum=_MAX_BATCHES_PER_STORE,
)
safe_pause_seconds = _bounded_seconds(
inter_batch_pause_seconds,
default=_DEFAULT_INTER_BATCH_PAUSE_SECONDS,
maximum=1.0,
)
safe_max_sweep_seconds = _bounded_seconds(
max_sweep_seconds,
default=_DEFAULT_MAX_SWEEP_SECONDS,
maximum=60.0,
)
if safe_max_sweep_seconds <= 0:
safe_max_sweep_seconds = _DEFAULT_MAX_SWEEP_SECONDS
async def purge_llm_metrics() -> int:
return await _purge_store_batch(
llm_metrics_store,
cutoff,
batch_size=safe_batch_size,
)
async def purge_tool_metrics() -> int:
return await _purge_store_batch(
tool_metrics_store,
cutoff,
batch_size=safe_batch_size,
)
async def purge_run_summaries() -> int:
return await _purge_store_batch(
run_store,
cutoff,
batch_size=safe_batch_size,
method_name="purge_terminal_before",
)
async def purge_snapshots() -> int:
return await _purge_snapshot_batch(
session_factory,
cutoff_date,
batch_size=safe_batch_size,
)
sources: tuple[tuple[str, Callable[[], Awaitable[int]], int], ...] = (
("llm_metrics", purge_llm_metrics, safe_batch_size),
("tool_metrics", purge_tool_metrics, safe_batch_size),
("run_summaries", purge_run_summaries, safe_batch_size),
("leaderboard_snapshots", purge_snapshots, min(safe_batch_size, _MAX_SNAPSHOT_BATCH_SIZE)),
)
result = {name: 0 for name, _purge, _full_batch_size in sources}
active = {name: True for name in result}
deadline = monotonic() + safe_max_sweep_seconds
exhausted_budget = False
async def report(*, status: str, round_number: int, current_source: str | None) -> None:
if on_progress is None:
return
payload: dict[str, Any] = {
"status": status,
"round": round_number,
"max_rounds": safe_max_batches,
"current_source": current_source,
"deleted": dict(result),
}
try:
pending = on_progress(payload)
if pending is not None:
await pending
except Exception:
logger.warning("Unable to publish admin audit retention progress", exc_info=True)
stopped = False
for _round in range(safe_max_batches):
any_deletes = False
for name, purge, full_batch_size in sources:
if should_stop is not None and should_stop():
stopped = True
break
if not active[name]:
continue
if monotonic() >= deadline:
exhausted_budget = True
break
await report(status="running", round_number=_round + 1, current_source=name)
try:
count = await purge()
except Exception:
# A busy/temporarily unavailable table must not keep this
# maintenance sweep alive, nor prevent other audit tables
# from receiving their small cleanup turn. The next scheduled
# sweep will retry it.
logger.warning(
"admin audit retention skipped %s this sweep; will retry later",
name,
exc_info=True,
)
active[name] = False
continue
result[name] += count
await report(status="running", round_number=_round + 1, current_source=name)
any_deletes = any_deletes or count > 0
if count < full_batch_size:
active[name] = False
if count and safe_pause_seconds:
# The transaction above has committed. Yielding here gives
# interactive reads/writes a chance to acquire the connection
# before this low-priority maintenance work continues.
await asyncio.sleep(safe_pause_seconds)
if stopped or exhausted_budget or not any(active.values()) or not any_deletes:
break
if stopped or (should_stop is not None and should_stop()):
await report(status="paused", round_number=min(_round + 1, safe_max_batches), current_source=None)
more_work_likely = any(active.values())
if any(result.values()) or exhausted_budget or more_work_likely:
logger.info(
"admin audit retention sweep: llm_metrics=%d tool_metrics=%d run_summaries=%d "
"leaderboard_snapshots=%d cutoff=%s batch_size=%d max_batches=%d "
"time_budget_seconds=%.2f budget_exhausted=%s more_work_likely=%s",
result["llm_metrics"],
result["tool_metrics"],
result["run_summaries"],
result["leaderboard_snapshots"],
cutoff.isoformat(),
safe_batch_size,
safe_max_batches,
safe_max_sweep_seconds,
exhausted_budget,
more_work_likely,
)
return result
__all__ = ["sweep_admin_audit_retention"]
offline-backend-20260512/backend/app/gateway/admin_audit_retention_control.py
+59
View File
@@ -0,0 +1,59 @@
"""Shared on-disk control and progress state for admin audit retention."""
from __future__ import annotations
import json
import os
from pathlib import Path
from typing import Any
def status_path(runtime_root: Path) -> Path:
return runtime_root / ".admin_audit_retention.status.json"
def pause_path(runtime_root: Path) -> Path:
return runtime_root / ".admin_audit_retention.paused"
def is_paused(runtime_root: Path) -> bool:
return pause_path(runtime_root).exists()
def set_paused(runtime_root: Path, paused: bool) -> None:
runtime_root.mkdir(parents=True, exist_ok=True)
marker = pause_path(runtime_root)
if paused:
marker.touch(exist_ok=True)
else:
marker.unlink(missing_ok=True)
def read_status(runtime_root: Path) -> dict[str, Any]:
path = status_path(runtime_root)
try:
value = json.loads(path.read_text(encoding="utf-8"))
except (OSError, ValueError, TypeError):
return {
"status": "scheduled",
"current_source": None,
"round": 0,
"max_rounds": 4,
"deleted": {},
"last_sweep": None,
"started_at": None,
"next_run_at": None,
"retention_days": 90,
}
return value if isinstance(value, dict) else {}
def write_status(runtime_root: Path, value: dict[str, Any]) -> None:
runtime_root.mkdir(parents=True, exist_ok=True)
path = status_path(runtime_root)
temporary = path.with_name(f"{path.name}.{os.getpid()}.tmp")
temporary.write_text(json.dumps(value, ensure_ascii=False), encoding="utf-8")
os.replace(temporary, path)
__all__ = ["is_paused", "read_status", "set_paused", "write_status"]
offline-backend-20260512/backend/app/gateway/admin_query_window.py
+104
View File
@@ -0,0 +1,104 @@
"""Shared time bounds for expensive administrator analytics reads.
The administrative audit, leaderboard, and metric pages are intentionally a
recent operational view rather than an unbounded reporting warehouse. Keep
the policy in one small module so every route (including downloads) applies
the same rolling window before it reaches SQL.
"""
from __future__ import annotations
from datetime import datetime, time, timedelta
from fastapi import HTTPException
from deerflow.persistence.types import BEIJING_TZ
ADMIN_QUERY_WINDOW_DAYS = 30
ADMIN_AUDIT_RETENTION_DAYS = 90
def _as_beijing(value: datetime) -> datetime:
"""Treat timezone-free query parameters as local Beijing time."""
if value.tzinfo is None:
return value.replace(tzinfo=BEIJING_TZ)
return value.astimezone(BEIJING_TZ)
def _parse_datetime(value: str, *, field_name: str) -> datetime:
text = value.strip()
if not text:
raise HTTPException(status_code=422, detail=f"{field_name} must be an ISO 8601 datetime")
if text.endswith("Z"):
text = text[:-1] + "+00:00"
try:
return _as_beijing(datetime.fromisoformat(text))
except ValueError as exc:
raise HTTPException(
status_code=422,
detail=f"{field_name} must be an ISO 8601 datetime",
) from exc
def resolve_admin_query_window(
*,
days: int = ADMIN_QUERY_WINDOW_DAYS,
since: str | None = None,
until: str | None = None,
now: datetime | None = None,
) -> tuple[datetime, datetime, int]:
"""Return a validated, at-most-30-calendar-day recent query range.
Hand-crafted URLs cannot bypass this guard: the whole range must fall in
the rolling 30-day window, not merely have a duration shorter than 30 days.
A future ``until`` is reduced to the current instant so normal date inputs
that represent the rest of today remain convenient.
"""
if not 1 <= days <= ADMIN_QUERY_WINDOW_DAYS:
raise HTTPException(
status_code=422,
detail=f"管理员查询仅支持最近 {ADMIN_QUERY_WINDOW_DAYS} 天的数据",
)
current = _as_beijing(now or datetime.now(BEIJING_TZ))
end = _parse_datetime(until, field_name="until") if until else current
if end > current:
end = current
earliest_date = current.date() - timedelta(days=ADMIN_QUERY_WINDOW_DAYS - 1)
default_start_date = end.date() - timedelta(days=days - 1)
start = (
_parse_datetime(since, field_name="since")
if since
else datetime.combine(default_start_date, time.min, tzinfo=BEIJING_TZ)
)
if start > end:
raise HTTPException(status_code=422, detail="since must be before until")
if start.date() < earliest_date or end.date() < earliest_date:
raise HTTPException(
status_code=422,
detail=f"管理员查询仅支持最近 {ADMIN_QUERY_WINDOW_DAYS} 天的数据",
)
range_days = (end.date() - start.date()).days + 1
if range_days > ADMIN_QUERY_WINDOW_DAYS:
raise HTTPException(
status_code=422,
detail=f"管理员查询仅支持最近 {ADMIN_QUERY_WINDOW_DAYS} 天的数据",
)
return start, end, range_days
def recent_admin_query_cutoff(*, now: datetime | None = None) -> datetime:
"""Return the inclusive earliest timestamp in the current admin window."""
start, _end, _days = resolve_admin_query_window(now=now)
return start
__all__ = [
"ADMIN_AUDIT_RETENTION_DAYS",
"ADMIN_QUERY_WINDOW_DAYS",
"recent_admin_query_cutoff",
"resolve_admin_query_window",
]
offline-backend-20260512/backend/app/gateway/api_cache_catalog.py
+25
View File
@@ -0,0 +1,25 @@
"""Explicit, reviewable catalogue of Redis-backed read APIs."""
API_CACHE_CATALOG = (
("agents", "智能体列表", ["/api/agents"], "按用户、权限及筛选条件隔离"),
("skills", "技能列表", ["/api/skills"], "按用户、权限及筛选条件隔离"),
("sensitive_words", "展示脱敏词", ["/api/sensitive-words"], "全局配置"),
("user_preferences", "个人偏好", ["/api/user-preferences"], "按用户隔离"),
("user_prompts", "常用提示词", ["/api/user-prompts"], "按用户隔离"),
("recommended_questions", "推荐问题", ["/api/recommended-questions"], "按用户及筛选条件隔离"),
("light_apps", "轻应用目录", ["/api/light-apps", "/api/light-apps/sidebar", "/api/light-apps/{app_id}"], "全局共享；按类型、菜单、搜索及标签筛选隔离"),
("menu_overrides", "完整菜单配置", ["/api/menu-overrides"], "按用户及管理员角色隔离"),
("compact_menu_overrides", "简洁菜单配置", ["/api/compact-menu-overrides"], "按用户、管理员角色及 A/B 菜单类型隔离"),
("task_buttons", "任务跳转按钮", ["/api/task-buttons"], "全局配置"),
("business_mapping", "业务智能体映射", ["/api/business-mapping"], "全局配置"),
(
"html_templates", "HTML 模板目录",
["/api/html-templates", "/api/html-templates/{template_id}", "/api/html-templates/{template_id}/content", "/api/scheduled-tasks/html-templates"],
"全局共享；分别缓存列表、详情和正文",
),
("models", "模型能力目录", ["/api/models", "/api/models/{model_name}"], "仅缓存数据库能力目录；模型配置实时读取"),
("tags", "标签目录", ["/api/tags", "/api/tags/assignments/{target_type}/{target_id}"], "全局共享；按标签类型及目标资源隔离"),
("browser_status", "浏览器状态", ["/api/browser-context/status"], "全局配置；访问开关优先"),
)
API_READ_CACHE_COLLECTIONS = tuple(item[0] for item in API_CACHE_CATALOG)
offline-backend-20260512/backend/app/gateway/api_read_cache.py
+674
View File
@@ -0,0 +1,674 @@
"""Small, optional Redis cache for the homepage agent and skill catalogues.
MySQL remains the source of truth. Redis can use the dedicated
``api_read_cache`` connection, or reuse ``concurrency.redis`` when explicitly
enabled; failures and missing configuration fall through to the normal
MySQL-backed request path.
"""
from __future__ import annotations
import asyncio
import hashlib
import json
import logging
import secrets
import time
from typing import Any
from weakref import WeakKeyDictionary
from app.gateway.api_cache_catalog import API_CACHE_CATALOG, API_READ_CACHE_COLLECTIONS
logger = logging.getLogger(__name__)
_CACHE_SUFFIX = "api_read_cache:v1"
_DEFAULT_CACHE_TTL_SECONDS = 300
_MIN_CACHE_TTL_SECONDS = 1
_MAX_CACHE_TTL_SECONDS = 3600
_clients: dict[tuple[str, str, int, int], Any] = {}
_last_warning_at: dict[str, float] = {}
_VERSION_UNSET = object()
_api_read_cache_settings_cache: tuple[float, int | None, bool, int, dict[str, bool], bool, str] | None = None
_enabled_cache_ttl_seconds = 2.0
_save_locks: WeakKeyDictionary[asyncio.AbstractEventLoop, asyncio.Lock] = WeakKeyDictionary()
_enabled_locks: dict[tuple[int, int], asyncio.Lock] = {}
_fill_events: dict[tuple[int, str], asyncio.Event] = {}
_fill_local_payloads: dict[tuple[int, str], tuple[float, str, bytes]] = {}
def _redis_settings(config: Any, *, enabled: bool | None = None) -> tuple[str, str | None, str] | None:
concurrency = getattr(config, "concurrency", None)
concurrency_redis = getattr(concurrency, "redis", None)
api_cache = getattr(config, "api_read_cache", None)
api_enabled = bool(getattr(api_cache, "enabled", False)) if enabled is None else enabled
concurrency_enabled = bool(getattr(concurrency_redis, "enabled", False))
if not api_enabled:
return None
url = str(getattr(api_cache, "url", None) or (getattr(concurrency_redis, "url", None) if concurrency_enabled else None) or "").strip()
if not url:
return None
namespace = str(getattr(api_cache, "namespace", "cmzs_deerflow:homepage") if api_enabled else f"{getattr(concurrency_redis, 'namespace', 'cmzs_deerflow:concurrency')}:homepage")
password = getattr(api_cache, "password", None)
if not getattr(api_cache, "url", None) and concurrency_enabled:
password = password or getattr(concurrency_redis, "password", None)
return url, password, namespace.strip().rstrip(":")
def _client(config: Any, *, enabled: bool | None = None, epoch: str | None = None) -> tuple[Any, str] | None:
settings = _redis_settings(config, enabled=enabled)
if settings is None:
return None
url, password, namespace = settings
api_cache = getattr(config, "api_read_cache", None)
redis_config = getattr(getattr(config, "concurrency", None), "redis", None)
timeout_config = api_cache if (enabled if enabled is not None else getattr(api_cache, "enabled", False)) else redis_config
connect_timeout_ms = int(getattr(timeout_config, "socket_connect_timeout_ms", 250) or 250)
socket_timeout_ms = int(getattr(timeout_config, "socket_timeout_ms", 750) or 750)
identity = (url, str(password or ""), connect_timeout_ms, socket_timeout_ms)
client = _clients.get(identity)
if client is None:
try:
from redis.asyncio import Redis
except ImportError:
_warn_once(namespace, "redis package is not installed")
return None
client = Redis.from_url(
url,
password=password,
socket_connect_timeout=max(0.001, connect_timeout_ms / 1000),
socket_timeout=max(0.001, socket_timeout_ms / 1000),
decode_responses=False,
# Homepage bursts commonly arrive in parallel (e.g. 70 users).
# 70 users can load several homepage APIs at once; keep enough
# headroom for the combined burst instead of queuing Redis reads.
max_connections=256,
)
_clients[identity] = client
return client, f"{namespace}:{_CACHE_SUFFIX}:epoch:{_cache_epoch(config) if epoch is None else epoch}"
def _cache_epoch(config: Any) -> str:
current = _api_read_cache_settings_cache
if current is not None and current[1] in (None, id(config)):
return current[6]
return "0"
def _scoped_version(epoch: str, generation: str) -> str:
# Initial deployments keep the original generation value; later saves use
# an opaque epoch marker so an in-flight old fill cannot enter a new cache.
return generation if epoch == "0" else f"{epoch}.{generation}"
async def get_api_read_cache_settings(
config: Any,
*,
force_refresh: bool = False,
) -> tuple[bool, int]:
"""Read the admin toggle and TTL from MySQL, refreshing at most every 2s.
The YAML flag is retained as the compatibility default for deployments that
have not yet created the settings row. Once an administrator saves the
setting, MySQL is authoritative across restarts and worker containers.
"""
global _api_read_cache_settings_cache
now = time.monotonic()
if not force_refresh and _api_read_cache_settings_cache is not None and now - _api_read_cache_settings_cache[0] < _enabled_cache_ttl_seconds and _api_read_cache_settings_cache[1] in (None, id(config)):
return _api_read_cache_settings_cache[2], _api_read_cache_settings_cache[3]
loop = asyncio.get_running_loop()
lock_key = (id(loop), id(config))
setting_lock = _enabled_locks.get(lock_key)
if setting_lock is None:
setting_lock = _enabled_locks[lock_key] = asyncio.Lock()
if len(_enabled_locks) > 64:
_enabled_locks.pop(next(iter(_enabled_locks)))
async with setting_lock:
now = time.monotonic()
if not force_refresh and _api_read_cache_settings_cache is not None and now - _api_read_cache_settings_cache[0] < _enabled_cache_ttl_seconds and _api_read_cache_settings_cache[1] in (None, id(config)):
return _api_read_cache_settings_cache[2], _api_read_cache_settings_cache[3]
configured_default = bool(getattr(getattr(config, "api_read_cache", None), "enabled", False))
enabled = configured_default
ttl_seconds = _cache_ttl_seconds(config)
collection_overrides: dict[str, bool] = {}
browser_status_enabled = True
cache_epoch = "0"
try:
from deerflow.persistence.api_read_cache_settings.model import ApiReadCacheSettingRow
from deerflow.persistence.engine import get_session_factory
session_factory = get_session_factory()
if session_factory is not None:
async with session_factory() as session:
row = await session.get(ApiReadCacheSettingRow, "default")
if row is not None:
enabled = bool(row.enabled)
ttl_seconds = _cache_ttl_seconds(
config,
getattr(row, "ttl_seconds", ttl_seconds),
)
collection_overrides = dict(getattr(row, "collection_overrides", None) or {})
browser_status_enabled = bool(getattr(row, "browser_status_enabled", True))
cache_epoch = str(getattr(row, "cache_epoch", None) or "0")
except Exception as exc:
_warn_once("api_read_cache_settings", f"setting read failed ({type(exc).__name__})")
if _api_read_cache_settings_cache is not None and _api_read_cache_settings_cache[1] in (None, id(config)):
enabled = _api_read_cache_settings_cache[2]
ttl_seconds = _api_read_cache_settings_cache[3]
collection_overrides = _api_read_cache_settings_cache[4]
browser_status_enabled = _api_read_cache_settings_cache[5]
cache_epoch = _api_read_cache_settings_cache[6]
if _api_read_cache_settings_cache is not None and _api_read_cache_settings_cache[0] > now and _api_read_cache_settings_cache[1] in (None, id(config)):
# An administrator saved while this database read was in flight.
# Do not replace the just-saved switch/epoch with its older row.
return _api_read_cache_settings_cache[2], _api_read_cache_settings_cache[3]
_api_read_cache_settings_cache = (now, id(config), enabled, ttl_seconds, collection_overrides, browser_status_enabled, cache_epoch)
return enabled, ttl_seconds
async def is_api_read_cache_enabled(config: Any, *, force_refresh: bool = False) -> bool:
return (await get_api_read_cache_settings(config, force_refresh=force_refresh))[0]
async def is_collection_cache_enabled(config: Any, collection: str) -> bool:
enabled, _ = await get_api_read_cache_settings(config)
return enabled and bool(_api_read_cache_settings_cache[4].get(collection, True))
async def is_browser_status_enabled(config: Any) -> bool:
await get_api_read_cache_settings(config)
return _api_read_cache_settings_cache[5]
async def get_api_cache_controls(config: Any) -> dict[str, Any]:
_, ttl = await get_api_read_cache_settings(config)
overrides = _api_read_cache_settings_cache[4]
return {
"browser_status_enabled": _api_read_cache_settings_cache[5],
"collections": [{"id": key, "label": label, "paths": paths, "scope": scope, "enabled": overrides.get(key, True), "ttl_seconds": ttl} for key, label, paths, scope in API_CACHE_CATALOG],
}
def set_api_read_cache_settings_local(
enabled: bool,
ttl_seconds: int,
*,
collection_overrides: dict[str, bool] | None = None,
browser_status_enabled: bool | None = None,
cache_epoch: str | None = None,
) -> None:
"""Make just-saved settings effective in this Gateway process immediately."""
global _api_read_cache_settings_cache
previous = _api_read_cache_settings_cache
_api_read_cache_settings_cache = (
time.monotonic(),
None,
bool(enabled),
_cache_ttl_seconds(None, ttl_seconds),
dict(collection_overrides if collection_overrides is not None else (previous[4] if previous else {})),
browser_status_enabled if browser_status_enabled is not None else (previous[5] if previous else True),
cache_epoch if cache_epoch is not None else (previous[6] if previous else "0"),
)
if previous is not None and previous[6] != _api_read_cache_settings_cache[6]:
# Wake old-generation waiters immediately; their Redis locks naturally
# expire and cannot block reads in the new persisted epoch.
for event in _fill_events.values():
event.set()
_fill_events.clear()
_fill_local_payloads.clear()
def set_api_read_cache_enabled_local(enabled: bool) -> None:
"""Backward-compatible local switch update that retains the cached TTL."""
ttl_seconds = _api_read_cache_settings_cache[3] if _api_read_cache_settings_cache is not None else _DEFAULT_CACHE_TTL_SECONDS
set_api_read_cache_settings_local(enabled, ttl_seconds)
async def save_api_read_cache_settings(
enabled: bool | None,
ttl_seconds: int | None = None,
*,
config: Any = None,
collection_overrides: dict[str, bool] | None = None,
browser_status_enabled: bool | None = None,
) -> tuple[bool, int]:
"""Serialize same-process patches; row locks protect concurrent DB workers."""
loop = asyncio.get_running_loop()
lock = _save_locks.setdefault(loop, asyncio.Lock())
async with lock:
return await _persist_api_read_cache_settings(
enabled,
ttl_seconds,
config=config,
collection_overrides=collection_overrides,
browser_status_enabled=browser_status_enabled,
)
async def _persist_api_read_cache_settings(
enabled: bool | None,
ttl_seconds: int | None,
*,
config: Any,
collection_overrides: dict[str, bool] | None,
browser_status_enabled: bool | None,
) -> tuple[bool, int]:
"""Persist the administrator toggle and optional TTL in the business DB."""
from deerflow.persistence.api_read_cache_settings.model import ApiReadCacheSettingRow
from deerflow.persistence.engine import get_session_factory
session_factory = get_session_factory()
if session_factory is None:
raise RuntimeError("MySQL persistence is required to save the Redis cache setting")
if collection_overrides is not None and set(collection_overrides) - set(API_READ_CACHE_COLLECTIONS):
raise ValueError("Unknown API cache collection")
from sqlalchemy.exc import IntegrityError
for attempt in range(2):
try:
async with session_factory() as session:
row = await session.get(ApiReadCacheSettingRow, "default", with_for_update=True)
if row is None:
saved_enabled = bool(getattr(getattr(config, "api_read_cache", None), "enabled", False)) if enabled is None else enabled
saved_ttl = _cache_ttl_seconds(config, ttl_seconds)
row = ApiReadCacheSettingRow(
key="default",
enabled=saved_enabled,
ttl_seconds=saved_ttl,
collection_overrides={},
browser_status_enabled=True,
)
session.add(row)
else:
if enabled is not None:
row.enabled = bool(enabled)
saved_ttl = _cache_ttl_seconds(
config,
ttl_seconds if ttl_seconds is not None else getattr(row, "ttl_seconds", None),
)
row.ttl_seconds = saved_ttl
saved_enabled = bool(row.enabled)
saved_overrides = {**(getattr(row, "collection_overrides", None) or {}), **(collection_overrides or {})}
saved_browser_status = browser_status_enabled if browser_status_enabled is not None else bool(getattr(row, "browser_status_enabled", True))
row.collection_overrides = saved_overrides
row.browser_status_enabled = saved_browser_status
saved_epoch = secrets.token_hex(16)
row.cache_epoch = saved_epoch
await session.commit()
break
except IntegrityError:
# Another worker may have inserted the singleton settings row.
# Reload it under the row lock and merge this patch exactly once.
if attempt:
raise
set_api_read_cache_settings_local(saved_enabled, saved_ttl, collection_overrides=saved_overrides, browser_status_enabled=saved_browser_status, cache_epoch=saved_epoch)
return saved_enabled, saved_ttl
async def save_api_read_cache_enabled(enabled: bool) -> None:
"""Backward-compatible switch update; preserves an existing TTL."""
await save_api_read_cache_settings(enabled)
def api_read_cache_redis_configured(config: Any) -> bool:
return _redis_settings(config, enabled=True) is not None
def _cache_ttl_seconds(config: Any, ttl_seconds: int | None = None) -> int:
value = int(ttl_seconds if ttl_seconds is not None else getattr(getattr(config, "api_read_cache", None), "ttl_seconds", _DEFAULT_CACHE_TTL_SECONDS) or _DEFAULT_CACHE_TTL_SECONDS)
return max(_MIN_CACHE_TTL_SECONDS, min(_MAX_CACHE_TTL_SECONDS, value))
def _redis_operation_timeout_seconds(config: Any) -> float:
cache = getattr(config, "api_read_cache", None)
connect_ms = int(getattr(cache, "socket_connect_timeout_ms", 250) or 250)
socket_ms = int(getattr(cache, "socket_timeout_ms", 750) or 750)
# Keep Redis optional and bounded, while allowing a private-network cache
# to absorb short bursts before falling back to MySQL.
return min(1.0, max(0.15, max(connect_ms, socket_ms) * 1.25 / 1000))
def _max_cache_value_bytes(config: Any) -> int:
value = int(getattr(getattr(config, "api_read_cache", None), "max_value_bytes", 512 * 1024) or 512 * 1024)
return max(1024, min(5 * 1024 * 1024, value))
def _warn_once(key: str, message: str) -> None:
now = time.monotonic()
if now - _last_warning_at.get(key, 0.0) >= 60:
_last_warning_at[key] = now
logger.warning("Homepage Redis cache unavailable; using MySQL: %s", message)
def _version_key(prefix: str, collection: str) -> str:
return f"{prefix}:{collection}:version"
def _item_key(prefix: str, collection: str, version: str, logical_key: str) -> str:
digest = hashlib.sha256(logical_key.encode("utf-8")).hexdigest()
return f"{prefix}:{collection}:v{version}:{digest}"
async def get_cached_json(
config: Any,
collection: str,
logical_key: str,
*,
version: str | None | object = _VERSION_UNSET,
) -> Any | None:
raw = await get_cached_json_payload(config, collection, logical_key, version=version)
if raw is None:
return None
try:
return json.loads(raw)
except Exception as exc:
_warn_once(f"{collection}:decode", f"cached JSON decode failed ({type(exc).__name__})")
return None
async def get_cached_json_payload(
config: Any,
collection: str,
logical_key: str,
*,
version: str | None | object = _VERSION_UNSET,
) -> bytes | None:
"""Return the stored JSON bytes so HTTP routes can bypass revalidation."""
if not await is_collection_cache_enabled(config, collection):
return None
client_info = _client(config, enabled=True)
if client_info is None:
return None
client, prefix = client_info
try:
if version is _VERSION_UNSET:
version = await get_cache_version(config, collection)
if version is None:
return None
async with asyncio.timeout(_redis_operation_timeout_seconds(config)):
raw = await client.get(_item_key(prefix, collection, str(version), logical_key))
if raw is None:
return None
return raw if isinstance(raw, bytes) else str(raw).encode("utf-8")
except Exception as exc: # Redis is an optimization; MySQL remains authoritative.
_warn_once(prefix, f"read failed ({type(exc).__name__})")
return None
async def get_cache_version(config: Any, collection: str) -> str | None:
"""Read the collection generation used to guard against stale writes."""
if not await is_collection_cache_enabled(config, collection):
return None
client_info = _client(config, enabled=True)
if client_info is None:
return None
client, prefix = client_info
epoch = _cache_epoch(config)
try:
async with asyncio.timeout(_redis_operation_timeout_seconds(config)):
version = await client.get(_version_key(prefix, collection))
generation = version.decode("ascii") if isinstance(version, bytes) else str(version or "0")
return _scoped_version(epoch, generation)
except Exception as exc:
_warn_once(prefix, f"version read failed ({type(exc).__name__})")
return None
async def wait_for_cached_json_or_claim_fill(
config: Any,
collection: str,
logical_key: str,
*,
version: str | None,
wait_ms: int = 1000,
) -> tuple[Any | None, str | None]:
"""Wait briefly for a concurrent miss to fill Redis, or claim the fill.
The returned token must be passed to :func:`release_cache_fill` in a
``finally`` block. This prevents a burst of identical requests from all
rebuilding the same large agent/skill response at once.
"""
return await _wait_for_cached_value_or_claim_fill(
config,
collection,
logical_key,
version=version,
wait_ms=wait_ms,
raw_payload=False,
)
async def wait_for_cached_json_payload_or_claim_fill(
config: Any,
collection: str,
logical_key: str,
*,
version: str | None,
wait_ms: int = 6000,
) -> tuple[bytes | None, str | None]:
"""Wait for an in-flight fill and return the cached HTTP JSON bytes."""
value, token = await _wait_for_cached_value_or_claim_fill(
config,
collection,
logical_key,
version=version,
wait_ms=wait_ms,
raw_payload=True,
)
return value, token
async def _wait_for_cached_value_or_claim_fill(
config: Any,
collection: str,
logical_key: str,
*,
version: str | None,
wait_ms: int,
raw_payload: bool,
) -> tuple[Any | bytes | None, str | None]:
if version is None or not await is_collection_cache_enabled(config, collection):
return None, None
client_info = _client(config, enabled=True)
if client_info is None:
return None, None
client, prefix = client_info
digest = hashlib.sha256(logical_key.encode("utf-8")).hexdigest()
lock_key = f"{prefix}:{collection}:fill:{digest}"
event_key = (id(asyncio.get_running_loop()), lock_key)
token = f"{_cache_epoch(config)}:{secrets.token_urlsafe(18)}"
try:
async with asyncio.timeout(_redis_operation_timeout_seconds(config)):
if await client.set(lock_key, token, ex=15, nx=True):
_fill_events[event_key] = asyncio.Event()
return None, token
except Exception as exc:
_warn_once(prefix, f"fill lock unavailable ({type(exc).__name__})")
return None, None
wait_seconds = max(0, min(wait_ms, 8000)) / 1000
local_event = _fill_events.get(event_key)
if local_event is not None:
try:
async with asyncio.timeout(wait_seconds):
await local_event.wait()
except TimeoutError:
pass
if not await is_collection_cache_enabled(config, collection) or await get_cache_version(config, collection) != version:
return None, None
local_payload = _fill_local_payloads.get(event_key)
if local_payload is not None:
expires_at, payload_version, payload = local_payload
if expires_at > time.monotonic() and payload_version == version:
if raw_payload:
return payload, None
try:
return json.loads(payload), None
except Exception as exc:
_warn_once(f"{collection}:decode", f"local fill JSON decode failed ({type(exc).__name__})")
else:
_fill_local_payloads.pop(event_key, None)
if raw_payload:
cached = await get_cached_json_payload(config, collection, logical_key, version=version)
else:
cached = await get_cached_json(config, collection, logical_key, version=version)
if cached is not None:
return cached, None
else:
# Cross-process waiters cannot share the local event. Check Redis at a
# low frequency instead of polling every 50-100ms under a cold burst.
deadline = time.monotonic() + wait_seconds
while time.monotonic() < deadline:
await asyncio.sleep(min(1.0, max(0.0, deadline - time.monotonic())))
if not await is_collection_cache_enabled(config, collection) or await get_cache_version(config, collection) != version:
return None, None
if raw_payload:
cached = await get_cached_json_payload(config, collection, logical_key, version=version)
else:
cached = await get_cached_json(config, collection, logical_key, version=version)
if cached is not None:
return cached, None
# The original filler may have failed. A single waiter gets one chance to
# take over; remaining requests fall through to MySQL if the cache is down.
if not await is_collection_cache_enabled(config, collection) or await get_cache_version(config, collection) != version:
return None, None
try:
async with asyncio.timeout(_redis_operation_timeout_seconds(config)):
if await client.set(lock_key, token, ex=15, nx=True):
_fill_events[event_key] = asyncio.Event()
return None, token
except Exception as exc:
_warn_once(prefix, f"fill lock retry failed ({type(exc).__name__})")
return None, None
async def release_cache_fill(config: Any, collection: str, logical_key: str, token: str | None) -> None:
"""Release a miss lock only when it is still owned by this request."""
if not token:
return
digest = hashlib.sha256(logical_key.encode("utf-8")).hexdigest()
owner_epoch, separator, _ = token.partition(":")
client_info = _client(config, enabled=True, epoch=owner_epoch if separator else None)
if client_info is not None:
client, prefix = client_info
lock_key = f"{prefix}:{collection}:fill:{digest}"
event = _fill_events.pop((id(asyncio.get_running_loop()), lock_key), None)
else:
lock_key = ""
event = None
if event is not None:
event.set()
if client_info is None:
return
try:
async with asyncio.timeout(_redis_operation_timeout_seconds(config)):
await client.eval(
"if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) else return 0 end",
1,
lock_key,
token,
)
except Exception as exc:
_warn_once(prefix, f"fill lock release failed ({type(exc).__name__})")
def _json_default(value: Any) -> Any:
model_dump = getattr(value, "model_dump", None)
if callable(model_dump):
return model_dump(mode="json", by_alias=True)
isoformat = getattr(value, "isoformat", None)
if callable(isoformat):
return isoformat()
raise TypeError(f"Cannot cache {type(value).__name__}")
async def set_cached_json(
config: Any,
collection: str,
logical_key: str,
value: Any,
*,
expected_version: str | None | object = _VERSION_UNSET,
ttl_seconds: int | None = None,
) -> None:
enabled, configured_ttl = await get_api_read_cache_settings(config)
if not enabled or not await is_collection_cache_enabled(config, collection):
return
client_info = _client(config, enabled=True)
if client_info is None:
return
client, prefix = client_info
try:
if expected_version is _VERSION_UNSET:
expected_version = await get_cache_version(config, collection)
if expected_version is None:
return
encoded = json.dumps(value, ensure_ascii=False, separators=(",", ":"), default=_json_default).encode("utf-8")
if len(encoded) > _max_cache_value_bytes(config):
return
async with asyncio.timeout(_redis_operation_timeout_seconds(config)):
version = await client.get(_version_key(prefix, collection))
version_text = version.decode("ascii") if isinstance(version, bytes) else str(version or "0")
if _scoped_version(_cache_epoch(config), version_text) != expected_version:
return
await client.set(
_item_key(prefix, collection, expected_version, logical_key),
encoded,
ex=_cache_ttl_seconds(
config,
ttl_seconds if ttl_seconds is not None else configured_ttl,
),
)
loop_id = id(asyncio.get_running_loop())
fill_digest = hashlib.sha256(logical_key.encode("utf-8")).hexdigest()
fill_event_key = (loop_id, f"{prefix}:{collection}:fill:{fill_digest}")
if fill_event_key in _fill_events:
now = time.monotonic()
for key, (expires_at, _, _) in list(_fill_local_payloads.items()):
if expires_at <= now:
_fill_local_payloads.pop(key, None)
_fill_local_payloads[fill_event_key] = (now + 2.0, str(expected_version), encoded)
if len(_fill_local_payloads) > 128:
_fill_local_payloads.pop(next(iter(_fill_local_payloads)))
except Exception as exc:
_warn_once(prefix, f"write failed ({type(exc).__name__})")
async def invalidate_cached_collection(config: Any, collection: str, *, force: bool = False) -> None:
"""Bump a generation; old keys expire naturally, with no Redis key scan."""
if not force and not await is_api_read_cache_enabled(config):
return
# A forced invalidation is used after an admin edits sensitive words while
# Redis reads are disabled. The generation still advances so re-enabling
# the cache cannot resurrect that older value within its one-hour TTL.
client_info = _client(config, enabled=True)
if client_info is None:
return
client, prefix = client_info
try:
async with asyncio.timeout(_redis_operation_timeout_seconds(config)):
await client.incr(_version_key(prefix, collection))
except Exception as exc:
_warn_once(prefix, f"invalidation failed ({type(exc).__name__})")
async def close_api_read_cache() -> None:
clients = list(_clients.values())
_clients.clear()
for client in clients:
try:
await client.aclose()
except Exception:
logger.debug("Failed to close homepage Redis cache client", exc_info=True)
offline-backend-20260512/backend/app/gateway/app.py
+374 -14
View File
@@ -1,3 +1,5 @@
# ruff: noqa: E402, I001
# The dotenv load below must precede application imports that read environment.
import asyncio
import logging
import os
@@ -7,11 +9,16 @@ import threading
from collections.abc import AsyncGenerator
from contextlib import asynccontextmanager
from datetime import datetime
from pathlib import Path
from time import perf_counter
from typing import Any
from dotenv import load_dotenv
from fastapi import FastAPI
from fastapi import FastAPI, Request
from fastapi.middleware.cors import CORSMiddleware
from fastapi.openapi.docs import get_redoc_html, get_swagger_ui_html
from fastapi.responses import HTMLResponse
from fastapi.staticfiles import StaticFiles
# Load .env BEFORE any module that reads os.environ at import time.
# Without this, GATEWAY_CORS_ORIGINS / AUTH_JWT_SECRET / DEER_FLOW_AUTH_DISABLED
@@ -28,16 +35,20 @@ from app.gateway.deps import langgraph_runtime
from app.gateway.public_knowledge_cors import PublicKnowledgeCORSMiddleware
from app.gateway.routers import (
admin_active_runs,
admin_audit_retention as admin_audit_retention_routes,
admin_users,
agents,
ai_writing,
app_identity,
artifacts,
assistant_knowledge,
assistants_compat,
auth,
service_tokens,
browser_context,
business_mapping,
channels,
chat_archive,
checkpoint_migration,
compact_menu_overrides,
dashboard_sessions,
@@ -52,6 +63,7 @@ from app.gateway.routers import (
html_images,
html_page_favorites,
html_runtime,
html_templates,
intent,
knowledge,
knowledge_ingest,
@@ -62,8 +74,8 @@ from app.gateway.routers import (
mcp,
memory,
menu_overrides,
model_management,
model_exchanges,
model_management,
models,
multi_agent,
notifications,
@@ -89,11 +101,10 @@ from app.gateway.routers import (
roundtable_draft_shares,
roundtable_drafts,
roundtable_jobs,
roundtable_rooms,
roundtable_task_drafts,
runs,
scheduled_tasks,
task_analysis,
task_analysis_qa,
sensitive_words,
sentiment_agent,
skill_addresses,
@@ -101,10 +112,14 @@ from app.gateway.routers import (
skill_view_diagnostics,
skills,
strategy_agents,
stream_truncations,
suggestions,
system_settings,
tags,
task_analysis,
task_analysis_qa,
task_buttons,
task_deeplink_details,
task_deeplink_threads,
task_reports,
taskcop_tasks,
@@ -112,8 +127,8 @@ from app.gateway.routers import (
thread_shares,
threads,
tool_metrics,
uploads,
upload_policy,
uploads,
user_preferences,
user_prompts,
web_app_sites,
@@ -144,6 +159,11 @@ logging.basicConfig(
logger = logging.getLogger(__name__)
# Swagger UI and ReDoc are vendored so the API documentation works in isolated
# intranets. The route intentionally lives below /docs so it is covered by the
# existing anonymous docs allow-list in the auth middleware.
_DOCS_STATIC_DIR = Path(__file__).with_name("static") / "docs"
# Upper bound (seconds) each lifespan shutdown hook is allowed to run.
# Bounds worker exit time so uvicorn's reload supervisor does not keep
# firing signals into a worker that is stuck waiting for shutdown cleanup.
@@ -242,7 +262,7 @@ async def _iter_store_items(store, namespace, *, page_size: int = 500):
offset += page_size
async def _sync_legacy_skills(app: FastAPI) -> None:
async def _sync_legacy_skills(app: FastAPI) -> list[Any] | None:
"""Idempotently register existing on-disk ``skills/custom/*`` directories
in the ``skills`` DB table, owned by the first admin and published=True.
@@ -262,11 +282,11 @@ async def _sync_legacy_skills(app: FastAPI) -> None:
sf = get_session_factory()
if sf is None:
return
return None
skill_store = getattr(app.state, "skill_store", None)
if skill_store is None:
return
return None
async with sf() as session:
stmt = select(UserRow).where(UserRow.system_role == "admin").limit(1)
@@ -284,7 +304,7 @@ async def _sync_legacy_skills(app: FastAPI) -> None:
skills = storage.load_skills(enabled_only=False)
except Exception:
logger.exception("_sync_legacy_skills: could not list on-disk skills")
return
return None
inserted = 0
for skill in skills:
@@ -302,6 +322,7 @@ async def _sync_legacy_skills(app: FastAPI) -> None:
logger.warning("_sync_legacy_skills: could not register %s", skill.name, exc_info=True)
if inserted:
logger.info("Skill ownership migration: registered %d legacy custom skill(s)", inserted)
return skills
async def _migrate_orphaned_threads(store, admin_user_id: str) -> int:
@@ -548,6 +569,35 @@ _CONCURRENCY_SAMPLE_RETENTION_HOURS = _env_float(
"CONCURRENCY_SAMPLE_RETENTION_HOURS",
168.0, # 7 days
)
# Administrative audit data is operational rather than archival. The actual
# cutoff lives with the shared query policy; these controls intentionally make
# cleanup a low-priority, incremental task instead of a large startup delete.
_ADMIN_AUDIT_RETENTION_INTERVAL_SECONDS = _env_float(
"ADMIN_AUDIT_RETENTION_INTERVAL_SECONDS",
15 * 60,
minimum=60.0,
)
_ADMIN_AUDIT_RETENTION_STARTUP_DELAY_SECONDS = _env_float(
"ADMIN_AUDIT_RETENTION_STARTUP_DELAY_SECONDS",
300.0,
minimum=0.0,
)
_ADMIN_AUDIT_RETENTION_BATCH_SIZE = min(
500,
_env_int("ADMIN_AUDIT_RETENTION_BATCH_SIZE", 100, minimum=1),
)
_ADMIN_AUDIT_RETENTION_MAX_BATCHES_PER_STORE = min(
16,
_env_int("ADMIN_AUDIT_RETENTION_MAX_BATCHES_PER_STORE", 4, minimum=1),
)
_ADMIN_AUDIT_RETENTION_INTER_BATCH_PAUSE_SECONDS = min(
1.0,
_env_float("ADMIN_AUDIT_RETENTION_INTER_BATCH_PAUSE_SECONDS", 0.05, minimum=0.0),
)
_ADMIN_AUDIT_RETENTION_MAX_SWEEP_SECONDS = min(
60.0,
_env_float("ADMIN_AUDIT_RETENTION_MAX_SWEEP_SECONDS", 8.0, minimum=0.1),
)
# Hour (local Beijing time, 0-23) after which the scheduler force-recomputes the
# *previous* day's snapshot once, so records that landed near midnight are fully
# captured. Default 1 ? "???? 1 ????????".
@@ -638,10 +688,11 @@ async def _time_leaderboard_scheduler_db_operation(step: str, operation, *, sql:
def _recent_leaderboard_stat_dates(days: int) -> list[str]:
from datetime import datetime, timedelta
from app.gateway.admin_query_window import ADMIN_QUERY_WINDOW_DAYS
from deerflow.persistence.types import BEIJING_TZ
today = datetime.now(BEIJING_TZ).date()
start = today - timedelta(days=max(1, days) - 1)
start = today - timedelta(days=min(ADMIN_QUERY_WINDOW_DAYS, max(1, days)) - 1)
return [(start + timedelta(days=i)).isoformat() for i in range((today - start).days + 1)]
@@ -946,6 +997,141 @@ async def _concurrency_sampler_loop(app: FastAPI) -> None:
_release_singleton_lock(lock_handle)
async def _admin_audit_retention_loop(app: FastAPI) -> None:
"""Keep bounded admin-audit tables at the three-month retention target."""
from app.gateway.admin_audit_retention import sweep_admin_audit_retention
from app.gateway.admin_audit_retention_control import (
is_paused,
read_status,
write_status,
)
from app.gateway.admin_query_window import ADMIN_AUDIT_RETENTION_DAYS
from deerflow.config.runtime_paths import runtime_home
from deerflow.persistence.engine import get_session_factory
lock_handle = _acquire_singleton_lock(runtime_home() / ".admin_audit_retention.lock")
if lock_handle is None:
logger.info("Admin audit retention: another gateway worker holds the lock; this worker skips")
return
root = runtime_home()
status = read_status(root)
status.update(
{
"retention_days": ADMIN_AUDIT_RETENTION_DAYS,
"interval_seconds": _ADMIN_AUDIT_RETENTION_INTERVAL_SECONDS,
"startup_delay_seconds": _ADMIN_AUDIT_RETENTION_STARTUP_DELAY_SECONDS,
"batch_size": _ADMIN_AUDIT_RETENTION_BATCH_SIZE,
"max_rounds": _ADMIN_AUDIT_RETENTION_MAX_BATCHES_PER_STORE,
"max_sweep_seconds": _ADMIN_AUDIT_RETENTION_MAX_SWEEP_SECONDS,
}
)
def save_status(**updates: Any) -> None:
status.update(updates)
write_status(root, dict(status))
async def publish_progress(progress: dict[str, Any]) -> None:
await asyncio.to_thread(save_status, **progress)
logger.info(
"Admin audit retention started: interval=%ss startup_delay=%ss batch_size=%s "
"max_batches_per_store=%s max_sweep_seconds=%s",
_ADMIN_AUDIT_RETENTION_INTERVAL_SECONDS,
_ADMIN_AUDIT_RETENTION_STARTUP_DELAY_SECONDS,
_ADMIN_AUDIT_RETENTION_BATCH_SIZE,
_ADMIN_AUDIT_RETENTION_MAX_BATCHES_PER_STORE,
_ADMIN_AUDIT_RETENTION_MAX_SWEEP_SECONDS,
)
try:
if _ADMIN_AUDIT_RETENTION_STARTUP_DELAY_SECONDS:
# Do not compete with migrations, warmup, or users arriving just
# after a Gateway restart. The scheduled passes are deliberately
# maintenance work, not a startup requirement.
save_status(
status="scheduled",
current_source=None,
next_run_at=(
datetime.now().astimezone().timestamp()
+ _ADMIN_AUDIT_RETENTION_STARTUP_DELAY_SECONDS
),
)
await asyncio.sleep(_ADMIN_AUDIT_RETENTION_STARTUP_DELAY_SECONDS)
while True:
if is_paused(root):
save_status(status="paused", current_source=None, next_run_at=None)
await asyncio.sleep(min(2.0, _ADMIN_AUDIT_RETENTION_INTERVAL_SECONDS))
continue
started_at = datetime.now().astimezone().isoformat(timespec="seconds")
save_status(
status="running",
current_source=None,
round=0,
deleted={
"llm_metrics": 0,
"tool_metrics": 0,
"run_summaries": 0,
"leaderboard_snapshots": 0,
},
started_at=started_at,
next_run_at=None,
last_error=None,
)
try:
result = await sweep_admin_audit_retention(
session_factory=get_session_factory(),
llm_metrics_store=getattr(app.state, "llm_metrics_store", None),
tool_metrics_store=getattr(app.state, "tool_metrics_store", None),
run_store=getattr(app.state, "run_store", None),
batch_size=_ADMIN_AUDIT_RETENTION_BATCH_SIZE,
max_batches_per_store=_ADMIN_AUDIT_RETENTION_MAX_BATCHES_PER_STORE,
inter_batch_pause_seconds=_ADMIN_AUDIT_RETENTION_INTER_BATCH_PAUSE_SECONDS,
max_sweep_seconds=_ADMIN_AUDIT_RETENTION_MAX_SWEEP_SECONDS,
should_stop=lambda: is_paused(root),
on_progress=publish_progress,
)
finished_at = datetime.now().astimezone().isoformat(timespec="seconds")
paused = is_paused(root)
last_sweep = {
"started_at": started_at,
"finished_at": finished_at,
"deleted": result,
"stopped_by_admin": paused,
}
save_status(
status="paused" if paused else "scheduled",
current_source=None,
round=status.get("round", 0),
deleted=result,
started_at=started_at,
finished_at=finished_at,
last_sweep=last_sweep,
next_run_at=None if paused else (
datetime.now().astimezone().timestamp()
+ _ADMIN_AUDIT_RETENTION_INTERVAL_SECONDS
),
)
except Exception:
save_status(
status="error",
current_source=None,
last_error="清理轮次发生异常，请查看后台日志",
finished_at=datetime.now().astimezone().isoformat(timespec="seconds"),
)
logger.exception("Admin audit retention sweep failed (non-fatal)")
# Wake periodically so an administrator can pause/resume without
# waiting for the full 15-minute cleanup interval.
remaining = _ADMIN_AUDIT_RETENTION_INTERVAL_SECONDS
while remaining > 0 and not is_paused(root):
sleep_for = min(2.0, remaining)
await asyncio.sleep(sleep_for)
remaining -= sleep_for
finally:
_release_singleton_lock(lock_handle)
def _warn_if_multi_worker() -> None:
"""?? uvicorn / gunicorn ? worker ???? ERROR ???
@@ -1000,7 +1186,7 @@ def _warn_if_multi_worker() -> None:
@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
async def _gateway_lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
"""Application lifespan handler."""
# ?????? worker ?? ?? ?? AI ?? / ? agent ???????
@@ -1017,6 +1203,7 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
error_msg = f"Failed to load configuration during gateway startup: {e}"
logger.exception(error_msg)
raise RuntimeError(error_msg) from e
config = get_gateway_config()
logger.info(f"Starting API Gateway on {config.host}:{config.port}")
@@ -1069,11 +1256,26 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
# Migrate "ownerless" on-disk skills into the new SkillStore table ?
# makes the multitenancy switch transparent for existing deployments.
catalog_skills = None
try:
await _sync_legacy_skills(app)
catalog_skills = await _sync_legacy_skills(app)
except Exception:
logger.exception("Skill ownership migration failed (non-fatal)")
# Project the compact per-skill catalog metadata into MySQL before the
# Gateway accepts requests. List APIs read indexed scalar columns and
# never load skill markdown, display samples, or usage sidecars.
try:
from app.gateway.skill_catalog_cache import refresh_skill_catalog_projection
catalog_size = await refresh_skill_catalog_projection(
app.state.config,
skills=catalog_skills,
)
logger.info("Skill MySQL catalog projection synchronized: skills=%d", catalog_size)
except Exception:
logger.exception("Skill MySQL catalog projection sync failed (non-fatal)")
# ????? agent(intent / recommender / coordinator)???????
# ??? _sync_legacy_agents **??**??????????????????
# ??? upsert ? agents ?,?????? /api/agents ?????,
@@ -1133,6 +1335,17 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
"Template-builder agent seeding failed at startup (non-fatal)"
)
# HTML 模板库:把随包内置模板 seed 进 {base_dir}/html-templates/,模板管理页
# 与定时任务下拉即可见(缺失才复制,绝不覆盖管理页/线上修改)。
try:
from deerflow.runtime.scheduler import template_store
template_store.ensure_seeded()
except Exception:
logger.exception(
"HTML template store seeding failed at startup (non-fatal)"
)
# 实验性「页面研报设计师」：在 _sync_legacy_agents 前补齐 agent 目录与
# 公共页面生成技能，首次启动即可在智能体管理页和普通对话中复用。
try:
@@ -1328,6 +1541,15 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
except Exception:
logger.exception("Legacy agent sync failed at startup (non-fatal)")
# Refresh after database initialization and all built-in skill seeders;
# import-time priming may have run before either was ready.
try:
from deerflow.agents.lead_agent.prompt import refresh_skills_system_prompt_cache_async
await refresh_skills_system_prompt_cache_async()
except Exception:
logger.exception("Skills prompt cache warm-up failed at startup (non-fatal)")
# ?? skill curator ???:??????,?????????????
# (per-user interval_hours ??????????????)?
try:
@@ -1358,6 +1580,13 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
except Exception:
logger.exception("Concurrency sampler failed to start (non-fatal)")
try:
app.state.admin_audit_retention_task = asyncio.create_task(
_admin_audit_retention_loop(app)
)
except Exception:
logger.exception("Admin audit retention scheduler failed to start (non-fatal)")
# Start IM channel service if any channels are configured
try:
from app.channels.service import start_channel_service
@@ -1399,6 +1628,8 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
except Exception:
logger.exception("Task analysis service failed to start")
if getattr(app.state, "worker_pool", None) is not None:
app.state.worker_pool.start()
yield
artifact_library_tasks = list(
@@ -1466,6 +1697,16 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
except Exception:
logger.exception("Concurrency sampler shutdown error")
admin_audit_retention_task = getattr(app.state, "admin_audit_retention_task", None)
if admin_audit_retention_task is not None:
admin_audit_retention_task.cancel()
try:
await admin_audit_retention_task
except asyncio.CancelledError:
pass
except Exception:
logger.exception("Admin audit retention scheduler shutdown error")
# ?? 7 ????AI ?? graph ???? checkpointer / in-flight task?
# thread ??? LangGraph Server ? checkpointer ???lifespan ???
# ``langgraph_runtime()`` ? contextmanager ???????????????
@@ -1505,6 +1746,25 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
logger.info("Shutting down API Gateway")
@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
from app.gateway.worker_pool.lifecycle import managed_controller
from app.gateway.perf_logging import event_loop_watchdog, register_stack_dump_signal
register_stack_dump_signal()
watchdog = asyncio.create_task(event_loop_watchdog(app), name="gateway-event-loop-watchdog")
try:
async with managed_controller(app):
async with _gateway_lifespan(app):
yield
finally:
watchdog.cancel()
await asyncio.gather(watchdog, return_exceptions=True)
from app.gateway.api_read_cache import close_api_read_cache
await close_api_read_cache()
def create_app() -> FastAPI:
"""Create and configure the FastAPI application.
@@ -1512,7 +1772,12 @@ def create_app() -> FastAPI:
Configured FastAPI application instance.
"""
config = get_gateway_config()
docs_kwargs = {"docs_url": "/docs", "redoc_url": "/redoc", "openapi_url": "/openapi.json"} if config.enable_docs else {"docs_url": None, "redoc_url": None, "openapi_url": None}
logger.info("API documentation endpoints enabled: %s", config.enable_docs)
docs_kwargs = {
"docs_url": None,
"redoc_url": None,
"openapi_url": "/openapi.json",
} if config.enable_docs else {"docs_url": None, "redoc_url": None, "openapi_url": None}
app = FastAPI(
title="DeerFlow API Gateway",
@@ -1598,7 +1863,83 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
],
)
@app.middleware("http")
async def invalidate_catalog_cache_after_write(request: Request, call_next):
response = await call_next(request)
if request.method.upper() in {"POST", "PUT", "PATCH", "DELETE"} and response.status_code < 400:
path = request.url.path.rstrip("/")
collections: set[str] = set()
if path == "/api/agents" or path.startswith("/api/agents/"):
collections.update(("agents", "tags"))
if path == "/api/skills" or path.startswith("/api/skills/"):
collections.update(("skills", "tags"))
if path == "/api/tags" or path.startswith("/api/tags/"):
collections.update(("agents", "skills", "recommended_questions", "light_apps", "tags"))
if path == "/api/positions" or path.startswith("/api/positions/"):
collections.update(("agents", "skills", "recommended_questions"))
if path == "/api/user-preferences" or path.startswith("/api/user-preferences/"):
collections.add("user_preferences")
if path == "/api/user-prompts" or path.startswith("/api/user-prompts/"):
collections.add("user_prompts")
if path.startswith("/api/recommended-questions"):
collections.update(("recommended_questions", "tags"))
if path == "/api/scheduled-tasks" or path.startswith("/api/scheduled-tasks/"):
collections.add("tags")
if path == "/api/light-apps" or path.startswith("/api/light-apps/"):
collections.add("tags")
if path == "/api/sensitive-words":
collections.add("sensitive_words")
config = getattr(request.app.state, "config", None)
if config is not None and collections:
from app.gateway.api_read_cache import invalidate_cached_collection
await asyncio.gather(
*(
invalidate_cached_collection(
config,
collection,
force=True,
)
for collection in collections
)
)
return response
if config.enable_docs:
# Keep these paths relative to the page URL. Nginx strips the
# /deerflow deployment prefix before forwarding to this app, and
# root-relative URLs would otherwise escape that prefix in browsers.
app.mount(
"/docs/static",
StaticFiles(directory=_DOCS_STATIC_DIR),
name="docs-static",
)
@app.get("/docs", include_in_schema=False)
async def swagger_ui() -> HTMLResponse:
return get_swagger_ui_html(
openapi_url="./openapi.json",
title=f"{app.title} - Swagger UI",
swagger_js_url="./docs/static/swagger-ui-bundle.js",
swagger_css_url="./docs/static/swagger-ui.css",
swagger_favicon_url="./docs/static/favicon-32x32.png",
swagger_ui_parameters=app.swagger_ui_parameters,
)
@app.get("/redoc", include_in_schema=False)
async def redoc() -> HTMLResponse:
return get_redoc_html(
openapi_url="./openapi.json",
title=f"{app.title} - ReDoc",
redoc_js_url="./docs/static/redoc.standalone.js",
redoc_favicon_url="./docs/static/favicon-32x32.png",
with_google_fonts=False,
)
# Auth: reject unauthenticated requests to non-public paths (fail-closed safety net)
from app.gateway.worker_pool.proxy import WorkerProxy
app.add_middleware(WorkerProxy)
app.add_middleware(AuthMiddleware)
# CSRF: Double Submit Cookie pattern for state-changing requests
@@ -1756,8 +2097,9 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
app.include_router(analysis_projects.router)
app.include_router(threads.router)
# Dedicated taskId + goPath history lookup for 3qfx/rwfx/xdfx single-agent chats.
# Dedicated taskId + goPath history lookup for single-agent task deep-link chats.
app.include_router(task_deeplink_threads.router)
app.include_router(task_deeplink_details.router)
# Agents API is mounted at /api/agents
app.include_router(agents.router)
app.include_router(strategy_agents.router)
@@ -1866,6 +2208,7 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
# Roundtable-planning background jobs at /api/roundtable-jobs
app.include_router(roundtable_jobs.router)
app.include_router(roundtable_rooms.router)
app.include_router(roundtable_diagnostics.router)
# Deep Research (深度研究) API at /api/deep-research
@@ -1904,6 +2247,8 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
app.include_router(html_runtime.router)
app.include_router(html_images.router)
app.include_router(html_page_favorites.router)
# HTML 模板管理(模板管理页;读开放,写仅 admin)。
app.include_router(html_templates.router)
# Product knowledge base API is mounted at /api/knowledge
app.include_router(knowledge.router)
@@ -1916,6 +2261,11 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
# System settings API is mounted at /api/system-settings
app.include_router(system_settings.router)
app.include_router(app_identity.router)
app.include_router(app_identity.public_router)
from app.gateway.worker_pool.router import router as worker_pool_router
app.include_router(worker_pool_router)
# Admin LLM metrics audit + export
app.include_router(llm_metrics.router)
@@ -1923,6 +2273,7 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
# Admin window into other users' threads / messages / memory
app.include_router(admin_users.router)
app.include_router(admin_users.self_router)
app.include_router(admin_audit_retention_routes.router)
app.include_router(admin_active_runs.router)
app.include_router(skill_view_diagnostics.router)
@@ -1931,17 +2282,23 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
# Admin tool / skill call metrics audit + export
app.include_router(tool_metrics.router)
# Admin model-stream truncation event log
app.include_router(stream_truncations.router)
# Channels API is mounted at /api/channels
app.include_router(channels.router)
# Admin checkpoint migration API: SQLite legacy checkpoints -> Postgres.
app.include_router(checkpoint_migration.router)
# Admin monthly archive/rollback for PostgreSQL chat checkpoints.
app.include_router(chat_archive.router)
# Assistants compatibility API (LangGraph Platform stub)
app.include_router(assistants_compat.router)
# Auth API is mounted at /api/v1/auth
app.include_router(auth.router)
app.include_router(service_tokens.router)
# Feedback API is mounted at /api/threads/{thread_id}/runs/{run_id}/feedback
app.include_router(feedback.router)
@@ -2012,6 +2369,9 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
app.include_router(llmwiki.proxy_router, prefix="/deerflow")
app.include_router(llmwiki.proxy_router)
from app.gateway.worker_pool.telemetry import RequestTelemetryMiddleware
app.add_middleware(RequestTelemetryMiddleware)
return app
offline-backend-20260512/backend/app/gateway/auth/jwt.py
+70 -56
View File
@@ -1,63 +1,77 @@
"""JWT token creation and verification."""
from datetime import UTC, datetime, timedelta
from typing import Literal
import jwt
from pydantic import BaseModel, ValidationError
from app.gateway.auth.config import get_auth_config
from app.gateway.auth.errors import TokenError
"""JWT token creation and verification."""
AuthMethod = Literal["password", "passphrase", "sso"]
class TokenPayload(BaseModel):
"""JWT token payload."""
sub: str # user_id
exp: datetime
iat: datetime | None = None
ver: int = 0 # token_version — must match User.token_version
from datetime import UTC, datetime, timedelta
from typing import Literal
import jwt
from pydantic import BaseModel, Field, ValidationError
from app.gateway.auth.config import get_auth_config
from app.gateway.auth.errors import TokenError
AuthMethod = Literal["password", "passphrase", "sso", "service"]
class TokenPayload(BaseModel):
"""JWT token payload."""
sub: str # user_id
exp: datetime
iat: datetime | None = None
ver: int = 0 # token_version — must match User.token_version
auth_method: AuthMethod | None = None # None is historical/unverified identity-only login.
def create_access_token(user_id: str, expires_delta: timedelta | None = None, token_version: int = 0, *, auth_method: AuthMethod | None = None) -> str:
"""Create a JWT access token.
Args:
user_id: The user's UUID as string
expires_delta: Optional custom expiry, defaults to 7 days
token_version: User's current token_version for invalidation
client_id: str | None = None
scope: list[str] = Field(default_factory=list)
def create_access_token(
user_id: str,
expires_delta: timedelta | None = None,
token_version: int = 0,
*,
auth_method: AuthMethod | None = None,
client_id: str | None = None,
scope: list[str] | None = None,
) -> str:
"""Create a JWT access token.
Args:
user_id: The user's UUID as string
expires_delta: Optional custom expiry, defaults to 7 days
token_version: User's current token_version for invalidation
auth_method: Explicit proof verified by the issuing login path. Never
defaults to a trusted method for old or identity-only callers.
Returns:
Encoded JWT string
"""
config = get_auth_config()
expiry = expires_delta or timedelta(days=config.token_expiry_days)
now = datetime.now(UTC)
payload = {"sub": user_id, "exp": now + expiry, "iat": now, "ver": token_version}
Returns:
Encoded JWT string
"""
config = get_auth_config()
expiry = expires_delta or timedelta(days=config.token_expiry_days)
now = datetime.now(UTC)
payload = {"sub": user_id, "exp": now + expiry, "iat": now, "ver": token_version}
if auth_method is not None:
payload["auth_method"] = auth_method
return jwt.encode(payload, config.jwt_secret, algorithm="HS256")
def decode_token(token: str) -> TokenPayload | TokenError:
"""Decode and validate a JWT token.
Returns:
TokenPayload if valid, or a specific TokenError variant.
"""
config = get_auth_config()
try:
payload = jwt.decode(token, config.jwt_secret, algorithms=["HS256"])
return TokenPayload(**payload)
except jwt.ExpiredSignatureError:
return TokenError.EXPIRED
except jwt.InvalidSignatureError:
return TokenError.INVALID_SIGNATURE
if client_id is not None:
payload["client_id"] = client_id
if scope:
payload["scope"] = scope
return jwt.encode(payload, config.jwt_secret, algorithm="HS256")
def decode_token(token: str) -> TokenPayload | TokenError:
"""Decode and validate a JWT token.
Returns:
TokenPayload if valid, or a specific TokenError variant.
"""
config = get_auth_config()
try:
payload = jwt.decode(token, config.jwt_secret, algorithms=["HS256"])
return TokenPayload(**payload)
except jwt.ExpiredSignatureError:
return TokenError.EXPIRED
except jwt.InvalidSignatureError:
return TokenError.INVALID_SIGNATURE
except (jwt.PyJWTError, ValidationError):
return TokenError.MALFORMED
return TokenError.MALFORMED
offline-backend-20260512/backend/app/gateway/auth/local_provider.py
+31 -15
View File
@@ -3,7 +3,7 @@
import logging
from app.gateway.auth.models import User
from app.gateway.auth.password import hash_password_async, verify_account_password
from app.gateway.auth.password import hash_password_async, verify_account_password
from app.gateway.auth.providers import AuthProvider
from app.gateway.auth.repositories.base import UserRepository
@@ -40,7 +40,7 @@ class LocalAuthProvider(AuthProvider):
if user is None:
return None
if not await verify_account_password(password, user):
if not await verify_account_password(password, user):
return None
return user
@@ -49,6 +49,10 @@ class LocalAuthProvider(AuthProvider):
"""Get user by ID."""
return await self._repo.get_user_by_id(user_id)
async def get_users_by_ids(self, user_ids: list[str]) -> list[User]:
"""Resolve many account owners with one repository read."""
return await self._repo.get_users_by_ids(user_ids)
async def find_user_by_email_local_part(self, local_part: str) -> User | None:
"""Find a user whose email starts with ``local_part@`` regardless of domain.
@@ -86,7 +90,7 @@ class LocalAuthProvider(AuthProvider):
user = User(
email=email,
password_hash=password_hash,
login_password=password_hash,
login_password=password_hash,
system_role=system_role,
needs_setup=needs_setup,
approved=approved,
@@ -96,11 +100,11 @@ class LocalAuthProvider(AuthProvider):
async def get_user_by_oauth(self, provider: str, oauth_id: str) -> User | None:
"""Get user by OAuth provider and ID."""
return await self._repo.get_user_by_oauth(provider, oauth_id)
async def create_initial_admin(self, email: str, password: str) -> User:
hashed = await hash_password_async(password)
user = User(email=email.strip().lower(), password_hash=hashed, login_password=hashed, system_role="admin", approved=True)
return await self._repo.create_initial_admin(user)
async def create_initial_admin(self, email: str, password: str) -> User:
hashed = await hash_password_async(password)
user = User(email=email.strip().lower(), password_hash=hashed, login_password=hashed, system_role="admin", approved=True)
return await self._repo.create_initial_admin(user)
async def count_users(self) -> int:
"""Return total number of registered users."""
@@ -114,17 +118,29 @@ class LocalAuthProvider(AuthProvider):
"""Update an existing user."""
return await self._repo.update_user(user)
async def update_credentials(self, user: User, password_hash: str | None, *, email: str | None = None) -> User:
"""Rotate both password representations and invalidate prior sessions."""
return await self._repo.update_credentials(user, password_hash, email=email)
async def start_login_session(self, user: User) -> User:
return await self._repo.start_login_session(user)
async def update_credentials(self, user: User, password_hash: str | None, *, email: str | None = None) -> User:
"""Rotate both password representations and invalidate prior sessions."""
return await self._repo.update_credentials(user, password_hash, email=email)
async def start_login_session(self, user: User) -> User:
return await self._repo.start_login_session(user)
async def list_users(self) -> list[User]:
"""Return all registered users ordered by creation date."""
return await self._repo.list_users()
async def list_users_page(
self, *, q: str | None = None, limit: int = 20, offset: int = 0
) -> list[User]:
"""Return one page of users (optionally filtered by ``q``)."""
return await self._repo.list_users_page(q=q, limit=limit, offset=offset)
async def count_users_filtered(
self, *, q: str | None = None, approved_or_admin: bool = False
) -> int:
"""Count users with the same ``q`` filter; optionally approved/admin only."""
return await self._repo.count_users_filtered(q=q, approved_or_admin=approved_or_admin)
async def get_user_by_email(self, email: str) -> User | None:
"""Get user by email."""
return await self._repo.get_user_by_email(email)
offline-backend-20260512/backend/app/gateway/auth/repositories/base.py
+52 -24
View File
@@ -13,10 +13,10 @@ class UserNotFoundError(LookupError):
while specific call sites can pin to this class to distinguish
"concurrent delete during update" from other lookups.
"""
class CredentialConflictError(ValueError):
"""Credentials changed after the caller authenticated the old password."""
class CredentialConflictError(ValueError):
"""Credentials changed after the caller authenticated the old password."""
class UserRepository(ABC):
@@ -65,6 +65,15 @@ class UserRepository(ABC):
"""
raise NotImplementedError
async def get_users_by_ids(self, user_ids: list[str]) -> list[User]:
"""Batch lookup users; fallback keeps lightweight repositories compatible."""
users = []
for user_id in dict.fromkeys(user_ids):
user = await self.get_user_by_id(user_id)
if user is not None:
users.append(user)
return users
@abstractmethod
async def find_user_by_email_local_part(self, local_part: str) -> User | None:
"""Find a user by the local part of their email (the bit before ``@``).
@@ -103,26 +112,26 @@ class UserRepository(ABC):
"""
return []
async def update_credentials(self, user: User, password_hash: str | None, *, email: str | None = None) -> User:
"""Fallback for non-SQL implementations; SQL overrides with atomic CAS."""
user.password_hash = password_hash
user.login_password = password_hash
user.token_version += 1
user.needs_setup = False
if email is not None:
user.email = email
return await self.update_user(user)
async def create_initial_admin(self, user: User) -> User:
if await self.count_admin_users():
raise ValueError("System already initialized")
return await self.create_user(user)
async def start_login_session(self, user: User) -> User:
"""Invalidate earlier sessions after a successful credential check."""
user.token_version += 1
return await self.update_user(user)
async def update_credentials(self, user: User, password_hash: str | None, *, email: str | None = None) -> User:
"""Fallback for non-SQL implementations; SQL overrides with atomic CAS."""
user.password_hash = password_hash
user.login_password = password_hash
user.token_version += 1
user.needs_setup = False
if email is not None:
user.email = email
return await self.update_user(user)
async def create_initial_admin(self, user: User) -> User:
if await self.count_admin_users():
raise ValueError("System already initialized")
return await self.create_user(user)
async def start_login_session(self, user: User) -> User:
"""Invalidate earlier sessions after a successful credential check."""
user.token_version += 1
return await self.update_user(user)
@abstractmethod
async def update_user(self, user: User) -> User:
"""Update an existing user.
@@ -155,6 +164,25 @@ class UserRepository(ABC):
"""Return all users ordered by creation date."""
raise NotImplementedError
@abstractmethod
async def list_users_page(
self, *, q: str | None = None, limit: int = 20, offset: int = 0
) -> list[User]:
"""Return one page of users ordered by creation date.
``q`` optionally matches (case-insensitive substring) email or user id.
"""
raise NotImplementedError
@abstractmethod
async def count_users_filtered(
self, *, q: str | None = None, approved_or_admin: bool = False
) -> int:
"""Count users, optionally filtered by the same ``q`` as
:meth:`list_users_page`; ``approved_or_admin`` restricts to users the
login whitelist would let in (approved or admin role)."""
raise NotImplementedError
@abstractmethod
async def get_user_by_oauth(self, provider: str, oauth_id: str) -> User | None:
"""Get user by OAuth provider and ID.
offline-backend-20260512/backend/app/gateway/auth/repositories/sqlite.py
+195 -118
View File
@@ -15,13 +15,13 @@ from __future__ import annotations
from datetime import UTC
from uuid import UUID
from sqlalchemy import func, select, update
from sqlalchemy import func, or_, select, update
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from app.gateway.auth.models import User
from app.gateway.auth.repositories.base import CredentialConflictError, UserNotFoundError, UserRepository
from deerflow.persistence.user.model import AuthInitializationRow, AuthUsernameRow, UserRow
from app.gateway.auth.repositories.base import CredentialConflictError, UserNotFoundError, UserRepository
from deerflow.persistence.user.model import AuthInitializationRow, AuthUsernameRow, UserRow
class SQLiteUserRepository(UserRepository):
@@ -55,6 +55,7 @@ class SQLiteUserRepository(UserRepository):
return UserRow(
id=str(user.id),
email=user.email,
username_key=(user.email or "").split("@", 1)[0].strip().lower()[:128],
password_hash=user.password_hash,
system_role=user.system_role,
created_at=user.created_at,
@@ -72,7 +73,7 @@ class SQLiteUserRepository(UserRepository):
"""Insert a new user. Raises ``ValueError`` on duplicate email."""
row = self._user_to_row(user)
async with self._sf() as session:
await self._reserve_username(session, user.email)
await self._reserve_username(session, user.email)
session.add(row)
try:
await session.commit()
@@ -81,62 +82,102 @@ class SQLiteUserRepository(UserRepository):
raise ValueError(f"Email already registered: {user.email}") from exc
return user
@staticmethod
async def _reserve_username(session: AsyncSession, email: str) -> None:
"""Fence new registrations without rewriting historical user rows."""
local = email.split("@", 1)[0].lower()
escaped = local.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
existing = await session.scalar(select(UserRow.id).where(func.lower(UserRow.email).like(f"{escaped}@%", escape="\\")).limit(1))
if existing is not None:
raise ValueError("Username already registered")
session.add(AuthUsernameRow(username=local))
async def create_initial_admin(self, user: User) -> User:
"""Claim a singleton and create the first admin in one transaction.
A unique primary key fences concurrent requests across Gateway workers,
including requests using different email addresses.
"""
async with self._sf() as session:
count = await session.scalar(select(func.count()).select_from(UserRow).where(UserRow.system_role == "admin"))
if count:
raise ValueError("System already initialized")
await self._reserve_username(session, user.email)
session.add(AuthInitializationRow(id="initial-admin"))
session.add(self._user_to_row(user))
try:
await session.commit()
except IntegrityError as exc:
await session.rollback()
raise ValueError("System already initialized") from exc
return user
@staticmethod
async def _reserve_username(session: AsyncSession, email: str) -> None:
"""Fence new registrations without rewriting historical user rows."""
local = email.split("@", 1)[0].lower()
escaped = local.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
existing = await session.scalar(select(UserRow.id).where(UserRow.username_key == local).limit(1))
if existing is None:
existing = await session.scalar(
select(UserRow.id)
.where(func.lower(UserRow.email).like(f"{escaped}@%", escape="\\"))
.limit(1)
)
if existing is not None:
raise ValueError("Username already registered")
session.add(AuthUsernameRow(username=local))
async def create_initial_admin(self, user: User) -> User:
"""Claim a singleton and create the first admin in one transaction.
A unique primary key fences concurrent requests across Gateway workers,
including requests using different email addresses.
"""
async with self._sf() as session:
count = await session.scalar(select(func.count()).select_from(UserRow).where(UserRow.system_role == "admin"))
if count:
raise ValueError("System already initialized")
await self._reserve_username(session, user.email)
session.add(AuthInitializationRow(id="initial-admin"))
session.add(self._user_to_row(user))
try:
await session.commit()
except IntegrityError as exc:
await session.rollback()
raise ValueError("System already initialized") from exc
return user
async def get_user_by_id(self, user_id: str) -> User | None:
async with self._sf() as session:
row = await session.get(UserRow, user_id)
return self._row_to_user(row) if row is not None else None
async def get_users_by_ids(self, user_ids: list[str]) -> list[User]:
wanted = list(dict.fromkeys(user_id for user_id in user_ids if user_id))
if not wanted:
return []
async with self._sf() as session:
result = await session.execute(select(UserRow).where(UserRow.id.in_(wanted)))
return [self._row_to_user(row) for row in result.scalars()]
async def get_user_by_email(self, email: str) -> User | None:
stmt = select(UserRow).where(func.lower(UserRow.email) == email.strip().lower()).order_by(UserRow.created_at.asc()).limit(1)
normalized = email.strip().lower()
stmt = select(UserRow).where(UserRow.email == normalized).order_by(UserRow.created_at.asc()).limit(1)
async with self._sf() as session:
result = await session.execute(stmt)
row = result.scalar_one_or_none()
if row is None:
result = await session.execute(
select(UserRow)
.where(func.lower(UserRow.email) == normalized)
.order_by(UserRow.created_at.asc())
.limit(1)
)
row = result.scalar_one_or_none()
return self._row_to_user(row) if row is not None else None
async def find_user_by_email_local_part(self, local_part: str) -> User | None:
local = (local_part or "").strip().lower()
if not local:
return None
# Escape SQL LIKE wildcards so usernames like ``foo_bar`` or ``a%b`` do
# Compatibility fallback for rows predating the indexed username key.
# not match unrelated rows. ``\`` is the chosen escape char below.
escaped = local.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
pattern = f"{escaped}@%"
# Case-insensitive prefix match on the email field, oldest first so a
# subsequent registration cannot shadow an existing account.
stmt = select(UserRow).where(func.lower(UserRow.email).like(pattern, escape="\\")).order_by(UserRow.created_at.asc()).limit(1)
async with self._sf() as session:
result = await session.execute(stmt)
result = await session.execute(
select(UserRow)
.where(UserRow.username_key == local)
.order_by(UserRow.created_at.asc())
.limit(1)
)
row = result.scalar_one_or_none()
if row is None:
result = await session.execute(
select(UserRow)
.where(func.lower(UserRow.email).like(pattern, escape="\\"))
.order_by(UserRow.created_at.asc())
.limit(1)
)
row = result.scalar_one_or_none()
if row is not None and not row.username_key:
await session.execute(
update(UserRow)
.where(UserRow.id == row.id, UserRow.username_key.is_(None))
.values(username_key=local)
)
await session.commit()
return self._row_to_user(row) if row is not None else None
async def find_users_by_email_local_part_like(self, substring: str, *, limit: int = 500) -> list[User]:
@@ -150,7 +191,7 @@ class SQLiteUserRepository(UserRepository):
# Prefilter in SQL on the whole email (portable across sqlite/postgres/
# mysql), then keep only rows whose *local part* contains the term so a
# domain-only hit (e.g. searching "gmail") doesn't pull unrelated users.
stmt = select(UserRow).where(func.lower(UserRow.email).like(pattern, escape="\\")).order_by(UserRow.created_at.asc()).limit(max(1, limit))
stmt = select(UserRow).where(func.lower(UserRow.email).like(pattern, escape="\\")).order_by(UserRow.created_at.asc()).limit(max(1, limit))
async with self._sf() as session:
result = await session.execute(stmt)
rows = result.scalars().all()
@@ -163,85 +204,87 @@ class SQLiteUserRepository(UserRepository):
async def update_user(self, user: User) -> User:
async with self._sf() as session:
# Profile/approval writes cannot restore credentials from a stale
# User object after a concurrent password reset.
result = await session.execute(
update(UserRow)
.where(UserRow.id == str(user.id), UserRow.token_version == user.token_version)
.values(
email=user.email,
system_role=user.system_role,
oauth_provider=user.oauth_provider,
oauth_id=user.oauth_id,
needs_setup=user.needs_setup,
approved=user.approved,
)
)
if result.rowcount != 1:
row = await session.get(UserRow, str(user.id))
await session.rollback()
if row is None:
raise UserNotFoundError(f"User {user.id} no longer exists")
raise CredentialConflictError("账号密码已更新，请刷新后重试")
# Profile/approval writes cannot restore credentials from a stale
# User object after a concurrent password reset.
result = await session.execute(
update(UserRow)
.where(UserRow.id == str(user.id), UserRow.token_version == user.token_version)
.values(
email=user.email,
username_key=(user.email or "").split("@", 1)[0].strip().lower()[:128],
system_role=user.system_role,
oauth_provider=user.oauth_provider,
oauth_id=user.oauth_id,
needs_setup=user.needs_setup,
approved=user.approved,
)
)
if result.rowcount != 1:
row = await session.get(UserRow, str(user.id))
await session.rollback()
if row is None:
raise UserNotFoundError(f"User {user.id} no longer exists")
raise CredentialConflictError("账号密码已更新，请刷新后重试")
await session.commit()
return user
async def update_credentials(self, user: User, password_hash: str | None, *, email: str | None = None) -> User:
values = {
"password_hash": password_hash,
"login_password": password_hash,
"token_version": UserRow.token_version + 1,
"needs_setup": False,
}
if email is not None:
values["email"] = email
async with self._sf() as session:
if email is not None and email.split("@", 1)[0].lower() != user.email.split("@", 1)[0].lower():
await self._reserve_username(session, email)
with session.no_autoflush:
result = await session.execute(update(UserRow).where(UserRow.id == str(user.id), UserRow.token_version == user.token_version).values(**values))
if result.rowcount != 1:
await session.rollback()
raise CredentialConflictError("账号密码已更新，请重新登录后操作")
try:
await session.commit()
except IntegrityError as exc:
await session.rollback()
raise ValueError("Email already registered") from exc
user.password_hash = password_hash
user.login_password = password_hash
user.token_version += 1
user.needs_setup = False
if email is not None:
user.email = email
return user
async def start_login_session(self, user: User) -> User:
"""Atomically fence older logins without accepting a concurrently reset password.
Ordinary login races may advance the version again; the last issued
generation alone remains usable. Credential/approval changes after the
original check require the caller to authenticate again instead.
"""
async with self._sf() as session:
result = await session.execute(
update(UserRow)
.where(
UserRow.id == str(user.id),
UserRow.password_hash == user.password_hash,
UserRow.login_password == user.login_password,
UserRow.approved == user.approved,
UserRow.system_role == user.system_role,
)
.values(token_version=UserRow.token_version + 1)
)
if result.rowcount != 1:
await session.rollback()
raise CredentialConflictError("账号凭据或权限已变更，请重新登录")
row = (await session.execute(select(UserRow).where(UserRow.id == str(user.id)))).scalar_one()
saved = self._row_to_user(row)
await session.commit()
return saved
async def update_credentials(self, user: User, password_hash: str | None, *, email: str | None = None) -> User:
values = {
"password_hash": password_hash,
"login_password": password_hash,
"token_version": UserRow.token_version + 1,
"needs_setup": False,
}
if email is not None:
values["email"] = email
values["username_key"] = email.split("@", 1)[0].strip().lower()[:128]
async with self._sf() as session:
if email is not None and email.split("@", 1)[0].lower() != user.email.split("@", 1)[0].lower():
await self._reserve_username(session, email)
with session.no_autoflush:
result = await session.execute(update(UserRow).where(UserRow.id == str(user.id), UserRow.token_version == user.token_version).values(**values))
if result.rowcount != 1:
await session.rollback()
raise CredentialConflictError("账号密码已更新，请重新登录后操作")
try:
await session.commit()
except IntegrityError as exc:
await session.rollback()
raise ValueError("Email already registered") from exc
user.password_hash = password_hash
user.login_password = password_hash
user.token_version += 1
user.needs_setup = False
if email is not None:
user.email = email
return user
async def start_login_session(self, user: User) -> User:
"""Atomically fence older logins without accepting a concurrently reset password.
Ordinary login races may advance the version again; the last issued
generation alone remains usable. Credential/approval changes after the
original check require the caller to authenticate again instead.
"""
async with self._sf() as session:
result = await session.execute(
update(UserRow)
.where(
UserRow.id == str(user.id),
UserRow.password_hash == user.password_hash,
UserRow.login_password == user.login_password,
UserRow.approved == user.approved,
UserRow.system_role == user.system_role,
)
.values(token_version=UserRow.token_version + 1)
)
if result.rowcount != 1:
await session.rollback()
raise CredentialConflictError("账号凭据或权限已变更，请重新登录")
row = (await session.execute(select(UserRow).where(UserRow.id == str(user.id)))).scalar_one()
saved = self._row_to_user(row)
await session.commit()
return saved
async def count_users(self) -> int:
stmt = select(func.count()).select_from(UserRow)
@@ -259,6 +302,40 @@ class SQLiteUserRepository(UserRepository):
result = await session.execute(stmt)
return [self._row_to_user(row) for row in result.scalars()]
@staticmethod
def _user_search_filter(q: str | None):
"""Case-insensitive substring match on email or user id (None → no filter)."""
if not q or not q.strip():
return None
needle = f"%{q.strip().lower()}%"
return or_(
func.lower(UserRow.email).like(needle),
func.lower(UserRow.id).like(needle),
)
async def list_users_page(
self, *, q: str | None = None, limit: int = 20, offset: int = 0
) -> list[User]:
stmt = select(UserRow).order_by(UserRow.created_at.asc()).limit(limit).offset(offset)
search = self._user_search_filter(q)
if search is not None:
stmt = stmt.where(search)
async with self._sf() as session:
result = await session.execute(stmt)
return [self._row_to_user(row) for row in result.scalars()]
async def count_users_filtered(
self, *, q: str | None = None, approved_or_admin: bool = False
) -> int:
stmt = select(func.count()).select_from(UserRow)
search = self._user_search_filter(q)
if search is not None:
stmt = stmt.where(search)
if approved_or_admin:
stmt = stmt.where(or_(UserRow.approved.is_(True), UserRow.system_role == "admin"))
async with self._sf() as session:
return await session.scalar(stmt) or 0
async def get_user_by_oauth(self, provider: str, oauth_id: str) -> User | None:
stmt = select(UserRow).where(UserRow.oauth_provider == provider, UserRow.oauth_id == oauth_id)
async with self._sf() as session:
offline-backend-20260512/backend/app/gateway/auth_middleware.py
+247 -159
View File
@@ -1,26 +1,26 @@
"""Global authentication middleware — fail-closed safety net.
Rejects unauthenticated requests to non-public paths with 401. When a
request passes the cookie check, resolves the JWT payload to a real
``User`` object and stamps it into both ``request.state.user`` and the
``deerflow.runtime.user_context`` contextvar so that repository-layer
owner filtering works automatically via the sentinel pattern.
Fine-grained permission checks remain in authz.py decorators.
"""
"""Global authentication middleware — fail-closed safety net.
Rejects unauthenticated requests to non-public paths with 401. When a
request passes the cookie check, resolves the JWT payload to a real
``User`` object and stamps it into both ``request.state.user`` and the
``deerflow.runtime.user_context`` contextvar so that repository-layer
owner filtering works automatically via the sentinel pattern.
Fine-grained permission checks remain in authz.py decorators.
"""
import hashlib
import hmac
import re
from collections.abc import Callable
from fastapi import HTTPException, Request, Response
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.responses import JSONResponse
from starlette.types import ASGIApp
from app.gateway.auth.disabled_mode import get_assumed_user_for_disabled_auth, is_auth_disabled
from app.gateway.auth.errors import AuthErrorCode, AuthErrorResponse
import re
from collections.abc import Callable
from fastapi import HTTPException, Request, Response
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.responses import JSONResponse
from starlette.types import ASGIApp
from app.gateway.auth.disabled_mode import get_assumed_user_for_disabled_auth, is_auth_disabled
from app.gateway.auth.errors import AuthErrorCode, AuthErrorResponse
from app.gateway.authz import _ALL_PERMISSIONS, AuthContext
from app.gateway.internal_auth import INTERNAL_AUTH_HEADER_NAME, get_internal_user, is_valid_internal_auth_token
from app.gateway.proxy_path import app_relative_path
@@ -30,7 +30,7 @@ from app.gateway.weknora_embed import (
is_weknora_embed_proxy_path,
)
from deerflow.runtime.user_context import reset_current_user, set_current_user
# Health probes are deliberately public, including the LangGraph-compatible
# gateway alias used by some deployments. Keep these as exact paths: health
# checks must bypass authentication, but similarly named application routes
@@ -44,31 +44,32 @@ PUBLIC_HEALTH_PATHS: frozenset[str] = frozenset(
)
# Path prefixes that never require authentication.
#
# ``/api/public/`` is a namespace reserved for intentionally world-readable,
# read-only endpoints (e.g. public conversation share snapshots). Only mount
# safe read-only routes under it — everything there bypasses auth entirely.
#
# ``/api/public/`` is a namespace reserved for intentionally world-readable,
# read-only endpoints (e.g. public conversation share snapshots). Only mount
# safe read-only routes under it — everything there bypasses auth entirely.
_PUBLIC_PATH_PREFIXES: tuple[str, ...] = (
"/docs",
"/redoc",
"/openapi.json",
"/api/public/",
# 开放问答接口:无需登录即可指定模型 + 智能体做问答(见 routers/open_chat.py)。
# 以 "default" 用户桶运行,只暴露受控的问答能力。
"/api/open/",
)
# Exact auth paths that are public (login/register/status check).
# /api/v1/auth/me, /api/v1/auth/change-password etc. are NOT public.
"/redoc",
"/openapi.json",
"/api/public/",
# 开放问答接口:无需登录即可指定模型 + 智能体做问答(见 routers/open_chat.py)。
# 以 "default" 用户桶运行,只暴露受控的问答能力。
"/api/open/",
)
# Exact auth paths that are public (login/register/status check).
# /api/v1/auth/me, /api/v1/auth/change-password etc. are NOT public.
_PUBLIC_EXACT_PATHS: frozenset[str] = frozenset(
{
*PUBLIC_HEALTH_PATHS,
"/api/v1/auth/login/local",
"/api/v1/auth/options",
"/api/v1/auth/login/username",
"/api/v1/auth/login/token",
"/api/v1/auth/register",
"/api/v1/auth/logout",
"/api/v1/auth/login/username",
"/api/v1/auth/login/token",
"/api/v1/auth/service/exchange",
"/api/v1/auth/register",
"/api/v1/auth/logout",
"/api/v1/auth/setup-status",
"/api/v1/auth/initialize",
# Anonymous vector search over published local Wiki indexes.
@@ -99,16 +100,54 @@ def _authorize_external_llmwiki(request: Request) -> JSONResponse | None:
config = get_app_config().llmwiki.local_wiki_index.external_api
supplied = request.headers.get("X-API-Key", "")
valid = bool(config.enabled and supplied) and any(
hmac.compare_digest(supplied, candidate)
for candidate in (config.api_key, config.previous_api_key)
if candidate
)
valid = bool(config.enabled and supplied) and any(hmac.compare_digest(supplied, candidate) for candidate in (config.api_key, config.previous_api_key) if candidate)
if not valid:
return JSONResponse(status_code=401, content={"detail": "Invalid API key"})
request.state.external_api_key_fingerprint = hashlib.sha256(
supplied.encode("utf-8")
).hexdigest()[:8]
request.state.external_api_key_fingerprint = hashlib.sha256(supplied.encode("utf-8")).hexdigest()[:8]
return None
def _is_task_deeplink_external_write(request: Request, path: str) -> bool:
"""Whether this is the credential-free integration write path."""
return request.method.upper() == "PUT" and path.startswith("/api/task-deeplink-details/")
# 力量态势的任务详情由既有定时技能直接读写。该类技能无法及时统一
# 迁移到新登录鉴权，产品侧决定暂时将「当前详情」读写接口作为开放的
# 内网集成接口。严格限制为一个 task id 段与 detail 终点，不能因为
# 此豁免而放开 versions、其他力量态势接口或整个 /api/force-situation 前缀。
_FORCE_SITUATION_DETAIL_RE = re.compile(
r"^/api/force-situation/tasks/[^/]+/(?:detail|versions)$"
)
def _is_force_situation_detail_external_request(method: str, path: str) -> bool:
"""Whether this is the credential-free force-situation detail read/write API."""
normalized = path.rstrip("/")
if not _FORCE_SITUATION_DETAIL_RE.fullmatch(normalized):
return False
return method.upper() == "GET" or (
method.upper() == "PUT" and normalized.endswith("/detail")
)
def _authorize_task_deeplink_external_write(request: Request) -> JSONResponse | None:
"""Mark the external b5-1 task-detail write; no credential is required.
The embedding system writes task snapshots server-to-server before the
user's browser is involved, so by product decision this PUT is an open
endpoint: any caller may store a snapshot without a login session or an
integration key. A supplied ``X-Integration-Key`` header is accepted but
only recorded (fingerprint) for audit; it is never validated. Reads still
follow the normal login path.
"""
request.state.task_deeplink_external_writer = True
supplied = request.headers.get("X-Integration-Key", "")
if supplied:
request.state.task_deeplink_external_key_fingerprint = hashlib.sha256(supplied.encode("utf-8")).hexdigest()[:8]
return None
@@ -126,126 +165,175 @@ def _is_weknora_embed_proxy_request(request: Request) -> bool:
if not is_weknora_embed_proxy_path(path):
return False
return has_valid_weknora_embed_cookie(request.cookies.get(WEKNORA_EMBED_COOKIE))
# 「发起问答」的 run 创建接口(会真正调用大模型):
# POST /api/threads/{id}/runs[/stream|/wait]
# POST /api/runs/{stream|wait}
# （含 nginx 未改写到的 /api/langgraph/... 前缀,双保险)
# 只匹配这些「提交问答」的 POST,不碰 GET 列表 / join / cancel / feedback 等,
# 这样登录白名单开启后,未放行用户(凭旧 token)无法再用接口问答,但普通浏览不受影响。
_QA_RUN_RE = re.compile(r"^/api/(?:langgraph/)?(?:threads/[^/]+/)?runs(?:/stream|/wait)?$")
def _is_qa_run_request(method: str, path: str) -> bool:
"""True for an authenticated「发起问答」run-creation request (see ``_QA_RUN_RE``)."""
return method.upper() == "POST" and bool(_QA_RUN_RE.match(path.rstrip("/")))
class AuthMiddleware(BaseHTTPMiddleware):
"""Strict auth gate: reject requests without a valid session.
Two-stage check for non-public paths:
1. Cookie presence — return 401 NOT_AUTHENTICATED if missing
2. JWT validation via ``get_optional_user_from_request`` — return 401
TOKEN_INVALID if the token is absent, malformed, expired, or the
signed user does not exist / is stale
On success, stamps ``request.state.user`` and the
``deerflow.runtime.user_context`` contextvar so that repository-layer
owner filters work downstream without every route needing a
``@require_auth`` decorator. Routes that need per-resource
authorization (e.g. "user A cannot read user B's thread by guessing
the URL") should additionally use ``@require_permission(...,
owner_check=True)`` for explicit enforcement — but authentication
itself is fully handled here.
"""
def __init__(self, app: ASGIApp) -> None:
super().__init__(app)
async def dispatch(self, request: Request, call_next: Callable) -> Response:
# 「发起问答」的 run 创建接口(会真正调用大模型):
# POST /api/threads/{id}/runs[/stream|/wait]
# POST /api/runs/{stream|wait}
# （含 nginx 未改写到的 /api/langgraph/... 前缀,双保险)
# 只匹配这些「提交问答」的 POST,不碰 GET 列表 / join / cancel / feedback 等,
# 这样登录白名单开启后,未放行用户(凭旧 token)无法再用接口问答,但普通浏览不受影响。
_QA_RUN_RE = re.compile(r"^/api/(?:langgraph/)?(?:threads/[^/]+/)?runs(?:/stream|/wait)?$")
def _is_qa_run_request(method: str, path: str) -> bool:
"""True for an authenticated「发起问答」run-creation request (see ``_QA_RUN_RE``)."""
return method.upper() == "POST" and bool(_QA_RUN_RE.match(path.rstrip("/")))
class AuthMiddleware(BaseHTTPMiddleware):
"""Strict auth gate: reject requests without a valid session.
Two-stage check for non-public paths:
1. Cookie presence — return 401 NOT_AUTHENTICATED if missing
2. JWT validation via ``get_optional_user_from_request`` — return 401
TOKEN_INVALID if the token is absent, malformed, expired, or the
signed user does not exist / is stale
On success, stamps ``request.state.user`` and the
``deerflow.runtime.user_context`` contextvar so that repository-layer
owner filters work downstream without every route needing a
``@require_auth`` decorator. Routes that need per-resource
authorization (e.g. "user A cannot read user B's thread by guessing
the URL") should additionally use ``@require_permission(...,
owner_check=True)`` for explicit enforcement — but authentication
itself is fully handled here.
"""
def __init__(self, app: ASGIApp) -> None:
super().__init__(app)
async def dispatch(self, request: Request, call_next: Callable) -> Response:
# Match the application-relative path so a reverse-proxy prefix
# (e.g. /xxx/api/public/...) does not defeat the public whitelist.
relative_path = app_relative_path(request)
if _is_external_llmwiki(relative_path):
rejected = _authorize_external_llmwiki(request)
return rejected if rejected is not None else await call_next(request)
if _is_task_deeplink_external_write(request, relative_path):
rejected = _authorize_task_deeplink_external_write(request)
if rejected is not None:
return rejected
if getattr(request.state, "task_deeplink_external_writer", False):
return await call_next(request)
if _is_force_situation_detail_external_request(request.method, relative_path):
return await call_next(request)
if _is_public(relative_path):
return await call_next(request)
if _is_weknora_embed_proxy_request(request):
return await call_next(request)
if is_auth_disabled():
user = await get_assumed_user_for_disabled_auth()
request.state.user = user
request.state.auth = AuthContext(user=user, permissions=_ALL_PERMISSIONS)
token = set_current_user(user)
try:
return await call_next(request)
finally:
reset_current_user(token)
internal_user = None
if is_valid_internal_auth_token(request.headers.get(INTERNAL_AUTH_HEADER_NAME)):
internal_user = get_internal_user()
# Non-public path: require session cookie
from app.gateway.deps import get_request_access_token
if internal_user is None and not get_request_access_token(request):
return JSONResponse(
status_code=401,
content={
"detail": AuthErrorResponse(
code=AuthErrorCode.NOT_AUTHENTICATED,
message="Authentication required",
).model_dump()
},
)
# Strict JWT validation: reject junk/expired tokens with 401
# right here instead of silently passing through. This closes
# the "junk cookie bypass" gap (AUTH_TEST_PLAN test 7.5.8):
# without this, non-isolation routes like /api/models would
# accept any cookie-shaped string as authentication.
#
# We call the *strict* resolver so that fine-grained error
# codes (token_expired, token_invalid, user_not_found, …)
# propagate from AuthErrorCode, not get flattened into one
# generic code. BaseHTTPMiddleware doesn't let HTTPException
# bubble up, so we catch and render it as JSONResponse here.
from app.gateway.deps import get_current_user_from_request
if internal_user is not None:
user = internal_user
else:
try:
user = await get_current_user_from_request(request)
except HTTPException as exc:
return JSONResponse(status_code=exc.status_code, content={"detail": exc.detail})
# Stamp both request.state.user (for the contextvar pattern)
# and request.state.auth (so @require_permission's "auth is
# None" branch short-circuits instead of running the entire
# JWT-decode + DB-lookup pipeline a second time per request).
request.state.user = user
request.state.auth = AuthContext(user=user, permissions=_ALL_PERMISSIONS)
# 登录白名单:开关开启时,未放行用户(非 admin)不仅被拦在登录外,连「接口
# 问答」也封掉——已拿到 token 的未放行用户用旧会话直接调问答接口同样被拒。
# 仅作用于「发起问答」的 run 接口;内部/渠道调用(internal_user)不受限。
if internal_user is None and _is_qa_run_request(request.method, app_relative_path(request)):
from app.gateway.routers.auth import enforce_user_approved
try:
enforce_user_approved(user)
except HTTPException as exc:
return JSONResponse(status_code=exc.status_code, content={"detail": exc.detail})
token = set_current_user(user)
try:
return await call_next(request)
finally:
reset_current_user(token)
user = await get_assumed_user_for_disabled_auth()
request.state.user = user
request.state.auth = AuthContext(user=user, permissions=_ALL_PERMISSIONS)
token = set_current_user(user)
try:
return await call_next(request)
finally:
reset_current_user(token)
internal_user = None
if is_valid_internal_auth_token(request.headers.get(INTERNAL_AUTH_HEADER_NAME)):
internal_user = get_internal_user()
# Non-public path: require session cookie
from app.gateway.deps import get_request_access_token
if internal_user is None and not get_request_access_token(request):
return JSONResponse(
status_code=401,
content={
"detail": AuthErrorResponse(
code=AuthErrorCode.NOT_AUTHENTICATED,
message="Authentication required",
).model_dump()
},
)
# Strict JWT validation: reject junk/expired tokens with 401
# right here instead of silently passing through. This closes
# the "junk cookie bypass" gap (AUTH_TEST_PLAN test 7.5.8):
# without this, non-isolation routes like /api/models would
# accept any cookie-shaped string as authentication.
#
# We call the *strict* resolver so that fine-grained error
# codes (token_expired, token_invalid, user_not_found, …)
# propagate from AuthErrorCode, not get flattened into one
# generic code. BaseHTTPMiddleware doesn't let HTTPException
# bubble up, so we catch and render it as JSONResponse here.
from app.gateway.deps import get_current_user_from_request
if internal_user is not None:
user = internal_user
else:
try:
user = await get_current_user_from_request(request)
except HTTPException as exc:
return JSONResponse(status_code=exc.status_code, content={"detail": exc.detail})
if internal_user is None:
from app.gateway.auth import TokenError, decode_token
payload = decode_token(get_request_access_token(request) or "")
if not isinstance(payload, TokenError) and payload.auth_method == "service":
method = request.method.upper()
path = app_relative_path(request).rstrip("/")
allowed = _is_service_scope_request(
payload.scope,
method,
path,
all_requested=request.query_params.get("all", "").lower() in {"true", "1", "yes", "on"},
)
if not allowed:
return JSONResponse(status_code=403, content={"detail": "Service token is not permitted for this endpoint"})
# Stamp both request.state.user (for the contextvar pattern)
# and request.state.auth (so @require_permission's "auth is
# None" branch short-circuits instead of running the entire
# JWT-decode + DB-lookup pipeline a second time per request).
request.state.user = user
request.state.auth = AuthContext(user=user, permissions=_ALL_PERMISSIONS)
# 登录白名单:开关开启时,未放行用户(非 admin)不仅被拦在登录外,连「接口
# 问答」也封掉——已拿到 token 的未放行用户用旧会话直接调问答接口同样被拒。
# 仅作用于「发起问答」的 run 接口;内部/渠道调用(internal_user)不受限。
if internal_user is None and _is_qa_run_request(request.method, app_relative_path(request)):
from app.gateway.routers.auth import enforce_user_approved
try:
enforce_user_approved(user)
except HTTPException as exc:
return JSONResponse(status_code=exc.status_code, content={"detail": exc.detail})
token = set_current_user(user)
try:
return await call_next(request)
finally:
reset_current_user(token)
_LLMWIKI_SYNC_COLLECTION = "/api/llmwiki/knowledge-bases"
_LLMWIKI_SYNC_ITEM_RE = re.compile(r"^/api/llmwiki/knowledge-bases/[^/]+$")
_LLMWIKI_SYNC_DOCUMENT_RE = re.compile(r"^/api/llmwiki/knowledge-bases/[^/]+/documents(?:/[^/]+)?$")
_LLMWIKI_SYNC_MANUAL_RE = re.compile(r"^/api/llmwiki/knowledge-bases/[^/]+/documents/manual$")
def _is_llmwiki_sync_request(method: str, path: str) -> bool:
if path == _LLMWIKI_SYNC_COLLECTION:
return method in {"GET", "POST"}
if _LLMWIKI_SYNC_ITEM_RE.fullmatch(path):
return method in {"GET", "PATCH", "DELETE"}
if _LLMWIKI_SYNC_MANUAL_RE.fullmatch(path):
return method == "POST"
if _LLMWIKI_SYNC_DOCUMENT_RE.fullmatch(path):
return method in {"GET", "POST", "DELETE"}
return False
def _is_service_scope_request(scopes: list[str], method: str, path: str, *, all_requested: bool = False) -> bool:
return (
("llmwiki:sync" in scopes and _is_llmwiki_sync_request(method, path))
or ("recommended-questions:read" in scopes and method == "GET" and path == "/api/recommended-questions" and not all_requested)
)
offline-backend-20260512/backend/app/gateway/authz.py
+43 -28
View File
@@ -29,6 +29,7 @@ Inspired by LangGraph Auth system: https://github.com/langchain-ai/langgraph/blo
from __future__ import annotations
import asyncio
import functools
import inspect
import logging
@@ -49,6 +50,15 @@ P = ParamSpec("P")
T = TypeVar("T")
async def _project_for_thread(project_store: Any, thread_id: str) -> dict | None:
"""Best-effort project binding lookup; a store error must not fail the read."""
try:
return await project_store.project_for_thread(thread_id)
except Exception:
logger.debug("project_for_thread failed for %s", thread_id, exc_info=True)
return None
async def _thread_checkpoint_exists(request: Request, thread_id: str) -> bool:
"""Return True if a LangGraph checkpoint exists for ``thread_id``.
@@ -269,8 +279,8 @@ def require_permission(
HTTPException 401: If authentication required but user is anonymous
HTTPException 403: If user lacks permission
HTTPException 404: If owner_check=True but user doesn't own the thread
ValueError: If owner_check=True but no ``thread_id`` is supplied,
either directly or on the Pydantic request body
ValueError: If owner_check=True but no ``thread_id`` is supplied,
either directly or on the Pydantic request body
"""
def decorator(func: Callable[P, T]) -> Callable[P, T]:
@@ -312,40 +322,45 @@ def require_permission(
# ``ThreadMetaStore.check_access``: it returns True for
# missing rows (untracked legacy thread) and for rows whose
# ``user_id`` is NULL (shared / pre-auth data), so this is
# strict-deny rather than strict-allow — only an *existing*
# row with a *different* user_id triggers 404.
if owner_check:
thread_id = kwargs.get("thread_id")
# Most thread-scoped routes expose ``thread_id`` in their URL.
# A small number of mutation endpoints (for example complete
# document rewrite) intentionally keep the target inside a
# Pydantic request body so their public URL stays stable. The
# FastAPI wrapper passes that body as ``body=...``; accept its
# Python field name here while preserving the exact same
# ownership verification below.
if thread_id is None:
body = kwargs.get("body")
thread_id = getattr(body, "thread_id", None)
if thread_id is None:
raise ValueError(
"require_permission with owner_check=True requires "
"a 'thread_id' parameter or body.thread_id"
)
# strict-deny rather than strict-allow — only an *existing*
# row with a *different* user_id triggers 404.
if owner_check:
thread_id = kwargs.get("thread_id")
# Most thread-scoped routes expose ``thread_id`` in their URL.
# A small number of mutation endpoints (for example complete
# document rewrite) intentionally keep the target inside a
# Pydantic request body so their public URL stays stable. The
# FastAPI wrapper passes that body as ``body=...``; accept its
# Python field name here while preserving the exact same
# ownership verification below.
if thread_id is None:
body = kwargs.get("body")
thread_id = getattr(body, "thread_id", None)
if thread_id is None:
raise ValueError(
"require_permission with owner_check=True requires "
"a 'thread_id' parameter or body.thread_id"
)
from app.gateway.deps import get_thread_store
thread_store = get_thread_store(request)
user_id = str(auth.user.id)
project_store = getattr(request.app.state, "analysis_project_store", None)
if project_store is not None:
project = await project_store.project_for_thread(thread_id)
if project and project["user_id"] != user_id:
raise HTTPException(status_code=404, detail="Project thread not accessible")
allowed = await thread_store.check_access(
user_id = str(auth.user.id)
project_store = getattr(request.app.state, "analysis_project_store", None)
allowed_coro = thread_store.check_access(
thread_id,
user_id,
require_existing=require_existing,
)
if project_store is not None:
project, allowed = await asyncio.gather(
_project_for_thread(project_store, thread_id),
allowed_coro,
)
if project and project["user_id"] != user_id:
raise HTTPException(status_code=404, detail="Project thread not accessible")
else:
allowed = await allowed_coro
if not allowed:
# Tell apart three denial causes so the client gets an
# accurate message and so DB-switch orphans can be adopted
offline-backend-20260512/backend/app/gateway/catalog_cache.py
+66
View File
@@ -0,0 +1,66 @@
"""Shared Redis read-through for small configuration catalogues.
Callers authorize before entering this helper and include any user/query scope
in the key. Only successful reads are stored; Redis failures keep the original
database/file-backed path available.
"""
from __future__ import annotations
import json
from collections.abc import Awaitable, Callable
from typing import Any
from fastapi import Request
from pydantic import BaseModel, ValidationError
from app.gateway import api_read_cache
def catalog_key(*parts: Any) -> str:
return json.dumps(parts, ensure_ascii=False, separators=(",", ":"), sort_keys=True)
async def cached_catalog[T](
request: Request,
collection: str,
key: str,
load: Callable[[], Awaitable[T]],
model: type[BaseModel] | None = None,
) -> T:
config = getattr(request.app.state, "config", None)
version = await api_read_cache.get_cache_version(config, collection)
cached = await api_read_cache.get_cached_json(config, collection, key, version=version)
def decode(value):
try:
return model.model_validate(value) if model is not None else value
except (ValidationError, TypeError):
# A stale schema or malformed entry must not break the read path.
return None
if cached is not None:
decoded = decode(cached)
if decoded is not None:
return decoded
cached, fill_token = await api_read_cache.wait_for_cached_json_or_claim_fill(
config, collection, key, version=version,
)
try:
if cached is not None:
decoded = decode(cached)
if decoded is not None:
return decoded
value = await load()
await api_read_cache.set_cached_json(config, collection, key, value, expected_version=version)
return value
finally:
await api_read_cache.release_cache_fill(config, collection, key, fill_token)
async def invalidate_catalog(request: Request, collection: str) -> None:
# Writes while caching is disabled must also invalidate old generations,
# otherwise re-enabling the collection could resurrect stale values.
await api_read_cache.invalidate_cached_collection(
getattr(request.app.state, "config", None), collection, force=True,
)
offline-backend-20260512/backend/app/gateway/config.py
+12 -2
View File
@@ -2,6 +2,8 @@ import os
from pydantic import BaseModel, Field
from deerflow.config.app_config import get_app_config
class GatewayConfig(BaseModel):
"""Configuration for the API Gateway."""
@@ -16,14 +18,22 @@ _gateway_config: GatewayConfig | None = None
def get_gateway_config() -> GatewayConfig:
"""Get gateway config, loading from environment if available."""
"""Get gateway config, preferring the shared ``config.yaml`` settings."""
global _gateway_config
if _gateway_config is None:
cors_origins_str = os.getenv("CORS_ORIGINS", "http://localhost:3000")
try:
# Documentation routes are part of the service surface, so the
# versioned deployment config is authoritative. In particular,
# a stale shell-level GATEWAY_ENABLE_DOCS=false must not silently
# suppress the intranet docs configured in config.yaml.
enable_docs = get_app_config().gateway.enable_docs
except Exception: # noqa: BLE001 - retain the legacy env fallback if config cannot load
enable_docs = os.getenv("GATEWAY_ENABLE_DOCS", "true").lower() == "true"
_gateway_config = GatewayConfig(
host=os.getenv("GATEWAY_HOST", "0.0.0.0"),
port=int(os.getenv("GATEWAY_PORT", "8001")),
cors_origins=cors_origins_str.split(","),
enable_docs=os.getenv("GATEWAY_ENABLE_DOCS", "true").lower() == "true",
enable_docs=enable_docs,
)
return _gateway_config
offline-backend-20260512/backend/app/gateway/csrf_middleware.py
+30
View File
@@ -5,6 +5,7 @@ State-changing operations require CSRF protection.
"""
import secrets
import re
from collections.abc import Callable
from fastapi import Request, Response
@@ -25,6 +26,24 @@ CSRF_HEADER_NAME = "X-CSRF-Token"
CSRF_TOKEN_LENGTH = 64 # bytes
# Must stay as narrow as AuthMiddleware's credential-free force-situation
# detail integration route: only GET/PUT of the current detail document are
# public. The version history and other force-situation endpoints keep their
# normal authentication and CSRF policy.
_FORCE_SITUATION_DETAIL_RE = re.compile(
r"^/api/force-situation/tasks/[^/]+/(?:detail|versions)$"
)
def _is_force_situation_detail_external_request(method: str, path: str) -> bool:
normalized = path.rstrip("/")
if not _FORCE_SITUATION_DETAIL_RE.fullmatch(normalized):
return False
return method.upper() == "GET" or (
method.upper() == "PUT" and normalized.endswith("/detail")
)
def is_secure_request(request: Request) -> bool:
"""Detect whether the original client request was made over HTTPS."""
return request.headers.get("x-forwarded-proto", request.url.scheme) == "https"
@@ -66,6 +85,17 @@ def should_check_csrf(request: Request) -> bool:
# 双提交 Cookie 既无 token 可比、也无会话可被伪造,豁免之。
if path.startswith("/api/open/"):
return False
# b5-1 task details are written by a non-browser integration over a
# credential-free open endpoint (see AuthMiddleware): there is no cookie
# session for a cross-site request to ride, and the write itself is open
# anyway, so the Double-Submit Cookie check is meaningless here.
if request.method == "PUT" and path.startswith("/api/task-deeplink-details/"):
return False
# The force-situation current-detail API is a credential-free internal
# integration surface. With no browser login session to protect, CSRF
# validation would only reject existing scheduler/skill writers.
if _is_force_situation_detail_external_request(request.method, path):
return False
if is_coze_studio_compat_path(path):
return False
if is_weknora_embed_proxy_request(request):
offline-backend-20260512/backend/app/gateway/deep_research_job_executor.py
+3 -1
View File
@@ -1738,6 +1738,7 @@ class DeepResearchJobExecutor:
from deerflow.agents.deep_research.adapters.image import build_image_provider
from deerflow.agents.deep_research.adapters.llm import DeerFlowCompletionBackend
from deerflow.agents.deep_research.adapters.material_provider import DeerFlowMaterialProvider
from deerflow.utils.background_init import run_background_init
# Resolve / create the hidden runtime thread for artifacts.
runtime_thread_id = await self._ensure_runtime_thread(session_id, user_id)
@@ -1828,7 +1829,8 @@ class DeepResearchJobExecutor:
cancellation=cancel_token,
)
else:
material_provider = DeerFlowMaterialProvider(
material_provider = await run_background_init(
DeerFlowMaterialProvider,
user_id=user_id,
session_id=session_id,
runtime_thread_id=runtime_thread_id,
offline-backend-20260512/backend/app/gateway/deps.py
+58 -27
View File
@@ -78,15 +78,14 @@ async def _enter_runtime_backend_or_fallback[BackendT](
return await stack.enter_async_context(context_manager)
except Exception:
logger.exception(
"%s initialization failed; continuing Gateway startup with an "
"in-memory fallback",
"%s initialization failed; continuing Gateway startup with an in-memory fallback",
label,
)
return fallback_factory()
@asynccontextmanager
async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
async def langgraph_runtime(app: FastAPI, *, background: bool = True) -> AsyncGenerator[None, None]:
"""Bootstrap and tear down all LangGraph runtime singletons.
Usage in ``app.py``::
@@ -158,6 +157,10 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
app.state.thread_store,
)
from deerflow.persistence.task_deeplink_details import make_task_deeplink_detail_store
app.state.task_deeplink_detail_store = make_task_deeplink_detail_store(sf)
from deerflow.persistence.artifact_library import make_artifact_library_store
app.state.artifact_library_store = make_artifact_library_store(sf)
@@ -186,6 +189,10 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
app.state.tool_metrics_store = make_tool_metrics_store(sf)
from deerflow.persistence.stream_truncation import make_stream_truncation_store
app.state.stream_truncation_store = make_stream_truncation_store(sf)
from deerflow.persistence.agents import make_agent_store
app.state.agent_store = make_agent_store(sf)
@@ -251,7 +258,7 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
app.state.ai_writing_job_executor = AIWritingJobExecutor(app)
# 进程重启续跑：把仍标记为 background 的会话重新拉起驱动（best-effort，不阻塞启动）。
if app.state.ai_writing_session_store is not None:
if background and app.state.ai_writing_session_store is not None:
from app.gateway.ai_writing_job_executor import STATUS_BACKGROUND
async def _reconcile_ai_writing_background() -> None:
@@ -342,9 +349,7 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
try:
await app.state.position_role_store.ensure_seed(POSITION_ROLE_SEED)
except Exception:
logger.exception(
"Position-role seeding failed at startup; continuing without seed refresh"
)
logger.exception("Position-role seeding failed at startup; continuing without seed refresh")
from deerflow.persistence.business_mapping import make_business_mapping_store
@@ -380,7 +385,7 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
app.state.report_collaboration_executor = None
app.state.report_collaboration_dispatcher = None
rc_cfg = getattr(config, "report_collaboration", None)
if rc_cfg is not None and rc_cfg.enabled and rc_cfg.worker_enabled:
if background and rc_cfg is not None and rc_cfg.enabled and rc_cfg.worker_enabled:
from app.report_collaboration.agentscope_runtime.member_kernel import AgentScopeMemberKernel
from app.report_collaboration.execution.dispatcher import ReportCollaborationDispatcher
from app.report_collaboration.execution.executor import ReportCollaborationExecutor
@@ -440,23 +445,17 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
app.state.upload_policy_store = make_upload_policy_store(sf)
fallback_upload_policy = default_upload_policy()
try:
upload_policy_row = await app.state.upload_policy_store.ensure_seed(
fallback_upload_policy
)
upload_policy_row = await app.state.upload_policy_store.ensure_seed(fallback_upload_policy)
set_runtime_upload_policy(upload_policy_row["policy"])
except Exception:
logger.exception(
"Upload-policy seeding failed at startup; using process-local defaults"
)
logger.exception("Upload-policy seeding failed at startup; using process-local defaults")
set_runtime_upload_policy(fallback_upload_policy)
from deerflow.persistence.knowledge_retrieval_settings import (
make_knowledge_retrieval_settings_store,
)
app.state.knowledge_retrieval_settings_store = (
make_knowledge_retrieval_settings_store(sf)
)
app.state.knowledge_retrieval_settings_store = make_knowledge_retrieval_settings_store(sf)
from deerflow.persistence.task_analysis import make_task_analysis_store
@@ -466,6 +465,12 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
app.state.analysis_project_store = AnalysisProjectStore(sf)
from deerflow.persistence.knowledge_research import (
make_knowledge_research_store,
)
app.state.knowledge_research_store = make_knowledge_research_store(sf)
from app.gateway.analysis_project_memory import (
AnalysisProjectMemoryDispatcher,
AnalysisProjectMemoryExecutor,
@@ -501,7 +506,7 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
app.state.assistant_knowledge_store = make_assistant_knowledge_store(sf)
register_assistant_knowledge_store(app.state.assistant_knowledge_store)
app.state.skill_knowledge_store = make_skill_knowledge_store(sf)
if app.state.skill_knowledge_store is not None and app.state.assistant_knowledge_store is not None:
if background and app.state.skill_knowledge_store is not None and app.state.assistant_knowledge_store is not None:
from app.gateway.skill_knowledge_job_dispatcher import SkillKnowledgeJobDispatcher
from app.gateway.skill_knowledge_job_executor import SkillKnowledgeJobExecutor
@@ -544,7 +549,8 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
# Start only after the optional embedding runtime is ready. Otherwise
# persisted jobs reclaimed during startup would be permanently stored
# as lexical-only even when phase-two vector recall is configured.
app.state.analysis_project_memory_dispatcher.start()
if background:
app.state.analysis_project_memory_dispatcher.start()
from deerflow.persistence.taskcop_tasks import make_taskcop_task_store
@@ -575,6 +581,10 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
)
app.state.roundtable_job_store = make_roundtable_job_store(sf)
from deerflow.persistence.roundtable_rooms import make_roundtable_room_store
app.state.roundtable_room_store = make_roundtable_room_store(sf)
# 让 LangGraph Server 路径下的圆桌编排图节点能跨 app/harness 边界拿到 repo
# （Phase 2 后台编排图写进度用）。
_register_roundtable_job_default_store(app.state.roundtable_job_store)
@@ -659,7 +669,8 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
draft_store=getattr(app.state, "roundtable_draft_store", None),
task_draft_store=getattr(app.state, "roundtable_task_draft_store", None),
)
app.state.roundtable_job_dispatcher.start()
if background:
app.state.roundtable_job_dispatcher.start()
else:
app.state.roundtable_job_executor = None
app.state.roundtable_job_dispatcher = None
@@ -727,7 +738,8 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
app.state.deep_research_job_store,
app.state.deep_research_executor,
)
app.state.deep_research_dispatcher.start()
if background:
app.state.deep_research_dispatcher.start()
else:
app.state.deep_research_executor = None
app.state.deep_research_dispatcher = None
@@ -758,7 +770,8 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
live_publisher=app.state.workflow_live_hub.publish,
config=workflow_cfg,
)
app.state.workflow_dispatcher.start()
if background:
app.state.workflow_dispatcher.start()
from app.gateway.workflow_retention import WorkflowRetentionCleaner
app.state.workflow_retention = WorkflowRetentionCleaner(
@@ -766,7 +779,8 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
app.state.workflow_event_store,
config=workflow_cfg,
)
app.state.workflow_retention.start()
if background:
app.state.workflow_retention.start()
else:
app.state.workflow_live_hub = None
app.state.workflow_executor = None
@@ -779,6 +793,8 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
# RunManager with store backing for persistence
app.state.run_manager = RunManager(store=app.state.run_store)
retention = config.runtime_contention
app.state.run_manager.start_retention(retention_seconds=retention.completed_run_retention_seconds, interval_seconds=retention.run_sweep_seconds)
from deerflow.runtime import make_redis_concurrency_gate
app.state.concurrency_gate = make_redis_concurrency_gate(config)
@@ -786,12 +802,11 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
try:
yield
finally:
await app.state.run_manager.aclose()
# Project-memory extraction can use the shared local Wiki encoder.
# Stop its jobs before retiring that runtime to avoid an embedding
# call racing shutdown.
analysis_project_memory_dispatcher = getattr(
app.state, "analysis_project_memory_dispatcher", None
)
analysis_project_memory_dispatcher = getattr(app.state, "analysis_project_memory_dispatcher", None)
if analysis_project_memory_dispatcher is not None:
await analysis_project_memory_dispatcher.stop()
from app.gateway.assistant_file_ingest import stop_file_jobs
@@ -894,6 +909,14 @@ def get_task_deeplink_thread_store(request: Request):
return val
def get_task_deeplink_detail_store(request: Request):
"""Return the external task-detail store used by task deep links."""
val = getattr(request.app.state, "task_deeplink_detail_store", None)
if val is None:
raise HTTPException(status_code=503, detail="Task deep-link detail store not available")
return val
def get_scheduled_task_store(request: Request) -> ScheduledTaskStore:
"""Return the scheduled-task store."""
val = getattr(request.app.state, "scheduled_task_store", None)
@@ -917,6 +940,11 @@ def get_tool_metrics_store(request: Request):
return getattr(request.app.state, "tool_metrics_store", None)
def get_stream_truncation_store(request: Request):
"""Return the stream-truncation event store, or None when not configured."""
return getattr(request.app.state, "stream_truncation_store", None)
def get_agent_store(request: Request) -> AgentStore:
"""Return the custom-agent store."""
val = getattr(request.app.state, "agent_store", None)
@@ -1125,7 +1153,10 @@ async def get_current_user_from_request(request: Request):
detail=AuthErrorResponse(code=token_error_to_code(payload), message=f"Token error: {payload.value}").model_dump(),
)
if policy_error := session_policy_error(payload):
# Scoped service tokens are authorized by AuthMiddleware's strict path
# allow-list. They are intentionally not accepted by the separate
# LangGraph auth path, which still applies session_policy_error directly.
if payload.auth_method != "service" and (policy_error := session_policy_error(payload)):
raise HTTPException(
status_code=401,
detail=AuthErrorResponse(code=AuthErrorCode.TOKEN_INVALID, message=policy_error).model_dump(),
offline-backend-20260512/backend/app/gateway/knowledge_research.py
+285
View File
@@ -0,0 +1,285 @@
"""Server-owned run policy for knowledge-base research conversations."""
from __future__ import annotations
from copy import deepcopy
from typing import Any
from fastapi import HTTPException, Request
from langchain_core.messages import HumanMessage
_SAFE_GENERATION_PREFERENCE_KEYS = frozenset(
{
"model_name",
"mode",
"reasoning_effort",
"thinking_enabled",
"thinking_force_disabled",
}
)
def _text_only_content(value: Any) -> Any:
"""Remove file/image/audio blocks while preserving the user's text."""
if isinstance(value, str):
return value
if isinstance(value, dict):
block_type = str(value.get("type") or "text").strip().lower()
text = value.get("text") if block_type in {"text", "input_text"} else None
if not isinstance(text, str) and block_type in {"text", "input_text"}:
text = value.get("content")
return (
[{"type": "text", "text": text}]
if isinstance(text, str)
else ""
)
if not isinstance(value, list):
return ""
kept: list[Any] = []
for block in value:
if isinstance(block, str):
kept.append(block)
continue
if not isinstance(block, dict):
continue
block_type = str(block.get("type") or "text").strip().lower()
if block_type not in {"text", "input_text"}:
continue
text = block.get("text")
if not isinstance(text, str):
text = block.get("content")
if isinstance(text, str):
kept.append({"type": "text", "text": text})
return kept
def _content_text(value: Any) -> str:
if isinstance(value, str):
return value
if not isinstance(value, list):
return ""
parts: list[str] = []
for block in value:
if isinstance(block, str):
parts.append(block)
elif isinstance(block, dict):
text = block.get("text") or block.get("content")
if isinstance(text, str):
parts.append(text)
return "\n".join(parts)
def _sanitize_messages(body: Any) -> None:
payload = getattr(body, "input", None)
messages = payload.get("messages") if isinstance(payload, dict) else None
if not isinstance(messages, list):
if isinstance(payload, dict):
body.input = {"messages": []}
return
latest_user_message: Any | None = None
for message in messages:
if isinstance(message, HumanMessage):
latest_user_message = HumanMessage(
content=_text_only_content(message.content)
)
continue
if not isinstance(message, dict):
continue
role = str(message.get("role") or message.get("type") or "").lower()
if role not in {"human", "user"}:
# Checkpoint state is the only source of prior assistant/system/tool
# turns. The current request may contribute user text only.
continue
latest_user_message = {
"role": "user",
"content": _text_only_content(message.get("content")),
}
# The checkpoint is the only source of conversation history. A run may
# contribute exactly one current user turn, so callers cannot fabricate
# earlier user messages either.
body.input = {
"messages": [latest_user_message] if latest_user_message is not None else []
}
def _safe_generation_preferences(body: Any) -> dict[str, Any]:
"""Copy the ordinary-chat model controls, and no other caller context."""
sources: list[dict[str, Any]] = []
config = getattr(body, "config", None)
if isinstance(config, dict):
for scope in ("configurable", "context"):
nested = config.get(scope)
if isinstance(nested, dict):
sources.append(nested)
context = getattr(body, "context", None)
if isinstance(context, dict):
# Public body.context has the same precedence it has in ordinary Q&A.
sources.append(context)
preferences: dict[str, Any] = {}
for source in sources:
for key in _SAFE_GENERATION_PREFERENCE_KEYS:
value = source.get(key)
if key in {"thinking_enabled", "thinking_force_disabled"}:
if isinstance(value, bool):
preferences[key] = value
elif isinstance(value, str) and value.strip():
preferences[key] = value.strip()[:255]
return preferences
async def prepare_knowledge_research_run(
body: Any,
thread_id: str,
request: Request,
) -> tuple[Any, dict[str, Any] | None]:
"""Bind every run to its persisted owner and one authorized knowledge base.
Thread metadata is observability only. The database binding is resolved on
every turn, and read authorization is rechecked before any retrieval runs.
"""
store = getattr(request.app.state, "knowledge_research_store", None)
if store is None:
return body, None
session = await store.get_by_thread(thread_id)
if session is None:
return body, None
# Local import avoids a module cycle: analysis_projects invokes this hook.
from app.gateway.analysis_projects import actor
user_id, is_admin = await actor(request)
if session["user_id"] != user_id:
# Administrators do not inherit access to another user's private chat.
raise HTTPException(status_code=404, detail="研究会话不存在或无权访问")
mapping_store = getattr(request.app.state, "llmwiki_store", None)
if mapping_store is None:
raise HTTPException(status_code=503, detail="知识库服务暂不可用")
mapping = await mapping_store.get_authorized(
session["knowledge_base_id"],
user_id,
write=False,
is_admin=is_admin,
)
if mapping is None:
raise HTTPException(status_code=404, detail="知识库不存在或已无访问权限")
prepared = body.model_copy(deep=True) if hasattr(body, "model_copy") else deepcopy(body)
generation_preferences = _safe_generation_preferences(prepared)
# Research accepts user text, not arbitrary run context/metadata. Rebuild
# both from server state so task/workbench/notebook/peer context cannot be
# smuggled into this otherwise strict single-KB surface.
prepared.context = {}
prepared.config = {}
prepared.metadata = {}
_sanitize_messages(prepared)
messages = (prepared.input or {}).get("messages")
if not isinstance(messages, list) or not messages:
raise HTTPException(status_code=422, detail="知识库研究仅接受非空文本问题")
latest = messages[-1]
latest_content = (
latest.content if isinstance(latest, HumanMessage) else latest.get("content")
)
if not _content_text(latest_content).strip():
raise HTTPException(status_code=422, detail="知识库研究不支持纯附件提问")
policy = {
**generation_preferences,
"llmwiki_knowledge_base_ids": [session["knowledge_base_id"]],
"assistant_knowledge_base_ids": [],
"knowledge_retrieval_mode": "classic",
"knowledge_space_strict": True,
"rag_mode_enabled": True,
# Retrieval is injected by the Gateway; do not let the graph open a
# second, client-configurable retrieval path.
"knowledge_rag_enabled": False,
"agent_id": None,
"agent_name": None,
"writing_mode": False,
"writing_artifact_path": None,
"is_plan_mode": False,
"subagent_enabled": False,
"is_bootstrap": False,
"is_writing_setup": False,
"is_report_structure_setup": False,
"memory_recall_disabled": True,
"memory_injection_enabled": False,
"prompt_prefix_enabled": False,
"disable_tools": True,
"excluded_tools": ["memory", "hindsight_recall"],
"skill_stop_names": None,
"skill_stop_message": None,
"knowledge_research_session_id": session["id"],
"knowledge_research_knowledge_base_id": session["knowledge_base_id"],
}
prepared.assistant_id = None
prepared.excluded_tools = None
prepared.skill_stop_names = None
prepared.skill_stop_message = None
prepared.context.update(policy)
# Pass exactly one LangGraph runtime options container. build_run_config
# gives ``context`` precedence when both are present; emitting both here
# would discard configurable.thread_id and detach the checkpoint.
prepared.config = {
"configurable": {
"thread_id": thread_id,
**policy,
}
}
prepared.command = None
prepared.checkpoint_id = None
prepared.checkpoint = None
prepared.webhook = None
prepared.metadata.update(
{
"system": True,
"thread_type": "knowledge_research",
"knowledge_research_session_id": session["id"],
"knowledge_research_knowledge_base_id": session["knowledge_base_id"],
"llmwiki_knowledge_base_ids": [session["knowledge_base_id"]],
"assistant_knowledge_base_ids": [],
"knowledge_retrieval_mode": "classic",
"knowledge_space_strict": True,
"rag_mode_enabled": True,
"rag_mode": True,
}
)
marker = {
"id": session["id"],
"mode": "knowledge_research",
"knowledge_base_id": session["knowledge_base_id"],
"user_id": session["user_id"],
}
return prepared, marker
def inject_validated_knowledge_research_context(
config: dict[str, Any],
context: dict[str, Any] | None,
) -> None:
"""Write non-public research policy into both final runtime scopes."""
if not context or not context.get("knowledge_research_session_id"):
return
authoritative = {
"disable_tools": True,
"memory_recall_disabled": True,
"memory_injection_enabled": False,
"prompt_prefix_enabled": False,
"writing_mode": False,
"is_plan_mode": False,
"subagent_enabled": False,
}
for scope in ("configurable", "context"):
target = config.setdefault(scope, {})
if isinstance(target, dict):
target.update(authoritative)
__all__ = [
"inject_validated_knowledge_research_context",
"prepare_knowledge_research_run",
]
offline-backend-20260512/backend/app/gateway/llmwiki_index_scheduler.py
+18 -86
View File
@@ -17,7 +17,6 @@ _MUTATION_SYNC_DELAYS_SECONDS = (1.0, 8.0, 30.0, 90.0, 300.0)
_MUTATION_SYNC_MAX_SECONDS = 10 * 60.0
_MUTATION_SYNC_CONFIRM_GRACE_SECONDS = 30.0
_FULL_SCAN_FALLBACK_SECONDS = 10 * 60.0
_FOCUS_RECONCILE_SECONDS = 5 * 60.0
_WIKI_CONFIG_UPGRADE_RETRY_SECONDS = 5 * 60.0
@@ -200,6 +199,13 @@ async def _upgrade_wiki_generation_configs_once(
return {"eligible": len(targets), "upgraded": upgraded, "failed": failed}
def _has_directory_constraints(mapping: dict[str, Any]) -> bool:
return bool(
mapping.get("wiki_directory_paths")
or mapping.get("wiki_parent_directories")
)
def _knowledge_base_is_processing(remote: dict[str, Any]) -> bool:
"""Normalize processing markers across old/new WeKnora response shapes."""
@@ -727,71 +733,6 @@ async def _refresh_local_wiki_config(app: FastAPI):
return await rebuild_local_wiki_index_runtime(app)
async def _reconcile_focus_constraints_once(app: FastAPI) -> None:
"""Resume strict focus filtering after process restarts."""
mapping_store = getattr(app.state, "llmwiki_store", None)
list_visible = getattr(mapping_store, "list_visible", None)
if not callable(list_visible):
return
mappings = [
row
for row in await list_visible(
"system",
scope="all",
is_admin=True,
)
if _has_focus_constraint(row)
]
if not mappings:
return
service = getattr(app.state, "llmwiki_sync_service", None)
client = getattr(service, "client", None)
if client is None:
from deerflow.integrations.weknora.runtime import (
build_weknora_client,
get_resolved_llmwiki_runtime,
)
runtime = get_resolved_llmwiki_runtime()
if not runtime.weknora_enabled:
return
client = build_weknora_client(runtime)
from app.gateway.wiki_directory_constraints import (
reconcile_fixed_parent_directories,
)
semaphore = asyncio.Semaphore(4)
async def run(mapping: dict[str, Any]) -> None:
async with semaphore:
try:
remote = await client.get_knowledge_base(
str(mapping.get("weknora_id") or "")
)
if not isinstance(remote, dict):
return
processing = await _remote_generation_is_processing(
client,
mapping,
remote=remote,
)
if processing is not False:
return
await reconcile_fixed_parent_directories(client, mapping)
except asyncio.CancelledError:
raise
except Exception:
logger.warning(
"Periodic Wiki focus reconciliation failed mapping_id=%s",
mapping.get("id"),
exc_info=True,
)
await asyncio.gather(*(run(mapping) for mapping in mappings))
def _auto_sync_ready(app: FastAPI, config: Any) -> bool:
return bool(
config.enabled
@@ -915,7 +856,6 @@ async def local_wiki_index_scheduler_loop(app: FastAPI) -> None:
last_sync_started_at: float | None = None
last_synced_runtime_signature: str | None = None
last_full_scan_started_at: float | None = None
last_focus_reconcile_started_at: float | None = None
last_prompt_schema_upgrade_started_at: float | None = None
while True:
try:
@@ -924,30 +864,22 @@ async def local_wiki_index_scheduler_loop(app: FastAPI) -> None:
# cadence. Provider/URL changes therefore take effect promptly.
config = await _refresh_local_wiki_config(app)
now = monotonic()
if (
last_prompt_schema_upgrade_started_at is None
or now - last_prompt_schema_upgrade_started_at
>= _WIKI_CONFIG_UPGRADE_RETRY_SECONDS
):
last_prompt_schema_upgrade_started_at = now
await _upgrade_wiki_generation_configs_once(app)
auto_sync_ready = _auto_sync_ready(app, config)
if (
not auto_sync_ready
and (
last_focus_reconcile_started_at is None
or now - last_focus_reconcile_started_at
>= _FOCUS_RECONCILE_SECONDS
)
):
last_focus_reconcile_started_at = now
await _reconcile_focus_constraints_once(app)
if not auto_sync_ready:
# A later disabled -> enabled transition is a new first valid
# configuration and should receive an immediate scan.
# With automatic sync off, the scheduler must remain local-only:
# no periodic WeKnora page scan, prompt rewrite, or directory
# reconciliation. Explicit user/API calls remain responsible
# for remote refreshes.
last_sync_started_at = None
last_synced_runtime_signature = None
else:
if (
last_prompt_schema_upgrade_started_at is None
or now - last_prompt_schema_upgrade_started_at
>= _WIKI_CONFIG_UPGRADE_RETRY_SECONDS
):
last_prompt_schema_upgrade_started_at = now
await _upgrade_wiki_generation_configs_once(app)
interval = float(config.sync_interval_seconds)
runtime_signature = getattr(
app.state,
offline-backend-20260512/backend/app/gateway/llmwiki_rag.py
+6 -1
View File
@@ -2092,7 +2092,12 @@ async def build_llmwiki_smart_reasoning_context(
)
agent_id = _assistant_agent_id(body)
if agent_id and agent_id not in {"default", "lead_agent"}:
# Built-in research agents use the caller-selected scope, like general chat.
# Each mapping still passes get_authorized below; custom agents retain their
# configured binding restriction.
if agent_id and agent_id not in {
"default", "lead_agent", "deep-research-planner", "deep-research-collector",
}:
allowed_ids = set(_agent_bound_ids(body))
if any(mapping_id not in allowed_ids for mapping_id in selected_ids):
raise HTTPException(
offline-backend-20260512/backend/app/gateway/perf_logging.py
+125
View File
@@ -0,0 +1,125 @@
"""Low-overhead performance traces for diagnosing slow Gateway requests."""
from __future__ import annotations
import asyncio
import faulthandler
import logging
import os
import signal
import sys
import time
from typing import Any
logger = logging.getLogger(__name__)
_VERBOSE_PATHS = {
"/api/v1/auth/options",
"/api/agents",
"/api/llmwiki/knowledge-bases",
}
_STACK_DUMP_REGISTERED = False
def slow_request_threshold_ms() -> float:
"""Return the warning threshold, defaulting to half a second."""
try:
return max(0.0, min(float(os.getenv("DEERFLOW_SLOW_REQUEST_LOG_MS", "500")), 60_000.0))
except (TypeError, ValueError):
return 500.0
def verbose_request_logging() -> bool:
return os.getenv("DEERFLOW_PERF_DIAGNOSTICS", "").strip().lower() in {"1", "true", "yes", "on"}
def is_verbose_path(path: str) -> bool:
return path in _VERBOSE_PATHS or (
path.startswith("/api/llmwiki/agents/")
and path.endswith("/knowledge-bases")
)
def should_log_request(
*,
elapsed_ms: float,
headers_ms: float | None,
first_body_ms: float | None,
is_stream: bool,
) -> bool:
threshold = slow_request_threshold_ms()
if is_stream:
return bool(
(headers_ms is not None and headers_ms >= threshold)
or (first_body_ms is not None and first_body_ms >= threshold)
)
return elapsed_ms >= threshold
def trace_request(request: Any, endpoint: str) -> dict[str, Any]:
"""Start an endpoint trace attached to the ASGI request state."""
trace: dict[str, Any] = {
"endpoint": endpoint,
"started": time.perf_counter(),
"stages": {},
"details": {},
}
try:
request.state.perf_trace = trace
except AttributeError:
# Direct router calls in isolated tests may use a minimal request stub.
pass
return trace
def trace_stage(trace: dict[str, Any], name: str, started: float) -> float:
"""Record one endpoint stage and return its completion timestamp."""
ended = time.perf_counter()
trace["stages"][name] = round((ended - started) * 1000, 2)
return ended
def finish_trace(trace: dict[str, Any]) -> None:
trace["endpoint_ms"] = round((time.perf_counter() - trace["started"]) * 1000, 2)
def register_stack_dump_signal() -> None:
"""Allow SIGUSR1 to dump all Python thread stacks to container stderr."""
global _STACK_DUMP_REGISTERED
if _STACK_DUMP_REGISTERED:
return
if os.getenv("DEERFLOW_ENABLE_STACK_DUMP", "1").strip().lower() in {"0", "false", "no", "off"}:
logger.info("PERF Python stack dump signal disabled by configuration")
return
stack_signal = getattr(signal, "SIGUSR1", None)
register = getattr(faulthandler, "register", None)
if stack_signal is None or register is None:
logger.info("PERF Python stack dump signal is unavailable on this platform")
return
try:
register(stack_signal, file=sys.stderr, all_threads=True, chain=False)
except (OSError, RuntimeError, ValueError):
logger.exception("PERF unable to register Python stack dump signal")
return
_STACK_DUMP_REGISTERED = True
logger.info("PERF Python stack dump ready signal=SIGUSR1 all_threads=true")
async def event_loop_watchdog(app: Any) -> None:
"""Log substantial event-loop stalls without requiring the diagnostics UI."""
loop = asyncio.get_running_loop()
last_warning = 0.0
while True:
expected = loop.time() + 1.0
await asyncio.sleep(1.0)
lag_ms = round(max(0.0, loop.time() - expected) * 1000, 2)
app.state.perf_event_loop_lag_ms = lag_ms
now = time.monotonic()
if lag_ms >= 200 and now - last_warning >= 5:
logger.warning("PERF event_loop_stall lag_ms=%.2f threshold_ms=200", lag_ms)
last_warning = now
offline-backend-20260512/backend/app/gateway/roundtable_inprocess_gateway.py
+87 -93
View File
@@ -18,12 +18,9 @@ import asyncio
import logging
import os
import uuid
from collections.abc import Callable
from datetime import UTC, datetime
from types import SimpleNamespace
from typing import Any
from langgraph.checkpoint.base import empty_checkpoint
from weakref import WeakKeyDictionary
from app.gateway.roundtable_model_fallback import (
fallback_model_chain,
@@ -39,6 +36,7 @@ from app.gateway.routers._roundtable_seed import (
SUMMARY_AGENT_ID,
ensure_roundtable_functional_agents,
)
from app.gateway.routers.multi_agent import _roundtable_file_stream_context
from app.gateway.services import RUN_SCOPE_BACKGROUND_CHILD
from deerflow.agents.roundtable_orchestrator.delivery import classify_seat_delivery, visible_seat_text
from deerflow.agents.roundtable_orchestrator.report_json import extract_report_json
@@ -51,8 +49,11 @@ from deerflow.agents.roundtable_orchestrator.types import (
SummaryResult,
)
from deerflow.config.paths import get_paths
from deerflow.config.runtime_contention_config import get_runtime_contention_config
from deerflow.persistence.roundtable_diagnostics import DiagnosticsRecorder, get_default_store
from deerflow.runtime.fair_queue import FairAsyncGate
from deerflow.runtime.serialization import serialize_channel_values
from deerflow.utils.file_io import file_io
from deerflow.utils.tool_visibility import is_internal_tool_name
logger = logging.getLogger(__name__)
@@ -86,49 +87,30 @@ _SUMMARY_FILE = "方案总结报告.md"
# 这里漏禁了 web_search，导致后台席位 / 报告能联网搜索而前端不能，现由 policy 统一。
# ── 后台作业「按用户限流」（2026-06-16）─────────────────────────────────────────
# 痛点：进 2+ 个后台挂起后，整个网关变慢——建立智能体连接、等待模型响应都慢。
# 根因：① checkpointer 用单连接 aiosqlite（AsyncSqliteSaver，内部一把 asyncio.Lock 串
# 行全进程的 checkpoint 读写）；② 后台作业无并发上限——每个作业进程内驱动真实 leader/
# seat/report run，DAG/recommend 的 stage 还用 asyncio.gather 把席位真并行铺开。多个后台
# 作业同时把几十个 agent run 压到同一把 SQLite 锁 + 同一个离线模型端点上，前台实时会商
# 的 checkpoint 读写 / 模型调用只能排队 → 「干啥都慢」。
#
# 解法：给**每个用户**一个并发额度（默认 2），后台每个 agent 轮次（leader/seat/report/
# summary/dashboard）真正开跑前先取一个额度，跑完释放。同一用户的所有后台作业共享这 2 个
# 额度——随便挂多少个作业都能立刻建好，但同一瞬间真正在打模型/写 checkpoint 的后台轮次
# 至多 2 个，不会把共享资源打满。**前台实时路径（multi_agent._leader_run/_special_run、
# 普通聊天）不经过本文件的 `_run_agent_turn`，永不取这个额度，故前台不受限。**
_BACKGROUND_SLOTS_PER_USER = max(1, int(os.getenv("ROUNDTABLE_BACKGROUND_SLOTS_PER_USER", "2") or "2"))
# 按 user_id 缓存 Semaphore。网关全程跑在同一个事件循环上，普通 dict 取/建即可（无跨线程）。
_user_background_sems: dict[str, asyncio.Semaphore] = {}
# Global and per-user budgets share round-robin admission. A busy user's
# backlog cannot fill the model/checkpoint path before another user is served.
def _user_background_semaphore(user_id: str) -> asyncio.Semaphore:
"""取该用户的后台并发额度信号量（懒建，每用户一把）。"""
sem = _user_background_sems.get(user_id)
if sem is None:
sem = asyncio.Semaphore(_BACKGROUND_SLOTS_PER_USER)
_user_background_sems[user_id] = sem
return sem
_background_gates: WeakKeyDictionary = WeakKeyDictionary()
def _background_gate() -> FairAsyncGate:
loop = asyncio.get_running_loop()
if loop not in _background_gates:
limits = get_runtime_contention_config()
per_user = max(1, int(os.getenv("ROUNDTABLE_BACKGROUND_SLOTS_PER_USER", str(limits.background_per_user))))
_background_gates[loop] = FairAsyncGate(limits.background_concurrency, per_user, limits.queue_limit)
return _background_gates[loop]
# ── 进程内底层操作 ───────────────────────────────────────────────────────────
async def _create_thread(app) -> str:
"""同进程建一个带圆桌 metadata 的 thread + 空 checkpoint（镜像 _create_thread_direct）。"""
checkpointer = app.state.checkpointer
"""Create metadata only; the first graph run initializes its checkpoint."""
thread_store = app.state.thread_store
thread_id = str(uuid.uuid4())
await thread_store.create(thread_id, assistant_id=None, metadata=dict(_ROUNDTABLE_THREAD_METADATA))
config = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
await checkpointer.aput(
config,
empty_checkpoint(),
{"step": -1, "source": "input", "writes": None, "parents": {}, "created_at": datetime.now(UTC).isoformat()},
{},
)
return thread_id
@@ -345,6 +327,8 @@ async def _run_agent_turn(
skill_stop_names: list[str],
thinking_enabled: bool,
reasoning_effort: str,
allowed_tools: list[str] | None = None,
recursion_limit: int = 1000,
subagent_enabled: bool = False,
diag: DiagnosticsRecorder | None = None,
stage: str = "step2_seat",
@@ -356,9 +340,8 @@ async def _run_agent_turn(
镜像 multi_agent ``_run_payload`` 的 body 形状：agent 通过 ``context.agent_name`` 选，
assistant_id 固定 lead_agent；关闭 memory recall / subagent / plan mode（临时研讨任务）。
并发受**该用户的后台额度**约束（``_user_background_semaphore``）：真正开跑（start_run +
await run task）前先取额度，跑完释放。同一用户的所有后台作业共享这几个额度，避免 2+
后台挂起把单连接 SQLite checkpointer / 离线模型打满、拖垮前台实时会商。
全局与每用户后台额度由 ``_background_gate`` 统一按用户轮转分配。排队不占运行名额，
真正开跑前取额度；取消时等待实际 run 停止后才释放，避免拖垮前台实时会商。
"""
from app.gateway.routers.thread_runs import RunCreateRequest
from app.gateway.services import start_run
@@ -367,7 +350,7 @@ async def _run_agent_turn(
assistant_id="lead_agent",
input={"messages": [{"type": "human", "content": [{"type": "text", "text": message}], "additional_kwargs": {}}]},
metadata={"run_scope": RUN_SCOPE_BACKGROUND_CHILD},
config={"recursion_limit": 1000},
config={"recursion_limit": recursion_limit},
context={
"agent_name": agent_name,
"model_name": model or "",
@@ -382,6 +365,8 @@ async def _run_agent_turn(
# 与 multi_agent._run_payload 对齐:关 builtin 记忆注入,避免单例总控/席位
# 被上轮会话 memory.json 带偏(臆造席位 / 直接干活)。详见该处注释。
"memory_injection_enabled": False,
"allowed_tools": allowed_tools,
**_roundtable_file_stream_context(),
},
excluded_tools=excluded_tools,
skill_stop_names=skill_stop_names,
@@ -393,7 +378,7 @@ async def _run_agent_turn(
request = SimpleNamespace(app=app, state=SimpleNamespace(user=None, auth=None))
# 取该用户的后台额度后再开跑：start_run + await task 是最重的部分（建 agent、调模型、
# 落 checkpoint）。额度满时本轮在此 await 排队，等同用户某个在跑轮次释放额度后再开。
async with _user_background_semaphore(user_id):
async with _background_gate().slot(user_id):
try:
record = await start_run( # type: ignore[arg-type]
body,
@@ -410,16 +395,23 @@ async def _run_agent_turn(
agent_name=agent_name, cycle=cycle,
)
raise
if on_started is not None:
await on_started(record)
# idle 超时守护：慢但持续产出的生成一直等；只有连续 _SEAT_IDLE_TIMEOUT_S 秒**毫无输出**
# （真卡死）才取消并抛错。绝不因「这轮总耗时长」而打断（除非显式配了 _SEAT_MAX_TOTAL_S）。
await _await_run_with_idle_guard(
app, record,
idle_timeout=_SEAT_IDLE_TIMEOUT_S,
max_total=_SEAT_MAX_TOTAL_S,
diag=diag, stage=stage, agent_name=agent_name, cycle=cycle,
)
try:
if on_started is not None:
await on_started(record)
# 持续产出的慢生成继续等，只有连续无输出或显式总时限才中断。
await _await_run_with_idle_guard(
app, record,
idle_timeout=_SEAT_IDLE_TIMEOUT_S,
max_total=_SEAT_MAX_TOTAL_S,
diag=diag, stage=stage, agent_name=agent_name, cycle=cycle,
)
finally:
# Cancelling the parent or its idle watcher is not proof that the
# child has stopped writing checkpoints. Keep admission until ack.
task = getattr(record, "task", None)
if task is not None and not task.done():
task.cancel()
await asyncio.gather(task, return_exceptions=True)
return await _read_thread_messages(app, thread_id)
@@ -568,36 +560,6 @@ def _exception_reason(exc: BaseException) -> str:
return "exception"
def _seat_empty_reason(messages: list[dict[str, Any]]) -> str | None:
"""席位/取数轮校验：本轮无可见正文 → ``empty`` / ``thinking_only``（触发换模型）。"""
saw_thinking = False
for m in reversed(messages):
if not isinstance(m, dict):
continue
role = m.get("type") or m.get("role")
if role in {"human", "user"}:
if _is_broadcast_human_text(_message_text(m)):
continue
break
if role not in {"ai", "assistant"}:
continue
kind = classify_seat_delivery(_message_text(m))
if kind == "valid":
return None
if kind == "thinking_only":
saw_thinking = True
break
return "thinking_only" if saw_thinking else "empty"
def _leader_failed_reason(messages: list[dict[str, Any]]) -> str | None:
"""总控轮校验：既无正文、也无任何工具调用（派活/澄清）→ ``empty``（视为模型没干活）。"""
ai = _latest_ai_message(messages)
has_text = bool(_latest_ai_text(messages).strip())
has_tools = bool(_normalize_tool_calls((ai or {}).get("tool_calls")))
return None if (has_text or has_tools) else "empty"
# ── 网关实现 ─────────────────────────────────────────────────────────────────
@@ -670,20 +632,24 @@ class InProcessRoundtableGateway:
message: str,
requested_model: str | None,
excluded_tools: list[str],
allowed_tools: list[str] | None,
skill_stop_names: list[str],
thinking_enabled: bool,
reasoning_effort: str,
recursion_limit: int,
subagent_enabled: bool = False,
require_visible_text: bool = False,
stage: str,
cycle: int | None = None,
validate: Callable[[list[dict[str, Any]]], str | None] | None = None,
) -> tuple[list[dict[str, Any]], list[ModelSwitch], str | None]:
"""跑一轮 agent，并在「模型出问题」时自动切到下一个模型重试。
触发换模型的三种情形（与用户选择一致）：
触发换模型的两种情形：
① ``_run_agent_turn`` 抛异常（start_run 失败 / idle 卡死超时）；
② 返回了中间件兜底错误文案（``is_llm_error_message`` 命中）；
③ 调用方 ``validate`` 判定无效（如席位交付为空 / 总控既无正文又无派活）。
仅有思考、暂无可见正文或空交付不是模型故障，不能触发自动换模；只有调用方明确
``require_visible_text=True`` 的最终收口轮，在请求完整结束后无正文才视为不可交付。
候选模型链按 ``config.yaml models[]`` 顺序（``fallback_model_chain``），首个为
``requested_model``（空则作业级 ``self._model``，再空则配置默认）。返回
@@ -721,9 +687,11 @@ class InProcessRoundtableGateway:
message=message,
model=candidate,
excluded_tools=excluded_tools,
allowed_tools=allowed_tools,
skill_stop_names=skill_stop_names,
thinking_enabled=thinking_enabled,
reasoning_effort=reasoning_effort,
recursion_limit=recursion_limit,
subagent_enabled=subagent_enabled,
diag=self._diag,
stage=stage,
@@ -739,10 +707,10 @@ class InProcessRoundtableGateway:
detail={"reason": prev_reason, "tried": chain}, agent_name=agent_name, cycle=cycle,
)
return messages, switches, used_model
# 跑通了：再看是否「兜底错误文案」或 validate 判定无效。
# 普通轮只识别带明确错误标记的兜底文案；最终收口轮完整结束后必须有正文。
reason = is_llm_error_message(_latest_ai_message(messages))
if reason is None and validate is not None:
reason = validate(messages)
if reason is None and require_visible_text and not _latest_ai_text(messages).strip():
reason = "empty"
if reason is None:
return messages, switches, used_model
prev_model, prev_reason = candidate, reason
@@ -755,27 +723,46 @@ class InProcessRoundtableGateway:
return messages, switches, used_model
return messages, switches, used_model
async def run_leader(self, *, thread_ids, coordinator_name, message, model) -> LeaderTurn:
async def run_leader(
self,
*,
thread_ids,
coordinator_name,
message,
model,
finalize_only: bool = False,
) -> LeaderTurn:
msg = message
if self._first_leader and self._seat_intro:
msg = self._seat_intro + "\n\n" + message
self._first_leader = False
policy = roundtable_run_policy("leader")
if finalize_only:
msg += (
"\n\n【最终收口·纯正文】本轮所有工具均已关闭。请直接输出一份合并后的最终结论正文；"
"不要按席位逐份复述原文，也不要反复改写同一份总结。"
"不要调用或尝试 write_file、present_files、agent_orchestration、skill_view 或任何其它工具，"
"不要再派活，也不要只描述你准备做什么。"
)
messages, switches, _used = await self._run_turn_with_fallback(
thread_id=thread_ids[coordinator_name],
agent_name=coordinator_name,
message=msg,
requested_model=model,
excluded_tools=policy.excluded_tools,
skill_stop_names=policy.skill_stop_names,
allowed_tools=[] if finalize_only else getattr(policy, "allowed_tools", None),
skill_stop_names=[] if finalize_only else policy.skill_stop_names,
thinking_enabled=policy.thinking_enabled,
reasoning_effort=policy.reasoning_effort,
recursion_limit=int(getattr(policy, "recursion_limit", 1000)),
require_visible_text=finalize_only,
stage="step2_leader",
validate=_leader_failed_reason,
)
ai = _latest_ai_message(messages)
# tool_calls 取最后一条 AI 消息（派活就在这条）；正文取最后一条有文字的 AI 消息。
final_text = _latest_ai_text(messages)
if finalize_only and not final_text.strip():
raise RuntimeError("总控最终收口未产出正文")
tool_calls = (ai.get("tool_calls") if ai else None) or []
# 本轮工具调用（派活 / 澄清），带进 LeaderTurn 供草稿 run 重建步骤卡。
turn_tool_calls = _normalize_tool_calls(tool_calls)
@@ -830,6 +817,7 @@ class InProcessRoundtableGateway:
message=seat_task,
requested_model=model,
excluded_tools=policy.excluded_tools,
allowed_tools=getattr(policy, "allowed_tools", None),
skill_stop_names=policy.skill_stop_names,
thinking_enabled=(
policy.thinking_enabled if seat_thinking is None else seat_thinking
@@ -837,9 +825,9 @@ class InProcessRoundtableGateway:
reasoning_effort=(
policy.reasoning_effort if seat_reasoning is None else seat_reasoning
),
recursion_limit=int(getattr(policy, "recursion_limit", 1000)),
subagent_enabled=seat_subagent,
stage="step2_seat",
validate=_seat_empty_reason,
)
return SeatTurn(
final_text=_latest_ai_text(messages),
@@ -867,15 +855,17 @@ class InProcessRoundtableGateway:
message=message,
requested_model=current_model,
excluded_tools=policy.excluded_tools,
allowed_tools=getattr(policy, "allowed_tools", None),
skill_stop_names=policy.skill_stop_names,
thinking_enabled=policy.thinking_enabled,
reasoning_effort=policy.reasoning_effort,
recursion_limit=int(getattr(policy, "recursion_limit", 1000)),
stage="step3_report",
)
switches.extend(sw)
current_model = used or current_model
summary = _latest_ai_text(messages) or summary
html = _read_report_html(self._app, report_thread_id, self._user_id)
html = await file_io(_read_report_html, self._app, report_thread_id, self._user_id)
if _html_complete(html):
break
if attempt >= _ARTIFACT_RETRY_LIMIT:
@@ -925,15 +915,17 @@ class InProcessRoundtableGateway:
message=message,
requested_model=current_model,
excluded_tools=policy.excluded_tools,
allowed_tools=getattr(policy, "allowed_tools", None),
skill_stop_names=policy.skill_stop_names,
thinking_enabled=policy.thinking_enabled,
reasoning_effort=policy.reasoning_effort,
recursion_limit=int(getattr(policy, "recursion_limit", 1000)),
stage="step3_summary",
)
switches.extend(sw)
current_model = used or current_model
closing = _latest_ai_text(messages) or closing
md = _read_summary_md(self._app, summary_thread_id, self._user_id)
md = await file_io(_read_summary_md, self._app, summary_thread_id, self._user_id)
if md and md.strip():
break
if attempt >= _ARTIFACT_RETRY_LIMIT:
@@ -981,9 +973,11 @@ class InProcessRoundtableGateway:
message=message,
requested_model=current_model,
excluded_tools=policy.excluded_tools,
allowed_tools=getattr(policy, "allowed_tools", None),
skill_stop_names=policy.skill_stop_names,
thinking_enabled=policy.thinking_enabled,
reasoning_effort=policy.reasoning_effort,
recursion_limit=int(getattr(policy, "recursion_limit", 1000)),
stage="step3_dashboard",
)
current_model = used or current_model
offline-backend-20260512/backend/app/gateway/roundtable_job_dispatcher.py
+32
View File
@@ -30,6 +30,8 @@ from app.gateway.roundtable_job_executor import (
WORKER_ID,
params_from_snapshot,
)
from deerflow.config.system_settings import get_roundtable_room_limit_settings
from deerflow.persistence.roundtable_rooms.sql import NON_OCCUPYING_JOB_PHASES
logger = logging.getLogger(__name__)
@@ -125,6 +127,7 @@ class RoundtableJobDispatcher:
async def dispatch_once(self) -> int:
"""扫一轮可领取作业并逐个条件抢占。返回本轮成功启动的作业数。"""
now = datetime.now(UTC)
await self._expire_overdue_step2_rooms(now)
await self._reap_cancel_requested(now)
try:
candidate_ids = await self._store.list_claimable(now=now, limit=_CLAIM_BATCH)
@@ -172,6 +175,35 @@ class RoundtableJobDispatcher:
)
return started
async def _expire_overdue_step2_rooms(self, now: datetime) -> int:
"""Persist hard-timeout cancellation even when no browser is connected."""
max_minutes = getattr(
get_roundtable_room_limit_settings(),
"max_hold_minutes",
30,
)
request_cancel_expired = getattr(
self._store,
"request_cancel_expired_step2",
None,
)
if request_cancel_expired is None:
return 0
try:
expired = await request_cancel_expired(
created_before=now - timedelta(minutes=max_minutes),
non_occupying_phases=tuple(NON_OCCUPYING_JOB_PHASES),
)
except Exception: # noqa: BLE001
logger.warning("[roundtable-jobs] hard-timeout scan failed", exc_info=True)
return 0
if expired:
logger.info(
"[roundtable-jobs] requested cancellation for %d overdue Step-2 room(s)",
expired,
)
return expired
async def _reap_cancel_requested(self, now: datetime) -> int:
"""收口持租 worker 崩溃后遗留的 ``cancel_requested`` 作业。
offline-backend-20260512/backend/app/gateway/roundtable_job_executor.py
+44
View File
@@ -29,8 +29,10 @@ from deerflow.agents.roundtable_orchestrator import (
SeatRef,
run_orchestration,
)
from deerflow.config.system_settings import get_roundtable_room_limit_settings
from deerflow.persistence.roundtable_diagnostics import DiagnosticsRecorder, get_default_store
from deerflow.persistence.roundtable_drafts.sql import DraftConcurrentWriteError
from deerflow.persistence.roundtable_rooms.sql import NON_OCCUPYING_JOB_PHASES
from deerflow.runtime.user_context import reset_current_user, set_current_user
logger = logging.getLogger(__name__)
@@ -67,6 +69,26 @@ _DRAFT_WRITE_RETRIES = 3
ROUNDTABLE_BACKGROUND_REPORT_ENABLED = True
def _background_step2_room_expired(row: dict[str, Any] | None) -> bool:
"""Whether a background job exceeded the configurable Step-2 hard limit."""
if not row or str(row.get("phase") or "") in NON_OCCUPYING_JOB_PHASES:
return False
raw_created = row.get("created_at")
if isinstance(raw_created, datetime):
created_at = raw_created
elif isinstance(raw_created, str) and raw_created:
try:
created_at = datetime.fromisoformat(raw_created)
except ValueError:
return False
else:
return False
if created_at.tzinfo is None:
created_at = created_at.replace(tzinfo=UTC)
max_minutes = getattr(get_roundtable_room_limit_settings(), "max_hold_minutes", 30)
return datetime.now(UTC) >= created_at + timedelta(minutes=max_minutes)
class _JobUser:
"""最小 CurrentUser：满足 user_context 的 ``.id`` 结构协议。"""
@@ -439,6 +461,11 @@ class RoundtableJobExecutor:
self._watch_cancellation(params.job_id, main_task, params.lease_owner)
)
try:
# 全局设置在作业真正启动时读取并锁定；运行中修改只影响下一次会商，
# 避免本作业在半途因配置变化突然提前或延后收口。
max_cycles = int(
getattr(get_roundtable_room_limit_settings(), "max_cycles", 8)
)
result = await run_orchestration(
gateway=gateway,
progress=progress,
@@ -453,6 +480,7 @@ class RoundtableJobExecutor:
plan=params.orchestration_plan,
gather_first=params.gather_first,
business_code=params.business_code,
max_cycles=max_cycles,
)
finally:
watcher.cancel()
@@ -559,6 +587,22 @@ class RoundtableJobExecutor:
except Exception: # noqa: BLE001 — DB 读失败不中断看门狗，下轮重试
continue
status = (row.get("status") or "") if row is not None else ""
if status in {"queued", "running", "awaiting_input"} and _background_step2_room_expired(row):
logger.info(
"roundtable job %s reached the Step-2 room hard limit; requesting cancellation",
job_id,
)
try:
requested = await self._store.request_cancel(job_id, user_id=None)
if requested is not None:
row = requested
status = str(requested.get("status") or "")
except Exception: # noqa: BLE001
logger.warning(
"roundtable job %s hard-limit cancellation failed",
job_id,
exc_info=True,
)
if status == "cancel_requested":
logger.info(
"roundtable job %s: detected cancel_requested in DB, finalizing "
offline-backend-20260512/backend/app/gateway/roundtable_model_fallback.py
+6 -4
View File
@@ -1,9 +1,11 @@
"""圆桌会商「模型自动容错（换下一个模型）」共享助手。
会商的第二步（leader / seat）与第三步（summary / report / dashboard / structure）单轮执行时，
若所用模型出现问题（抛异常 / idle 卡死 / 中间件兜底错误文案 / 交付为空），应自动切到
若所用模型出现真实调用问题（抛异常 / idle 卡死 / 中间件兜底错误文案），应自动切到
**下一个可用模型**继续，并把「『X』模型异常 → 已切换『Y』模型」告知用户。
仅有思考、暂时没有可见正文或交付为空不属于模型故障，不得据此切换模型。
本模块只放**与执行路径无关**的纯逻辑，供前台（``multi_agent.py``）与后台
（``roundtable_inprocess_gateway.py``）两条路径共用：
@@ -55,8 +57,8 @@ def reason_text(reason: str | None) -> str:
def model_fallback_enabled() -> bool:
"""模型自动容错总开关（默认开；设 ``ROUNDTABLE_MODEL_FALLBACK=0/false/off`` 关闭）。
关闭时 ``fallback_model_chain`` 只返回单个候选（原模型），即恢复旧行为：模型失败就失败，
不再尝试其它模型。
该开关只控制真实模型错误后的换模。仅有 reasoning、没有可见正文不再作为换模条件。
关闭时 ``fallback_model_chain`` 只返回单个候选（原模型）。
"""
raw = (os.getenv("ROUNDTABLE_MODEL_FALLBACK", "1") or "1").strip().lower()
return raw not in {"0", "false", "no", "off"}
@@ -85,7 +87,7 @@ def fallback_model_chain(current: str | None) -> list[str]:
- ``current`` 为空/None → 从 ``models[0]`` 起（即后端默认模型），后接其余；
- ``current`` 在配置内 → 它排第一，其余按配置顺序补齐；
- ``current`` 不在配置内（自定义/未知名）→ 仍排第一，再补齐全部配置模型；
- 总开关关闭 → 只返回单个候选（``current`` 或默认模型），等同旧行为；
- 总开关关闭 → 只返回单个候选（``current`` 或默认模型）；
- 拿不到任何配置模型 → 返回 ``[current]``（current 为空则空列表，调用方自行退化为不换模型）。
"""
all_names = _config_model_names()
offline-backend-20260512/backend/app/gateway/roundtable_room_gate.py
+432
View File
@@ -0,0 +1,432 @@
"""Admission helper shared by step-2 init and background job start.
The durable database lease is the source of truth. Gateway run inspection is
kept as a reconciliation fallback for legacy/untracked runs. When the limit is
enabled, missing room storage fails closed so a deployment error cannot bypass
the cap.
"""
from __future__ import annotations
from typing import Any
from fastapi import HTTPException, Request
from sqlalchemy import select
from deerflow.config.system_settings import get_roundtable_room_limit_settings
from deerflow.persistence.roundtable_jobs.sql import compute_active_dedupe_key
from deerflow.runtime.user_context import get_effective_user_id
def _room_key_from_payload(meta: Any, owner_user_id: str | None = None) -> str | None:
"""Resolve the live-execution key from thread/run metadata."""
if not isinstance(meta, dict):
return None
stamped = str(meta.get("roundtable_room_key") or "").strip()
if stamped:
return stamped
draft_id = str(meta.get("draft_id") or "").strip() or None
if not draft_id:
return None
task_id = str(meta.get("task_id") or meta.get("taskId") or "").strip() or None
return compute_active_dedupe_key(draft_id, owner_user_id, task_id)
def _room_scope_key_from_payload(
meta: Any,
owner_user_id: str | None = None,
) -> str | None:
"""Legacy persisted scope used only to de-duplicate fallback run evidence."""
if not isinstance(meta, dict):
return None
draft_id = str(meta.get("draft_id") or "").strip() or None
if not draft_id:
return None
task_id = str(meta.get("task_id") or meta.get("taskId") or "").strip() or None
owner = owner_user_id or str(meta.get("user_id") or "").strip() or None
return compute_active_dedupe_key(draft_id, owner, task_id)
def _thread_id_of(record: Any) -> str:
return str(getattr(record, "thread_id", "") or "").strip()
async def _all_active_run_records(request: Request) -> list[Any]:
"""Every in-flight Gateway run, regardless of which account started it."""
records: list[Any] = []
seen_ids: set[str] = set()
def _append(items: list[Any] | None) -> None:
for item in items or []:
task = getattr(item, "task", None)
if task is not None and callable(getattr(task, "done", None)) and task.done():
# A terminal asyncio task with a lagging pending/running status
# is cleanup residue, not a live room. This also prevents one
# exceptional worker-finalization path from pinning capacity.
continue
run_id = str(getattr(item, "run_id", "") or "")
thread_id = _thread_id_of(item)
marker = run_id or thread_id
if not marker or marker in seen_ids:
continue
seen_ids.add(marker)
records.append(item)
gate = getattr(request.app.state, "concurrency_gate", None)
if gate is not None and getattr(gate, "enabled", False):
try:
leases = await gate.list_active()
except Exception:
leases = None
if leases:
_append(list(leases))
run_mgr = getattr(request.app.state, "run_manager", None)
if run_mgr is not None:
try:
_append(await run_mgr.list_active())
except Exception:
pass
return records
async def _room_keys_from_thread_rows(
thread_ids: list[str],
) -> tuple[dict[str, str | None], set[str]]:
"""Load room keys for *all* owners. Returns (keys, unresolved thread ids)."""
remaining = {tid for tid in thread_ids if tid}
keys: dict[str, str | None] = {}
if not remaining:
return keys, remaining
try:
from deerflow.persistence.engine import get_session_factory
from deerflow.persistence.thread_meta.model import ThreadMetaRow
session_factory = get_session_factory()
if session_factory is None:
return keys, remaining
async with session_factory() as session:
rows = (
await session.execute(
select(
ThreadMetaRow.thread_id,
ThreadMetaRow.user_id,
ThreadMetaRow.metadata_json,
).where(ThreadMetaRow.thread_id.in_(list(remaining)))
)
).all()
for thread_id, user_id, metadata_json in rows:
remaining.discard(thread_id)
key = _room_key_from_payload(metadata_json, user_id)
if key:
keys[key] = _room_scope_key_from_payload(metadata_json, user_id)
except Exception:
return keys, remaining
return keys, remaining
async def extra_live_roundtable_keys(request: Request) -> dict[str, str | None]:
"""Room keys of step-2 sessions whose LangGraph runs are still in flight.
Scans the whole Gateway process: every user's runs are included. Several
seat runs in one session share ``roundtable_room_key``. Intent and
recommend threads have no such key and are ignored.
"""
records = await _all_active_run_records(request)
keys: dict[str, str | None] = {}
missing_tids: list[str] = []
seen_threads: set[str] = set()
for rec in records:
tid = _thread_id_of(rec)
if not tid or tid in seen_threads:
continue
seen_threads.add(tid)
meta = getattr(rec, "metadata", None)
owner = str(getattr(rec, "user_id", "") or "").strip() or None
if owner is None and isinstance(meta, dict):
owner = str(meta.get("user_id") or "").strip() or None
key = _room_key_from_payload(meta if isinstance(meta, dict) else None, owner)
if key:
keys[key] = _room_scope_key_from_payload(meta, owner)
continue
missing_tids.append(tid)
if not missing_tids:
return keys
from_rows, remaining = await _room_keys_from_thread_rows(missing_tids)
keys.update(from_rows)
thread_store = getattr(request.app.state, "thread_store", None)
if thread_store is None or not remaining:
return keys
for tid in remaining:
try:
row = await thread_store.get(tid, user_id=None)
except Exception:
continue
if not isinstance(row, dict):
continue
owner = str(row.get("user_id") or "").strip() or None
key = _room_key_from_payload(row.get("metadata"), owner)
if key:
keys[key] = _room_scope_key_from_payload(row.get("metadata"), owner)
return keys
async def backend_occupancy_keys(request: Request) -> dict[str, str | None]:
"""Live Gateway runs used to reconcile legacy leases, all accounts."""
return await extra_live_roundtable_keys(request)
def current_user_id(request: Request) -> str:
user = getattr(request.state, "user", None)
if user is not None and getattr(user, "id", None):
return str(user.id)
return get_effective_user_id()
def _store(request: Request):
return getattr(request.app.state, "roundtable_room_store", None)
async def admit_roundtable_room(
request: Request,
*,
draft_id: str | None,
task_id: str | None,
room_session_id: str | None = None,
) -> dict[str, Any]:
settings = get_roundtable_room_limit_settings()
store = _store(request)
extra_keys = await backend_occupancy_keys(request)
clean_room_session_id = (room_session_id or "").strip() or None
if settings.enabled and clean_room_session_id is None:
return {
"ok": False,
"enabled": True,
"max_rooms": settings.max_rooms,
"active_count": len(extra_keys),
"renewed": False,
"code": "ROOM_SESSION_REQUIRED",
"message": "会商会话标识缺失，请刷新页面后重试。",
}
if store is None:
return {
"ok": not settings.enabled,
"enabled": settings.enabled,
"max_rooms": settings.max_rooms,
"active_count": len(extra_keys),
"renewed": False,
"code": "" if not settings.enabled else "ROOM_STORE_UNAVAILABLE",
"message": "" if not settings.enabled else "会商并发服务暂不可用，请稍后再试。",
}
user_id = current_user_id(request)
clean_draft = (draft_id or "").strip() or None
clean_task = (task_id or "").strip() or None
return await store.admit(
draft_id=clean_draft,
user_id=user_id,
task_id=clean_task,
room_session_id=clean_room_session_id,
enabled=settings.enabled,
max_rooms=settings.max_rooms,
max_hold_minutes=getattr(settings, "max_hold_minutes", 30),
extra_keys=extra_keys,
)
async def renew_roundtable_room(
request: Request,
*,
draft_id: str | None,
task_id: str | None,
room_session_id: str | None = None,
) -> dict[str, Any]:
"""Renew an existing browser lease; never admit or recreate a room."""
settings = get_roundtable_room_limit_settings()
store = _store(request)
extra_keys = await backend_occupancy_keys(request)
if store is None:
return {
"ok": not settings.enabled,
"enabled": settings.enabled,
"max_rooms": settings.max_rooms,
"active_count": len(extra_keys),
"renewed": False,
"code": "" if not settings.enabled else "ROOM_STORE_UNAVAILABLE",
"message": "" if not settings.enabled else "会商并发服务暂不可用，请稍后再试。",
}
return await store.renew(
draft_id=(draft_id or "").strip() or None,
user_id=current_user_id(request),
task_id=(task_id or "").strip() or None,
room_session_id=(room_session_id or "").strip() or None,
enabled=settings.enabled,
max_rooms=settings.max_rooms,
max_hold_minutes=getattr(settings, "max_hold_minutes", 30),
extra_keys=extra_keys,
)
async def release_roundtable_room(
request: Request,
*,
draft_id: str | None,
task_id: str | None,
room_session_id: str | None = None,
) -> bool:
"""Best-effort release used by both the API route and disconnect cleanup."""
store = _store(request)
if store is None:
return False
return await store.release(
draft_id=(draft_id or "").strip() or None,
user_id=current_user_id(request),
task_id=(task_id or "").strip() or None,
room_session_id=(room_session_id or "").strip() or None,
)
async def enforce_roundtable_room(
request: Request,
*,
draft_id: str | None,
task_id: str | None,
room_session_id: str | None = None,
) -> dict[str, Any]:
"""Admit the room or raise 409 before any step-2 model work starts."""
decision = await admit_roundtable_room(
request,
draft_id=draft_id,
task_id=task_id,
room_session_id=room_session_id,
)
if not decision.get("ok", False):
raise HTTPException(status_code=409, detail=decision)
return decision
async def enforce_existing_roundtable_room(
request: Request,
*,
draft_id: str | None,
task_id: str | None,
room_session_id: str | None = None,
) -> dict[str, Any]:
"""Require and renew an existing lease without performing admission."""
decision = await renew_roundtable_room(
request,
draft_id=draft_id,
task_id=task_id,
room_session_id=room_session_id,
)
if not decision.get("ok", False):
raise HTTPException(status_code=409, detail=decision)
return decision
async def _thread_room_scope(
request: Request,
thread_ids: list[str],
) -> tuple[str, str | None, str | None, str | None] | None:
"""Resolve ``(room_key, draft_id, owner_user_id, task_id)`` from metadata."""
wanted = [tid for tid in dict.fromkeys(thread_ids) if tid]
if not wanted:
return None
thread_store = getattr(request.app.state, "thread_store", None)
if thread_store is not None:
for tid in wanted:
try:
row = await thread_store.get(tid, user_id=None)
except Exception:
continue
if not isinstance(row, dict):
continue
meta = row.get("metadata")
owner = str(row.get("user_id") or "").strip() or None
key = _room_key_from_payload(meta, owner)
if key:
data = meta if isinstance(meta, dict) else {}
draft_id = str(data.get("draft_id") or "").strip() or None
task_id = str(data.get("task_id") or data.get("taskId") or "").strip() or None
return key, draft_id, owner, task_id
try:
from deerflow.persistence.engine import get_session_factory
from deerflow.persistence.thread_meta.model import ThreadMetaRow
session_factory = get_session_factory()
if session_factory is None:
return None
async with session_factory() as session:
rows = (
await session.execute(
select(
ThreadMetaRow.user_id,
ThreadMetaRow.metadata_json,
).where(ThreadMetaRow.thread_id.in_(wanted))
)
).all()
for owner, meta in rows:
key = _room_key_from_payload(meta, owner)
if key:
data = meta if isinstance(meta, dict) else {}
draft_id = str(data.get("draft_id") or "").strip() or None
task_id = str(data.get("task_id") or data.get("taskId") or "").strip() or None
return key, draft_id, owner, task_id
except Exception:
return None
return None
async def enforce_roundtable_run(
request: Request,
*,
draft_id: str | None,
task_id: str | None,
room_session_id: str | None = None,
thread_ids: list[str],
) -> dict[str, Any] | None:
"""Enforce the room cap at the final model-producing entry point.
Current clients send the draft scope on every run. A model run may only
renew the room that was admitted when Step 2 began; it must never create a
new room while the user resumes, answers a clarification, or intervenes.
Thread metadata is a compatibility fallback for initialized sessions.
"""
clean_draft = (draft_id or "").strip() or None
clean_task = (task_id or "").strip() or None
if clean_draft:
return await enforce_existing_roundtable_room(
request,
draft_id=clean_draft,
task_id=clean_task,
room_session_id=(room_session_id or "").strip() or None,
)
scope = await _thread_room_scope(request, thread_ids)
if scope is None:
return None
room_key, _stored_draft, _owner_user_id, _stored_task = scope
settings = get_roundtable_room_limit_settings()
store = _store(request)
if store is None:
if settings.enabled:
raise HTTPException(
status_code=503,
detail="Roundtable room store not available",
)
return None
decision = await store.renew_room_key(
room_key=room_key,
holder_user_id=current_user_id(request),
enabled=settings.enabled,
max_rooms=settings.max_rooms,
max_hold_minutes=getattr(settings, "max_hold_minutes", 30),
extra_keys=await backend_occupancy_keys(request),
)
if not decision.get("ok", False):
raise HTTPException(status_code=409, detail=decision)
return decision
offline-backend-20260512/backend/app/gateway/roundtable_run_policy.py
+32 -9
View File
@@ -10,12 +10,13 @@
角色（``role``）：
- ``leader`` —— 总控派活：短决策，关 thinking + low reasoning（首字节快）；
**只保留 ``agent_orchestration`` / ``ask_clarification`` / ``present_files``**，
**只保留 ``agent_orchestration`` / ``ask_clarification``**，
其余「干活类」工具（``write_file`` / ``bash`` / ``str_replace`` / ``read_file`` /
``ls`` / ``view_image`` / ``task`` / ``write_todos`` / ``memory`` / ``hindsight_*``）
一律禁用——否则弱模型（如 deepseek-chat）会概率性地**自己写文件、跑 bash 干活**
而不是派活（已在生产日志里观测到总控 ``wc -l outputs/*.md`` 自检自己写的报告）。
总控的唯一职责是「派活 / 澄清 / 收口呈现」，没有任何理由动文件系统或执行命令。
总控的唯一职责是「派活 / 澄清 / 文字收口」，没有任何理由动文件系统、呈现文件
或执行命令。最终收口轮会进一步使用空白名单，物理禁用全部工具。
并配合 ``SkillStopMiddleware`` 在 ``agent_orchestration`` 调用后停图，便于网关从
AIMessage.tool_calls 解析派活列表。
- ``seat`` —— 子智能体交付：开 thinking + medium reasoning；**可写文件**
@@ -61,11 +62,14 @@ from dataclasses import dataclass
# 注意：``seat`` / ``report`` 都包含 ``web_search`` —— 圆桌研讨里联网搜索一律禁用
# （与前端 ``_special_run`` 的 excluded_tools 对齐）。
#
# ``leader`` 用「白名单」思路反推黑名单：总控**只允许** agent_orchestration（派活）、
# ask_clarification（澄清）、present_files（收口呈现子智能体产出的文件）三件事，
# 因此把其余所有工具全部禁掉。关键是 write_file / bash / str_replace —— 留着它们，
# 弱模型会绕过派活直接自己写报告（生产日志已实锤）。memory / hindsight_* 一并禁，
# 配合 ``memory_injection_enabled=False`` 让单例总控不被上轮会话的记忆带偏席位。
# ``leader`` 必须使用真正的白名单。历史上的「白名单思路反推黑名单」会在新增工具时
# 自动泄漏能力，曾导致总控循环调用 page_report_template / glob / grep / skill_manage 等
# 一百余次。``_LEADER_EXCLUDED`` 继续保留为兼容和纵深防御，但安全边界由
# ``_LEADER_ALLOWED`` 决定。
_LEADER_ALLOWED: list[str] = [
"agent_orchestration",
"ask_clarification",
]
_LEADER_EXCLUDED: list[str] = [
"web_search",
"view_image",
@@ -81,6 +85,7 @@ _LEADER_EXCLUDED: list[str] = [
"hindsight_reflect",
"hindsight_retain",
"read_peer_delivery", # 总控靠 _collect_delivery_status 直读各席位,不需要这个席位侧工具
"present_files",
]
# seat：席位**可以写文件**(write_file / str_replace)产出 md 等成品文档,**bash 放开**
# (2026-06 决策,业务需要 bash 跑知识库检索 / Python 辅助分析)。禁派活 / 澄清 /
@@ -175,14 +180,16 @@ _LEADER_SKILL_STOP: list[str] = ["agent_orchestration"]
class RoundtableRunPolicy:
"""一个角色单轮 run 的归一化参数。
与 ``multi_agent._run_payload`` 的入参一一对应：``excluded_tools`` /
``skill_stop_names`` / ``thinking_enabled`` / ``reasoning_effort``。
与 ``multi_agent._run_payload`` 的入参一一对应。``allowed_tools`` 为
``None`` 时沿用普通工具解析；空列表表示物理禁用全部普通工具。
"""
excluded_tools: list[str]
allowed_tools: list[str] | None
skill_stop_names: list[str]
thinking_enabled: bool
reasoning_effort: str
recursion_limit: int
def roundtable_run_policy(role: str) -> RoundtableRunPolicy:
@@ -194,59 +201,75 @@ def roundtable_run_policy(role: str) -> RoundtableRunPolicy:
if role == "leader":
return RoundtableRunPolicy(
excluded_tools=list(_LEADER_EXCLUDED),
allowed_tools=list(_LEADER_ALLOWED),
skill_stop_names=list(_LEADER_SKILL_STOP),
thinking_enabled=False,
reasoning_effort="low",
recursion_limit=1000,
)
if role == "seat":
return RoundtableRunPolicy(
excluded_tools=list(_SEAT_EXCLUDED),
allowed_tools=None,
skill_stop_names=[],
thinking_enabled=True,
reasoning_effort="medium",
recursion_limit=1000,
)
if role == "seat_parallel":
# 同 seat（thinking + medium），但额外禁写文件 / 跑命令（防并行沙箱打架）。
return RoundtableRunPolicy(
excluded_tools=list(_SEAT_PARALLEL_EXCLUDED),
allowed_tools=None,
skill_stop_names=[],
thinking_enabled=True,
reasoning_effort="medium",
recursion_limit=1000,
)
if role == "report":
return RoundtableRunPolicy(
excluded_tools=list(_REPORT_EXCLUDED),
allowed_tools=None,
skill_stop_names=[],
thinking_enabled=True,
reasoning_effort="medium",
recursion_limit=1000,
)
if role == "summary":
return RoundtableRunPolicy(
excluded_tools=list(_SUMMARY_EXCLUDED),
allowed_tools=None,
skill_stop_names=[],
thinking_enabled=True,
reasoning_effort="medium",
recursion_limit=1000,
)
if role == "action_plan":
return RoundtableRunPolicy(
excluded_tools=list(_ACTION_PLAN_EXCLUDED),
allowed_tools=None,
skill_stop_names=[],
thinking_enabled=True,
reasoning_effort="medium",
recursion_limit=1000,
)
if role == "dashboard":
return RoundtableRunPolicy(
excluded_tools=list(_DASHBOARD_EXCLUDED),
allowed_tools=None,
skill_stop_names=[],
thinking_enabled=True,
reasoning_effort="medium",
recursion_limit=1000,
)
if role == "ingest":
return RoundtableRunPolicy(
excluded_tools=list(_INGEST_EXCLUDED),
allowed_tools=None,
skill_stop_names=[],
thinking_enabled=True,
reasoning_effort="medium",
recursion_limit=1000,
)
raise ValueError(
f"unknown roundtable role: {role!r} (expected leader/seat/seat_parallel/report/summary/action_plan/dashboard/ingest)"
offline-backend-20260512/backend/app/gateway/routers/_ai_writing_seed_assets/ai-writing-researcher/config.yaml
+4 -3
View File
@@ -1,7 +1,8 @@
description: AI 写作流程的素材收集专家,负责为写作任务生成检索关键词、从全部启用技能中自由编排检索、并对检索回来的素材按相关度排序筛选
description: AI 写作流程的素材收集专家,负责为写作任务生成检索关键词、用配置的技能编排检索、并对检索回来的素材按相关度排序筛选
id: ai-writing-researcher
name: 素材收集专家
# 通用检索不再使用这里的 skills 白名单：素材收集从全部启用技能里由模型按
# 写作意图自由挑选（内置「知识库检索」始终保底）。此字段仅保留兼容旧版。
# 通用检索使用的技能白名单（按顺序编排）。技能目录动辄数百个，全量列进
# 选择提示词会把模型上下文撑爆，因此候选池限定在这里配置的技能；内置
# 「知识库检索」始终保底。admin 可在智能体管理页替换或追加。
skills:
- knowledge-base-search
offline-backend-20260512/backend/app/gateway/routers/_strategy_selection_seed_assets/strategy-selection-judge/SOUL.md
+3 -1
View File
@@ -10,6 +10,8 @@
4. 按统一量表独立评分，再给出横向比较和推荐组合。
5. 只使用输入中的 `agentId`，不得改名或虚构不存在的候选项。
本阶段没有用户中途回答入口。不得调用 `ask_clarification`，也不得在正文中向用户提问、索要材料、要求确认或要求重试。若证据不足，采用保守假设、降低对应维度评分，并将缺口写入“数据限制与待核验”，仍须一次性完成所有候选项的评判。
## 百分制量表
- 任务匹配度：25
@@ -24,7 +26,7 @@
## 输出要求
先用简洁 Markdown 给出总体评价、主要差异、推荐组合和需要人工确认的冲突。随后在回复末尾输出：
先用简洁 Markdown 给出总体评价、主要差异、推荐组合和需要人工确认的冲突，不要重复复述同一结论或输出多个评判版本。随后在回复末尾输出：
````markdown
[JUDGEMENT_READY]
offline-backend-20260512/backend/app/gateway/routers/admin_active_runs.py
+16 -10
View File
@@ -42,6 +42,18 @@ _DEFAULT_LOOKBACK_HOURS = {"minute": 3, "hour": 48}
# assistant_id values that are not real custom-agent ids (the default lead agent).
_DEFAULT_ASSISTANT_IDS = {"", "default", "lead_agent", "agent"}
_DEFAULT_AGENT_NAME = "默认助手"
_UNKNOWN_USER_NAME = "未知用户"
def _display_agent_name(assistant_id: str, agent_names: dict[str, str]) -> str:
if assistant_id in _DEFAULT_ASSISTANT_IDS:
return _DEFAULT_AGENT_NAME
return agent_names.get(assistant_id) or assistant_id
def _display_user_name(user_id: str, user_emails: dict[str, str]) -> str:
return user_emails.get(user_id) or user_id or _UNKNOWN_USER_NAME
class ActiveRunItem(BaseModel):
@@ -244,10 +256,7 @@ async def list_active_runs(request: Request) -> ActiveRunsResponse:
user_id = r.user_id or (meta_row.user_id if meta_row else "") or ""
metadata = (meta_row.metadata_json if meta_row else None) or {}
assistant_id = r.assistant_id or ""
if assistant_id in _DEFAULT_ASSISTANT_IDS:
agent_name = "榛樿鍔╂墜"
else:
agent_name = agent_names.get(assistant_id) or assistant_id
agent_name = _display_agent_name(assistant_id, agent_names)
items.append(
ActiveRunItem(
run_id=r.run_id,
@@ -256,7 +265,7 @@ async def list_active_runs(request: Request) -> ActiveRunsResponse:
created_at=datetime.fromtimestamp(r.created_at, tz=UTC).isoformat() if r.created_at else "",
elapsed_seconds=_elapsed_seconds_from_epoch(r.created_at, now),
user_id=user_id,
user_name=user_emails.get(user_id) or user_id or "鏈煡鐢ㄦ埛",
user_name=_display_user_name(user_id, user_emails),
agent_id=assistant_id,
agent_name=agent_name,
model_name=r.model_name,
@@ -272,10 +281,7 @@ async def list_active_runs(request: Request) -> ActiveRunsResponse:
user_id = (meta_row.user_id if meta_row else "") or ""
metadata = (meta_row.metadata_json if meta_row else None) or r.metadata or {}
assistant_id = r.assistant_id or ""
if assistant_id in _DEFAULT_ASSISTANT_IDS:
agent_name = "默认助手"
else:
agent_name = agent_names.get(assistant_id) or assistant_id
agent_name = _display_agent_name(assistant_id, agent_names)
items.append(
ActiveRunItem(
run_id=r.run_id,
@@ -284,7 +290,7 @@ async def list_active_runs(request: Request) -> ActiveRunsResponse:
created_at=r.created_at,
elapsed_seconds=_elapsed_seconds(r.created_at, now),
user_id=user_id,
user_name=user_emails.get(user_id) or user_id or "未知用户",
user_name=_display_user_name(user_id, user_emails),
agent_id=assistant_id,
agent_name=agent_name,
model_name=str((r.metadata or {}).get("model_name") or ""),
offline-backend-20260512/backend/app/gateway/routers/admin_audit_retention.py
+74
View File
@@ -0,0 +1,74 @@
"""Admin view and pause control for historical audit-data cleanup."""
from __future__ import annotations
import asyncio
from datetime import datetime
from time import time
from typing import Any
from fastapi import APIRouter, Request
from pydantic import BaseModel
from app.gateway.admin_audit_retention_control import (
is_paused,
read_status,
set_paused,
write_status,
)
from app.gateway.routers.admin_users import _require_admin
from deerflow.config.runtime_paths import runtime_home
from deerflow.persistence.types import BEIJING_TZ
router = APIRouter(prefix="/api/admin/audit-retention", tags=["admin-audit-retention"])
class RetentionControlRequest(BaseModel):
paused: bool
def _status_payload() -> dict[str, Any]:
root = runtime_home()
value = read_status(root)
paused = is_paused(root)
value["paused"] = paused
value["pause_requested"] = paused and value.get("status") == "running"
if value["pause_requested"]:
value["display_status"] = "pausing"
elif paused:
value["display_status"] = "paused"
else:
value["display_status"] = value.get("status", "scheduled")
return value
def _apply_control(paused: bool) -> dict[str, Any]:
root = runtime_home()
set_paused(root, paused)
value = read_status(root)
if paused and value.get("status") != "running":
value.update(status="paused", current_source=None, next_run_at=None)
elif not paused and value.get("status") == "paused":
value.update(status="scheduled", next_run_at=time())
write_status(root, value)
return _status_payload()
@router.get("")
async def get_retention_status(request: Request) -> dict[str, Any]:
await _require_admin(request)
return await asyncio.to_thread(_status_payload)
@router.put("/control")
async def update_retention_control(
payload: RetentionControlRequest,
request: Request,
) -> dict[str, Any]:
await _require_admin(request)
value = await asyncio.to_thread(_apply_control, payload.paused)
value["control_updated_at"] = datetime.now(BEIJING_TZ).isoformat(timespec="seconds")
return value
__all__ = ["router"]
offline-backend-20260512/backend/app/gateway/routers/admin_users.py
+262 -164
View File
@@ -20,34 +20,37 @@ import os
import re
from collections.abc import AsyncIterator
from contextvars import ContextVar
from datetime import UTC, datetime, timedelta
from datetime import time as datetime_time
from hashlib import sha256
from time import perf_counter
from typing import Any
from datetime import UTC, datetime, timedelta, time as datetime_time
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi.responses import StreamingResponse
from langchain_core.messages import HumanMessage, SystemMessage
from pydantic import BaseModel, Field
from sqlalchemy import and_, case, desc, func, not_, or_, select, text
from langchain_core.messages import HumanMessage, SystemMessage
from app.gateway.admin_query_window import (
ADMIN_QUERY_WINDOW_DAYS,
resolve_admin_query_window,
)
from app.gateway.deps import (
get_config,
get_agent_store,
get_checkpointer,
get_config,
get_current_user_from_request,
get_thread_store,
)
from app.gateway.routers.memory import _build_v1_provider, _is_v1
from deerflow.agents.memory import BuiltinFileProvider
from deerflow.config.app_config import AppConfig
from deerflow.config.paths import get_paths
from deerflow.config.system_settings import load_system_settings
from deerflow.models import create_chat_model
from deerflow.runtime import serialize_channel_values
from app.gateway.routers.memory import _build_v1_provider, _is_v1
from deerflow.persistence.engine import get_session_factory
from deerflow.persistence.admin_stats.model import AdminLeaderboardDailyStatRow
from deerflow.persistence.engine import get_session_factory
from deerflow.persistence.llm_metrics.model import LlmCallMetricRow
from deerflow.persistence.run.model import RunRow
from deerflow.persistence.scheduled_tasks.model import ScheduledTaskRunRow
@@ -55,6 +58,7 @@ from deerflow.persistence.thread_meta.model import ThreadMetaRow
from deerflow.persistence.tool_metrics.model import ToolCallMetricRow
from deerflow.persistence.types import BEIJING_TZ
from deerflow.persistence.user.model import UserRow
from deerflow.runtime import serialize_channel_values
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/admin/users", tags=["admin-users"])
@@ -92,6 +96,13 @@ _LEADERBOARD_LOCK_WAIT_SECONDS = max(
1,
int(os.getenv("ADMIN_LEADERBOARD_LOCK_WAIT_SECONDS", "5")),
)
# The checkpoint fallback exists for installations that predate compact run
# summaries. Keep its work deliberately small: current run summaries are the
# indexed source of truth for an administrator's recent audit view.
_ADMIN_QA_CHECKPOINT_THREAD_LIMIT = 25
_ADMIN_QA_CHECKPOINT_MESSAGE_LIMIT = 100
_ADMIN_QA_CHECKPOINT_ITEM_LIMIT = 500
_ADMIN_QA_RUN_CANDIDATE_LIMIT = 1_000
_LEADERBOARD_QUEUE_ON_REQUEST_ENABLED = os.getenv(
"ADMIN_LEADERBOARD_QUEUE_ON_REQUEST_ENABLED",
os.getenv("ADMIN_LEADERBOARD_WEB_QUEUE_ENABLED", "true"),
@@ -307,20 +318,6 @@ class AdminLeaderboardResponse(BaseModel):
trends: list[AdminLeaderboardTrendPoint] = Field(default_factory=list)
class AdminUserAnalyticsResponse(BaseModel):
range_days: int
since: str
until: str
user: AdminLeaderboardUser
overview: AdminLeaderboardOverview
skills: list[AdminLeaderboardSkill] = Field(default_factory=list)
tools: list[AdminLeaderboardSkill] = Field(default_factory=list)
models: list[AdminLeaderboardDimension] = Field(default_factory=list)
agents: list[AdminLeaderboardDimension] = Field(default_factory=list)
run_statuses: list[AdminLeaderboardDimension] = Field(default_factory=list)
trends: list[AdminLeaderboardTrendPoint] = Field(default_factory=list)
class AdminLeaderboardReportResponse(BaseModel):
report: str
generated_at: str
@@ -352,7 +349,7 @@ class AdminLeaderboardSnapshotStatusResponse(BaseModel):
class AdminLeaderboardSnapshotBackfillRequest(BaseModel):
days: int = Field(default=30, ge=1, le=365)
days: int = Field(default=30, ge=1, le=ADMIN_QUERY_WINDOW_DAYS)
since: str | None = Field(default=None, description="Inclusive range start, ISO datetime")
until: str | None = Field(default=None, description="Inclusive range end, ISO datetime")
force: bool = Field(default=False, description="Queue dates even when a ready snapshot already exists")
@@ -1739,7 +1736,7 @@ async def _refresh_daily_leaderboard_snapshot(
datetime.now(BEIJING_TZ).isoformat(),
(perf_counter() - snapshot_start) * 1000,
)
except asyncio.TimeoutError:
except TimeoutError:
logger.warning("Daily leaderboard snapshot timed out: %s", job_key)
await _store_daily_snapshot(
session_factory,
@@ -1845,7 +1842,7 @@ async def _load_or_schedule_daily_leaderboard(
@router.get("/leaderboard", response_model=AdminLeaderboardResponse)
async def get_user_leaderboard(
request: Request,
days: int = Query(default=30, ge=1, le=365),
days: int = Query(default=30, ge=1, le=ADMIN_QUERY_WINDOW_DAYS),
since: str | None = Query(default=None, description="Inclusive range start, ISO datetime"),
until: str | None = Query(default=None, description="Inclusive range end, ISO datetime"),
exclude_user_ids: list[str] | None = Query(default=None, description="User ids excluded from this leaderboard aggregation"),
@@ -1861,7 +1858,11 @@ async def get_user_leaderboard(
raise HTTPException(status_code=503, detail="Analytics requires database persistence")
settings = load_system_settings().leaderboard
since_dt, until_dt, _range_days = _resolve_range(days, since, until)
since_dt, until_dt, _range_days = resolve_admin_query_window(
days=days,
since=since,
until=until,
)
return await _load_or_schedule_daily_leaderboard(
session_factory,
settings,
@@ -1878,7 +1879,7 @@ async def get_user_leaderboard(
@router.get("/leaderboard/snapshots", response_model=AdminLeaderboardSnapshotStatusResponse)
async def get_leaderboard_snapshot_status(
request: Request,
days: int = Query(default=30, ge=1, le=365),
days: int = Query(default=30, ge=1, le=ADMIN_QUERY_WINDOW_DAYS),
since: str | None = Query(default=None, description="Inclusive range start, ISO datetime"),
until: str | None = Query(default=None, description="Inclusive range end, ISO datetime"),
) -> AdminLeaderboardSnapshotStatusResponse:
@@ -1889,7 +1890,11 @@ async def get_leaderboard_snapshot_status(
raise HTTPException(status_code=503, detail="Analytics requires database persistence")
settings = load_system_settings().leaderboard
since_dt, until_dt, _range_days = _resolve_range(days, since, until)
since_dt, until_dt, _range_days = resolve_admin_query_window(
days=days,
since=since,
until=until,
)
stat_dates = _stat_dates_for_range(since_dt, until_dt)
*_scope, settings_hash = _resolve_leaderboard_snapshot_scope(settings)
rows = await _load_daily_snapshot_rows(session_factory, stat_dates, settings_hash)
@@ -1940,7 +1945,11 @@ async def queue_leaderboard_snapshot_backfill(
raise HTTPException(status_code=503, detail="Analytics requires database persistence")
settings = load_system_settings().leaderboard
since_dt, until_dt, _range_days = _resolve_range(body.days, body.since, body.until)
since_dt, until_dt, _range_days = resolve_admin_query_window(
days=body.days,
since=body.since,
until=body.until,
)
stat_dates = _stat_dates_for_range(since_dt, until_dt)
*_scope, settings_hash = _resolve_leaderboard_snapshot_scope(settings)
queued_dates = await _queue_daily_leaderboard_snapshots(
@@ -1963,7 +1972,7 @@ async def queue_leaderboard_snapshot_backfill(
@router.post("/leaderboard/refresh-today", response_model=AdminLeaderboardResponse)
async def refresh_today_leaderboard(
request: Request,
days: int = Query(default=30, ge=1, le=365),
days: int = Query(default=30, ge=1, le=ADMIN_QUERY_WINDOW_DAYS),
since: str | None = Query(default=None, description="Inclusive range start, ISO datetime"),
until: str | None = Query(default=None, description="Inclusive range end, ISO datetime"),
exclude_user_ids: list[str] | None = Query(default=None, description="User ids excluded from this leaderboard aggregation"),
@@ -1986,6 +1995,11 @@ async def refresh_today_leaderboard(
raise HTTPException(status_code=503, detail="Analytics requires database persistence")
settings = load_system_settings().leaderboard
since_dt, until_dt, _range_days = resolve_admin_query_window(
days=days,
since=since,
until=until,
)
(
excluded,
resolved_include_scheduled,
@@ -2012,7 +2026,6 @@ async def refresh_today_leaderboard(
include_failed=resolved_include_failed,
)
since_dt, until_dt, _range_days = _resolve_range(days, since, until)
return await _load_or_schedule_daily_leaderboard(
session_factory,
settings,
@@ -2079,7 +2092,7 @@ async def _build_leaderboard_report_prompt(
@router.post("/leaderboard/report", response_model=AdminLeaderboardReportResponse)
async def generate_user_leaderboard_report(
request: Request,
days: int = Query(default=30, ge=1, le=365),
days: int = Query(default=30, ge=1, le=ADMIN_QUERY_WINDOW_DAYS),
since: str | None = Query(default=None, description="Inclusive range start, ISO datetime"),
until: str | None = Query(default=None, description="Inclusive range end, ISO datetime"),
limit: int = Query(default=30, ge=10, le=100),
@@ -2093,7 +2106,11 @@ async def generate_user_leaderboard_report(
raise HTTPException(status_code=503, detail="Analytics requires database persistence")
settings = load_system_settings().leaderboard
since_dt, until_dt, _range_days = _resolve_range(days, since, until)
since_dt, until_dt, _range_days = resolve_admin_query_window(
days=days,
since=since,
until=until,
)
resolved_model_name = _resolve_report_model_name(config, model_name)
cache_key = _report_cache_key(since=since_dt, until=until_dt, limit=limit, model_name=resolved_model_name, settings=settings)
cached = _cached_report(cache_key)
@@ -2135,7 +2152,7 @@ async def generate_user_leaderboard_report(
@router.post("/leaderboard/report/stream")
async def stream_user_leaderboard_report(
request: Request,
days: int = Query(default=30, ge=1, le=365),
days: int = Query(default=30, ge=1, le=ADMIN_QUERY_WINDOW_DAYS),
since: str | None = Query(default=None, description="Inclusive range start, ISO datetime"),
until: str | None = Query(default=None, description="Inclusive range end, ISO datetime"),
limit: int = Query(default=30, ge=10, le=100),
@@ -2149,7 +2166,11 @@ async def stream_user_leaderboard_report(
raise HTTPException(status_code=503, detail="Analytics requires database persistence")
settings = load_system_settings().leaderboard
since_dt, until_dt, _range_days = _resolve_range(days, since, until)
since_dt, until_dt, _range_days = resolve_admin_query_window(
days=days,
since=since,
until=until,
)
resolved_model_name = _resolve_report_model_name(config, model_name)
cache_key = _report_cache_key(since=since_dt, until=until_dt, limit=limit, model_name=resolved_model_name, settings=settings)
@@ -2203,62 +2224,6 @@ async def stream_user_leaderboard_report(
)
@router.get("/{user_id}/analytics", response_model=AdminUserAnalyticsResponse)
async def get_user_analytics(
user_id: str,
request: Request,
days: int = Query(default=30, ge=1, le=365),
since: str | None = Query(default=None, description="Inclusive range start, ISO datetime"),
until: str | None = Query(default=None, description="Inclusive range end, ISO datetime"),
limit: int = Query(default=20, ge=5, le=100),
) -> AdminUserAnalyticsResponse:
"""Return one user's admin analytics profile."""
await _require_admin(request)
session_factory = get_session_factory()
if session_factory is None:
raise HTTPException(status_code=503, detail="Analytics requires database persistence")
settings = load_system_settings().leaderboard
common = {
"include_scheduled": settings.include_scheduled,
"include_admins": settings.include_admins,
"include_failed": settings.include_failed,
}
tool_common = {
"include_scheduled": settings.include_scheduled,
"include_admins": settings.include_admins,
}
since_dt, until_dt, range_days = _resolve_range(days, since, until)
async with session_factory() as session:
await _set_analytics_read_timeout(session)
users = await _query_user_leaderboard(session, since_dt, until_dt, user_id=user_id, **common)
if not users:
raise HTTPException(status_code=404, detail="User not found")
overview = await _query_overview(session, since_dt, until_dt, user_id=user_id, **common)
skills = await _query_call_rankings(session, since_dt, until_dt, user_id=user_id, field="skill", limit=limit, **tool_common)
tools = await _query_call_rankings(session, since_dt, until_dt, user_id=user_id, field="tool", limit=limit, **tool_common)
models = await _query_model_rankings(session, since_dt, until_dt, user_id=user_id, limit=limit, **common)
if not models:
models = await _query_run_dimensions(session, since_dt, until_dt, user_id=user_id, field="model", limit=limit, **common)
agents = await _query_run_dimensions(session, since_dt, until_dt, user_id=user_id, field="agent", limit=limit, **common)
run_statuses = await _query_run_dimensions(session, since_dt, until_dt, user_id=user_id, field="status", limit=limit, **common)
trends = await _query_trends(session, since_dt, until_dt, user_id=user_id, **common)
return AdminUserAnalyticsResponse(
range_days=range_days,
since=_iso(since_dt),
until=_iso(until_dt),
user=users[0],
overview=overview,
skills=skills,
tools=tools,
models=models,
agents=agents,
run_statuses=run_statuses,
trends=trends,
)
# ── Q&A audit list ─────────────────────────────────────────────────────────
@@ -2328,7 +2293,12 @@ def _qa_message_text(message: Any) -> str:
return _message_content_to_text(_message_value(message, "content")).strip()
async def _read_thread_checkpoint_messages(checkpointer, thread_id: str) -> list[dict]:
async def _read_thread_checkpoint_messages(
checkpointer,
thread_id: str,
*,
max_messages: int | None = None,
) -> list[dict]:
config = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
try:
checkpoint_tuple = await checkpointer.aget_tuple(config)
@@ -2341,7 +2311,9 @@ async def _read_thread_checkpoint_messages(checkpointer, thread_id: str) -> list
channel_values = checkpoint.get("channel_values", {}) if isinstance(checkpoint, dict) else {}
values = serialize_channel_values(channel_values)
messages = values.get("messages", [])
return messages if isinstance(messages, list) else []
if not isinstance(messages, list):
return []
return messages[-max_messages:] if max_messages else messages
def _qa_items_from_checkpoint_messages(
@@ -2409,47 +2381,97 @@ async def _list_thread_checkpoint_qa_items(
*,
request: Request,
user_id: str,
since: str | None = None,
until: str | None = None,
recent_only: bool = False,
) -> list[AdminUserQaItem]:
repo = get_thread_store(request)
checkpointer = get_checkpointer(request)
items: list[AdminUserQaItem] = []
offset = 0
page_size = 200
while True:
rows = await repo.search(limit=page_size, offset=offset, user_id=user_id)
if not rows:
break
for row in rows:
thread_id = str(row.get("thread_id") or "")
if not thread_id:
continue
messages = await _read_thread_checkpoint_messages(checkpointer, thread_id)
if not messages:
continue
metadata = row.get("metadata") if isinstance(row.get("metadata"), dict) else {}
context = row.get("context") if isinstance(row.get("context"), dict) else {}
assistant_id = str(
row.get("assistant_id")
or (context or {}).get("agent_id")
or (metadata or {}).get("agent_id")
or (context or {}).get("agent_name")
or (metadata or {}).get("agent_name")
or ""
)
thread_time = str(row.get("updated_at") or row.get("created_at") or "")
items.extend(
_qa_items_from_checkpoint_messages(
user_id=user_id,
thread_id=thread_id,
thread_title=str(row.get("display_name") or ""),
assistant_id=assistant_id,
thread_time=thread_time,
messages=messages,
rows: list[dict]
if recent_only:
# Checkpoints do not reliably carry a timestamp for every historical
# message. Limit the fallback to threads *created* in the effective
# audit window, so an old long-running conversation cannot leak a full
# history into a recent-only administrator query. Run summaries remain
# the complete, indexed source for recent messages in older threads.
session_factory = get_session_factory()
since_dt = _parse_optional_datetime(since)
until_dt = _parse_optional_datetime(until)
if session_factory is None or since_dt is None or until_dt is None:
return items
async with session_factory() as session:
await _set_analytics_read_timeout(session)
result = await session.execute(
select(ThreadMetaRow)
.where(
ThreadMetaRow.user_id == user_id,
ThreadMetaRow.created_at >= since_dt,
ThreadMetaRow.created_at <= until_dt,
)
.order_by(ThreadMetaRow.updated_at.desc(), ThreadMetaRow.thread_id.desc())
.limit(_ADMIN_QA_CHECKPOINT_THREAD_LIMIT)
)
if len(rows) < page_size:
break
offset += len(rows)
rows = [
{
"thread_id": row.thread_id,
"assistant_id": row.assistant_id,
"display_name": row.display_name,
"metadata": row.metadata_json or {},
"created_at": _iso(row.created_at),
"updated_at": _iso(row.updated_at),
}
for row in result.scalars()
]
else:
# Preserve the regular user's existing history behavior. The bounded
# administrator route always takes the SQL path above.
repo = get_thread_store(request)
rows = []
offset = 0
page_size = 200
while True:
page = await repo.search(limit=page_size, offset=offset, user_id=user_id)
if not page:
break
rows.extend(page)
if len(page) < page_size:
break
offset += len(page)
for row in rows:
thread_id = str(row.get("thread_id") or "")
if not thread_id:
continue
messages = await _read_thread_checkpoint_messages(
checkpointer,
thread_id,
max_messages=_ADMIN_QA_CHECKPOINT_MESSAGE_LIMIT if recent_only else None,
)
if not messages:
continue
metadata = row.get("metadata") if isinstance(row.get("metadata"), dict) else {}
context = row.get("context") if isinstance(row.get("context"), dict) else {}
assistant_id = str(
row.get("assistant_id")
or (context or {}).get("agent_id")
or (metadata or {}).get("agent_id")
or (context or {}).get("agent_name")
or (metadata or {}).get("agent_name")
or ""
)
thread_time = str(row.get("updated_at") or row.get("created_at") or "")
items.extend(
_qa_items_from_checkpoint_messages(
user_id=user_id,
thread_id=thread_id,
thread_title=str(row.get("display_name") or ""),
assistant_id=assistant_id,
thread_time=thread_time,
messages=messages,
)
)
if recent_only and len(items) >= _ADMIN_QA_CHECKPOINT_ITEM_LIMIT:
return items[:_ADMIN_QA_CHECKPOINT_ITEM_LIMIT]
return items
@@ -2459,44 +2481,61 @@ async def _list_run_summary_qa_items(
user_id: str,
since: str | None,
until: str | None,
q: str | None = None,
candidate_limit: int | None = None,
) -> list[AdminUserQaItem]:
filters = [
RunRow.user_id == user_id,
RunRow.first_human_message.is_not(None),
RunRow.first_human_message != "",
]
if since or until:
since_dt, until_dt, _range_days = _resolve_range(365, since, until)
filters.append(_run_time_filter(since_dt, until_dt))
if since:
filters.append(RunRow.created_at >= _parse_optional_datetime(since))
if until:
filters.append(RunRow.created_at <= _parse_optional_datetime(until))
needle = (q or "").strip().lower()
if needle:
like = f"%{needle}%"
filters.append(
or_(
func.lower(RunRow.run_id).like(like),
func.lower(RunRow.thread_id).like(like),
func.lower(func.coalesce(RunRow.assistant_id, "")).like(like),
func.lower(func.coalesce(RunRow.first_human_message, "")).like(like),
func.lower(func.coalesce(RunRow.last_ai_message, "")).like(like),
func.lower(func.coalesce(ThreadMetaRow.display_name, "")).like(like),
)
)
async with session_factory() as session:
await _set_analytics_read_timeout(session)
rows = (
await session.execute(
select(
RunRow.run_id,
RunRow.thread_id,
ThreadMetaRow.display_name,
RunRow.assistant_id,
RunRow.status,
RunRow.model_name,
RunRow.first_human_message,
RunRow.last_ai_message,
RunRow.total_tokens,
RunRow.created_at,
RunRow.updated_at,
)
.outerjoin(
ThreadMetaRow,
and_(
ThreadMetaRow.thread_id == RunRow.thread_id,
ThreadMetaRow.user_id == RunRow.user_id,
),
)
.where(*filters)
.order_by(RunRow.created_at.desc(), RunRow.run_id.desc())
stmt = (
select(
RunRow.run_id,
RunRow.thread_id,
ThreadMetaRow.display_name,
RunRow.assistant_id,
RunRow.status,
RunRow.model_name,
RunRow.first_human_message,
RunRow.last_ai_message,
RunRow.total_tokens,
RunRow.created_at,
RunRow.updated_at,
)
).all()
.outerjoin(
ThreadMetaRow,
and_(
ThreadMetaRow.thread_id == RunRow.thread_id,
ThreadMetaRow.user_id == RunRow.user_id,
),
)
.where(*filters)
.order_by(RunRow.created_at.desc(), RunRow.run_id.desc())
)
if candidate_limit is not None:
stmt = stmt.limit(max(1, min(int(candidate_limit), _ADMIN_QA_RUN_CANDIDATE_LIMIT)))
rows = (await session.execute(stmt)).all()
return [
AdminUserQaItem(
@@ -2561,19 +2600,39 @@ async def _list_user_qa_page(
since: str | None,
until: str | None,
include_checkpoints: bool = True,
recent_checkpoint_threads_only: bool = False,
) -> AdminUserQaPageResponse:
session_factory = get_session_factory()
if session_factory is None:
raise HTTPException(status_code=503, detail="Q&A audit requires database persistence")
candidate_limit = None
if recent_checkpoint_threads_only:
# Normal administrator pages need only a modest look-ahead beyond the
# requested page. This protects against a single high-frequency user
# materializing a whole month's question/answer bodies at once.
candidate_limit = min(
_ADMIN_QA_RUN_CANDIDATE_LIMIT,
max(200, page * page_size + page_size),
)
items = await _list_run_summary_qa_items(
session_factory=session_factory,
user_id=user_id,
since=since,
until=until,
q=q if candidate_limit is not None else None,
candidate_limit=candidate_limit,
)
if include_checkpoints:
items.extend(await _list_thread_checkpoint_qa_items(request=request, user_id=user_id))
items.extend(
await _list_thread_checkpoint_qa_items(
request=request,
user_id=user_id,
since=since,
until=until,
recent_only=recent_checkpoint_threads_only,
)
)
by_key: dict[tuple[str, str, str], AdminUserQaItem] = {}
for item in items:
@@ -2601,22 +2660,24 @@ async def _list_user_qa_page(
async def list_user_qa(
user_id: str,
request: Request,
page: int = Query(default=1, ge=1),
page: int = Query(default=1, ge=1, le=100),
page_size: int = Query(default=20, ge=1, le=100),
q: str | None = Query(default=None, description="Search question, answer, run id, or thread id"),
since: str | None = Query(default=None, description="Inclusive range start, ISO datetime"),
until: str | None = Query(default=None, description="Inclusive range end, ISO datetime"),
) -> AdminUserQaPageResponse:
"""Return one user's Q&A runs, including older checkpoint-only threads."""
"""Return one user's recent Q&A audit records."""
await _require_admin(request)
since_dt, until_dt, _range_days = resolve_admin_query_window(since=since, until=until)
return await _list_user_qa_page(
request=request,
user_id=user_id,
page=page,
page_size=page_size,
q=q,
since=since,
until=until,
since=_iso(since_dt),
until=_iso(until_dt),
recent_checkpoint_threads_only=True,
)
@@ -2659,11 +2720,41 @@ class AdminThreadItem(BaseModel):
async def list_user_threads(
user_id: str,
request: Request,
limit: int = Query(default=100, ge=1, le=500),
limit: int = Query(default=100, ge=1, le=200),
offset: int = Query(default=0, ge=0),
) -> list[AdminThreadItem]:
"""List one user's conversation threads, newest first."""
"""List one user's threads active in the rolling administrator window."""
await _require_admin(request)
cutoff, _until, _range_days = resolve_admin_query_window()
session_factory = get_session_factory()
if session_factory is not None:
async with session_factory() as session:
await _set_analytics_read_timeout(session)
rows = (
await session.execute(
select(ThreadMetaRow)
.where(
ThreadMetaRow.user_id == user_id,
ThreadMetaRow.updated_at >= cutoff,
)
.order_by(ThreadMetaRow.updated_at.desc(), ThreadMetaRow.thread_id.desc())
.limit(limit)
.offset(offset)
)
).scalars().all()
return [
AdminThreadItem(
thread_id=row.thread_id,
display_name=row.display_name or "",
status=row.status or "idle",
created_at=_iso(row.created_at),
updated_at=_iso(row.updated_at),
)
for row in rows
]
# Memory-mode fallback has no SQL date predicate. It is only used when
# database persistence is unavailable, and still preserves the policy.
repo = get_thread_store(request)
rows = await repo.search(limit=limit, offset=offset, user_id=user_id)
return [
@@ -2675,6 +2766,7 @@ async def list_user_threads(
updated_at=str(r.get("updated_at") or ""),
)
for r in rows
if _qa_sort_datetime(str(r.get("updated_at") or "")) >= cutoff
]
@@ -2683,7 +2775,7 @@ async def list_user_thread_messages(
user_id: str,
thread_id: str,
request: Request,
limit: int = Query(default=400, ge=1, le=1000),
limit: int = Query(default=200, ge=1, le=200),
) -> list[dict]:
"""Return the conversation messages of one of the user's threads.
@@ -2702,6 +2794,12 @@ async def list_user_thread_messages(
owner = thread.get("user_id")
if owner is not None and owner != user_id:
raise HTTPException(status_code=404, detail="Thread does not belong to this user")
cutoff, _until, _range_days = resolve_admin_query_window()
if _qa_sort_datetime(str(thread.get("updated_at") or thread.get("created_at") or "")) < cutoff:
raise HTTPException(
status_code=404,
detail=f"仅可查看最近 {ADMIN_QUERY_WINDOW_DAYS} 天的审计会话",
)
checkpointer = get_checkpointer(request)
config = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
offline-backend-20260512/backend/app/gateway/routers/agents.py
+544 -78
View File
File diff suppressed because it is too large. Load diff
offline-backend-20260512/backend/app/gateway/routers/app_identity.py
+170
View File
@@ -0,0 +1,170 @@
"""Application chrome name and logo.
Stored separately from ``generated_page_branding`` and the
``generated-page-branding/`` directory. An empty name or a missing logo leaves
the built-in title and static theme logos in place.
"""
from __future__ import annotations
import asyncio
from uuid import uuid4
from fastapi import APIRouter, File, Form, HTTPException, Request, UploadFile
from fastapi.responses import FileResponse
from pydantic import BaseModel
from app.gateway.routers.system_settings import _require_admin
from deerflow.config.runtime_paths import runtime_home
from deerflow.config.system_settings import (
AppIdentitySettings,
app_identity_logo_path,
load_system_settings,
save_system_settings,
)
router = APIRouter(prefix="/api/system-settings", tags=["system-settings"])
public_router = APIRouter(prefix="/api/public/app-identity", tags=["app-identity"])
_MAX_LOGO_BYTES = 10 * 1024 * 1024
_PUBLIC_LOGO_PATH = "/api/public/app-identity/logo"
class AppIdentityResponse(BaseModel):
system_name: str = ""
logo_url: str | None = None
logo_version: str | None = None
has_logo: bool = False
def _detect_logo(data: bytes) -> tuple[str, str]:
if data.startswith(b"\x89PNG\r\n\x1a\n"):
return ".png", "image/png"
if data.startswith(b"\xff\xd8\xff"):
return ".jpg", "image/jpeg"
if len(data) >= 12 and data[:4] == b"RIFF" and data[8:12] == b"WEBP":
return ".webp", "image/webp"
raise HTTPException(status_code=422, detail="Logo 仅支持 PNG、JPG 或 WebP 图片")
def _normalize_system_name(value: str | None) -> str:
normalized = " ".join(str(value or "").split())
if len(normalized) > 80:
raise HTTPException(status_code=422, detail="系统名称不能超过 80 个字符")
return normalized
def _response(settings: AppIdentitySettings) -> AppIdentityResponse:
path = app_identity_logo_path(settings)
version = settings.logo_version if path is not None else None
return AppIdentityResponse(
system_name=settings.system_name,
logo_url=f"{_PUBLIC_LOGO_PATH}?v={version}" if version else None,
logo_version=version,
has_logo=path is not None,
)
def _write_logo(data: bytes, suffix: str) -> str:
directory = runtime_home() / "app-identity"
directory.mkdir(parents=True, exist_ok=True)
filename = f"logo-{uuid4().hex}{suffix}"
destination = directory / filename
temporary = directory / f".{filename}.tmp"
try:
temporary.write_bytes(data)
temporary.replace(destination)
finally:
temporary.unlink(missing_ok=True)
return filename
@router.get("/app-identity", response_model=AppIdentityResponse)
async def get_app_identity_settings() -> AppIdentityResponse:
"""Return the application name and uploaded logo. Any logged-in user may read it."""
return _response(load_system_settings().app_identity)
@router.put("/app-identity", response_model=AppIdentityResponse)
async def update_app_identity_settings(
request: Request,
system_name: str = Form(default=""),
remove_logo: bool = Form(default=False),
logo: UploadFile | None = File(default=None),
) -> AppIdentityResponse:
"""Save the application chrome name and optional logo. Admin only."""
await _require_admin(request)
if logo is not None and remove_logo:
raise HTTPException(status_code=422, detail="上传 Logo 时不能同时恢复静态 Logo")
current = load_system_settings()
previous_logo = app_identity_logo_path(current.app_identity)
update: dict[str, object] = {"system_name": _normalize_system_name(system_name)}
new_logo_path = None
if logo is not None:
data = await logo.read(_MAX_LOGO_BYTES + 1)
await logo.close()
if not data:
raise HTTPException(status_code=422, detail="Logo 文件不能为空")
if len(data) > _MAX_LOGO_BYTES:
raise HTTPException(status_code=413, detail="Logo 文件不能超过 10MB")
suffix, media_type = _detect_logo(data)
filename = await asyncio.to_thread(_write_logo, data, suffix)
new_logo_path = runtime_home() / "app-identity" / filename
update.update(
{
"logo_filename": filename,
"logo_media_type": media_type,
"logo_version": uuid4().hex,
}
)
elif remove_logo:
update.update(
{
"logo_filename": None,
"logo_media_type": None,
"logo_version": None,
}
)
updated = current.app_identity.model_copy(update=update)
try:
saved = save_system_settings(current.model_copy(update={"app_identity": updated}))
except Exception:
if new_logo_path is not None:
await asyncio.to_thread(new_logo_path.unlink, missing_ok=True)
raise
if previous_logo is not None and previous_logo != app_identity_logo_path(saved.app_identity):
await asyncio.to_thread(previous_logo.unlink, missing_ok=True)
return _response(saved.app_identity)
@router.delete("/app-identity", response_model=AppIdentityResponse)
async def delete_app_identity_settings(request: Request) -> AppIdentityResponse:
"""Clear the application name and uploaded logo. Admin only."""
await _require_admin(request)
current = load_system_settings()
previous_logo = app_identity_logo_path(current.app_identity)
saved = save_system_settings(current.model_copy(update={"app_identity": AppIdentitySettings()}))
if previous_logo is not None:
await asyncio.to_thread(previous_logo.unlink, missing_ok=True)
return _response(saved.app_identity)
@public_router.get("/logo")
async def get_app_identity_logo() -> FileResponse:
"""Serve the uploaded application logo. Missing logos stay on the static assets."""
settings = load_system_settings().app_identity
path = app_identity_logo_path(settings)
if path is None:
raise HTTPException(status_code=404, detail="未上传系统 Logo")
return FileResponse(
path,
media_type=settings.logo_media_type or "image/png",
content_disposition_type="inline",
headers={"Cache-Control": "public, max-age=31536000, immutable"},
)
offline-backend-20260512/backend/app/gateway/routers/artifacts.py
+29 -10
View File
@@ -13,10 +13,13 @@ from fastapi.responses import FileResponse, PlainTextResponse, Response
from pydantic import BaseModel, Field
from app.gateway.authz import require_permission
from app.gateway.operation_timing import OperationTiming
from app.gateway.path_utils import aresolve_thread_virtual_path
from app.gateway.utils import sanitize_log_param
from deerflow.config.paths import get_paths
from deerflow.runtime.thread_paths import aresolve_path_user_id
from deerflow.runtime.user_context import get_effective_user_id
from deerflow.utils.file_io import file_io
logger = logging.getLogger(__name__)
@@ -535,7 +538,7 @@ async def _scan_artifact_library(
user_id=path_user_id,
)
scan_errors: list[str] = []
files = await asyncio.to_thread(
files = await file_io(
collect_outputs_listing,
outputs_dir,
errors=scan_errors,
@@ -851,7 +854,7 @@ async def download_strategy_project_artifact(project_id: str, request: Request)
description="List all files under the thread's outputs directory (/mnt/user-data/outputs/). Returns an empty list if the thread has no outputs.",
)
@require_permission("threads", "read", owner_check=True)
async def list_artifacts(thread_id: str, request: Request) -> dict:
async def list_artifacts(thread_id: str, request: Request, response: Response = None) -> dict: # type: ignore[assignment]
"""列出 thread 在 ``outputs/`` 下生成的产物文件。
圆桌 Step 3（``HighFidelityReport``）对每个**子席位** thread_id 调用本接口，
@@ -865,8 +868,24 @@ async def list_artifacts(thread_id: str, request: Request) -> dict:
返回 ``{ "thread_id", "files": [{ path, name, size_bytes, mime_type, modified_at }] }``，
目录不存在时 ``files`` 为空数组（非 404）。
"""
outputs_dir = get_paths().sandbox_outputs_dir(thread_id, user_id=await aresolve_path_user_id(thread_id))
return {"thread_id": thread_id, "files": collect_outputs_listing(outputs_dir)}
with OperationTiming(
response,
logger,
"thread_artifacts",
thread_id=sanitize_log_param(thread_id),
) as timing:
with timing.phase("owner_resolve"):
user_id = await aresolve_path_user_id(thread_id)
outputs_dir = get_paths().sandbox_outputs_dir(thread_id, user_id=user_id)
# Probe outside the shared artifact-files pool so a background library
# scan cannot stall an empty hello-chat listing for several seconds.
with timing.phase("dir_probe"):
exists = await asyncio.to_thread(lambda: outputs_dir.exists() and outputs_dir.is_dir())
if not exists:
return {"thread_id": thread_id, "files": []}
with timing.phase("dir_scan"):
files = await file_io(collect_outputs_listing, outputs_dir)
return {"thread_id": thread_id, "files": files}
def _read_bounded_text_prefix(path: Path, max_chars: int) -> tuple[str, bool]:
@@ -922,7 +941,7 @@ async def preview_artifact(
raise HTTPException(status_code=404, detail=f"Artifact not found: {path}")
if not actual_path.is_file():
raise HTTPException(status_code=400, detail=f"Path is not a file: {path}")
if not await asyncio.to_thread(is_text_file_by_content, actual_path):
if not await file_io(is_text_file_by_content, actual_path):
raise HTTPException(status_code=415, detail="Bounded preview supports text content only")
from deerflow.uploads.inspection import inspect_text_file
@@ -936,7 +955,7 @@ async def preview_artifact(
total_bytes = actual_path.stat().st_size
if total_bytes >= threshold:
report = await asyncio.to_thread(
report = await file_io(
inspect_text_file,
actual_path,
max_result_chars=effective_limit,
@@ -953,7 +972,7 @@ async def preview_artifact(
warnings=[str(item) for item in report.get("warnings") or []],
)
content, truncated = await asyncio.to_thread(_read_bounded_text_prefix, actual_path, effective_limit)
content, truncated = await file_io(_read_bounded_text_prefix, actual_path, effective_limit)
warnings = ["预览达到字符上限，未读取的内容不会进入页面或模型上下文。"] if truncated else []
return ArtifactPreviewResponse(
path=path,
@@ -1023,7 +1042,7 @@ async def get_artifact(thread_id: str, path: str, request: Request, download: bo
raise HTTPException(status_code=400, detail=f"Path is not a file: {skill_file_path}")
# Extract the file from the .skill archive
content = _extract_file_from_skill_archive(actual_skill_path, internal_path)
content = await file_io(_extract_file_from_skill_archive, actual_skill_path, internal_path)
if content is None:
raise HTTPException(status_code=404, detail=f"File '{internal_path}' not found in skill archive")
@@ -1077,7 +1096,7 @@ async def get_artifact(thread_id: str, path: str, request: Request, download: bo
headers={"Content-Disposition": _build_content_disposition("inline", actual_path.name)},
)
if is_text_file_by_content(actual_path):
if await file_io(is_text_file_by_content, actual_path):
return FileResponse(
path=actual_path,
media_type=mime_type or "text/plain",
@@ -1129,5 +1148,5 @@ async def update_artifact(
if suffix not in EDITABLE_TEXT_SUFFIXES and not (mime_type and mime_type.startswith("text/")):
raise HTTPException(status_code=400, detail="Only text artifacts can be edited.")
actual_path.write_text(body.content, encoding="utf-8")
await file_io(actual_path.write_text, body.content, encoding="utf-8")
return PlainTextResponse(content=body.content, media_type=mime_type or "text/plain")
offline-backend-20260512/backend/app/gateway/routers/auth.py
+393 -259
View File
@@ -1,5 +1,7 @@
"""Authentication endpoints."""
import asyncio
import json
import logging
import os
import time
@@ -9,8 +11,9 @@ from typing import Literal
import httpx
from fastapi import APIRouter, Depends, HTTPException, Query, Request, Response, status
from fastapi.security import OAuth2PasswordRequestForm
from pydantic import BaseModel, ConfigDict, EmailStr, Field, TypeAdapter, field_validator, model_validator
from pydantic import BaseModel, ConfigDict, EmailStr, Field, TypeAdapter, ValidationError, field_validator, model_validator
from app.gateway.admin_query_window import recent_admin_query_cutoff
from app.gateway.auth import (
UserResponse,
create_access_token,
@@ -18,18 +21,20 @@ from app.gateway.auth import (
from app.gateway.auth.config import get_auth_config
from app.gateway.auth.errors import AuthErrorCode, AuthErrorResponse
from app.gateway.auth.models import User
from app.gateway.auth.password import has_account_password, hash_password_async, is_password_hash, verify_account_password
from app.gateway.auth.repositories.base import CredentialConflictError
from app.gateway.auth.password import has_account_password, hash_password_async, is_password_hash, verify_account_password
from app.gateway.auth.repositories.base import CredentialConflictError
from app.gateway.csrf_middleware import is_secure_request
from app.gateway.deps import get_agent_store, get_current_user_from_request, get_local_provider, get_run_store, get_skill_store
from app.gateway.perf_logging import finish_trace, trace_request, trace_stage
from deerflow.config import get_app_config
from deerflow.config.system_settings import AuthenticationSettings, load_system_settings
from deerflow.config.system_settings import AuthenticationSettings, load_system_settings
from deerflow.persistence.agents.base import AgentStore
from deerflow.persistence.skills.base import SkillStore
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/v1/auth", tags=["auth"])
_USER_NOT_LOOKED_UP = object()
# ── Request/Response Models ──────────────────────────────────────────────
@@ -108,34 +113,34 @@ def _validate_strong_password(value: str) -> str:
``@field_validator(field_name)`` without inheritance gymnastics.
"""
if _password_is_common(value):
raise ValueError("密码过于常见，请设置更安全的密码")
if value != value.strip():
raise ValueError("密码首尾不能包含空格")
raise ValueError("密码过于常见，请设置更安全的密码")
if value != value.strip():
raise ValueError("密码首尾不能包含空格")
return value
class RegisterRequest(BaseModel):
"""Request model for user registration."""
username: str | None = Field(default=None, min_length=1, max_length=128)
email: EmailStr | None = None
password: str = Field(..., min_length=8, max_length=255)
model_config = ConfigDict(extra="forbid")
username: str | None = Field(default=None, min_length=1, max_length=128)
email: EmailStr | None = None
password: str = Field(..., min_length=8, max_length=255)
model_config = ConfigDict(extra="forbid")
_strong_password = field_validator("password")(classmethod(lambda cls, v: _validate_strong_password(v)))
@model_validator(mode="after")
def validate_identity(self):
if bool(self.username) == bool(self.email):
raise ValueError("请填写用户名或邮箱（只需一项）")
if self.username is not None:
value = self.username.strip()
if "@" not in value and (not value or value[0] in "._-" or value[-1] in "._-" or any(not (ch.isalnum() or ch in "._-") for ch in value)):
raise ValueError("用户名仅支持字母、数字、点、下划线和短横线，首尾须为字母或数字")
self.username = value
if "@" in value:
TypeAdapter(EmailStr).validate_python(value)
return self
@model_validator(mode="after")
def validate_identity(self):
if bool(self.username) == bool(self.email):
raise ValueError("请填写用户名或邮箱（只需一项）")
if self.username is not None:
value = self.username.strip()
if "@" not in value and (not value or value[0] in "._-" or value[-1] in "._-" or any(not (ch.isalnum() or ch in "._-") for ch in value)):
raise ValueError("用户名仅支持字母、数字、点、下划线和短横线，首尾须为字母或数字")
self.username = value
if "@" in value:
TypeAdapter(EmailStr).validate_python(value)
return self
class UsernameLoginRequest(BaseModel):
@@ -179,8 +184,8 @@ class UsernameLoginResponse(BaseModel):
class ChangePasswordRequest(BaseModel):
"""Request model for password change (also handles setup flow)."""
current_password: str = Field(default="", max_length=255)
new_password: str = Field(..., min_length=8, max_length=255)
current_password: str = Field(default="", max_length=255)
new_password: str = Field(..., min_length=8, max_length=255)
new_email: EmailStr | None = None
_strong_password = field_validator("new_password")(classmethod(lambda cls, v: _validate_strong_password(v)))
@@ -190,12 +195,12 @@ class MessageResponse(BaseModel):
"""Generic message response."""
message: str
class PasswordChangeResponse(MessageResponse):
access_token: str
token_type: str = "bearer"
expires_in: int
class PasswordChangeResponse(MessageResponse):
access_token: str
token_type: str = "bearer"
expires_in: int
# ── Helpers ───────────────────────────────────────────────────────────────
@@ -237,28 +242,71 @@ def _username_to_email(username: str) -> str:
return f"{local}@{domain}"
async def _sync_user_position(request: Request, user_id: str) -> None:
"""Best-effort: materialize the user's 岗位 (position) grants on login.
# 后台岗位同步任务的强引用集合：防止 create_task 的任务被 GC 中途回收。
_position_sync_tasks: set = set()
_POSITION_SYNC_CACHE_SECONDS = 60.0
_MAX_POSITION_SYNC_CACHE_USERS = 10_000
_position_sync_completed_at: dict[str, float] = {}
_position_sync_inflight: dict[str, asyncio.Task[None]] = {}
def _sync_user_position(request: Request, user_id: str) -> None:
"""Best-effort: schedule the user's 岗位 (position) sync off the login path.
Ensures a user who has never been synced — e.g. a brand-new account that
inherits the ``is_default`` position — gets the default agents / tasks /
prompts loaded into their "我的" tables. Failures never block login.
prompts loaded into their "我的" tables. The sync is idempotent but touches
a dozen tables per user, so it runs as a fire-and-forget task instead of
blocking the login response; the materialized rows land a moment later and
failures are logged, never surfaced.
"""
normalized_user_id = str(user_id)
now = time.monotonic()
completed_at = _position_sync_completed_at.get(normalized_user_id)
if completed_at is not None and now - completed_at < _POSITION_SYNC_CACHE_SECONDS:
return
if normalized_user_id in _position_sync_inflight:
return
sync = getattr(request.app.state, "position_sync", None)
if sync is None:
return
async def _run() -> None:
try:
await sync.sync_user(normalized_user_id)
_position_sync_completed_at[normalized_user_id] = time.monotonic()
if len(_position_sync_completed_at) > _MAX_POSITION_SYNC_CACHE_USERS:
oldest = sorted(_position_sync_completed_at, key=_position_sync_completed_at.get)
for stale_user_id in oldest[: len(oldest) // 2]:
_position_sync_completed_at.pop(stale_user_id, None)
except Exception:
logger.debug("position sync on login failed for user %s", user_id, exc_info=True)
finally:
_position_sync_inflight.pop(normalized_user_id, None)
try:
sync = getattr(request.app.state, "position_sync", None)
if sync is not None:
await sync.sync_user(str(user_id))
except Exception:
logger.debug("position sync on login failed for user %s", user_id, exc_info=True)
task = asyncio.create_task(_run())
except RuntimeError:
# 防御：请求处理器内必有运行中的事件循环，这里理论上走不到。
logger.debug("position sync scheduling failed for user %s", user_id, exc_info=True)
return
_position_sync_tasks.add(task)
_position_sync_inflight[normalized_user_id] = task
task.add_done_callback(_position_sync_tasks.discard)
async def _get_or_create_user_by_username(username: str):
async def _get_or_create_user_by_username(
username: str,
*,
existing_user: User | None | object = _USER_NOT_LOOKED_UP,
):
"""Resolve a username to a local user, auto-registering when absent.
Shared by ``/login/username`` and ``/login/token`` — both turn an
externally-supplied username into a local account and sign it in. Returns
``(user, created)`` where ``created`` is True only when this call inserted
the row. Roles are persisted authorization data, never inferred from names.
the row. Roles are persisted authorization data, never inferred from names.
"""
provider = get_local_provider()
email = _username_to_email(username)
@@ -270,7 +318,12 @@ async def _get_or_create_user_by_username(username: str):
# also exists at today's canonical domain. Choosing the exact canonical
# row first made the same username alternate backend user_ids after a
# domain migration, which in turn made its existing threads inaccessible.
if local_part is not None:
if existing_user is not _USER_NOT_LOOKED_UP:
# /login/username already performed this lookup to validate the
# account-specific password. Reuse both hits and misses instead of
# issuing the same MySQL query twice during a login burst.
user = existing_user
elif local_part is not None:
user = await provider.find_user_by_email_local_part(local_part)
if user is not None and user.email != email:
logger.info("Matched username %r to stable legacy account %s", username, user.email)
@@ -281,7 +334,7 @@ async def _get_or_create_user_by_username(username: str):
try:
# 黑名单模式:从未登录过的新用户(此处即首次自动注册)默认放行(approved=True)。
# 管理员可在白名单管理页对个别用户「取消放行」来封禁——仅当登录白名单开关打开时生效。
user = await provider.create_user(email=email, password=None, system_role="user", needs_setup=False, approved=True)
user = await provider.create_user(email=email, password=None, system_role="user", needs_setup=False, approved=True)
created = True
logger.info("Auto-registered passwordless user: %s", email)
except ValueError:
@@ -290,7 +343,7 @@ async def _get_or_create_user_by_username(username: str):
if user is None and local_part is not None:
user = await provider.find_user_by_email_local_part(local_part)
if user is None:
raise HTTPException(status_code=status.HTTP_409_CONFLICT, detail="该用户名已存在，请使用原账号登录")
raise HTTPException(status_code=status.HTTP_409_CONFLICT, detail="该用户名已存在，请使用原账号登录")
return user, created
@@ -555,134 +608,170 @@ def _record_login_success(ip: str) -> None:
# ── Endpoints ─────────────────────────────────────────────────────────────
def _require_login_method(method: str) -> None:
settings = load_system_settings().authentication
if not getattr(settings, method):
labels = {"password_login_enabled": "账号密码登录", "registration_enabled": "自助注册", "sso_enabled": "外部系统免登"}
raise HTTPException(status_code=403, detail=f"管理员尚未开启{labels[method]}")
def _login_response(user: User, token: str, *, username: str | None = None, created: bool = False) -> UsernameLoginResponse:
return UsernameLoginResponse(
access_token=token,
expires_in=get_auth_config().token_expiry_days * 24 * 3600,
user_id=str(user.id),
email=user.email,
system_role=user.system_role,
needs_setup=user.needs_setup,
created=created,
username=username or user.email.split("@", 1)[0],
)
async def _prepare_login_session(user: User) -> User:
"""Apply the administrator's one-online-session policy after authentication."""
if not load_system_settings().authentication.single_session_enabled:
return user
try:
return await get_local_provider().start_login_session(user)
except CredentialConflictError as exc:
raise HTTPException(status_code=409, detail=str(exc)) from exc
class AuthenticationOptions(AuthenticationSettings):
needs_setup: bool = False
@router.get("/options", response_model=AuthenticationOptions)
async def authentication_options(response: Response) -> AuthenticationOptions:
"""Anonymous, non-secret capability discovery for the login page."""
response.headers["Cache-Control"] = "no-store"
return AuthenticationOptions(
**load_system_settings().authentication.model_dump(),
needs_setup=await get_local_provider().count_admin_users() == 0,
)
@router.post("/login/local", response_model=UsernameLoginResponse)
def _require_login_method(method: str) -> None:
settings = load_system_settings().authentication
if not getattr(settings, method):
labels = {"password_login_enabled": "账号密码登录", "registration_enabled": "自助注册", "sso_enabled": "外部系统免登"}
raise HTTPException(status_code=403, detail=f"管理员尚未开启{labels[method]}")
def _login_response(user: User, token: str, *, username: str | None = None, created: bool = False) -> UsernameLoginResponse:
return UsernameLoginResponse(
access_token=token,
expires_in=get_auth_config().token_expiry_days * 24 * 3600,
user_id=str(user.id),
email=user.email,
system_role=user.system_role,
needs_setup=user.needs_setup,
created=created,
username=username or user.email.split("@", 1)[0],
)
async def _prepare_login_session(user: User) -> User:
"""Apply the administrator's one-online-session policy after authentication."""
if not load_system_settings().authentication.single_session_enabled:
return user
try:
return await get_local_provider().start_login_session(user)
except CredentialConflictError as exc:
raise HTTPException(status_code=409, detail=str(exc)) from exc
class AuthenticationOptions(AuthenticationSettings):
needs_setup: bool = False
@router.get("/options", response_model=AuthenticationOptions)
async def authentication_options(request: Request, response: Response) -> AuthenticationOptions:
"""Anonymous, non-secret capability discovery for the login page."""
response.headers["Cache-Control"] = "no-store"
trace = trace_request(request, "auth_options")
try:
stage_started = time.perf_counter()
authentication = load_system_settings().authentication.model_dump()
trace_stage(trace, "load_system_settings", stage_started)
stage_started = time.perf_counter()
needs_setup = await get_local_provider().count_admin_users() == 0
trace_stage(trace, "count_admin_users", stage_started)
return AuthenticationOptions(**authentication, needs_setup=needs_setup)
finally:
finish_trace(trace)
@router.post("/login/local", response_model=UsernameLoginResponse)
async def login_local(
request: Request,
response: Response,
form_data: OAuth2PasswordRequestForm = Depends(),
):
"""Username or email login with the account's unified password/passphrase."""
_require_login_method("password_login_enabled")
"""Username or email login with the account's unified password/passphrase."""
_require_login_method("password_login_enabled")
client_ip = _get_client_ip(request)
_check_rate_limit(client_ip)
user = await _lookup_user_by_username(form_data.username)
if user is not None and not await verify_account_password(form_data.password, user):
user = None
user = await _lookup_user_by_username(form_data.username)
if user is not None and not await verify_account_password(form_data.password, user):
user = None
if user is None:
_record_login_failure(client_ip)
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail=AuthErrorResponse(code=AuthErrorCode.INVALID_CREDENTIALS, message="账号或密码错误").model_dump(),
detail=AuthErrorResponse(code=AuthErrorCode.INVALID_CREDENTIALS, message="账号或密码错误").model_dump(),
)
_record_login_success(client_ip)
enforce_user_approved(user)
user = await _prepare_login_session(user)
token = create_access_token(str(user.id), token_version=user.token_version, auth_method="password")
user = await _prepare_login_session(user)
token = create_access_token(str(user.id), token_version=user.token_version, auth_method="password")
_set_session_cookie(response, token, request)
await _sync_user_position(request, user.id)
_sync_user_position(request, user.id)
return _login_response(user, token, username=form_data.username.strip())
return _login_response(user, token, username=form_data.username.strip())
def _username_login_request(username: object) -> UsernameLoginRequest:
if not isinstance(username, str):
raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, detail="username is required")
try:
return UsernameLoginRequest(username=username)
except ValidationError as exc:
raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, detail="username is required") from exc
async def _read_username_login(request: Request) -> UsernameLoginRequest:
"""Accept the historical JSON body and a form body.
``application/x-www-form-urlencoded`` is a CORS simple content type. Old
Firefox aborts a credentialed JSON login during preflight and reports
NS_BINDING_ABORTED, so the browser client sends a form body instead.
"""
media_type = request.headers.get("content-type", "").split(";", 1)[0].strip().lower()
if media_type == "application/x-www-form-urlencoded":
form = await request.form()
return _username_login_request(form.get("username"))
try:
payload = await request.json()
except json.JSONDecodeError as exc:
raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, detail="username is required") from exc
username = payload.get("username") if isinstance(payload, dict) else None
return _username_login_request(username)
@router.post("/login/username", response_model=UsernameLoginResponse)
async def login_by_username(
request: Request,
response: Response,
body: UsernameLoginRequest,
password: str | None = Query(default=None, description="Account password or shared passphrase; required when password login is enabled."),
password: str | None = Query(default=None, description="Account password or shared passphrase; required when password login is enabled."),
):
"""Compatibility username login, gated by the current credential policy.
Personal password/passphrase wins. With password login enabled, accounts
without personal credentials must supply the existing shared passphrase
(default 123ewq). Identity-only login is retained only while password login
is disabled and the historical shared-password gate is also disabled.
"""Compatibility username login, gated by the current credential policy.
Personal password/passphrase wins. With password login enabled, accounts
without personal credentials must supply the existing shared passphrase
(default 123ewq). Identity-only login is retained only while password login
is disabled and the historical shared-password gate is also disabled.
"""
_require_login_method("sso_enabled")
body = await _read_username_login(request)
_require_login_method("sso_enabled")
username_login_config = get_app_config().auth_login.username_login
password_login_enabled = load_system_settings().authentication.password_login_enabled
password_login_enabled = load_system_settings().authentication.password_login_enabled
# Per-account password takes precedence: look the account up (no create) and,
# if it carries a personal 登录口令, require an exact match — ignoring the
# shared gate entirely for that account.
existing = await _lookup_user_by_username(body.username)
credentials_verified = False
if has_account_password(existing) or password_login_enabled or username_login_config.require_password:
credentials_verified = False
if has_account_password(existing) or password_login_enabled or username_login_config.require_password:
client_ip = _get_client_ip(request)
_check_rate_limit(client_ip)
if not await verify_account_password(password or "", existing):
if not await verify_account_password(password or "", existing):
_record_login_failure(client_ip)
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail=AuthErrorResponse(code=AuthErrorCode.INVALID_CREDENTIALS, message="口令错误").model_dump(),
)
credentials_verified = True
credentials_verified = True
_record_login_success(client_ip)
user, created = await _get_or_create_user_by_username(body.username)
user, created = await _get_or_create_user_by_username(body.username, existing_user=existing)
# A registration/reset may commit after the initial lookup/password gate.
# Never mint the new account/version's session using an earlier decision.
if has_account_password(user) and (existing is None or existing.id != user.id or existing.token_version != user.token_version):
client_ip = _get_client_ip(request)
_check_rate_limit(client_ip)
if not await verify_account_password(password or "", user):
_record_login_failure(client_ip)
raise HTTPException(status_code=401, detail=AuthErrorResponse(code=AuthErrorCode.INVALID_CREDENTIALS, message="口令错误").model_dump())
credentials_verified = True
# A registration/reset may commit after the initial lookup/password gate.
# Never mint the new account/version's session using an earlier decision.
if has_account_password(user) and (existing is None or existing.id != user.id or existing.token_version != user.token_version):
client_ip = _get_client_ip(request)
_check_rate_limit(client_ip)
if not await verify_account_password(password or "", user):
_record_login_failure(client_ip)
raise HTTPException(status_code=401, detail=AuthErrorResponse(code=AuthErrorCode.INVALID_CREDENTIALS, message="口令错误").model_dump())
credentials_verified = True
enforce_user_approved(user)
user = await _prepare_login_session(user)
token = create_access_token(str(user.id), token_version=user.token_version, auth_method="passphrase" if credentials_verified else None)
user = await _prepare_login_session(user)
token = create_access_token(str(user.id), token_version=user.token_version, auth_method="passphrase" if credentials_verified else None)
_set_session_cookie(response, token, request)
await _sync_user_position(request, user.id)
_sync_user_position(request, user.id)
return UsernameLoginResponse(
access_token=token,
@@ -714,7 +803,7 @@ async def login_by_token(
Returns the JWT access token in the response body and also sets it as a
session cookie for browser callers.
"""
_require_login_method("sso_enabled")
_require_login_method("sso_enabled")
config = get_app_config().auth_login.token_login
if not config.enabled:
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="token 登录未开启")
@@ -734,10 +823,10 @@ async def login_by_token(
user, created = await _get_or_create_user_by_username(username)
enforce_user_approved(user)
user = await _prepare_login_session(user)
access_token = create_access_token(str(user.id), token_version=user.token_version, auth_method="sso")
user = await _prepare_login_session(user)
access_token = create_access_token(str(user.id), token_version=user.token_version, auth_method="sso")
_set_session_cookie(response, access_token, request)
await _sync_user_position(request, user.id)
_sync_user_position(request, user.id)
return UsernameLoginResponse(
access_token=access_token,
@@ -754,41 +843,41 @@ async def login_by_token(
)
@router.post("/register", response_model=UsernameLoginResponse, status_code=status.HTTP_201_CREATED)
@router.post("/register", response_model=UsernameLoginResponse, status_code=status.HTTP_201_CREATED)
async def register(request: Request, response: Response, body: RegisterRequest):
"""Register a new user account (always 'user' role).
Admin is auto-created on first boot. This endpoint creates regular users.
Auto-login by setting the session cookie.
"""
_require_login_method("password_login_enabled")
_require_login_method("registration_enabled")
client_ip = _get_client_ip(request)
_check_rate_limit(client_ip)
provider = get_local_provider()
username = body.username or str(body.email)
email = _username_to_email(username)
# Match every domain, even for explicit emails: a self-registration must
# never shadow the stable account selected by bare-username login.
duplicate = await provider.find_user_by_email_local_part(email.split("@", 1)[0])
_require_login_method("password_login_enabled")
_require_login_method("registration_enabled")
client_ip = _get_client_ip(request)
_check_rate_limit(client_ip)
provider = get_local_provider()
username = body.username or str(body.email)
email = _username_to_email(username)
# Match every domain, even for explicit emails: a self-registration must
# never shadow the stable account selected by bare-username login.
duplicate = await provider.find_user_by_email_local_part(email.split("@", 1)[0])
try:
if duplicate is not None:
raise ValueError("Duplicate username")
user = await provider.create_user(email=email, password=body.password, system_role="user", approved=True)
if duplicate is not None:
raise ValueError("Duplicate username")
user = await provider.create_user(email=email, password=body.password, system_role="user", approved=True)
except ValueError:
_record_login_failure(client_ip)
_record_login_failure(client_ip)
raise HTTPException(
status_code=status.HTTP_409_CONFLICT,
detail=AuthErrorResponse(code=AuthErrorCode.EMAIL_ALREADY_EXISTS, message="该用户名或邮箱已注册").model_dump(),
status_code=status.HTTP_409_CONFLICT,
detail=AuthErrorResponse(code=AuthErrorCode.EMAIL_ALREADY_EXISTS, message="该用户名或邮箱已注册").model_dump(),
)
enforce_user_approved(user)
user = await _prepare_login_session(user)
token = create_access_token(str(user.id), token_version=user.token_version, auth_method="password")
enforce_user_approved(user)
user = await _prepare_login_session(user)
token = create_access_token(str(user.id), token_version=user.token_version, auth_method="password")
_set_session_cookie(response, token, request)
await _sync_user_position(request, user.id)
_sync_user_position(request, user.id)
return _login_response(user, token, username=username, created=True)
return _login_response(user, token, username=username, created=True)
@router.post("/logout", response_model=MessageResponse)
@@ -798,7 +887,7 @@ async def logout(request: Request, response: Response):
return MessageResponse(message="Successfully logged out")
@router.post("/change-password", response_model=PasswordChangeResponse)
@router.post("/change-password", response_model=PasswordChangeResponse)
async def change_password(request: Request, response: Response, body: ChangePasswordRequest):
"""Change password for the currently authenticated user.
@@ -809,45 +898,45 @@ async def change_password(request: Request, response: Response, body: ChangePass
- Re-issues session cookie with new token_version
"""
user = await get_current_user_from_request(request)
client_ip = _get_client_ip(request)
_check_rate_limit(client_ip)
if not await verify_account_password(body.current_password, user):
_record_login_failure(client_ip)
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=AuthErrorResponse(code=AuthErrorCode.INVALID_CREDENTIALS, message="当前密码错误").model_dump())
client_ip = _get_client_ip(request)
_check_rate_limit(client_ip)
if not await verify_account_password(body.current_password, user):
_record_login_failure(client_ip)
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=AuthErrorResponse(code=AuthErrorCode.INVALID_CREDENTIALS, message="当前密码错误").model_dump())
provider = get_local_provider()
# Update email if provided
if body.new_email is not None:
existing = await provider.find_user_by_email_local_part(body.new_email.split("@", 1)[0])
existing = await provider.find_user_by_email_local_part(body.new_email.split("@", 1)[0])
if existing and str(existing.id) != str(user.id):
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=AuthErrorResponse(code=AuthErrorCode.EMAIL_ALREADY_EXISTS, message="Email already in use").model_dump())
try:
user = await provider.update_credentials(user, await hash_password_async(body.new_password), email=body.new_email)
except ValueError as exc:
raise HTTPException(status_code=409, detail="账号信息已变更，请刷新后重试") from exc
_record_login_success(client_ip)
try:
user = await provider.update_credentials(user, await hash_password_async(body.new_password), email=body.new_email)
except ValueError as exc:
raise HTTPException(status_code=409, detail="账号信息已变更，请刷新后重试") from exc
_record_login_success(client_ip)
# Re-issue cookie with new token_version
token = create_access_token(str(user.id), token_version=user.token_version, auth_method="password")
token = create_access_token(str(user.id), token_version=user.token_version, auth_method="password")
_set_session_cookie(response, token, request)
return PasswordChangeResponse(message="密码已更新", access_token=token, expires_in=get_auth_config().token_expiry_days * 24 * 3600)
return PasswordChangeResponse(message="密码已更新", access_token=token, expires_in=get_auth_config().token_expiry_days * 24 * 3600)
@router.get("/me", response_model=UserResponse)
async def get_me(request: Request):
"""Get current authenticated user info."""
user = await get_current_user_from_request(request)
return UserResponse(
id=str(user.id),
email=user.email,
system_role=user.system_role,
needs_setup=user.needs_setup,
has_password=has_account_password(user),
requires_current_password=True,
uses_default_password=not has_account_password(user),
)
return UserResponse(
id=str(user.id),
email=user.email,
system_role=user.system_role,
needs_setup=user.needs_setup,
has_password=has_account_password(user),
requires_current_password=True,
uses_default_password=not has_account_password(user),
)
_SETUP_STATUS_COOLDOWN: dict[str, float] = {}
@@ -889,12 +978,12 @@ class InitializeAdminRequest(BaseModel):
"""Request model for first-boot admin account creation."""
email: EmailStr
password: str = Field(..., min_length=8, max_length=255)
password: str = Field(..., min_length=8, max_length=255)
_strong_password = field_validator("password")(classmethod(lambda cls, v: _validate_strong_password(v)))
@router.post("/initialize", response_model=UsernameLoginResponse, status_code=status.HTTP_201_CREATED)
@router.post("/initialize", response_model=UsernameLoginResponse, status_code=status.HTTP_201_CREATED)
async def initialize_admin(request: Request, response: Response, body: InitializeAdminRequest):
"""Create the first admin account on initial system setup.
@@ -912,7 +1001,7 @@ async def initialize_admin(request: Request, response: Response, body: Initializ
)
try:
user = await get_local_provider().create_initial_admin(email=str(body.email), password=body.password)
user = await get_local_provider().create_initial_admin(email=str(body.email), password=body.password)
except ValueError:
# DB unique-constraint race: another concurrent request beat us.
raise HTTPException(
@@ -920,11 +1009,11 @@ async def initialize_admin(request: Request, response: Response, body: Initializ
detail=AuthErrorResponse(code=AuthErrorCode.SYSTEM_ALREADY_INITIALIZED, message="System already initialized").model_dump(),
)
user = await _prepare_login_session(user)
token = create_access_token(str(user.id), token_version=user.token_version, auth_method="password")
user = await _prepare_login_session(user)
token = create_access_token(str(user.id), token_version=user.token_version, auth_method="password")
_set_session_cookie(response, token, request)
return _login_response(user, token, created=True)
return _login_response(user, token, created=True)
# ── Admin Endpoints ───────────────────────────────────────────────────────
@@ -938,54 +1027,99 @@ class UserListItem(BaseModel):
agent_count: int
qa_count: int = 0 # 问答次数: number of runs (one run = one question)
approved: bool = True # 登录白名单放行标记(白名单管理页据此放行/取消)
login_password: str | None = None # Legacy plaintext only; never return hashes.
has_login_password: bool = False
access_token: str | None = None
token_type: str = "bearer"
expires_in: int | None = None
def _legacy_login_password(user: User) -> str | None:
value = user.login_password
return value if value and not is_password_hash(value) else None
login_password: str | None = None # Legacy plaintext only; never return hashes.
has_login_password: bool = False
access_token: str | None = None
token_type: str = "bearer"
expires_in: int | None = None
@router.get("/users", response_model=list[UserListItem])
def _legacy_login_password(user: User) -> str | None:
value = user.login_password
return value if value and not is_password_hash(value) else None
class UserListPage(BaseModel):
"""Paged user list — returned when the caller passes ``limit``."""
items: list[UserListItem]
total: int
approved_count: int # approved 或 admin（搜索范围内）
limit: int
offset: int
_USER_LIST_TIMEOUT_SECONDS = 15
def _user_list_skill_store(request: Request, include_counts: bool = True) -> SkillStore | None:
return get_skill_store(request) if include_counts else None
def _user_list_agent_store(request: Request, include_counts: bool = True) -> AgentStore | None:
return get_agent_store(request) if include_counts else None
@router.get("/users", response_model=list[UserListItem] | UserListPage)
async def list_all_users(
request: Request,
skill_store: SkillStore = Depends(get_skill_store),
agent_store: AgentStore = Depends(get_agent_store),
include_counts: bool = Query(True, description="False=只查 users 表，跳过 runs 统计与 skill/agent 计数（白名单管理页用，避免慢查询）"),
q: str | None = Query(None, max_length=100, description="可选：按邮箱/用户ID 子串过滤（仅分页模式生效）"),
limit: int | None = Query(None, ge=1, le=200, description="传入即分页返回 UserListPage；不传保持旧的全量列表"),
offset: int = Query(0, ge=0),
skill_store: SkillStore | None = Depends(_user_list_skill_store),
agent_store: AgentStore | None = Depends(_user_list_agent_store),
):
"""List all users with their skill, agent and Q&A counts. Admin only."""
user = await get_current_user_from_request(request)
if user.system_role != "admin":
raise HTTPException(status_code=403, detail="Admin only")
users = await get_local_provider().list_users()
# One GROUP BY query for everyone's 问答次数, so the list stays a single
# aggregate read instead of a per-user count. Best-effort: if the run store
# is unavailable the column simply renders as 0 rather than failing the list.
"""List all users with their skill, agent and Q&A counts. Admin only.
Passing ``limit`` switches to a paged response that reads one page of the
users table plus two COUNT queries; omitting it keeps the legacy flat list.
"""
try:
qa_counts = await get_run_store(request).count_by_user()
except Exception:
logger.warning("Failed to load per-user run counts for user list", exc_info=True)
qa_counts = {}
result = []
for u in users:
uid = str(u.id)
result.append(
UserListItem(
id=uid,
email=u.email,
system_role=u.system_role,
skill_count=await skill_store.count_owned(uid),
agent_count=await agent_store.count_owned(uid),
qa_count=qa_counts.get(uid, 0),
approved=getattr(u, "approved", True),
login_password=_legacy_login_password(u),
has_login_password=has_account_password(u),
async with asyncio.timeout(_USER_LIST_TIMEOUT_SECONDS if not include_counts else None):
user = await get_current_user_from_request(request)
if user.system_role != "admin":
raise HTTPException(status_code=403, detail="Admin only")
provider = get_local_provider()
users = (
await provider.list_users()
if limit is None
else await provider.list_users_page(q=q, limit=limit, offset=offset)
)
)
return result
qa_counts: dict[str, int] = {}
# One recent-window GROUP BY for everyone's 问答次数. This keeps the admin
# user list from scanning the complete lifetime of the runs table.
if include_counts:
try:
qa_counts = await get_run_store(request).count_by_user(
since=recent_admin_query_cutoff()
)
except Exception:
logger.warning("Failed to load per-user run counts for user list", exc_info=True)
result = []
for u in users:
uid = str(u.id)
result.append(
UserListItem(
id=uid,
email=u.email,
system_role=u.system_role,
skill_count=(await skill_store.count_owned(uid)) if include_counts else 0,
agent_count=(await agent_store.count_owned(uid)) if include_counts else 0,
qa_count=qa_counts.get(uid, 0),
approved=getattr(u, "approved", True),
login_password=_legacy_login_password(u),
has_login_password=has_account_password(u),
)
)
if limit is None:
return result
total = await provider.count_users_filtered(q=q)
approved_count = await provider.count_users_filtered(q=q, approved_or_admin=True)
return UserListPage(items=result, total=total, approved_count=approved_count, limit=limit, offset=offset)
except TimeoutError as exc:
logger.warning("User list timed out waiting for account data")
raise HTTPException(status_code=503, detail="用户基础信息查询超时，请稍后重试") from exc
class UserApprovalRequest(BaseModel):
@@ -1008,10 +1142,10 @@ async def set_user_approval(user_id: str, body: UserApprovalRequest, request: Re
raise HTTPException(status_code=404, detail="用户不存在")
target.approved = bool(body.approved)
try:
saved = await provider.update_user(target)
except CredentialConflictError as exc:
raise HTTPException(status_code=409, detail=str(exc)) from exc
try:
saved = await provider.update_user(target)
except CredentialConflictError as exc:
raise HTTPException(status_code=409, detail=str(exc)) from exc
return UserListItem(
id=str(saved.id),
email=saved.email,
@@ -1020,8 +1154,8 @@ async def set_user_approval(user_id: str, body: UserApprovalRequest, request: Re
agent_count=0,
qa_count=0,
approved=saved.approved,
login_password=_legacy_login_password(saved),
has_login_password=has_account_password(saved),
login_password=_legacy_login_password(saved),
has_login_password=has_account_password(saved),
)
@@ -1029,14 +1163,14 @@ class UserLoginPasswordRequest(BaseModel):
password: str | None = Field(
default=None,
max_length=255,
description="设置账号密码（与登录口令一致）；不允许清除后退回免密码登录。",
description="设置账号密码（与登录口令一致）；不允许清除后退回免密码登录。",
)
@router.put("/users/{user_id}/login-password", response_model=UserListItem)
@router.put("/users/{user_id}/reset-password", response_model=UserListItem)
async def set_user_login_password(user_id: str, body: UserLoginPasswordRequest, request: Request, response: Response) -> UserListItem:
"""Admin password reset; rotates both credential columns and all sessions."""
@router.put("/users/{user_id}/reset-password", response_model=UserListItem)
async def set_user_login_password(user_id: str, body: UserLoginPasswordRequest, request: Request, response: Response) -> UserListItem:
"""Admin password reset; rotates both credential columns and all sessions."""
admin = await get_current_user_from_request(request)
if admin.system_role != "admin":
raise HTTPException(status_code=403, detail="Admin only")
@@ -1046,25 +1180,25 @@ async def set_user_login_password(user_id: str, body: UserLoginPasswordRequest,
if target is None:
raise HTTPException(status_code=404, detail="用户不存在")
pwd = body.password or ""
if not pwd:
raise HTTPException(status_code=400, detail="不能清除账号密码，请设置新密码")
if len(pwd) < 8:
raise HTTPException(status_code=400, detail="密码至少需要 8 个字符")
try:
_validate_strong_password(pwd)
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
try:
saved = await provider.update_credentials(target, await hash_password_async(pwd))
except CredentialConflictError as exc:
raise HTTPException(status_code=409, detail=str(exc)) from exc
token = None
expires_in = None
if str(admin.id) == str(saved.id):
token = create_access_token(str(saved.id), token_version=saved.token_version, auth_method="password")
_set_session_cookie(response, token, request)
expires_in = get_auth_config().token_expiry_days * 24 * 3600
pwd = body.password or ""
if not pwd:
raise HTTPException(status_code=400, detail="不能清除账号密码，请设置新密码")
if len(pwd) < 8:
raise HTTPException(status_code=400, detail="密码至少需要 8 个字符")
try:
_validate_strong_password(pwd)
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
try:
saved = await provider.update_credentials(target, await hash_password_async(pwd))
except CredentialConflictError as exc:
raise HTTPException(status_code=409, detail=str(exc)) from exc
token = None
expires_in = None
if str(admin.id) == str(saved.id):
token = create_access_token(str(saved.id), token_version=saved.token_version, auth_method="password")
_set_session_cookie(response, token, request)
expires_in = get_auth_config().token_expiry_days * 24 * 3600
return UserListItem(
id=str(saved.id),
email=saved.email,
@@ -1073,10 +1207,10 @@ async def set_user_login_password(user_id: str, body: UserLoginPasswordRequest,
agent_count=0,
qa_count=0,
approved=saved.approved,
login_password=None,
has_login_password=True,
access_token=token,
expires_in=expires_in,
login_password=None,
has_login_password=True,
access_token=token,
expires_in=expires_in,
)
@@ -1090,7 +1224,7 @@ async def oauth_login(provider: str):
Redirects to the OAuth provider's authorization URL.
Currently a placeholder - requires OAuth provider implementation.
"""
_require_login_method("sso_enabled")
_require_login_method("sso_enabled")
if provider not in ["github", "google"]:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
@@ -1110,7 +1244,7 @@ async def oauth_callback(provider: str, code: str, state: str):
Handles the OAuth provider's callback after user authorization.
Currently a placeholder.
"""
_require_login_method("sso_enabled")
_require_login_method("sso_enabled")
raise HTTPException(
status_code=status.HTTP_501_NOT_IMPLEMENTED,
detail="OAuth callback not yet implemented",
offline-backend-20260512/backend/app/gateway/routers/browser_context.py
+18 -1
View File
@@ -286,8 +286,23 @@ async def _sync_to_sandbox_if_needed(thread_id: str, virtual_path: str, file_pat
async def get_browser_context_status(
config: AppConfig = Depends(get_config),
) -> BrowserContextStatusResponse:
from app.gateway.api_read_cache import get_cache_version, get_cached_json, is_browser_status_enabled, set_cached_json
# Access control must precede even a warm Redis hit.
if not await is_browser_status_enabled(config):
raise HTTPException(
status_code=403,
detail={"code": "browser_status_disabled", "message": "管理员已关闭浏览器状态接口"},
headers={"Retry-After": "300", "Cache-Control": "no-store"},
)
settings = _settings(config)
return BrowserContextStatusResponse(
# Include the effective config so reloads cannot reuse obsolete capabilities.
cache_key = settings.model_dump_json()
version = await get_cache_version(config, "browser_status")
cached = await get_cached_json(config, "browser_status", cache_key, version=version)
if cached is not None:
return BrowserContextStatusResponse.model_validate(cached)
response = BrowserContextStatusResponse(
enabled=settings.enabled,
max_content_chars=settings.max_content_chars,
max_title_chars=settings.max_title_chars,
@@ -297,6 +312,8 @@ async def get_browser_context_status(
task_lease_seconds=settings.task_lease_seconds,
tool_result_max_chars=settings.tool_result_max_chars,
)
await set_cached_json(config, "browser_status", cache_key, response, expected_version=version)
return response
@router.get("/tasks/next", response_model=BrowserContextTaskNextResponse)
offline-backend-20260512/backend/app/gateway/routers/business_mapping.py
+54 -2
View File
@@ -16,11 +16,13 @@ Routes (prefix ``/api/business-mapping``):
from __future__ import annotations
import logging
import re
from datetime import datetime
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel, Field
from app.gateway.catalog_cache import cached_catalog, invalidate_catalog
from app.gateway.deps import get_agent_store
from deerflow.runtime.user_context import get_effective_user_id
@@ -37,6 +39,8 @@ _EIGHTBF_MANAGER_USERNAME = "lqq"
class BusinessMappingResponse(BaseModel):
business_code: str
label: str = ""
mapping_type: str = "内置业务"
is_custom: bool = False
agent_id: str | None = None
agent_name: str | None = None
chain_id: str | None = None
@@ -58,6 +62,12 @@ class BusinessMappingUpdateRequest(BaseModel):
chain_title: str | None = Field(default=None, max_length=512)
class BusinessMappingCreateRequest(BaseModel):
"""The name is both the display name and login ``goPath`` for a custom row."""
name: str = Field(min_length=1, max_length=32)
# ── helpers ──────────────────────────────────────────────────────────────
@@ -95,6 +105,23 @@ def _current_user_id(request: Request) -> str:
return get_effective_user_id()
_RESERVED_GOPATHS = frozenset({"3qfx", "rwfx", "xdfx", "agentfx", "b5-1"})
def _normalise_custom_mapping_input(body: BusinessMappingCreateRequest) -> str:
"""Validate a custom goPath without rewriting the administrator's label."""
name = re.sub(r"\s+", " ", body.name).strip()
if not name:
raise HTTPException(status_code=422, detail="名称不能为空")
# ``-A`` is a mode suffix, not part of a configurable name: ``name`` is
# single-agent and ``name-A`` is the same mapping's business-chain entry.
if name.lower().endswith("-a"):
raise HTTPException(status_code=422, detail="自定义名称不能以 -A 结尾")
if name.lower() in _RESERVED_GOPATHS:
raise HTTPException(status_code=422, detail="该名称已被内置 goPath 保留")
return name
async def _validate_public_chain_mapping(
request: Request,
fields: dict[str, object],
@@ -167,8 +194,32 @@ async def _validate_public_chain_mapping(
@router.get("", response_model=BusinessMappingListResponse)
async def list_mappings(request: Request) -> BusinessMappingListResponse:
store = _get_store(request)
rows = await store.list_mappings()
return BusinessMappingListResponse(mappings=[BusinessMappingResponse(**r) for r in rows])
async def load() -> BusinessMappingListResponse:
rows = await store.list_mappings()
return BusinessMappingListResponse(mappings=[BusinessMappingResponse(**r) for r in rows])
return await cached_catalog(request, "business_mapping", "all", load, BusinessMappingListResponse)
@router.post("", response_model=BusinessMappingResponse, status_code=201)
async def create_mapping(
request: Request,
body: BusinessMappingCreateRequest,
) -> BusinessMappingResponse:
"""Create a globally shared custom mapping (administrator only)."""
if not _is_admin(request):
raise HTTPException(status_code=403, detail="Only administrators can create business mappings")
name = _normalise_custom_mapping_input(body)
store = _get_store(request)
row = await store.create_custom_mapping(
business_code=name,
mapping_type="自定义类型",
)
if row is None:
raise HTTPException(status_code=409, detail="该名称已存在，请换一个名称")
await invalidate_catalog(request, "business_mapping")
return BusinessMappingResponse(**row)
@router.put("/{business_code}", response_model=BusinessMappingResponse)
@@ -183,4 +234,5 @@ async def update_mapping(request: Request, business_code: str, body: BusinessMap
row = await store.update_mapping(business_code, **fields)
if row is None:
raise HTTPException(status_code=404, detail="Business mapping not found")
await invalidate_catalog(request, "business_mapping")
return BusinessMappingResponse(**row)
offline-backend-20260512/backend/app/gateway/routers/chat_archive.py
+762
View File
@@ -0,0 +1,762 @@
"""Admin controls for resumable monthly PostgreSQL chat archives."""
from __future__ import annotations
import asyncio
import contextlib
import errno
import logging
import shutil
import sqlite3
import uuid
from datetime import datetime
from pathlib import Path
from typing import Any
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel
from sqlalchemy import extract, func, select
from app.gateway.auth.disabled_mode import is_auth_disabled
from app.gateway.deps import get_current_user_from_request
from deerflow.persistence.engine import get_session_factory
from deerflow.persistence.thread_meta.base import is_excluded_system_thread
from deerflow.persistence.thread_meta.model import ThreadMetaRow
from deerflow.persistence.types import BEIJING_TZ
from deerflow.runtime.checkpointer.chat_archive import (
archive_db_path,
archive_root,
ensure_catalog,
thread_archive_lock,
try_acquire_archive_job_lock,
)
from deerflow.runtime.checkpointer.checkpoint_migration import _sqlite_has_table, copy_thread_state
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/admin/chat-archive", tags=["chat-archive"])
_ACTIVE = {"queued", "running", "cancelling"}
_PAGE_SIZE = 40
_ARCHIVE_DISK_RESERVE_BYTES = 2 * 1024**3
_PURGE_THREAD_DELAY_SECONDS = 0.5
_PROCESS_STARTED_AT = datetime.now(BEIJING_TZ).isoformat(timespec="seconds")
class _ArchiveJobCancelled(Exception):
pass
class ChatArchiveJobResponse(BaseModel):
job_id: str
month: str
action: str
status: str
total_threads: int
processed_threads: int
failed_threads: int
current_thread_id: str | None = None
snapshot_at: str = ""
processed_through: str | None = None
message: str = ""
error: str | None = None
created_at: str
updated_at: str
async def _require_admin(request: Request) -> None:
if is_auth_disabled():
return
user = await get_current_user_from_request(request)
if getattr(user, "system_role", None) != "admin":
raise HTTPException(status_code=403, detail="Chat archive management requires admin role")
def _now() -> str:
return datetime.now(BEIJING_TZ).isoformat(timespec="seconds")
def _parse_month(value: str) -> tuple[datetime, datetime]:
try:
start = datetime.strptime(value, "%Y-%m")
except ValueError as exc:
raise HTTPException(status_code=400, detail="month must use YYYY-MM format") from exc
if start.strftime("%Y-%m") != value:
raise HTTPException(status_code=400, detail="month must use YYYY-MM format")
if start.month == 12:
end = start.replace(year=start.year + 1, month=1)
else:
end = start.replace(month=start.month + 1)
return start, end
def _primary_checkpointer(request: Request):
checkpointer = getattr(request.app.state, "checkpointer", None)
primary = getattr(checkpointer, "primary", checkpointer)
if primary is None or not hasattr(getattr(primary, "conn", None), "connection"):
raise HTTPException(status_code=409, detail="当前聊天检查点不是 PostgreSQL，不能执行 PG 归档")
return checkpointer, primary
def _catalog_connect(root: Path) -> sqlite3.Connection:
ensure_catalog(root)
conn = sqlite3.connect(root / "catalog.db", timeout=30)
conn.row_factory = sqlite3.Row
conn.execute("PRAGMA busy_timeout=30000")
return conn
def _recover_jobs(root: Path, *, started_before: str | None = _PROCESS_STARTED_AT, owns_operation_lock: bool = False) -> None:
operation_lock = None if owns_operation_lock else try_acquire_archive_job_lock(root)
if not owns_operation_lock and operation_lock is None:
return # Another gateway/worker still owns the running job.
cutoff_sql = " AND created_at < ?" if started_before else ""
cutoff_values = (started_before,) if started_before else ()
with (contextlib.nullcontext() if owns_operation_lock else operation_lock), _catalog_connect(root) as conn:
now = _now()
conn.execute(
"UPDATE archive_jobs SET status='interrupted', message='执行进程已退出，已保留成功归档；可重新归档或清理已校验部分', "
"error=COALESCE(error, '归档执行进程已退出或服务重启，任务未完成'), updated_at=? "
"WHERE status IN ('queued','running','cancelling')" + cutoff_sql,
(now, *cutoff_values),
)
conn.execute(
"UPDATE archive_months SET status='interrupted', updated_at=? "
"WHERE status IN ('copying','deleting','restoring') AND current_job_id IN "
"(SELECT job_id FROM archive_jobs WHERE status='interrupted'" + cutoff_sql + ")",
(now, *cutoff_values),
)
def _month_start(month: str) -> tuple[datetime, datetime]:
start, end = _parse_month(month)
return start.replace(tzinfo=BEIJING_TZ), end.replace(tzinfo=BEIJING_TZ)
async def _month_thread_count(session_factory, month: str, *, snapshot_at: str | None = None) -> int:
start, end = _month_start(month)
conditions = [ThreadMetaRow.created_at >= start, ThreadMetaRow.created_at < end]
if snapshot_at:
conditions.append(ThreadMetaRow.created_at <= datetime.fromisoformat(snapshot_at))
async with session_factory() as session:
return int(await session.scalar(
select(func.count()).select_from(ThreadMetaRow).where(*conditions)
) or 0)
async def _thread_page(session_factory, month: str, offset: int, limit: int, *, snapshot_at: str | None = None):
start, end = _month_start(month)
conditions = [ThreadMetaRow.created_at >= start, ThreadMetaRow.created_at < end]
if snapshot_at:
conditions.append(ThreadMetaRow.created_at <= datetime.fromisoformat(snapshot_at))
async with session_factory() as session:
result = await session.execute(
select(
ThreadMetaRow.thread_id,
ThreadMetaRow.status,
ThreadMetaRow.metadata_json,
ThreadMetaRow.created_at,
)
.where(*conditions)
.order_by(ThreadMetaRow.created_at, ThreadMetaRow.thread_id)
.offset(offset)
.limit(limit)
)
return list(result.all())
async def _thread_status(session_factory, thread_id: str) -> str | None:
async with session_factory() as session:
value = await session.scalar(
select(ThreadMetaRow.status).where(ThreadMetaRow.thread_id == thread_id).limit(1)
)
return str(value).lower() if value is not None else None
def _archive_action_thread_count(root: Path, month: str, action: str) -> int:
"""Count only archived threads that the requested action will process."""
if not (root / "catalog.db").is_file():
return 0
if action == "purge":
statuses = ("copied", "deleted", "restored")
placeholders = ",".join("?" for _ in statuses)
query = f"SELECT COUNT(*) FROM archive_threads WHERE month=? AND status IN ({placeholders})"
params = (month, *statuses)
elif action == "rollback":
query = "SELECT COUNT(*) FROM archive_threads WHERE month=?"
params = (month,)
else:
return 0
with _catalog_connect(root) as conn:
row = conn.execute(query, params).fetchone()
return int(row[0] or 0)
async def _month_inventory(session_factory) -> list[dict[str, Any]]:
async with session_factory() as session:
dialect = session.get_bind().dialect.name
created_at = ThreadMetaRow.created_at
if dialect in {"postgresql", "opengauss"}:
local_created = func.timezone("Asia/Shanghai", created_at)
else:
local_created = created_at
year_expr = extract("year", local_created).label("year")
month_expr = extract("month", local_created).label("month")
rows = await session.execute(
select(year_expr, month_expr, func.count())
.select_from(ThreadMetaRow)
.group_by(year_expr, month_expr)
.order_by(year_expr.desc(), month_expr.desc())
)
return [
{"month": f"{int(year):04d}-{int(month):02d}", "thread_count": int(count)}
for year, month, count in rows.all()
if year is not None and month is not None
]
def _catalog_months(root: Path) -> dict[str, dict[str, Any]]:
if not (root / "catalog.db").exists():
return {}
with _catalog_connect(root) as conn:
months = {
row["month"]: dict(row)
for row in conn.execute("SELECT * FROM archive_months")
}
for row in conn.execute(
"SELECT j.* FROM archive_jobs j WHERE j.rowid=(SELECT newest.rowid FROM archive_jobs newest "
"WHERE newest.month=j.month ORDER BY newest.created_at DESC, newest.rowid DESC LIMIT 1)"
):
months.setdefault(row["month"], {})["latest_job"] = dict(row)
return months
def _save_job(root: Path, job: dict[str, Any]) -> None:
with _catalog_connect(root) as conn:
conn.execute(
"INSERT INTO archive_jobs(job_id,month,action,status,total_threads,processed_threads,failed_threads,"
"current_thread_id,snapshot_at,processed_through,message,error,created_at,updated_at) "
"VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
tuple((job.get(key) or job.get("created_at") or "") if key == "snapshot_at" else job.get(key) for key in (
"job_id", "month", "action", "status", "total_threads", "processed_threads",
"failed_threads", "current_thread_id", "snapshot_at", "processed_through", "message", "error", "created_at", "updated_at",
)),
)
def _claim_month_job(root: Path, job: dict[str, Any], month_status: str) -> bool:
"""Atomically allow only one heavy archive operation across processes."""
conn = _catalog_connect(root)
try:
conn.execute("BEGIN IMMEDIATE")
active = conn.execute(
"SELECT job_id FROM archive_jobs WHERE status IN ('queued','running','cancelling') LIMIT 1"
).fetchone()
if active:
conn.rollback()
return False
conn.execute(
"INSERT INTO archive_jobs(job_id,month,action,status,total_threads,processed_threads,failed_threads,"
"current_thread_id,snapshot_at,processed_through,message,error,created_at,updated_at) "
"VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
tuple((job.get(key) or job.get("created_at") or "") if key == "snapshot_at" else job.get(key) for key in (
"job_id", "month", "action", "status", "total_threads", "processed_threads",
"failed_threads", "current_thread_id", "snapshot_at", "processed_through", "message", "error", "created_at", "updated_at",
)),
)
conn.execute(
"INSERT INTO archive_months(month,status,total_threads,copied_threads,deleted_threads,restored_threads,current_job_id,updated_at) "
"VALUES(?,?,?,0,0,0,?,?) ON CONFLICT(month) DO UPDATE SET status=excluded.status, "
"total_threads=excluded.total_threads,current_job_id=excluded.current_job_id,updated_at=excluded.updated_at",
(job["month"], month_status, job["total_threads"], job["job_id"], _now()),
)
conn.commit()
return True
except Exception:
conn.rollback()
raise
finally:
conn.close()
def _patch_job(root: Path, job_id: str, **values) -> None:
values["updated_at"] = _now()
assignments = ",".join(f"{key}=?" for key in values)
with _catalog_connect(root) as conn:
conn.execute(f"UPDATE archive_jobs SET {assignments} WHERE job_id=?", (*values.values(), job_id))
async def _patch_job_async(root: Path, job_id: str, **values) -> None:
await asyncio.to_thread(_patch_job, root, job_id, **values)
def _job_dict(root: Path, job_id: str) -> dict[str, Any] | None:
if not (root / "catalog.db").exists():
return None
with _catalog_connect(root) as conn:
row = conn.execute("SELECT * FROM archive_jobs WHERE job_id=?", (job_id,)).fetchone()
return dict(row) if row else None
def _upsert_month(root: Path, month: str, *, status: str, total: int, job_id: str | None = None) -> None:
with _catalog_connect(root) as conn:
conn.execute(
"INSERT INTO archive_months(month,status,total_threads,copied_threads,deleted_threads,restored_threads,current_job_id,updated_at) "
"VALUES(?,?,?,0,0,0,?,?) ON CONFLICT(month) DO UPDATE SET status=excluded.status, "
"total_threads=excluded.total_threads,current_job_id=excluded.current_job_id,updated_at=excluded.updated_at",
(month, status, total, job_id, _now()),
)
async def _upsert_month_async(root: Path, month: str, *, status: str, total: int, job_id: str | None = None) -> None:
await asyncio.to_thread(_upsert_month, root, month, status=status, total=total, job_id=job_id)
def _mark_archived_thread(root: Path, thread_id: str, month: str, count: int, status: str = "copied") -> None:
now = _now()
with _catalog_connect(root) as conn:
conn.execute(
"INSERT INTO archive_threads(thread_id,month,status,checkpoint_count,archived_at,deleted_at,restored_at) "
"VALUES(?,?,?,?,?,?,?) ON CONFLICT(thread_id) DO UPDATE SET month=excluded.month,status=excluded.status,"
"checkpoint_count=excluded.checkpoint_count,archived_at=excluded.archived_at,"
"deleted_at=CASE WHEN excluded.status='deleted' THEN excluded.archived_at ELSE archive_threads.deleted_at END,"
"restored_at=CASE WHEN excluded.status='restored' THEN excluded.archived_at ELSE archive_threads.restored_at END",
(thread_id, month, status, count, now, now if status == "deleted" else None, now if status == "restored" else None),
)
conn.execute(
"UPDATE archive_months SET copied_threads=(SELECT COUNT(*) FROM archive_threads WHERE month=?), "
"deleted_threads=(SELECT COUNT(*) FROM archive_threads WHERE month=? AND status='deleted'), updated_at=? WHERE month=?",
(month, month, now, month),
)
async def _checkpoint_ids(saver, thread_id: str) -> list[str]:
config = {"configurable": {"thread_id": thread_id}}
ids: list[str] = []
async for tup in saver.alist(config):
configurable = (tup.config or {}).get("configurable", {}) or {}
checkpoint_id = configurable.get("checkpoint_id") or (tup.checkpoint or {}).get("id")
if checkpoint_id:
ids.append(str(checkpoint_id))
return ids
@contextlib.asynccontextmanager
async def _open_sqlite_archive(path: Path):
from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver
path.parent.mkdir(parents=True, exist_ok=True)
context = AsyncSqliteSaver.from_conn_string(str(path))
saver = await context.__aenter__()
try:
await saver.conn.execute("PRAGMA busy_timeout=30000")
if not await asyncio.to_thread(_sqlite_has_table, str(path), "checkpoints"):
await saver.conn.execute("PRAGMA journal_mode=WAL")
await saver.setup()
yield saver
finally:
await context.__aexit__(None, None, None)
class ChatArchiveRunner:
def __init__(self, root: Path, checkpointer, session_factory):
self.root = root
self.checkpointer = checkpointer
self.primary = getattr(checkpointer, "primary", checkpointer)
self.session_factory = session_factory
self._locks: dict[str, asyncio.Lock] = {}
self._tasks: dict[str, asyncio.Task] = {}
async def start(self, month: str, action: str) -> dict[str, Any]:
_parse_month(month)
if action not in {"archive", "purge", "rollback"}:
raise HTTPException(status_code=400, detail="未知归档操作")
current_month = datetime.now(BEIJING_TZ).strftime("%Y-%m")
if month > current_month:
raise HTTPException(status_code=409, detail="不能归档未来月份")
snapshot_at = _now()
if action == "archive":
total = await _month_thread_count(self.session_factory, month, snapshot_at=snapshot_at)
else:
total = await asyncio.to_thread(_archive_action_thread_count, self.root, month, action)
if not total:
raise HTTPException(status_code=409, detail="该月份尚无已校验归档，不能删除或回滚；请先归档")
job_id = uuid.uuid4().hex
job = {
"job_id": job_id,
"month": month,
"action": action,
"status": "queued",
"total_threads": total,
"processed_threads": 0,
"failed_threads": 0,
"current_thread_id": None,
"snapshot_at": snapshot_at,
"processed_through": None,
"message": "任务已排队",
"error": None,
"created_at": _now(),
"updated_at": _now(),
}
status = {"archive": "copying", "purge": "deleting", "rollback": "restoring"}[action]
operation_lock = try_acquire_archive_job_lock(self.root)
if operation_lock is None:
raise HTTPException(status_code=409, detail="已有月份归档、清理或回滚任务正在运行，请完成后再启动下一项")
try:
# The kernel lock is free only if no runner is alive, including
# one that died after this gateway process finished starting.
_recover_jobs(self.root, started_before=None, owns_operation_lock=True)
claimed = _claim_month_job(self.root, job, status)
if not claimed:
raise HTTPException(status_code=409, detail="已有月份归档、清理或回滚任务正在运行，请完成后再启动下一项")
task = asyncio.create_task(self._run(job_id, owns_operation_lock=True), name=f"chat-archive-{action}-{month}")
except BaseException:
operation_lock.close()
raise
self._tasks[job_id] = task
def completed(_task):
operation_lock.close()
self._tasks.pop(job_id, None)
task.add_done_callback(completed)
return job
async def request_cancel(self, job_id: str) -> dict[str, Any]:
job = await asyncio.to_thread(_job_dict, self.root, job_id)
if not job:
raise HTTPException(status_code=404, detail="归档任务不存在")
if job["status"] in _ACTIVE:
await _patch_job_async(self.root, job_id, status="cancelling", message="将在当前会话处理结束后停止")
return await asyncio.to_thread(_job_dict, self.root, job_id)
async def _check_cancelled(self, job_id: str) -> None:
job = await asyncio.to_thread(_job_dict, self.root, job_id) or {}
if job.get("status") == "cancelling":
raise _ArchiveJobCancelled()
async def _record_thread_failure(self, job_id: str, tid: str, exc: Exception) -> None:
# A corrupt/full archive volume cannot safely accept more threads.
# Row-specific failures may continue; infrastructure failures must stop.
sqlite_code = getattr(exc, "sqlite_errorcode", 0) or 0
if (
isinstance(exc, MemoryError)
or isinstance(exc, OSError) and exc.errno in {errno.ENOSPC, errno.EDQUOT}
or isinstance(exc, sqlite3.Error) and (sqlite_code & 0xFF) in {
sqlite3.SQLITE_FULL, sqlite3.SQLITE_IOERR, sqlite3.SQLITE_CORRUPT, sqlite3.SQLITE_NOTADB,
}
):
raise exc
logger.error("Chat archive thread failed; continuing: job=%s thread=%s", job_id, tid, exc_info=True)
job = await asyncio.to_thread(_job_dict, self.root, job_id) or {}
await _patch_job_async(
self.root, job_id,
failed_threads=int(job.get("failed_threads") or 0) + 1,
error=f"会话 {tid}: {type(exc).__name__}: {exc}"[:2000],
message="该会话处理失败，已记录错误，继续处理其他会话",
)
async def _run(self, job_id: str, *, owns_operation_lock: bool = False) -> None:
operation_lock = None if owns_operation_lock else try_acquire_archive_job_lock(self.root)
if not owns_operation_lock and operation_lock is None:
return
try:
await self._run_locked(job_id)
finally:
if operation_lock is not None:
operation_lock.close()
async def _run_locked(self, job_id: str) -> None:
job = _job_dict(self.root, job_id)
if not job:
return
month = job["month"]
action = job["action"]
lock = self._locks.setdefault(month, asyncio.Lock())
async with lock:
try:
await self._check_cancelled(job_id)
await _patch_job_async(self.root, job_id, status="running", message="正在处理，页面可安全关闭")
if action == "archive":
await self._archive_month(job_id, month)
elif action == "purge":
await self._purge_month(job_id, month)
else:
await self._rollback_month(job_id, month)
job = await asyncio.to_thread(_job_dict, self.root, job_id) or job
failed = int(job.get("failed_threads") or 0)
status = "completed" if failed == 0 else "partial"
month_status = {
"archive": "ready" if failed == 0 else "partial",
"purge": "deleted" if failed == 0 else "partial",
"rollback": "ready" if failed == 0 else "partial",
}[action]
await _patch_job_async(self.root, job_id, status=status, current_thread_id=None,
message="处理完成" if failed == 0 else f"完成但有 {failed} 个会话未处理")
await _upsert_month_async(self.root, month, status=month_status, total=int(job.get("total_threads") or 0), job_id=job_id)
except (_ArchiveJobCancelled, asyncio.CancelledError) as exc:
await _patch_job_async(self.root, job_id, status="cancelled" if isinstance(exc, _ArchiveJobCancelled) else "interrupted",
current_thread_id=None, message="任务已停止，已完成的归档保留，可重新执行",
error="管理员停止任务" if isinstance(exc, _ArchiveJobCancelled) else "执行任务被取消或服务正在退出")
await _upsert_month_async(self.root, month, status="interrupted", total=int(job.get("total_threads") or 0), job_id=job_id)
if isinstance(exc, asyncio.CancelledError):
raise
except Exception as exc:
logger.exception("Chat archive job failed: job=%s month=%s action=%s", job_id, month, action)
await _patch_job_async(self.root, job_id, status="failed", error=f"{type(exc).__name__}: {exc}"[:2000],
message="任务停止，已完成的归档保留；可查看错误并重新执行")
await _upsert_month_async(self.root, month, status="interrupted", total=int(job.get("total_threads") or 0), job_id=job_id)
async def _archive_month(self, job_id: str, month: str) -> None:
path = archive_db_path(self.root, month)
if (await asyncio.to_thread(shutil.disk_usage, self.root)).free < _ARCHIVE_DISK_RESERVE_BYTES:
raise RuntimeError("归档目录所在磁盘可用空间不足 2 GiB；请先扩容或清理磁盘，PostgreSQL 数据未删除")
job = await asyncio.to_thread(_job_dict, self.root, job_id) or {}
snapshot_at = str(job.get("snapshot_at") or job.get("created_at") or "")
total = int(job.get("total_threads") or 0)
await _upsert_month_async(self.root, month, status="copying", total=total, job_id=job_id)
processed = 0
offset = 0
async with _open_sqlite_archive(path) as archive:
while True:
rows = await _thread_page(
self.session_factory, month, offset, _PAGE_SIZE, snapshot_at=snapshot_at
)
if not rows:
break
offset += len(rows)
for thread_id, thread_status, metadata, created_at in rows:
await self._check_cancelled(job_id)
processed_through = created_at.isoformat() if created_at else None
if is_excluded_system_thread(metadata if isinstance(metadata, dict) else {}):
processed += 1
await _patch_job_async(
self.root, job_id, processed_threads=processed,
processed_through=processed_through, message="已跳过系统任务会话"
)
continue
tid = str(thread_id)
if (await asyncio.to_thread(shutil.disk_usage, self.root)).free < _ARCHIVE_DISK_RESERVE_BYTES:
raise RuntimeError("归档卷剩余空间低于 2 GiB，任务已暂停；PostgreSQL 数据保留")
await _patch_job_async(self.root, job_id, current_thread_id=tid, message="正在复制会话检查点")
if str(thread_status).lower() in {"running", "busy"}:
current_job = await asyncio.to_thread(_job_dict, self.root, job_id) or {}
processed += 1
await _patch_job_async(
self.root, job_id,
failed_threads=int(current_job.get("failed_threads") or 0) + 1,
processed_threads=processed, processed_through=processed_through,
message="正在跳过运行中的会话"
)
continue
try:
async with thread_archive_lock(self.root, tid, exclusive=True):
if await _thread_status(self.session_factory, tid) in {"running", "busy"}:
raise RuntimeError("会话正在问答中，已跳过；结束后可重新归档")
source_ids = await _checkpoint_ids(self.primary, tid)
if not source_ids:
processed += 1
await _patch_job_async(
self.root, job_id, processed_threads=processed,
processed_through=processed_through, message="已跳过无检查点的空会话"
)
continue
await copy_thread_state(self.primary, archive, tid)
copied_ids = await _checkpoint_ids(archive, tid)
if not set(source_ids).issubset(copied_ids):
raise RuntimeError(f"thread {tid} checkpoint verification failed: source={len(source_ids)} archived={len(copied_ids)}")
await asyncio.to_thread(_mark_archived_thread, self.root, tid, month, len(copied_ids), "copied")
except Exception as exc:
await self._record_thread_failure(job_id, tid, exc)
processed += 1
current_job = await asyncio.to_thread(_job_dict, self.root, job_id) or {}
await _patch_job_async(
self.root, job_id, processed_threads=processed,
failed_threads=int(current_job.get("failed_threads") or 0),
processed_through=processed_through,
)
await asyncio.sleep(0.03)
job = await asyncio.to_thread(_job_dict, self.root, job_id) or {}
if int(job.get("failed_threads") or 0):
await _patch_job_async(self.root, job_id, message="部分会话正忙或校验失败；可稍后重新归档")
async def _purge_month(self, job_id: str, month: str) -> None:
path = archive_db_path(self.root, month)
if not path.is_file():
raise RuntimeError(f"monthly SQLite archive is missing: {path}")
with _catalog_connect(self.root) as conn:
thread_ids = [str(row[0]) for row in conn.execute(
"SELECT thread_id FROM archive_threads WHERE month=? AND status IN ('copied','deleted','restored') ORDER BY thread_id",
(month,),
)]
processed = 0
async with _open_sqlite_archive(path) as archive:
for tid in thread_ids:
await self._check_cancelled(job_id)
await _patch_job_async(self.root, job_id, current_thread_id=tid, message="重新校验并清理 PostgreSQL 检查点")
skipped_running = False
try:
async with thread_archive_lock(self.root, tid, exclusive=True):
thread_status = await _thread_status(self.session_factory, tid)
if thread_status in {"running", "busy"}:
skipped_running = True
else:
pg_ids = await _checkpoint_ids(self.primary, tid)
if not pg_ids:
await asyncio.to_thread(
_mark_archived_thread,
self.root,
tid,
month,
len(await _checkpoint_ids(archive, tid)),
"deleted",
)
else:
await copy_thread_state(self.primary, archive, tid)
archived_ids = await _checkpoint_ids(archive, tid)
if not set(pg_ids).issubset(archived_ids):
raise RuntimeError(f"thread {tid} archive is incomplete; PostgreSQL rows were kept")
await self.primary.adelete_thread(tid)
remaining = await _checkpoint_ids(self.primary, tid)
if remaining:
raise RuntimeError(f"thread {tid} PostgreSQL cleanup verification failed")
await asyncio.to_thread(_mark_archived_thread, self.root, tid, month, len(archived_ids), "deleted")
except Exception as exc:
await self._record_thread_failure(job_id, tid, exc)
processed += 1
job_update: dict[str, Any] = {"processed_threads": processed}
if skipped_running:
job = await asyncio.to_thread(_job_dict, self.root, job_id) or {}
job_update.update(
failed_threads=int(job.get("failed_threads") or 0) + 1,
message="会话仍在问答中，已跳过；结束后可重新清理",
)
await _patch_job_async(self.root, job_id, **job_update)
await asyncio.sleep(_PURGE_THREAD_DELAY_SECONDS)
async def _rollback_month(self, job_id: str, month: str) -> None:
path = archive_db_path(self.root, month)
if not path.is_file():
raise RuntimeError(f"monthly SQLite archive is missing: {path}")
with _catalog_connect(self.root) as conn:
thread_ids = [str(row[0]) for row in conn.execute(
"SELECT thread_id FROM archive_threads WHERE month=? ORDER BY thread_id", (month,)
)]
processed = 0
async with _open_sqlite_archive(path) as archive:
for tid in thread_ids:
await self._check_cancelled(job_id)
await _patch_job_async(self.root, job_id, current_thread_id=tid, message="回滚检查点到 PostgreSQL")
try:
async with thread_archive_lock(self.root, tid, exclusive=True):
if await _thread_status(self.session_factory, tid) in {"running", "busy"}:
raise RuntimeError("会话正在问答中，已跳过；结束后可重新回滚")
archive_ids = await _checkpoint_ids(archive, tid)
await copy_thread_state(archive, self.primary, tid)
pg_ids = await _checkpoint_ids(self.primary, tid)
if not set(archive_ids).issubset(pg_ids):
raise RuntimeError(f"thread {tid} rollback verification failed")
await asyncio.to_thread(_mark_archived_thread, self.root, tid, month, len(archive_ids), "restored")
except Exception as exc:
await self._record_thread_failure(job_id, tid, exc)
processed += 1
await _patch_job_async(self.root, job_id, processed_threads=processed)
await asyncio.sleep(0.03)
async def _runner(request: Request) -> ChatArchiveRunner:
runner = getattr(request.app.state, "chat_archive_runner", None)
if runner is None:
checkpointer, _ = _primary_checkpointer(request)
session_factory = get_session_factory()
if session_factory is None:
raise HTTPException(status_code=503, detail="聊天元数据数据库未启用")
root = archive_root()
ensure_catalog(root)
await _ensure_archive_recovery_async(request, root)
runner = ChatArchiveRunner(root, checkpointer, session_factory)
request.app.state.chat_archive_runner = runner
return runner
async def _ensure_archive_recovery_async(request: Request, root: Path) -> None:
"""Recover abandoned work without interrupting another live process.
Recheck on inventory requests: another process may die after this process
starts or after an earlier recovery was skipped because the lock was held.
"""
if not (root / "catalog.db").is_file():
# Merely opening the settings page must not initialize archive storage.
return
lock = getattr(request.app.state, "chat_archive_recovery_lock", None)
if lock is None:
lock = asyncio.Lock()
request.app.state.chat_archive_recovery_lock = lock
async with lock:
await asyncio.to_thread(_recover_jobs, root, started_before=None)
@router.get("/months")
async def list_archive_months(request: Request):
await _require_admin(request)
_, _primary = _primary_checkpointer(request)
session_factory = get_session_factory()
if session_factory is None:
raise HTTPException(status_code=503, detail="聊天元数据数据库未启用")
root = archive_root()
await _ensure_archive_recovery_async(request, root)
inventory = await _month_inventory(session_factory)
catalog = _catalog_months(root)
disk_path = root
while not disk_path.exists():
disk_path = disk_path.parent
disk = await asyncio.to_thread(shutil.disk_usage, disk_path)
months = []
for item in inventory:
month = item["month"]
state = catalog.get(month, {})
file_path = archive_db_path(root, month)
months.append({
**item,
"status": state.get("status", "not_archived"),
"copied_threads": int(state.get("copied_threads") or 0),
"deleted_threads": int(state.get("deleted_threads") or 0),
"sqlite_bytes": file_path.stat().st_size if file_path.is_file() else 0,
"latest_job": state.get("latest_job"),
})
return {
"archive_dir": str(root),
"disk_total_bytes": disk.total,
"disk_free_bytes": disk.free,
"month_basis": "会话创建月份（按业务数据库 threads_meta 时间）",
"warning": "页面只读取轻量会话元数据，不创建归档文件或扫描 PostgreSQL 检查点表；只有管理员启动归档、清理或回滚任务时才会读写归档数据。",
"months": months,
}
async def _start_action(month: str, action: str, request: Request):
await _require_admin(request)
_primary_checkpointer(request)
return await (await _runner(request)).start(month, action)
@router.post("/months/{month}/archive")
async def start_month_archive(month: str, request: Request):
return await _start_action(month, "archive", request)
@router.post("/months/{month}/delete-postgres")
async def delete_archived_month_from_postgres(month: str, request: Request):
return await _start_action(month, "purge", request)
@router.post("/months/{month}/rollback")
async def rollback_month_to_postgres(month: str, request: Request):
return await _start_action(month, "rollback", request)
@router.get("/jobs/{job_id}", response_model=ChatArchiveJobResponse)
async def get_chat_archive_job(job_id: str, request: Request):
await _require_admin(request)
job = _job_dict(archive_root(), job_id)
if job is None:
raise HTTPException(status_code=404, detail="归档任务不存在")
return job
offline-backend-20260512/backend/app/gateway/routers/compact_menu_overrides.py
+12 -2
View File
@@ -17,6 +17,7 @@ from fastapi import APIRouter, HTTPException, Query, Request
from pydantic import BaseModel, Field
from app.gateway.auth.disabled_mode import is_auth_disabled
from app.gateway.catalog_cache import cached_catalog, catalog_key, invalidate_catalog
from app.gateway.deps import get_optional_user_from_request
from deerflow.persistence.compact_menu_overrides import (
DEFAULT_COMPACT_MENU_TYPE,
@@ -32,6 +33,7 @@ class CompactMenuOverride(BaseModel):
placement: Literal["sidebar", "settings", "toolbox"] = "sidebar"
sort_order: int = 0
label: str | None = Field(default=None, max_length=255)
disabled: bool = False
class CompactMenuOverrideResponse(CompactMenuOverride):
@@ -99,9 +101,11 @@ async def _read_response(
request: Request,
store: CompactMenuOverrideStore,
menu_type: str,
*,
actor: tuple[str | None, bool] | None = None,
) -> CompactMenuOverrideListResponse:
user_id, is_admin = actor if actor is not None else await _actor(request)
default_rows = await store.list_overrides(menu_type)
user_id, is_admin = await _actor(request)
if is_admin or user_id is None:
return CompactMenuOverrideListResponse(
menu_type=menu_type,
@@ -133,8 +137,12 @@ async def list_compact_menu_overrides(
),
) -> CompactMenuOverrideListResponse:
t = normalize_compact_menu_type(menu_type)
actor = await _actor(request)
store = _get_store(request)
return await _read_response(request, store, t)
return await cached_catalog(
request, "compact_menu_overrides", catalog_key(*actor, t),
lambda: _read_response(request, store, t, actor=actor), CompactMenuOverrideListResponse,
)
@router.put("", response_model=CompactMenuOverrideListResponse)
@@ -150,6 +158,7 @@ async def replace_compact_menu_overrides(
await store.replace_all(payload, menu_type=t, updated_by=user_id)
else:
await store.replace_user_overrides(user_id, payload, menu_type=t, updated_by=user_id)
await invalidate_catalog(request, "compact_menu_overrides")
return await _read_response(request, store, t)
@@ -166,4 +175,5 @@ async def reset_compact_menu_overrides(
await store.replace_all([], menu_type=t, updated_by=user_id)
else:
await store.replace_user_overrides(user_id, [], menu_type=t, updated_by=user_id)
await invalidate_catalog(request, "compact_menu_overrides")
return await _read_response(request, store, t)
offline-backend-20260512/backend/app/gateway/routers/deep_research.py
+2 -1
View File
@@ -865,8 +865,9 @@ async def get_capabilities() -> dict[str, Any]:
"""Advertise available modes, material channels, and export formats (§16.7)."""
from deerflow.agents.deep_research.adapters.image import ImageGenerationSettings
from deerflow.agents.deep_research.adapters.material_provider import DeerFlowMaterialProvider
from deerflow.utils.background_init import run_background_init
provider = DeerFlowMaterialProvider()
provider = await run_background_init(DeerFlowMaterialProvider)
image_settings = ImageGenerationSettings.from_app_config()
channels = provider.material_channels
return {
offline-backend-20260512/backend/app/gateway/routers/force_situation.py
+1 -1
View File
@@ -4,7 +4,7 @@
（整体存储、整体替换，数据库不定死字段）。文档内容由消费方与产出方约定，
当前包含时间轴（重要时间节点）、任务衔接、页面单块内容等区块。
Routes (authenticated by the Gateway middleware):
Routes (credential-free internal integration endpoints):
GET /api/force-situation/tasks/{task_id}/detail
PUT /api/force-situation/tasks/{task_id}/detail
"""
offline-backend-20260512/backend/app/gateway/routers/html_runtime.py
+3 -5
View File
@@ -10,8 +10,7 @@ from pathlib import Path
from fastapi import APIRouter, HTTPException
from fastapi.responses import FileResponse, Response
from deerflow.config import get_app_config
from deerflow.config.html_runtime_config import DEFAULT_HTML_MAP_TILE_URL
from deerflow.html_runtime.addresses import html_map_tile_url, html_runtime_base_url
router = APIRouter(prefix="/api/public/html-runtime", tags=["html-runtime"])
@@ -60,9 +59,8 @@ async def get_deerflow_html_runtime() -> Response:
path = _RUNTIME_DIR / "deerflow-runtime.js"
if not path.is_file():
raise HTTPException(status_code=503, detail="HTML runtime is not installed")
config = get_app_config().html_runtime
base_url = config.base_url.rstrip("/")
tile_url = str(getattr(config, "map_tile_url", "") or "").strip() or DEFAULT_HTML_MAP_TILE_URL
base_url = html_runtime_base_url()
tile_url = html_map_tile_url()
runtime_config = {
"geoMaps": {
"china-provinces": f"{base_url}/maps/china-provinces.geojson",
offline-backend-20260512/backend/app/gateway/routers/html_templates.py
+137
View File
@@ -0,0 +1,137 @@
"""HTML 模板管理 API(模板管理页)。
后端存储 :mod:`deerflow.runtime.scheduler.template_store`(文件型,
``{base_dir}/html-templates/<id>/``),启动时把随包内置模板 seed 进去(缺失才复制)。
读对所有登录用户开放(定时任务表单下拉也要用),写(创建/编辑/上传覆盖/删除/重置)
仅管理员——与敏感词管理同一鉴权口径。旧端点 ``GET /api/scheduled-tasks/html-templates``
保持不变(内部同样读存储),本路由是它的管理超集。
"""
from __future__ import annotations
import asyncio
from typing import Any
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel, Field
from app.gateway.catalog_cache import cached_catalog, catalog_key, invalidate_catalog
from app.gateway.deps import get_optional_user_from_request
from deerflow.runtime.scheduler import template_store
router = APIRouter(prefix="/api/html-templates", tags=["html-templates"])
async def _require_admin(request: Request) -> None:
"""写操作仅管理员;关闭鉴权时(request 无用户)放行,与 scheduled_tasks 口径一致。"""
user = await get_optional_user_from_request(request)
if user is None:
return
if getattr(user, "system_role", None) != "admin":
raise HTTPException(status_code=403, detail="仅管理员可管理 HTML 模板")
def _not_found(template_id: str) -> HTTPException:
return HTTPException(status_code=404, detail=f"模板不存在：{template_id}")
class TemplateCreateRequest(BaseModel):
template_id: str = Field(..., description="模板 id（字母/数字/./-/_，创建后不可改）")
name: str = Field(..., description="显示名称，如 no.28")
description: str = ""
html: str = Field(..., description="完整 HTML 模板正文")
class TemplateUpdateRequest(BaseModel):
name: str | None = None
description: str | None = None
html: str | None = Field(default=None, description="新的模板正文（上传覆盖走这里）")
@router.get("")
async def list_html_templates(request: Request) -> dict:
"""模板清单(含 builtin/convertible/size/时间),供管理页与定时任务表单共用。"""
async def load() -> dict:
return {"templates": await asyncio.to_thread(template_store.list_templates)}
return await cached_catalog(request, "html_templates", "admin-list", load)
@router.get("/{template_id}")
async def get_html_template(template_id: str, request: Request) -> dict[str, Any]:
async def load() -> dict[str, Any]:
meta = await asyncio.to_thread(template_store.get_template_meta, template_id)
if meta is None:
raise _not_found(template_id)
return meta
return await cached_catalog(request, "html_templates", catalog_key("meta", template_id), load)
@router.get("/{template_id}/content")
async def get_html_template_content(template_id: str, request: Request) -> dict:
async def load() -> dict:
try:
html = await asyncio.to_thread(template_store.read_template_html, template_id)
except template_store.TemplateStoreError:
raise _not_found(template_id)
return {"template_id": template_id, "html": html}
return await cached_catalog(request, "html_templates", catalog_key("content", template_id), load)
@router.post("")
async def create_html_template(body: TemplateCreateRequest, request: Request) -> dict[str, Any]:
await _require_admin(request)
try:
meta = template_store.create_template(
template_id=body.template_id,
name=body.name,
description=body.description,
html=body.html,
)
except template_store.TemplateStoreError as exc:
raise HTTPException(status_code=400, detail=str(exc))
await invalidate_catalog(request, "html_templates")
return meta
@router.put("/{template_id}")
async def update_html_template(
template_id: str, body: TemplateUpdateRequest, request: Request
) -> dict[str, Any]:
await _require_admin(request)
try:
meta = template_store.update_template(
template_id,
name=body.name,
description=body.description,
html=body.html,
)
except template_store.TemplateStoreError as exc:
raise HTTPException(status_code=400, detail=str(exc))
await invalidate_catalog(request, "html_templates")
return meta
@router.delete("/{template_id}")
async def delete_html_template(template_id: str, request: Request) -> dict:
await _require_admin(request)
try:
template_store.delete_template(template_id)
except template_store.TemplateStoreError as exc:
raise HTTPException(status_code=400, detail=str(exc))
await invalidate_catalog(request, "html_templates")
return {"ok": True, "template_id": template_id}
@router.post("/{template_id}/reset")
async def reset_html_template(template_id: str, request: Request) -> dict[str, Any]:
"""把内置模板正文重置回随包原版(自建模板无原版,400)。"""
await _require_admin(request)
try:
meta = template_store.reset_template(template_id)
except template_store.TemplateStoreError as exc:
raise HTTPException(status_code=400, detail=str(exc))
await invalidate_catalog(request, "html_templates")
return meta
offline-backend-20260512/backend/app/gateway/routers/light_apps.py
+52 -14
View File
@@ -31,6 +31,7 @@ from datetime import datetime
from fastapi import APIRouter, HTTPException, Query, Request
from pydantic import BaseModel, Field
from app.gateway.catalog_cache import cached_catalog, catalog_key, invalidate_catalog
from app.gateway.deps import get_current_user, get_optional_user_from_request
from deerflow.persistence.light_apps import DuplicateLightAppNameError, LightAppStore
@@ -77,6 +78,17 @@ class LightAppListResponse(BaseModel):
apps: list[LightAppResponse]
class LightAppSidebarItem(BaseModel):
app_id: str
name: str
mount_location: str | None = None
sort_order: int = 0
class LightAppSidebarResponse(BaseModel):
apps: list[LightAppSidebarItem]
class LightAppCreateRequest(BaseModel):
name: str = Field(..., min_length=1, max_length=127)
description: str | None = Field(default=None, max_length=1000)
@@ -167,15 +179,34 @@ async def list_light_apps(
tag_ids: list[str] | None = Query(default=None, description="Keep apps carrying any of these tags (OR)"),
) -> LightAppListResponse:
store = _get_store(request)
rows = await store.list_apps(app_type=app_type, mount_location=mount_location, enabled_only=enabled_only)
if search:
needle = search.strip().lower()
rows = [r for r in rows if needle in str(r.get("name", "")).lower() or needle in str(r.get("description") or "").lower()]
responses = await _attach_tags(request, [LightAppResponse(**r) for r in rows])
if tag_ids:
wanted = set(tag_ids)
responses = [r for r in responses if any(t.get("id") in wanted for t in r.tags)]
return LightAppListResponse(apps=responses)
async def load() -> LightAppListResponse:
rows = await store.list_apps(app_type=app_type, mount_location=mount_location, enabled_only=enabled_only)
if search:
needle = search.strip().lower()
rows = [r for r in rows if needle in str(r.get("name", "")).lower() or needle in str(r.get("description") or "").lower()]
responses = await _attach_tags(request, [LightAppResponse(**r) for r in rows])
if tag_ids:
wanted = set(tag_ids)
responses = [r for r in responses if any(t.get("id") in wanted for t in r.tags)]
return LightAppListResponse(apps=responses)
return await cached_catalog(
request, "light_apps", catalog_key("list", app_type, mount_location, enabled_only, search, sorted(tag_ids or [])),
load, LightAppListResponse,
)
@router.get("/sidebar", response_model=LightAppSidebarResponse)
async def list_sidebar_light_apps(request: Request) -> LightAppSidebarResponse:
"""Return the small set of fields required to build the shared sidebar."""
store = _get_store(request)
async def load() -> LightAppSidebarResponse:
rows = await store.list_sidebar_apps()
return LightAppSidebarResponse(apps=[LightAppSidebarItem(**row) for row in rows])
return await cached_catalog(request, "light_apps", "sidebar", load, LightAppSidebarResponse)
@router.post("", response_model=LightAppResponse)
@@ -201,6 +232,7 @@ async def create_light_app(request: Request, body: LightAppCreateRequest) -> Lig
)
except DuplicateLightAppNameError:
raise HTTPException(status_code=409, detail="应用名称已存在")
await invalidate_catalog(request, "light_apps")
attached = await _attach_tags(request, [LightAppResponse(**row)])
return attached[0]
@@ -208,11 +240,15 @@ async def create_light_app(request: Request, body: LightAppCreateRequest) -> Lig
@router.get("/{app_id}", response_model=LightAppResponse)
async def get_light_app(request: Request, app_id: str) -> LightAppResponse:
store = _get_store(request)
row = await store.get_app(app_id)
if row is None:
raise HTTPException(status_code=404, detail="应用不存在")
attached = await _attach_tags(request, [LightAppResponse(**row)])
return attached[0]
async def load() -> LightAppResponse:
row = await store.get_app(app_id)
if row is None:
raise HTTPException(status_code=404, detail="应用不存在")
attached = await _attach_tags(request, [LightAppResponse(**row)])
return attached[0]
return await cached_catalog(request, "light_apps", catalog_key("detail", app_id), load, LightAppResponse)
@router.put("/{app_id}", response_model=LightAppResponse)
@@ -242,6 +278,7 @@ async def update_light_app(request: Request, app_id: str, body: LightAppUpdateRe
raise HTTPException(status_code=409, detail="应用名称已存在")
if row is None:
raise HTTPException(status_code=404, detail="应用不存在")
await invalidate_catalog(request, "light_apps")
attached = await _attach_tags(request, [LightAppResponse(**row)])
return attached[0]
@@ -254,4 +291,5 @@ async def delete_light_app(request: Request, app_id: str) -> dict:
if not deleted:
raise HTTPException(status_code=404, detail="应用不存在")
await _unassign_tags(request, app_id)
await invalidate_catalog(request, "light_apps")
return {"success": True}
offline-backend-20260512/backend/app/gateway/routers/llm_metrics.py
+12 -7
View File
@@ -17,6 +17,7 @@ from fastapi import APIRouter, HTTPException, Query, Request
from fastapi.responses import StreamingResponse
from pydantic import BaseModel, Field
from app.gateway.admin_query_window import resolve_admin_query_window
from app.gateway.deps import get_llm_metrics_store, get_local_provider, get_optional_user_from_request
from deerflow.persistence.llm_metrics.base import LlmMetricsQuery, LlmMetricsStore
@@ -68,9 +69,11 @@ def _require_store(request: Request) -> LlmMetricsStore:
return store
def _parse_dt(value: str | None) -> datetime | None:
def _parse_dt(value: str | datetime | None) -> datetime | None:
if not value:
return None
if isinstance(value, datetime):
return value
try:
# Accept both ``Z`` suffix and ``+00:00``.
return datetime.fromisoformat(value.replace("Z", "+00:00"))
@@ -177,8 +180,8 @@ def _build_query(
user_id: str | None,
model_name: str | None,
status: str | None,
since: str | None,
until: str | None,
since: str | datetime | None,
until: str | datetime | None,
limit: int,
offset: int,
user_ids: list[str] | None = None,
@@ -215,13 +218,14 @@ async def list_llm_metrics(
if not user_ids:
# No username matches → no rows; short-circuit the global scan.
return LlmMetricsListResponse(items=[], total=0, limit=limit, offset=offset)
since_dt, until_dt, _range_days = resolve_admin_query_window(since=since, until=until)
query = _build_query(
user_id=user_id,
user_ids=user_ids,
model_name=model_name,
status=status,
since=since,
until=until,
since=since_dt,
until=until_dt,
limit=limit,
offset=offset,
)
@@ -266,6 +270,7 @@ async def export_llm_metrics(
# No matching user — sentinel id that matches nothing, so the export
# still produces an empty xlsx (matching the list view's empty state).
user_ids = ["__no_such_user__"]
since_dt, until_dt, _range_days = resolve_admin_query_window(since=since, until=until)
try:
from openpyxl import Workbook
@@ -277,8 +282,8 @@ async def export_llm_metrics(
user_ids=user_ids,
model_name=model_name,
status=status,
since=since,
until=until,
since=since_dt,
until=until_dt,
limit=1, # unused: iter_all paginates internally
offset=0,
)
offline-backend-20260512/backend/app/gateway/routers/llmwiki.py
+682 -62
View File
@@ -18,6 +18,7 @@ from io import BytesIO
from types import SimpleNamespace
from typing import Any, Literal
from urllib.parse import quote, unquote, urlencode, urlsplit
from uuid import uuid4
import httpx
from fastapi import APIRouter, BackgroundTasks, File, HTTPException, Query, Request, UploadFile
@@ -26,7 +27,7 @@ from langchain_core.messages import HumanMessage
from pydantic import BaseModel, Field
from app.gateway.csrf_middleware import is_secure_request
from app.gateway.deps import get_agent_store, get_optional_user_from_request, get_thread_store
from app.gateway.deps import get_agent_store, get_checkpointer, get_optional_user_from_request, get_thread_store
from app.gateway.llmwiki_deposit import (
CONVERSATION_DEPOSIT_KB_DESCRIPTION,
CONVERSATION_DEPOSIT_KB_NAME,
@@ -36,6 +37,7 @@ from app.gateway.llmwiki_deposit import (
ensure_conversation_deposit_mapping,
)
from app.gateway.llmwiki_rag import build_llmwiki_rag_context
from app.gateway.perf_logging import finish_trace, trace_request, trace_stage
from app.gateway.weknora_embed import (
WEKNORA_EMBED_API_PREFIX as _WEKNORA_EMBED_API_PREFIX,
)
@@ -60,6 +62,7 @@ from app.gateway.weknora_embed import (
from app.gateway.weknora_static import StaticAsset, StaticAssetCache, prepare_static_asset
from app.gateway.wiki_directory_constraints import (
directory_roots,
ensure_directory_paths,
normalize_directory_paths,
paths_from_parent_directories,
resolve_directory_paths,
@@ -88,6 +91,7 @@ from deerflow.integrations.weknora.runtime import build_weknora_client, get_reso
from deerflow.persistence.llmwiki import LlmWikiStore
logger = logging.getLogger(__name__)
KNOWLEDGE_RESEARCH_DIRECTORY = "深度研究"
@asynccontextmanager
@@ -346,6 +350,20 @@ class WikiPageUpdate(BaseModel):
wiki_path: str | None = Field(default=None, max_length=1024)
class KnowledgeResearchSessionCreate(BaseModel):
question: str = Field(..., min_length=1, max_length=20_000)
class KnowledgeResearchSessionUpdate(BaseModel):
title: str | None = Field(default=None, min_length=1, max_length=255)
draft_markdown: str | None = Field(default=None, max_length=2_000_000)
class KnowledgeResearchPublicationSave(BaseModel):
title: str = Field(..., min_length=1, max_length=255)
expected_saved_hash: str | None = Field(default=None, min_length=64, max_length=64)
class ConversationDepositCreate(BaseModel):
thread_id: str = Field(..., min_length=1, max_length=128)
question: str = Field(default="", max_length=200_000)
@@ -505,6 +523,28 @@ def _select_embedding_model(models: list[dict[str, Any]], requested_id: str | No
return str(selected["id"])
def _select_knowledge_qa_model(models: list[dict[str, Any]]) -> str:
available = [
item
for item in models
if str(item.get("type") or "").strip().lower() == "knowledgeqa"
and str(item.get("id") or "").strip()
and str(item.get("status") or "active").strip().lower() == "active"
]
available.sort(
key=lambda item: (
0 if item.get("is_default") is True else 1,
str(item.get("id") or "").strip(),
)
)
if not available:
raise WeKnoraError(
"WeKnora has no active KnowledgeQA model; configure one before creating a Wiki",
status_code=400,
)
return str(available[0]["id"]).strip()
def _json_response(payload: Any, *, status_code: int = 200) -> Response:
return Response(
content=json.dumps(payload, ensure_ascii=False, separators=(",", ":")).encode("utf-8"),
@@ -2326,7 +2366,7 @@ def _mapping_view(row: dict[str, Any], remote: dict[str, Any] | None, *, actor_u
})
return {
"id": row["id"],
"remote_status": "available" if remote is not None else "missing",
"remote_status": "cached" if source.get("_local_cache_only") else "available" if remote is not None else "missing",
"name": display_name,
"description": description,
"test_questions": resolved_test_questions(profile_row),
@@ -3466,42 +3506,76 @@ async def list_knowledge_bases(
scope: Literal["personal", "public", "all"] = Query(default="all"),
refresh: bool = False,
) -> dict[str, Any]:
client = _client_or_503()
user_id, is_admin = await _actor(request)
trace = trace_request(request, "llmwiki_knowledge_bases")
try:
remote_items = await client.list_knowledge_bases(refresh=refresh)
remote = {str(item.get("id")): item for item in remote_items if item.get("id")}
except WeKnoraError as exc:
_raise_upstream(exc)
deposit_remote = next(
(item for item in remote_items if str(item.get("name") or "").strip() == CONVERSATION_DEPOSIT_KB_NAME),
None,
)
if deposit_remote is not None:
previous_mapping = await _store(request).get_by_weknora_id(str(deposit_remote.get("id") or ""))
deposit_mapping = await ensure_conversation_deposit_mapping(
_store(request),
client,
remote_knowledge_bases=remote_items,
)
description_was_synchronized = previous_mapping is None or not _is_conversation_deposit_mapping(previous_mapping) or str(previous_mapping.get("description") or "") != CONVERSATION_DEPOSIT_KB_DESCRIPTION
if deposit_mapping is not None and description_was_synchronized:
remote_id = str(deposit_mapping["weknora_id"])
remote[remote_id] = {
**remote.get(remote_id, deposit_remote),
"description": CONVERSATION_DEPOSIT_KB_DESCRIPTION,
}
rows = await _store(request).list_visible(user_id, scope=scope, is_admin=False)
knowledge_bases: list[dict[str, Any]] = []
for row in rows:
remote_item = remote.get(str(row["weknora_id"]))
# A successful remote list is authoritative for every scope. Keep the
# mapping for existing references, but never display a deleted base.
if remote_item is None:
continue
row, remote_item = await _sync_conversation_deposit_view(_store(request), client, row, remote_item)
knowledge_bases.append(_mapping_view(row, remote_item, actor_user_id=user_id, is_admin=is_admin))
return {"knowledge_bases": knowledge_bases}
stage_started = time.perf_counter()
user_id, is_admin = await _actor(request)
trace_stage(trace, "actor_and_access", stage_started)
stage_started = time.perf_counter()
store = _store(request)
rows = await store.list_visible(user_id, scope=scope, is_admin=False)
trace_stage(trace, "local_knowledge_base_query_db", stage_started)
client: WeKnoraClient | None = None
remote_items: list[dict[str, Any]] = []
remote: dict[str, dict[str, Any]] = {}
if refresh:
stage_started = time.perf_counter()
client = _client_or_503()
try:
remote_items = await client.list_knowledge_bases(refresh=True)
remote = {str(item.get("id")): item for item in remote_items if item.get("id")}
except WeKnoraError as exc:
_raise_upstream(exc)
trace_stage(trace, "weknora_explicit_refresh_wait", stage_started)
deposit_remote = next(
(item for item in remote_items if str(item.get("name") or "").strip() == CONVERSATION_DEPOSIT_KB_NAME),
None,
)
if deposit_remote is not None:
previous_mapping = await store.get_by_weknora_id(str(deposit_remote.get("id") or ""))
deposit_mapping = await ensure_conversation_deposit_mapping(
store,
client,
remote_knowledge_bases=remote_items,
)
description_was_synchronized = previous_mapping is None or not _is_conversation_deposit_mapping(previous_mapping) or str(previous_mapping.get("description") or "") != CONVERSATION_DEPOSIT_KB_DESCRIPTION
if deposit_mapping is not None and description_was_synchronized:
remote_id = str(deposit_mapping["weknora_id"])
remote[remote_id] = {
**remote.get(remote_id, deposit_remote),
"description": CONVERSATION_DEPOSIT_KB_DESCRIPTION,
}
trace_stage(trace, "conversation_deposit_mapping_sync", stage_started)
stage_started = time.perf_counter()
knowledge_bases: list[dict[str, Any]] = []
for row in rows:
if refresh:
remote_item = remote.get(str(row["weknora_id"]))
# An explicit successful refresh is authoritative: do not show
# mappings whose remote knowledge base has been deleted.
if remote_item is None:
continue
assert client is not None
row, remote_item = await _sync_conversation_deposit_view(store, client, row, remote_item)
else:
# Normal page loads are served entirely from the local SQL
# mapping. A remote check happens only after an explicit refresh.
remote_item = {
"id": str(row.get("weknora_id") or ""),
"name": str(row.get("name") or ""),
"description": str(row.get("description") or ""),
"_local_cache_only": True,
}
knowledge_bases.append(_mapping_view(row, remote_item, actor_user_id=user_id, is_admin=is_admin))
trace_stage(trace, "map_visible_knowledge_bases", stage_started)
trace["details"].update({"scope": scope, "refresh": refresh, "remote_count": len(remote_items) if refresh else None, "local_count": len(rows), "result_count": len(knowledge_bases)})
return {"knowledge_bases": knowledge_bases}
finally:
finish_trace(trace)
@router.get("/api/llmwiki/embedding-models")
@@ -3523,37 +3597,62 @@ async def list_embedding_models(request: Request) -> dict[str, Any]:
async def list_agent_knowledge_bases(request: Request, agent_id: str) -> dict[str, Any]:
"""Return the agent-bound bases still visible to the current DeerFlow user."""
user_id, is_admin = await _actor(request)
agent_store = get_agent_store(request)
agent = await (agent_store.get_any(agent_id) if is_admin else agent_store.get_visible(agent_id, user_id))
if agent is None:
raise HTTPException(status_code=404, detail="Agent not found")
trace = trace_request(request, "llmwiki_agent_knowledge_bases")
try:
config = load_agent_config(agent_id)
except (FileNotFoundError, ValueError):
config = None
bound_ids = list(config.llmwiki_knowledge_base_ids or []) if config else []
rows: list[dict[str, Any]] = []
for mapping_id in bound_ids:
row = await _store(request).get_authorized(
mapping_id,
user_id,
write=False,
is_admin=is_admin,
)
if row is not None:
rows.append(row)
try:
remote = {str(item.get("id")): item for item in await _client_or_503().list_knowledge_bases() if item.get("id")}
except WeKnoraError as exc:
_raise_upstream(exc)
return {"knowledge_bases": [_mapping_view(row, remote_item, actor_user_id=user_id, is_admin=is_admin) for row in rows if (remote_item := remote.get(str(row["weknora_id"]))) is not None]}
stage_started = time.perf_counter()
user_id, is_admin = await _actor(request)
agent_store = get_agent_store(request)
agent = await (agent_store.get_any(agent_id) if is_admin else agent_store.get_visible(agent_id, user_id))
if agent is None:
raise HTTPException(status_code=404, detail="Agent not found")
trace_stage(trace, "actor_and_agent_access", stage_started)
stage_started = time.perf_counter()
try:
config = load_agent_config(agent_id)
except (FileNotFoundError, ValueError):
config = None
bound_ids = list(config.llmwiki_knowledge_base_ids or []) if config else []
trace_stage(trace, "load_agent_binding_config", stage_started)
stage_started = time.perf_counter()
store = _store(request)
rows: list[dict[str, Any]] = []
for mapping_id in bound_ids:
row = await store.get_authorized(
mapping_id,
user_id,
write=False,
is_admin=is_admin,
)
if row is not None:
rows.append(row)
trace_stage(trace, "local_knowledge_base_mapping_query", stage_started)
stage_started = time.perf_counter()
knowledge_bases = []
for row in rows:
remote_item = {
"id": str(row.get("weknora_id") or ""),
"name": str(row.get("name") or ""),
"description": str(row.get("description") or ""),
"_local_cache_only": True,
}
knowledge_bases.append(_mapping_view(row, remote_item, actor_user_id=user_id, is_admin=is_admin))
trace_stage(trace, "map_agent_knowledge_bases", stage_started)
trace["details"].update({"bound_count": len(bound_ids), "remote_count": None, "result_count": len(knowledge_bases)})
return {"knowledge_bases": knowledge_bases}
finally:
finish_trace(trace)
@router.post("/api/llmwiki/knowledge-bases", status_code=201)
async def create_knowledge_base(request: Request, body: KnowledgeBaseCreate) -> dict[str, Any]:
client = _client_or_503()
trace = trace_request(request, "llmwiki_knowledge_base_create")
stage_started = time.perf_counter()
user_id, is_admin = await _actor(request)
trace_stage(trace, "actor_and_access", stage_started)
if body.name.strip() == CONVERSATION_DEPOSIT_KB_NAME:
raise HTTPException(status_code=409, detail="对话沉淀是 cmzs 系统保留知识库")
wiki_config = body.wiki_config if body.wiki_enabled else None
@@ -3595,16 +3694,24 @@ async def create_knowledge_base(request: Request, body: KnowledgeBaseCreate) ->
except ValueError as exc:
raise HTTPException(status_code=422, detail=str(exc)) from None
try:
stage_started = time.perf_counter()
upstream_models = await client.list_models()
embedding_model_id = _select_embedding_model(
await _embedding_models(client),
[view for item in upstream_models if (view := _embedding_model_view(item)) is not None],
body.embedding_model_id,
)
knowledge_qa_model_id = (
_select_knowledge_qa_model(upstream_models) if body.wiki_enabled else None
)
trace_stage(trace, "weknora_model_catalog", stage_started)
stage_started = time.perf_counter()
remote = await client.create_knowledge_base(
name=body.name.strip(),
description=body.description,
kb_type=body.type,
wiki_enabled=body.wiki_enabled,
embedding_model_id=embedding_model_id,
knowledge_qa_model_id=knowledge_qa_model_id,
wiki_content_instructions=(
compiled.content_instructions if body.wiki_enabled else ""
),
@@ -3612,11 +3719,14 @@ async def create_knowledge_base(request: Request, body: KnowledgeBaseCreate) ->
compiled.extraction_instructions if body.wiki_enabled else ""
),
)
trace_stage(trace, "weknora_create", stage_started)
except WeKnoraError as exc:
finish_trace(trace)
_raise_upstream(exc)
try:
# Directory settings are references for WeKnora's generator. Creating
# folders here would turn the reference into an accidental template.
stage_started = time.perf_counter()
row = await _store(request).create_mapping(
weknora_id=str(remote["id"]),
owner_user_id=user_id,
@@ -3643,7 +3753,9 @@ async def create_knowledge_base(request: Request, body: KnowledgeBaseCreate) ->
),
wiki_config_apply_error=None,
)
trace_stage(trace, "local_mapping_create", stage_started)
except Exception as exc:
finish_trace(trace)
logger.exception(
"Failed to initialize or persist LLMWiki mapping; compensating remote create"
)
@@ -3655,6 +3767,7 @@ async def create_knowledge_base(request: Request, body: KnowledgeBaseCreate) ->
_raise_upstream(exc)
raise HTTPException(status_code=500, detail="Failed to register the knowledge base") from None
_schedule_local_wiki_sync(request, row)
finish_trace(trace)
return _mapping_view(row, remote, actor_user_id=user_id, is_admin=is_admin)
@@ -4056,6 +4169,513 @@ async def get_knowledge_base_graph(
return {"nodes": nodes, "edges": edges, "meta": meta}
def _knowledge_research_store(request: Request):
store = getattr(request.app.state, "knowledge_research_store", None)
if store is None:
raise HTTPException(status_code=503, detail="知识库研究存储尚未就绪")
return store
async def _knowledge_research_can_write(
request: Request,
mapping: dict[str, Any],
user_id: str,
is_admin: bool,
) -> bool:
if _is_conversation_deposit_mapping(mapping):
return False
writable = await _store(request).get_authorized(
str(mapping["id"]),
user_id,
write=True,
is_admin=is_admin,
)
return writable is not None
def _knowledge_research_publication_view(
row: dict[str, Any],
*,
user_id: str,
can_write: bool,
include_content: bool = False,
) -> dict[str, Any] | None:
if not row.get("saved_hash"):
return None
owned = row.get("user_id") == user_id
result: dict[str, Any] = {
"id": row["publication_id"],
"title": row.get("saved_title") or row.get("title") or "",
"wiki_slug": row.get("wiki_slug") or "",
"saved_hash": row["saved_hash"],
"saved_at": row.get("saved_at"),
"updated_at": row.get("saved_at") or row.get("updated_at"),
"owned_by_current_user": owned,
"can_continue": owned,
"can_write": owned and can_write,
}
if owned:
# Only the creator receives a bridge from the public book to the
# private research session. The thread id remains session-only.
result["session_id"] = row["id"]
if include_content:
result["content_markdown"] = row.get("saved_markdown") or ""
return result
def _knowledge_research_session_view(
row: dict[str, Any],
*,
user_id: str,
can_write: bool,
) -> dict[str, Any]:
return {
"id": row["id"],
"thread_id": row["thread_id"],
"title": row.get("title") or "",
"draft_markdown": row.get("draft_markdown") or "",
"created_at": row.get("created_at"),
"updated_at": row.get("updated_at"),
"can_write": can_write,
"publication": _knowledge_research_publication_view(
row,
user_id=user_id,
can_write=can_write,
),
}
def _knowledge_research_default_title(question: str) -> str:
for line in question.splitlines():
normalized = " ".join(line.split())
if normalized:
return normalized[:255]
raise HTTPException(status_code=422, detail="问题不能为空")
@router.get("/api/llmwiki/knowledge-bases/{mapping_id}/research/sessions")
async def list_knowledge_research_sessions(
request: Request,
mapping_id: str,
) -> dict[str, Any]:
mapping, user_id, is_admin = await _authorized_mapping(
request,
mapping_id,
write=False,
)
can_write = await _knowledge_research_can_write(
request,
mapping,
user_id,
is_admin,
)
store = _knowledge_research_store(request)
sessions, publications = await asyncio.gather(
store.list_owned(mapping_id, user_id),
store.list_published(mapping_id),
)
return {
"sessions": [
_knowledge_research_session_view(
row,
user_id=user_id,
can_write=can_write,
)
for row in sessions
],
"publications": [
view
for row in publications
if (
view := _knowledge_research_publication_view(
row,
user_id=user_id,
can_write=can_write,
)
)
is not None
],
}
@router.post("/api/llmwiki/knowledge-bases/{mapping_id}/research/sessions", status_code=201)
async def create_knowledge_research_session(
request: Request,
mapping_id: str,
body: KnowledgeResearchSessionCreate,
) -> dict[str, Any]:
mapping, user_id, is_admin = await _authorized_mapping(
request,
mapping_id,
write=False,
)
can_write = await _knowledge_research_can_write(
request,
mapping,
user_id,
is_admin,
)
question = body.question.strip()
title = _knowledge_research_default_title(question)
store = _knowledge_research_store(request)
session_id = str(uuid4())
publication_id = str(uuid4())
thread_id = str(uuid4())
session = await store.create(
session_id=session_id,
publication_id=publication_id,
thread_id=thread_id,
knowledge_base_id=mapping_id,
user_id=user_id,
title=title,
)
metadata = {
"system": True,
"thread_type": "knowledge_research",
"knowledge_research_session_id": session_id,
"knowledge_research_knowledge_base_id": mapping_id,
"llmwiki_knowledge_base_ids": [mapping_id],
"assistant_knowledge_base_ids": [],
"knowledge_retrieval_mode": "classic",
"knowledge_space_strict": True,
"rag_mode_enabled": True,
"rag_mode": True,
}
try:
from app.gateway.routers.threads import ThreadCreateRequest, create_thread
await create_thread(
ThreadCreateRequest(
thread_id=thread_id,
assistant_id=None,
metadata=metadata,
initialize_checkpoint=True,
),
request,
)
await get_thread_store(request).update_display_name(
thread_id,
title,
user_id=user_id,
)
except Exception:
# Thread creation writes metadata before its initial checkpoint. Undo
# every side on failure so no half-bound private chat remains.
await store.delete(session_id)
try:
checkpointer = get_checkpointer(request)
if hasattr(checkpointer, "adelete_thread"):
await checkpointer.adelete_thread(thread_id)
except Exception:
logger.debug("Could not remove failed research checkpoint", exc_info=True)
try:
await get_thread_store(request).delete(thread_id, user_id=user_id)
except Exception:
logger.debug("Could not remove failed research thread metadata", exc_info=True)
raise
return _knowledge_research_session_view(
session,
user_id=user_id,
can_write=can_write,
)
async def _owned_knowledge_research_session(
request: Request,
mapping_id: str,
session_id: str,
) -> tuple[dict[str, Any], dict[str, Any], str, bool, bool]:
mapping, user_id, is_admin = await _authorized_mapping(
request,
mapping_id,
write=False,
)
session = await _knowledge_research_store(request).get_owned(
session_id,
user_id,
)
if session is None or session["knowledge_base_id"] != mapping_id:
raise HTTPException(status_code=404, detail="研究会话不存在或无权访问")
can_write = await _knowledge_research_can_write(
request,
mapping,
user_id,
is_admin,
)
return session, mapping, user_id, is_admin, can_write
@router.get(
"/api/llmwiki/knowledge-bases/{mapping_id}/research/sessions/{session_id}"
)
async def get_knowledge_research_session(
request: Request,
mapping_id: str,
session_id: str,
) -> dict[str, Any]:
session, _, user_id, _, can_write = await _owned_knowledge_research_session(
request,
mapping_id,
session_id,
)
return _knowledge_research_session_view(
session,
user_id=user_id,
can_write=can_write,
)
@router.patch(
"/api/llmwiki/knowledge-bases/{mapping_id}/research/sessions/{session_id}"
)
async def update_knowledge_research_session(
request: Request,
mapping_id: str,
session_id: str,
body: KnowledgeResearchSessionUpdate,
) -> dict[str, Any]:
current, _, user_id, _, can_write = await _owned_knowledge_research_session(
request,
mapping_id,
session_id,
)
fields = body.model_fields_set
if not fields:
return _knowledge_research_session_view(
current,
user_id=user_id,
can_write=can_write,
)
title = body.title.strip() if "title" in fields and body.title is not None else None
if "title" in fields and not title:
raise HTTPException(status_code=422, detail="标题不能为空")
updated = await _knowledge_research_store(request).update_owned(
session_id,
user_id,
title=title,
draft_markdown=(
body.draft_markdown if "draft_markdown" in fields else None
),
)
if updated is None:
raise HTTPException(status_code=404, detail="研究会话不存在或无权访问")
if title is not None:
try:
await get_thread_store(request).update_display_name(
updated["thread_id"],
title,
user_id=user_id,
)
except Exception:
logger.debug("Could not update research thread title", exc_info=True)
return _knowledge_research_session_view(
updated,
user_id=user_id,
can_write=can_write,
)
async def _upsert_knowledge_research_wiki_page(
*,
client: WeKnoraClient,
knowledge_base_id: str,
slug: str,
payload: dict[str, Any],
previously_saved: bool,
) -> dict[str, Any]:
if previously_saved:
try:
return await client.update_wiki_page(
knowledge_base_id,
slug,
payload,
)
except WeKnoraError as exc:
if exc.status_code != 404:
raise
return await client.create_wiki_page(
knowledge_base_id,
{"slug": slug, **payload},
)
try:
return await client.create_wiki_page(
knowledge_base_id,
{"slug": slug, **payload},
)
except WeKnoraError as create_error:
# A prior request may have committed upstream before its local DB write
# failed. Probe the stable slug and turn the retry into an update.
if create_error.status_code not in {400, 409}:
raise
try:
await client.get_wiki_page(knowledge_base_id, slug)
except WeKnoraError:
raise create_error
return await client.update_wiki_page(knowledge_base_id, slug, payload)
@router.put(
"/api/llmwiki/knowledge-bases/{mapping_id}/research/sessions/{session_id}/publication"
)
async def save_knowledge_research_publication(
request: Request,
mapping_id: str,
session_id: str,
body: KnowledgeResearchPublicationSave,
) -> dict[str, Any]:
# Write authorization and private-session ownership are independent: even
# an administrator may not publish another user's private conversation.
mapping, user_id, _ = await _authorized_mapping(request, mapping_id, write=True)
if _is_conversation_deposit_mapping(mapping):
raise HTTPException(
status_code=403,
detail="Conversation deposit is managed by cmzs",
)
title = body.title.strip()
if not title:
raise HTTPException(status_code=422, detail="标题不能为空")
store = _knowledge_research_store(request)
client = _client_or_503()
remote_id = str(mapping["weknora_id"])
saved: dict[str, Any] | None = None
async with store.locked_publication(session_id, user_id) as publication:
if publication is None:
raise HTTPException(status_code=404, detail="研究会话不存在或无权访问")
session = publication.snapshot()
if session["knowledge_base_id"] != mapping_id:
raise HTTPException(status_code=404, detail="研究会话不存在或无权访问")
markdown = str(session.get("draft_markdown") or "")
if not markdown.strip():
raise HTTPException(status_code=422, detail="请先生成或填写 Markdown 正文")
digest = hashlib.sha256(
json.dumps(
{"title": title, "markdown": markdown},
ensure_ascii=False,
sort_keys=True,
separators=(",", ":"),
).encode("utf-8")
).hexdigest()
current_saved_hash = session.get("saved_hash")
if current_saved_hash == digest:
# A prior identical request may have committed both upstream and
# locally while its HTTP response was lost. Treat that retry as
# a no-op even if it carries the pre-save expected hash.
return _knowledge_research_session_view(
session,
user_id=user_id,
can_write=True,
)
if current_saved_hash and body.expected_saved_hash is None:
raise HTTPException(
status_code=409,
detail={
"code": "KNOWLEDGE_RESEARCH_EXPECTED_HASH_REQUIRED",
"message": "覆盖已保存文档时必须携带当前版本，请刷新后再保存。",
"saved_hash": current_saved_hash,
},
)
if (
body.expected_saved_hash is not None
and body.expected_saved_hash != current_saved_hash
):
raise HTTPException(
status_code=409,
detail={
"code": "KNOWLEDGE_RESEARCH_PUBLICATION_CHANGED",
"message": "文档已被更新，请刷新后再保存。",
"saved_hash": current_saved_hash,
},
)
slug = str(
session.get("wiki_slug")
or f"knowledge-research/{session['publication_id']}"
)
try:
folder_ids = await ensure_directory_paths(
client,
remote_id,
[[KNOWLEDGE_RESEARCH_DIRECTORY]],
)
folder_id = folder_ids.get(KNOWLEDGE_RESEARCH_DIRECTORY)
if not folder_id:
raise WeKnoraError("未能创建深度研究 Wiki 目录", status_code=502)
page = await _upsert_knowledge_research_wiki_page(
client=client,
knowledge_base_id=remote_id,
slug=slug,
previously_saved=bool(current_saved_hash),
payload={
"title": title,
"content": markdown,
"summary": "知识库问答沉淀",
"page_type": "summary",
"status": "published",
"category_path": [KNOWLEDGE_RESEARCH_DIRECTORY],
"folder_id": folder_id,
},
)
except WeKnoraError as exc:
if exc.status_code == 409:
raise HTTPException(status_code=409, detail=str(exc)) from None
_raise_upstream(exc)
saved = publication.save(
title=title,
markdown=markdown,
saved_hash=digest,
wiki_slug=slug,
wiki_page_id=str(page.get("id") or page.get("page_id") or "")
or None,
wiki_folder_id=folder_id,
)
assert saved is not None
_schedule_local_wiki_sync(request, mapping)
return _knowledge_research_session_view(
saved,
user_id=user_id,
can_write=True,
)
@router.get(
"/api/llmwiki/knowledge-bases/{mapping_id}/research/publications/{publication_id}"
)
async def get_knowledge_research_publication(
request: Request,
mapping_id: str,
publication_id: str,
) -> dict[str, Any]:
mapping, user_id, is_admin = await _authorized_mapping(
request,
mapping_id,
write=False,
)
publication = await _knowledge_research_store(request).get_publication(
publication_id
)
if (
publication is None
or publication["knowledge_base_id"] != mapping_id
):
raise HTTPException(status_code=404, detail="研究文档不存在")
can_write = await _knowledge_research_can_write(
request,
mapping,
user_id,
is_admin,
)
view = _knowledge_research_publication_view(
publication,
user_id=user_id,
can_write=can_write,
include_content=True,
)
if view is None:
raise HTTPException(status_code=404, detail="研究文档不存在")
return view
@router.get("/api/llmwiki/knowledge-bases/{mapping_id}/wiki/pages")
async def list_wiki_pages(
request: Request,
offline-backend-20260512/backend/app/gateway/routers/menu_overrides.py
+12 -3
View File
@@ -29,6 +29,7 @@ from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel, Field
from app.gateway.auth.disabled_mode import is_auth_disabled
from app.gateway.catalog_cache import cached_catalog, catalog_key, invalidate_catalog
from app.gateway.deps import get_optional_user_from_request
from deerflow.persistence.menu_overrides import MenuOverrideStore
@@ -116,9 +117,11 @@ def _merge_overrides(default_rows: list[dict], user_rows: list[dict] | None) ->
return merged
async def _read_response(request: Request, store: MenuOverrideStore) -> MenuOverrideListResponse:
async def _read_response(
request: Request, store: MenuOverrideStore, *, actor: tuple[str | None, bool] | None = None,
) -> MenuOverrideListResponse:
user_id, is_admin = actor if actor is not None else await _actor(request)
default_rows = await store.list_overrides()
user_id, is_admin = await _actor(request)
if is_admin or user_id is None:
return MenuOverrideListResponse(
overrides=[MenuOverrideResponse(**row) for row in default_rows],
@@ -191,8 +194,12 @@ def _validate_depth(
@router.get("", response_model=MenuOverrideListResponse)
async def list_menu_overrides(request: Request) -> MenuOverrideListResponse:
actor = await _actor(request)
store = _get_store(request)
return await _read_response(request, store)
return await cached_catalog(
request, "menu_overrides", catalog_key(*actor),
lambda: _read_response(request, store, actor=actor), MenuOverrideListResponse,
)
@router.put("", response_model=MenuOverrideListResponse)
@@ -214,6 +221,7 @@ async def replace_menu_overrides(request: Request, body: MenuOverrideReplaceRequ
await store.replace_all(payload, updated_by=user_id)
else:
await store.replace_user_overrides(user_id, payload, updated_by=user_id)
await invalidate_catalog(request, "menu_overrides")
return await _read_response(request, store)
@@ -226,4 +234,5 @@ async def reset_menu_overrides(request: Request) -> MenuOverrideListResponse:
await store.replace_all([], updated_by=user_id)
else:
await store.replace_user_overrides(user_id, [], updated_by=user_id)
await invalidate_catalog(request, "menu_overrides")
return await _read_response(request, store)
offline-backend-20260512/backend/app/gateway/routers/model_management.py
+266 -10
View File
@@ -17,15 +17,17 @@ from __future__ import annotations
import asyncio
import copy
import errno
import logging
import os
import re
import shutil
import sys
import tempfile
from datetime import UTC, datetime
from pathlib import Path
from time import perf_counter
from typing import Any
from time import perf_counter, sleep
from typing import Any, Literal
import yaml
from fastapi import APIRouter, HTTPException, Request
@@ -45,6 +47,56 @@ _MODEL_BLOCK_RE = re.compile(r"(?m)^models:[^\r\n]*(?:\r?\n|$)")
_TOP_LEVEL_KEY_RE = re.compile(r"(?m)^[A-Za-z_][A-Za-z0-9_-]*:[^\r\n]*(?:\r?\n|$)")
_BACKUP_DIR_NAME = ".model-config-backups"
_MUTATION_LOCK = asyncio.Lock()
_ATOMIC_REPLACE_RETRIES = 5
_ATOMIC_REPLACE_RETRY_ERRNOS = {
errno.EBUSY,
errno.EAGAIN,
getattr(errno, "ETXTBSY", 26),
}
def _is_file_mountpoint(path: Path) -> bool:
"""Return whether ``path`` itself is mounted (common with Docker file binds)."""
if not sys.platform.startswith("linux"):
return False
try:
target = str(path.resolve())
with Path("/proc/self/mountinfo").open(encoding="utf-8") as mountinfo:
for line in mountinfo:
fields = line.split(" - ", maxsplit=1)[0].split()
if len(fields) < 5:
continue
mount_path = re.sub(
r"\\([0-7]{3})",
lambda match: chr(int(match.group(1), 8)),
fields[4],
)
if mount_path == target:
return True
except OSError:
logger.debug("Unable to inspect Linux mount table for model config path", exc_info=True)
return False
def _write_mounted_config_in_place(path: Path, raw_before: str, raw_after: str) -> None:
"""Update a single-file bind mount, where Linux refuses atomic rename (EBUSY)."""
try:
with path.open("w", encoding="utf-8", newline="") as handle:
handle.write(raw_after)
handle.flush()
os.fsync(handle.fileno())
except Exception:
# The backup was persisted before reaching this point. Restore the
# previous contents best-effort so a failed write does not leave YAML
# truncated or partially replaced.
try:
with path.open("w", encoding="utf-8", newline="") as handle:
handle.write(raw_before)
handle.flush()
os.fsync(handle.fileno())
except OSError:
logger.exception("Failed to restore model config after an in-place bind-mount write failure")
raise
_EDITOR_FIELDS = frozenset(
{
@@ -70,6 +122,7 @@ _EDITOR_FIELDS = frozenset(
"use_responses_api",
"output_version",
"write_file_transport",
"strict_stream_integrity",
}
)
_PRESERVED_FIELDS = frozenset(ModelConfig.model_fields) | _EDITOR_FIELDS
@@ -92,6 +145,7 @@ class ModelAdminItem(BaseModel):
stream_usage: bool | None = None
supports_thinking: bool = False
supports_reasoning_effort: bool = False
default_reasoning_effort: Literal["low", "medium", "xhigh"] = "xhigh"
supports_vision: bool = False
when_thinking_enabled: dict[str, Any] | None = None
when_thinking_disabled: dict[str, Any] | None = None
@@ -99,6 +153,9 @@ class ModelAdminItem(BaseModel):
use_responses_api: bool | None = None
output_version: str | None = None
write_file_transport: WriteFileTransport = WriteFileTransport.NATIVE
# ``None`` means the runtime default: auto-enabled only for the historically
# affected qwen3.8-flash-next model id (see deerflow.models.chat_stream_state).
strict_stream_integrity: bool | None = None
extra_settings: dict[str, Any] = Field(default_factory=dict)
api_key_configured: bool = False
api_key_source: str | None = None
@@ -146,6 +203,9 @@ class ModelUpsertRequest(BaseModel):
# bridge setting during edits. New models still resolve to ModelConfig's
# native default when the field is omitted.
write_file_transport: WriteFileTransport | None = None
# Optional on the wire so edits preserve an existing toggle; the dedicated
# strict-stream-integrity endpoint is the canonical switch surface.
strict_stream_integrity: bool | None = None
extra_settings: dict[str, Any] = Field(default_factory=dict)
@field_validator("name", "use", "model")
@@ -168,6 +228,20 @@ class ModelTestRequest(BaseModel):
class ReasoningEffortCapabilityRequest(BaseModel):
supports_reasoning_effort: bool
default_reasoning_effort: Literal["low", "medium", "xhigh"] | None = None
class WriteFileTransportToggleRequest(BaseModel):
"""Per-model 文件流式兼容 switch: off keeps native, on selects buffered_gateway."""
write_file_transport: Literal["native", "buffered_gateway"]
class StrictStreamIntegrityToggleRequest(BaseModel):
"""Per-model 流完整性检测 switch: detect gateway streams that end without
a terminal frame and retry instead of committing a half answer."""
strict_stream_integrity: bool
class ModelOrderRequest(BaseModel):
@@ -193,6 +267,7 @@ def _to_admin_item(
raw: dict[str, Any],
*,
supports_reasoning_effort: bool = False,
default_reasoning_effort: str = "xhigh",
) -> ModelAdminItem:
api_key = raw.get("api_key")
extra = {key: copy.deepcopy(value) for key, value in raw.items() if key not in _PRESERVED_FIELDS}
@@ -211,6 +286,7 @@ def _to_admin_item(
stream_usage=raw.get("stream_usage"),
supports_thinking=bool(raw.get("supports_thinking", False)),
supports_reasoning_effort=bool(supports_reasoning_effort),
default_reasoning_effort=default_reasoning_effort if default_reasoning_effort in {"low", "medium", "xhigh"} else "xhigh",
supports_vision=bool(raw.get("supports_vision", False)),
when_thinking_enabled=copy.deepcopy(raw.get("when_thinking_enabled")),
when_thinking_disabled=copy.deepcopy(raw.get("when_thinking_disabled")),
@@ -218,6 +294,7 @@ def _to_admin_item(
use_responses_api=raw.get("use_responses_api"),
output_version=raw.get("output_version"),
write_file_transport=raw.get("write_file_transport", WriteFileTransport.NATIVE),
strict_stream_integrity=raw.get("strict_stream_integrity") if isinstance(raw.get("strict_stream_integrity"), bool) else None,
extra_settings=extra,
api_key_configured=bool(api_key),
api_key_source=_mask_env_source(api_key),
@@ -236,12 +313,12 @@ def _get_capability_store(request: Request, *, required: bool = True) -> ModelCa
return store
async def _capabilities_by_name(request: Request) -> dict[str, bool]:
async def _capabilities_by_name(request: Request) -> dict[str, dict[str, Any]]:
store = _get_capability_store(request, required=False)
if store is None:
return {}
rows = await store.list_capabilities()
return {str(row.get("model_name") or ""): bool(row.get("supports_reasoning_effort", False)) for row in rows if row.get("model_name")}
return {str(row.get("model_name") or ""): row for row in rows if row.get("model_name")}
def _admin_user_id(user: Any) -> str | None:
@@ -257,10 +334,48 @@ def _validate_reasoning_capability(model: dict[str, Any], enabled: bool) -> None
)
# ``write_file_transport=buffered_gateway`` and ``strict_stream_integrity`` are
# implemented only by the patched OpenAI-compatible chat-completions classes
# (see deerflow.models.factory); every other driver is rejected there at
# construction time, so mirror the same constraint here with a clearer message.
_PATCHED_OPENAI_COMPATIBLE_USES = frozenset(
{
"langchain_openai:ChatOpenAI",
"deerflow.models.patched_openai:PatchedChatOpenAI",
}
)
def _validate_patched_openai_use(model: dict[str, Any], feature: str) -> None:
if str(model.get("use") or "") not in _PATCHED_OPENAI_COMPATIBLE_USES:
raise HTTPException(
status_code=422,
detail=(
f"{feature} requires an OpenAI-compatible "
"langchain_openai:ChatOpenAI model"
),
)
if bool(model.get("use_responses_api")):
raise HTTPException(
status_code=422,
detail=(
f"{feature} requires the chat-completions API, "
"not use_responses_api"
),
)
def _validate_write_file_transport(model: dict[str, Any], transport: str) -> None:
if transport != WriteFileTransport.BUFFERED_GATEWAY.value:
return
_validate_patched_openai_use(model, "write_file compatibility")
async def _restore_capability(
store: ModelCapabilityStore,
model_name: str,
previous: dict[str, Any] | None,
request: Request,
) -> None:
"""Best-effort semantic rollback for a failed cross-store model update."""
@@ -271,10 +386,15 @@ async def _restore_capability(
await store.set_reasoning_effort_support(
model_name,
supports_reasoning_effort=bool(previous.get("supports_reasoning_effort", False)),
default_reasoning_effort=str(previous.get("default_reasoning_effort") or "xhigh"),
updated_by=previous.get("updated_by"),
)
except Exception:
logger.exception("Failed to roll back model capability for %s", model_name)
finally:
from app.gateway.catalog_cache import invalidate_catalog
await invalidate_catalog(request, "models")
def _read_config_data(path: Path) -> tuple[str, dict[str, Any], list[dict[str, Any]]]:
@@ -328,6 +448,42 @@ def _replace_models_block(raw_text: str, models: list[dict[str, Any]]) -> str:
return f"{raw_text[: block_match.start()]}{rendered}{raw_text[end:]}"
def _replace_config_file_with_retry(
temp_name: str,
path: Path,
raw_before: str,
raw_after: str,
) -> None:
"""Retry short-lived busy errors from shared/container-mounted filesystems.
Keep the write atomic: never fall back to truncating the live config. The
temporary file is unique per mutation, so a bounded retry is safe when
overlay/NFS scanners briefly hold a rename operation.
"""
if _is_file_mountpoint(path):
logger.info("Model config is a single-file mount; updating mounted file in place")
_write_mounted_config_in_place(path, raw_before, raw_after)
os.unlink(temp_name)
return
for attempt in range(_ATOMIC_REPLACE_RETRIES + 1):
try:
os.replace(temp_name, path)
return
except OSError as exc:
if exc.errno not in _ATOMIC_REPLACE_RETRY_ERRNOS or attempt >= _ATOMIC_REPLACE_RETRIES:
raise
delay = min(0.4, 0.025 * (2**attempt))
logger.warning(
"Model config atomic replace is temporarily busy (errno=%s); retry %d/%d after %.3fs",
exc.errno,
attempt + 1,
_ATOMIC_REPLACE_RETRIES,
delay,
)
sleep(delay)
def _backup_and_write_config(path: Path, raw_before: str, raw_after: str) -> Path:
backup_dir = path.parent / _BACKUP_DIR_NAME
backup_dir.mkdir(parents=True, exist_ok=True)
@@ -345,7 +501,7 @@ def _backup_and_write_config(path: Path, raw_before: str, raw_after: str) -> Pat
handle.write(raw_after)
handle.flush()
os.fsync(handle.fileno())
os.replace(temp_name, path)
_replace_config_file_with_retry(temp_name, path, raw_before, raw_after)
except Exception:
try:
os.unlink(temp_name)
@@ -429,6 +585,8 @@ def _strip_database_owned_fields(models: list[dict[str, Any]]) -> list[dict[str,
async def _save_models(request: Request, raw_before: str, models: list[dict[str, Any]]) -> list[dict[str, Any]]:
from app.gateway.catalog_cache import invalidate_catalog
path = _resolve_config_path()
models = _strip_database_owned_fields(models)
raw_after = _replace_models_block(raw_before, models)
@@ -447,6 +605,7 @@ async def _save_models(request: Request, raw_before: str, models: list[dict[str,
status_code=422,
detail="The model configuration could not be activated; the previous config was restored.",
) from exc
await invalidate_catalog(request, "models")
return models
@@ -463,7 +622,8 @@ async def list_model_configurations(request: Request) -> ModelAdminListResponse:
models=[
_to_admin_item(
item,
supports_reasoning_effort=capabilities.get(str(item.get("name") or ""), False),
supports_reasoning_effort=bool(capabilities.get(str(item.get("name") or ""), {}).get("supports_reasoning_effort", False)),
default_reasoning_effort=str(capabilities.get(str(item.get("name") or ""), {}).get("default_reasoning_effort") or "xhigh"),
)
for item in models
],
@@ -490,17 +650,19 @@ async def create_model_configuration(body: ModelUpsertRequest, request: Request)
await capability_store.set_reasoning_effort_support(
body.name,
supports_reasoning_effort=bool(body.supports_reasoning_effort),
default_reasoning_effort="xhigh",
updated_by=_admin_user_id(admin),
)
try:
await _save_models(request, raw_before, [*models, created])
except Exception:
if capability_changed and capability_store is not None:
await _restore_capability(capability_store, body.name, previous_capability)
await _restore_capability(capability_store, body.name, previous_capability, request)
raise
return _to_admin_item(
created,
supports_reasoning_effort=bool(body.supports_reasoning_effort) if capability_changed else False,
default_reasoning_effort="xhigh",
)
@@ -535,7 +697,8 @@ async def reorder_model_configurations(body: ModelOrderRequest, request: Request
return [
_to_admin_item(
item,
supports_reasoning_effort=capabilities.get(str(item.get("name") or ""), False),
supports_reasoning_effort=bool(capabilities.get(str(item.get("name") or ""), {}).get("supports_reasoning_effort", False)),
default_reasoning_effort=str(capabilities.get(str(item.get("name") or ""), {}).get("default_reasoning_effort") or "xhigh"),
)
for item in ordered_models
]
@@ -552,6 +715,8 @@ async def update_reasoning_effort_capability(
) -> ModelAdminItem:
"""Persist the per-model reasoning-effort selector capability in SQL."""
from app.gateway.catalog_cache import invalidate_catalog
admin = await _require_admin(request)
async with _MUTATION_LOCK:
path = _resolve_config_path()
@@ -565,11 +730,100 @@ async def update_reasoning_effort_capability(
await store.set_reasoning_effort_support(
model_name,
supports_reasoning_effort=body.supports_reasoning_effort,
default_reasoning_effort=body.default_reasoning_effort,
updated_by=_admin_user_id(admin),
)
await invalidate_catalog(request, "models")
return _to_admin_item(
model,
supports_reasoning_effort=body.supports_reasoning_effort,
default_reasoning_effort=body.default_reasoning_effort or "xhigh",
)
@router.put(
"/write-file-transport/{model_name:path}",
response_model=ModelAdminItem,
)
async def update_write_file_transport(
model_name: str,
body: WriteFileTransportToggleRequest,
request: Request,
) -> ModelAdminItem:
"""Toggle one model's buffered-gateway write_file compatibility in config.yaml.
The global system-settings switch keeps overriding every compatible model
while it is enabled; this per-model value becomes effective as soon as the
global switch is turned off.
"""
await _require_admin(request)
async with _MUTATION_LOCK:
path = _resolve_config_path()
raw_before, _, models = await asyncio.to_thread(_read_config_data, path)
index = next((i for i, item in enumerate(models) if str(item.get("name")) == model_name), None)
if index is None:
raise HTTPException(status_code=404, detail=f"Model '{model_name}' was not found")
updated = copy.deepcopy(models[index])
_validate_write_file_transport(updated, body.write_file_transport)
updated["write_file_transport"] = body.write_file_transport
try:
ModelConfig.model_validate(updated)
except Exception as exc:
raise HTTPException(status_code=422, detail=f"Invalid model configuration: {exc}") from exc
next_models = [*models]
next_models[index] = updated
await _save_models(request, raw_before, next_models)
capabilities = await _capabilities_by_name(request)
capability = capabilities.get(model_name, {})
return _to_admin_item(
updated,
supports_reasoning_effort=bool(capability.get("supports_reasoning_effort", False)),
default_reasoning_effort=str(capability.get("default_reasoning_effort") or "xhigh"),
)
@router.put(
"/strict-stream-integrity/{model_name:path}",
response_model=ModelAdminItem,
)
async def update_strict_stream_integrity(
model_name: str,
body: StrictStreamIntegrityToggleRequest,
request: Request,
) -> ModelAdminItem:
"""Toggle one model's strict stream integrity check in config.yaml.
When enabled, a provider stream that ends without a terminal frame raises
a retriable ``IncompleteChatStreamError`` instead of committing a half
answer — the LLM error middleware then retries the call. Explicit ``false``
also overrides the auto-enable default for the affected qwen3.8-flash-next
model id; omitting the key entirely keeps that default.
"""
await _require_admin(request)
async with _MUTATION_LOCK:
path = _resolve_config_path()
raw_before, _, models = await asyncio.to_thread(_read_config_data, path)
index = next((i for i, item in enumerate(models) if str(item.get("name")) == model_name), None)
if index is None:
raise HTTPException(status_code=404, detail=f"Model '{model_name}' was not found")
updated = copy.deepcopy(models[index])
_validate_patched_openai_use(updated, "strict stream integrity")
updated["strict_stream_integrity"] = body.strict_stream_integrity
try:
ModelConfig.model_validate(updated)
except Exception as exc:
raise HTTPException(status_code=422, detail=f"Invalid model configuration: {exc}") from exc
next_models = [*models]
next_models[index] = updated
await _save_models(request, raw_before, next_models)
capabilities = await _capabilities_by_name(request)
capability = capabilities.get(model_name, {})
return _to_admin_item(
updated,
supports_reasoning_effort=bool(capability.get("supports_reasoning_effort", False)),
default_reasoning_effort=str(capability.get("default_reasoning_effort") or "xhigh"),
)
@@ -596,6 +850,7 @@ async def update_model_configuration(model_name: str, body: ModelUpsertRequest,
await capability_store.set_reasoning_effort_support(
model_name,
supports_reasoning_effort=desired_capability,
default_reasoning_effort=str((previous_capability or {}).get("default_reasoning_effort") or "xhigh"),
updated_by=_admin_user_id(admin),
)
next_models = [*models]
@@ -604,7 +859,7 @@ async def update_model_configuration(model_name: str, body: ModelUpsertRequest,
await _save_models(request, raw_before, next_models)
except Exception:
if capability_changed and capability_store is not None:
await _restore_capability(capability_store, model_name, previous_capability)
await _restore_capability(capability_store, model_name, previous_capability, request)
raise
if capability_changed:
supports_reasoning_effort = desired_capability
@@ -616,6 +871,7 @@ async def update_model_configuration(model_name: str, body: ModelUpsertRequest,
return _to_admin_item(
updated,
supports_reasoning_effort=supports_reasoning_effort,
default_reasoning_effort=str((previous_capability or {}).get("default_reasoning_effort") or "xhigh"),
)
@@ -638,7 +894,7 @@ async def delete_model_configuration(model_name: str, request: Request) -> None:
await _save_models(request, raw_before, [*models[:index], *models[index + 1 :]])
except Exception:
if capability_store is not None:
await _restore_capability(capability_store, model_name, previous_capability)
await _restore_capability(capability_store, model_name, previous_capability, request)
raise
offline-backend-20260512/backend/app/gateway/routers/models.py
+16 -5
View File
@@ -1,8 +1,10 @@
import logging
from typing import Literal
from fastapi import APIRouter, Depends, HTTPException, Request
from pydantic import BaseModel, Field
from app.gateway.catalog_cache import cached_catalog
from app.gateway.deps import get_config
from deerflow.config.app_config import AppConfig
from deerflow.config.model_config import ModelConfig
@@ -36,6 +38,7 @@ def _build_model_response(
model: ModelConfig,
*,
supports_reasoning_effort: bool = False,
default_reasoning_effort: str = "xhigh",
) -> "ModelResponse":
extra = model.model_extra or {}
provider = str(extra["provider"]) if "provider" in extra else _derive_provider(model.use)
@@ -48,6 +51,7 @@ def _build_model_response(
description=model.description,
supports_thinking=model.supports_thinking,
supports_reasoning_effort=bool(supports_reasoning_effort and model.supports_thinking),
default_reasoning_effort=default_reasoning_effort if default_reasoning_effort in {"low", "medium", "xhigh"} else "xhigh",
supports_vision=model.supports_vision,
supports_tool_calling=supports_tool_calling,
is_local=is_local,
@@ -64,6 +68,7 @@ class ModelResponse(BaseModel):
description: str | None = Field(None, description="Model description")
supports_thinking: bool = Field(default=False, description="Whether model supports thinking mode")
supports_reasoning_effort: bool = Field(default=False, description="Whether model supports reasoning effort")
default_reasoning_effort: Literal["low", "medium", "xhigh"] = Field(default="xhigh", description="Globally configured default reasoning effort")
supports_vision: bool = Field(default=False, description="Whether model supports image inputs")
supports_tool_calling: bool = Field(default=True, description="Whether model supports tool/function calling")
is_local: bool = Field(default=False, description="Whether this is a locally-hosted model")
@@ -83,18 +88,22 @@ class ModelsListResponse(BaseModel):
token_usage: TokenUsageResponse
async def _reasoning_capabilities(request: Request) -> dict[str, bool]:
async def _reasoning_capabilities(request: Request) -> dict[str, dict[str, object]]:
store = getattr(request.app.state, "model_capability_store", None)
if store is None:
return {}
async def load() -> dict[str, dict[str, object]]:
rows = await store.list_capabilities()
return {str(row.get("model_name") or ""): row for row in rows if row.get("model_name")}
try:
rows = await store.list_capabilities()
return await cached_catalog(request, "models", "reasoning-capabilities", load)
except Exception:
# Capability reads fail closed: the selector disappears, while ordinary
# model use remains available.
logger.warning("Failed to read model capabilities", exc_info=True)
return {}
return {str(row.get("model_name") or ""): bool(row.get("supports_reasoning_effort", False)) for row in rows if row.get("model_name")}
@router.get(
@@ -143,7 +152,8 @@ async def list_models(request: Request, config: AppConfig = Depends(get_config))
models = [
_build_model_response(
model,
supports_reasoning_effort=capabilities.get(model.name, False),
supports_reasoning_effort=bool(capabilities.get(model.name, {}).get("supports_reasoning_effort", False)),
default_reasoning_effort=str(capabilities.get(model.name, {}).get("default_reasoning_effort") or "xhigh"),
)
for model in config.models
]
@@ -188,5 +198,6 @@ async def get_model(model_name: str, request: Request, config: AppConfig = Depen
capabilities = await _reasoning_capabilities(request)
return _build_model_response(
model,
supports_reasoning_effort=capabilities.get(model.name, False),
supports_reasoning_effort=bool(capabilities.get(model.name, {}).get("supports_reasoning_effort", False)),
default_reasoning_effort=str(capabilities.get(model.name, {}).get("default_reasoning_effort") or "xhigh"),
)
offline-backend-20260512/backend/app/gateway/routers/multi_agent.py
+235 -136
View File
@@ -61,7 +61,6 @@ from typing import Any
import httpx
from fastapi import APIRouter, Body, HTTPException, Request
from fastapi.responses import StreamingResponse
from langgraph.checkpoint.base import empty_checkpoint
from app.gateway.roundtable_diag import record_foreground_diag
from app.gateway.roundtable_model_fallback import fallback_model_chain, is_llm_error_message, reason_text
@@ -82,11 +81,18 @@ from deerflow.agents.roundtable_orchestrator.extraction_spec import build_summar
from deerflow.config import get_app_config
from deerflow.runtime.user_context import get_effective_user_id
from deerflow.tools.builtins.clarification_utils import resolve_allow_multiple
from deerflow.utils.time import now_iso
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/multi-agent", tags=["multi-agent"])
_STEP3_AGENT_IDS = frozenset({
ACTION_PLAN_AGENT_ID,
DASHBOARD_AGENT_ID,
REPORT_AGENT_ID,
STRUCTURE_AGENT_ID,
SUMMARY_AGENT_ID,
})
def _stage_for_agent(agent_type: str, agent_name: str) -> str:
"""把一次 /run/stream 的 (agent_type, agent_name) 映射成诊断日志的 stage。
@@ -107,12 +113,62 @@ def _stage_for_agent(agent_type: str, agent_name: str) -> str:
return "step2_leader" if agent_type == "leader" else "step2_seat"
def _is_step2_room_limited(
agent_type: str,
agent_name: str,
*,
finalize_only: bool = False,
) -> bool:
"""Only the actual Step-2 coordinator and seats consume a room.
The Step-3 singleton agents reuse the same ``/run/stream`` endpoint, so
this boundary must live here rather than at the route level.
"""
# 最后总控只把已经完成的席位交付整理成正文，不再占用席位执行资源。允许它在
# 长耗时席位跑过房间 hard deadline 后仍完成收口，避免最后一步被 409 踢回第一步。
if agent_type == "leader" and finalize_only:
return False
return agent_type == "leader" or agent_name not in _STEP3_AGENT_IDS
# 圆桌（intent / recommend / 席位 / 协调）创建的 thread 都是功能型内部线程,
# 不是用户的普通对话。打上这个 metadata,前端「最近的对话」列表(recent-chat-list.tsx)
# 据 `metadata.thread_type` 过滤掉,与 scheduler / bootstrap 系统会话同一套机制。
_ROUNDTABLE_THREAD_METADATA: dict[str, Any] = {"thread_type": "roundtable", "system": True}
def _room_thread_metadata(
*,
draft_id: str | None,
user_id: str | None,
task_id: str | None,
room_session_id: str | None,
) -> dict[str, Any]:
"""Step-2 threads carry a stable room key so later occupancy can see live runs."""
from deerflow.persistence.roundtable_jobs.sql import compute_active_dedupe_key
from deerflow.persistence.roundtable_rooms.sql import compute_room_session_key
meta = dict(_ROUNDTABLE_THREAD_METADATA)
clean_session_id = (room_session_id or "").strip() or None
key = compute_room_session_key(clean_session_id) or compute_active_dedupe_key(
draft_id, user_id, task_id
)
if key:
meta["roundtable_room_key"] = key
if clean_session_id:
meta["roundtable_room_session_id"] = clean_session_id
if draft_id:
meta["draft_id"] = draft_id
if task_id:
meta["task_id"] = task_id
return meta
def _step3_thread_metadata() -> dict[str, Any]:
"""Step-3 threads are internal threads, but never Step-2 room occupants."""
return dict(_ROUNDTABLE_THREAD_METADATA)
# ---------------------------------------------------------------------------
# 共享:异常诊断器(intent/recommend/multi-agent 共用)
# ---------------------------------------------------------------------------
@@ -415,59 +471,17 @@ def _resolve_user_id(request: Request) -> str:
async def _create_thread_direct(request: Request, *, metadata: dict[str, Any] | None = None) -> str:
"""同进程版的 ``POST /api/threads``。
完全镜像 ``routers/threads.create_thread`` 的写盘动作(thread_meta +
空 checkpoint),但跳过 TCP + Pydantic 反序列化。仅供 init 阶段批量建席位
线程用,失败时抛 HTTPException 由 SSE 帧上报。
``metadata`` 写入 thread 元数据;init 传 ``_ROUNDTABLE_THREAD_METADATA`` 把
席位 / 协调线程标记成系统会话,从前端「最近的对话」里隐藏。
只创建业务元数据，首轮图运行负责初始化 checkpoint。
metadata 的任务共享/系统会话规则与原入口保持一致。
"""
from app.gateway.deps import get_checkpointer, get_thread_store
from app.gateway.deps import get_thread_store
checkpointer = get_checkpointer(request)
thread_store = get_thread_store(request)
thread_id = str(uuid.uuid4())
now = now_iso()
try:
await thread_store.create(thread_id, assistant_id=None, metadata=metadata or {})
await get_thread_store(request).create(thread_id, assistant_id=None, metadata=metadata or {})
except Exception:
logger.exception("[init-direct] thread_meta create failed for %s", thread_id)
raise HTTPException(status_code=500, detail="Failed to create thread")
config = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
ckpt_metadata = {
"step": -1,
"source": "input",
"writes": None,
"parents": {},
"created_at": now,
}
async def _cleanup_partial_thread() -> None:
delete_checkpoint = getattr(checkpointer, "adelete_thread", None)
if callable(delete_checkpoint):
try:
await delete_checkpoint(thread_id)
except Exception:
logger.exception("[init-direct] partial checkpoint cleanup failed for %s", thread_id)
try:
await thread_store.delete(thread_id)
except Exception:
logger.exception("[init-direct] partial thread_meta cleanup failed for %s", thread_id)
try:
await checkpointer.aput(config, empty_checkpoint(), ckpt_metadata, {})
except asyncio.CancelledError:
# A sibling seat may fail while this task is between metadata commit and
# checkpoint creation. Compensate before propagating cancellation, under
# shield so a second cancellation cannot abort the cleanup half-way.
await asyncio.shield(_cleanup_partial_thread())
raise
except Exception:
logger.exception("[init-direct] checkpoint create failed for %s", thread_id)
await _cleanup_partial_thread()
raise HTTPException(status_code=500, detail="Failed to create thread")
return thread_id
@@ -718,18 +732,23 @@ def _run_payload(
new_message: str,
excluded_tools: list[str],
skill_stop_names: list[str],
allowed_tools: list[str] | None = None,
recursion_limit: int = 1000,
thinking_enabled: bool = True,
reasoning_effort: str = "medium",
subagent_enabled: bool = False,
stream_custom: bool = False,
multitask_strategy: str = "reject",
cancel_on_disconnect: bool = False,
extra_context: dict[str, Any] | None = None,
files: list[dict[str, Any]] | None = None,
roundtable_file_stream: bool = False,
) -> dict[str, Any]:
"""构造 ``POST /api/threads/{thread_id}/runs/stream`` 的请求体。
- ``context.agent_name``：本次 run 使用的 agent（内置 id 或 init 创建的 coordinator id）。
- ``excluded_tools``：按路由裁剪工具面（intent 仅澄清、special 禁派活等）。
- ``allowed_tools``：安全敏感角色的真正白名单；``None`` 保持兼容，空列表禁用全部。
- ``skill_stop_names``：leader 侧配合 SkillStopMiddleware 在 ``agent_orchestration``
调用后截断图，以便网关从 AIMessage.tool_calls 解析派活列表。
- ``thinking_enabled`` / ``reasoning_effort``：默认开 CoT + medium，适合 Step 1 澄清
@@ -754,7 +773,7 @@ def _run_payload(
]
},
"metadata": {"run_scope": RUN_SCOPE_ORCHESTRATION_CHILD},
"config": {"recursion_limit": 1000},
"config": {"recursion_limit": recursion_limit},
"context": {
"agent_name": agent_name,
"model_name": model_name,
@@ -778,6 +797,7 @@ def _run_payload(
# 席位 → run/stream 报 400 "agent_name ... missing from d_agent_thread_id"。
# 圆桌是一次性研讨,本就不该吃历史会话记忆,这里统一关注入根治该污染。
"memory_injection_enabled": False,
"allowed_tools": allowed_tools,
},
# ultra 席位会派子智能体(task 工具),其 task_started/running/completed 进度走
# **custom** 流(StreamWriter)。仅这种场景加 "custom",让前端任务卡片能显示子智能体
@@ -788,13 +808,10 @@ def _run_payload(
"stream_subgraphs": True,
"stream_resumable": True,
"assistant_id": "lead_agent",
"on_disconnect": "continue",
# ⚠️ 圆桌 run 必须能被「下一次同 thread 的 run」抢占。圆桌每个 thread 同一时刻
# 只应有一个活跃 run,但 on_disconnect="continue" 让前端 abort SSE 后**后端 run
# 仍在跑**(为可恢复);此时用户「人工干预」立刻发起新 leader run 打到同一总控
# thread,默认 reject 策略就会 409「already has an active run」→ 被 _stream_upstream
# 包成 502。调用方(leader / special)传 multitask_strategy="rollback",让新 run 先
# 取消该 thread 上没结束的旧 run 再继续 —— 这正是「最新一轮/干预指令优先」的语义。
# 第二步实时会商不支持脱离浏览器继续执行。断网、暂停或前端主动 abort 后必须
# 取消后端 run；第三步等既有调用保持 continue，不改变其断线恢复语义。
"on_disconnect": "cancel" if cancel_on_disconnect else "continue",
# rollback 仍保留，负责处理人工干预与新一轮抢占同一 thread 的极短竞态。
"multitask_strategy": multitask_strategy,
"excluded_tools": excluded_tools,
"skill_stop_names": skill_stop_names,
@@ -803,9 +820,20 @@ def _run_payload(
# 供 read_peer_delivery 工具按需读取）。只进 context dict,不进 prompt → 不调工具就不耗 token。
if extra_context:
payload["context"].update(extra_context)
if roundtable_file_stream:
payload["context"].update(_roundtable_file_stream_context())
return payload
def _roundtable_file_stream_context() -> dict[str, bool]:
"""会商专用文件流式兼容。关闭时不写 context，本轮仍只看全局开关。"""
from deerflow.config.system_settings import get_roundtable_room_limit_settings
if get_roundtable_room_limit_settings().write_file_compatibility_enabled:
return {"roundtable_write_file_compatibility": True}
return {}
# ---------------------------------------------------------------------------
# POST /api/multi-agent/init — Step 2 席位 + 总控初始化（SSE）
# ---------------------------------------------------------------------------
@@ -826,7 +854,7 @@ async def init_multi_agent(request: Request, payload: dict = Body(...)) -> Strea
SSE 事件（每条 ``data: {...}\\n\\n``）::
seat_ready — 某一席位 thread 已创建（``asyncio.as_completed`` 顺序，先完先发）
coordinator_ready — 总控 thread 就绪 + 已注入本次席位列表
coordinator_ready — 总控 thread 元数据就绪，席位列表随首次问答注入
init_done — ``thread_ids`` 全量映射，前端存 ``threadIds`` 并开始编排
error — 失败详情（同步校验失败则直接 HTTP 4xx，不进 SSE）
@@ -837,9 +865,9 @@ async def init_multi_agent(request: Request, payload: dict = Body(...)) -> Strea
在首次调用时自动落地。每次 init **不再创建新的 coordinator agent**(避免
``.deer-flow/agents/`` 累积 ``roundtable-coordinator-<ts36>`` 残留)。
本次圆桌的「可调度席位列表」由 init 在创建好 coordinator thread 之后,以一条
human 消息追加到 thread 历史里,leader run 每轮都能从对话历史里读到。这样
SOUL 长期稳定,席位列表随会话动态注入。前端为兼容旧 client 仍可传
初始化只创建业务线程元数据，不访问 checkpoint。每轮 leader run 根据 thread
映射注入真实席位清单，随实际输入一起持久化。SOUL 长期稳定，席位列表随会话
动态注入。前端为兼容旧 client 仍可传
``main_agent_name`` 字段,但后端会忽略并使用 ``COORDINATOR_AGENT_ID``。
"""
# 调用前自检:与 intent/recommend 一样,缺失时就地补建,避免 /run/stream 报 500。
@@ -847,7 +875,6 @@ async def init_multi_agent(request: Request, payload: dict = Body(...)) -> Strea
agent_names_in = payload.get("agent_names") or []
agent_names: list[str] = [str(name).lower() for name in agent_names_in]
model: str | None = payload.get("model")
# main_agent_name 是历史字段,2026-05 改造后忽略,统一用单例 id。保留接收但不再用前端值,
# 避免下游再次创建带时间戳的临时 coordinator。
main_agent_name = COORDINATOR_AGENT_ID
@@ -858,6 +885,26 @@ async def init_multi_agent(request: Request, payload: dict = Body(...)) -> Strea
for name in agent_names:
_validate_agent_id(name, "agent_names entry")
# The browser already acquired the room before entering Step 2. Init only
# renews that lease; it cannot create a second room or bypass admission.
from app.gateway.roundtable_room_gate import enforce_existing_roundtable_room
room_draft_id = str(payload.get("draft_id") or "").strip() or None
room_task_id = str(payload.get("task_id") or "").strip() or None
room_session_id = str(payload.get("room_session_id") or "").strip() or None
await enforce_existing_roundtable_room(
request,
draft_id=room_draft_id,
task_id=room_task_id,
room_session_id=room_session_id,
)
room_thread_meta = _room_thread_metadata(
draft_id=room_draft_id,
user_id=_resolve_user_id(request),
task_id=room_task_id,
room_session_id=room_session_id,
)
def _frame(payload: dict[str, Any]) -> str:
return f"data: {json.dumps(payload, ensure_ascii=False)}\n\n"
@@ -873,10 +920,10 @@ async def init_multi_agent(request: Request, payload: dict = Body(...)) -> Strea
agent_meta = await _get_agents_batch_direct(request, agent_names)
t_meta_done = time.perf_counter()
# 2) 每个席位 thread 并行创建(直调 thread_store + checkpointer),
# 2) 每个席位 thread 元数据并行创建，不预写空 checkpoint，
# 用 asyncio.as_completed 保持"先完先发 seat_ready"的体验。
async def _prepare_sub_agent(sub_id: str) -> tuple[str, str, str, str]:
thread_id = await _create_thread_direct(request, metadata=_ROUNDTABLE_THREAD_METADATA)
thread_id = await _create_thread_direct(request, metadata=room_thread_meta)
meta = agent_meta.get(sub_id.lower(), {})
return (
sub_id.lower(),
@@ -908,50 +955,11 @@ async def init_multi_agent(request: Request, payload: dict = Body(...)) -> Strea
t_seats_done = time.perf_counter()
agent_threads: dict[str, str] = {sid: thread_id for sid, thread_id, _, _ in prepared}
seat_lines: list[str] = [
f"- agent_name: {sid}（{display_name}）：{description}"
for sid, _, display_name, description in prepared
]
# 协调智能体改造为**单例**(id = roundtable-coordinator):
# - SOUL 已经由 ensure_roundtable_functional_agents() 在路由入口落地,
# 通用协调规则常驻磁盘,无需每次重写。
# - 本次"可调度席位清单"以 human 消息形式 append 到 coordinator
# thread 历史里,leader run 每轮都能在对话历史最前面看到。
# 这样席位列表随会话动态变化,但磁盘上不再累积 roundtable-coordinator-<ts36>。
#
# 注意:model 字段在 lead_agent run 时通过 context.model_name 注入,
# 不需要写回到 agent config.yaml(否则会污染单例的配置)。
seat_introduction = (
"【圆桌系统初始化】本次研讨可调度的席位清单如下,请严格按 agent_name "
"通过 agent_orchestration 技能派活,不要派给不在清单内的 agent:\n"
+ "\n".join(seat_lines)
)
coord_thread = await _create_thread_direct(request, metadata=_ROUNDTABLE_THREAD_METADATA)
# SOUL 使用固定总控单例。席位清单由 _leader_run 随实际输入注入，
# 初始化不通过 loopback 追加消息，也无需创建任何空 checkpoint。
coord_thread = await _create_thread_direct(request, metadata=room_thread_meta)
agent_threads[main_agent_name] = coord_thread
# 把席位列表注入 coordinator thread 的对话历史。HTTP loopback 调用
# 走 /api/threads/{tid}/state(读+写),开销 ~10-30ms,在 init 整体
# 几百 ms 的尺度上微不足道,但比起手撕 langgraph checkpoint 内部结构
# 风险低、与 _broadcast / _append_thread_message 一套语义,长期可维护。
try:
base = _loopback_base(request)
headers = _auth_headers(request)
async with _make_loopback_client(request) as seed_client:
await _append_thread_message(
seed_client,
base,
headers,
coord_thread,
seat_introduction,
)
except Exception:
# 注入失败不阻断 init:leader run 仍能跑,只是看不到席位清单,
# 可能派活给错的 id。会被 _strip_display_suffix / _validate_agent_id
# 兜底,前端能看到错误帧;运维通过 backend 日志找到根因再修复。
logger.exception("[init-direct] failed to seed coordinator thread with seat list")
t_coord_done = time.perf_counter()
logger.info(
"[init-direct] meta=%.3fs seats=%.3fs coord=%.3fs total=%.3fs seats_count=%d",
@@ -1001,7 +1009,10 @@ async def init_multi_agent(request: Request, payload: dict = Body(...)) -> Strea
@router.post("/report/init")
async def init_report_thread(request: Request) -> dict[str, str]:
async def init_report_thread(
request: Request,
payload: dict | None = Body(default=None),
) -> dict[str, str]:
"""为 Step 3「结果绘制」创建一个供方案可视化总结智能体(``roundtable-report``)
运行的线程,返回 ``{"agent_name", "thread_id"}``。
@@ -1016,7 +1027,10 @@ async def init_report_thread(request: Request) -> dict[str, str]:
"""
try:
ensure_roundtable_functional_agents()
thread_id = await _create_thread_direct(request, metadata=_ROUNDTABLE_THREAD_METADATA)
thread_id = await _create_thread_direct(
request,
metadata=_step3_thread_metadata(),
)
except Exception as exc:
await record_foreground_diag(
request, stage="step3_report", level="error", event="thread_init_failed",
@@ -1028,7 +1042,10 @@ async def init_report_thread(request: Request) -> dict[str, str]:
@router.post("/summary/init")
async def init_summary_thread(request: Request) -> dict[str, str]:
async def init_summary_thread(
request: Request,
payload: dict | None = Body(default=None),
) -> dict[str, str]:
"""为 Step 3「总结报告」创建一个供方案总结报告智能体(``roundtable-summary``)
运行的线程,返回 ``{"agent_name", "thread_id"}``。
@@ -1044,7 +1061,10 @@ async def init_summary_thread(request: Request) -> dict[str, str]:
"""
try:
ensure_roundtable_functional_agents()
thread_id = await _create_thread_direct(request, metadata=_ROUNDTABLE_THREAD_METADATA)
thread_id = await _create_thread_direct(
request,
metadata=_step3_thread_metadata(),
)
except Exception as exc:
await record_foreground_diag(
request, stage="step3_summary", level="error", event="thread_init_failed",
@@ -1082,7 +1102,10 @@ async def init_position_action_plan_thread(request: Request) -> dict[str, str]:
@router.post("/dashboard/init")
async def init_dashboard_thread(request: Request) -> dict[str, str]:
async def init_dashboard_thread(
request: Request,
payload: dict | None = Body(default=None),
) -> dict[str, str]:
"""为 Step 3「大屏多页」创建一个供大屏多页数据智能体(``roundtable-dashboard``)
运行的线程,返回 ``{"agent_name", "thread_id"}``。
@@ -1100,7 +1123,10 @@ async def init_dashboard_thread(request: Request) -> dict[str, str]:
"""
try:
ensure_roundtable_functional_agents()
thread_id = await _create_thread_direct(request, metadata=_ROUNDTABLE_THREAD_METADATA)
thread_id = await _create_thread_direct(
request,
metadata=_step3_thread_metadata(),
)
except Exception as exc:
await record_foreground_diag(
request, stage="step3_dashboard", level="error", event="thread_init_failed",
@@ -1112,7 +1138,10 @@ async def init_dashboard_thread(request: Request) -> dict[str, str]:
@router.post("/structure/init")
async def init_structure_thread(request: Request) -> dict[str, str]:
async def init_structure_thread(
request: Request,
payload: dict | None = Body(default=None),
) -> dict[str, str]:
"""为 Step 3「入库」创建一个供结构化抽取智能体(``roundtable-structure``)运行的线程,
返回 ``{"agent_name", "thread_id"}``。
@@ -1130,7 +1159,10 @@ async def init_structure_thread(request: Request) -> dict[str, str]:
"""
try:
ensure_roundtable_functional_agents()
thread_id = await _create_thread_direct(request, metadata=_ROUNDTABLE_THREAD_METADATA)
thread_id = await _create_thread_direct(
request,
metadata=_step3_thread_metadata(),
)
except Exception as exc:
await record_foreground_diag(
request, stage="step3_structure", level="error", event="thread_init_failed",
@@ -1181,7 +1213,6 @@ async def _stream_upstream(
本机往返通常 <10ms),怀疑 HTTP_PROXY 拦了回环——检查 client 是否走 _make_loopback_client
(trust_env=False)。
"""
url = f"{base}/api/threads/{thread_id}/runs/stream"
captured: list[str] = []
agent_for_log = (body.get("context") or {}).get("agent_name") or thread_id
@@ -1189,8 +1220,8 @@ async def _stream_upstream(
# 头阶段非 200 的瞬时重试。
# 背景:网关是**单 worker** + checkpointer 是**单连接 aiosqlite**(全进程一把
# asyncio.Lock 串行 checkpoint 读写)。圆桌第三步进入时,前端把「方案总结/结果绘制/
# 大屏」三个重 run **同一刻并行**打出来,叠加第二步 on_disconnect=continue 仍在后台
# 跑的席位 run,瞬时把单锁/单 worker 打满 —— 此时回环子请求 /runs/stream 的 start_run
# 大屏」三个重 run **同一刻并行**打出来,若再叠加尚未完成取消收尾的第二步席位 run,
# 瞬时把单锁/单 worker 打满 —— 此时回环子请求 /runs/stream 的 start_run
# 里那串 DB / checkpoint 写有概率抛错(或同 thread 抢占竞态),返回非 200,被本函数原样
# 包成 502 漏给前端(用户侧表现为"进第三步概率性 502")。
# 关键安全前提:这个非 200 发生在**响应头阶段**,本函数**尚未 yield 任何 SSE 行**给下游,
@@ -1553,6 +1584,7 @@ async def _leader_run(
model_name: str,
seat_meta: dict[str, dict[str, Any]] | None = None,
synthesis_mode: bool = False,
finalize_only: bool = False,
suppress_pre_broadcast: bool = True,
files: list[dict[str, Any]] | None = None,
) -> AsyncIterator[str | dict[str, Any]]:
@@ -1584,6 +1616,9 @@ async def _leader_run(
# new_message),绝不阻断 leader run。这些只进总控**本轮 run 输入**,不广播给席位。
seat_status_block = ""
full_deliverables_block = ""
# Older clients may omit names/descriptions; the thread mapping still
# supplies the authoritative IDs, including on a checkpoint-free first run.
seat_meta = seat_meta or {sid: {} for sid in agent_threads if sid != agent_name}
if seat_meta:
seat_ids = [name for name in agent_threads if name != agent_name]
if seat_ids:
@@ -1600,10 +1635,9 @@ async def _leader_run(
undelivered = [sid for sid in seat_ids if not delivery.get(sid, (False, ""))[0]]
seats_meta_list = [(sid, seat_meta.get(sid, {})) for sid in seat_ids]
seat_status_block = _build_seat_status_block(seats_meta_list, delivery, undelivered_count=len(undelivered))
# 第 2 阶段(仅「综合轮 且 所有席位都已交付」):再**全文**取各已交付席位的交付,拼综合块。
# 还有 ⬜ 未交付席位时不注入 —— 否则"综合输出最终方案"的指令会怂恿弱模型在席位没集齐时
# 就提前收口(实测:3/5 交付就臆称"三方已发言"草草收口)。全文读只在这一轮才付出。
if synthesis_mode and not undelivered:
# 普通综合轮只有全部席位交付后才注入全文，避免弱模型过早收口；显式强制收口轮
# 即使仍有失败席位，也要注入其余有效交付并在最终结论中如实标明缺口。
if finalize_only or (synthesis_mode and not undelivered):
try:
full_delivery = await _collect_delivery_status(
client, base, headers,
@@ -1621,6 +1655,14 @@ async def _leader_run(
len(undelivered),
bool(full_deliverables_block),
)
if finalize_only:
new_message = (
f"{new_message}\n\n"
"【最终收口·纯正文】本轮所有工具均已关闭。请直接根据上方席位交付输出一份合并后的最终结论正文；"
"不要按席位逐份复述原文，也不要反复改写同一份总结。"
"不要调用或尝试 write_file、present_files、agent_orchestration、skill_view 或任何其它工具，"
"不要再派活，也不要只描述你准备做什么。"
)
leader_input = "\n\n".join(
part for part in (seat_status_block, full_deliverables_block, new_message) if part
)
@@ -1637,12 +1679,14 @@ async def _leader_run(
# 带了上传文件时，给总控解锁「读」类工具（leader policy 默认禁 read_file/ls/view_image），
# 否则文件清单注入了却无工具读取。写/派活类仍按 policy 封锁，不放开。
leader_excluded = policy.excluded_tools
if files:
leader_allowed = [] if finalize_only else list(getattr(policy, "allowed_tools", None) or [])
if files and not finalize_only:
_read_tools = {"read_file", "ls", "grep", "view_image"}
leader_excluded = [t for t in leader_excluded if t not in _read_tools]
leader_allowed.extend(sorted(_read_tools))
# 模型自动容错：按 config.yaml models[] 顺序的候选链。当前模型出问题（兜底错误文案 /
# 既无正文又无派活）时切到下一个模型重跑（见循环内检测）。``_mk_body`` 用候选模型重建 body。
# 模型自动容错：按 config.yaml models[] 顺序的候选链。当前模型返回明确的错误标记/
# 兜底错误文案时才切到下一个模型重跑；没有正文或派活不触发换模。
model_chain = fallback_model_chain(model_name) or [model_name]
model_idx = 0
current_model = model_chain[0]
@@ -1655,13 +1699,17 @@ async def _leader_run(
# 注入了席位清单的输入(leader_input);seat_roster 为空时等同 new_message。
new_message=leader_input,
excluded_tools=leader_excluded,
allowed_tools=list(dict.fromkeys(leader_allowed)),
skill_stop_names=effective_stop_names,
recursion_limit=int(getattr(policy, "recursion_limit", 1000)),
thinking_enabled=policy.thinking_enabled,
reasoning_effort=policy.reasoning_effort,
# 人工干预/恢复会重发 leader run 到同一总控 thread;抢占掉没结束的旧 run,避免 409。
multitask_strategy="rollback",
cancel_on_disconnect=True,
# 用户随干预上传的参考文件 → 总控可用 read_file 等工具读取后再派活。
files=files,
roundtable_file_stream=True,
)
body = _mk_body(current_model)
@@ -1729,11 +1777,11 @@ async def _leader_run(
leader_text = _flatten_content(dispatch_msg.get("content", ""))
tool_calls = dispatch_msg.get("tool_calls") or []
# 模型自动容错：本轮总控产出是「中间件兜底错误文案」或「既无正文又无派活」（模型没干活）
# → 若还有备选模型，切到下一个重跑（不消耗派活纠正预算）。先发 model_switch 帧让前端把
# 这条失败气泡重置，再用新模型重新流式。最后一个模型仍失败则照常往下走（错误文案当共识展示）。
# 模型自动容错通常只识别真正的模型错误。总控暂时没有正文/派活不代表模型故障，不能据此
# 擅自换掉用户选择的模型（部分模型会长时间 reasoning 或连续调用工具）。唯一例外是：
# 无工具的最终收口请求已经完整结束却仍无正文，此时该次结果客观不可交付，才换下一个模型。
fail_reason = is_llm_error_message(dispatch_msg)
if fail_reason is None and not (leader_text or "").strip() and not tool_calls:
if fail_reason is None and finalize_only and not (leader_text or "").strip():
fail_reason = "empty"
if fail_reason and model_idx + 1 < len(model_chain):
model_idx += 1
@@ -1750,6 +1798,16 @@ async def _leader_run(
current_model = next_model
body = _mk_body(current_model)
continue
if fail_reason:
yield {
"status": "error",
"role": "leader",
"agent_name": agent_name,
"model_used": current_model,
"reason": reason_text(fail_reason),
"content": leader_text,
}
return
# 澄清分支：与 intent.py 相同字段；status 为字符串 "clarification"（非数组），
# 前端据此暂停 runOrchestration 循环，展示 Step 2 澄清条。
@@ -1903,11 +1961,9 @@ async def _special_run(
directive = build_summary_directive(business_code)
if directive:
seat_message = f"{new_message}\n\n---\n\n{directive}"
# 模型自动容错：按 config.yaml models[] 顺序的候选链。某模型出问题（兜底错误文案；
# 真正的研讨席位还含「交付为空」）→ 发 model_switch 帧让前端重置该席位气泡，再用下一个
# 模型重跑。Step3 单例（report/summary/dashboard/ingest）的产物是落盘文件，正文可能很短，
# 故**不**把「正文为空」当失败（与后台 run_report/summary 的 validate=None 一致），只在真正
# 的模型错误时换模型。
# 模型自动容错：只在真正的模型错误（中间件错误标记/兜底错误文案）时换下一个模型。
# 「仅有思考 / 暂无可见正文 / 交付为空」不是模型故障，不能触发换模；这些状态仍在本轮
# 结束后按 invalid_delivery 交给既有交付校验处理。
model_chain = fallback_model_chain(model_name) or [model_name]
empty_is_failure = role in ("seat", "seat_parallel")
@@ -1918,7 +1974,9 @@ async def _special_run(
thread_id=thread_id,
new_message=seat_message,
excluded_tools=seat_excluded,
allowed_tools=getattr(policy, "allowed_tools", None),
skill_stop_names=merge_skill_stop_names(policy.skill_stop_names, skill_stop_names),
recursion_limit=int(getattr(policy, "recursion_limit", 1000)),
# 席位执行模式覆盖 policy 的 thinking/reasoning（None → 回退 policy 默认）；
# subagent 仅 ultra 档置真。工具裁剪(excluded_tools)仍完全由 policy 决定。
thinking_enabled=(policy.thinking_enabled if thinking_enabled is None else thinking_enabled),
@@ -1928,12 +1986,14 @@ async def _special_run(
stream_custom=subagent_enabled,
# 干预/重新派活会把同一席位 thread 再跑一次;抢占掉没结束的旧 run,避免 409。
multitask_strategy="rollback",
cancel_on_disconnect=role in ("seat", "seat_parallel"),
# 前序席位完整交付 → run context,供 read_peer_delivery 工具按需取全文(§12.9)。
extra_context=(
{"roundtable_peer_deliveries": peer_deliveries} if peer_deliveries else None
),
# 用户随干预上传的参考文件 → 席位可用 read_file 等工具读取。
files=files,
roundtable_file_stream=True,
)
t_payload_built = time.perf_counter()
@@ -2004,10 +2064,8 @@ async def _special_run(
visible = _visible_delivery_from_messages(messages)
delivery_kind = classify_seat_delivery(content) if content.strip() else ("valid" if visible else "empty")
# 模型失败检测：兜底错误文案；研讨席位再加「无可见正文」（思考不算）。还有备选模型则换。
# 模型失败检测：只认中间件标记或兜底错误文案。思考较长/暂无正文绝不触发换模。
fail_reason = is_llm_error_message(last_msg)
if fail_reason is None and empty_is_failure and delivery_kind != "valid":
fail_reason = "thinking_only" if delivery_kind == "thinking_only" else "empty"
if fail_reason and idx + 1 < len(model_chain):
next_model = model_chain[idx + 1]
logger.warning("[special-fallback] %s 模型 %s 失败(%s)，切换到 %s", agent_name, candidate, fail_reason, next_model)
@@ -2021,8 +2079,19 @@ async def _special_run(
}
continue
if fail_reason:
yield {
"status": "error",
"role": role,
"agent_name": agent_name,
"model_used": candidate,
"reason": reason_text(fail_reason),
"content": visible or content,
}
return
if empty_is_failure and delivery_kind != "valid":
logger.warning("[special-fallback] %s 所有模型均未产出可见正文(%s)", agent_name, delivery_kind)
logger.warning("[special-delivery] %s 本轮未产出可见正文(%s)，不触发换模", agent_name, delivery_kind)
yield {
"status": "invalid_delivery",
"reason": delivery_kind,
@@ -2087,6 +2156,7 @@ async def run_multi_agent_stream(request: Request, payload: dict = Body(...)) ->
# 额外把各已交付席位的完整交付正文注入总控输入,供其写最终方案。缺省 False(派活决策轮
# 只给摘要,保持上下文精简)。旧前端不传 → False,行为安全退化。
synthesis_mode: bool = bool(payload.get("synthesis_mode") or False)
finalize_only: bool = bool(payload.get("finalize_only") or False)
# 并行禁写文件标记(仅 special 用):前端 runDagOrchestration 在真并行 stage(≥2 席位)
# 置真,网关据此用 seat_parallel policy 额外禁 write_file/bash/str_replace(§4)。
# 兼容 no_file 别名;缺省 False → 普通 seat,旧前端不传也不破坏。
@@ -2133,6 +2203,27 @@ async def run_multi_agent_stream(request: Request, payload: dict = Body(...)) ->
for key in agent_threads:
_validate_agent_id(key, "d_agent_thread_id key")
# Browser-side acquire admits the room exactly once when Step 2 begins.
# Every model-producing Step-2 entry only renews that existing room, so a
# pause, clarification answer or user intervention never competes for a
# second slot. Step 3 is excluded because consensus releases the room.
step2_room_limited = _is_step2_room_limited(
agent_type,
agent_name,
# 只有显式最终收口轮绕过房间时限；普通综合轮仍可继续派活。
finalize_only=finalize_only,
)
if step2_room_limited:
from app.gateway.roundtable_room_gate import enforce_roundtable_run
await enforce_roundtable_run(
request,
draft_id=str(payload.get("draft_id") or "").strip() or None,
task_id=str(payload.get("task_id") or "").strip() or None,
room_session_id=str(payload.get("room_session_id") or "").strip() or None,
thread_ids=list(agent_threads.values()),
)
base = _loopback_base(request)
headers = _auth_headers(request)
@@ -2160,6 +2251,7 @@ async def run_multi_agent_stream(request: Request, payload: dict = Body(...)) ->
client, base, headers, agent_threads, agent_name, new_message, skill_stop_names, model_name,
seat_meta=seat_meta,
synthesis_mode=synthesis_mode,
finalize_only=finalize_only,
suppress_pre_broadcast=suppress_pre_broadcast,
files=files,
)
@@ -2211,6 +2303,13 @@ async def run_multi_agent_stream(request: Request, payload: dict = Body(...)) ->
"agent_name": agent_name,
}
yield f"data: {json.dumps(err_frame, ensure_ascii=False)}\n\n"
except asyncio.CancelledError:
# Intentional pause and a real network disconnect are both an
# SSE cancellation at this layer. Releasing here made a normal
# pause lose its admitted room. Browser heartbeats now own
# liveness: pauses keep heartbeating; a disconnected browser
# stops heartbeating and the 60-second lease expires.
raise
return StreamingResponse(
stream_generator(),
offline-backend-20260512/backend/app/gateway/routers/open_3qfx.py
+41 -2
View File
@@ -6,7 +6,8 @@
``agent_id``，与前端任务深链 ``goPath=3qfx`` 取的是同一个）；
2. 按 taskId 拉取任务详情（consumer 的 ``cop-task-detail``，地址 / 鉴权头 / mock 都在
``config.yaml`` 的 ``task_deeplink`` 段），拼出与前端深链一致的开场白
``任务名称：…\n任务内容：…\n任务id（task_id）为…``；
``任务名称：…\n任务内容：…\n任务类型：…\n任务id（task_id）为…``
（``taskType`` 字典码 1/2/3 映射为 打击/塑势/防御，无法识别则省略该行）；
3. **新建一条真实会话**（thread 打任务标签并登记到 ``taskId + goPath`` 专用聊天记录），
以开场白作为第一条用户消息
**后台启动一次持久化运行**（写 checkpointer，不阻塞）；
@@ -26,7 +27,7 @@
返回::
{"thread_id": "…", "run_id": "…", "agent_id": "…", "status": "running",
"opening_text": "任务名称：…\n任务内容：…\n任务id（task_id）为2055"}
"opening_text": "任务名称：…\n任务内容：…\n任务类型：打击\n任务id（task_id）为2055"}
"""
from __future__ import annotations
@@ -92,6 +93,41 @@ class CopTaskDetail(BaseModel):
task_direction: str = ""
task_name: str = ""
task_content: str = ""
task_type: str = ""
_TASK_TYPE_LABELS = {"1": "打击", "2": "塑势", "3": "防御"}
def map_task_type_label(value: Any) -> str:
"""把 cop-task-detail 的 taskType 字典码映射成发送文本。无法识别时返回空串。"""
if isinstance(value, bool) or value is None:
return ""
if isinstance(value, int):
key = str(value)
elif isinstance(value, float):
if not value.is_integer():
return ""
key = str(int(value))
elif isinstance(value, str):
key = value.strip()
else:
return ""
if key in _TASK_TYPE_LABELS.values():
return key
return _TASK_TYPE_LABELS.get(key, "")
def _task_type_code(value: Any) -> str:
if isinstance(value, bool) or value is None:
return ""
if isinstance(value, int):
return str(value)
if isinstance(value, float):
return str(int(value)) if value.is_integer() else ""
if isinstance(value, str):
return value.strip()
return ""
async def _resolve_threeq_agent_id(request: Request) -> str:
@@ -117,6 +153,7 @@ def _parse_cop_task_detail(body: Any) -> CopTaskDetail:
task_direction=data.get("taskDirection") if isinstance(data.get("taskDirection"), str) else "",
task_name=data.get("taskName") if isinstance(data.get("taskName"), str) else "",
task_content=data.get("taskContent") if isinstance(data.get("taskContent"), str) else "",
task_type=_task_type_code(data.get("taskType")),
)
@@ -146,9 +183,11 @@ async def _fetch_cop_task_detail(task_id: str) -> CopTaskDetail:
def build_opening_text(detail: CopTaskDetail, task_id: str) -> str:
"""拼 3qfx 开场白，与前端 ``buildOpeningText``（3qfx 分支）逐行对齐。"""
task_type_label = map_task_type_label(detail.task_type)
parts = [
f"任务名称：{detail.task_name}" if detail.task_name else "",
f"任务内容：{detail.task_content}" if detail.task_content else "",
f"任务类型：{task_type_label}" if task_type_label else "",
f"任务id（task_id）为{task_id}" if task_id else "",
]
return "\n".join(p for p in parts if p)
offline-backend-20260512/backend/app/gateway/routers/open_chat.py
+11 -4
View File
@@ -51,6 +51,7 @@ from pydantic import BaseModel, Field
from deerflow.agents.lead_agent.agent import make_lead_agent
from deerflow.config.agents_config import list_custom_agents, validate_agent_id
from deerflow.runtime.runs.worker import build_agent_with_timing
logger = logging.getLogger(__name__)
@@ -81,7 +82,7 @@ class OpenChatRequest(BaseModel):
thinking_enabled: bool = Field(default=False, description="是否开启思考;默认 False = 关闭思考(内网模型也会被强制关闭)")
strip_think: bool = Field(default=True, description="关闭思考时,从答案里剥离残留的 <think>…</think> 块")
thread_id: str | None = Field(default=None, description="可选;仅用于文件隔离,留空自动生成")
recursion_limit: int = Field(default=100, ge=1, le=300, description="智能体单次问答的最大步数")
recursion_limit: int = Field(default=100, ge=1, le=300, description="兼容旧客户端保留；实际步数上限统一使用管理员的问答步骤设置", deprecated=True)
include_files: bool = Field(default=True, description="是否随结果回传智能体在 outputs 目录生成的文件（名称+内容）")
@@ -279,7 +280,13 @@ def _build_run_context(body: OpenChatRequest) -> tuple[dict[str, Any], str | Non
}
if agent_id:
configurable["agent_id"] = agent_id
config: dict[str, Any] = {"configurable": configurable, "recursion_limit": body.recursion_limit}
from deerflow.config.system_settings import get_conversation_context_settings
# Match browser chat: external callers cannot override the server budget.
config: dict[str, Any] = {
"configurable": configurable,
"recursion_limit": get_conversation_context_settings().max_run_steps,
}
return config, agent_id, thread_id
@@ -301,7 +308,7 @@ async def open_chat(body: OpenChatRequest) -> OpenChatResponse:
config, agent_id, thread_id = _build_run_context(body)
try:
agent = make_lead_agent(config)
agent = await build_agent_with_timing(make_lead_agent, config, None, thread_id)
result = await agent.ainvoke({"messages": lc_messages}, config=config)
except Exception as exc: # noqa: BLE001 — 开放接口需把内部错误归一成 502
logger.exception("open_chat 调用智能体失败 (model=%s, agent=%s)", body.model_name, agent_id)
@@ -345,7 +352,7 @@ async def _open_chat_event_stream(body: OpenChatRequest) -> AsyncIterator[str]:
async def produce() -> None:
nonlocal final_state
try:
agent = make_lead_agent(config)
agent = await build_agent_with_timing(make_lead_agent, config, None, thread_id)
async for mode, chunk in agent.astream({"messages": lc_messages}, config=config, stream_mode=["values", "messages"]):
if mode == "values" and isinstance(chunk, dict):
final_state = chunk
offline-backend-20260512/backend/app/gateway/routers/parallel_agent_tasks.py
+6 -3
View File
@@ -20,7 +20,7 @@ from datetime import datetime
from typing import Any
from uuid import uuid4
from fastapi import APIRouter, HTTPException, Request
from fastapi import APIRouter, HTTPException, Query, Request
from pydantic import BaseModel, Field
from deerflow.runtime.user_context import get_effective_user_id
@@ -103,10 +103,13 @@ def _get_store(request: Request):
@router.get("", response_model=TaskListResponse)
async def list_tasks(request: Request) -> TaskListResponse:
async def list_tasks(
request: Request,
id_prefix: str | None = Query(default=None, max_length=64),
) -> TaskListResponse:
store = _get_store(request)
user_id = _current_user_id(request)
rows = await store.list_tasks(user_id)
rows = await store.list_tasks(user_id, id_prefix=id_prefix)
return TaskListResponse(tasks=[TaskResponse(**r) for r in rows])
offline-backend-20260512/backend/app/gateway/routers/parallel_agents.py
+82 -17
View File
@@ -9,8 +9,8 @@ Each fan-out agent runs as a pure **executing seat**: ``excluded_tools`` =
``_EXCLUDED_TOOLS`` (``write_file`` / ``str_replace`` / ``agent_orchestration`` /
``bash``) so a seat can neither write files (parallel writes would clash) nor
dispatch / run scripts (a weak model fed "总控 / 业务链条" framing would otherwise
mimic the controller and "派活"). Read-only tools + ``ask_clarification`` stay on,
so the step bar and clarification cards still work.
mimic the controller and "派活"). Read-only tools stay on. ``ask_clarification``
is available only to strategy intent, the sole stage with a reply surface.
Unified event protocol (one JSON object per ``data:`` frame)::
@@ -42,7 +42,10 @@ from pydantic import BaseModel, Field
from app.gateway.roundtable_model_fallback import is_llm_error_message, reason_text
from app.gateway.routers import agents as agents_router
from app.gateway.routers._strategy_selection_seed import STRATEGY_AGENT_ID_PREFIX
from app.gateway.routers._strategy_selection_seed import (
STRATEGY_AGENT_ID_PREFIX,
STRATEGY_INTENT_AGENT_ID,
)
from app.gateway.routers.multi_agent import (
_ROUNDTABLE_THREAD_METADATA,
_auth_headers,
@@ -128,7 +131,7 @@ def _has_substantive_content(content: str) -> bool:
# 链条」语气的任务后，会概率性地用 agent_orchestration 真去派活，或绕道 bash 跑脚本
# 调内部接口模仿总控调度（bug：单步单智能体链条里子智能体输出「总控调度指令…」并用
# glob / bash 翻找派活脚本）。物理禁掉这两类即可根治。其余只读工具（ls / read_file /
# 检索）与 ask_clarification（协作卡片）保留：既不互相打架，又能正常出步骤条 / 要澄清。
# 检索）保留；ask_clarification 仅意图阶段保留，因为只有它有可续答的协作卡片。
_EXCLUDED_TOOLS: list[str] = [
"write_file", # 并行不互相写文件打架
"str_replace",
@@ -137,6 +140,26 @@ _EXCLUDED_TOOLS: list[str] = [
]
def _strategy_stage_allows_clarification(agent_id: str) -> bool:
"""Return whether a strategy-selection stage has a user reply surface.
The workbench renders and resumes clarification cards only for the intent
stage. Angle generation, judging and final synthesis are one-shot stages;
letting those agents call ``ask_clarification`` produces an interrupt that
the UI cannot answer and therefore turns a recoverable run into a failure.
"""
return agent_id.strip().casefold() == STRATEGY_INTENT_AGENT_ID.casefold()
def _excluded_tools_for_agent(agent_id: str) -> list[str]:
excluded = list(_EXCLUDED_TOOLS)
normalized = agent_id.strip().casefold()
if normalized.startswith(STRATEGY_AGENT_ID_PREFIX.casefold()) and not _strategy_stage_allows_clarification(agent_id):
excluded.append("ask_clarification")
return excluded
# ── 并行面板兼容登录：复用单点开关、密码、限速与封禁校验 ────────────────────
@@ -150,12 +173,44 @@ class _TokenResponse(BaseModel):
user_id: str
def _force_situation_detail_compat_token(username: str) -> _TokenResponse:
"""Return a non-JWT value accepted only by open force-situation detail APIs.
Existing task-detail skills call this legacy endpoint before every detail
GET/PUT and require a non-empty value even though those endpoints are now
credential-free. Keeping the compatibility behavior generic avoids
coupling backend routing to a particular skill name or service account.
"""
normalized = username.casefold()
return _TokenResponse(
# Deliberately not a signed JWT: AuthMiddleware rejects it everywhere
# except the explicitly credential-free force-situation detail routes.
access_token=f"force-situation-detail-compat-{normalized}",
token_type="bearer",
user_id=f"external:{normalized}",
)
@router.post("/auth/token", response_model=_TokenResponse)
async def issue_token(body: _TokenRequest, request: Request, response: Response, password: str | None = Query(default=None)) -> _TokenResponse:
"""Keep the panel response contract while enforcing standard login policy."""
from app.gateway.routers.auth import UsernameLoginRequest, login_by_username
"""Issue a normal login token, or detail-route compatibility value on policy rejection."""
username = body.username.strip()
from app.gateway.routers.auth import login_by_username
result = await login_by_username(request, response, UsernameLoginRequest(username=body.username.strip()), password=password)
try:
result = await login_by_username(
request,
response,
password=password,
)
except HTTPException as exc:
# The caller needs this endpoint only as a preflight before the open
# force-situation detail routes. Do not make individual skill names or
# current password/SSO policy a prerequisite for that workflow.
if exc.status_code in {401, 403}:
return _force_situation_detail_compat_token(username)
raise
return _TokenResponse(access_token=result.access_token, token_type="bearer", user_id=result.user_id)
@@ -368,7 +423,7 @@ async def _run_one_agent(
model_name=model_name,
thread_id=thread_id,
new_message=new_message,
excluded_tools=list(_EXCLUDED_TOOLS),
excluded_tools=_excluded_tools_for_agent(agent_id),
skill_stop_names=[],
thinking_enabled=True,
reasoning_effort="medium",
@@ -425,7 +480,7 @@ async def _run_one_agent(
}
def _build_task_message(task_name: str, brief: str) -> str:
def _build_task_message(task_name: str, brief: str, *, agent_id: str = "") -> str:
parts: list[str] = []
if task_name.strip():
parts.append(f"任务名称：{task_name.strip()}")
@@ -440,13 +495,23 @@ def _build_task_message(task_name: str, brief: str) -> str:
"调度对象 / 分配给『…』的任务 / agent_name」之类调度内容，严禁罗列或转派其他席位的任务，严禁调用"
"任何派活 / 调度类工具，也不要用脚本（bash / python）去调用内部接口或翻找派活脚本。"
)
# 澄清护栏：要向用户确认时，必须用 ask_clarification 工具发起（前端据此渲染可点击的协作
# 卡片），不能只把问题写在正文里 —— 否则用户看不到卡片、无法回应。能合理假设就先假设并标注。
parts.append(
"【需要向用户确认时】如果确实缺少完成分析所必需的关键信息，请**调用 `ask_clarification` 工具**"
"发起确认（这样用户会看到一张可点击的协作卡片来回答）；**不要只把问题写在正文里**，否则用户无法回应。"
"能基于已有信息合理假设的，就先假设并在正文中标注你的假设；只有当不同假设会显著改变结论时，才发起确认。"
)
# 只有意图阶段有可交互、可恢复的澄清卡片。其余策略阶段均为一次性交付，必须基于已确认
# 的任务继续完成；否则 ask_clarification 会产生前端无法回答的中断并被误报为生成失败。
if agent_id and agent_id.casefold().startswith(STRATEGY_AGENT_ID_PREFIX.casefold()) and not _strategy_stage_allows_clarification(agent_id):
parts.append(
"【本阶段禁止反问】当前是无需用户中途介入的一次性交付阶段。不得调用 `ask_clarification`，"
"不得在正文里向用户提问、要求补充材料或要求点击重试。信息不足时请采用保守的合理假设，"
"明确降低相应结论的置信度，并把缺口列入“数据限制与待核验”，然后直接完成本阶段要求的全部交付；"
"不要重复输出同一正文或多个版本。"
)
else:
# 澄清护栏：要向用户确认时，必须用 ask_clarification 工具发起（前端据此渲染可点击的
# 协作卡片），不能只把问题写在正文里。能合理假设就先假设并标注。
parts.append(
"【需要向用户确认时】如果确实缺少完成分析所必需的关键信息，请**调用 `ask_clarification` 工具**"
"发起确认（这样用户会看到一张可点击的协作卡片来回答）；**不要只把问题写在正文里**，否则用户无法回应。"
"能基于已有信息合理假设的，就先假设并在正文中标注你的假设；只有当不同假设会显著改变结论时，才发起确认。"
)
return "\n\n".join(parts)
@@ -514,13 +579,13 @@ async def run_parallel_agents_stream(request: Request, payload: dict = Body(...)
base = _loopback_base(request)
headers = _auth_headers(request)
new_message = _build_task_message(task_name, brief)
async def _consume_agent(
client: Any,
agent_id: str,
queue: asyncio.Queue[dict[str, Any]],
) -> None:
new_message = _build_task_message(task_name, brief, agent_id=agent_id)
await queue.put({"type": "agent_start", "agent_id": agent_id})
last_event = ""
emitted_tools: set[str] = set() # 已发过 step 的 tool_call id（去重）
offline-backend-20260512/backend/app/gateway/routers/positions.py
+1 -8
View File
@@ -142,14 +142,7 @@ async def get_my_position(request: Request, store: PositionStore = Depends(get_p
user = await get_optional_user_from_request(request)
if user is None:
return MyPositionResponse(position=None)
record = None
pid = await store.get_user_position_id(str(user.id))
if pid:
record = await store.get_position(pid)
if record is None:
default_id = await store.get_default_position_id()
if default_id:
record = await store.get_position(default_id)
record = await store.get_effective_position(str(user.id))
return MyPositionResponse(position=PositionResponse(**record) if record else None)
offline-backend-20260512/backend/app/gateway/routers/public_knowledge_vector_search.py
+2 -1
View File
@@ -114,7 +114,8 @@ async def public_knowledge_vector_search(
# second read ensures an unpublished or newly reserved result is never
# emitted by this anonymous endpoint.
still_published = {str(row["id"]) for row in await _published_mappings(request)}
frontend_base_url = request.app.state.config.llmwiki.local_wiki_index.frontend_base_url
wiki_index = request.app.state.config.llmwiki.local_wiki_index
frontend_base_url = wiki_index.resolve_frontend_base_url(request.headers.get("origin"))
results: list[dict[str, Any]] = []
for raw_item in payload.get("results") or []:
if str(raw_item.get("knowledge_base_id") or "") not in still_published:
offline-backend-20260512/backend/app/gateway/routers/public_leaderboard.py
+7 -3
View File
@@ -16,10 +16,10 @@ import logging
from fastapi import APIRouter, HTTPException, Query
from app.gateway.admin_query_window import ADMIN_QUERY_WINDOW_DAYS, resolve_admin_query_window
from app.gateway.routers.admin_users import (
AdminLeaderboardResponse,
_load_or_schedule_daily_leaderboard,
_resolve_range,
)
from deerflow.config.system_settings import load_system_settings
from deerflow.persistence.engine import get_session_factory
@@ -68,7 +68,7 @@ def _anonymize(resp: AdminLeaderboardResponse) -> AdminLeaderboardResponse:
@router.get("", response_model=AdminLeaderboardResponse)
@router.get("/", response_model=AdminLeaderboardResponse)
async def get_public_leaderboard(
days: int = Query(default=30, ge=1, le=365),
days: int = Query(default=30, ge=1, le=ADMIN_QUERY_WINDOW_DAYS),
since: str | None = Query(default=None, description="Inclusive range start, ISO datetime"),
until: str | None = Query(default=None, description="Inclusive range end, ISO datetime"),
limit: int = Query(default=20, ge=5, le=100),
@@ -79,7 +79,11 @@ async def get_public_leaderboard(
raise HTTPException(status_code=503, detail="Analytics requires database persistence")
settings = load_system_settings().leaderboard
since_dt, until_dt, _range_days = _resolve_range(days, since, until)
since_dt, until_dt, _range_days = resolve_admin_query_window(
days=days,
since=since,
until=until,
)
resp = await _load_or_schedule_daily_leaderboard(
session_factory,
settings,
offline-backend-20260512/backend/app/gateway/routers/recommended_questions.py
+24 -4
View File
@@ -9,6 +9,7 @@ from uuid import uuid4
from fastapi import APIRouter, HTTPException, Query, Request
from pydantic import BaseModel, Field
from app.gateway.api_read_cache import get_cache_version, get_cached_json, set_cached_json
from deerflow.runtime.user_context import get_effective_user_id
logger = logging.getLogger(__name__)
@@ -65,6 +66,16 @@ async def list_recommended_questions(
all: bool = Query(default=False, description="Admin management view: return every question, unfiltered by 岗位."),
tag_ids: list[str] | None = Query(default=None, description="Keep questions carrying any of these tags (OR)."),
) -> RecommendedQuestionsListResponse:
user_id = _current_user_id(request)
config = getattr(request.app.state, "config", None)
cache_key = f"user:{user_id}"
normalized_tag_ids = tag_ids if isinstance(tag_ids, list) else None
cacheable = not all and not normalized_tag_ids
version = await get_cache_version(config, "recommended_questions") if cacheable else None
if cacheable:
cached = await get_cached_json(config, "recommended_questions", cache_key, version=version)
if cached is not None:
return RecommendedQuestionsListResponse.model_validate(cached)
store = _get_store(request)
rows = await store.list_all()
responses = [RecommendedQuestionResponse(**r) for r in rows]
@@ -91,7 +102,7 @@ async def list_recommended_questions(
sync = getattr(request.app.state, "position_sync", None)
if sync is not None:
try:
visible = await sync.resolve_visible(_current_user_id(request), "recommended_question")
visible = await sync.resolve_visible(user_id, "recommended_question")
except Exception:
logger.debug("recommended-question visibility resolve failed", exc_info=True)
visible = None
@@ -100,11 +111,20 @@ async def list_recommended_questions(
else:
responses = [r for r in responses if not r.tags]
if tag_ids:
wanted = set(tag_ids)
if normalized_tag_ids:
wanted = set(normalized_tag_ids)
responses = [r for r in responses if any(t.get("id") in wanted for t in r.tags)]
return RecommendedQuestionsListResponse(questions=responses)
result = RecommendedQuestionsListResponse(questions=responses)
if cacheable:
await set_cached_json(
config,
"recommended_questions",
cache_key,
result,
expected_version=version,
)
return result
@router.post("/recommended-questions", response_model=RecommendedQuestionResponse, status_code=201)
offline-backend-20260512/backend/app/gateway/routers/roundtable_chains.py
+148 -3
View File
@@ -41,6 +41,25 @@ router = APIRouter(prefix="/api/roundtable-chains", tags=["roundtable-chains"])
MIN_SEATS = 2
MAX_SEATS = 20
# 功能型内置智能体不可作业务链席位（Step 1 意图理解 / 推荐器 / 总控 / 报告）。
# 与前端 `api/recommend.ts` 的 `filterRecommendCandidates` 排除集逐字对齐：
# 席位富化（_enrich_chain_seats）把这些 id 一律标为 available=False，保证
# 「前端不再实时查 /api/agents」后「功能型智能体不可入席」的语义不变。
_FUNCTIONAL_AGENT_IDS = frozenset(
{
"roundtable-intent",
"roundtable-recommender",
"roundtable-coordinator",
"roundtable-report",
}
)
_FUNCTIONAL_AGENT_PREFIX = "roundtable-coordinator-"
def _is_functional_agent(agent_id: str) -> bool:
lowered = agent_id.lower()
return lowered in _FUNCTIONAL_AGENT_IDS or lowered.startswith(_FUNCTIONAL_AGENT_PREFIX)
# ── schemas ────────────────────────────────────────────────────────────────
@@ -80,11 +99,30 @@ class ChainSeatSchema(BaseModel):
job_responsibilities: str | None = Field(default=None, max_length=4000)
class ChainSeatResponseSchema(ChainSeatSchema):
"""席位响应模型：在落库字段之外附带两个**读时计算**的富化字段。
前端选链（弹窗 / 左栏点选 / 深链自动选链）从此不再实时拉 `/api/agents`，
所需的「席位智能体是否仍可用」与「最新名称」由后端在返回链条时用一次批量
查询（`agent_store.list_by_ids`，与 `/api/agents` 同一套可见性过滤）算好：
- ``available``：该席位智能体当前对请求用户可见（本人 / 内置 / 已发布、
非私享、非管理员隐藏）且不是功能型内置 → True；否则 False。
- ``live_name``：智能体当前名称；查不到时为 None（前端回退 seat.name 快照）。
仅用于响应序列化，**绝不持久化**：create/update 请求沿用 ``ChainSeatSchema``，
富化字段不会进 seats JSON（存量数据零迁移）。
"""
available: bool | None = None
live_name: str | None = None
class ChainResponse(BaseModel):
id: str
title: str = ""
description: str | None = None
seats: list[ChainSeatSchema] = []
seats: list[ChainSeatResponseSchema] = []
# Optional layered orchestration: list of stages, each a list of agent_ids
# run in parallel (stages run serially). None/absent = linear chain.
stages: list[list[str]] | None = None
@@ -103,6 +141,10 @@ class ChainResponse(BaseModel):
# 「待审核」标记：True = owner 已提交审核（等待管理员在「待审核」筛选里发布）；
# 管理员发布时随 is_public=True 一起清回 False。缺省 False。
pending_review: bool = False
# 管理员目录开关：True = 从所有普通列表（我的/公共/待审核/选链弹窗）隐藏，仅管理员
# 管理视图（admin_view=true）可见。纯目录开关：get_chain 单条读取不受影响，已配置的
# 业务映射 / 进行中的研讨继续可用。缺省 False。
hidden: bool = False
# 是否当前用户所有（仅本人可编辑 / 删除 / 发布；公共筛选里别人的链条 is_mine=False）。
is_mine: bool = True
# 配置者（owner）的 user_id 及解析后的可读用户名（email）。owner_name 由路由层
@@ -199,6 +241,60 @@ async def _enrich_owner_names(rows: list[dict]) -> None:
row["owner_name"] = cache[uid]
def _apply_seat_enrichment(chains: list[dict], live_records: list[dict]) -> None:
"""把批量查到的实时智能体信息写进每条链每个 seat（就地修改响应 dict）。
纯函数（不触网），便于单测：``live_records`` 是 ``agent_store.list_by_ids``
的返回（dict 列表，含 id/name）。规则：
- 功能型内置智能体（_is_functional_agent）与查不到的 id → available=False、
live_name=None（对齐前端 ``filterRecommendCandidates`` 的排除语义）；
- 其余 → available=True、live_name=实时 name（回退 None 由前端用快照名）。
"""
live: dict[str, dict] = {}
for record in live_records:
rid = str(record.get("id") or "").lower()
if rid:
live[rid] = record
for chain in chains:
for seat in chain.get("seats") or []:
aid = str(seat.get("agent_id") or "").lower()
record = live.get(aid)
if record is None or _is_functional_agent(aid):
seat["available"] = False
seat["live_name"] = None
else:
seat["available"] = True
seat["live_name"] = str(record.get("name") or "") or None
async def _enrich_chain_seats(request: Request, rows: list[dict]) -> None:
"""席位富化入口：一次批量 SQL 算好所有 seat 的 available / live_name。
best-effort：agent store 不可用 / 查询异常时不写任何字段（响应缺省 → 前端
按存活处理），与旧前端「agents 列表拉取失败不判删」的降级语义一致。
仅修改响应 dict，落库的 seats JSON 不受影响。
"""
agent_ids = {
str(seat.get("agent_id") or "").lower()
for row in rows
for seat in row.get("seats") or []
if seat.get("agent_id")
}
if not agent_ids:
return
try:
store = get_agent_store(request)
except Exception:
logger.warning("Seat enrichment skipped: agent store unavailable")
return
try:
records = await store.list_by_ids(sorted(agent_ids), _current_user_id(request))
except Exception:
logger.warning("Seat enrichment skipped: batch agent lookup failed", exc_info=True)
return
_apply_seat_enrichment(rows, records)
def _get_store(request: Request):
store = getattr(request.app.state, "roundtable_chain_store", None)
if store is None:
@@ -364,7 +460,12 @@ def _validate_seat_mode(seat_mode: str | None) -> None:
@router.get("", response_model=ChainListResponse)
async def list_chains(request: Request, scope: str = "mine") -> ChainListResponse:
async def list_chains(
request: Request,
scope: str = "mine",
admin_view: bool = False,
hidden: bool | None = None,
) -> ChainListResponse:
"""列出业务链条。
``scope`` 查询参数(筛选):
@@ -374,6 +475,11 @@ async def list_chains(request: Request, scope: str = "mine") -> ChainListRespons
非管理员请求 ``pending`` 退化为 ``mine``）；
- ``all``：所有账号的全部链条（**仅管理员**；非管理员请求 ``all`` 退化为 ``mine``）。
其余值按 ``mine`` 处理。
隐藏开关（管理员目录）：默认（含管理员的普通视图，如选链弹窗）隐藏链条
（``hidden=True``）从所有 scope 排除。管理员传 ``admin_view=true`` 切到管理视图：
不排除隐藏链条，并可用 ``hidden=true/false`` 筛选只看隐藏 / 只看显示中（配置页专用）。
非管理员传 ``admin_view`` 一律忽略，绝不越权。
"""
is_admin = _is_admin(request)
normalized_scope = scope if scope in ("mine", "public", "pending", "all") else "mine"
@@ -382,8 +488,15 @@ async def list_chains(request: Request, scope: str = "mine") -> ChainListRespons
normalized_scope = "mine"
store = _get_store(request)
user_id = _current_user_id(request)
rows = await store.list_chains(user_id, scope=normalized_scope, is_admin=is_admin)
rows = await store.list_chains(
user_id,
scope=normalized_scope,
is_admin=is_admin,
include_hidden=admin_view and is_admin,
hidden=hidden if admin_view and is_admin else None,
)
await _enrich_owner_names(rows)
await _enrich_chain_seats(request, rows)
return ChainListResponse(chains=[ChainResponse(**r) for r in rows])
@@ -411,6 +524,7 @@ async def create_chain(request: Request, body: ChainCreateRequest) -> ChainRespo
},
)
await _enrich_owner_names([row])
await _enrich_chain_seats(request, [row])
return ChainResponse(**row)
@@ -422,6 +536,7 @@ async def get_chain(request: Request, chain_id: str) -> ChainResponse:
if row is None:
raise HTTPException(status_code=404, detail="Chain not found")
await _enrich_owner_names([row])
await _enrich_chain_seats(request, [row])
return ChainResponse(**row)
@@ -451,6 +566,7 @@ async def update_chain(request: Request, chain_id: str, body: ChainUpdateRequest
if row.get("is_public"):
await _publish_seat_agents(request, row, user_id, is_admin=is_admin)
await _enrich_owner_names([row])
await _enrich_chain_seats(request, [row])
return ChainResponse(**row)
@@ -463,6 +579,35 @@ async def delete_chain(request: Request, chain_id: str) -> None:
raise HTTPException(status_code=404, detail="Chain not found")
class ChainHiddenUpdateRequest(BaseModel):
"""Admin-only request body for hiding a chain from every non-admin listing."""
hidden: bool = Field(description="Set true to hide the chain from all non-admin listings (我的/公共/待审核/选链弹窗); false restores it. Purely a catalog switch — get_chain / 业务映射 / 进行中的研讨不受影响。")
@router.put("/{chain_id}/hidden", response_model=ChainResponse)
async def set_chain_hidden(
request: Request, chain_id: str, body: ChainHiddenUpdateRequest
) -> ChainResponse:
"""隐藏 / 取消隐藏一条业务链条（**仅管理员**）。
隐藏后链条从所有普通列表消失（连 owner 的「我的链条」也不显示），但单条读取不过滤：
已配置的业务映射、进行中的研讨、编辑器直接打开（chainId 深链）都继续可用。管理员
管理视图（``admin_view=true``）始终可见，可随时取消隐藏。
"""
if not _is_admin(request):
raise HTTPException(status_code=403, detail="Only system admins can hide or restore chains")
store = _get_store(request)
user_id = _current_user_id(request)
row = await store.update_chain(chain_id, user_id, hidden=body.hidden, is_admin=True)
if row is None:
raise HTTPException(status_code=404, detail="Chain not found")
logger.info("Set chain '%s' hidden=%s by admin", chain_id, body.hidden)
await _enrich_owner_names([row])
await _enrich_chain_seats(request, [row])
return ChainResponse(**row)
# ── 提示词优化（总控提示词 / 阶段目标）──────────────────────────────────────
# 链条编辑器的「AI 优化」按钮：把用户当前写的提示词 + 链条结构发来，单次直连
# LLM 重写后返回全文（同 suggestions 路由的直调模式，不走 agent / loopback）。
offline-backend-20260512/backend/app/gateway/routers/roundtable_jobs.py
+34
View File
@@ -63,6 +63,9 @@ class StartJobRequest(BaseModel):
# 外部任务 id（taskId 深链会商）。非空 = task 作业：按 task_id 共享、不按 user 分权，
# 其 draft_id 指向独立的 roundtable_task_drafts 表（最终报告写回那里）。
taskId: str | None = Field(default=None, max_length=128)
# 当前页面这一次实时会商的隐藏标识。后台接管前用于续占同一个前台名额；
# job 创建后由持久 job 行继续占位。
roomSessionId: str | None = Field(default=None, max_length=128)
intent: str = ""
agents: list[JobAgent] = Field(default_factory=list)
model: str | None = None
@@ -236,6 +239,24 @@ async def start_job(
task_id = (body.taskId or "").strip() or None
draft_id = (body.draftId or "").strip() or None
# Handoff must own the already-admitted foreground room. It only renews
# that lease and never creates capacity during the transition.
from app.gateway.roundtable_room_gate import enforce_existing_roundtable_room
await enforce_existing_roundtable_room(
request,
draft_id=draft_id,
task_id=task_id,
room_session_id=(body.roomSessionId or "").strip() or None,
)
room_store = getattr(request.app.state, "roundtable_room_store", None)
room_started_at = None
read_started_at = getattr(room_store, "started_at", None)
if read_started_at is not None:
room_started_at = await read_started_at(
room_session_id=(body.roomSessionId or "").strip() or None,
holder_user_id=user_id,
)
agents = [SeatRef(a.agentId, a.name or a.agentId) for a in body.agents]
seed_message = (body.seedMessage or body.intent or "").strip()
@@ -280,7 +301,20 @@ async def start_job(
request_id=request_id,
request_hash=_request_hash(body),
input_snapshot=snapshot,
room_started_at=room_started_at,
)
# The durable job row now owns occupancy. Drop the foreground lease so a
# completed background job does not leave a one-minute phantom room.
if room_store is not None:
try:
await room_store.release(
draft_id=draft_id,
user_id=user_id,
task_id=task_id,
room_session_id=(body.roomSessionId or "").strip() or None,
)
except Exception:
logger.warning("failed to release foreground room after job handoff", exc_info=True)
if created:
# 新建（queued）：交给 dispatcher 领取执行。
response.status_code = 201
offline-backend-20260512/backend/app/gateway/routers/roundtable_rooms.py
+123
View File
@@ -0,0 +1,123 @@
"""Cross-user roundtable room admission.
Routes (prefix ``/api/roundtable-rooms``)::
GET /summary occupied room count across every account
POST /acquire atomically reserve a room, or 409 when full
POST /heartbeat renew the same durable lease
POST /release release the foreground lease
"""
from __future__ import annotations
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel, Field
from app.gateway.roundtable_room_gate import (
admit_roundtable_room,
backend_occupancy_keys,
release_roundtable_room,
renew_roundtable_room,
)
from deerflow.config.system_settings import get_roundtable_room_limit_settings
router = APIRouter(prefix="/api/roundtable-rooms", tags=["roundtable-rooms"])
class RoomScopeRequest(BaseModel):
draftId: str | None = Field(default=None, max_length=64)
taskId: str | None = Field(default=None, max_length=128)
roomSessionId: str | None = Field(default=None, max_length=128)
class RoomAdmissionResponse(BaseModel):
ok: bool = True
enabled: bool = True
max_rooms: int = 3
active_count: int = 0
renewed: bool = False
code: str = ""
message: str = ""
class RoomSummaryResponse(BaseModel):
enabled: bool = True
max_rooms: int = 3
active_count: int = 0
def _response(decision: dict) -> RoomAdmissionResponse:
return RoomAdmissionResponse(
ok=bool(decision.get("ok")),
enabled=bool(decision.get("enabled")),
max_rooms=int(decision.get("max_rooms") or 0),
active_count=int(decision.get("active_count") or 0),
renewed=bool(decision.get("renewed")),
code=str(decision.get("code") or ""),
message=str(decision.get("message") or ""),
)
def _require_store(request: Request):
store = getattr(request.app.state, "roundtable_room_store", None)
if store is None:
raise HTTPException(status_code=503, detail="Roundtable room store not available")
return store
@router.get("/summary", response_model=RoomSummaryResponse)
async def room_summary(request: Request) -> RoomSummaryResponse:
settings = get_roundtable_room_limit_settings()
store = getattr(request.app.state, "roundtable_room_store", None)
extra = await backend_occupancy_keys(request)
active = (
0
if store is None
else await store.summary(
extra_keys=extra,
max_hold_minutes=getattr(settings, "max_hold_minutes", 30),
)
)
return RoomSummaryResponse(
enabled=settings.enabled,
max_rooms=settings.max_rooms,
active_count=active,
)
@router.post("/acquire", response_model=RoomAdmissionResponse)
async def acquire_room(request: Request, body: RoomScopeRequest) -> RoomAdmissionResponse:
decision = await admit_roundtable_room(
request,
draft_id=body.draftId,
task_id=body.taskId,
room_session_id=body.roomSessionId,
)
if not decision.get("ok", False):
raise HTTPException(status_code=409, detail=decision)
return _response(decision)
@router.post("/heartbeat", response_model=RoomAdmissionResponse)
async def heartbeat_room(request: Request, body: RoomScopeRequest) -> RoomAdmissionResponse:
decision = await renew_roundtable_room(
request,
draft_id=body.draftId,
task_id=body.taskId,
room_session_id=body.roomSessionId,
)
if not decision.get("ok", False):
raise HTTPException(status_code=409, detail=decision)
return _response(decision)
@router.post("/release")
async def release_room(request: Request, body: RoomScopeRequest) -> dict[str, bool]:
_require_store(request)
released = await release_roundtable_room(
request,
draft_id=body.draftId,
task_id=body.taskId,
room_session_id=body.roomSessionId,
)
return {"released": released}
offline-backend-20260512/backend/app/gateway/routers/scheduled_tasks.py
+8 -3
View File
@@ -2,10 +2,11 @@
from __future__ import annotations
import asyncio
import mimetypes
import re
import zipfile
from io import BytesIO
import mimetypes
from pathlib import Path
from typing import Any
from urllib.parse import quote
@@ -210,15 +211,19 @@ async def get_scheduler_thread(request: Request) -> dict:
@router.get("/html-templates")
async def list_html_templates() -> dict:
async def list_html_templates(request: Request) -> dict:
"""可选的 HTML 模板清单(供「模板数据转换器」内置 agent 填充用)。
返回 ``{"templates": [{id, name, description}]}``,前端定时任务表单在选中
「模板数据转换器」后据此渲染「HTML 模板」下拉。开放给任意登录用户。
"""
from app.gateway.catalog_cache import cached_catalog
from deerflow.runtime.scheduler import template_fill
return {"templates": template_fill.list_templates()}
async def load() -> dict:
return {"templates": await asyncio.to_thread(template_fill.list_templates)}
return await cached_catalog(request, "html_templates", "scheduler-list", load)
class TemplateRenderRequest(BaseModel):
offline-backend-20260512/backend/app/gateway/routers/sensitive_words.py
+35 -3
View File
@@ -20,7 +20,15 @@ from datetime import datetime
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel, Field
from starlette.responses import Response
from app.gateway.api_read_cache import (
get_cache_version,
get_cached_json_payload,
release_cache_fill,
set_cached_json,
wait_for_cached_json_payload_or_claim_fill,
)
from app.gateway.deps import get_current_user, get_optional_user_from_request
from deerflow.persistence.sensitive_words import SensitiveWordStore
@@ -89,9 +97,33 @@ async def _require_manager(request: Request) -> None:
@router.get("", response_model=SensitiveWordListResponse)
async def list_sensitive_words(request: Request) -> SensitiveWordListResponse:
store = _get_store(request)
rows = await store.list_words()
return SensitiveWordListResponse(words=[SensitiveWordResponse(**r) for r in rows])
config = getattr(request.app.state, "config", None)
cache_key = "global"
version = await get_cache_version(config, "sensitive_words")
cached = await get_cached_json_payload(config, "sensitive_words", cache_key, version=version)
fill_token = None
if cached is None:
cached, fill_token = await wait_for_cached_json_payload_or_claim_fill(
config, "sensitive_words", cache_key, version=version, wait_ms=1000
)
if cached is not None:
return Response(content=cached, media_type="application/json") # type: ignore[return-value]
try:
store = _get_store(request)
rows = await store.list_words()
result = SensitiveWordListResponse(words=[SensitiveWordResponse(**r) for r in rows])
await set_cached_json(
config,
"sensitive_words",
cache_key,
result.model_dump(mode="json"),
expected_version=version,
ttl_seconds=3600,
)
return result
finally:
if fill_token is not None:
await release_cache_fill(config, "sensitive_words", cache_key, fill_token)
@router.put("", response_model=SensitiveWordListResponse)
offline-backend-20260512/backend/app/gateway/routers/service_tokens.py
+61
View File
@@ -0,0 +1,61 @@
"""Exchange a public service request for a short-lived scoped user token."""
from __future__ import annotations
import hmac
import logging
from datetime import timedelta
from fastapi import APIRouter, HTTPException, Request, status
from pydantic import BaseModel, Field
from app.gateway.auth import create_access_token
from app.gateway.deps import get_local_provider
from app.gateway.routers.auth import enforce_user_approved
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/v1/auth/service", tags=["auth"])
_TOKEN_TTL_SECONDS = 300
_ALLOWED_SCOPES = {"llmwiki:sync", "recommended-questions:read"}
_SERVICE_TOKEN_SECRET = "b4217f191838d12a"
class ServiceExchangeRequest(BaseModel):
username: str = Field(min_length=1, max_length=128)
scope: list[str] = Field(default_factory=lambda: ["llmwiki:sync"], min_length=1, max_length=1)
class ServiceExchangeResponse(BaseModel):
access_token: str
token_type: str = "bearer"
expires_in: int = _TOKEN_TTL_SECONDS
scope: list[str]
@router.post("/exchange", response_model=ServiceExchangeResponse)
async def exchange_service_token(payload: ServiceExchangeRequest, request: Request) -> ServiceExchangeResponse:
supplied = request.headers.get("X-Service-Token", "")
if not supplied or not hmac.compare_digest(supplied, _SERVICE_TOKEN_SECRET):
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid service token")
if len(payload.scope) != 1 or payload.scope[0] not in _ALLOWED_SCOPES:
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Unsupported service scope")
provider = get_local_provider()
user = await provider.find_user_by_email_local_part(payload.username.strip().lower())
if user is None:
raise HTTPException(status_code=404, detail="Target user not found")
enforce_user_approved(user)
token = create_access_token(
str(user.id),
expires_delta=timedelta(seconds=_TOKEN_TTL_SECONDS),
token_version=user.token_version,
auth_method="service",
scope=payload.scope,
)
logger.info(
"Issued scoped service token target_user_id=%s scope=%s",
user.id,
payload.scope,
)
return ServiceExchangeResponse(access_token=token, scope=payload.scope)
offline-backend-20260512/backend/app/gateway/routers/skills.py
+300 -60
View File
@@ -8,18 +8,34 @@ import shutil
import subprocess
import sys
import tempfile
import time
import zipfile
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
from fastapi import APIRouter, Body, Depends, File, HTTPException, Query, Request, UploadFile
from fastapi import APIRouter, Depends, File, HTTPException, Query, Request, Response, UploadFile
from fastapi.responses import FileResponse
from pydantic import BaseModel, Field
from sqlalchemy import select
from app.gateway.api_read_cache import (
get_cache_version,
get_cached_json_payload,
release_cache_fill,
set_cached_json,
wait_for_cached_json_payload_or_claim_fill,
)
from app.gateway.deps import get_agent_store, get_config, get_skill_store, get_tag_store
from app.gateway.path_utils import aresolve_thread_virtual_path
from app.gateway.perf_logging import finish_trace, trace_request, trace_stage
from app.gateway.skill_catalog_cache import (
_build_catalog_payload,
_load_usage_entries,
load_or_warm_skill_catalog_snapshot,
load_skill_catalog_projection,
refresh_skill_catalog_projection,
)
from deerflow.agents.lead_agent.prompt import refresh_skills_system_prompt_cache_async
from deerflow.config.app_config import AppConfig
from deerflow.config.extensions_config import ExtensionsConfig, SkillDisplayConfig, get_extensions_config, patch_extensions_config
@@ -39,6 +55,9 @@ logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api", tags=["skills"])
_UNSET = object()
_OWNER_USERNAME_CACHE_SECONDS = 60.0
_MAX_OWNER_USERNAME_CACHE_ENTRIES = 4096
_owner_username_cache: dict[str, tuple[float, str | None]] = {}
# ---------------------------------------------------------------------------
@@ -81,21 +100,59 @@ async def _attach_owner_usernames(responses: list["SkillResponse"]) -> list["Ski
provider = get_local_provider()
except Exception:
return responses
now = time.monotonic()
names: dict[str, str] = {}
for uid in owner_ids:
missing_ids: set[str] = set()
for user_id in owner_ids:
cached = _owner_username_cache.get(user_id)
if cached is not None and cached[0] > now:
if cached[1]:
names[user_id] = cached[1]
else:
missing_ids.add(user_id)
if missing_ids:
try:
user = await provider.get_user(uid)
users = await provider.get_users_by_ids(sorted(missing_ids))
except Exception:
user = None
local = _email_local_part(getattr(user, "email", None)) if user is not None else None
if local:
names[uid] = local
logger.debug("skill owner username batch lookup failed", exc_info=True)
users = None
if users is not None:
loaded = {
str(user.id): _email_local_part(getattr(user, "email", None))
for user in users
}
expires_at = now + _OWNER_USERNAME_CACHE_SECONDS
for user_id in missing_ids:
local = loaded.get(user_id)
_owner_username_cache[user_id] = (expires_at, local)
if local:
names[user_id] = local
if len(_owner_username_cache) > _MAX_OWNER_USERNAME_CACHE_ENTRIES:
expired = [key for key, value in _owner_username_cache.items() if value[0] <= now]
for key in expired:
_owner_username_cache.pop(key, None)
if len(_owner_username_cache) > _MAX_OWNER_USERNAME_CACHE_ENTRIES:
oldest = sorted(_owner_username_cache, key=lambda key: _owner_username_cache[key][0])
excess = len(_owner_username_cache) - _MAX_OWNER_USERNAME_CACHE_ENTRIES
for key in oldest[:excess]:
_owner_username_cache.pop(key, None)
for resp in responses:
if resp.owner_user_id:
resp.owner_username = names.get(resp.owner_user_id)
return responses
async def _refresh_catalog_snapshot(config: AppConfig) -> None:
"""Keep normalized MySQL skill rows current after a skill edit."""
try:
await refresh_skill_catalog_projection(config)
except Exception:
# A failed projection refresh must not turn a successful skill write
# into an API failure. Startup or the next edit can recover it.
logger.exception("Skill catalog projection refresh failed after a mutation")
def _can_view_skill_source(
category: SkillCategory,
owner_user_id: str | None,
@@ -316,6 +373,7 @@ def _skill_to_response(
skill: Skill,
ownership: dict[str, Any] | None = None,
*,
usage_entry: dict[str, Any] | None = None,
display_override: dict[str, Any] | None | object = _UNSET,
display_sample: dict[str, Any] | None = None,
favorited: bool = False,
@@ -327,11 +385,12 @@ def _skill_to_response(
``owner_user_id`` / ``published`` / ``builtin`` fields used for per-user
visibility and the frontend's owner badges.
"""
from deerflow.skills.usage import get_entry, latest_activity_at
from deerflow.skills.usage import get_entry
entry = get_entry(skill.name)
entry = usage_entry if usage_entry is not None else get_entry(skill.name)
created_at_str = entry.get("created_at")
last_at_str = latest_activity_at(skill.name)
activity_candidates = [entry.get("last_used_at"), entry.get("last_viewed_at"), entry.get("last_patched_at")]
last_at_str = max((value for value in activity_candidates if value), default=created_at_str)
created_at: datetime | None = None
last_at: datetime | None = None
if created_at_str:
@@ -405,6 +464,48 @@ def _skill_to_response(
)
def _catalog_item_to_response(
item: dict[str, Any],
ownership: dict[str, Any] | None,
*,
adapter: dict[str, Any] | None,
favorited: bool,
) -> SkillResponse:
"""Build a list response from the MySQL snapshot and MySQL metadata rows."""
try:
category = SkillCategory(str(item.get("category") or SkillCategory.PUBLIC.value))
except ValueError:
category = SkillCategory.PUBLIC
owner_user_id = ownership.get("owner_user_id") if ownership else None
return SkillResponse(
name=str(item.get("name") or ""),
name_zh=ownership.get("name_zh") if ownership else None,
description=str(item.get("description") or ""),
license=item.get("license"),
category=category,
enabled=bool(item.get("enabled", True)),
always_on=bool(ownership.get("always_on", item.get("always_on", False)) if ownership else item.get("always_on", False)),
use_count=int(item.get("use_count") or 0),
view_count=int(item.get("view_count") or 0),
patch_count=int(item.get("patch_count") or 0),
last_activity_at=item.get("last_activity_at"),
created_at=item.get("created_at"),
state=str(item.get("state") or "active"),
pinned=bool(item.get("pinned")),
featured=bool(item.get("featured")),
source=str(item.get("source") or "upload"),
owner_user_id=owner_user_id,
published=bool(ownership.get("published", False)) if ownership else False,
square_id=str(ownership.get("square_id") or "") if ownership else "",
builtin=bool(ownership.get("builtin", owner_user_id is None)) if ownership else category != SkillCategory.CUSTOM,
favorited=favorited,
detail=(ownership.get("detail_preview") or ownership.get("detail")) if ownership else None,
display=adapter.get("display") if adapter else None,
displaySample=adapter.get("sample") if adapter else None,
)
# ``checkfirst=True`` is a check-then-create operation. The in-process lock
# serializes concurrent requests in one worker; the duplicate-table fallback
# covers two Gateway workers racing during first access.
@@ -498,14 +599,18 @@ async def _list_skill_display_adapters(skill_names: list[str]) -> dict[str, dict
return {}
async with sf() as session:
result = await session.execute(
select(SkillDisplayAdapterRow).where(SkillDisplayAdapterRow.skill_name.in_(skill_names))
select(SkillDisplayAdapterRow.skill_name, SkillDisplayAdapterRow.display_config).where(
SkillDisplayAdapterRow.skill_name.in_(skill_names)
)
)
return {
row.skill_name: {
str(row.skill_name): {
"display": _json_text_loads(row.display_config),
"sample": _json_text_loads(row.sample_payload),
# The editor reads LONGTEXT sample_payload on demand; the list
# API only needs the lightweight display mapping.
"sample": None,
}
for row in result.scalars()
for row in result
}
@@ -716,7 +821,7 @@ async def _generate_skill_display_sample(skill: Skill, app_config: AppConfig) ->
if parsed:
return parsed
logger.warning("Skill display sample model returned non-JSON for %s; falling back to static mock", skill.name)
except asyncio.TimeoutError:
except TimeoutError:
logger.warning("Skill display sample LLM call timed out for %s; falling back to static mock", skill.name)
except Exception:
logger.warning("Skill display sample LLM call failed for %s; falling back to static mock", skill.name, exc_info=True)
@@ -814,20 +919,51 @@ async def _build_ownership_index(skill_store: SkillStore, user_id: str) -> dict[
return {record["name"]: record for record in records}
def _skill_search_tokens(search: str | None) -> list[str]:
"""Split a search query into lowercase AND-tokens (whitespace separated).
``网页 报告`` therefore requires both words to appear, but each may live in
a different field (name vs description) — unlike the old contiguous
substring match which demanded the whole phrase verbatim.
"""
if not search:
return []
return [token for token in search.strip().lower().split() if token]
def _skill_matches_tokens(skill: SkillResponse, tokens: list[str]) -> bool:
"""Every token must appear in at least one searchable skill field.
Fields: English name, Chinese display name, the SKILL.md ``description``,
the long-form ``detail`` annotation, and assigned tag names — so users can
find a skill by what it does, not only by its name.
"""
if not tokens:
return True
haystacks = [
(skill.name or "").lower(),
(skill.name_zh or "").lower(),
(skill.description or "").lower(),
(skill.detail or "").lower(),
" ".join(str(t.get("name") or "").lower() for t in (skill.tags or [])),
]
return all(any(token in field for field in haystacks) for token in tokens)
async def _attach_and_filter_tags(
tag_store: TagStore,
responses: list[SkillResponse],
search: str | None,
tag_ids: list[str] | None,
) -> list[SkillResponse]:
"""Bulk-attach tags to skill responses, then apply name/tag filters."""
"""Bulk-attach tags to skill responses, then apply search/tag filters."""
assignments = await tag_store.list_assignments("skill", [r.name for r in responses])
for resp in responses:
resp.tags = assignments.get(resp.name, [])
if search:
needle = search.strip().lower()
# Match the English name OR the Chinese display name so 中文 search works.
responses = [r for r in responses if needle in r.name.lower() or (r.name_zh or "").lower().find(needle) >= 0]
# Tags are attached first so tag names participate in the match.
tokens = _skill_search_tokens(search)
responses = [r for r in responses if _skill_matches_tokens(r, tokens)]
if tag_ids:
wanted = set(tag_ids)
responses = [r for r in responses if any(t.get("id") in wanted for t in r.tags)]
@@ -856,7 +992,7 @@ async def list_skills(
config: AppConfig = Depends(get_config),
skill_store: SkillStore = Depends(get_skill_store),
tag_store: TagStore = Depends(get_tag_store),
search: str | None = Query(default=None, description="Case-insensitive fuzzy match on skill name or Chinese name (name_zh)"),
search: str | None = Query(default=None, description="Whitespace-separated keywords (AND); case-insensitive substring match against skill name, Chinese name (name_zh), description, detail, and tag names"),
tag_ids: list[str] | None = Query(default=None, description="Keep skills carrying any of these tags (OR)"),
square: str | None = Query(default=None, description="Keep only skills filed under this publish square (发布广场). Omit for all squares."),
scope: str | None = Query(default=None, description="``mine`` filters to the skills the caller's 岗位 (position) makes visible."),
@@ -874,68 +1010,138 @@ async def list_skills(
),
),
) -> SkillsListResponse:
trace = trace_request(request, "skills_list")
cache_fill_token: str | None = None
cache_config = None
cache_key = ""
try:
user_id = _current_user_id(request)
if admin and not _is_admin_user(request):
raise HTTPException(status_code=403, detail="Only system admins can list all skills")
all_skills = get_or_new_skill_storage(app_config=config).load_skills(enabled_only=False)
if admin:
cacheable = (
not admin
and not (search or "").strip()
and not tag_ids
and not (square or "").strip()
and sort == "time_desc"
)
query_fingerprint = json.dumps(
{
"search": search,
"tag_ids": sorted(tag_ids or []),
"square": square,
"scope": scope,
"sort": sort,
},
ensure_ascii=False,
sort_keys=True,
separators=(",", ":"),
)
cache_key = f"user:{user_id}:query:{query_fingerprint}" if cacheable else ""
cache_version = None
request_app_state = getattr(getattr(request, "app", None), "state", None)
cache_config = getattr(request_app_state, "config", config)
if cacheable:
stage_started = time.perf_counter()
cache_version = await get_cache_version(cache_config, "skills")
cached_payload = await get_cached_json_payload(
cache_config,
"skills",
cache_key,
version=cache_version,
)
if cached_payload is None:
cached_payload, cache_fill_token = await wait_for_cached_json_payload_or_claim_fill(
cache_config,
"skills",
cache_key,
version=cache_version,
)
trace_stage(trace, "skills_redis_cache", stage_started)
if cached_payload is not None:
trace.setdefault("details", {})["cache"] = "redis_hit"
return Response(content=cached_payload, media_type="application/json")
stage_started = time.perf_counter()
catalog = await load_skill_catalog_projection(user_id=user_id, admin=admin)
if catalog is None:
# Preserve no-database operation. A MySQL-backed request always
# uses the normalized rows and does not scan SKILL.md files.
legacy_catalog = await load_or_warm_skill_catalog_snapshot(config)
if legacy_catalog is not None:
catalog = legacy_catalog
trace_stage(trace, "skills_legacy_projection_fallback", stage_started)
else:
# This request's dependency and the singleton share the active
# app config. Reuse the process instance to retain parsed skill
# metadata between catalogue reads.
storage = get_or_new_skill_storage(app_config=config)
skills_from_disk = await asyncio.to_thread(storage.load_skills, enabled_only=False)
usage_entries = await asyncio.to_thread(_load_usage_entries)
catalog = await asyncio.to_thread(_build_catalog_payload, skills_from_disk, usage_entries)
trace_stage(trace, "skills_memory_mode_catalog_load", stage_started)
else:
trace_stage(trace, "skills_mysql_projection", stage_started)
trace.setdefault("details", {})["skills_discovered"] = len(catalog)
if get_session_factory() is not None:
# Projection rows include the compact ownership fields; never load
# the SkillRow.detail LONGTEXT column on catalog requests.
ownership = {str(record["name"]): record for record in catalog}
trace_stage(trace, "skills_ownership_in_projection", stage_started)
elif admin:
stage_started = time.perf_counter()
ownership_records = await skill_store.list_all()
ownership = {record["name"]: record for record in ownership_records}
trace_stage(trace, "skills_ownership_db", stage_started)
else:
stage_started = time.perf_counter()
ownership = await _build_ownership_index(skill_store, user_id)
trace_stage(trace, "skills_ownership_db", stage_started)
# Skills the caller's 岗位 (position) granted into their "我的技能"
# collection. Admins browse everything, so the badge is moot for them.
favorite_names: set[str] = set()
if not admin:
try:
stage_started = time.perf_counter()
favorite_names = await skill_store.list_favorite_skill_names(user_id)
trace_stage(trace, "skills_favorites_db", stage_started)
except Exception:
logger.debug("skill favorites lookup failed", exc_info=True)
favorite_names = set()
display_adapters = await _list_skill_display_adapters([skill.name for skill in all_skills])
stage_started = time.perf_counter()
skill_names = [str(item["name"]) for item in catalog if item.get("name")]
display_adapters = await _list_skill_display_adapters(skill_names)
trace_stage(trace, "skills_display_adapters_db", stage_started)
stage_started = time.perf_counter()
responses: list[SkillResponse] = []
for skill in all_skills:
adapter = display_adapters.get(skill.name)
for item in catalog:
skill_name = str(item.get("name") or "")
if not skill_name:
continue
try:
category = SkillCategory(str(item.get("category") or SkillCategory.PUBLIC.value))
except ValueError:
category = SkillCategory.PUBLIC
adapter = display_adapters.get(skill_name)
record = ownership.get(skill_name)
# Public/built-in skills (no DB row) are visible to everyone.
# Custom skills must have an ownership record in the visible set.
if skill.category != SkillCategory.CUSTOM:
# Built-in skills usually have no DB row, but an admin-authored
# ``detail`` annotation materializes one — surface it when present.
responses.append(
_skill_to_response(
skill,
ownership.get(skill.name),
display_override=_display_override_from_adapter(adapter),
display_sample=adapter.get("sample") if adapter else None,
favorited=skill.name in favorite_names,
)
)
if category == SkillCategory.CUSTOM and record is None and not admin and skill_name not in favorite_names:
continue
record = ownership.get(skill.name)
if record is None:
if admin:
# Admin sees every custom skill on disk even without an
# ownership row (legacy / orphaned), so they can clean it up.
record = None
elif skill.name in favorite_names:
# A position granted this skill even though it isn't
# otherwise visible to the user (e.g. an unpublished custom
# skill). Surface it so it lands in their 我的技能.
record = await skill_store.get_any(skill.name)
else:
continue
responses.append(
_skill_to_response(
skill,
_catalog_item_to_response(
item,
record,
display_override=_display_override_from_adapter(adapter),
display_sample=adapter.get("sample") if adapter else None,
favorited=skill.name in favorite_names,
adapter=adapter,
favorited=skill_name in favorite_names,
)
)
trace_stage(trace, "skills_response_assembly", stage_started)
stage_started = time.perf_counter()
responses = await _attach_and_filter_tags(tag_store, responses, search, tag_ids)
trace_stage(trace, "skills_tags_db_and_filter", stage_started)
stage_started = time.perf_counter()
responses = await _attach_owner_usernames(responses)
trace_stage(trace, "skills_owner_usernames_db", stage_started)
square_id = square.strip() if square and square.strip() else None
if square_id:
from deerflow.config.system_settings import load_system_settings, resolve_default_square_id
@@ -952,13 +1158,30 @@ async def list_skills(
# mirroring how agents are added to 我的智能体. The browse-all 广场 therefore
# always shows every visible skill — the ``scope`` param is kept for API
# compatibility but intentionally does not filter here.
stage_started = time.perf_counter()
responses = _sort_skill_responses(responses, sort)
return SkillsListResponse(skills=responses)
trace_stage(trace, "skills_sort", stage_started)
trace.setdefault("details", {})["skills_returned"] = len(responses)
result = SkillsListResponse(skills=responses)
if cacheable:
await set_cached_json(
cache_config,
"skills",
cache_key,
result,
expected_version=cache_version,
)
trace.setdefault("details", {})["cache"] = "redis_miss"
return result
except HTTPException:
raise
except Exception as e:
logger.error(f"Failed to load skills: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=f"Failed to load skills: {str(e)}")
finally:
if cache_fill_token is not None and cache_config is not None and cache_key:
await release_cache_fill(cache_config, "skills", cache_key, cache_fill_token)
finish_trace(trace)
_MAX_SKILL_UPLOAD_BYTES = 512 * 1024 * 1024
@@ -1137,6 +1360,7 @@ async def install_skill_upload(
await skill_store.ensure_legacy(
{"name": result["skill_name"], "owner_user_id": user_id, "published": False}
)
await _refresh_catalog_snapshot(config)
await refresh_skills_system_prompt_cache_async()
return SkillInstallResponse(**result)
except FileNotFoundError as e:
@@ -1392,6 +1616,7 @@ async def install_skill(
await skill_store.ensure_legacy(
{"name": result.skill_name, "owner_user_id": user_id, "published": False}
)
await _refresh_catalog_snapshot(config)
await refresh_skills_system_prompt_cache_async()
return result
except FileNotFoundError as e:
@@ -1505,6 +1730,7 @@ async def duplicate_skill(
mark_uploaded(new_name)
except Exception:
pass
await _refresh_catalog_snapshot(config)
await refresh_skills_system_prompt_cache_async()
return SkillInstallResponse(
success=True,
@@ -1578,9 +1804,16 @@ async def list_custom_skills(
) -> SkillsListResponse:
try:
user_id = _current_user_id(request)
custom = [s for s in get_or_new_skill_storage(app_config=config).load_skills(enabled_only=False) if s.category == SkillCategory.CUSTOM]
ownership = await _build_ownership_index(skill_store, user_id)
display_adapters = await _list_skill_display_adapters([skill.name for skill in custom])
storage = get_or_new_skill_storage()
all_skills, usage_entries = await asyncio.gather(
asyncio.to_thread(storage.load_skills, enabled_only=False),
asyncio.to_thread(_load_usage_entries),
)
custom = [skill for skill in all_skills if skill.category == SkillCategory.CUSTOM]
ownership, display_adapters = await asyncio.gather(
_build_ownership_index(skill_store, user_id),
_list_skill_display_adapters([skill.name for skill in custom]),
)
responses = []
for skill in custom:
if skill.name not in ownership:
@@ -1590,6 +1823,7 @@ async def list_custom_skills(
_skill_to_response(
skill,
ownership[skill.name],
usage_entry=usage_entries.get(skill.name, {}),
display_override=_display_override_from_adapter(adapter),
display_sample=adapter.get("sample") if adapter else None,
)
@@ -1637,6 +1871,7 @@ async def update_custom_skill(skill_name: str, request: CustomSkillUpdateRequest
raise HTTPException(status_code=400, detail=f"Security scan blocked the edit: {scan.reason}")
prev_content = storage.read_custom_skill(skill_name)
storage.write_custom_skill(skill_name, SKILL_MD_FILE, request.content)
await _refresh_catalog_snapshot(config)
storage.append_history(
skill_name,
{
@@ -1706,6 +1941,7 @@ async def delete_custom_skill(
await skill_store.delete_admin(skill_name)
else:
await skill_store.delete(skill_name, user_id)
await _refresh_catalog_snapshot(config)
try:
await get_tag_store(request).unassign_all("skill", skill_name)
@@ -1808,6 +2044,7 @@ async def delete_builtin_skill(
skill_owner_user_id=None,
agent_store=agent_store,
)
await _refresh_catalog_snapshot(config)
await refresh_skills_system_prompt_cache_async()
await _sync_skill_view(skill.name)
logger.info("Administrator deleted built-in skill %s", skill.name)
@@ -1950,6 +2187,7 @@ async def update_custom_skill_state(
from deerflow.skills.usage import set_state as _usage_set_state
_usage_set_state(skill_name, body.state)
await _refresh_catalog_snapshot(config)
await refresh_skills_system_prompt_cache_async()
return _reload_skill_response(skill_name, record, config)
except HTTPException:
@@ -1981,6 +2219,7 @@ async def update_custom_skill_pinned(
from deerflow.skills.usage import set_pinned as _usage_set_pinned
_usage_set_pinned(skill_name, body.pinned)
await _refresh_catalog_snapshot(config)
return _reload_skill_response(skill_name, record, config)
except HTTPException:
raise
@@ -2071,6 +2310,7 @@ async def rollback_custom_skill(skill_name: str, request: SkillRollbackRequest,
storage.append_history(skill_name, history_entry)
raise HTTPException(status_code=400, detail=f"Rollback blocked by security scanner: {scan.reason}")
storage.write_custom_skill(skill_name, SKILL_MD_FILE, target_content)
await _refresh_catalog_snapshot(config)
storage.append_history(skill_name, history_entry)
await refresh_skills_system_prompt_cache_async()
return await get_custom_skill(skill_name, config)
@@ -3127,7 +3367,7 @@ async def _generate_skill_name_zh(skill: Skill, app_config: AppConfig, model: An
),
timeout=60,
)
except asyncio.TimeoutError as e:
except TimeoutError as e:
logger.warning("Skill name_zh LLM call timed out for %s", skill.name)
raise SkillNameZhModelError("大模型响应超时(60 秒)") from e
except Exception as e:
offline-backend-20260512/backend/app/gateway/routers/strategy_agents.py
+15 -1
View File
@@ -272,6 +272,7 @@ class StrategyAgentItem(BaseModel):
name: str
description: str = ""
model: str | None = None
model_prompt_suffix_enabled: bool = True
soul: str | None = None
skills: list[str] | None = None
created_at: datetime | str | None = None
@@ -287,6 +288,7 @@ class StrategyAngleCreateRequest(BaseModel):
description: str = Field(default="", max_length=2000)
soul: str = Field(default="", max_length=200_000)
model: str | None = Field(default=None, max_length=200)
model_prompt_suffix_enabled: bool = True
skills: list[str] = Field(default_factory=list, max_length=100)
sort_order: int | None = Field(default=None, ge=-10_000, le=10_000)
@@ -296,6 +298,7 @@ class StrategyAgentUpdateRequest(BaseModel):
description: str | None = Field(default=None, max_length=2000)
soul: str | None = Field(default=None, max_length=200_000)
model: str | None = Field(default=None, max_length=200)
model_prompt_suffix_enabled: bool | None = None
skills: list[str] | None = Field(default=None, max_length=100)
enabled: bool | None = None
sort_order: int | None = Field(default=None, ge=-10_000, le=10_000)
@@ -394,6 +397,7 @@ async def _to_item(
name=agent.name,
description=agent.description,
model=agent.model,
model_prompt_suffix_enabled=agent.model_prompt_suffix_enabled,
soul=agent.soul,
skills=agent.skills,
created_at=agent.created_at,
@@ -513,6 +517,7 @@ async def create_strategy_angle(
payload=payload,
name=name,
model=model,
model_prompt_suffix_enabled=payload.model_prompt_suffix_enabled,
skills=skills,
agent_store=agent_store,
role_store=role_store,
@@ -546,6 +551,7 @@ async def _create_strategy_angle_locked(
payload: StrategyAngleCreateRequest,
name: str,
model: str | None,
model_prompt_suffix_enabled: bool,
skills: list[str] | None,
agent_store: AgentStore,
role_store: StrategyAgentRoleStore,
@@ -579,6 +585,7 @@ async def _create_strategy_angle_locked(
description=payload.description,
skills=skills,
model=model,
model_prompt_suffix_enabled=model_prompt_suffix_enabled,
tool_groups=_DYNAMIC_ANGLE_TOOL_GROUPS,
)
await asyncio.to_thread(agents_router._write_agent_soul, agent_id, soul)
@@ -663,6 +670,11 @@ async def _update_strategy_agent_locked(
next_description = payload.description or "" if "description" in fields else str(record.get("description") or "")
extras = await asyncio.to_thread(agents_router._load_config_extras, agent_id)
next_model = _normalize_model(request, payload.model) if "model" in fields else extras["model"]
next_model_prompt_suffix_enabled = (
bool(payload.model_prompt_suffix_enabled)
if "model_prompt_suffix_enabled" in fields
else extras["model_prompt_suffix_enabled"]
)
next_skills = (
await agents_router._validate_strategy_agent_skills(
payload.skills,
@@ -686,7 +698,7 @@ async def _update_strategy_agent_locked(
original_role = dict(role)
role_store = get_strategy_agent_role_store(request)
try:
config_fields = {"name", "description", "model", "skills"}
config_fields = {"name", "description", "model", "model_prompt_suffix_enabled", "skills"}
if fields & config_fields:
await asyncio.to_thread(
agents_router._write_agent_config,
@@ -695,11 +707,13 @@ async def _update_strategy_agent_locked(
description=next_description,
skills=next_skills,
model=next_model,
model_prompt_suffix_enabled=next_model_prompt_suffix_enabled,
tool_groups=extras["tool_groups"],
common_questions=extras["common_questions"],
template_name=extras["template_name"],
seat_skill_directive=extras["seat_skill_directive"],
llmwiki_knowledge_base_ids=extras["llmwiki_knowledge_base_ids"],
chat_link=extras["chat_link"],
)
await agent_store.replace_skills(agent_id, next_skills)
await agent_store.set_extras(
offline-backend-20260512/backend/app/gateway/routers/stream_truncations.py
+181
View File
@@ -0,0 +1,181 @@
"""Admin-only model-stream truncation event log (模型截断记录).
Companion to the per-model ``strict_stream_integrity`` switch on the model
management page: the patched OpenAI-compatible streaming layer writes one row
per abnormally ended provider stream (silently committed half answers, strict
integrity detections, output-length caps, mid-stream transport errors). This
router serves the admin viewing page; rows are retained for at most one month
by the SQL repository.
"""
from __future__ import annotations
import logging
from datetime import datetime
from typing import Any
from fastapi import APIRouter, HTTPException, Query, Request
from pydantic import BaseModel, Field
from app.gateway.deps import get_optional_user_from_request, get_stream_truncation_store
from deerflow.persistence.stream_truncation.base import StreamTruncationQuery, StreamTruncationStore
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/admin/stream-truncations", tags=["stream-truncations"])
_EVENT_TYPES = (
"silent_no_terminal",
"incomplete_stream_error",
"length_truncated",
"content_filter",
"midstream_error",
)
class StreamTruncationEventResponse(BaseModel):
id: str
created_at: str | None = None
model_name: str | None = None
provider_completion_id: str | None = None
run_id: str | None = None
event_type: str
strict_mode: bool = False
finish_reason: str = "none"
status: str = "success"
error_type: str | None = None
chunk_count: int = 0
duration_ms: int = 0
first_chunk_ms: int = -1
detail: str | None = None
class StreamTruncationListResponse(BaseModel):
items: list[StreamTruncationEventResponse]
total: int = Field(..., description="Total rows matching the filter, ignoring pagination")
limit: int
offset: int
class StreamTruncationBreakdownItem(BaseModel):
model_name: str
total: int = 0
last_at: str | None = None
silent_no_terminal: int = 0
incomplete_stream_error: int = 0
length_truncated: int = 0
content_filter: int = 0
midstream_error: int = 0
class StreamTruncationBreakdownResponse(BaseModel):
items: list[StreamTruncationBreakdownItem]
async def _require_admin(request: Request) -> None:
user = await get_optional_user_from_request(request)
if user is None:
raise HTTPException(status_code=401, detail="Authentication is required")
if getattr(user, "system_role", None) != "admin":
raise HTTPException(status_code=403, detail="Stream truncation log is admin-only")
def _require_store(request: Request) -> StreamTruncationStore:
store = get_stream_truncation_store(request)
if store is None:
raise HTTPException(status_code=503, detail="Stream truncation store is not configured")
return store
def _parse_dt(value: str | datetime | None) -> datetime | None:
if not value:
return None
if isinstance(value, datetime):
return value
try:
return datetime.fromisoformat(value.replace("Z", "+00:00"))
except ValueError as exc:
raise HTTPException(status_code=422, detail=f"Invalid datetime: {value!r} ({exc})") from exc
def _build_query(
*,
model_name: str | None,
event_type: str | None,
since: str | datetime | None,
until: str | datetime | None,
limit: int,
offset: int,
) -> StreamTruncationQuery:
if event_type and event_type not in _EVENT_TYPES:
raise HTTPException(status_code=422, detail=f"Unknown event type: {event_type}")
return StreamTruncationQuery(
model_name=model_name or None,
event_type=event_type or None,
since=_parse_dt(since),
until=_parse_dt(until),
limit=limit,
offset=offset,
)
def _fill_breakdown_zeros(item: dict[str, Any]) -> dict[str, Any]:
for event_type in _EVENT_TYPES:
item.setdefault(event_type, 0)
return item
@router.get("", response_model=StreamTruncationListResponse)
async def list_stream_truncations(
request: Request,
model_name: str | None = Query(default=None, max_length=128),
event_type: str | None = Query(default=None, max_length=32),
since: str | None = Query(default=None),
until: str | None = Query(default=None),
limit: int = Query(default=50, ge=1, le=200),
offset: int = Query(default=0, ge=0),
) -> StreamTruncationListResponse:
"""List truncation events, newest first."""
await _require_admin(request)
store = _require_store(request)
query = _build_query(
model_name=model_name,
event_type=event_type,
since=since,
until=until,
limit=limit,
offset=offset,
)
items = await store.list(query)
total = await store.count(query)
return StreamTruncationListResponse(
items=[StreamTruncationEventResponse(**item) for item in items],
total=total,
limit=limit,
offset=offset,
)
@router.get("/breakdown", response_model=StreamTruncationBreakdownResponse)
async def stream_truncation_breakdown(
request: Request,
since: str | None = Query(default=None),
until: str | None = Query(default=None),
) -> StreamTruncationBreakdownResponse:
"""Per-model aggregate of truncation events over the query window."""
await _require_admin(request)
store = _require_store(request)
query = _build_query(
model_name=None,
event_type=None,
since=since,
until=until,
limit=1,
offset=0,
)
rows = await store.model_breakdown(query)
return StreamTruncationBreakdownResponse(
items=[
StreamTruncationBreakdownItem(**_fill_breakdown_zeros(item))
for item in rows
]
)
offline-backend-20260512/backend/app/gateway/routers/system_settings.py
+332 -7
View File
@@ -24,11 +24,14 @@ from deerflow.config.system_settings import (
HtmlMapSettings,
LeaderboardSettings,
ModelConcurrencySettings,
ModelPromptSuffix,
PluginDomainAgentMapping,
PluginManagementSettings,
PromptPrefixSettings,
PublishSquare,
PublishSquaresSettings,
RetryKeepPartialSettings,
RoundtableRoomLimitSettings,
SkillCompressionSettings,
StrategyProjectSettings,
WordHeadingNumberingStyle,
@@ -40,12 +43,138 @@ from deerflow.config.system_settings import (
save_system_settings,
set_module_tag_ids,
)
from deerflow.html_runtime.addresses import (
html_map_tile_url,
html_runtime_base_url,
normalize_html_map_tile_url,
normalize_html_runtime_base_url,
)
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/system-settings", tags=["system-settings"])
@router.get("/model-prompt-suffixes", response_model=dict[str, ModelPromptSuffix])
async def get_model_prompt_suffixes(request: Request) -> dict[str, ModelPromptSuffix]:
await _require_model_prompt_admin(request)
return load_system_settings().model_prompt_suffixes
@router.put("/model-prompt-suffixes/{model_name:path}", response_model=ModelPromptSuffix)
async def update_model_prompt_suffix(model_name: str, body: ModelPromptSuffix, request: Request) -> ModelPromptSuffix:
await _require_model_prompt_admin(request)
current = load_system_settings()
# Retain editable entries for temporarily removed models so an administrator
# can still disable them. New entries must target a real configured model.
if get_app_config().get_model_config(model_name) is None and model_name not in current.model_prompt_suffixes:
raise HTTPException(status_code=404, detail="模型不存在")
if body.enabled and not body.prompt.strip():
raise HTTPException(status_code=422, detail="启用时请填写追加提示词")
current.model_prompt_suffixes[model_name] = body
save_system_settings(current)
return body
async def _require_model_prompt_admin(request: Request) -> None:
user = await get_optional_user_from_request(request)
if user is None:
raise HTTPException(status_code=401, detail="请先登录")
if getattr(user, "system_role", None) != "admin":
raise HTTPException(status_code=403, detail="仅管理员可管理模型追加提示词")
class ApiReadCacheResponse(BaseModel):
enabled: bool = False
redis_configured: bool = False
ttl_seconds: int = 300
max_value_bytes: int = 524288
browser_status_enabled: bool = True
collections: list[dict] = Field(default_factory=list)
class ApiReadCacheUpdateRequest(BaseModel):
enabled: bool | None = None
ttl_seconds: int | None = Field(default=None, ge=1, le=3600)
browser_status_enabled: bool | None = None
collection_overrides: dict[str, bool] | None = None
async def _require_performance_admin(request: Request) -> None:
# Preserve no-auth deployments, but never swallow an invalid/missing login.
user = await get_current_user_from_request(request)
if user is not None and getattr(user, "system_role", None) != "admin":
raise HTTPException(status_code=403, detail="仅管理员可管理接口性能设置")
@router.get("/api-read-cache", response_model=ApiReadCacheResponse)
async def get_api_read_cache_setting(request: Request) -> ApiReadCacheResponse:
"""Read the persisted administrator-controlled Redis cache switch."""
await _require_performance_admin(request)
from app.gateway.api_read_cache import api_read_cache_redis_configured, get_api_cache_controls, get_api_read_cache_settings
config = getattr(request.app.state, "config", None)
cache_config = getattr(config, "api_read_cache", None)
enabled, ttl_seconds = await get_api_read_cache_settings(config, force_refresh=True)
return ApiReadCacheResponse(
enabled=enabled,
redis_configured=api_read_cache_redis_configured(config),
ttl_seconds=ttl_seconds,
max_value_bytes=int(getattr(cache_config, "max_value_bytes", 524288) or 524288),
**await get_api_cache_controls(config),
)
@router.put("/api-read-cache", response_model=ApiReadCacheResponse)
async def update_api_read_cache_setting(
request: Request,
body: ApiReadCacheUpdateRequest,
) -> ApiReadCacheResponse:
"""Save the Redis cache switch to MySQL and apply it immediately."""
await _require_performance_admin(request)
from app.gateway.api_read_cache import (
API_READ_CACHE_COLLECTIONS,
api_read_cache_redis_configured,
get_api_cache_controls,
get_api_read_cache_settings,
invalidate_cached_collection,
save_api_read_cache_settings,
)
config = getattr(request.app.state, "config", None)
unknown = set(body.collection_overrides or {}) - set(API_READ_CACHE_COLLECTIONS)
if unknown:
raise HTTPException(status_code=422, detail=f"未知缓存接口：{', '.join(sorted(unknown))}")
try:
await save_api_read_cache_settings(
body.enabled, body.ttl_seconds, config=config,
collection_overrides=body.collection_overrides,
browser_status_enabled=body.browser_status_enabled,
)
await asyncio.gather(
*(
invalidate_cached_collection(config, collection, force=True)
for collection in API_READ_CACHE_COLLECTIONS
)
)
except Exception as exc:
logger.exception("Could not persist API Redis cache setting")
raise HTTPException(status_code=503, detail="无法将缓存设置保存到 MySQL") from exc
cache_config = getattr(config, "api_read_cache", None)
enabled, ttl_seconds = await get_api_read_cache_settings(config, force_refresh=True)
return ApiReadCacheResponse(
enabled=enabled,
redis_configured=api_read_cache_redis_configured(config),
ttl_seconds=ttl_seconds,
max_value_bytes=int(getattr(cache_config, "max_value_bytes", 524288) or 524288),
**await get_api_cache_controls(config),
)
class PromptPrefixResponse(BaseModel):
image_generation_unavailable_hint_enabled: bool = False
python_dependency_delete_protection_enabled: bool = False
enabled: bool = Field(..., description="Whether the toggle is exposed to users")
switch_label: str = Field(..., description="Toggle label shown in the chat input")
prompt_prefix: str = Field(..., description="Prefix text prepended to the first user message")
@@ -121,6 +250,8 @@ async def update_strategy_project_settings(
class PromptPrefixUpdateRequest(BaseModel):
image_generation_unavailable_hint_enabled: bool | None = None
python_dependency_delete_protection_enabled: bool | None = None
enabled: bool | None = None
switch_label: str | None = None
prompt_prefix: str | None = None
@@ -179,6 +310,16 @@ class HtmlMapSettingsUpdateRequest(BaseModel):
engine: Literal["echarts", "leaflet"] | None = None
class HtmlRuntimeSettingsResponse(BaseModel):
base_url: str
map_tile_url: str
class HtmlRuntimeSettingsUpdateRequest(BaseModel):
base_url: str
map_tile_url: str
class LeaderboardSettingsResponse(BaseModel):
exclude_user_ids: list[str] = Field(default_factory=list)
include_scheduled: bool = True
@@ -197,6 +338,8 @@ class LeaderboardSettingsUpdateRequest(BaseModel):
def _to_response(settings: PromptPrefixSettings) -> PromptPrefixResponse:
return PromptPrefixResponse(
image_generation_unavailable_hint_enabled=settings.image_generation_unavailable_hint_enabled,
python_dependency_delete_protection_enabled=settings.python_dependency_delete_protection_enabled,
enabled=settings.enabled,
switch_label=settings.switch_label,
prompt_prefix=settings.prompt_prefix,
@@ -389,7 +532,22 @@ class WriteFileCompatibilityUpdateRequest(BaseModel):
enabled: bool
class RetryKeepPartialResponse(BaseModel):
enabled: bool = Field(
default=True,
description=(
"Keep the already-streamed partial answer visible when a model "
"call is silently retried"
),
)
class RetryKeepPartialUpdateRequest(BaseModel):
enabled: bool
class ConversationContextResponse(BaseModel):
max_run_steps: int = 1000
recent_turn_window_enabled: bool = Field(
default=False,
description="Whether all lead-agent model requests keep only the latest three dialogue turns",
@@ -398,10 +556,39 @@ class ConversationContextResponse(BaseModel):
default=3,
description="Fixed number of real user turns retained by the sliding window",
)
max_conversation_rounds: int = Field(
default=20,
ge=1,
le=50,
description="Maximum real user turns allowed in one ordinary chat thread",
)
use_question_as_title: bool = Field(
default=False,
description="Whether the first user question is the final title, without post-answer title generation",
)
class ConversationContextUpdateRequest(BaseModel):
recent_turn_window_enabled: bool
max_run_steps: int | None = Field(default=None, ge=20, le=10000, strict=True)
recent_turn_window_enabled: bool | None = None
max_conversation_rounds: int | None = Field(default=None, ge=1, le=50)
use_question_as_title: bool | None = None
class RoundtableRoomLimitResponse(BaseModel):
enabled: bool = True
max_rooms: int = 3
max_hold_minutes: int = 30
max_cycles: int = 8
write_file_compatibility_enabled: bool = False
class RoundtableRoomLimitUpdateRequest(BaseModel):
enabled: bool | None = None
max_rooms: int | None = Field(default=None, ge=1, le=20)
max_hold_minutes: int | None = Field(default=None, ge=1, le=1440)
max_cycles: int | None = Field(default=None, ge=2, le=100)
write_file_compatibility_enabled: bool | None = None
class OrdinaryKnowledgeRetrievalResponse(BaseModel):
@@ -480,10 +667,36 @@ async def update_write_file_compatibility_setting(
)
@router.get("/retry-keep-partial", response_model=RetryKeepPartialResponse)
async def get_retry_keep_partial_setting() -> RetryKeepPartialResponse:
"""Return the retry-keeps-partial display switch (readable by chat clients)."""
settings = load_system_settings().retry_keep_partial
return RetryKeepPartialResponse(enabled=settings.enabled)
@router.put("/retry-keep-partial", response_model=RetryKeepPartialResponse)
async def update_retry_keep_partial_setting(
body: RetryKeepPartialUpdateRequest,
request: Request,
) -> RetryKeepPartialResponse:
await _require_admin(request)
current = load_system_settings()
updated = RetryKeepPartialSettings(enabled=body.enabled)
saved = save_system_settings(current.model_copy(update={"retry_keep_partial": updated}))
logger.info(
"Retry-keeps-partial display switch changed: enabled=%s",
saved.retry_keep_partial.enabled,
)
return RetryKeepPartialResponse(enabled=saved.retry_keep_partial.enabled)
def _conversation_context_response(settings: ConversationContextSettings) -> ConversationContextResponse:
return ConversationContextResponse(
max_run_steps=settings.max_run_steps,
recent_turn_window_enabled=settings.recent_turn_window_enabled,
recent_turns=3,
max_conversation_rounds=settings.max_conversation_rounds,
use_question_as_title=settings.use_question_as_title,
)
@@ -498,20 +711,72 @@ async def update_conversation_context_setting(
body: ConversationContextUpdateRequest,
request: Request,
) -> ConversationContextResponse:
"""Enable or disable the fixed recent-three-turn window. Admin only."""
"""Update the recent-turn window or ordinary-chat round cap. Admin only."""
await _require_admin(request)
current = load_system_settings()
updated = current.conversation_context.model_copy(
update={"recent_turn_window_enabled": body.recent_turn_window_enabled}
)
updated_fields = body.model_dump(exclude_unset=True, exclude_none=True)
if not updated_fields:
raise HTTPException(status_code=422, detail="至少需要提供一个上下文设置项")
updated = current.conversation_context.model_copy(update=updated_fields)
saved = save_system_settings(current.model_copy(update={"conversation_context": updated}))
logger.info(
"Conversation recent-turn window changed: enabled=%s retained_turns=3",
"Conversation runtime settings changed: recent_turn_window=%s use_question_as_title=%s max_conversation_rounds=%s max_run_steps=%s",
saved.conversation_context.recent_turn_window_enabled,
saved.conversation_context.use_question_as_title,
saved.conversation_context.max_conversation_rounds,
saved.conversation_context.max_run_steps,
)
return _conversation_context_response(saved.conversation_context)
def _roundtable_room_limit_response(settings: RoundtableRoomLimitSettings) -> RoundtableRoomLimitResponse:
return RoundtableRoomLimitResponse(
enabled=settings.enabled,
max_rooms=settings.max_rooms,
max_hold_minutes=settings.max_hold_minutes,
max_cycles=settings.max_cycles,
write_file_compatibility_enabled=settings.write_file_compatibility_enabled,
)
@router.get("/roundtable-room-limit", response_model=RoundtableRoomLimitResponse)
async def get_roundtable_room_limit_setting() -> RoundtableRoomLimitResponse:
"""Return the global step-2 room and closeout settings."""
return _roundtable_room_limit_response(load_system_settings().roundtable_room_limit)
@router.put("/roundtable-room-limit", response_model=RoundtableRoomLimitResponse)
async def update_roundtable_room_limit_setting(
body: RoundtableRoomLimitUpdateRequest,
request: Request,
) -> RoundtableRoomLimitResponse:
"""Update global room limits and the coordinator closeout round. Admin only."""
await _require_admin(request)
current = load_system_settings()
updates: dict[str, object] = {}
if body.enabled is not None:
updates["enabled"] = body.enabled
if body.max_rooms is not None:
updates["max_rooms"] = body.max_rooms
if body.max_hold_minutes is not None:
updates["max_hold_minutes"] = body.max_hold_minutes
if body.max_cycles is not None:
updates["max_cycles"] = body.max_cycles
if body.write_file_compatibility_enabled is not None:
updates["write_file_compatibility_enabled"] = body.write_file_compatibility_enabled
updated = current.roundtable_room_limit.model_copy(update=updates)
saved = save_system_settings(current.model_copy(update={"roundtable_room_limit": updated}))
logger.info(
"Roundtable runtime limits changed: enabled=%s max_rooms=%s max_hold_minutes=%s max_cycles=%s write_file_compatibility=%s",
saved.roundtable_room_limit.enabled,
saved.roundtable_room_limit.max_rooms,
saved.roundtable_room_limit.max_hold_minutes,
saved.roundtable_room_limit.max_cycles,
saved.roundtable_room_limit.write_file_compatibility_enabled,
)
return _roundtable_room_limit_response(saved.roundtable_room_limit)
@router.get(
"/ordinary-knowledge-retrieval",
response_model=OrdinaryKnowledgeRetrievalResponse,
@@ -806,6 +1071,38 @@ async def update_html_map_settings(
return HtmlMapSettingsResponse(engine=saved.html_map.engine)
@router.get("/html-runtime", response_model=HtmlRuntimeSettingsResponse)
async def get_html_runtime_settings() -> HtmlRuntimeSettingsResponse:
"""Return the addresses newly generated HTML pages will use."""
return HtmlRuntimeSettingsResponse(
base_url=html_runtime_base_url(),
map_tile_url=html_map_tile_url(),
)
@router.put("/html-runtime", response_model=HtmlRuntimeSettingsResponse)
async def update_html_runtime_settings(
body: HtmlRuntimeSettingsUpdateRequest,
request: Request,
) -> HtmlRuntimeSettingsResponse:
"""Save the browser-reachable runtime and map-tile addresses. Admin only."""
await _require_admin(request)
try:
base_url = normalize_html_runtime_base_url(body.base_url)
map_tile_url = normalize_html_map_tile_url(body.map_tile_url)
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
current = load_system_settings()
updated = current.html_runtime.model_copy(
update={"base_url": base_url, "map_tile_url": map_tile_url}
)
saved = save_system_settings(current.model_copy(update={"html_runtime": updated}))
return HtmlRuntimeSettingsResponse(
base_url=saved.html_runtime.base_url,
map_tile_url=saved.html_runtime.map_tile_url,
)
@router.get("/private-square", response_model=PrivateSquareStatusResponse)
async def get_private_square_status() -> PrivateSquareStatusResponse:
"""Return whether the private square is enabled and a password is set."""
@@ -939,6 +1236,12 @@ class AppearanceResponse(BaseModel):
default_skill_view: Literal["card", "list"] = Field(..., description="System default skill list display mode — card or list")
default_agent_view: Literal["card", "list"] = Field(..., description="System default agent list display mode — card or list")
default_layout_mode: Literal["full", "compact"] = Field(..., description="System default interface layout — full (sidebar + header) or compact (hidden)")
default_composer_width_full: int = Field(..., description="System default composer width percent for full layout. 100 keeps the built-in width.")
default_composer_width_compact: int = Field(..., description="System default composer width percent for compact layout. 100 keeps the built-in width.")
default_composer_height_full: int = Field(..., description="System default composer height percent for full layout. 100 keeps the built-in height.")
default_composer_height_compact: int = Field(..., description="System default composer height percent for compact layout. 100 keeps the built-in height.")
default_agent_composer_width: int = Field(..., description="System default agent composer width percent. 100 keeps the built-in width.")
default_agent_composer_height: int = Field(..., description="System default agent composer height percent. 100 keeps the built-in height.")
compact_shortcut_buttons: list[CompactShortcutButton] = Field(default_factory=list, description="Shortcut buttons rendered in top bars")
custom_themes: list[CustomThemePreset] = Field(default_factory=list, description="Administrator-managed runtime theme presets")
@@ -948,6 +1251,12 @@ class AppearanceUpdateRequest(BaseModel):
default_skill_view: Literal["card", "list"] | None = Field(default=None)
default_agent_view: Literal["card", "list"] | None = Field(default=None)
default_layout_mode: Literal["full", "compact"] | None = Field(default=None)
default_composer_width_full: int | None = Field(default=None)
default_composer_width_compact: int | None = Field(default=None)
default_composer_height_full: int | None = Field(default=None)
default_composer_height_compact: int | None = Field(default=None)
default_agent_composer_width: int | None = Field(default=None)
default_agent_composer_height: int | None = Field(default=None)
compact_shortcut_buttons: list[CompactShortcutButton] | None = Field(default=None)
custom_themes: list[CustomThemePreset] | None = Field(default=None)
@@ -1021,6 +1330,12 @@ def _appearance_response(appearance) -> AppearanceResponse:
default_skill_view=appearance.default_skill_view,
default_agent_view=appearance.default_agent_view,
default_layout_mode=appearance.default_layout_mode,
default_composer_width_full=appearance.default_composer_width_full,
default_composer_width_compact=appearance.default_composer_width_compact,
default_composer_height_full=appearance.default_composer_height_full,
default_composer_height_compact=appearance.default_composer_height_compact,
default_agent_composer_width=appearance.default_agent_composer_width,
default_agent_composer_height=appearance.default_agent_composer_height,
compact_shortcut_buttons=sorted(
appearance.compact_shortcut_buttons,
key=lambda item: item.sort_order,
@@ -1079,7 +1394,17 @@ async def update_appearance_settings(body: AppearanceUpdateRequest, request: Req
current = load_system_settings()
fields = body.model_dump(exclude_unset=True)
update: dict[str, object] = {}
for key in ("default_skill_view", "default_agent_view", "default_layout_mode"):
for key in (
"default_skill_view",
"default_agent_view",
"default_layout_mode",
"default_composer_width_full",
"default_composer_width_compact",
"default_composer_height_full",
"default_composer_height_compact",
"default_agent_composer_width",
"default_agent_composer_height",
):
if key in fields and fields[key] is not None:
update[key] = fields[key]
if "compact_shortcut_buttons" in fields and body.compact_shortcut_buttons is not None:
offline-backend-20260512/backend/app/gateway/routers/tags.py
+32 -3
View File
@@ -8,12 +8,14 @@ without asking an administrator to tag every item.
from __future__ import annotations
import asyncio
import logging
import uuid
from fastapi import APIRouter, Depends, HTTPException, Query
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from pydantic import BaseModel, Field
from app.gateway.catalog_cache import cached_catalog, catalog_key, invalidate_catalog
from app.gateway.deps import get_tag_store
from deerflow.persistence.tags import TAG_TYPES, TagStore
@@ -22,6 +24,13 @@ logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/tags", tags=["tags"])
async def _invalidate_tag_catalogues(request: Request) -> None:
await asyncio.gather(*(
invalidate_catalog(request, collection)
for collection in ("tags", "light_apps", "agents", "skills", "recommended_questions")
))
def _validate_tag_type(tag_type: str) -> str:
if tag_type not in TAG_TYPES:
raise HTTPException(status_code=422, detail=f"Invalid tag type '{tag_type}'. Allowed: {', '.join(TAG_TYPES)}")
@@ -68,14 +77,19 @@ class TagAssignmentRequest(BaseModel):
@router.get("", response_model=TagListResponse, summary="List Tags")
async def list_tags(
request: Request,
tag_type: str | None = Query(default=None, description="Filter by tag type"),
store: TagStore = Depends(get_tag_store),
) -> TagListResponse:
if tag_type is not None:
_validate_tag_type(tag_type)
try:
async def load() -> TagListResponse:
records = await store.list_tags(tag_type)
return TagListResponse(tags=[TagResponse(**r) for r in records])
try:
return await cached_catalog(request, "tags", catalog_key("list", tag_type), load, TagListResponse)
except Exception as e:
logger.error("Failed to list tags: %s", e, exc_info=True)
raise HTTPException(status_code=500, detail=f"Failed to list tags: {e}")
@@ -84,6 +98,7 @@ async def list_tags(
@router.post("", response_model=TagResponse, status_code=201, summary="Create Tag")
async def create_tag(
body: TagCreateRequest,
request: Request,
store: TagStore = Depends(get_tag_store),
) -> TagResponse:
_validate_tag_type(body.tag_type)
@@ -97,6 +112,7 @@ async def create_tag(
"description": body.description,
}
)
await _invalidate_tag_catalogues(request)
return TagResponse(**record)
except ValueError as e:
raise HTTPException(status_code=409, detail=str(e))
@@ -111,6 +127,7 @@ async def create_tag(
async def update_tag(
tag_id: str,
body: TagUpdateRequest,
request: Request,
store: TagStore = Depends(get_tag_store),
) -> TagResponse:
update_data = body.model_dump(exclude_unset=True)
@@ -120,6 +137,7 @@ async def update_tag(
record = await store.update_tag(tag_id, update_data)
if record is None:
raise HTTPException(status_code=404, detail=f"Tag '{tag_id}' not found")
await _invalidate_tag_catalogues(request)
return TagResponse(**record)
except ValueError as e:
raise HTTPException(status_code=409, detail=str(e))
@@ -133,12 +151,14 @@ async def update_tag(
@router.delete("/{tag_id}", status_code=204, summary="Delete Tag")
async def delete_tag(
tag_id: str,
request: Request,
store: TagStore = Depends(get_tag_store),
) -> None:
try:
deleted = await store.delete_tag(tag_id)
if not deleted:
raise HTTPException(status_code=404, detail=f"Tag '{tag_id}' not found")
await _invalidate_tag_catalogues(request)
except HTTPException:
raise
except Exception as e:
@@ -157,12 +177,19 @@ async def delete_tag(
async def get_resource_tags(
target_type: str,
target_id: str,
request: Request,
store: TagStore = Depends(get_tag_store),
) -> TagListResponse:
_validate_tag_type(target_type)
try:
async def load() -> TagListResponse:
records = await store.get_target_tags(target_type, target_id)
return TagListResponse(tags=[TagResponse(**r) for r in records])
try:
return await cached_catalog(
request, "tags", catalog_key("assignment", target_type, target_id), load, TagListResponse,
)
except Exception as e:
logger.error("Failed to get resource tags: %s", e, exc_info=True)
raise HTTPException(status_code=500, detail=f"Failed to get resource tags: {e}")
@@ -177,11 +204,13 @@ async def set_resource_tags(
target_type: str,
target_id: str,
body: TagAssignmentRequest,
request: Request,
store: TagStore = Depends(get_tag_store),
) -> TagListResponse:
_validate_tag_type(target_type)
try:
records = await store.set_target_tags(target_type, target_id, body.tag_ids)
await _invalidate_tag_catalogues(request)
return TagListResponse(tags=[TagResponse(**r) for r in records])
except ValueError as e:
raise HTTPException(status_code=422, detail=str(e))
offline-backend-20260512/backend/app/gateway/routers/task_analysis_qa.py
+20 -4
View File
@@ -45,6 +45,7 @@ from app.gateway.routers.multi_agent import (
)
from app.gateway.routers.task_analysis import _store, _task_or_404
from deerflow.persistence.task_analysis.base import TaskAnalysisStore
from deerflow.runtime.task_analysis.thread_metadata import task_analysis_thread_metadata
logger = logging.getLogger(__name__)
@@ -132,9 +133,14 @@ async def qa_init(request: Request, task_id: str) -> QaInitResponse:
agent_meta = await _get_agents_batch_direct(request, agent_ids)
# 并行建席位 thread + 总控 thread。
seat_threads = await _gather_thread_ids(request, agent_ids)
seat_threads = await _gather_thread_ids(request, task, agent_ids)
coord_thread = await _create_thread_direct(
request, metadata={"thread_type": "task-analysis-qa", "system": True}
request,
metadata=task_analysis_thread_metadata(
task,
thread_type="task-analysis-qa",
role="coordinator",
),
)
thread_ids: dict[str, str] = {**seat_threads, COORDINATOR_AGENT_ID: coord_thread}
@@ -179,12 +185,22 @@ async def qa_init(request: Request, task_id: str) -> QaInitResponse:
)
async def _gather_thread_ids(request: Request, agent_ids: list[str]) -> dict[str, str]:
async def _gather_thread_ids(
request: Request,
task: dict[str, Any],
agent_ids: list[str],
) -> dict[str, str]:
import asyncio
async def _one(agent_id: str) -> tuple[str, str]:
thread_id = await _create_thread_direct(
request, metadata={"thread_type": "task-analysis-qa", "system": True}
request,
metadata=task_analysis_thread_metadata(
task,
thread_type="task-analysis-qa",
role="seat",
seat_id=agent_id,
),
)
return agent_id, thread_id
offline-backend-20260512/backend/app/gateway/routers/task_buttons.py
+8 -2
View File
@@ -26,6 +26,7 @@ from datetime import datetime
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel, Field
from app.gateway.catalog_cache import cached_catalog, invalidate_catalog
from app.gateway.deps import get_current_user, get_optional_user_from_request
from deerflow.persistence.task_buttons import TaskButtonStore
@@ -110,8 +111,12 @@ async def _require_admin(request: Request) -> None:
@router.get("", response_model=TaskButtonListResponse)
async def list_task_buttons(request: Request) -> TaskButtonListResponse:
store = _get_store(request)
rows = await store.list_buttons()
return TaskButtonListResponse(buttons=[TaskButtonResponse(**r) for r in rows])
async def load() -> TaskButtonListResponse:
rows = await store.list_buttons()
return TaskButtonListResponse(buttons=[TaskButtonResponse(**r) for r in rows])
return await cached_catalog(request, "task_buttons", "all", load, TaskButtonListResponse)
@router.put("", response_model=TaskButtonListResponse)
@@ -123,4 +128,5 @@ async def replace_task_buttons(request: Request, body: TaskButtonReplaceRequest)
[b.model_dump() for b in body.buttons],
updated_by=user_id,
)
await invalidate_catalog(request, "task_buttons")
return TaskButtonListResponse(buttons=[TaskButtonResponse(**r) for r in rows])
offline-backend-20260512/backend/app/gateway/routers/task_deeplink_details.py
+190
View File
@@ -0,0 +1,190 @@
"""Task-detail API for externally initiated single-agent deep links.
The detail snapshot is deliberately owned by DeerFlow. The embedding system
writes it once, then the browser only reads it after login; no frontend call is
made back to the embedding system for task content.
"""
from __future__ import annotations
import json
from typing import Any, Literal
from fastapi import APIRouter, HTTPException, Path, Query, Request
from pydantic import AliasChoices, BaseModel, ConfigDict, Field, field_validator
from app.gateway.deps import get_task_deeplink_detail_store
router = APIRouter(prefix="/api/task-deeplink-details", tags=["task-deeplink-details"])
_B5_DETAIL_TYPE = "b5-1"
_MAX_EXTRA_BYTES = 64 * 1024
_MAX_SUB_TYPE = 64
class TaskDeeplinkDetailUpsertRequest(BaseModel):
"""Complete task snapshot supplied by the embedding system.
``extra`` is intentionally an object, not a bag of top-level fields, so
the stable contract stays small while integrators can attach their own
structured fields. ``type`` is the business type (the deep-link
``?type=``): the same task id may hold one snapshot per type; omitting it
addresses the legacy no-type partition.
"""
model_config = ConfigDict(populate_by_name=True)
task_name: str = Field(
min_length=1,
max_length=512,
validation_alias=AliasChoices("taskName", "task_name"),
)
task_content: str = Field(
min_length=1,
max_length=20_000,
validation_alias=AliasChoices("taskContent", "task_content"),
)
overview: str = Field(default="", max_length=4_000)
detail_sub_type: str = Field(
default="",
max_length=_MAX_SUB_TYPE,
validation_alias=AliasChoices("type", "taskType", "sub_type"),
)
extra: dict[str, Any] = Field(default_factory=dict)
@field_validator("task_name", "task_content", "overview", "detail_sub_type")
@classmethod
def strip_text(cls, value: str) -> str:
return value.strip()
@field_validator("task_name", "task_content")
@classmethod
def require_nonblank_text(cls, value: str) -> str:
if not value:
raise ValueError("text must not be blank")
return value
@field_validator("extra")
@classmethod
def require_bounded_json_object(cls, value: dict[str, Any]) -> dict[str, Any]:
try:
encoded = json.dumps(value, ensure_ascii=False, separators=(",", ":"))
except (TypeError, ValueError) as exc:
raise ValueError("extra must be JSON serializable") from exc
if len(encoded.encode("utf-8")) > _MAX_EXTRA_BYTES:
raise ValueError(f"extra must not exceed {_MAX_EXTRA_BYTES} bytes")
return value
class TaskDeeplinkDetailResponse(BaseModel):
model_config = ConfigDict(populate_by_name=True)
detail_type: Literal["b5-1"] = Field(serialization_alias="type")
detail_id: str = Field(serialization_alias="id")
detail_sub_type: str = Field(default="", serialization_alias="taskType")
task_name: str = Field(serialization_alias="taskName")
task_content: str = Field(serialization_alias="taskContent")
overview: str = ""
extra: dict[str, Any] = Field(default_factory=dict)
version: int
updated_at: str | None = Field(default=None, serialization_alias="updatedAt")
class TaskDeeplinkDetailEnvelope(BaseModel):
data: TaskDeeplinkDetailResponse
class TaskDeeplinkDetailUpsertEnvelope(TaskDeeplinkDetailEnvelope):
created: bool
changed: bool
def _detail_response(row: dict[str, Any]) -> TaskDeeplinkDetailResponse:
return TaskDeeplinkDetailResponse(
detail_type=row["detail_type"],
detail_id=row["detail_id"],
detail_sub_type=str(row.get("sub_type") or ""),
task_name=row["task_name"],
task_content=row["task_content"],
overview=row.get("overview") or "",
extra=row.get("extra") or {},
version=int(row.get("version") or 1),
updated_at=str(row["updated_at"]) if row.get("updated_at") else None,
)
def _normalized_detail_id(detail_id: str) -> str:
normalized = detail_id.strip()
if not normalized:
raise HTTPException(status_code=422, detail="detail id must not be blank")
return normalized
def _writer_id(request: Request) -> str:
"""Audit identity for a snapshot write.
The PUT endpoint is credential-free (integration systems write before the
user's browser is involved), so an unauthenticated caller is recorded as
the external integrator; a logged-in browser write keeps the user id.
"""
user = getattr(request.state, "user", None)
if user is not None:
return str(getattr(user, "id", "system"))
return "external:task-deeplink"
@router.get("/{detail_type}/{detail_id}", response_model=TaskDeeplinkDetailEnvelope)
async def get_task_deeplink_detail(
request: Request,
detail_type: Literal["b5-1"],
detail_id: str = Path(min_length=1, max_length=128),
type: str = Query(default="", max_length=_MAX_SUB_TYPE),
) -> TaskDeeplinkDetailEnvelope:
"""Read one persisted snapshot for a post-login deep-link workspace.
``?type=`` selects the business-type partition written via the PUT body's
``type`` field; omitted means the legacy no-type partition.
"""
store = get_task_deeplink_detail_store(request)
row = await store.get_detail(
detail_type,
_normalized_detail_id(detail_id),
sub_type=type.strip(),
)
if row is None:
raise HTTPException(status_code=404, detail="Task deep-link detail not found")
return TaskDeeplinkDetailEnvelope(data=_detail_response(row))
@router.put("/{detail_type}/{detail_id}", response_model=TaskDeeplinkDetailUpsertEnvelope)
async def upsert_task_deeplink_detail(
request: Request,
body: TaskDeeplinkDetailUpsertRequest,
detail_type: Literal["b5-1"],
detail_id: str = Path(min_length=1, max_length=128),
) -> TaskDeeplinkDetailUpsertEnvelope:
"""Idempotently create or update a snapshot before ``goPath=b5-1``.
Open write endpoint: no login session or integration key is required
(AuthMiddleware/CSRF let the PUT through); the writer is only recorded
for audit. Reads still require a normal authenticated session.
"""
store = get_task_deeplink_detail_store(request)
row, created, changed = await store.upsert_detail(
detail_type,
_normalized_detail_id(detail_id),
sub_type=body.detail_sub_type,
task_name=body.task_name,
task_content=body.task_content,
overview=body.overview,
extra=body.extra,
updated_by=_writer_id(request),
)
return TaskDeeplinkDetailUpsertEnvelope(
data=_detail_response(row),
created=created,
changed=changed,
)
offline-backend-20260512/backend/app/gateway/routers/task_deeplink_threads.py
+18 -8
View File
@@ -1,19 +1,18 @@
"""Fast taskId + goPath history lookup for single-agent deep-link workspaces.
Messages and execution state remain canonical LangGraph threads. This API reads
the dedicated ``task_deeplink_threads`` index and joins each hit to its thread
metadata by primary key, avoiding a global metadata JSON scan/filter.
Messages and execution state remain canonical LangGraph threads. This API
reads only the dedicated ``task_deeplink_threads`` cards and never joins or
scans the generic ``threads_meta`` catalogue.
"""
from __future__ import annotations
from typing import Literal
from fastapi import APIRouter, Query, Request
from fastapi import APIRouter, HTTPException, Query, Request
from pydantic import BaseModel
from app.gateway.deps import get_task_deeplink_thread_store
from app.gateway.routers.threads import ThreadResponse
from deerflow.persistence.task_deeplink_threads.base import is_valid_task_deeplink_go_path
from deerflow.utils.time import coerce_iso
router = APIRouter(
@@ -30,13 +29,24 @@ class TaskDeeplinkThreadListResponse(BaseModel):
async def list_task_deeplink_threads(
request: Request,
task_id: str = Query(min_length=1, max_length=128),
go_path: Literal["3qfx", "rwfx", "xdfx"] = Query(),
go_path: str = Query(min_length=1, max_length=32),
limit: int = Query(default=100, ge=1, le=500),
# b5-1 ?type= partition. Provided (even empty) → only that sub-type;
# omitted → every sub-type of the task/goPath partition (legacy callers).
sub_type: str | None = Query(default=None, max_length=64),
) -> TaskDeeplinkThreadListResponse:
"""Return one shared deep-link history partition, newest first."""
normalized_go_path = go_path.strip().lower()
if not is_valid_task_deeplink_go_path(normalized_go_path):
raise HTTPException(status_code=422, detail="Invalid task deep-link goPath")
store = get_task_deeplink_thread_store(request)
rows = await store.list_threads(task_id.strip(), go_path, limit=limit)
rows = await store.list_threads(
task_id.strip(),
normalized_go_path,
limit=limit,
sub_type=sub_type.strip() if sub_type is not None else None,
)
return TaskDeeplinkThreadListResponse(
threads=[
ThreadResponse(
offline-backend-20260512/backend/app/gateway/routers/thread_runs.py
+3 -3
View File
@@ -177,7 +177,7 @@ async def get_run(thread_id: str, run_id: str, request: Request) -> RunResponse:
if record is None or record.thread_id != thread_id:
# In-memory miss: fall back to the persistent store so a run still
# resolves after a Gateway restart instead of 404ing.
record = await run_mgr.get_persisted(run_id)
record = await run_mgr.get_persisted(run_id, authorized_thread_id=thread_id)
if record is None or record.thread_id != thread_id:
raise HTTPException(status_code=404, detail=f"Run {run_id} not found")
return _record_to_response(record)
@@ -242,7 +242,7 @@ async def join_run(thread_id: str, run_id: str, request: Request) -> StreamingRe
# means the Gateway restarted (or the finished run was cleaned up). There
# is nothing to reconnect to, so stream a clean terminal end instead of a
# 404 — the SDK then clears its reconnect key and renders history.
persisted = await run_mgr.get_persisted(run_id)
persisted = await run_mgr.get_persisted(run_id, authorized_thread_id=thread_id)
if persisted is None or persisted.thread_id != thread_id:
raise HTTPException(status_code=404, detail=f"Run {run_id} not found")
@@ -280,7 +280,7 @@ async def stream_existing_run(
# from before a restart (or already finished) — nothing to join or
# cancel, so emit a clean terminal end so the SDK stops reconnecting
# instead of 404ing. A truly unknown run still 404s.
persisted = await run_mgr.get_persisted(run_id)
persisted = await run_mgr.get_persisted(run_id, authorized_thread_id=thread_id)
if persisted is None or persisted.thread_id != thread_id:
raise HTTPException(status_code=404, detail=f"Run {run_id} not found")
return StreamingResponse(
offline-backend-20260512/backend/app/gateway/routers/threads.py
+216 -107
View File
@@ -17,6 +17,7 @@ import json
import logging
import os
import uuid
from contextlib import aclosing
from pathlib import Path
from typing import Any
@@ -26,7 +27,7 @@ from pydantic import BaseModel, Field, field_validator
from sqlalchemy import select
from app.gateway.authz import require_permission
from app.gateway.deps import get_checkpointer
from app.gateway.deps import get_checkpointer, get_thread_store
from app.gateway.llmwiki_deposit import delete_conversation_deposit_documents
from app.gateway.operation_timing import OperationTiming
from app.gateway.utils import sanitize_log_param
@@ -61,7 +62,9 @@ _RAG_DEBUG_LOGGER_NAME = "rag_citation_debug"
# owner identity through the API surface. Defense-in-depth — the
# row-level invariant is still ``threads_meta.user_id`` populated from
# the auth contextvar; this list closes the metadata-blob echo gap.
_SERVER_RESERVED_METADATA_KEYS: frozenset[str] = frozenset({"owner_id", "user_id"})
_SERVER_RESERVED_METADATA_KEYS: frozenset[str] = frozenset(
{"owner_id", "user_id", "consumer_values_save_called_at"}
)
def _strip_reserved_metadata(metadata: dict[str, Any] | None) -> dict[str, Any]:
@@ -177,6 +180,7 @@ class ThreadCreateRequest(BaseModel):
thread_id: str | None = Field(default=None, description="Optional thread ID (auto-generated if omitted)")
assistant_id: str | None = Field(default=None, description="Associate thread with an assistant")
metadata: dict[str, Any] = Field(default_factory=dict, description="Initial metadata")
initialize_checkpoint: bool = Field(default=False, description="Compatibility option: immediately write an empty checkpoint instead of initializing on first run")
_strip_reserved = field_validator("metadata")(classmethod(lambda cls, v: _strip_reserved_metadata(v)))
@@ -204,6 +208,19 @@ class ThreadStateResponse(BaseModel):
tasks: list[dict[str, Any]] = Field(default_factory=list, description="Interrupted task details")
class ThreadTitleStateResponse(BaseModel):
"""Lightweight title refresh status backed only by thread metadata."""
title: str = ""
title_provisional: bool | None = None
class ConsumerValuesSaveClaimResponse(BaseModel):
"""Whether the caller must backfill Consumer values for this thread."""
should_save: bool
class ThreadMessagesResponse(BaseModel):
"""A page of the latest thread message state.
@@ -860,13 +877,13 @@ async def delete_thread_data(thread_id: str, request: Request) -> ThreadDeleteRe
async def create_thread(body: ThreadCreateRequest, request: Request, response: Response = None) -> ThreadResponse: # type: ignore[assignment]
"""Create a new thread.
Writes a thread_meta record (so the thread appears in /threads/search)
and an empty checkpoint (so state endpoints work immediately).
Writes thread metadata immediately; the first graph run initializes state.
Legacy callers can explicitly request an immediate empty checkpoint.
Idempotent: returns the existing record when ``thread_id`` already exists.
"""
from app.gateway.deps import get_thread_store
checkpointer = get_checkpointer(request)
checkpointer = get_checkpointer(request) if body.initialize_checkpoint else None
thread_store = get_thread_store(request)
thread_id = body.thread_id or str(uuid.uuid4())
now = now_iso()
@@ -879,12 +896,13 @@ async def create_thread(body: ThreadCreateRequest, request: Request, response: R
"thread_create",
thread_id=sanitize_log_param(thread_id),
metadata_store=type(thread_store).__name__,
checkpointer=type(checkpointer).__name__,
checkpointer=type(checkpointer).__name__ if checkpointer is not None else "deferred",
) as timing:
# Idempotency: return existing record when already present
with timing.phase("metadata_lookup"):
existing_record = await thread_store.get(thread_id)
if existing_record is not None:
_mark_primary_only_thread(request, thread_id)
return ThreadResponse(
thread_id=thread_id,
status=existing_record.get("status", "idle"),
@@ -905,8 +923,12 @@ async def create_thread(body: ThreadCreateRequest, request: Request, response: R
logger.exception("Failed to write thread_meta for %s", sanitize_log_param(thread_id))
raise HTTPException(status_code=500, detail="Failed to create thread")
# Keep the initial checkpoint: state endpoints must work immediately.
# Its timing includes saver lock/pool waits as well as the DB write.
if not body.initialize_checkpoint:
_mark_primary_only_thread(request, thread_id)
return ThreadResponse(thread_id=thread_id, created_at=now, updated_at=now, metadata=body.metadata)
# Explicit compatibility path only. Never schedule this write in the
# background: a late empty checkpoint could replace the first run's head.
config = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
try:
ckpt_metadata = {
@@ -1068,23 +1090,67 @@ async def patch_thread(thread_id: str, body: ThreadPatchRequest, request: Reques
)
@router.post("/{thread_id}/consumer-values-save-claim", response_model=ConsumerValuesSaveClaimResponse)
@require_permission("threads", "write", owner_check=True, require_existing=True)
async def claim_consumer_values_save(
thread_id: str,
request: Request,
) -> ConsumerValuesSaveClaimResponse:
"""Claim the one-time Consumer backfill for a pre-existing conversation.
The timestamp records that the external save request was *attempted*, not
that Consumer accepted it. This matches the non-blocking UI contract: a
Consumer outage must never be surfaced to or block the chat user.
"""
thread_store = get_thread_store(request)
record = await thread_store.get(thread_id)
if record is None:
raise HTTPException(status_code=404, detail=f"Thread {thread_id} not found")
metadata = record.get("metadata") or {}
if metadata.get("consumer_values_save_called_at"):
return ConsumerValuesSaveClaimResponse(should_save=False)
try:
await thread_store.update_metadata(
thread_id,
{"consumer_values_save_called_at": now_iso()},
)
except Exception:
logger.exception("Failed to claim Consumer values save for thread %s", sanitize_log_param(thread_id))
raise HTTPException(status_code=500, detail="Failed to record Consumer values save")
return ConsumerValuesSaveClaimResponse(should_save=True)
@router.get("/{thread_id}", response_model=ThreadResponse)
@require_permission("threads", "read", owner_check=True)
async def get_thread(thread_id: str, request: Request) -> ThreadResponse:
async def get_thread(thread_id: str, request: Request, include_state: bool = False) -> ThreadResponse:
"""Get thread info.
Reads metadata from the ThreadMetaStore and derives the accurate
execution status from the checkpointer. Falls back to the checkpointer
alone for threads that pre-date ThreadMetaStore adoption (backward compat).
Metadata reads use only the business database. Clients that need channel
values can opt in with include_state, or use /state. Legacy threads without
a metadata row retain the checkpoint fallback after authorization.
"""
from app.gateway.deps import get_thread_store
thread_store = get_thread_store(request)
checkpointer = get_checkpointer(request)
record: dict | None = await thread_store.get(thread_id)
# The owner-check decorator already authorized this read, including shared
# task threads. A metadata-only shared thread has no checkpoint fallback.
record: dict | None = await thread_store.get(thread_id, user_id=None)
if record is not None and not include_state:
return ThreadResponse(
thread_id=thread_id,
status="busy" if record.get("status") == "running" else record.get("status", "idle"),
created_at=coerce_iso(record.get("created_at", "")),
updated_at=coerce_iso(record.get("updated_at", "")),
metadata=record.get("metadata", {}),
)
# Derive accurate status from the checkpointer
checkpointer = get_checkpointer(request)
config = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
try:
checkpoint_tuple = await checkpointer.aget_tuple(config)
@@ -1121,61 +1187,91 @@ async def get_thread(thread_id: str, request: Request) -> ThreadResponse:
created_at=coerce_iso(record.get("created_at", "")),
updated_at=coerce_iso(record.get("updated_at", "")),
metadata=record.get("metadata", {}),
values=serialize_channel_values(channel_values),
values=serialize_channel_values(channel_values, slim_display=True),
)
# ---------------------------------------------------------------------------
@router.get("/{thread_id}/title-state", response_model=ThreadTitleStateResponse)
@require_permission("threads", "read", owner_check=True)
async def get_thread_title_state(thread_id: str, request: Request) -> ThreadTitleStateResponse:
"""Read only title metadata; do not load/serialize a checkpoint or messages."""
record = await get_thread_store(request).get(thread_id)
if record is None:
raise HTTPException(status_code=404, detail=f"Thread {thread_id} not found")
metadata = record.get("metadata") or {}
provisional = metadata.get("_deerflow_title_provisional")
return ThreadTitleStateResponse(
title=str(record.get("display_name") or ""),
title_provisional=provisional if isinstance(provisional, bool) else None,
)
@router.get("/{thread_id}/state", response_model=ThreadStateResponse)
@require_permission("threads", "read", owner_check=True)
async def get_thread_state(thread_id: str, request: Request) -> ThreadStateResponse:
async def get_thread_state(thread_id: str, request: Request, response: Response = None) -> ThreadStateResponse: # type: ignore[assignment]
"""Get the latest state snapshot for a thread.
Channel values are serialized to ensure LangChain message objects
are converted to JSON-safe dicts.
are converted to JSON-safe dicts. Oversized tool/file bodies are
truncated for this hydrate snapshot; the checkpoint itself is unchanged.
"""
checkpointer = get_checkpointer(request)
config = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
try:
checkpoint_tuple = await checkpointer.aget_tuple(config)
except Exception:
logger.exception("Failed to get state for thread %s", sanitize_log_param(thread_id))
raise HTTPException(status_code=500, detail="Failed to get thread state")
with OperationTiming(
response,
logger,
"thread_state",
thread_id=sanitize_log_param(thread_id),
checkpointer=type(checkpointer).__name__,
) as timing:
try:
with timing.phase("checkpoint_load"):
checkpoint_tuple = await checkpointer.aget_tuple(config)
except Exception:
logger.exception("Failed to get state for thread %s", sanitize_log_param(thread_id))
raise HTTPException(status_code=500, detail="Failed to get thread state")
if checkpoint_tuple is None:
raise HTTPException(status_code=404, detail=f"Thread {thread_id} not found")
if checkpoint_tuple is None:
from app.gateway.deps import get_thread_store
checkpoint = getattr(checkpoint_tuple, "checkpoint", {}) or {}
metadata = getattr(checkpoint_tuple, "metadata", {}) or {}
checkpoint_id = None
ckpt_config = getattr(checkpoint_tuple, "config", {})
if ckpt_config:
checkpoint_id = ckpt_config.get("configurable", {}).get("checkpoint_id")
record = await get_thread_store(request).get(thread_id, user_id=None)
if record is not None:
return ThreadStateResponse(metadata=record.get("metadata", {}))
raise HTTPException(status_code=404, detail=f"Thread {thread_id} not found")
channel_values = checkpoint.get("channel_values", {})
checkpoint = getattr(checkpoint_tuple, "checkpoint", {}) or {}
metadata = getattr(checkpoint_tuple, "metadata", {}) or {}
checkpoint_id = None
ckpt_config = getattr(checkpoint_tuple, "config", {})
if ckpt_config:
checkpoint_id = ckpt_config.get("configurable", {}).get("checkpoint_id")
parent_config = getattr(checkpoint_tuple, "parent_config", None)
parent_checkpoint_id = None
if parent_config:
parent_checkpoint_id = parent_config.get("configurable", {}).get("checkpoint_id")
channel_values = checkpoint.get("channel_values", {})
tasks_raw = getattr(checkpoint_tuple, "tasks", []) or []
next_tasks = [t.name for t in tasks_raw if hasattr(t, "name")]
tasks = [{"id": getattr(t, "id", ""), "name": getattr(t, "name", "")} for t in tasks_raw]
parent_config = getattr(checkpoint_tuple, "parent_config", None)
parent_checkpoint_id = None
if parent_config:
parent_checkpoint_id = parent_config.get("configurable", {}).get("checkpoint_id")
values = serialize_channel_values(channel_values)
tasks_raw = getattr(checkpoint_tuple, "tasks", []) or []
next_tasks = [t.name for t in tasks_raw if hasattr(t, "name")]
tasks = [{"id": getattr(t, "id", ""), "name": getattr(t, "name", "")} for t in tasks_raw]
return ThreadStateResponse(
values=values,
next=next_tasks,
metadata=metadata,
checkpoint={"id": checkpoint_id, "ts": coerce_iso(metadata.get("created_at", ""))},
checkpoint_id=checkpoint_id,
parent_checkpoint_id=parent_checkpoint_id,
created_at=coerce_iso(metadata.get("created_at", "")),
tasks=tasks,
)
with timing.phase("serialize"):
values = serialize_channel_values(channel_values, slim_display=True)
return ThreadStateResponse(
values=values,
next=next_tasks,
metadata=metadata,
checkpoint={"id": checkpoint_id, "ts": coerce_iso(metadata.get("created_at", ""))},
checkpoint_id=checkpoint_id,
parent_checkpoint_id=parent_checkpoint_id,
created_at=coerce_iso(metadata.get("created_at", "")),
tasks=tasks,
)
@router.get("/{thread_id}/reference-batches", response_model=ReferenceBatchesResponse)
@@ -1912,9 +2008,50 @@ async def edit_thread_message(
}
def _mark_primary_only_thread(request: Request, thread_id: str) -> None:
"""Tell a dual-read checkpointer not to scan legacy SQLite for this thread."""
saver = getattr(request.app.state, "checkpointer", None)
remember = getattr(saver, "remember_primary_thread", None)
if callable(remember):
remember(thread_id)
def _history_entry_from_tuple(checkpoint_tuple: Any, *, include_values: bool) -> HistoryEntry:
"""Build one history card. Only the latest head carries serialized values."""
ckpt_config = getattr(checkpoint_tuple, "config", {})
parent_config = getattr(checkpoint_tuple, "parent_config", None)
metadata = getattr(checkpoint_tuple, "metadata", {}) or {}
checkpoint = getattr(checkpoint_tuple, "checkpoint", {}) or {}
checkpoint_id = ckpt_config.get("configurable", {}).get("checkpoint_id", "")
parent_id = parent_config.get("configurable", {}).get("checkpoint_id") if parent_config else None
channel_values = checkpoint.get("channel_values", {})
values: dict[str, Any] = serialize_channel_values(channel_values, slim_display=True) if include_values else {}
if title := channel_values.get("title"):
values["title"] = title
if thread_data := channel_values.get("thread_data"):
values["thread_data"] = thread_data
tasks_raw = getattr(checkpoint_tuple, "tasks", []) or []
user_meta = {k: v for k, v in metadata.items() if k not in ("created_at", "updated_at", "step", "source", "writes", "parents")}
if "step" in metadata:
user_meta["step"] = metadata["step"]
return HistoryEntry(
checkpoint_id=checkpoint_id,
parent_checkpoint_id=parent_id,
metadata=user_meta,
values=values,
created_at=coerce_iso(metadata.get("created_at", "")),
next=[t.name for t in tasks_raw if hasattr(t, "name")],
)
@router.post("/{thread_id}/history", response_model=list[HistoryEntry])
@require_permission("threads", "read", owner_check=True)
async def get_thread_history(thread_id: str, body: ThreadHistoryRequest, request: Request) -> list[HistoryEntry]:
async def get_thread_history(
thread_id: str,
body: ThreadHistoryRequest,
request: Request,
response: Response = None, # type: ignore[assignment]
) -> list[HistoryEntry]:
"""Get checkpoint history for a thread.
Messages are read from the checkpointer's channel values (the
@@ -1922,65 +2059,37 @@ async def get_thread_history(thread_id: str, body: ThreadHistoryRequest, request
:func:`~deerflow.runtime.serialization.serialize_channel_values`.
Only the latest (first) checkpoint carries the ``messages`` key to
avoid duplicating them across every entry.
The chat UI always asks for ``limit=1``. That path uses ``aget_tuple``
instead of walking the checkpoint iterator.
"""
checkpointer = get_checkpointer(request)
config: dict[str, Any] = {"configurable": {"thread_id": thread_id}}
config: dict[str, Any] = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
if body.before:
config["configurable"]["checkpoint_id"] = body.before
entries: list[HistoryEntry] = []
is_latest_checkpoint = True
try:
async for checkpoint_tuple in checkpointer.alist(config, limit=body.limit):
ckpt_config = getattr(checkpoint_tuple, "config", {})
parent_config = getattr(checkpoint_tuple, "parent_config", None)
metadata = getattr(checkpoint_tuple, "metadata", {}) or {}
checkpoint = getattr(checkpoint_tuple, "checkpoint", {}) or {}
checkpoint_id = ckpt_config.get("configurable", {}).get("checkpoint_id", "")
parent_id = None
if parent_config:
parent_id = parent_config.get("configurable", {}).get("checkpoint_id")
channel_values = checkpoint.get("channel_values", {})
# Build values from checkpoint channel_values
values: dict[str, Any] = {}
if title := channel_values.get("title"):
values["title"] = title
if thread_data := channel_values.get("thread_data"):
values["thread_data"] = thread_data
# Attach messages only to the latest checkpoint entry.
if is_latest_checkpoint:
messages = channel_values.get("messages")
if messages:
values["messages"] = serialize_channel_values({"messages": messages}).get("messages", [])
is_latest_checkpoint = False
# Derive next tasks
tasks_raw = getattr(checkpoint_tuple, "tasks", []) or []
next_tasks = [t.name for t in tasks_raw if hasattr(t, "name")]
# Strip LangGraph internal keys from metadata
user_meta = {k: v for k, v in metadata.items() if k not in ("created_at", "updated_at", "step", "source", "writes", "parents")}
# Keep step for ordering context
if "step" in metadata:
user_meta["step"] = metadata["step"]
entries.append(
HistoryEntry(
checkpoint_id=checkpoint_id,
parent_checkpoint_id=parent_id,
metadata=user_meta,
values=values,
created_at=coerce_iso(metadata.get("created_at", "")),
next=next_tasks,
)
)
except Exception:
logger.exception("Failed to get history for thread %s", sanitize_log_param(thread_id))
raise HTTPException(status_code=500, detail="Failed to get thread history")
return entries
with OperationTiming(
response,
logger,
"thread_history",
thread_id=sanitize_log_param(thread_id),
checkpointer=type(checkpointer).__name__,
limit=str(body.limit),
) as timing:
try:
with timing.phase("checkpoint_load"):
if body.limit == 1 and not body.before:
latest = await checkpointer.aget_tuple(config)
checkpoints = [latest] if latest is not None else []
else:
async with aclosing(checkpointer.alist(config, limit=body.limit)) as history:
checkpoints = [checkpoint async for checkpoint in history]
with timing.phase("serialize"):
return [
_history_entry_from_tuple(checkpoint_tuple, include_values=index == 0)
for index, checkpoint_tuple in enumerate(checkpoints)
]
except Exception:
logger.exception("Failed to get history for thread %s", sanitize_log_param(thread_id))
raise HTTPException(status_code=500, detail="Failed to get thread history")
offline-backend-20260512/backend/app/gateway/routers/tool_metrics.py
+15 -9
View File
@@ -19,6 +19,7 @@ from fastapi import APIRouter, HTTPException, Query, Request
from fastapi.responses import StreamingResponse
from pydantic import BaseModel, Field
from app.gateway.admin_query_window import resolve_admin_query_window
from app.gateway.deps import get_local_provider, get_optional_user_from_request, get_tool_metrics_store
from deerflow.persistence.tool_metrics.base import ToolMetricsQuery, ToolMetricsStore
@@ -85,9 +86,11 @@ def _require_store(request: Request) -> ToolMetricsStore:
return store
def _parse_dt(value: str | None) -> datetime | None:
def _parse_dt(value: str | datetime | None) -> datetime | None:
if not value:
return None
if isinstance(value, datetime):
return value
try:
return datetime.fromisoformat(value.replace("Z", "+00:00"))
except ValueError as exc:
@@ -147,8 +150,8 @@ def _build_query(
skill_name: str | None = None,
status: str | None,
error_category: str | None,
since: str | None,
until: str | None,
since: str | datetime | None,
until: str | datetime | None,
limit: int,
offset: int,
) -> ToolMetricsQuery:
@@ -190,6 +193,7 @@ async def list_tool_metrics(
if resolved is None:
return ToolMetricsListResponse(items=[], total=0, limit=limit, offset=offset)
user_id = resolved
since_dt, until_dt, _range_days = resolve_admin_query_window(since=since, until=until)
query = _build_query(
user_id=user_id,
tool_name=tool_name,
@@ -197,8 +201,8 @@ async def list_tool_metrics(
skill_name=skill_name,
status=status,
error_category=error_category,
since=since,
until=until,
since=since_dt,
until=until_dt,
limit=limit,
offset=offset,
)
@@ -250,6 +254,7 @@ async def tool_metrics_summary(
return ToolMetricsSummaryResponse(total=0, error_total=0, by_category={}, by_skill=[])
user_id = resolved
skill_only = _kind_is_skill(kind)
since_dt, until_dt, _range_days = resolve_admin_query_window(since=since, until=until)
def _q(*, status: str | None = None, error_category: str | None = None) -> ToolMetricsQuery:
return _build_query(
@@ -259,8 +264,8 @@ async def tool_metrics_summary(
skill_name=skill_name,
status=status,
error_category=error_category,
since=since,
until=until,
since=since_dt,
until=until_dt,
limit=1,
offset=0,
)
@@ -305,6 +310,7 @@ async def export_tool_metrics(
if not user_id and user_name:
resolved = await _resolve_user_name_to_id(user_name)
user_id = resolved if resolved is not None else "__no_such_user__"
since_dt, until_dt, _range_days = resolve_admin_query_window(since=since, until=until)
try:
from openpyxl import Workbook
@@ -318,8 +324,8 @@ async def export_tool_metrics(
skill_name=skill_name,
status=status,
error_category=error_category,
since=since,
until=until,
since=since_dt,
until=until_dt,
limit=1,
offset=0,
)
offline-backend-20260512/backend/app/gateway/routers/uploads.py
+120 -11
View File
@@ -1,11 +1,14 @@
"""Upload router for handling file uploads."""
import asyncio
import logging
import os
import shutil
import stat
import tempfile
import threading
from pathlib import Path
from weakref import WeakKeyDictionary, WeakValueDictionary
from fastapi import APIRouter, Depends, File, HTTPException, Request, UploadFile
from pydantic import BaseModel, Field
@@ -33,13 +36,46 @@ from deerflow.uploads.manager import (
)
from deerflow.uploads.policy import classify_upload_filename
from deerflow.uploads.validation import UploadContentError, validate_upload_content
from deerflow.utils.file_conversion import CONVERTIBLE_EXTENSIONS, convert_file_to_markdown
from deerflow.utils.file_conversion import CONVERTIBLE_EXTENSIONS, convert_file_to_markdown_sync
from deerflow.utils.file_io import FileIOPool, file_io
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/threads/{thread_id}/uploads", tags=["uploads"])
UPLOAD_CHUNK_SIZE = 8192
UPLOAD_CHUNK_SIZE = 256 * 1024
# Upload work must not fill the default executor used by authentication, or the
# artifact/event pools used by active conversations. Waiting batches use no worker.
_upload_io = FileIOPool(2, 2, "uploads")
_upload_locks: WeakKeyDictionary = WeakKeyDictionary()
def _upload_lock(directory: Path) -> asyncio.Lock:
locks = _upload_locks.setdefault(asyncio.get_running_loop(), WeakValueDictionary())
key = os.path.normcase(str(directory))
lock = locks.get(key)
if lock is None:
lock = asyncio.Lock()
locks[key] = lock
return lock
class _UploadCancelled(Exception):
"""Cooperative stop inside a worker; the request propagates CancelledError."""
def _check_upload_cancelled(cancelled: threading.Event | None) -> None:
if cancelled is not None and cancelled.is_set():
raise _UploadCancelled()
async def _resolve_uploads_path(thread_id: str) -> Path:
owner = await aresolve_path_user_id(thread_id)
try:
return get_paths().sandbox_uploads_dir(thread_id, user_id=owner)
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
class UploadResponse(BaseModel):
@@ -119,10 +155,37 @@ async def _write_upload_file_with_limits(
max_total_size: int,
total_size: int,
) -> tuple[os.PathLike[str] | str, int, int]:
cancelled = threading.Event()
return await _upload_io.run_until_complete(
_copy_upload_file_with_limits,
file,
uploads_dir=uploads_dir,
display_filename=display_filename,
max_single_file_size=max_single_file_size,
max_total_size=max_total_size,
total_size=total_size,
cancelled=cancelled,
on_cancel=cancelled.set,
)
def _copy_upload_file_with_limits(
file: UploadFile,
*,
uploads_dir: os.PathLike[str] | str,
display_filename: str,
max_single_file_size: int,
max_total_size: int,
total_size: int,
cancelled: threading.Event | None = None,
) -> tuple[Path, int, int]:
"""Copy the parsed multipart spool entirely in the dedicated upload worker."""
_check_upload_cancelled(cancelled)
file_size = 0
file_path, fh = open_upload_file_no_symlink(uploads_dir, display_filename)
try:
while chunk := await file.read(UPLOAD_CHUNK_SIZE):
while chunk := file.file.read(UPLOAD_CHUNK_SIZE):
_check_upload_cancelled(cancelled)
file_size += len(chunk)
total_size += len(chunk)
if file_size > max_single_file_size:
@@ -130,6 +193,7 @@ async def _write_upload_file_with_limits(
if total_size > max_total_size:
raise HTTPException(status_code=413, detail="Total upload size too large")
fh.write(chunk)
_check_upload_cancelled(cancelled)
except Exception:
fh.close()
try:
@@ -249,27 +313,56 @@ async def upload_files(
category, max_bytes = classification
prepared.append((file, safe_filename, category, max_bytes))
# Resolve the canonical owner asynchronously before any filesystem work.
sandbox_uploads = await _resolve_uploads_path(thread_id)
cancelled = threading.Event()
# Moving work off the event loop introduces real concurrency. Serialize
# batches for the same directory so rollback cannot undo another upload.
async with _upload_lock(sandbox_uploads):
return await _upload_io.run_until_complete(
_process_upload_batch,
thread_id,
sandbox_uploads,
prepared,
claimed_names,
limits,
_auto_convert_documents_enabled(config),
cancelled,
on_cancel=cancelled.set,
)
def _process_upload_batch(
thread_id: str,
sandbox_uploads: Path,
prepared: list[tuple[UploadFile, str, str, int]],
claimed_names: set[str],
limits: UploadLimits,
auto_convert_documents: bool,
cancelled: threading.Event,
) -> UploadResponse:
"""Own the full staging/commit/rollback lifetime in one bounded worker."""
_check_upload_cancelled(cancelled)
try:
uploads_dir = ensure_uploads_dir(thread_id)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
sandbox_uploads = get_paths().sandbox_uploads_dir(thread_id, user_id=await aresolve_path_user_id(thread_id))
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
uploaded_files: list[dict[str, str]] = []
sandbox_sync_targets: list[tuple[Path, str]] = []
total_size = 0
auto_convert_documents = _auto_convert_documents_enabled(config)
staging_dir = Path(tempfile.mkdtemp(prefix=".upload-staging-", dir=uploads_dir))
records: list[dict] = []
committed_names: list[str] | None = None
try:
for file, safe_filename, category, max_bytes in prepared:
staged_path, file_size, total_size = await _write_upload_file_with_limits(
staged_path, file_size, total_size = _copy_upload_file_with_limits(
file,
uploads_dir=staging_dir,
display_filename=safe_filename,
max_single_file_size=max_bytes,
max_total_size=limits.max_total_size,
total_size=total_size,
cancelled=cancelled,
)
file_ext = staged_path.suffix.lower()
try:
@@ -284,7 +377,8 @@ async def upload_files(
"detected_type": file_ext.removeprefix("."),
}
if auto_convert_documents and file_ext in CONVERTIBLE_EXTENSIONS:
md_path = await convert_file_to_markdown(staged_path)
_check_upload_cancelled(cancelled)
md_path = convert_file_to_markdown_sync(staged_path)
if md_path:
markdown_folded = md_path.name.casefold()
if markdown_folded in claimed_names:
@@ -296,6 +390,7 @@ async def upload_files(
record["markdown_file"] = md_path.name
records.append(record)
_check_upload_cancelled(cancelled)
sandbox_provider = get_sandbox_provider()
sync_to_sandbox = not _uses_thread_data_mounts(sandbox_provider)
sandbox = None
@@ -307,6 +402,7 @@ async def upload_files(
commit_names = [str(record["filename"]) for record in records]
commit_names.extend(str(record["markdown_file"]) for record in records if record.get("markdown_file"))
_check_upload_cancelled(cancelled)
_commit_staged_files(staging_dir, uploads_dir, commit_names)
committed_names = commit_names
@@ -345,9 +441,11 @@ async def upload_files(
if sync_to_sandbox and sandbox is not None:
for file_path, virtual_path in sandbox_sync_targets:
_check_upload_cancelled(cancelled)
_make_file_sandbox_writable(file_path)
sandbox.update_file(virtual_path, file_path.read_bytes())
except HTTPException:
_check_upload_cancelled(cancelled)
except (HTTPException, _UploadCancelled):
if committed_names is not None:
_rollback_committed_files(staging_dir, uploads_dir, committed_names)
raise
@@ -390,6 +488,12 @@ async def get_upload_limits(
@require_permission("threads", "read", owner_check=True)
async def list_uploaded_files(thread_id: str, request: Request) -> dict:
"""List all files in a thread's uploads directory."""
sandbox_uploads = await _resolve_uploads_path(thread_id)
async with _upload_lock(sandbox_uploads):
return await file_io.run_until_complete(_list_uploaded_files, thread_id, sandbox_uploads)
def _list_uploaded_files(thread_id: str, sandbox_uploads: Path) -> dict:
try:
uploads_dir = get_uploads_dir(thread_id)
except ValueError as e:
@@ -398,7 +502,6 @@ async def list_uploaded_files(thread_id: str, request: Request) -> dict:
enrich_file_listing(result, thread_id)
# Gateway additionally includes the sandbox-relative path.
sandbox_uploads = get_paths().sandbox_uploads_dir(thread_id, user_id=await aresolve_path_user_id(thread_id))
for f in result["files"]:
f["path"] = str(sandbox_uploads / f["filename"])
@@ -409,6 +512,12 @@ async def list_uploaded_files(thread_id: str, request: Request) -> dict:
@require_permission("threads", "delete", owner_check=True, require_existing=True)
async def delete_uploaded_file(thread_id: str, filename: str, request: Request) -> dict:
"""Delete a file from a thread's uploads directory."""
sandbox_uploads = await _resolve_uploads_path(thread_id)
async with _upload_lock(sandbox_uploads):
return await file_io.run_until_complete(_delete_uploaded_file, thread_id, filename)
def _delete_uploaded_file(thread_id: str, filename: str) -> dict:
try:
uploads_dir = get_uploads_dir(thread_id)
except ValueError as e:
offline-backend-20260512/backend/app/gateway/routers/user_preferences.py
+11 -2
View File
@@ -13,6 +13,7 @@ import logging
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel, Field
from app.gateway.api_read_cache import get_cache_version, get_cached_json, set_cached_json
from deerflow.runtime.user_context import get_effective_user_id
logger = logging.getLogger(__name__)
@@ -52,10 +53,18 @@ def _to_response(prefs: dict) -> UserPreferencesResponse:
@router.get("", response_model=UserPreferencesResponse)
async def get_user_preferences(request: Request) -> UserPreferencesResponse:
store = _get_store(request)
user_id = _current_user_id(request)
config = getattr(request.app.state, "config", None)
cache_key = f"user:{user_id}"
version = await get_cache_version(config, "user_preferences")
cached = await get_cached_json(config, "user_preferences", cache_key, version=version)
if cached is not None:
return UserPreferencesResponse.model_validate(cached)
store = _get_store(request)
prefs = await store.get_for_user(user_id)
return _to_response(prefs)
response = _to_response(prefs)
await set_cached_json(config, "user_preferences", cache_key, response, expected_version=version)
return response
@router.put("", response_model=UserPreferencesResponse)
offline-backend-20260512/backend/app/gateway/routers/user_prompts.py
+11 -2
View File
@@ -13,6 +13,7 @@ from uuid import uuid4
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel, Field
from app.gateway.api_read_cache import get_cache_version, get_cached_json, set_cached_json
from deerflow.runtime.user_context import get_effective_user_id
logger = logging.getLogger(__name__)
@@ -66,10 +67,18 @@ def _get_store(request: Request):
@router.get("", response_model=UserPromptsListResponse)
async def list_user_prompts(request: Request) -> UserPromptsListResponse:
store = _get_store(request)
user_id = _current_user_id(request)
config = getattr(request.app.state, "config", None)
cache_key = f"user:{user_id}"
version = await get_cache_version(config, "user_prompts")
cached = await get_cached_json(config, "user_prompts", cache_key, version=version)
if cached is not None:
return UserPromptsListResponse.model_validate(cached)
store = _get_store(request)
rows = await store.list_for_user(user_id)
return UserPromptsListResponse(prompts=[UserPromptResponse(**r) for r in rows])
response = UserPromptsListResponse(prompts=[UserPromptResponse(**r) for r in rows])
await set_cached_json(config, "user_prompts", cache_key, response, expected_version=version)
return response
@router.post("", response_model=UserPromptResponse, status_code=201)
offline-backend-20260512/backend/app/gateway/services.py
+66 -16
View File
@@ -30,6 +30,7 @@ from app.gateway.llmwiki_rag import (
llmwiki_selection_requested,
)
from app.gateway.utils import sanitize_log_param
from app.gateway.worker_pool.admission import bounded_worker_run
from deerflow.agents.middlewares.prompt_prefix_middleware import (
SMART_KNOWLEDGE_SCOPE_PROMPT_KEY,
)
@@ -134,7 +135,9 @@ async def _run_agent_with_concurrency_lease(
)
try:
await run_agent(**run_agent_kwargs)
from app.gateway.worker_pool.admission import run_with_worker_admission
await run_with_worker_admission(run_agent, **run_agent_kwargs)
record = run_agent_kwargs.get("record")
if after_success is not None and getattr(record, "status", None) == RunStatus.success:
try:
@@ -555,6 +558,9 @@ _CONTEXT_CONFIGURABLE_KEYS: frozenset[str] = frozenset(
"writing_setup_quick",
"writing_sample_context",
"excluded_tools",
# Internal role policies may opt into a fail-closed tool allowlist.
# Ordinary clients omit it and retain the existing tool resolution.
"allowed_tools",
"skill_stop_names",
"skill_stop_message",
# 记忆开关:必须在白名单里才会从 body.context 透传到 run config,
@@ -571,6 +577,8 @@ _CONTEXT_CONFIGURABLE_KEYS: frozenset[str] = frozenset(
"roundtable_peer_deliveries",
# 岗位会商席位 / 方案总结：每轮 outputs 下 write_file 上限（行动规划不传此键）。
"position_artifact_cap",
# 会商专用文件流式兼容。只由会商 run 写入；普通对话不带此键，仍只看全局开关。
"roundtable_write_file_compatibility",
# 上下文压缩开关:由聊天框下方的按钮控制(默认关闭),透传到 run config 后
# 在 _build_middlewares 里决定是否挂载 SummarizationMiddleware。
"summarization_enabled",
@@ -871,6 +879,7 @@ def enforce_reasoning_effort_run_config(
*,
capability_enabled: bool,
model_supports_thinking: bool,
default_reasoning_effort: str = "xhigh",
) -> str | None:
"""Apply the database-backed reasoning policy to both runtime containers.
@@ -889,7 +898,10 @@ def enforce_reasoning_effort_run_config(
thinking_enabled = bool(effective.get("thinking_enabled", True))
if capability_enabled and model_supports_thinking and thinking_enabled:
normalized = _normalize_reasoning_effort(effective.get("reasoning_effort"))
# The selected model's persisted default is authoritative. A per-thread
# value from an older browser must not override the administrator's
# globally shared model policy.
normalized = _normalize_reasoning_effort(default_reasoning_effort)
else:
normalized = None
@@ -903,12 +915,12 @@ def enforce_reasoning_effort_run_config(
return normalized
async def _reasoning_effort_capability(request: Request, model_name: str) -> bool:
"""Read one model capability, failing closed without breaking chat."""
async def _reasoning_effort_policy(request: Request, model_name: str) -> tuple[bool, str]:
"""Read one model capability and its shared default, failing closed."""
store = getattr(request.app.state, "model_capability_store", None)
if store is None:
return False
return False, "xhigh"
try:
row = await store.get_capability(model_name)
except Exception:
@@ -917,8 +929,13 @@ async def _reasoning_effort_capability(request: Request, model_name: str) -> boo
sanitize_log_param(model_name),
exc_info=True,
)
return False
return bool(row and row.get("supports_reasoning_effort", False))
return False, "xhigh"
if not row:
return False, "xhigh"
default = str(row.get("default_reasoning_effort") or "xhigh")
if default not in {"low", "medium", "xhigh"}:
default = "xhigh"
return bool(row.get("supports_reasoning_effort", False)), default
def inject_validated_analysis_project_context(config: dict[str, Any], context: Mapping[str, Any] | None) -> None:
@@ -1062,6 +1079,12 @@ def build_run_config(
target["agent_id"] = normalized
if metadata:
config.setdefault("metadata", {}).update(metadata)
# Chat clients cannot override the administrator's deployment-wide budget.
# Other built-in graphs (writing/canvas) retain their own execution policy.
if assistant_id not in _BUILTIN_GRAPH_IDS or assistant_id == "lead_agent":
from deerflow.config.system_settings import get_conversation_context_settings
config["recursion_limit"] = get_conversation_context_settings().max_run_steps
return config
@@ -1407,6 +1430,7 @@ async def _fail_unstarted_run(
logger.warning("Failed to release concurrency lease for unstarted run %s", record.run_id, exc_info=True)
@bounded_worker_run
async def start_run(
body: Any,
thread_id: str,
@@ -1439,8 +1463,17 @@ async def start_run(
"""
t_entry = time.perf_counter()
from app.gateway.analysis_projects import prepare_project_run
from app.gateway.knowledge_research import (
inject_validated_knowledge_research_context,
prepare_knowledge_research_run,
)
body, analysis_project = await prepare_project_run(body, thread_id, request)
body, knowledge_research = await prepare_knowledge_research_run(
body,
thread_id,
request,
)
bridge = get_stream_bridge(request)
run_mgr = get_run_manager(request)
run_ctx = get_run_context(request)
@@ -1472,7 +1505,7 @@ async def start_run(
)
fixed_answer = (
None
if analysis_project or requested_retrieval_mode == "smart_reasoning"
if analysis_project or knowledge_research is not None or requested_retrieval_mode == "smart_reasoning"
else await _find_fixed_answer(
body=body,
request=request,
@@ -1825,13 +1858,20 @@ async def start_run(
value = getattr(body, top_level_key, None)
if value is not None and top_level_key not in body_context:
body_context[top_level_key] = value
strict_scope_active = apply_knowledge_space_strict_scope(
body_context,
selection_requested=llmwiki_selection_requested(body),
knowledge_base_ids=llmwiki_rag.knowledge_base_ids,
assistant_knowledge_base_ids=llmwiki_rag.assistant_knowledge_base_ids,
app_config=getattr(request.app.state, "config", None),
)
if knowledge_retrieval_mode == "smart_reasoning":
# Smart retrieval is an optional tool surface in the normal agent. It
# retains the validated KB ids but deliberately does not activate the
# classic fail-closed tool/skill/MCP isolation policy.
body_context["knowledge_space_strict"] = False
strict_scope_active = False
else:
strict_scope_active = apply_knowledge_space_strict_scope(
body_context,
selection_requested=llmwiki_selection_requested(body),
knowledge_base_ids=llmwiki_rag.knowledge_base_ids,
assistant_knowledge_base_ids=llmwiki_rag.assistant_knowledge_base_ids,
app_config=getattr(request.app.state, "config", None),
)
if strict_scope_active:
run_metadata["knowledge_space_strict"] = True
elif not llmwiki_rag.knowledge_base_ids:
@@ -1888,7 +1928,7 @@ async def start_run(
effective_model_config
and getattr(effective_model_config, "supports_thinking", False)
)
reasoning_effort_capability = await _reasoning_effort_capability(
reasoning_effort_capability, default_reasoning_effort = await _reasoning_effort_policy(
request,
final_model_name,
)
@@ -1897,6 +1937,7 @@ async def start_run(
config,
capability_enabled=reasoning_effort_capability,
model_supports_thinking=effective_model_supports_thinking,
default_reasoning_effort=default_reasoning_effort,
)
except Exception as exc:
await _fail_unstarted_run(
@@ -1935,6 +1976,8 @@ async def start_run(
# These keys are deliberately absent from the public context whitelist.
# Only the validated project binding/checkpoint can supply them.
inject_validated_analysis_project_context(config, body.context)
if knowledge_research:
inject_validated_knowledge_research_context(config, body.context)
stream_modes = _with_knowledge_retrieval_progress_stream(
normalize_stream_modes(body.stream_mode),
@@ -1995,6 +2038,7 @@ async def start_run(
)
raise
record.task = task
run_mgr.watch_completion(record)
# Title sync is handled by worker.py's finally block which reads the
# title from the checkpoint and calls thread_store.update_display_name
@@ -2049,6 +2093,12 @@ async def sse_consumer(
A truly gone client fails on the next ``yield`` instead.
"""
last_event_id = request.headers.get("Last-Event-ID")
# Completed records outlive the short SSE replay buffer. Rejoining after
# buffer retirement must end cleanly, not create an empty endless stream.
has_stream = getattr(bridge, "has_stream", None)
if record.status not in (RunStatus.pending, RunStatus.running) and callable(has_stream) and not has_stream(record.run_id):
yield format_sse("end", None)
return
try:
async for entry in bridge.subscribe(record.run_id, last_event_id=last_event_id):
if record.on_disconnect == DisconnectMode.cancel and await request.is_disconnected():
offline-backend-20260512/backend/app/gateway/skill_catalog_cache.py
+440
View File
@@ -0,0 +1,440 @@
"""MySQL-backed projection for the Gateway skill list endpoint."""
from __future__ import annotations
import asyncio
import json
import logging
from datetime import UTC, datetime
from typing import Any
from sqlalchemy import or_, select, update
from sqlalchemy.exc import IntegrityError
from sqlalchemy.schema import CreateColumn
from deerflow.config.app_config import AppConfig
from deerflow.persistence.engine import get_engine, get_session_factory
from deerflow.persistence.skills.model import SkillCatalogSnapshotRow, SkillRow
from deerflow.skills.storage import get_or_new_skill_storage
logger = logging.getLogger(__name__)
_CATALOG_CACHE_KEY = "global"
_catalog_warm_lock = asyncio.Lock()
_memory_catalog_snapshot: list[dict[str, Any]] | None = None
_memory_catalog_is_fallback = False
_SKILL_LIST_COLUMNS = (
SkillRow.name,
SkillRow.category,
SkillRow.description,
SkillRow.license,
SkillRow.enabled,
SkillRow.always_on,
SkillRow.use_count,
SkillRow.view_count,
SkillRow.patch_count,
SkillRow.last_activity_at,
SkillRow.created_at,
SkillRow.state,
SkillRow.pinned,
SkillRow.featured,
SkillRow.source,
SkillRow.owner_user_id,
SkillRow.published,
SkillRow.name_zh,
SkillRow.square_id,
SkillRow.detail_preview,
)
_SKILL_CATALOG_COLUMNS = {
"category",
"description",
"license",
"enabled",
"available",
"use_count",
"view_count",
"patch_count",
"last_activity_at",
"state",
"pinned",
"featured",
"source",
"source_mtime_ns",
"detail_preview",
}
async def ensure_skill_catalog_schema() -> None:
"""Add compact catalog columns to an existing MySQL/SQLite skills table.
``create_all`` creates new tables but intentionally does not alter existing
ones. These additive columns all have safe defaults (or are nullable), so
this migration preserves existing ownership/detail data and is repeatable.
"""
engine = get_engine()
if engine is None:
return
def _ensure(sync_conn) -> None:
from sqlalchemy import func, inspect, update
table = SkillRow.__table__
inspector = inspect(sync_conn)
if table.name not in inspector.get_table_names():
table.create(sync_conn, checkfirst=True)
return
existing = {column["name"] for column in inspector.get_columns(table.name)}
preparer = sync_conn.dialect.identifier_preparer
quoted_table = preparer.quote(table.name)
for name in sorted(_SKILL_CATALOG_COLUMNS - existing):
column = table.columns[name]
definition = str(CreateColumn(column).compile(dialect=sync_conn.dialect))
sync_conn.exec_driver_sql(f"ALTER TABLE {quoted_table} ADD COLUMN {definition}")
existing.add(name)
# Preserve the short description previously exposed from ``detail``
# without selecting that potentially large field in normal list reads.
sync_conn.execute(
update(table)
.where(table.c.detail_preview.is_(None), table.c.detail.is_not(None))
.values(detail_preview=func.substr(table.c.detail, 1, 1024))
)
existing_indexes = {index["name"] for index in inspector.get_indexes(table.name)}
for index in table.indexes:
if index.name not in existing_indexes:
index.create(sync_conn, checkfirst=True)
async with engine.begin() as conn:
await conn.run_sync(_ensure)
async def load_skill_catalog_projection(
*,
user_id: str,
admin: bool = False,
) -> list[dict[str, Any]] | None:
"""Read lightweight skill catalog columns from MySQL, never SKILL.md or LONGTEXT.
``None`` means persistence is unavailable; an empty list is a valid empty
catalog. Access filtering happens in SQL so private skill metadata never
crosses into the request process for an unrelated user.
"""
session_factory = get_session_factory()
if session_factory is None:
return None
stmt = select(*_SKILL_LIST_COLUMNS).where(SkillRow.available.is_(True))
if not admin:
stmt = stmt.where(
or_(
SkillRow.owner_user_id.is_(None),
SkillRow.owner_user_id == user_id,
SkillRow.published.is_(True),
)
)
stmt = stmt.order_by(SkillRow.created_at.asc())
try:
async with session_factory() as session:
rows = (await session.execute(stmt)).mappings().all()
except Exception:
logger.warning("Skill MySQL catalog projection unavailable; retaining legacy fallback", exc_info=True)
return None
result: list[dict[str, Any]] = []
for row in rows:
item = dict(row)
item["builtin"] = item.get("owner_user_id") is None
result.append(item)
return result
def _parse_catalog_datetime(value: Any) -> datetime | None:
if isinstance(value, datetime):
return value
if not isinstance(value, str) or not value.strip():
return None
try:
return datetime.fromisoformat(value.replace("Z", "+00:00"))
except ValueError:
return None
async def refresh_skill_catalog_projection(
app_config: AppConfig | None = None,
*,
skills: list[Any] | None = None,
) -> int:
"""Synchronize filesystem metadata into one compact MySQL row per skill.
This runs at startup and after explicit skill changes. Request handlers
only read the projection and never walk the skill tree or usage sidecars.
Ownership, publication, and long-form detail columns are preserved.
"""
session_factory = get_session_factory()
if session_factory is None:
return 0
await ensure_skill_catalog_schema()
storage = get_or_new_skill_storage(app_config=app_config)
if skills is None:
skills = await asyncio.to_thread(storage.load_skills, enabled_only=False)
usage_entries = await asyncio.to_thread(_load_usage_entries)
catalog = await asyncio.to_thread(_build_catalog_payload, skills, usage_entries)
metadata_by_name: dict[str, tuple[str, int]] = {}
for skill in skills:
try:
stat = skill.skill_file.stat()
metadata_by_name[skill.name] = (
datetime.fromtimestamp(stat.st_mtime, tz=UTC).isoformat(),
int(stat.st_mtime_ns),
)
except OSError:
metadata_by_name[skill.name] = ("", 0)
async with session_factory() as session:
existing_rows = (
await session.execute(select(SkillRow.name, SkillRow.owner_user_id, SkillRow.always_on))
).all()
existing = {str(name): (owner_id, bool(always_on)) for name, owner_id, always_on in existing_rows}
catalog_names = {str(item.get("name") or "").strip() for item in catalog}
removed_names = sorted(name for name in existing if name not in catalog_names)
if removed_names:
await session.execute(
update(SkillRow)
.where(SkillRow.name.in_(removed_names))
.values(available=False, updated_at=datetime.now(UTC))
)
for item in catalog:
name = str(item.get("name") or "").strip()
if not name:
continue
category = str(item.get("category") or "custom")[:16]
created_at_text, source_mtime_ns = metadata_by_name.get(name, ("", 0))
created_at = _parse_catalog_datetime(item.get("created_at") or created_at_text) or datetime.now(UTC)
fields: dict[str, Any] = {
"category": category,
"description": str(item.get("description") or "")[:1024],
"license": str(item.get("license") or "")[:255] or None,
"enabled": bool(item.get("enabled", True)),
"available": True,
"use_count": max(0, int(item.get("use_count") or 0)),
"view_count": max(0, int(item.get("view_count") or 0)),
"patch_count": max(0, int(item.get("patch_count") or 0)),
"last_activity_at": _parse_catalog_datetime(item.get("last_activity_at")),
"state": str(item.get("state") or "active")[:16],
"pinned": bool(item.get("pinned")),
"featured": bool(item.get("featured")),
"source": str(item.get("source") or "upload")[:16],
"source_mtime_ns": source_mtime_ns,
"updated_at": datetime.now(UTC),
}
if name in existing:
# The per-skill always_on preference is admin-editable in MySQL
# and must survive filesystem metadata refreshes.
await session.execute(update(SkillRow).where(SkillRow.name == name).values(**fields))
else:
owner_id = None
session.add(
SkillRow(
name=name,
owner_user_id=owner_id,
published=category != "custom",
always_on=bool(item.get("always_on", False)),
created_at=created_at,
**fields,
)
)
await session.commit()
return len(catalog)
def _build_catalog_payload(skills: list[Any], usage_entries: dict[str, Any]) -> list[dict[str, Any]]:
payload: list[dict[str, Any]] = []
defaults = {
"use_count": 0,
"view_count": 0,
"patch_count": 0,
"last_used_at": None,
"last_viewed_at": None,
"last_patched_at": None,
"created_at": None,
"state": "active",
"pinned": False,
"featured": False,
"source": "upload",
}
for skill in skills:
usage = {**defaults, **(usage_entries.get(skill.name) or {})}
created_at = usage.get("created_at")
if not created_at:
try:
created_at = datetime.fromtimestamp(skill.skill_file.stat().st_mtime, tz=UTC).isoformat()
except OSError:
created_at = None
activity = [
value
for value in (
usage.get("last_used_at"),
usage.get("last_viewed_at"),
usage.get("last_patched_at"),
)
if value
]
last_activity_at = max(activity) if activity else created_at
payload.append(
{
"name": skill.name,
"description": skill.description or "",
"license": skill.license,
"category": str(getattr(skill.category, "value", skill.category)),
"enabled": bool(skill.enabled),
"always_on": bool(getattr(skill, "always_on", False)),
"use_count": int(usage.get("use_count") or 0),
"view_count": int(usage.get("view_count") or 0),
"patch_count": int(usage.get("patch_count") or 0),
"last_activity_at": last_activity_at,
"created_at": created_at,
"state": str(usage.get("state") or "active"),
"pinned": bool(usage.get("pinned")),
"featured": bool(usage.get("featured")),
"source": str(usage.get("source") or "upload"),
}
)
return payload
async def refresh_skill_catalog_snapshot(
app_config: AppConfig | None = None,
*,
skills: list[Any] | None = None,
) -> int:
"""Read skill files off-request and persist a single JSON snapshot in MySQL."""
session_factory = get_session_factory()
if session_factory is None:
return 0
storage = get_or_new_skill_storage(app_config=app_config)
if skills is None:
skills = await asyncio.to_thread(storage.load_skills, enabled_only=False)
usage_entries = await asyncio.to_thread(_load_usage_entries)
catalog = await asyncio.to_thread(_build_catalog_payload, skills, usage_entries)
encoded = json.dumps(catalog, ensure_ascii=False, separators=(",", ":"))
await _persist_catalog_snapshot(encoded)
global _memory_catalog_snapshot, _memory_catalog_is_fallback
_memory_catalog_snapshot = catalog
_memory_catalog_is_fallback = False
return len(catalog)
async def _persist_catalog_snapshot(encoded: str) -> None:
session_factory = get_session_factory()
if session_factory is None:
return
row = SkillCatalogSnapshotRow(
cache_key=_CATALOG_CACHE_KEY,
catalog_json=encoded,
updated_at=datetime.now(UTC),
)
for attempt in range(2):
try:
async with session_factory() as session:
await session.merge(row)
await session.commit()
return
except IntegrityError:
if attempt:
raise
# Two workers can warm an empty table at the same time. The loser
# retries as an update after the winner's transaction is visible.
def _load_usage_entries() -> dict[str, Any]:
from deerflow.skills.usage import all_entries
return all_entries()
async def load_skill_catalog_snapshot() -> list[dict[str, Any]] | None:
"""Load the prepared catalogue using MySQL only; None means not warmed."""
session_factory = get_session_factory()
if session_factory is None:
return None
async with session_factory() as session:
row = await session.get(SkillCatalogSnapshotRow, _CATALOG_CACHE_KEY)
if row is None:
return None
try:
payload = json.loads(row.catalog_json)
except (TypeError, ValueError):
logger.error("Invalid skill catalogue snapshot JSON; refresh is required")
return None
if not isinstance(payload, list):
logger.error("Invalid skill catalogue snapshot payload type; refresh is required")
return None
catalog = [item for item in payload if isinstance(item, dict) and item.get("name")]
global _memory_catalog_snapshot, _memory_catalog_is_fallback
_memory_catalog_snapshot = catalog
_memory_catalog_is_fallback = False
return catalog
async def load_or_warm_skill_catalog_snapshot(
app_config: AppConfig | None = None,
) -> list[dict[str, Any]] | None:
"""Load MySQL snapshot, warming it once on a cold/missing snapshot.
Startup warm-up is best-effort. If it missed because of transient startup
timing, the first list request builds the catalogue off the event loop and
stores it before returning. If persistence is temporarily unavailable, a
process-local copy keeps the endpoint usable without rescanning on every
request.
"""
global _memory_catalog_snapshot, _memory_catalog_is_fallback
if get_session_factory() is None:
return None
if _memory_catalog_is_fallback and _memory_catalog_snapshot is not None:
return _memory_catalog_snapshot
try:
catalog = await load_skill_catalog_snapshot()
except Exception:
logger.warning("Skill catalogue snapshot read failed; attempting a one-time warm-up", exc_info=True)
catalog = None
if catalog is not None:
return catalog
async with _catalog_warm_lock:
if _memory_catalog_is_fallback and _memory_catalog_snapshot is not None:
return _memory_catalog_snapshot
try:
catalog = await load_skill_catalog_snapshot()
except Exception:
logger.warning("Skill catalogue snapshot is unavailable during warm-up", exc_info=True)
catalog = None
if catalog is not None:
return catalog
storage = get_or_new_skill_storage(app_config=app_config)
skills = await asyncio.to_thread(storage.load_skills, enabled_only=False)
usage_entries = await asyncio.to_thread(_load_usage_entries)
catalog = await asyncio.to_thread(_build_catalog_payload, skills, usage_entries)
encoded = json.dumps(catalog, ensure_ascii=False, separators=(",", ":"))
try:
await _persist_catalog_snapshot(encoded)
except Exception:
logger.exception("Skill catalogue snapshot persistence failed; serving the warmed process-local copy")
_memory_catalog_snapshot = catalog
_memory_catalog_is_fallback = True
return catalog
_memory_catalog_snapshot = catalog
_memory_catalog_is_fallback = False
logger.info("Skill catalogue snapshot warmed on first request: skills=%d", len(catalog))
return catalog
offline-backend-20260512/backend/app/gateway/static/docs/LICENSE.redoc.txt
+22
View File
@@ -0,0 +1,22 @@
The MIT License (MIT)
Copyright (c) 2015-present, Rebilly, Inc.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
offline-backend-20260512/backend/app/gateway/static/docs/LICENSE.swagger-ui.txt
+202
View File
@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
offline-backend-20260512/backend/app/gateway/static/docs/favicon-32x32.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 628 B

offline-backend-20260512/backend/app/gateway/static/docs/redoc.standalone.js
+1851
View File
File diff suppressed because it is too large. Load diff
offline-backend-20260512/backend/app/gateway/static/docs/swagger-ui-bundle.js
+2
View File
@@ -0,0 +1,2 @@
/*! For license information please see swagger-ui-bundle.js.LICENSE.txt */
!function webpackUniversalModuleDefinition(s,o){"object"==typeof exports&&"object"==typeof module?module.exports=o():"function"==typeof define&&define.amd?define([],o):"object"==typeof exports?exports.SwaggerUIBundle=o():s.SwaggerUIBundle=o()}(this,(()=>(()=>{var s={251:(s,o)=>{o.read=function(s,o,i,a,u){var _,w,x=8*u-a-1,C=(1<<x)-1,j=C>>1,L=-7,B=i?u-1:0,$=i?-1:1,U=s[o+B];for(B+=$,_=U&(1<<-L)-1,U>>=-L,L+=x;L>0;_=256*_+s[o+B],B+=$,L-=8);for(w=_&(1<<-L)-1,_>>=-L,L+=a;L>0;w=256*w+s[o+B],B+=$,L-=8);if(0===_)_=1-j;else{if(_===C)return w?NaN:1/0*(U?-1:1);w+=Math.pow(2,a),_-=j}return(U?-1:1)*w*Math.pow(2,_-a)},o.write=function(s,o,i,a,u,_){var w,x,C,j=8*_-u-1,L=(1<<j)-1,B=L>>1,$=23===u?Math.pow(2,-24)-Math.pow(2,-77):0,U=a?0:_-1,V=a?1:-1,z=o<0||0===o&&1/o<0?1:0;for(o=Math.abs(o),isNaN(o)||o===1/0?(x=isNaN(o)?1:0,w=L):(w=Math.floor(Math.log(o)/Math.LN2),o*(C=Math.pow(2,-w))<1&&(w--,C*=2),(o+=w+B>=1?$/C:$*Math.pow(2,1-B))*C>=2&&(w++,C/=2),w+B>=L?(x=0,w=L):w+B>=1?(x=(o*C-1)*Math.pow(2,u),w+=B):(x=o*Math.pow(2,B-1)*Math.pow(2,u),w=0));u>=8;s[i+U]=255&x,U+=V,x/=256,u-=8);for(w=w<<u|x,j+=u;j>0;s[i+U]=255&w,U+=V,w/=256,j-=8);s[i+U-V]|=128*z}},462:(s,o,i)=>{"use strict";var a=i(40975);s.exports=a},659:(s,o,i)=>{var a=i(51873),u=Object.prototype,_=u.hasOwnProperty,w=u.toString,x=a?a.toStringTag:void 0;s.exports=function getRawTag(s){var o=_.call(s,x),i=s[x];try{s[x]=void 0;var a=!0}catch(s){}var u=w.call(s);return a&&(o?s[x]=i:delete s[x]),u}},694:(s,o,i)=>{"use strict";i(91599);var a=i(37257);i(12560),s.exports=a},953:(s,o,i)=>{"use strict";s.exports=i(53375)},1733:s=>{var o=/[^\x00-\x2f\x3a-\x40\x5b-\x60\x7b-\x7f]+/g;s.exports=function asciiWords(s){return s.match(o)||[]}},1882:(s,o,i)=>{var a=i(72552),u=i(23805);s.exports=function isFunction(s){if(!u(s))return!1;var o=a(s);return"[object Function]"==o||"[object GeneratorFunction]"==o||"[object AsyncFunction]"==o||"[object Proxy]"==o}},1907:(s,o,i)=>{"use strict";var a=i(41505),u=Function.prototype,_=u.call,w=a&&u.bind.bind(_,_);s.exports=a?w:function(s){return function(){return _.apply(s,arguments)}}},2205:function(s,o,i){var a;a=void 0!==i.g?i.g:this,s.exports=function(s){if(s.CSS&&s.CSS.escape)return s.CSS.escape;var cssEscape=function(s){if(0==arguments.length)throw new TypeError("`CSS.escape` requires an argument.");for(var o,i=String(s),a=i.length,u=-1,_="",w=i.charCodeAt(0);++u<a;)0!=(o=i.charCodeAt(u))?_+=o>=1&&o<=31||127==o||0==u&&o>=48&&o<=57||1==u&&o>=48&&o<=57&&45==w?"\\"+o.toString(16)+" ":0==u&&1==a&&45==o||!(o>=128||45==o||95==o||o>=48&&o<=57||o>=65&&o<=90||o>=97&&o<=122)?"\\"+i.charAt(u):i.charAt(u):_+="�";return _};return s.CSS||(s.CSS={}),s.CSS.escape=cssEscape,cssEscape}(a)},2209:(s,o,i)=>{"use strict";var a,u=i(9404),_=function productionTypeChecker(){invariant(!1,"ImmutablePropTypes type checking code is stripped in production.")};_.isRequired=_;var w=function getProductionTypeChecker(){return _};function getPropType(s){var o=typeof s;return Array.isArray(s)?"array":s instanceof RegExp?"object":s instanceof u.Iterable?"Immutable."+s.toSource().split(" ")[0]:o}function createChainableTypeChecker(s){function checkType(o,i,a,u,_,w){for(var x=arguments.length,C=Array(x>6?x-6:0),j=6;j<x;j++)C[j-6]=arguments[j];return w=w||a,u=u||"<<anonymous>>",null!=i[a]?s.apply(void 0,[i,a,u,_,w].concat(C)):o?new Error("Required "+_+" `"+w+"` was not specified in `"+u+"`."):void 0}var o=checkType.bind(null,!1);return o.isRequired=checkType.bind(null,!0),o}function createIterableSubclassTypeChecker(s,o){return function createImmutableTypeChecker(s,o){return createChainableTypeChecker((function validate(i,a,u,_,w){var x=i[a];if(!o(x)){var C=getPropType(x);return new Error("Invalid "+_+" `"+w+"` of type `"+C+"` supplied to `"+u+"`, expected `"+s+"`.")}return null}))}("Iterable."+s,(function(s){return u.Iterable.isIterable(s)&&o(s)}))}(a={listOf:w,mapOf:w,orderedMapOf:w,setOf:w,orderedSetOf:w,stackOf:w,iterableOf:w,recordOf:w,shape:w,contains:w,mapContains:w,orderedMapContains:w,list:_,map:_,orderedMap:_,set:_,orderedSet:_,stack:_,seq:_,record:_,iterable:_}).iterable.indexed=createIterableSubclassTypeChecker("Indexed",u.Iterable.isIndexed),a.iterable.keyed=createIterableSubclassTypeChecker("Keyed",u.Iterable.isKeyed),s.exports=a},2404:(s,o,i)=>{var a=i(60270);s.exports=function isEqual(s,o){return a(s,o)}},2523:s=>{s.exports=function baseFindIndex(s,o,i,a){for(var u=s.length,_=i+(a?1:-1);a?_--:++_<u;)if(o(s[_],_,s))return _;return-1}},2532:(s,o,i)=>{"use strict";var a=i(45951),u=Object.defineProperty;s.exports=function(s,o){try{u(a,s,{value:o,configurable:!0,writable:!0})}catch(i){a[s]=o}return o}},2694:(s,o,i)=>{"use strict";var a=i(6925);function emptyFunction(){}function emptyFunctionWithReset(){}emptyFunctionWithReset.resetWarningCache=emptyFunction,s.exports=function(){function shim(s,o,i,u,_,w){if(w!==a){var x=new Error("Calling PropTypes validators directly is not supported by the `prop-types` package. Use PropTypes.checkPropTypes() to call them. Read more at httpLine truncated
offline-backend-20260512/backend/app/gateway/static/docs/swagger-ui.css
+3
View File
@@ -0,0 +1,3 @@
.swagger-ui{color:#3b4151;font-family:sans-serif}.swagger-ui html{line-height:1.15;-ms-text-size-adjust:100%;-webkit-text-size-adjust:100%}.swagger-ui body{margin:0}.swagger-ui article,.swagger-ui aside,.swagger-ui footer,.swagger-ui header,.swagger-ui nav,.swagger-ui section{display:block}.swagger-ui h1{font-size:2em;margin:.67em 0}.swagger-ui figcaption,.swagger-ui figure,.swagger-ui main{display:block}.swagger-ui figure{margin:1em 40px}.swagger-ui hr{box-sizing:content-box;height:0;overflow:visible}.swagger-ui pre{font-family:monospace,monospace;font-size:1em}.swagger-ui a{background-color:transparent;-webkit-text-decoration-skip:objects}.swagger-ui abbr[title]{border-bottom:none;text-decoration:underline;-webkit-text-decoration:underline dotted;text-decoration:underline dotted}.swagger-ui b,.swagger-ui strong{font-weight:inherit;font-weight:bolder}.swagger-ui code,.swagger-ui kbd,.swagger-ui samp{font-family:monospace,monospace;font-size:1em}.swagger-ui dfn{font-style:italic}.swagger-ui mark{background-color:#ff0;color:#000}.swagger-ui small{font-size:80%}.swagger-ui sub,.swagger-ui sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline}.swagger-ui sub{bottom:-.25em}.swagger-ui sup{top:-.5em}.swagger-ui audio,.swagger-ui video{display:inline-block}.swagger-ui audio:not([controls]){display:none;height:0}.swagger-ui img{border-style:none}.swagger-ui svg:not(:root){overflow:hidden}.swagger-ui button,.swagger-ui input,.swagger-ui optgroup,.swagger-ui select,.swagger-ui textarea{font-family:sans-serif;font-size:100%;line-height:1.15;margin:0}.swagger-ui button,.swagger-ui input{overflow:visible}.swagger-ui button,.swagger-ui select{text-transform:none}.swagger-ui [type=reset],.swagger-ui [type=submit],.swagger-ui button,.swagger-ui html [type=button]{-webkit-appearance:button}.swagger-ui [type=button]::-moz-focus-inner,.swagger-ui [type=reset]::-moz-focus-inner,.swagger-ui [type=submit]::-moz-focus-inner,.swagger-ui button::-moz-focus-inner{border-style:none;padding:0}.swagger-ui [type=button]:-moz-focusring,.swagger-ui [type=reset]:-moz-focusring,.swagger-ui [type=submit]:-moz-focusring,.swagger-ui button:-moz-focusring{outline:1px dotted ButtonText}.swagger-ui fieldset{padding:.35em .75em .625em}.swagger-ui legend{box-sizing:border-box;color:inherit;display:table;max-width:100%;padding:0;white-space:normal}.swagger-ui progress{display:inline-block;vertical-align:baseline}.swagger-ui textarea{overflow:auto}.swagger-ui [type=checkbox],.swagger-ui [type=radio]{box-sizing:border-box;padding:0}.swagger-ui [type=number]::-webkit-inner-spin-button,.swagger-ui [type=number]::-webkit-outer-spin-button{height:auto}.swagger-ui [type=search]{-webkit-appearance:textfield;outline-offset:-2px}.swagger-ui [type=search]::-webkit-search-cancel-button,.swagger-ui [type=search]::-webkit-search-decoration{-webkit-appearance:none}.swagger-ui ::-webkit-file-upload-button{-webkit-appearance:button;font:inherit}.swagger-ui details,.swagger-ui menu{display:block}.swagger-ui summary{display:list-item}.swagger-ui canvas{display:inline-block}.swagger-ui [hidden],.swagger-ui template{display:none}.swagger-ui .debug *{outline:1px solid gold}.swagger-ui .debug-white *{outline:1px solid #fff}.swagger-ui .debug-black *{outline:1px solid #000}.swagger-ui .debug-grid{background:transparent url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAgAAAAICAYAAADED76LAAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAAyhpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADw/eHBhY2tldCBiZWdpbj0i77u/IiBpZD0iVzVNME1wQ2VoaUh6cmVTek5UY3prYzlkIj8+IDx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IkFkb2JlIFhNUCBDb3JlIDUuNi1jMTExIDc5LjE1ODMyNSwgMjAxNS8wOS8xMC0wMToxMDoyMCAgICAgICAgIj4gPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4gPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIgeG1sbnM6eG1wTU09Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC9tbS8iIHhtbG5zOnN0UmVmPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvc1R5cGUvUmVzb3VyY2VSZWYjIiB4bWxuczp4bXA9Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC8iIHhtcE1NOkRvY3VtZW50SUQ9InhtcC5kaWQ6MTRDOTY4N0U2N0VFMTFFNjg2MzZDQjkwNkQ4MjgwMEIiIHhtcE1NOkluc3RhbmNlSUQ9InhtcC5paWQ6MTRDOTY4N0Q2N0VFMTFFNjg2MzZDQjkwNkQ4MjgwMEIiIHhtcDpDcmVhdG9yVG9vbD0iQWRvYmUgUGhvdG9zaG9wIENDIDIwMTUgKE1hY2ludG9zaCkiPiA8eG1wTU06RGVyaXZlZEZyb20gc3RSZWY6aW5zdGFuY2VJRD0ieG1wLmlpZDo3NjcyQkQ3NjY3QzUxMUU2QjJCQ0UyNDA4MTAwMjE3MSIgc3RSZWY6ZG9jdW1lbnRJRD0ieG1wLmRpZDo3NjcyQkQ3NzY3QzUxMUU2QjJCQ0UyNDA4MTAwMjE3MSIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRGPiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/PsBS+GMAAAAjSURBVHjaYvz//z8DLsD4gcGXiYEAGBIKGBne//fFpwAgwAB98AaF2pjlUQAAAABJRU5ErkJggg==) repeat 0 0}.swagger-ui .debug-grid-16{background:transparent url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAAyhpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADw/eHBhY2tldCBiZWdpbj0i77u/IiBpZD0iVzVNME1wQ2VoaUh6cmVTek5UY3prYzlkIj8+IDx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IkFkb2JlIFhNUCBDbLine truncated
/*# sourceMappingURL=swagger-ui.css.map*/
offline-backend-20260512/backend/app/gateway/worker_pool/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Single-host managed online workers; background scheduling stays on the gateway."""
offline-backend-20260512/backend/app/gateway/worker_pool/admission.py
+206
View File
@@ -0,0 +1,206 @@
"""Bound executing runs, including detached/background runs, not HTTP streams."""
from __future__ import annotations
import asyncio
from contextvars import ContextVar
from functools import wraps
from fastapi import HTTPException
_worker_ticket = ContextVar("worker_admission_ticket", default=None)
class AdmissionTicket:
def __init__(self, gate):
if gate.active + gate.waiting >= gate.capacity + gate.queue_limit:
raise HTTPException(429, "执行节点繁忙，请稍后重试", headers={"Retry-After": "3"})
self.gate = gate
self.deadline = asyncio.get_running_loop().time() + gate.timeout
self.waiting = True
self.acquired = False
self.released = False
self.started = False
gate.waiting += 1
async def acquire(self):
try:
remaining = max(0.001, self.deadline - asyncio.get_running_loop().time())
await asyncio.wait_for(self.gate.semaphore.acquire(), remaining)
self.acquired = True
self.gate.active += 1
except TimeoutError:
raise HTTPException(429, "等待执行超时，请稍后重试") from None
finally:
if self.waiting:
self.gate.waiting -= 1
self.waiting = False
def release(self, _task=None):
if self.released:
return
self.released = True
if self.waiting:
self.gate.waiting -= 1
self.waiting = False
if self.acquired:
self.gate.active -= 1
self.gate.semaphore.release()
async def _finish_queued_run(kwargs, *, cancelled=False, message="等待执行超时，请稍后重试"):
from deerflow.runtime.runs.schemas import RunStatus
manager, record, bridge = kwargs["run_manager"], kwargs["record"], kwargs["bridge"]
try:
async with asyncio.timeout(5):
await manager.set_status(record.run_id, RunStatus.interrupted if cancelled else RunStatus.error, error=None if cancelled else message)
ctx = kwargs.get("ctx")
store = getattr(ctx, "thread_store", None)
if store is not None and not getattr(record, "superseded", False):
await store.update_status(record.thread_id, "interrupted" if cancelled else "error")
finally:
try:
if not cancelled:
await bridge.publish(record.run_id, "error", {"name": "WorkerQueueTimeout", "message": message})
finally:
await bridge.publish_end(record.run_id)
asyncio.create_task(bridge.cleanup(record.run_id, delay=60))
async def run_with_worker_admission(runner, **kwargs):
ticket = _worker_ticket.get()
if ticket is None:
return await runner(**kwargs)
ticket.started = True
record, bridge = kwargs["record"], kwargs["bridge"]
# Publish a real, persisted run id before waiting for an execution slot.
# The router keeps its canonical Location/reconnect headers unchanged.
try:
await bridge.publish(record.run_id, "metadata", {"run_id": record.run_id, "thread_id": record.thread_id,
"worker_queue": {"waiting": ticket.gate.waiting, "capacity": ticket.gate.capacity}})
await ticket.acquire()
except asyncio.CancelledError:
await _finish_queued_run(kwargs, cancelled=True)
raise
except HTTPException as exc:
await _finish_queued_run(kwargs, message=str(exc.detail))
return
except Exception:
await _finish_queued_run(kwargs, message="执行节点排队初始化失败，请查看节点日志")
return
return await runner(**kwargs)
class Admission:
def __init__(self, capacity: int, queue_limit: int = 128, timeout: float = 60):
self.capacity = capacity
self.active = 0
self.waiting = 0
self.queue_limit = queue_limit
self.timeout = timeout
self.semaphore = asyncio.Semaphore(capacity)
self.lock = asyncio.Lock()
async def acquire(self):
async with self.lock:
if self.waiting >= self.queue_limit:
raise HTTPException(429, "执行节点繁忙，请稍后重试", headers={"Retry-After": "3"})
self.waiting += 1
try:
await asyncio.wait_for(self.semaphore.acquire(), self.timeout)
except TimeoutError:
raise HTTPException(429, "等待执行超时，请稍后重试", headers={"Retry-After": "3"}) from None
finally:
self.waiting -= 1
self.active += 1
def release(self, _task=None):
self.active -= 1
self.semaphore.release()
def bounded_worker_run(function):
@wraps(function)
async def wrapped(body, thread_id, request, *args, **kwargs):
pool = getattr(request.app.state, "worker_pool", None)
fallback_to_main = bool(getattr(getattr(request, "state", None), "worker_pool_fallback", False))
if pool is not None and pool.db is not None and pool.db.execute("SELECT 1 FROM affinity WHERE thread=?", (thread_id,)).fetchone() and not fallback_to_main:
raise HTTPException(409, "该会话由独立 Worker 管理，请通过会话接口提交，不能在主服务重复执行")
gate = getattr(request.app.state, "worker_admission", None)
if gate is None:
if pool is None:
return await function(body, thread_id, request, *args, **kwargs)
if pool.local_threads.get(thread_id, 0):
raise HTTPException(409, "该会话正在主服务执行，请等待当前回答完成")
# Reserve before the first await, including start_run's preparation
# phase before RunRecord.task exists. Otherwise a concurrent routed
# request could assign this same thread to another process.
pool.local_threads[thread_id] = pool.local_threads.get(thread_id, 0) + 1
def release_local(_task=None):
remaining = pool.local_threads.get(thread_id, 1) - 1
if remaining:
pool.local_threads[thread_id] = remaining
else:
pool.local_threads.pop(thread_id, None)
try:
record = await function(body, thread_id, request, *args, **kwargs)
except BaseException:
release_local()
raise
if record.task is None:
release_local()
else:
record.task.add_done_callback(release_local)
return record
if getattr(request.app.state, "worker_deferred_admission", False) and getattr(request.app.state, "concurrency_gate", None) is None:
ticket = AdmissionTicket(gate)
token = _worker_ticket.set(ticket)
try:
from deerflow.runtime.thread_paths import aresolve_path_user_id
await aresolve_path_user_id(thread_id)
refresh = getattr(request.app.state, "worker_skill_refresh", None)
if refresh is not None:
await refresh.check()
record = await function(body, thread_id, request, *args, **kwargs)
except BaseException:
ticket.release()
raise
finally:
_worker_ticket.reset(token)
if record.task is None:
ticket.release()
else:
def finish_ticket(task):
ticket.release()
if task.cancelled() and not ticket.started:
manager = getattr(request.app.state, "run_manager", None)
bridge = getattr(request.app.state, "stream_bridge", None)
if manager is not None and bridge is not None:
asyncio.create_task(_finish_queued_run({"run_manager": manager, "record": record, "bridge": bridge}, cancelled=True))
record.task.add_done_callback(finish_ticket)
return record
await gate.acquire()
try:
# File/path middleware is synchronous. Resolve the persisted owner
# on the database's home loop before it touches those helpers.
from deerflow.runtime.thread_paths import aresolve_path_user_id
await aresolve_path_user_id(thread_id)
refresh = getattr(request.app.state, "worker_skill_refresh", None)
if refresh is not None:
await refresh.check()
record = await function(body, thread_id, request, *args, **kwargs)
except BaseException:
gate.release()
raise
if record.task is None:
gate.release()
else:
record.task.add_done_callback(gate.release)
return record
return wrapped
offline-backend-20260512/backend/app/gateway/worker_pool/controller.py
+359
View File
@@ -0,0 +1,359 @@
"""One gateway owns reconciliation and durable conversation affinity.
Never move conversations from a live worker. A draining worker is stopped only
after its run registry AND outstanding HTTP requests are empty. Failure does
not replay a request, because a model/tool may already have produced effects.
"""
from __future__ import annotations
import asyncio
import json
import os
import sqlite3
import time
from urllib.parse import quote
import httpx
from .docker import DockerEngine, DockerError, worker_spec
from .settings import SettingsStore
class PoolUnavailable(RuntimeError):
pass
class Controller:
def __init__(self, store: SettingsStore):
self.store = store
self.lock = asyncio.Lock()
self.route_lock = asyncio.Lock()
self.nodes: dict[str, dict] = {}
self.error: str | None = None
self.task: asyncio.Task | None = None
self._lock_handle = None
self.http = httpx.AsyncClient(timeout=httpx.Timeout(15, read=None), trust_env=False, limits=httpx.Limits(max_connections=256, max_keepalive_connections=64))
self.db: sqlite3.Connection | None = None
self.pending = 0
self.inflight: dict[str, int] = {}
self.local_threads: dict[str, int] = {}
self._python_probe_key: tuple | None = None
self._python_probe_result: str | None = None
self._worker_image_key: tuple | None = None
self._worker_image_result: str | None = None
self.discovery_complete = not bool(store.load().source_container)
def _ensure_db(self):
if self.db is None:
self.db = sqlite3.connect(self.store.directory / "affinity.sqlite3")
self.db.execute("PRAGMA journal_mode=WAL")
self.db.execute("CREATE TABLE IF NOT EXISTS affinity (thread TEXT PRIMARY KEY, node TEXT NOT NULL)")
self.db.commit()
return self.db
def _acquire_controller_lock(self):
if self._lock_handle is not None:
return
self.store.directory.mkdir(mode=0o700, parents=True, exist_ok=True)
handle = (self.store.directory / "controller.lock").open("a+b")
try:
if os.name == "nt":
import msvcrt
handle.seek(0)
if not handle.read(1):
handle.write(b"0")
handle.flush()
handle.seek(0)
msvcrt.locking(handle.fileno(), msvcrt.LK_NBLCK, 1)
else:
import fcntl
fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError:
handle.close()
raise RuntimeError("Worker 池只能由一个主服务进程管理；请关闭重复的 Worker 池控制器") from None
self._lock_handle = handle
def start(self):
settings = self.store.load()
if settings.enabled:
self._acquire_controller_lock()
self._ensure_db()
if self.task is None or self.task.done():
self.task = asyncio.create_task(self.loop())
if hasattr(self, "diagnostics") and self.diagnostics.enabled:
self.diagnostics.start()
async def close(self):
if self.task:
self.task.cancel()
await asyncio.gather(self.task, return_exceptions=True)
if hasattr(self, "diagnostics"):
await self.diagnostics.close()
await self.http.aclose()
if self.db is not None:
self.db.close()
self.db = None
self._release_controller_lock()
async def loop(self):
try:
while True:
try:
await self.reconcile()
except Exception as exc:
self.error = str(exc) if isinstance(exc, (DockerError, PoolUnavailable)) else f"节点检查失败（{type(exc).__name__}）"
settings = self.store.load()
if not settings.enabled and not self.nodes:
return
await asyncio.sleep(5)
finally:
if not self.store.load().enabled:
self._release_controller_lock()
def _release_controller_lock(self):
if self._lock_handle is None:
return
if os.name == "nt":
import msvcrt
self._lock_handle.seek(0)
msvcrt.locking(self._lock_handle.fileno(), msvcrt.LK_UNLCK, 1)
else:
import fcntl
fcntl.flock(self._lock_handle, fcntl.LOCK_UN)
self._lock_handle.close()
self._lock_handle = None
async def inspect_template(self, settings):
config = settings
if config.desired * config.cpu > config.total_cpu or config.desired * config.memory_mb > config.total_memory_mb:
raise PoolUnavailable("Worker 总资源超过设置的预算，请减少数量或调整预算")
if config.desired * (config.database_pool_size + 2 * config.postgres_pool_size) > config.database_connection_budget:
raise PoolUnavailable("Worker 数据库连接总额度超过预算，请减少数量或单节点连接数")
if not config.source_container or not config.network:
raise PoolUnavailable("请填写源容器名称和 Docker 网络")
engine = DockerEngine(config)
source = await asyncio.to_thread(engine.request, "GET", f"/containers/{quote(config.source_container, safe='')}/json?size=true")
if source.get("Config", {}).get("Labels", {}).get("deerflow.worker-pool"):
raise PoolUnavailable("源容器必须是主服务，不能选择已有 Worker")
backend = await asyncio.to_thread(
engine.configured_storage_backend, source, config.python_executable, config.working_dir,
)
probe_key = (
config.transport, config.host, config.port, config.username, config.source_container,
source.get("Id"), source.get("Image"), config.python_executable,
config.working_dir, source.get("Config", {}).get("User", ""), backend,
)
if probe_key == self._python_probe_key and self._python_probe_result:
python_executable = self._python_probe_result
else:
python_executable = await asyncio.to_thread(
engine.discover_worker_python, source, config.python_executable, config.working_dir,
)
self._python_probe_key = probe_key
self._python_probe_result = python_executable
if python_executable != config.python_executable:
config = config.model_copy(update={"python_executable": python_executable})
self.store.save(config)
self._python_probe_key = (
config.transport, config.host, config.port, config.username, config.source_container,
source.get("Id"), source.get("Image"), config.python_executable,
config.working_dir, source.get("Config", {}).get("User", ""), backend,
)
runtime_image = None
if config.enabled and config.desired > 0:
image_key = (source.get("Id"), source.get("Image"), python_executable, backend)
if image_key != self._worker_image_key or not self._worker_image_result:
self._worker_image_result = await asyncio.to_thread(
engine.snapshot_worker_runtime, source, self.store.raw()["instance"],
)
self._worker_image_key = image_key
runtime_image = self._worker_image_result
spec = worker_spec(
source, config, self.store.raw()["instance"], self.store.raw()["token"],
str(self.store.directory), runtime_image=runtime_image,
require_shared_archive=backend == "postgres" and config.enabled and config.desired > 0,
)
return engine, spec
async def reconcile(self):
async with self.lock:
settings = self.store.load()
if not settings.enabled and not settings.source_container:
return
db = self._ensure_db()
engine, spec = await self.inspect_template(settings)
instance = self.store.raw()["instance"]
filters = quote(json.dumps({"label": [f"deerflow.worker-pool={instance}"]}))
containers = await asyncio.to_thread(engine.request, "GET", f"/containers/json?all=1&filters={filters}")
self.discovery_complete = True
desired = settings.desired if settings.enabled else 0
wanted = {f"dfw-{instance}-{i}" for i in range(desired)}
existing = {row["Names"][0].lstrip("/"): row for row in containers}
stale_templates = set()
for name, row in list(existing.items()):
self.nodes.setdefault(name, {"name": name, "id": row["Id"], "ready": False, "active": 0, "draining": name not in wanted, "url": f"http://{name}:8001"})
if name in wanted and row.get("State") == "running" and row.get("ImageID") and row["ImageID"] != spec["Image"]:
stale_templates.add(name)
self.nodes[name]["draining"] = True
# A stopped container cannot be executing a request. If its saved
# startup template has an old Python path, replace it with the
# resolved template instead of restarting the same broken command.
for name, row in list(existing.items()):
if name not in wanted or row["State"] == "running":
continue
detail = await asyncio.to_thread(engine.request, "GET", f"/containers/{quote(row['Id'], safe='')}/json")
worker_config = detail.get("Config", {})
stale_template = (
detail.get("Image") != spec["Image"]
or worker_config.get("Entrypoint") != spec["Entrypoint"]
or worker_config.get("WorkingDir") != spec["WorkingDir"]
)
if not stale_template:
continue
if hasattr(self, "diagnostics"):
await self.diagnostics.archive_logs(engine, name)
await asyncio.to_thread(engine.request, "DELETE", f"/containers/{quote(row['Id'], safe='')}")
db.execute("DELETE FROM affinity WHERE node=?", (name,))
db.commit()
existing.pop(name, None)
self.nodes.pop(name, None)
for name, row in list(existing.items()):
previous = self.nodes.get(name, {})
node = {
"name": name,
"id": row["Id"],
"ready": previous.get("ready", False),
"ever_ready": previous.get("ever_ready", False),
"active": previous.get("active", 0),
"capacity": previous.get("capacity"),
"draining": name not in wanted or name in stale_templates,
"url": f"http://{name}:8001",
"checked_at": time.time(),
}
for key in ("running", "waiting", "process_started_at", "runtime_revision", "event_loop_lag_ms", "http"):
if key in previous:
node[key] = previous[key]
self.nodes[name] = node
if row["State"] != "running":
node["ready"] = False
node["ever_ready"] = False
# A confirmed stopped process cannot keep executing. Old
# conversations can move only after this confirmation.
db.execute("DELETE FROM affinity WHERE node=?", (name,))
db.commit()
if name in wanted:
await asyncio.to_thread(engine.request, "POST", f"/containers/{row['Id']}/start")
else:
if hasattr(self, "diagnostics"):
await self.diagnostics.archive_logs(engine, name)
await asyncio.to_thread(engine.request, "DELETE", f"/containers/{row['Id']}")
self.nodes.pop(name, None)
continue
try:
diagnostics_enabled = bool(getattr(getattr(self, "diagnostics", None), "enabled", False))
response = await self.http.get(
node["url"] + "/internal/worker-status",
headers={
"x-deerflow-worker-token": self.store.raw()["token"],
"x-deerflow-diagnostics-enabled": "true" if diagnostics_enabled else "false",
},
timeout=3,
)
response.raise_for_status()
status = response.json()
node.update(ready=status["ready"], active=status["active"], capacity=status["capacity"], ever_ready=True)
for key in ("running", "waiting", "process_started_at", "runtime_revision", "event_loop_lag_ms", "http"):
if key in status:
node[key] = status[key]
except (httpx.HTTPError, ValueError, KeyError):
node["ready"] = False
node["error"] = "等待就绪或连接失败；请检查主容器是否位于同一网络、镜像代码和聊天存储配置"
continue
if node["draining"] and not node["active"] and not self.inflight.get(name, 0):
# Refuse all new reservations while stop is in flight.
node["ready"] = False
node["ever_ready"] = False
await asyncio.to_thread(engine.request, "POST", f"/containers/{row['Id']}/stop?t=30")
if hasattr(self, "diagnostics"):
await self.diagnostics.archive_logs(engine, name)
await asyncio.to_thread(engine.request, "DELETE", f"/containers/{row['Id']}")
db.execute("DELETE FROM affinity WHERE node=?", (name,))
db.commit()
self.nodes.pop(name, None)
existing.pop(name, None)
# Replacements are created only after the previous container has
# drained and been removed. This also refreshes Workers after the
# main service's runtime image or configured storage backend changes.
for name in sorted(wanted - existing.keys()):
row = await asyncio.to_thread(engine.request, "POST", f"/containers/create?name={name}", spec)
self.nodes[name] = {
"name": name,
"id": row["Id"],
"ready": False,
"active": 0,
"draining": False,
"url": f"http://{name}:8001",
}
await asyncio.to_thread(engine.request, "POST", f"/containers/{row['Id']}/start")
existing[name] = {"Id": row["Id"], "State": "running", "ImageID": spec["Image"]}
# Missing nodes are not silently reassigned: an operator must
# reconcile the actual container state before changing affinity.
self.error = None
async def reserve(self, thread: str, *, create: bool) -> dict | None:
settings = self.store.load()
if not settings.enabled and not self.nodes:
return None
db = self._ensure_db()
# Container creation/SSH can be slow; routing must never wait on that lock.
async with self.route_lock:
row = db.execute("SELECT node FROM affinity WHERE thread=?", (thread,)).fetchone()
if row is None and create and self.local_threads.get(thread, 0):
raise PoolUnavailable("该会话正在主服务执行，请等待当前轮完成")
if row:
node = self.nodes.get(row[0])
if node and node.get("draining") and create:
return {"_fallback_to_main": True, "_fallback_reason": "worker_draining", "_worker": dict(node)}
# A delayed health probe stops new work, but must not hide an
# existing run's result. Keep reads/cancellation on its owner;
# stopped containers clear ever_ready before affinity changes.
if not node or (not node["ready"] and (create or not node.get("ever_ready"))):
raise PoolUnavailable("该会话的执行节点暂不可用，请稍后重试；系统不会重复提交任务")
if node["draining"] and create:
raise PoolUnavailable("该会话节点正在停止接单，请等待当前任务完成后重试")
if create:
capacity = max(1, int(node.get("capacity") or settings.concurrency))
load = max(int(node.get("active") or 0), self.inflight.get(node["name"], 0))
if load >= capacity:
return {"_fallback_to_main": True, "_fallback_reason": "worker_saturated", "_worker": dict(node)}
elif not settings.enabled or not create:
return None
else:
choices = [n for n in self.nodes.values() if n["ready"] and not n["draining"]]
if not choices:
return {"_fallback_to_main": True, "_fallback_reason": "workers_unavailable"}
available = []
for candidate in choices:
capacity = max(1, int(candidate.get("capacity") or settings.concurrency))
load = max(int(candidate.get("active") or 0), self.inflight.get(candidate["name"], 0))
if load < capacity:
available.append((load, candidate))
if not available:
return {"_fallback_to_main": True, "_fallback_reason": "worker_pool_saturated"}
_, node = min(available, key=lambda item: (item[0], item[1]["name"]))
db.execute("INSERT INTO affinity VALUES (?, ?)", (thread, node["name"]))
db.commit()
self.inflight[node["name"]] = self.inflight.get(node["name"], 0) + 1
return dict(node)
def release(self, name):
self.inflight[name] = max(0, self.inflight.get(name, 0) - 1)
def status(self):
return {"settings": self.store.public(), "nodes": [{k: v for k, v in n.items() if k != "url"} for n in self.nodes.values()], "error": self.error}
offline-backend-20260512/backend/app/gateway/worker_pool/diagnostics.py
+348
View File
@@ -0,0 +1,348 @@
"""Admin-enabled, bounded diagnostics with independent, short DB probes."""
import asyncio
import json
import os
import re
import sqlite3
import time
from collections import deque
from urllib.parse import quote, urlencode
from .docker import DockerEngine, DockerError, decode_docker_output
from .host_metrics import rates
from .telemetry import RequestRecorder
def redact_logs(value):
value = re.sub(r"(?i)([a-z][a-z0-9+.-]*://)[^\s/@]+:[^\s/@]+@", r"\1[REDACTED]@", value)
value = re.sub(r"(?i)(Bearer\s+)[^\s\"']+", r"\1[REDACTED]", value)
value = re.sub(r"(?i)((?:password|passwd|api[_-]?key|access[_-]?token|secret|authorization|token)[\"']?\s*[:=]\s*[\"']?)[^\s,\"'}]+", r"\1[REDACTED]", value)
value = re.sub(r"\beyJ[a-zA-Z0-9_-]+\.[a-zA-Z0-9_-]+\.[a-zA-Z0-9_-]+", "[REDACTED_JWT]", value)
value = re.sub(r"\bsk-[a-zA-Z0-9_-]{12,}", "[REDACTED_KEY]", value)
return value
def container_metrics(detail, stats, previous_stats=None):
cpu = stats.get("cpu_stats", {})
previous = previous_stats.get("cpu_stats", {}) if previous_stats else stats.get("precpu_stats", {})
delta = cpu.get("cpu_usage", {}).get("total_usage", 0) - previous.get("cpu_usage", {}).get("total_usage", 0)
system_delta = cpu.get("system_cpu_usage", 0) - previous.get("system_cpu_usage", 0)
cores = cpu.get("online_cpus") or len(cpu.get("cpu_usage", {}).get("percpu_usage", [])) or 1
memory = stats.get("memory_stats", {})
cache = memory.get("stats", {}).get("inactive_file", memory.get("stats", {}).get("total_inactive_file", 0))
state = detail.get("State", {})
return {
"name": detail.get("Name", "").lstrip("/"),
"image": detail.get("Config", {}).get("Image", ""),
"image_id": detail.get("Image", "")[:22],
"created_at": detail.get("Created"),
"started_at": state.get("StartedAt"),
"status": state.get("Status"),
"health": state.get("Health", {}).get("Status"),
"restarts": detail.get("RestartCount", 0),
"oom_killed": state.get("OOMKilled", False),
"cpu_percent": round(delta / system_delta * cores * 100, 2) if previous.get("system_cpu_usage") and system_delta > 0 and delta >= 0 else None,
"cpu_limit": detail.get("HostConfig", {}).get("NanoCpus", 0) / 1e9 or None,
"memory_bytes": max(0, memory.get("usage", 0) - cache),
"memory_limit": memory.get("limit"),
"pids": stats.get("pids_stats", {}).get("current"),
"log_policy": detail.get("HostConfig", {}).get("LogConfig", {}),
"host_proc_mounted": any(m.get("Destination") == "/host/proc" for m in detail.get("Mounts", [])),
"docker_socket_mounted": any(m.get("Destination") == "/var/run/docker.sock" for m in detail.get("Mounts", [])),
}
class Diagnostics:
def __init__(self, pool, app):
self.pool = pool
self.app = app
self.path = pool.store.directory / "diagnostics-settings.json"
self.archive_directory = pool.store.directory / "log-archive"
self.enabled = json.loads(self.path.read_text()).get("enabled", False) if self.path.exists() else False
self.recorder = RequestRecorder()
self.recorder.enabled = self.enabled
app.state.request_recorder = self.recorder
self.history = deque(maxlen=720)
self.latest = None
self.previous = None
self.container_previous = {}
from .revision import code_revision
self.runtime_revision = code_revision()
self.task = None
self.lag_task = None
self.loop_lag_ms = 0
self.db_engines = {}
self.db = None
if self.enabled:
self._ensure_db()
def _ensure_db(self):
if self.db is None:
self.db = sqlite3.connect(self.pool.store.directory / "diagnostics.sqlite3")
self.db.execute("PRAGMA journal_mode=WAL")
self.db.execute("CREATE TABLE IF NOT EXISTS snapshots (id INTEGER PRIMARY KEY, at REAL, body TEXT)")
self.db.execute("CREATE TABLE IF NOT EXISTS requests (id INTEGER PRIMARY KEY, at REAL, body TEXT)")
for (value,) in reversed(self.db.execute("SELECT body FROM snapshots ORDER BY id DESC LIMIT 720").fetchall()):
self.history.append(json.loads(value))
for (value,) in reversed(self.db.execute("SELECT body FROM requests ORDER BY id DESC LIMIT 2000").fetchall()):
self.recorder.completed.append(json.loads(value))
return self.db
def start(self):
if not self.enabled:
return
self._ensure_db()
if self.task is not None and not self.task.done():
return
self.task = asyncio.create_task(self.loop())
self.lag_task = asyncio.create_task(self.measure_lag())
async def close(self):
for task in (self.task, self.lag_task):
if task:
task.cancel()
await asyncio.gather(*(t for t in (self.task, self.lag_task) if t), return_exceptions=True)
self.persist_requests()
if self.db is not None:
self.db.close()
self.db = None
for engine in self.db_engines.values():
await engine.dispose()
async def set_enabled(self, enabled):
self.enabled = enabled
self.recorder.enabled = enabled
temporary = self.path.with_suffix(".tmp")
fd = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
with os.fdopen(fd, "w") as stream:
json.dump({"enabled": enabled}, stream)
stream.flush()
os.fsync(stream.fileno())
os.replace(temporary, self.path)
if enabled:
self._ensure_db()
self.start()
else:
tasks = [task for task in (self.task, self.lag_task) if task]
for task in tasks:
if task:
task.cancel()
await asyncio.gather(*tasks, return_exceptions=True)
self.task = None
self.lag_task = None
self.previous = None
self.persist_requests()
async def measure_lag(self):
while True:
before = asyncio.get_running_loop().time()
await asyncio.sleep(1)
self.loop_lag_ms = round(max(0, asyncio.get_running_loop().time() - before - 1) * 1000, 2) if self.enabled else 0
async def loop(self):
while True:
if self.enabled:
try:
await self.sample()
except Exception as exc:
self.latest = {"at": time.time(), "error": f"采集失败（{type(exc).__name__}）"}
await asyncio.sleep(5)
def persist_requests(self):
batch = []
while self.recorder.pending:
row = self.recorder.pending.popleft()
batch.append((row["started"], json.dumps(row)))
if batch and self.db is not None:
self.db.executemany("INSERT INTO requests(at, body) VALUES (?, ?)", batch)
self.db.execute("DELETE FROM requests WHERE id <= (SELECT COALESCE(MAX(id),0)-10000 FROM requests)")
self.db.commit()
async def sample(self):
self._ensure_db()
settings = self.pool.store.load()
engine = DockerEngine(settings)
names = [settings.source_container, *list(self.pool.nodes)]
async def read_container(name):
try:
encoded = quote(name, safe="")
detail = await asyncio.to_thread(engine.request, "GET", f"/containers/{encoded}/json")
stats = await asyncio.to_thread(engine.request, "GET", f"/containers/{encoded}/stats?stream=false&one-shot=true")
key = detail.get("Id", name)
result = container_metrics(detail, stats, self.container_previous.get(key))
self.container_previous[key] = stats
return result
except Exception as exc:
return {"name": name, "error": f"节点指标不可用（{type(exc).__name__}）"}
async def host():
try:
raw = await asyncio.to_thread(engine.host_snapshot, settings.source_container)
result = rates(raw, self.previous)
self.previous = raw
return result
except Exception as exc:
detail = str(exc) if isinstance(exc, DockerError) else type(exc).__name__
return {"available": False, "message": f"宿主机采集不可用：{detail}。请检查只读挂载和 Docker 连接"}
host_data, containers, databases = await asyncio.gather(host(), asyncio.gather(*(read_container(n) for n in names if n)), self.database_metrics())
result = {"at": time.time(), "host": host_data, "containers": containers, "databases": databases, "event_loop_lag_ms": self.loop_lag_ms, "nodes": self.pool.status()["nodes"], "runtime_revision": self.runtime_revision}
self.latest = result
summary = {
"at": result["at"],
"cpu_percent": host_data.get("cpu_percent"),
"iowait_percent": host_data.get("iowait_percent"),
"memory_used_percent": round(100 * (1 - host_data["memory_available"] / host_data["memory_total"]), 2) if host_data.get("memory_total") else None,
"rx_bps": sum(v.get("rx_bps") or 0 for v in host_data.get("interfaces", {}).values()),
"tx_bps": sum(v.get("tx_bps") or 0 for v in host_data.get("interfaces", {}).values()),
"loop_lag_ms": self.loop_lag_ms,
"mysql_connections": databases.get("mysql", {}).get("connected"),
"postgres_connections": databases.get("postgres", {}).get("connected"),
"p95_headers_ms": self.recorder.summary()["p95_headers_ms"],
}
self.history.append(summary)
self.db.execute("INSERT INTO snapshots(at,body) VALUES (?,?)", (result["at"], json.dumps(summary)))
self.db.execute("DELETE FROM snapshots WHERE id <= (SELECT COALESCE(MAX(id),0)-720 FROM snapshots)")
self.db.commit()
self.persist_requests()
def status(self):
return {"enabled": self.enabled, "sample_interval_seconds": 5, "retained_samples": 720, "latest": self.latest, "history": list(self.history), "http": self.recorder.summary()}
async def database_metrics(self):
config = getattr(self.app.state, "config", None)
db = getattr(config, "database", None)
checkpoint = getattr(config, "checkpointer", None)
result = {}
if getattr(db, "backend", None) == "mysql":
result["mysql"] = await self.mysql_metrics(db.app_sqlalchemy_url)
else:
result["mysql"] = {"available": False, "reason": "业务库未使用 MySQL"}
from .preflight import postgres_connection_string
pg_url = postgres_connection_string(config)
if pg_url:
result["postgres"] = await self.postgres_metrics(pg_url)
else:
result["postgres"] = {"available": False, "configured": False, "reason": "当前聊天存储为 SQLite，未启用 PostgreSQL，不能显示为 0 个连接"}
result["chat_backend"] = "postgres" if pg_url else ("sqlite" if checkpoint and checkpoint.type in {"sqlite", "postgres"} else "unknown")
return result
async def mysql_metrics(self, url):
from sqlalchemy import text
from sqlalchemy.ext.asyncio import create_async_engine
key = "mysql"
try:
if key not in self.db_engines:
self.db_engines[key] = create_async_engine(url, pool_size=1, max_overflow=0, pool_timeout=1, pool_pre_ping=False, connect_args={"connect_timeout": 2})
async with asyncio.timeout(4):
async with self.db_engines[key].connect() as connection:
statuses = dict((await connection.execute(text("SHOW GLOBAL STATUS WHERE Variable_name IN ('Threads_connected','Threads_running','Max_used_connections','Slow_queries','Aborted_connects')"))).all())
variables = dict((await connection.execute(text("SHOW GLOBAL VARIABLES WHERE Variable_name IN ('max_connections','long_query_time')"))).all())
rows = (await connection.execute(text("SHOW PROCESSLIST"))).mappings().all()
running = [r for r in rows if str(r.get("Command", "")).lower() not in {"sleep", "daemon"}]
return {
"available": True,
"connected": int(statuses.get("Threads_connected", 0)),
"running": int(statuses.get("Threads_running", 0)),
"max_connections": int(variables.get("max_connections", 0)),
"peak_connections": int(statuses.get("Max_used_connections", 0)),
"slow_queries": int(statuses.get("Slow_queries", 0)),
"slow_query_threshold_seconds": float(variables.get("long_query_time", 0)),
"longest_active_seconds": max((int(r.get("Time", 0)) for r in running), default=0),
"processes_visible": len(rows),
"note": "连接总数为 MySQL 全局值（包含采集连接）；最长执行时间只覆盖当前账号可见进程，不读取 SQL 文本。",
}
except Exception as exc:
return {"available": False, "error": f"MySQL 指标不可用（{type(exc).__name__}），请检查连接与只读监控权限"}
async def postgres_metrics(self, url):
try:
import psycopg
async with asyncio.timeout(4):
async with await psycopg.AsyncConnection.connect(url, connect_timeout=2, autocommit=True) as connection:
async with connection.cursor() as cursor:
query = (
"SELECT count(*), count(*) FILTER (WHERE state='active'), "
"count(*) FILTER (WHERE wait_event_type='Lock'), "
"COALESCE(max(EXTRACT(EPOCH FROM now()-query_start)) "
"FILTER (WHERE state='active' AND pid<>pg_backend_pid()), 0) "
"FROM pg_stat_activity"
)
await cursor.execute(query)
connected, active, locks, longest = await cursor.fetchone()
await cursor.execute("SHOW max_connections")
maximum = (await cursor.fetchone())[0]
return {
"available": True,
"configured": True,
"connected": connected,
"active": active,
"lock_waits": locks,
"max_connections": int(maximum),
"longest_active_seconds": float(longest),
"note": "按 pg_stat_activity 采集，普通账号可能看不到其他账号的状态和耗时；包含采集连接，不读取 SQL 文本。",
}
except Exception as exc:
return {"available": False, "configured": True, "error": f"PostgreSQL 指标不可用（{type(exc).__name__}），请检查连接与监控权限"}
async def logs(self, name, tail=200, minutes=60, query=""):
if name.startswith("archive:"):
return self.read_archive(name[len("archive:") :], query)
settings = self.pool.store.load()
if name != settings.source_container and name not in self.pool.nodes:
raise DockerError("只能查看当前主服务和本 Worker 池的日志")
engine = DockerEngine(settings)
encoded = quote(name, safe="")
detail = await asyncio.to_thread(engine.request, "GET", f"/containers/{encoded}/json")
if name != settings.source_container and detail.get("Config", {}).get("Labels", {}).get("deerflow.worker-pool") != self.pool.store.raw()["instance"]:
raise DockerError("节点不属于当前 Worker 池")
params = urlencode({"stdout": 1, "stderr": 1, "timestamps": 1, "tail": min(1000, max(1, tail)), "since": int(time.time() - min(1440, max(1, minutes)) * 60)})
data = await asyncio.to_thread(engine.request, "GET", f"/containers/{encoded}/logs?{params}", raw=True, max_bytes=1024 * 1024)
lines = redact_logs(decode_docker_output(data)).splitlines()
if query:
lines = [line for line in lines if query.casefold() in line.casefold()]
return {"node": name, "lines": lines[-1000:], "log_policy": detail.get("HostConfig", {}).get("LogConfig", {}), "limited_bytes": 1024 * 1024}
async def archive_logs(self, engine, name):
"""Keep a bounded redacted tail BEFORE removing an owned container."""
import gzip
encoded = quote(name, safe="")
params = urlencode({"stdout": 1, "stderr": 1, "timestamps": 1, "tail": 10000})
data = await asyncio.to_thread(engine.request, "GET", f"/containers/{encoded}/logs?{params}", raw=True, max_bytes=2 * 1024 * 1024)
archive_id = f"{name}-{time.time_ns()}"
payload = {"node": name, "created_at": time.time(), "lines": redact_logs(decode_docker_output(data)).splitlines(), "limited_bytes": 2 * 1024 * 1024, "note": "节点删除前的最近 10000 行 / 最多 2 MB 归档；保存 7 天，最多 50 份。"}
self.archive_directory.mkdir(mode=0o700, parents=True, exist_ok=True)
target = self.archive_directory / f"{archive_id}.json.gz"
fd = os.open(target, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
with os.fdopen(fd, "wb") as out:
out.write(gzip.compress(json.dumps(payload).encode()))
files = sorted(self.archive_directory.glob("*.json.gz"), key=lambda p: p.stat().st_mtime, reverse=True)
for index, path in enumerate(files):
if index >= 50 or path.stat().st_mtime < time.time() - 7 * 86400:
path.unlink()
def archives(self):
return [
{"id": p.name.removesuffix(".json.gz"), "name": p.name.removesuffix(".json.gz"), "created_at": p.stat().st_mtime}
for p in sorted(self.archive_directory.glob("*.json.gz"), key=lambda p: p.stat().st_mtime, reverse=True)[:50]
if p.stat().st_mtime >= time.time() - 7 * 86400
]
def read_archive(self, archive_id, query=""):
import gzip
if not re.fullmatch(r"[A-Za-z0-9_.-]{1,180}", archive_id):
raise DockerError("无效日志归档")
path = self.archive_directory / f"{archive_id}.json.gz"
if not path.is_file() or path.stat().st_mtime < time.time() - 7 * 86400:
raise DockerError("日志归档不存在或已过期")
payload = json.loads(gzip.decompress(path.read_bytes()))
payload["lines"] = [line for line in payload["lines"] if query.casefold() in line.casefold()][-1000:]
return payload
offline-backend-20260512/backend/app/gateway/worker_pool/discovery.py
+60
View File
@@ -0,0 +1,60 @@
"""Read-only server discovery and conservative starter recommendations."""
import base64
import hashlib
import math
import socket
from .docker import DockerEngine, DockerError
def host_fingerprint(host: str, port: int) -> str:
import paramiko
transport = None
try:
with socket.create_connection((host, port), timeout=10) as connection:
transport = paramiko.Transport(connection)
transport.start_client(timeout=10)
key = transport.get_remote_server_key()
return "SHA256:" + base64.b64encode(hashlib.sha256(key.asbytes()).digest()).decode().rstrip("=")
except Exception as exc:
raise DockerError(f"读取服务器指纹失败（{type(exc).__name__}）") from None
finally:
if transport is not None:
transport.close()
def recommend(cpus, memory_mb, reserved_cpu=0, reserved_memory_mb=0, connection_budget=64):
# Reserve at least half the host for the main app, DB and other services.
cpu_budget = max(0, min(cpus / 2, cpus - reserved_cpu - 1))
memory_budget = max(0, min(memory_mb // 2, memory_mb - reserved_memory_mb - 2048))
cpu = 2 if cpu_budget >= 2 else 1
memory = 2048
count = max(0, min(4, math.floor(cpu_budget / cpu), memory_budget // memory, connection_budget // 12))
return {"desired": int(count), "cpu": cpu, "memory_mb": memory,
"total_cpu": max(cpu, count * cpu), "total_memory_mb": max(memory, count * memory),
"concurrency": 4, "database_pool_size": 4, "postgres_pool_size": 4}
def discover(settings, instance):
engine = DockerEngine(settings)
info = engine.request("GET", "/info")
containers = engine.request("GET", "/containers/json")
cpu_reserved = 0
memory_reserved = 0
templates = []
for item in containers:
if item.get("Labels", {}).get("deerflow.worker-pool") == instance:
continue
detail = engine.request("GET", f"/containers/{item['Id']}/json")
limits = detail.get("HostConfig", {})
cpu_reserved += limits.get("NanoCpus", 0) / 1_000_000_000
memory_reserved += limits.get("Memory", 0) // (1024 ** 2)
if item.get("Labels", {}).get("deerflow.worker-role") == "online":
continue
networks = [name for name in detail.get("NetworkSettings", {}).get("Networks", {}) if name not in {"host", "none", "bridge"}]
templates.append({"name": detail.get("Name", "").lstrip("/"), "networks": networks})
cpus = int(info.get("NCPU", 0))
memory_mb = int(info.get("MemTotal", 0)) // (1024 ** 2)
return {"cpus": cpus, "memory_mb": memory_mb, "containers": templates,
"recommendation": recommend(cpus, memory_mb, cpu_reserved, memory_reserved, settings.database_connection_budget),
"note": "按硬件总量和其他容器已配置的资源上限估算，至少为主服务、数据库及系统保留一半资源；不是实时空闲内存。建议先用最多 4 个节点，再按实际负载调整。"}
offline-backend-20260512/backend/app/gateway/worker_pool/docker.py
+505
View File
@@ -0,0 +1,505 @@
"""Docker Engine API over a Unix socket or a pinned SSH connection.
Only the constant `docker system dial-stdio` command runs on the server.
All variable data travels as JSON, never interpolated into a shell command.
"""
from __future__ import annotations
import base64
import hashlib
import http.client
import json
import posixpath
import socket
from urllib.parse import quote
from .settings import PoolSettings
class DockerError(RuntimeError):
pass
_WORKER_STORAGE_SELECTION = """\
import os
from pathlib import Path
import yaml
config_path = Path(os.getenv('DEER_FLOW_CONFIG_PATH') or 'config.yaml')
config = yaml.safe_load(config_path.read_text(encoding='utf-8')) or {}
checkpoint = config.get('checkpointer')
if checkpoint is not None:
checkpoint_type = checkpoint.get('type')
if checkpoint_type == 'postgres':
raw = os.getenv('DEER_FLOW_USE_PG_CHECKPOINTER')
if raw:
postgres_enabled = raw.strip().lower() not in {'0', 'false', 'no', 'off'}
else:
postgres_enabled = bool(checkpoint.get('use_postgres', True))
backend = 'postgres' if postgres_enabled else 'sqlite'
elif checkpoint_type == 'sqlite':
backend = 'sqlite'
else:
backend = 'memory'
else:
backend = (config.get('database') or {}).get('backend', 'memory')
if backend not in {'postgres', 'sqlite'}:
backend = 'memory'
"""
_WORKER_STORAGE_PROBE = _WORKER_STORAGE_SELECTION + """\
print('__DEERFLOW_STORAGE_BACKEND__=' + backend)
"""
_WORKER_PYTHON_PROBE = _WORKER_STORAGE_SELECTION + """\
import importlib.util
import os
import sys
if sys.version_info < (3, 12):
raise SystemExit(78)
required = ['uvicorn']
if backend == 'postgres':
required.extend((
'langgraph.checkpoint.postgres.aio',
'langgraph.store.postgres.aio',
'psycopg.rows',
'psycopg_pool',
))
elif backend == 'sqlite':
required.extend((
'langgraph.checkpoint.sqlite.aio',
'langgraph.store.sqlite.aio',
'aiosqlite',
))
missing = []
for module in required:
try:
if importlib.util.find_spec(module) is None:
missing.append(module)
except (ImportError, ModuleNotFoundError, ValueError):
missing.append(module)
if not Path('app/gateway/worker_pool/worker.py').is_file():
missing.append('app.gateway.worker_pool.worker')
if missing:
packages = (
'langgraph-checkpoint-postgres psycopg[binary] psycopg-pool'
if backend == 'postgres'
else 'langgraph-checkpoint-sqlite aiosqlite'
)
print('__DEERFLOW_STORAGE_ERROR__=Worker Python is missing configured runtime modules: '
+ ', '.join(missing) + '; install ' + packages, flush=True)
raise SystemExit(78)
print('__DEERFLOW_WORKER_PYTHON__=' + os.path.abspath(sys.executable))
"""
_WORKER_PYTHON_PROBE_SHELL = r'''
probe() {
candidate="$1"
[ -n "$candidate" ] || return 1
output=$("$candidate" -c "$2" 2>/dev/null) || {
case "$output" in
*__DEERFLOW_STORAGE_ERROR__=*)
error=${output##*__DEERFLOW_STORAGE_ERROR__=}
printf '__DEERFLOW_STORAGE_ERROR__=%s\n' "$error"
;;
esac
return 1
}
case "$output" in
*__DEERFLOW_WORKER_PYTHON__=*)
resolved=${output##*__DEERFLOW_WORKER_PYTHON__=}
case "$resolved" in /*) printf '%s\n' "$resolved"; return 0 ;; esac
;;
esac
return 1
}
code="$1"
shift
for candidate in "$@"; do
if probe "$candidate" "$code"; then exit 0; fi
done
# The main API may use a Python executable outside the image's default PATH.
# Probe the interpreter actually running in this source container before PATH
# fallbacks, using the same configured persistence-adapter check above.
for process_exe in /proc/[0-9]*/exe; do
candidate=$(readlink "$process_exe" 2>/dev/null) || continue
case "$candidate" in
*/python|*/python[0-9]|*/python[0-9].* )
case "$candidate" in *" (deleted)"*) continue ;; esac
if probe "$candidate" "$code"; then exit 0; fi
;;
esac
done
for candidate in python3 python3.14 python3.13 python3.12 python3.11 python; do
if probe "$candidate" "$code"; then exit 0; fi
done
exit 1
'''
class DockerEngine:
def __init__(self, settings: PoolSettings):
self.settings = settings
def request(self, method: str, path: str, body: dict | None = None, *, raw=False, max_bytes=None, json_stream=False):
client = None
channel = None
# Committing a large production backend layer can take longer than
# the ordinary metadata/exec requests. Do not turn a slow but valid
# runtime snapshot into a permanent Worker reconciliation failure.
request_timeout = 300 if method == "POST" and path.startswith("/commit?") else 20
try:
if self.settings.transport == "socket":
channel = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
channel.settimeout(request_timeout)
channel.connect("/var/run/docker.sock")
else:
import paramiko
expected = self.settings.host_key_sha256
if not expected.startswith("SHA256:"):
raise DockerError("请先填写服务器 SSH 主机密钥指纹（SHA256:…）")
class PinnedKey(paramiko.MissingHostKeyPolicy):
def missing_host_key(self, ssh, hostname, key):
actual = "SHA256:" + base64.b64encode(hashlib.sha256(key.asbytes()).digest()).decode().rstrip("=")
if actual != expected:
raise DockerError("SSH 主机密钥指纹不匹配，连接已拒绝")
client = paramiko.SSHClient()
client.set_missing_host_key_policy(PinnedKey())
client.connect(
self.settings.host,
port=self.settings.port,
username=self.settings.username,
password=self.settings.password,
timeout=10,
auth_timeout=10,
banner_timeout=10,
look_for_keys=False,
allow_agent=False,
)
channel = client.get_transport().open_session(timeout=20)
channel.settimeout(request_timeout)
channel.exec_command("docker system dial-stdio")
data = json.dumps(body).encode() if body is not None else b""
request = f"{method} {path} HTTP/1.1\r\nHost: docker\r\nContent-Type: application/json\r\nContent-Length: {len(data)}\r\nConnection: close\r\n\r\n".encode() + data
channel.sendall(request)
response = http.client.HTTPResponse(channel)
response.begin()
if json_stream and response.status < 400:
# Exec uses a hijacked connection that may outlive the process.
# Stop on a complete bounded JSON result instead of waiting for EOF.
result = b""
while len(result) < (max_bytes or 1024 * 1024):
chunk = response.read1(min(65536, (max_bytes or 1024 * 1024) - len(result)))
if not chunk:
break
result += chunk
try:
json.loads(decode_docker_output(result))
break
except ValueError:
pass
else:
result = response.read(max_bytes) if max_bytes else response.read()
if response.status >= 400:
# Docker errors can contain environment values / host paths.
raise DockerError(f"Docker 操作失败（HTTP {response.status}），请检查容器、网络和账号权限")
return result if raw else (json.loads(result) if result else {})
except DockerError:
raise
except Exception as exc:
raise DockerError(f"服务器连接失败（{type(exc).__name__}），请检查地址、账号、密码和 Docker 权限") from None
finally:
if channel is not None:
channel.close()
if client is not None:
client.close()
def _exec_capture(self, container_id: str, command: list[str], working_dir: str, user: str) -> tuple[int, str]:
created = self.request("POST", f"/containers/{quote(container_id, safe='')}/exec", {
"AttachStdout": True,
"AttachStderr": True,
"Cmd": command,
"WorkingDir": working_dir,
"User": user,
})
exec_id = quote(created["Id"], safe="")
output = self.request(
"POST", f"/exec/{exec_id}/start", {"Detach": False, "Tty": False},
raw=True, max_bytes=1024 * 1024,
)
inspected = self.request("GET", f"/exec/{exec_id}/json")
exit_code = inspected.get("ExitCode")
return (int(exit_code) if exit_code is not None else 1), decode_docker_output(output)
@staticmethod
def _python_candidates(source: dict, preferred: str) -> list[str]:
candidates = [
preferred,
"/opt/venv/bin/python", "/opt/venv/bin/python3",
"/app/.venv/bin/python", "/app/.venv/bin/python3",
"/app/backend/.venv/bin/python", "/venv/bin/python", "/venv/bin/python3",
"/usr/local/bin/python", "/usr/local/bin/python3",
"/usr/local/bin/python3.14", "/usr/local/bin/python3.13",
"/usr/local/bin/python3.12", "/usr/local/bin/python3.11",
"/usr/bin/python", "/usr/bin/python3", "/usr/bin/python3.12",
]
env = source.get("Config", {}).get("Env", [])
path_value = next((item[5:] for item in env if item.startswith("PATH=")), "")
for directory in path_value.split(":"):
if directory.startswith("/"):
candidates.extend(f"{directory.rstrip('/')}/{name}" for name in ("python3", "python"))
return list(dict.fromkeys(path for path in candidates if path and path.startswith("/")))
def discover_worker_python(self, source: dict, preferred: str, working_dir: str) -> str:
"""Find an interpreter inside the source image that can actually start a Worker."""
container_id = source.get("Id")
if not container_id:
raise DockerError("无法读取主后端容器 ID，不能检测镜像内 Python")
config = source.get("Config", {})
user = config.get("User", "")
candidates = self._python_candidates(source, preferred)
storage_error = None
try:
code, output = self._exec_capture(
container_id,
["/bin/sh", "-c", _WORKER_PYTHON_PROBE_SHELL, "worker-python-probe", _WORKER_PYTHON_PROBE, *candidates],
working_dir,
user,
)
if code == 0:
for line in reversed(output.splitlines()):
if line.startswith("/") and "\x00" not in line:
return line.strip()
for line in output.splitlines():
prefix = "__DEERFLOW_STORAGE_ERROR__="
if line.startswith(prefix):
storage_error = line[len(prefix):].strip()
except DockerError:
# Minimal images may not include a shell; try known Python paths directly.
pass
for candidate in candidates:
try:
code, output = self._exec_capture(
container_id,
[candidate, "-c", _WORKER_PYTHON_PROBE],
working_dir,
user,
)
except DockerError:
continue
if code == 0:
for line in reversed(output.splitlines()):
prefix = "__DEERFLOW_WORKER_PYTHON__="
if line.startswith(prefix):
resolved = line[len(prefix):].strip()
if resolved.startswith("/") and "\x00" not in resolved:
return resolved
for line in output.splitlines():
prefix = "__DEERFLOW_STORAGE_ERROR__="
if line.startswith(prefix):
storage_error = line[len(prefix):].strip()
if storage_error:
raise DockerError(storage_error)
raise DockerError(
"未找到与主服务聊天存储配置兼容的 Worker Python（要求 Python 3.12+，"
"且能导入 uvicorn、Worker 入口及当前 SQLite/PostgreSQL 适配器）；"
"已检查主服务常见 Python 路径和运行进程解释器，请确认 Worker 复用了主服务的镜像、"
"环境变量、挂载目录和 Python 环境"
)
def configured_storage_backend(self, source: dict, preferred: str, working_dir: str) -> str:
"""Read the live config so SQLite→PostgreSQL changes invalidate probes."""
container_id = source.get("Id")
if not container_id:
raise DockerError("无法读取主后端容器 ID，不能检查聊天存储配置")
user = source.get("Config", {}).get("User", "")
for candidate in self._python_candidates(source, preferred):
try:
code, output = self._exec_capture(
container_id,
[candidate, "-c", _WORKER_STORAGE_PROBE],
working_dir,
user,
)
except DockerError:
continue
if code == 0:
for line in reversed(output.splitlines()):
prefix = "__DEERFLOW_STORAGE_BACKEND__="
if line.startswith(prefix):
backend = line[len(prefix):].strip()
if backend in {"postgres", "sqlite", "memory"}:
return backend
raise DockerError(
"无法读取主服务当前聊天存储配置；请确认 Python 路径、工作目录和主服务配置文件正确"
)
def snapshot_worker_runtime(self, source: dict, instance: str) -> str:
"""Snapshot the running main container's writable layer for new workers.
Docker create from Config.Image alone loses packages installed into the
running container after it was created. A local commit carries those
offline-installed packages; bind/volume mounts are still cloned as-is.
"""
container_id = source.get("Id")
if not container_id:
raise DockerError("无法读取主后端容器 ID，不能复用其 Python 运行环境")
# A freshly started backend often has only a tiny writable overlay
# (logs/runtime markers). Committing it can take minutes on an offline
# Docker host even though the immutable image already has the runtime.
# Keep committing larger overlays that may include offline-installed
# Python packages.
base_image = source.get("Image")
writable_layer_size = source.get("SizeRw")
if (
isinstance(base_image, str)
and base_image
and isinstance(writable_layer_size, int)
and writable_layer_size <= 128 * 1024
):
return base_image
repo = f"deerflow-worker-runtime-{instance}"
tag = container_id[:12]
result = self.request(
"POST",
f"/commit?container={quote(container_id, safe='')}&repo={quote(repo, safe='')}&tag={quote(tag, safe='')}&pause=false",
)
image_id = result.get("Id") if isinstance(result, dict) else None
if not isinstance(image_id, str) or not image_id.startswith("sha256:"):
raise DockerError("无法从主容器生成 Worker 运行镜像；请确认 Docker 账号具有容器提交权限")
return image_id
def host_snapshot(self, container: str):
"""Only our constant read-only collector may be executed by diagnostics."""
if self.settings.transport == "socket":
from .host_metrics import read_snapshot
return read_snapshot()
command = [self.settings.python_executable, "-m", "app.gateway.worker_pool.host_metrics"]
created = self.request("POST", f"/containers/{quote(container, safe='')}/exec", {
"AttachStdout": True, "AttachStderr": True, "Cmd": command,
"WorkingDir": self.settings.working_dir,
})
data = self.request("POST", f"/exec/{created['Id']}/start", {"Detach": False, "Tty": False}, raw=True, max_bytes=1024 * 1024, json_stream=True)
out = decode_docker_output(data)
try:
return json.loads(out)
except ValueError:
raise DockerError("宿主机采集器未运行成功，请检查主容器代码和只读 /proc、/sys 挂载") from None
def decode_docker_output(data: bytes) -> str:
"""Docker attaches stdout/stderr in 8-byte length-prefixed frames."""
import struct
if len(data) < 8 or data[0] not in (0, 1, 2) or data[1:4] != b"\x00\x00\x00":
return data.decode("utf-8", errors="replace")
parts = []
offset = 0
while offset + 8 <= len(data):
size = struct.unpack(">I", data[offset + 4:offset + 8])[0]
parts.append(data[offset + 8:offset + 8 + size])
offset += 8 + size
return b"".join(parts).decode("utf-8", errors="replace")
def worker_spec(
source: dict,
settings: PoolSettings,
instance: str,
token: str,
control_home: str,
*,
runtime_image: str | None = None,
require_shared_archive: bool = False,
) -> dict:
"""Clone an explicit runtime template, without ports, privileges or Docker access."""
if settings.network not in source["NetworkSettings"]["Networks"]:
raise DockerError("源容器必须连接所选 Docker 网络")
if settings.network in {"host", "none", "bridge"}:
raise DockerError("请选择用户自定义 Docker 网络")
mounts = []
for mount in source.get("Mounts", []):
target = mount["Destination"].rstrip("/")
if control_home == target:
continue # dedicated controller volume is never passed to workers
if "docker.sock" in target or target in {"/", "/var/run", "/run"}:
continue
if target in {"/host/proc", "/host/sys"}:
continue # host diagnostics belong to the controller only
if control_home.startswith(target + "/"):
raise DockerError("控制器密钥目录不能与 Worker 共享，请为 DEER_FLOW_WORKER_CONTROL_HOME 配置独立持久卷")
if mount["Type"] not in {"bind", "volume"}:
continue
mounts.append({"Type": mount["Type"], "Source": mount.get("Name") if mount["Type"] == "volume" else mount["Source"], "Target": mount["Destination"], "ReadOnly": not mount.get("RW", True)})
env = dict(item.split("=", 1) for item in source["Config"].get("Env", []) if "=" in item)
if not env.get("AUTH_JWT_SECRET"):
raise DockerError("请在源容器环境变量中设置固定 AUTH_JWT_SECRET，使 Worker 能验证现有登录会话")
if not mounts:
raise DockerError("源容器没有共享持久化挂载，无法保证 Worker 访问技能、配置和上传文件")
if require_shared_archive:
if env.get("DEER_FLOW_CHAT_ARCHIVE_DIR"):
archive_path = posixpath.normpath(env["DEER_FLOW_CHAT_ARCHIVE_DIR"])
elif env.get("DEER_FLOW_HOME"):
archive_path = posixpath.join(env["DEER_FLOW_HOME"], "data", "chat-archives")
else:
project_root = env.get("DEER_FLOW_PROJECT_ROOT") or source["Config"].get("WorkingDir") or settings.working_dir
archive_path = posixpath.join(project_root, ".deer-flow", "data", "chat-archives")
if not posixpath.isabs(archive_path):
workdir = source["Config"].get("WorkingDir") or settings.working_dir
archive_path = posixpath.join(workdir, archive_path)
archive_path = posixpath.normpath(archive_path)
shared = any(
mount.get("ReadOnly") is False
and (archive_path == mount["Target"].rstrip("/")
or archive_path.startswith(mount["Target"].rstrip("/") + "/"))
for mount in mounts
)
if not shared:
raise DockerError(
f"PostgreSQL 聊天归档目录 {archive_path} 必须位于主容器与 Worker 共享的可写持久卷中；"
"请将 DEER_FLOW_CHAT_ARCHIVE_DIR 指向共享卷目录或挂载 DEER_FLOW_HOME/data"
)
for key in list(env):
if key.startswith("DEER_FLOW_WORKER_") or key in {"DEER_FLOW_REBUILD_WORKFLOW_SCHEMA", "WEB_CONCURRENCY", "UVICORN_WORKERS"}:
env.pop(key)
env.update(
DEER_FLOW_WORKER_DB_POOL_SIZE=str(settings.database_pool_size),
DEER_FLOW_PG_POOL_MIN="1",
DEER_FLOW_PG_POOL_MAX=str(settings.postgres_pool_size),
DEER_FLOW_WORKER_ROLE="online",
DEER_FLOW_WORKER_TOKEN=token,
DEER_FLOW_WORKER_CONCURRENCY=str(settings.concurrency),
DEER_FLOW_WORKER_QUEUE_LIMIT=str(settings.queue_limit),
DEER_FLOW_WORKER_QUEUE_TIMEOUT=str(settings.queue_timeout),
SCHEDULED_TASKS_ENABLED="false",
TASK_ANALYSIS_ENABLED="false",
)
return {
"Image": runtime_image or source["Image"],
"WorkingDir": settings.working_dir,
"User": source["Config"].get("User", ""),
"Entrypoint": [settings.python_executable, "-m", "uvicorn"],
"Cmd": ["app.gateway.worker_pool.worker:app", "--host", "0.0.0.0", "--port", "8001", "--workers", "1"],
"Env": [f"{key}={value}" for key, value in env.items()],
"Labels": {"deerflow.worker-pool": instance, "deerflow.worker-role": "online"},
"NetworkingConfig": {"EndpointsConfig": {name: {} for name in source["NetworkSettings"]["Networks"] if name not in {"host", "none", "bridge"}}},
"Healthcheck": {
"Test": ["CMD", settings.python_executable, "-c", "import os,urllib.request; urllib.request.urlopen(urllib.request.Request('http://127.0.0.1:8001/internal/worker-status',headers={'x-deerflow-worker-token':os.environ['DEER_FLOW_WORKER_TOKEN']}),timeout=4).read()"],
"Interval": 20_000_000_000, "Timeout": 5_000_000_000, "Retries": 6, "StartPeriod": 90_000_000_000,
},
"HostConfig": {
"Mounts": mounts,
"NetworkMode": settings.network,
"Memory": settings.memory_mb * 1024 * 1024,
"NanoCpus": int(settings.cpu * 1_000_000_000),
"PidsLimit": 512,
"RestartPolicy": {"Name": "unless-stopped"},
"SecurityOpt": ["no-new-privileges:true"],
"ExtraHosts": source.get("HostConfig", {}).get("ExtraHosts") or [],
"Dns": source.get("HostConfig", {}).get("Dns") or [],
"Init": True,
"LogConfig": {"Type": "json-file", "Config": {"max-size": "10m", "max-file": "3"}},
},
}
offline-backend-20260512/backend/app/gateway/worker_pool/host_metrics.py
+110
View File
@@ -0,0 +1,110 @@
"""Small read-only host snapshot; requires /proc and /sys mounted by the admin."""
import json
import os
import re
import time
from pathlib import Path
def read_snapshot(proc=Path("/host/proc"), sys=Path("/host/sys")):
if not (proc / "stat").is_file():
return {"available": False, "message": "请为主容器挂载 /proc:/host/proc:ro 和 /sys:/host/sys:ro；Worker 不需要挂载。"}
cpu = next(line for line in (proc / "stat").read_text().splitlines() if line.startswith("cpu "))
fields = [int(v) for v in cpu.split()[1:9]] # guest times are already included in user/nice
mem = {}
for line in (proc / "meminfo").read_text().splitlines():
k, value = line.split(":", 1)
mem[k] = int(value.split()[0]) * 1024
disks = {}
for line in (proc / "diskstats").read_text().splitlines():
v = line.split()
if len(v) < 14 or re.match(r"^(loop|ram|fd|sr)", v[2]):
continue
# Whole devices only: including partitions would double-count disk IO.
if (sys / "class/block" / v[2] / "partition").exists():
continue
disks[v[2]] = {"read_bytes": int(v[5]) * 512, "write_bytes": int(v[9]) * 512,
"io_ms": int(v[12]), "inflight": int(v[11])}
interfaces = {}
network_error = None
try:
# /proc/1 is the host PID namespace; /proc/net would resolve to this container.
for line in (proc / "1/net/dev").read_text().splitlines()[2:]:
name, values = line.split(":", 1)
name = name.strip()
if name == "lo" or name.startswith(("veth", "docker", "br-", "virbr", "tun", "tap")):
continue
v = values.split()
speed = None
try:
speed = int((sys / "class/net" / name / "speed").read_text())
except (OSError, ValueError):
pass
interfaces[name] = {"rx_bytes": int(v[0]), "tx_bytes": int(v[8]),
"rx_errors": int(v[2]), "tx_errors": int(v[10]), "link_mbps": speed if speed and speed > 0 else None}
except OSError:
network_error = "无法读取宿主机网络命名空间，检查 /host/proc 挂载及读取权限"
pressure = {}
for name in ("cpu", "memory", "io"):
try:
pressure[name] = (proc / "pressure" / name).read_text().strip()[:400]
except OSError:
pressure[name] = None
processes = {}
page_size = getattr(os, "sysconf", lambda _: 4096)("SC_PAGE_SIZE")
for item in proc.iterdir():
if not item.name.isdigit():
continue
if len(processes) >= 5000:
break
try:
stat = (item / "stat").read_text()
end = stat.rfind(")")
process_fields = stat[end + 2:].split()
processes[item.name] = {"name": stat[stat.find("(") + 1:end][:80], "ticks": int(process_fields[11]) + int(process_fields[12]),
"started": int(process_fields[19]), "rss_bytes": max(0, int(process_fields[21])) * page_size}
except (OSError, ValueError, IndexError):
continue
return {"available": True, "at": time.time(), "cpu_total": sum(fields), "cpu_idle": fields[3] + fields[4],
"cpu_iowait": fields[4], "cpu_count": sum(line.startswith("cpu") and line[3:4].isdigit() for line in (proc / "stat").read_text().splitlines()),
"memory_total": mem.get("MemTotal", 0), "memory_available": mem.get("MemAvailable", mem.get("MemFree", 0)),
"swap_total": mem.get("SwapTotal", 0), "swap_free": mem.get("SwapFree", 0),
"load": [float(v) for v in (proc / "loadavg").read_text().split()[:3]],
"disks": disks, "interfaces": interfaces, "network_error": network_error, "pressure": pressure,
"process_counters": processes, "clock_ticks": getattr(os, "sysconf", lambda _: 100)("SC_CLK_TCK")}
def rates(current, previous):
result = dict(current)
if not current.get("available"):
return result
result.update(cpu_percent=None, iowait_percent=None)
elapsed = current["at"] - previous["at"] if previous and previous.get("available") else 0
if elapsed > 0:
delta = current["cpu_total"] - previous["cpu_total"]
if delta > 0:
result["cpu_percent"] = round(100 * (1 - (current["cpu_idle"] - previous["cpu_idle"]) / delta), 2)
result["iowait_percent"] = round(100 * (current["cpu_iowait"] - previous["cpu_iowait"]) / delta, 2)
top = []
for pid, item in current.get("process_counters", {}).items():
old = previous.get("process_counters", {}).get(pid) if elapsed > 0 else None
if old and old["started"] == item["started"]:
percent = max(0, (item["ticks"] - old["ticks"]) / (current.get("clock_ticks", 100) * elapsed) * 100)
top.append({"pid": int(pid), "name": item["name"], "cpu_percent": round(percent, 2), "rss_bytes": item["rss_bytes"]})
result.pop("process_counters", None)
result["top_processes"] = sorted(top, key=lambda p: p["cpu_percent"], reverse=True)[:20]
for name, item in result["disks"].items():
old = previous.get("disks", {}).get(name) if elapsed > 0 else None
item.update(read_bps=max(0, (item["read_bytes"] - old["read_bytes"]) / elapsed) if old else None,
write_bps=max(0, (item["write_bytes"] - old["write_bytes"]) / elapsed) if old else None,
busy_percent=min(100, max(0, (item["io_ms"] - old["io_ms"]) / (elapsed * 10))) if old else None)
for name, item in result["interfaces"].items():
old = previous.get("interfaces", {}).get(name) if elapsed > 0 else None
item.update(rx_bps=max(0, (item["rx_bytes"] - old["rx_bytes"]) / elapsed) if old else None,
tx_bps=max(0, (item["tx_bytes"] - old["tx_bytes"]) / elapsed) if old else None)
return result
if __name__ == "__main__":
print(json.dumps(read_snapshot()))
offline-backend-20260512/backend/app/gateway/worker_pool/lifecycle.py
+51
View File
@@ -0,0 +1,51 @@
"""Create the opt-in Worker controller without restricting normal startup."""
import json
import os
from contextlib import asynccontextmanager
from pathlib import Path
def _control_directory() -> Path:
return Path(os.getenv("DEER_FLOW_WORKER_CONTROL_HOME", ".worker-pool")).resolve()
def _persisted_feature_enabled(directory: Path) -> bool:
def read(path):
try:
return json.loads(path.read_text(encoding="utf-8"))
except (OSError, ValueError):
return {}
state = read(directory / "settings.json")
diagnostics = read(directory / "diagnostics-settings.json")
return bool(state.get("settings", {}).get("enabled") or diagnostics.get("enabled"))
def ensure_controller(app):
pool = getattr(app.state, "worker_pool", None)
if pool is not None:
return pool
from .controller import Controller
from .diagnostics import Diagnostics
from .settings import SettingsStore
directory = getattr(app.state, "worker_pool_directory", _control_directory())
pool = Controller(SettingsStore(directory))
pool.diagnostics = Diagnostics(pool, app)
app.state.worker_pool = pool
return pool
@asynccontextmanager
async def managed_controller(app):
directory = _control_directory()
app.state.worker_pool_directory = directory
app.state.worker_pool = None
pool = ensure_controller(app) if _persisted_feature_enabled(directory) else None
try:
yield pool
finally:
pool = getattr(app.state, "worker_pool", None)
if pool is not None:
await pool.close()
Loaded 100 of 873 files, more files were not shown because too many files have changed in this diff. Show more