wsk/deerflow-code
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/master' into xuhd

Browse Source 
# Conflicts:
#	offline-backend-20260512/backend/config.yaml
This commit is contained in:
xuhd committed 2026-06-09 15:43:40 +08:00
2 parents 9adb0fee28 2082d20b88
commit bd0eceda61
267 files changed
+41209 -14541
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

offline-backend-20260512/.env
+4 -2
View File
@@ -17,7 +17,7 @@ SERPER_API_KEY=your-serper-api-key
# AUTH_JWT_SECRET must stay stable across restarts, or existing logins expire.
# GATEWAY_CORS_ORIGINS must include the exact frontend origins, separated by commas.
AUTH_JWT_SECRET=i3_BiBK44sYHiFIkChhDs46y57ZewJKO3diQDw01anI
GATEWAY_CORS_ORIGINS=http://localhost:5173,http://127.0.0.1:5173,http://localhost:5174,http://127.0.0.1:5174
GATEWAY_CORS_ORIGINS=http://localhost:5173,http://127.0.0.1:5173,http://localhost:5183,http://localhost:5174,http://127.0.0.1:5174,http://localhost:4200
GATEWAY_CORS_ALLOW_ALL=0
GATEWAY_ENABLE_DOCS=true
GATEWAY_WORKERS=2
@@ -59,4 +59,6 @@ ELEMENT_STATE_FILE=/app/backend/.deer-flow/element-bot-token
# PYTHON_EXTRA_INDEX_URL=
# PYTHON_TRUSTED_HOST=your-python-mirror
MYSQL_DATABASE_URL=mysql+asyncmy://deerflow:deerflow123@192.168.0.123:3306/deerflow?charset=utf8mb4
DEER_FLOW_NOTEBOOK_API_BASE=http://127.0.0.1:8003
DEER_FLOW_NOTEBOOK_API_BASE=http://127.0.0.1:8003
# 圆桌后台作业网关：inprocess=进程内驱动真模型；simulated=模拟（默认）
ROUNDTABLE_JOB_GATEWAY=inprocess
offline-backend-20260512/backend/CLAUDE.md
+14
View File
@@ -245,6 +245,8 @@ FastAPI application on port 8001 with health check at `GET /health`. Set `GATEWA
| **Thread Shares** (`/api/thread-shares`) | Per-conversation sharing with two modes (`thread_shares.mode`). **`import`**: `POST /` `{thread_id, mode}` - create (or reuse) a revocable share code; `GET /{code}/preview` + `POST /{code}/import` - copy that conversation into the caller's account (fresh `thread_id` via `app/gateway/thread_copy.py` + new `threads_meta` row + `user-data` copy; dedup on `metadata.shared_origin`). **`view`**: a public, read-only static page — see Public Shares below. Both modes: `GET /` - list my codes; `DELETE /{code}` - revoke. |
| **Public Shares** (`/api/public/shares`) | **Unauthenticated** (the `/api/public/` prefix is whitelisted in `auth_middleware._PUBLIC_PATH_PREFIXES`). `GET /{code}` - read-only conversation snapshot (title + serialized messages + artifacts) for a `view`-mode share; messages are read straight from the latest checkpoint. `GET /{code}/files/{path}` - serves a file referenced by that conversation (images, Markdown, text, PDF, …) so the shared page's links are openable; path resolution is confined to the owner's thread directory. Active content (`text/html`, `application/xhtml+xml`, `image/svg+xml`) is forced to download (`_public_share_disposition`); everything else is served inline. Missing/revoked/non-`view` codes all return 404. |
| **Roundtable Drafts** (`/api/roundtable-drafts`) | Per-user persistence for the roundtable-planning page (replaces the old browser `localStorage`). `GET /` - list drafts (lightweight meta); `POST /` - create; `GET/PUT/DELETE /{id}` - full read / partial update / delete. A draft = one full session (Step 1 intent + Step 2 roundtable), step snapshots stored as JSON in `PortableLongText`. Recommendation history bound to a draft: `GET /{id}/recommendations` - list newest-first; `POST /{id}/recommendations` - append one run (objective/status/rationale/picks/candidates). Backed by `deerflow.persistence.roundtable_drafts` (tables `roundtable_drafts`, `roundtable_recommend_history`); store wired in `deps.py` as `app.state.roundtable_draft_store`. |
| **Roundtable Draft Shares** (`/api/roundtable-draft-shares`) | Per-draft sharing for the roundtable-planning page, mirroring Thread Shares but the shared resource is a `roundtable_drafts` row. Two modes (`roundtable_draft_shares.mode`). **`import`**: `POST /` `{draft_id, mode}` - create (or reuse) a revocable code; `GET /{code}/preview` + `POST /{code}/import` - copy that draft (step1/2/3) into the caller's account under a deterministic per-(code,recipient) id (`imp_<sha1>` → re-import is a no-op). **`view`**: a public, read-only result page — see Public Roundtable Shares. Both: `GET /` - list my codes; `DELETE /{code}` - revoke. Backed by `deerflow.persistence.roundtable_draft_shares` (table `roundtable_draft_shares`, migration `20260606_01`); store wired in `deps.py` as `app.state.roundtable_draft_share_store`. Tests: `tests/test_roundtable_draft_shares.py`. |
| **Public Roundtable Shares** (`/api/public/roundtable-shares`) | **Unauthenticated** (`/api/public/` whitelisted). `GET /{code}` - read-only snapshot for a `view`-mode draft share: returns the Step 3「结果绘制」report HTML (`step3.html`, a self-contained document) + title/summary/`has_report`. The frontend renders it in a sandboxed `iframe` (no `allow-scripts`) at `#/roundtable-share/<code>`. Missing/revoked/non-`view` codes all return 404. |
| **Scheduled Tasks** (`/api/scheduled-tasks`) | Per-user cron-like agent tasks. `GET/POST /` - list/create owned tasks; `PATCH/DELETE /{id}` - update/delete (owner); `POST /{id}/{pause,resume,trigger}`; `GET /{id}/runs`, `GET /runs/{rid}`, run-file preview/edit/download; `GET /runs/{rid}/messages` - agent messages for the run's underlying agent run (used to inspect a stuck/running run; authorized via `_load_run`); `GET /runs/{rid}/html` - generated HTML document for an `html_page` run; `DELETE /runs/{rid}` - owner-scoped, with an admin fallback so admins can delete any user's run; publish/subscribe under `/{id}/{publish,unpublish,subscribe}` and `GET /published`. **Admin (cross-user):** `GET /admin/all` - every task; `PATCH/DELETE /admin/{id}` - edit/delete any task; `POST /admin/{id}/trigger`; `GET /admin/{id}/runs`. Admin routes require `system_role == "admin"`. |
| **HTML Page Favorites** (`/api/html-page-favorites`) | Bookmarked generated HTML pages, owner-scoped. `GET /` - list; `POST /` `{page_id, title?, description?, tags?}` - bookmark a `scheduled_html_pages` row (extracts a reusable style/layout summary via GLM5 at favorite time); a page can only be favorited once per user (`DuplicateFavoriteError` → HTTP 409). `GET/DELETE /{id}` - read/delete (the create-task reference picker deletes favorites here). A favorite can be selected as a style reference when creating a new `html_page` task. |
| **Public Scheduled Tasks** (`/api/public/scheduled-tasks`) | **Unauthenticated** mirror for embedding. Adds `GET /runs/{rid}/html` and `GET /{task_id}/latest-html` for the `html_page` preview; the run result keeps `result.html` (only file paths are redacted). |
@@ -261,12 +263,22 @@ Proxied through nginx: `/api/langgraph/*` → LangGraph, all other `/api/*` →
**HTML page tasks (`task_kind == "html_page"`)**: a second task kind that renders a full HTML document each run instead of a Markdown report. The kind + its config (`html_config`: `page_prompt`, `negative_prompt`, `reference_favorite_id`, `model_name`) live inside `execution_context` — no schema change to `scheduled_tasks`. Data gathering is delegated to the agent's own search skills (driven by `page_prompt`), so there is no explicit search-query/data-source field; the render model is user-selectable (`model_name`, default `zai-org/GLM-5-FP8`). `_run_task_attempt()` dispatches by kind; `_run_html_page_attempt()` runs the pipeline: gather real data via the lead agent (its search tools) → optionally load a reference favorite's reusable style/layout → render with the chosen model (thinking off) → fault-tolerant JSON parse → sanitize (BeautifulSoup, strips `<script>`/`<iframe>`/`on*`/`javascript:`) → structural validate → one GLM5 repair pass if invalid → **completeness gate**: if the HTML is still not a full valid document (missing doctype/html/head/body, empty body, or residual scripts) after the repair pass, raise `HtmlGenerationError` so the run is marked **failed** (a page only counts as successful when it is a complete, previewable HTML document) → best-effort Playwright screenshot (no-op when Playwright absent) → archive `index.html` → persist a `scheduled_html_pages` row → return `result` with `format: "html"` and the full HTML inline. Pure helpers live in `deerflow/runtime/scheduler/html_page.py`. The frontend previews the HTML in a sandboxed `iframe` (no `allow-scripts`) on the run-detail and public pages; a user can bookmark a page (style summary extracted via GLM5) and later pick it as a style reference. New tables `scheduled_html_pages` + `html_page_favorites` (`deerflow/persistence/html_pages/`, wired as `app.state.html_page_store`) are created by `Base.metadata.create_all` with no migration. Tests: `tests/test_html_page_tasks.py`.
**Unified output switches (`require_html` + `require_markdown`).** The task form no longer splits "普通任务 / HTML 页面" into separate kinds. `execution_context` now carries two **independent** booleans that may both be on: `require_html` (必须生成 HTML) and `require_markdown` (必须生成 Markdown). `_task_requires_html()` returns true for `require_html` **or** legacy `task_kind == "html_page"` (so old tasks need no data migration), and `_run_task_attempt()` dispatches to `_run_html_page_attempt` when it is true, else the regular agent attempt. When **both** switches are on, `_gather_html_data()` returns `(text, files)` — the gather agent run is additionally told to save a `.md` file — and `_run_html_page_attempt` validates it (`MarkdownRequirementError` on failure) and surfaces it alongside the generated page, so the run delivers both artifacts. The single main `prompt` doubles as the HTML page requirement (`html_config.page_prompt`); there is no separate page-prompt field. Tests: `tests/test_scheduled_task_require_html.py` + the markdown-coexistence cases in `tests/test_html_page_tasks.py`.
### Per-user Prompts & UI Preferences
**Custom prompts can be scoped to one agent.** `user_custom_prompts` gains a nullable `agent_id` column (migration `20260603_02`): `NULL` = global (the chip shows in every chat), otherwise the prompt only surfaces when that agent is selected. The visibility filter is applied in the frontend `CustomPromptBar` (it receives the current agent id from `ChatPage`'s selector or `AgentChatPage`'s `assistantId`); the backend (`/api/user-prompts` CRUD) just stores/returns/clears `agent_id` (update accepts an explicit `null` to clear). Tests: `tests/test_user_prompts_agent_id.py`.
**Per-user UI preferences** (`deerflow/persistence/user_preferences/`, wired as `app.state.user_preferences_store`; migration `20260603_03`, also auto-created by `create_all`): one JSON-blob row per user so new toggles need no schema change. Router `/api/user-preferences` (`GET`/`PUT`, account-scoped). Current key: `hide_admin_prompt_prefix` — when true the chat input hides the admin-configured prompt-prefix switch and never applies the prefix (the user uses only their own prompts; enforced in `ChatPage` for both `shouldApplyPrefix` and `promptPrefixProp`). Toggled from the "管理常用提示词" dialog. Tests: `tests/test_user_preferences.py`.
**HTML skills (tools the render step uses).** Two offline skills under `skills/public/` shape the generated HTML and are toggled in the skills settings page (enabled by default): `html-page-builder` (a self-contained inline-CSS design system — colors/cards/tables/KPI/responsive) and `html-charts` (inline-SVG charts that render in the no-`allow-scripts` preview iframe; optional local ECharts documented in its `assets/README.md`). A skill opts into HTML injection via `metadata.kind: html_page` in its SKILL.md frontmatter; `_collect_html_skill_instructions()` reads the bodies of **enabled** such skills and `build_generation_prompt(skill_instructions=…)` injects them, so the (raw-model) render step honors the user's skill selection. The data-gathering step and the render/repair/style passes all use the task's selected `model_name` (no silent fallback to the default model — and there is **no hardcoded GLM-5**; an unset selection falls back to config `models[0]`). To stop auxiliary LLM calls from leaking onto the default model, **`TitleMiddleware` is skipped when `is_scheduled_run` is set** (`_build_middlewares` in `lead_agent/agent.py`) — scheduled/HTML runs use throwaway threads whose titles are never shown, and `config.title.model_name` is usually unset (→ would call `models[0]`, not the chosen model). The completeness gate (`validate_html`) also requires a non-empty `<title>` and forbids external stylesheets, on top of the doctype/html/head/body + non-empty-body + no-script checks.
### Custom Agent Persistence
Custom agent metadata lives in the `agents` ORM table (`deerflow.persistence.agents`). `agents.id` is the stable runtime id and the filesystem directory name under `.deer-flow/agents/{id}/`; `agents.name` is display-only, can contain Chinese characters, and is not unique. `user_id IS NULL` marks built-in agents, user-owned agents are visible only to their owner unless `published = true`, and update/delete operations require ownership. The runtime accepts `agent_id` in context/configurable, and non-default `assistant_id` values are treated as agent ids after visibility checks.
**Listing cache + pagination** — to keep `GET /api/agents` fast, the gallery's per-card extras (skills + tool_groups) are denormalized into two cache tables instead of reading every agent's `config.yaml` on each request: `agent_skills` (relational, one row per (agent, skill) with `position`) and `agent_extras` (one row per agent: `tool_groups` JSON + `model` + `skills_synced_at`). `config.yaml` remains authoritative for the agent runtime; these tables are a read cache, write-through on create/update and backfilled from disk by `_sync_legacy_agents` (mtime-vs-`skills_synced_at` drift check). The list endpoint enriches via `AgentStore.get_extras_for([ids])` (two batched `IN` queries). Both tables are auto-created by `Base.metadata.create_all` (no migration needed). `GET /api/agents` supports **opt-in server-side pagination**: `scope` (`mine` = owned + favorited / `square` = built-in/published public), `page`, `page_size`, returning `{agents, total, page, page_size}`; omitting `scope`/`page` preserves the legacy "return everything" behavior. Name `search` matches the raw OR desensitized name (the frontend's static term→abbrev map is mirrored server-side in `deerflow/config/agent_name_desensitize.py`); `tag_ids` is an SQL OR-filter. `POST /api/agents/private-square/list` takes the same pagination params. Repository entry point: `AgentStore.list_paginated(...)`.
**Homepage selector (agent picker on the new-chat page)** — single admin-curated list backed by `agents.featured_order` (nullable BIGINT). NULL = not featured. Only `published = true` agents can be featured (router enforces on writes); any published agent is eligible regardless of ownership. Cleared with `order: null`. The list rendered to end users is `WHERE featured_order IS NOT NULL` + visibility-filter, sorted by `featured_order ASC`. Admins reorder by sending the desired ids to `PUT /api/agents/featured/order` (assigns indices 0..N-1). BIGINT is used rather than INT so the management UI can seed initial values from `Date.now()` (overflows INT on MySQL).
### Portable Column Types (`deerflow/persistence/types.py`)
@@ -509,6 +521,8 @@ Per-user runtime overrides (whitelisted fields) live at `{base_dir}/users/{user_
Both endpoints are whitelisted in `auth_middleware._PUBLIC_EXACT_PATHS`. The shared `_get_or_create_user_by_username()` helper (in `routers/auth.py`) backs both `/login/username` and `/login/token`.
- `log_system` - Admin-only external log-system entry (`deerflow/config/log_system_config.py`): `{enabled, url, label}`. Surfaced read-only by `GET /api/system-settings/log-system` (admin only); the frontend renders it as a "日志系统" item in the bottom-right settings menu (`workspace-nav-menu.tsx`) that opens `url` in a new tab. Disabled by default.
**`extensions_config.json`**:
- `mcpServers` - Map of server name → config (enabled, type, command, args, env, url, headers, oauth, description)
- `skills` - Map of skill name → state (enabled)
offline-backend-20260512/backend/_reqs.txt
+760
View File
@@ -0,0 +1,760 @@
# This file was autogenerated by uv via the following command:
# uv export --no-hashes --no-emit-project --no-dev -o _reqs.txt
-e ./packages/harness
# via deer-flow
agent-client-protocol==0.10.1
# via deerflow-harness
agent-sandbox==0.0.30
# via deerflow-harness
aiohappyeyeballs==2.6.2
# via aiohttp
aiohttp==3.14.1
# via
# aiohttp-retry
# dingtalk-stream
# firecrawl-py
# hindsight-client
# kubernetes
# wecom-aibot-python-sdk
aiohttp-retry==2.9.1
# via hindsight-client
aiosignal==1.4.0
# via aiohttp
aiosqlite==0.22.1
# via
# deerflow-harness
# langgraph-checkpoint-sqlite
alembic==1.18.4
# via deerflow-harness
annotated-doc==0.0.4
# via fastapi
annotated-types==0.7.0
# via pydantic
anthropic==0.107.1
# via langchain-anthropic
anyio==4.13.0
# via
# anthropic
# google-genai
# httpx
# mcp
# openai
# sse-starlette
# starlette
# watchfiles
asyncmy==0.2.11
# via deer-flow
attrs==26.1.0
# via
# aiohttp
# jsonschema
# referencing
audioop-lts==0.2.2 ; python_full_version >= '3.13'
# via
# speechrecognition
# standard-aifc
azure-ai-documentintelligence==1.0.2
# via markitdown
azure-core==1.41.0
# via
# azure-ai-documentintelligence
# azure-identity
azure-identity==1.25.3
# via markitdown
backoff==2.2.1
# via langfuse
bcrypt==5.0.0
# via deer-flow
beautifulsoup4==4.15.0
# via
# markdownify
# markitdown
# readabilipy
blockbuster==1.5.26
# via langgraph-runtime-inmem
brotli==1.2.0 ; platform_python_implementation == 'CPython'
# via httpx
brotlicffi==1.2.0.1 ; platform_python_implementation != 'CPython'
# via httpx
certifi==2026.5.20
# via
# httpcore
# httpx
# kubernetes
# requests
# volcengine-python-sdk
# wecom-aibot-python-sdk
cffi==2.0.0
# via
# brotlicffi
# cryptography
charset-normalizer==3.4.7
# via
# markitdown
# pdfminer-six
# requests
click==8.4.1
# via
# ddgs
# langgraph-cli
# magika
# uvicorn
cloudpickle==3.1.2
# via langgraph-api
cobble==0.1.4
# via mammoth
colorama==0.4.6 ; sys_platform == 'win32'
# via
# click
# tqdm
# uvicorn
coloredlogs==15.0.1
# via onnxruntime
croniter==6.2.2
# via
# deer-flow
# langgraph-runtime-inmem
cryptography==48.0.0
# via
# azure-identity
# google-auth
# langgraph-api
# msal
# pdfminer-six
# pyjwt
# wecom-aibot-python-sdk
ddgs==9.14.4
# via deerflow-harness
defusedxml==0.7.1
# via
# markitdown
# youtube-transcript-api
dingtalk-stream==0.24.3
# via deer-flow
distro==1.9.0
# via
# anthropic
# google-genai
# openai
dnspython==2.8.0
# via email-validator
docstring-parser==0.18.0
# via anthropic
dotenv==0.9.9
# via deerflow-harness
duckdb==1.5.3
# via deerflow-harness
durationpy==0.10
# via kubernetes
email-validator==2.3.0
# via deer-flow
et-xmlfile==2.0.0
# via openpyxl
exa-py==2.13.0
# via deerflow-harness
fake-useragent==2.2.0
# via ddgs
fastapi==0.136.3
# via deer-flow
filetype==1.2.0
# via langchain-google-genai
firecrawl-py==4.28.2
# via deerflow-harness
flatbuffers==25.12.19
# via onnxruntime
forbiddenfruit==0.1.4 ; implementation_name == 'cpython'
# via blockbuster
frozenlist==1.8.0
# via
# aiohttp
# aiosignal
google-auth==2.53.0
# via google-genai
google-genai==2.8.0
# via langchain-google-genai
googleapis-common-protos==1.75.0
# via opentelemetry-exporter-otlp-proto-http
greenlet==3.5.1
# via sqlalchemy
grpcio==1.80.0
# via
# grpcio-health-checking
# grpcio-tools
# langgraph-api
grpcio-health-checking==1.80.0
# via langgraph-api
grpcio-tools==1.80.0
# via langgraph-api
h11==0.16.0
# via
# httpcore
# uvicorn
h2==4.3.0
# via httpx
hindsight-client==0.7.2
# via deer-flow
hpack==4.1.0
# via h2
html5lib==1.1
# via readabilipy
httpcore==1.0.9
# via
# exa-py
# httpx
# python-telegram-bot
httptools==0.8.0
# via
# langgraph-api
# uvicorn
httpx==0.28.1
# via
# agent-sandbox
# anthropic
# ddgs
# deer-flow
# deerflow-harness
# exa-py
# firecrawl-py
# google-genai
# langfuse
# langgraph-api
# langgraph-cli
# langgraph-sdk
# langsmith
# lark-oapi
# mcp
# openai
# python-telegram-bot
# tavily-python
httpx-sse==0.4.3
# via mcp
humanfriendly==10.0
# via coloredlogs
hyperframe==6.1.0
# via h2
idna==3.18
# via
# anyio
# email-validator
# httpx
# requests
# yarl
isodate==0.7.2
# via azure-ai-documentintelligence
jiter==0.15.0
# via
# anthropic
# openai
jsonpatch==1.33
# via langchain-core
jsonpointer==3.1.1
# via jsonpatch
jsonschema==4.26.0
# via mcp
jsonschema-rs==0.44.1
# via langgraph-api
jsonschema-specifications==2025.9.1
# via jsonschema
kubernetes==36.0.2
# via deerflow-harness
langchain==1.3.4
# via deerflow-harness
langchain-anthropic==1.4.4
# via deerflow-harness
langchain-core==1.4.1
# via
# langchain
# langchain-anthropic
# langchain-deepseek
# langchain-google-genai
# langchain-mcp-adapters
# langchain-openai
# langgraph
# langgraph-api
# langgraph-checkpoint
# langgraph-prebuilt
# langgraph-sdk
langchain-deepseek==1.1.0
# via deerflow-harness
langchain-google-genai==4.2.4
# via deerflow-harness
langchain-mcp-adapters==0.2.2
# via deerflow-harness
langchain-openai==1.2.2
# via
# deerflow-harness
# langchain-deepseek
langchain-protocol==0.0.16
# via
# langchain-core
# langgraph-api
# langgraph-sdk
langfuse==4.7.1
# via deerflow-harness
langgraph==1.2.4
# via
# deerflow-harness
# langchain
# langgraph-api
# langgraph-runtime-inmem
langgraph-api==0.9.0
# via deerflow-harness
langgraph-checkpoint==4.1.1
# via
# langgraph
# langgraph-api
# langgraph-checkpoint-postgres
# langgraph-checkpoint-sqlite
# langgraph-prebuilt
# langgraph-runtime-inmem
langgraph-checkpoint-postgres==3.1.0
# via deerflow-harness
langgraph-checkpoint-sqlite==3.1.0
# via deerflow-harness
langgraph-cli==0.4.27
# via deerflow-harness
langgraph-prebuilt==1.1.0
# via langgraph
langgraph-runtime-inmem==0.29.0
# via
# deerflow-harness
# langgraph-api
langgraph-sdk==0.4.2
# via
# deer-flow
# deerflow-harness
# langgraph
# langgraph-api
# langgraph-cli
langsmith==0.8.9
# via
# langchain-core
# langgraph-api
lark-oapi==1.6.8
# via deer-flow
lxml==6.1.1
# via
# ddgs
# markitdown
# python-pptx
# readabilipy
magika==0.6.3
# via markitdown
mako==1.3.12
# via alembic
mammoth==1.11.0
# via markitdown
markdown-to-mrkdwn==0.3.3
# via deer-flow
markdownify==1.2.2
# via
# deerflow-harness
# markitdown
markitdown==0.1.5
# via deerflow-harness
markupsafe==3.0.3
# via mako
mcp==1.27.2
# via langchain-mcp-adapters
mpmath==1.3.0
# via sympy
msal==1.37.0
# via
# azure-identity
# msal-extensions
msal-extensions==1.3.1
# via azure-identity
multidict==6.7.1
# via
# aiohttp
# yarl
nest-asyncio==1.6.0
# via firecrawl-py
numpy==2.4.6
# via
# magika
# onnxruntime
# pandas
oauthlib==3.3.1
# via requests-oauthlib
olefile==0.47
# via markitdown
onnxruntime==1.20.1
# via magika
openai==2.41.0
# via
# exa-py
# langchain-openai
openpyxl==3.1.5
# via markitdown
opentelemetry-api==1.42.1
# via
# langfuse
# langgraph-api
# langsmith
# opentelemetry-exporter-otlp-proto-http
# opentelemetry-sdk
# opentelemetry-semantic-conventions
opentelemetry-exporter-otlp-proto-common==1.42.1
# via opentelemetry-exporter-otlp-proto-http
opentelemetry-exporter-otlp-proto-http==1.42.1
# via
# langfuse
# langgraph-api
# langsmith
opentelemetry-proto==1.42.1
# via
# opentelemetry-exporter-otlp-proto-common
# opentelemetry-exporter-otlp-proto-http
opentelemetry-sdk==1.42.1
# via
# langfuse
# langgraph-api
# langsmith
# opentelemetry-exporter-otlp-proto-http
opentelemetry-semantic-conventions==0.63b1
# via opentelemetry-sdk
orjson==3.11.9
# via
# langgraph-api
# langgraph-checkpoint-postgres
# langgraph-sdk
# langsmith
ormsgpack==1.12.2
# via langgraph-checkpoint
packaging==26.2
# via
# langchain-core
# langfuse
# langsmith
# onnxruntime
pandas==3.0.3
# via markitdown
pathspec==1.1.1
# via langgraph-cli
pdfminer-six==20251230
# via
# markitdown
# pdfplumber
pdfplumber==0.11.9
# via markitdown
pillow==12.2.0
# via
# pdfplumber
# python-pptx
primp==1.3.1
# via ddgs
propcache==0.5.2
# via
# aiohttp
# yarl
protobuf==6.33.6
# via
# googleapis-common-protos
# grpcio-health-checking
# grpcio-tools
# langgraph-api
# onnxruntime
# opentelemetry-proto
psycopg==3.3.4
# via
# deerflow-harness
# langgraph-checkpoint-postgres
psycopg-binary==3.3.4 ; implementation_name != 'pypy'
# via psycopg
psycopg-pool==3.3.1
# via
# deerflow-harness
# langgraph-checkpoint-postgres
pyasn1==0.6.3
# via pyasn1-modules
pyasn1-modules==0.4.2
# via google-auth
pycparser==3.0 ; implementation_name != 'PyPy'
# via cffi
pycryptodome==3.23.0
# via lark-oapi
pydantic==2.13.4
# via
# agent-client-protocol
# agent-sandbox
# anthropic
# deerflow-harness
# exa-py
# fastapi
# firecrawl-py
# google-genai
# hindsight-client
# langchain
# langchain-anthropic
# langchain-core
# langchain-google-genai
# langfuse
# langgraph
# langsmith
# mcp
# openai
# pydantic-settings
pydantic-core==2.46.4
# via pydantic
pydantic-settings==2.14.1
# via mcp
pydub==0.25.1
# via markitdown
pyee==13.0.1
# via wecom-aibot-python-sdk
pyjwt==2.13.0
# via
# deer-flow
# langgraph-api
# mcp
# msal
pypdfium2==5.9.0
# via pdfplumber
pyreadline3==3.5.6 ; sys_platform == 'win32'
# via humanfriendly
python-dateutil==2.9.0.post0
# via
# croniter
# hindsight-client
# kubernetes
# pandas
# volcengine-python-sdk
python-dotenv==1.2.2
# via
# dotenv
# exa-py
# firecrawl-py
# langgraph-cli
# magika
# pydantic-settings
# uvicorn
python-multipart==0.0.32
# via
# deer-flow
# mcp
python-pptx==1.0.2
# via markitdown
python-telegram-bot==22.7
# via deer-flow
pywin32==312 ; sys_platform == 'win32'
# via mcp
pyyaml==6.0.3
# via
# deerflow-harness
# kubernetes
# langchain-core
# uvicorn
readabilipy==0.3.0
# via deerflow-harness
referencing==0.37.0
# via
# jsonschema
# jsonschema-specifications
regex==2026.5.9
# via
# readabilipy
# tiktoken
requests==2.34.2
# via
# azure-core
# dingtalk-stream
# exa-py
# firecrawl-py
# google-auth
# google-genai
# kubernetes
# langsmith
# lark-oapi
# markitdown
# msal
# opentelemetry-exporter-otlp-proto-http
# requests-oauthlib
# requests-toolbelt
# tavily-python
# tiktoken
# youtube-transcript-api
requests-oauthlib==2.0.0
# via kubernetes
requests-toolbelt==1.0.0
# via
# langsmith
# lark-oapi
rpds-py==2026.5.1
# via
# jsonschema
# referencing
setuptools==82.0.1
# via grpcio-tools
six==1.17.0
# via
# html5lib
# kubernetes
# markdownify
# python-dateutil
# volcengine-python-sdk
slack-sdk==3.42.0
# via deer-flow
sniffio==1.3.1
# via
# anthropic
# google-genai
# openai
socksio==1.0.0
# via httpx
soupsieve==2.8.4
# via beautifulsoup4
speechrecognition==3.16.1
# via markitdown
sqlalchemy==2.0.50
# via
# alembic
# deerflow-harness
sqlite-vec==0.1.9
# via langgraph-checkpoint-sqlite
sse-starlette==3.3.4
# via
# deer-flow
# langgraph-api
# langgraph-runtime-inmem
# mcp
standard-aifc==3.13.0 ; python_full_version >= '3.13'
# via speechrecognition
standard-chunk==3.13.0 ; python_full_version >= '3.13'
# via standard-aifc
starlette==1.2.1
# via
# fastapi
# langgraph-api
# langgraph-runtime-inmem
# mcp
# sse-starlette
structlog==25.5.0
# via
# langgraph-api
# langgraph-runtime-inmem
sympy==1.14.0
# via onnxruntime
tavily-python==0.7.25
# via deerflow-harness
tenacity==9.1.4
# via
# google-genai
# langchain-core
# langgraph-api
tiktoken==0.13.0
# via
# deerflow-harness
# langchain-openai
# tavily-python
tqdm==4.68.1
# via openai
truststore==0.10.4
# via langgraph-api
typing-extensions==4.15.0
# via
# aiohttp
# aiosignal
# alembic
# anthropic
# anyio
# azure-ai-documentintelligence
# azure-core
# azure-identity
# beautifulsoup4
# exa-py
# fastapi
# google-genai
# grpcio
# hindsight-client
# langchain-core
# langchain-mcp-adapters
# langchain-protocol
# mcp
# openai
# opentelemetry-api
# opentelemetry-exporter-otlp-proto-http
# opentelemetry-sdk
# opentelemetry-semantic-conventions
# psycopg
# psycopg-pool
# pydantic
# pydantic-core
# pyee
# python-pptx
# referencing
# speechrecognition
# sqlalchemy
# starlette
# typing-inspection
typing-inspection==0.4.2
# via
# fastapi
# mcp
# pydantic
# pydantic-settings
tzdata==2026.2 ; sys_platform == 'emscripten' or sys_platform == 'win32'
# via
# pandas
# psycopg
urllib3==2.7.0
# via
# hindsight-client
# kubernetes
# requests
# volcengine-python-sdk
uuid-utils==0.16.0
# via
# langchain-core
# langgraph-api
# langsmith
uvicorn==0.49.0
# via
# deer-flow
# langgraph-api
# mcp
uvloop==0.22.1 ; sys_platform != 'win32'
# via
# langgraph-api
# uvicorn
volcengine-python-sdk==5.0.33
# via agent-sandbox
watchfiles==1.2.0
# via
# langgraph-api
# uvicorn
webencodings==0.5.1
# via html5lib
websocket-client==1.9.0
# via kubernetes
websockets==15.0.1
# via
# dingtalk-stream
# firecrawl-py
# google-genai
# langgraph-api
# langgraph-sdk
# langsmith
# lark-oapi
# uvicorn
# wecom-aibot-python-sdk
wecom-aibot-python-sdk==1.0.2
# via deer-flow
wrapt==1.17.3
# via langfuse
xlrd==2.0.2
# via markitdown
xlsxwriter==3.2.9
# via python-pptx
xxhash==3.7.0
# via
# langgraph
# langsmith
yarl==1.24.2
# via aiohttp
youtube-transcript-api==1.0.3
# via markitdown
zstandard==0.25.0
# via
# langgraph-api
# langsmith
offline-backend-20260512/backend/app/gateway/app.py
+276
View File
@@ -31,6 +31,7 @@ from app.gateway.routers import (
assistants_compat,
auth,
channels,
checkpoint_migration,
feedback,
html_page_favorites,
intent,
@@ -40,11 +41,14 @@ from app.gateway.routers import (
models,
multi_agent,
notifications,
parallel_agent_tasks,
parallel_agents,
public_embed,
public_scheduled_tasks,
recommend,
recommended_questions,
roundtable_chains,
roundtable_draft_shares,
roundtable_drafts,
roundtable_jobs,
runs,
@@ -58,6 +62,7 @@ from app.gateway.routers import (
thread_shares,
threads,
uploads,
user_preferences,
user_prompts,
writing,
)
@@ -405,6 +410,243 @@ async def _curator_scheduler_loop() -> None:
_release_singleton_lock(lock_handle)
def _env_bool(name: str, default: bool) -> bool:
raw = os.getenv(name)
if raw is None:
return default
return raw.strip().lower() not in {"0", "false", "no", "off"}
def _env_int(name: str, default: int, *, alias: str | None = None, minimum: int = 1) -> int:
raw = os.getenv(name)
if raw is None and alias:
raw = os.getenv(alias)
if raw is None:
raw = str(default)
try:
value = int(raw)
except ValueError:
return default
return max(minimum, value)
def _env_float(name: str, default: float, *, alias: str | None = None, minimum: float = 0.0) -> float:
raw = os.getenv(name)
if raw is None and alias:
raw = os.getenv(alias)
if raw is None:
raw = str(default)
try:
value = float(raw)
except ValueError:
return default
return max(minimum, value)
_LEADERBOARD_SNAPSHOT_BACKGROUND_ENABLED = _env_bool("ADMIN_LEADERBOARD_BACKGROUND_ENABLED", True)
_LEADERBOARD_SNAPSHOT_PREWARM_DAYS = _env_int(
"ADMIN_LEADERBOARD_BACKGROUND_PREWARM_DAYS",
30,
)
_LEADERBOARD_SNAPSHOT_BATCH_SIZE = _env_int(
"ADMIN_LEADERBOARD_BACKGROUND_BATCH_SIZE",
1,
)
_LEADERBOARD_SNAPSHOT_INTERVAL_SECONDS = _env_float(
"ADMIN_LEADERBOARD_BACKGROUND_INTERVAL_SECONDS",
300.0,
)
_LEADERBOARD_SNAPSHOT_SLEEP_SECONDS = _env_float(
"ADMIN_LEADERBOARD_BACKGROUND_SLEEP_SECONDS",
2.0,
)
_LEADERBOARD_SNAPSHOT_RUNNING_TTL_SECONDS = _env_int(
"ADMIN_LEADERBOARD_BACKGROUND_RUNNING_TTL_SECONDS",
1800,
)
_LEADERBOARD_SNAPSHOT_STARTUP_DELAY_SECONDS = _env_float(
"ADMIN_LEADERBOARD_BACKGROUND_STARTUP_DELAY_SECONDS",
60.0,
)
def _recent_leaderboard_stat_dates(days: int) -> list[str]:
from datetime import datetime, timedelta
from deerflow.persistence.types import BEIJING_TZ
today = datetime.now(BEIJING_TZ).date()
start = today - timedelta(days=max(1, days) - 1)
return [(start + timedelta(days=i)).isoformat() for i in range((today - start).days + 1)]
async def _leaderboard_snapshot_due_dates(
session_factory,
*,
settings_hash: str,
limit: int,
running_ttl_seconds: int,
) -> list[str]:
from datetime import UTC, datetime, timedelta
from sqlalchemy import and_, or_, select
from deerflow.persistence.admin_stats.model import AdminLeaderboardDailyStatRow
cutoff = datetime.now(UTC) - timedelta(seconds=running_ttl_seconds)
async with session_factory() as session:
rows = (
await session.execute(
select(AdminLeaderboardDailyStatRow.stat_date)
.where(
AdminLeaderboardDailyStatRow.settings_hash == settings_hash,
or_(
AdminLeaderboardDailyStatRow.status.in_(["queued", "error"]),
and_(
AdminLeaderboardDailyStatRow.status == "running",
AdminLeaderboardDailyStatRow.updated_at < cutoff,
),
),
)
.order_by(AdminLeaderboardDailyStatRow.stat_date.desc())
.limit(limit)
)
).all()
return [str(row[0]) for row in rows]
async def _claim_leaderboard_snapshot(
session_factory,
*,
stat_date: str,
settings_hash: str,
running_ttl_seconds: int,
) -> bool:
from datetime import UTC, datetime
from app.gateway.routers.admin_users import _snapshot_is_fresh
from deerflow.persistence.admin_stats.model import AdminLeaderboardDailyStatRow
from deerflow.persistence.types import BEIJING_TZ
async with session_factory() as session:
row = await session.get(
AdminLeaderboardDailyStatRow,
{"stat_date": stat_date, "settings_hash": settings_hash},
)
if row is None or _snapshot_is_fresh(row, stat_date):
return False
if row.status == "running":
updated_at = row.updated_at
if isinstance(updated_at, datetime):
if updated_at.tzinfo is None:
updated_at = updated_at.replace(tzinfo=BEIJING_TZ)
age = (datetime.now(BEIJING_TZ) - updated_at.astimezone(BEIJING_TZ)).total_seconds()
if age < running_ttl_seconds:
return False
row.status = "running"
row.message = "排行榜后台统计任务已认领。"
row.error = None
row.updated_at = datetime.now(UTC)
await session.commit()
return True
async def _run_leaderboard_snapshot_tick() -> tuple[int, int]:
from app.gateway.routers.admin_users import (
_daily_job_key,
_queue_daily_leaderboard_snapshots,
_refresh_daily_leaderboard_snapshot,
_resolve_leaderboard_snapshot_scope,
)
from deerflow.config.system_settings import load_system_settings
from deerflow.persistence.engine import get_session_factory
session_factory = get_session_factory()
if session_factory is None:
return 0, 0
settings = load_system_settings().leaderboard
excluded, include_scheduled, include_admins, include_failed, settings_hash = _resolve_leaderboard_snapshot_scope(settings)
stat_dates = _recent_leaderboard_stat_dates(_LEADERBOARD_SNAPSHOT_PREWARM_DAYS)
queued_dates = await _queue_daily_leaderboard_snapshots(
session_factory,
stat_dates,
settings_hash,
message="排行榜后台统计任务定期预热/补算队列。",
)
due_dates = await _leaderboard_snapshot_due_dates(
session_factory,
settings_hash=settings_hash,
limit=_LEADERBOARD_SNAPSHOT_BATCH_SIZE,
running_ttl_seconds=_LEADERBOARD_SNAPSHOT_RUNNING_TTL_SECONDS,
)
processed = 0
for stat_date in due_dates:
claimed = await _claim_leaderboard_snapshot(
session_factory,
stat_date=stat_date,
settings_hash=settings_hash,
running_ttl_seconds=_LEADERBOARD_SNAPSHOT_RUNNING_TTL_SECONDS,
)
if not claimed:
continue
logger.info("Leaderboard snapshot scheduler: building date=%s hash=%s", stat_date, settings_hash[:10])
await _refresh_daily_leaderboard_snapshot(
_daily_job_key(stat_date, settings_hash),
session_factory=session_factory,
settings=settings,
stat_date=stat_date,
settings_hash=settings_hash,
excluded=excluded,
include_scheduled=include_scheduled,
include_admins=include_admins,
include_failed=include_failed,
)
processed += 1
if _LEADERBOARD_SNAPSHOT_SLEEP_SECONDS > 0 and processed < len(due_dates):
await asyncio.sleep(_LEADERBOARD_SNAPSHOT_SLEEP_SECONDS)
return processed, len(queued_dates)
async def _leaderboard_snapshot_scheduler_loop() -> None:
"""Background task inside the existing gateway process for daily snapshots."""
if not _LEADERBOARD_SNAPSHOT_BACKGROUND_ENABLED:
logger.info("Leaderboard snapshot scheduler disabled by ADMIN_LEADERBOARD_BACKGROUND_ENABLED")
return
from deerflow.config.runtime_paths import runtime_home
lock_handle = _acquire_singleton_lock(runtime_home() / ".leaderboard_snapshot_scheduler.lock")
if lock_handle is None:
logger.info("Leaderboard snapshot scheduler: another gateway worker holds the lock; this worker skips")
return
logger.info(
"Leaderboard snapshot scheduler started: prewarm_days=%d batch_size=%d interval=%.1fs",
_LEADERBOARD_SNAPSHOT_PREWARM_DAYS,
_LEADERBOARD_SNAPSHOT_BATCH_SIZE,
_LEADERBOARD_SNAPSHOT_INTERVAL_SECONDS,
)
try:
if _LEADERBOARD_SNAPSHOT_STARTUP_DELAY_SECONDS > 0:
await asyncio.sleep(_LEADERBOARD_SNAPSHOT_STARTUP_DELAY_SECONDS)
while True:
try:
processed, queued = await _run_leaderboard_snapshot_tick()
if processed or queued:
logger.info(
"Leaderboard snapshot scheduler tick: queued=%d processed=%d",
queued,
processed,
)
except Exception:
logger.exception("Leaderboard snapshot scheduler tick failed (non-fatal)")
await asyncio.sleep(_LEADERBOARD_SNAPSHOT_INTERVAL_SECONDS)
finally:
_release_singleton_lock(lock_handle)
def _warn_if_multi_worker() -> None:
"""检测 uvicorn / gunicorn 多 worker 部署并打 ERROR 日志。
@@ -566,6 +808,11 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
except Exception:
logger.exception("AI writing cleanup scheduler failed to start (non-fatal)")
try:
app.state.leaderboard_snapshot_scheduler_task = asyncio.create_task(_leaderboard_snapshot_scheduler_loop())
except Exception:
logger.exception("Leaderboard snapshot scheduler failed to start (non-fatal)")
# Start IM channel service if any channels are configured
try:
from app.channels.service import start_channel_service
@@ -614,6 +861,16 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
except Exception:
logger.exception("AI writing cleanup scheduler shutdown error")
leaderboard_snapshot_task = getattr(app.state, "leaderboard_snapshot_scheduler_task", None)
if leaderboard_snapshot_task is not None:
leaderboard_snapshot_task.cancel()
try:
await leaderboard_snapshot_task
except asyncio.CancelledError:
pass
except Exception:
logger.exception("Leaderboard snapshot scheduler shutdown error")
# 阶段 7 删冗余：AI 写作 graph 不再自管 checkpointer / in-flight task；
# thread 状态由 LangGraph Server 主 checkpointer 接管，lifespan 出口由
# ``langgraph_runtime()`` 的 contextmanager 统一关闭，无需在这里特殊处理。
@@ -858,12 +1115,19 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
app.include_router(user_prompts.router)
# Per-user UI preferences at /api/user-preferences
app.include_router(user_preferences.router)
# Roundtable-planning drafts + recommendation history at /api/roundtable-drafts
app.include_router(roundtable_drafts.router)
# Roundtable-planning business chains at /api/roundtable-chains
app.include_router(roundtable_chains.router)
# Standalone parallel multi-agent panel: fan-out streaming + saved tasks
app.include_router(parallel_agents.router)
app.include_router(parallel_agent_tasks.router)
# Roundtable-planning background jobs at /api/roundtable-jobs
app.include_router(roundtable_jobs.router)
@@ -903,6 +1167,9 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
# Channels API is mounted at /api/channels
app.include_router(channels.router)
# Admin checkpoint migration API: SQLite legacy checkpoints -> Postgres.
app.include_router(checkpoint_migration.router)
# Assistants compatibility API (LangGraph Platform stub)
app.include_router(assistants_compat.router)
@@ -925,6 +1192,11 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
# Public, unauthenticated read-only share snapshots (/api/public/shares/*).
app.include_router(thread_shares.public_router)
# 圆桌会商草稿分享（import 复制副本 / view 只读结果报告页）
app.include_router(roundtable_draft_shares.router)
# Public, unauthenticated read-only roundtable result (/api/public/roundtable-shares/*).
app.include_router(roundtable_draft_shares.public_router)
# Stateless Runs API (stream/wait without a pre-existing thread)
app.include_router(runs.router)
@@ -936,7 +1208,11 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
app.include_router(recommend.router) # Step 1→2：/api/recommend/stream（无 init）
# Step 3 产物列表/下载复用下方 artifacts.router（/api/threads/{id}/artifacts）
# 探活端点：无需鉴权(见 auth_middleware._PUBLIC_PATH_PREFIXES)。
# 同时暴露裸 ``/health`` 与带 API 前缀的 ``/api/health``,方便监控按
# 任意 base URL 直接探活,不会被认证中间件拦成 401。
@app.get("/health", tags=["health"])
@app.get("/api/health", tags=["health"])
async def health_check() -> dict:
"""Health check endpoint.
offline-backend-20260512/backend/app/gateway/auth/local_provider.py
+8
View File
@@ -71,6 +71,14 @@ class LocalAuthProvider(AuthProvider):
"""
return await self._repo.find_user_by_email_local_part(local_part)
async def find_users_by_email_local_part_like(self, substring: str, *, limit: int = 500) -> list[User]:
"""Fuzzy-match users by a substring of their email local part.
Backs the admin LLM-metrics username filter — returns every account
whose username contains ``substring`` (case-insensitive).
"""
return await self._repo.find_users_by_email_local_part_like(substring, limit=limit)
async def create_user(self, email: str, password: str | None = None, system_role: str = "user", needs_setup: bool = False) -> User:
"""Create a new local user.
offline-backend-20260512/backend/app/gateway/auth/repositories/base.py
+18
View File
@@ -81,6 +81,24 @@ class UserRepository(ABC):
"""
raise NotImplementedError
async def find_users_by_email_local_part_like(self, substring: str, *, limit: int = 500) -> list["User"]:
"""Fuzzy-match users whose email local part *contains* ``substring``.
Backs the admin LLM-metrics username filter (按用户名模糊搜索). Unlike
:meth:`find_user_by_email_local_part` (exact prefix → single account),
this returns every matching account. Case-insensitive substring match
against the part before ``@``. Non-overriding repositories return an
empty list.
Args:
substring: Case-insensitive substring to match within the local part.
limit: Maximum number of users to return.
Returns:
Matching users (may be empty).
"""
return []
@abstractmethod
async def update_user(self, user: User) -> User:
"""Update an existing user.
offline-backend-20260512/backend/app/gateway/auth/repositories/sqlite.py
+27
View File
@@ -109,6 +109,33 @@ class SQLiteUserRepository(UserRepository):
row = result.scalar_one_or_none()
return self._row_to_user(row) if row is not None else None
async def find_users_by_email_local_part_like(self, substring: str, *, limit: int = 500) -> list[User]:
term = (substring or "").strip().lower()
if not term:
return []
# Escape SQL LIKE wildcards so a literal ``_``/``%`` in the query does
# not turn into a wildcard. ``\`` is the chosen escape char.
escaped = term.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
pattern = f"%{escaped}%"
# Prefilter in SQL on the whole email (portable across sqlite/postgres/
# mysql), then keep only rows whose *local part* contains the term so a
# domain-only hit (e.g. searching "gmail") doesn't pull unrelated users.
stmt = (
select(UserRow)
.where(func.lower(UserRow.email).like(pattern, escape="\\"))
.order_by(UserRow.created_at.asc())
.limit(max(1, limit))
)
async with self._sf() as session:
result = await session.execute(stmt)
rows = result.scalars().all()
out: list[User] = []
for row in rows:
local = (row.email or "").split("@", 1)[0].lower()
if term in local:
out.append(self._row_to_user(row))
return out
async def update_user(self, user: User) -> User:
async with self._sf() as session:
row = await session.get(UserRow, str(user.id))
offline-backend-20260512/backend/app/gateway/auth_middleware.py
+3
View File
@@ -29,6 +29,7 @@ from deerflow.runtime.user_context import reset_current_user, set_current_user
# safe read-only routes under it — everything there bypasses auth entirely.
_PUBLIC_PATH_PREFIXES: tuple[str, ...] = (
"/health",
"/api/health",
"/docs",
"/redoc",
"/openapi.json",
@@ -46,6 +47,8 @@ _PUBLIC_EXACT_PATHS: frozenset[str] = frozenset(
"/api/v1/auth/logout",
"/api/v1/auth/setup-status",
"/api/v1/auth/initialize",
# 并行多智能体面板专用的「绕过常规登录」取 token 端点（内网部署限制了标准登录时用）。
"/api/parallel-agents/auth/token",
}
)
offline-backend-20260512/backend/app/gateway/csrf_middleware.py
+2
View File
@@ -59,6 +59,8 @@ _AUTH_EXEMPT_PATHS: frozenset[str] = frozenset(
"/api/v1/auth/logout",
"/api/v1/auth/register",
"/api/v1/auth/initialize",
# 并行多智能体面板专用取 token 端点（公共 + CSRF 豁免）。
"/api/parallel-agents/auth/token",
}
)
offline-backend-20260512/backend/app/gateway/deps.py
+20
View File
@@ -163,14 +163,26 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
app.state.user_prompt_store = make_user_prompt_store(sf)
from deerflow.persistence.user_preferences import make_user_preferences_store
app.state.user_preferences_store = make_user_preferences_store(sf)
from deerflow.persistence.roundtable_drafts import make_roundtable_draft_store
app.state.roundtable_draft_store = make_roundtable_draft_store(sf)
from deerflow.persistence.roundtable_draft_shares import make_roundtable_draft_share_store
app.state.roundtable_draft_share_store = make_roundtable_draft_share_store(sf)
from deerflow.persistence.roundtable_chains import make_roundtable_chain_store
app.state.roundtable_chain_store = make_roundtable_chain_store(sf)
from deerflow.persistence.parallel_agent_tasks import make_parallel_agent_task_store
app.state.parallel_agent_task_store = make_parallel_agent_task_store(sf)
from deerflow.persistence.roundtable_jobs import (
make_roundtable_job_store,
register_default_store as _register_roundtable_job_default_store,
@@ -330,6 +342,14 @@ def get_thread_share_store(request: Request):
return val
def get_roundtable_draft_share_store(request: Request):
"""Return the roundtable-draft share-link store."""
val = getattr(request.app.state, "roundtable_draft_share_store", None)
if val is None:
raise HTTPException(status_code=503, detail="Roundtable draft share store not available")
return val
def get_embed_session_store(request: Request):
"""Return the iframe embed session → thread mapping store."""
val = getattr(request.app.state, "embed_session_store", None)
offline-backend-20260512/backend/app/gateway/roundtable_inprocess_gateway.py
+103 -32
View File
@@ -22,6 +22,12 @@ from typing import Any
from langgraph.checkpoint.base import empty_checkpoint
from app.gateway.roundtable_run_policy import roundtable_run_policy
from app.gateway.routers._roundtable_seed import (
COORDINATOR_AGENT_ID,
REPORT_AGENT_ID,
ensure_roundtable_functional_agents,
)
from deerflow.agents.roundtable_orchestrator.types import (
LeaderTurn,
ReportResult,
@@ -31,22 +37,14 @@ from deerflow.agents.roundtable_orchestrator.types import (
from deerflow.config.paths import get_paths
from deerflow.runtime.serialization import serialize_channel_values
from app.gateway.routers._roundtable_seed import (
COORDINATOR_AGENT_ID,
REPORT_AGENT_ID,
ensure_roundtable_functional_agents,
)
logger = logging.getLogger(__name__)
_ROUNDTABLE_THREAD_METADATA: dict[str, Any] = {"thread_type": "roundtable", "system": True}
_REPORT_FILE = "方案总览.html"
# 各角色的工具裁剪（镜像 multi_agent 的 _leader_run / _special_run）。
_LEADER_EXCLUDED = ["web_search"]
_LEADER_SKILL_STOP = ["agent_orchestration"]
_SEAT_EXCLUDED = ["ask_clarification", "present_files", "view_image", "agent_orchestration"]
_REPORT_EXCLUDED = [*_SEAT_EXCLUDED, "bash", "str_replace"]
# 各角色的工具裁剪 / thinking / reasoning 全部取自单一数据源 roundtable_run_policy
# （与 multi_agent 的 _leader_run / _special_run 同源），不再在本文件硬编码——历史上
# 这里漏禁了 web_search，导致后台席位 / 报告能联网搜索而前端不能，现由 policy 统一。
# ── 进程内底层操作 ───────────────────────────────────────────────────────────
@@ -101,6 +99,56 @@ def _message_text(message: dict[str, Any] | None) -> str:
return ""
def _latest_ai_text(messages: list[dict[str, Any]]) -> str:
"""取**最后一条有正文**的 AI 消息文本。
席位常以「调一堆工具」的 AIMessage 收尾（正文为空、只有 tool_calls，甚至 dangling），
直接取最后一条 AI 消息会拿到空串 → 报告就拿不到席位产物。这里向前找到真正有文字的那条。
"""
for m in reversed(messages):
if not isinstance(m, dict):
continue
if (m.get("type") or m.get("role")) not in {"ai", "assistant"}:
continue
text = _message_text(m)
if text.strip():
return text
return ""
def _normalize_tool_calls(tool_calls: Any) -> list[dict[str, Any]]:
"""把 checkpoint 里的 tool_calls 归一化成 {name, args, id}（丢掉无名项）。"""
out: list[dict[str, Any]] = []
for tc in tool_calls or []:
if not isinstance(tc, dict):
continue
name = tc.get("name") or ""
if not name:
continue
out.append({"name": name, "args": tc.get("args") or {}, "id": tc.get("id") or ""})
return out
def _tool_calls_since_last_human(messages: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""收集「最后一条 human 消息之后」各 AI 消息的 tool_calls（即**本轮**调用），保序。
席位 thread 跨轮累积历史，直接扫全量会把往轮的工具调用也算进来；以最后一条 human
（本轮 run 的输入）为界，只取其后的 AI 工具调用，对应前端「这一条气泡的步骤卡」。
"""
last_human = -1
for i, m in enumerate(messages):
if isinstance(m, dict) and (m.get("type") or m.get("role")) in {"human", "user"}:
last_human = i
out: list[dict[str, Any]] = []
for m in messages[last_human + 1 :]:
if not isinstance(m, dict):
continue
if (m.get("type") or m.get("role")) not in {"ai", "assistant"}:
continue
out.extend(_normalize_tool_calls(m.get("tool_calls")))
return out
async def _run_agent_turn(
app,
*,
@@ -135,6 +183,9 @@ async def _run_agent_turn(
"subagent_enabled": False,
"thread_id": thread_id,
"memory_recall_disabled": True,
# 与 multi_agent._run_payload 对齐:关 builtin 记忆注入,避免单例总控/席位
# 被上轮会话 memory.json 带偏(臆造席位 / 直接干活)。详见该处注释。
"memory_injection_enabled": False,
},
excluded_tools=excluded_tools,
skill_stop_names=skill_stop_names,
@@ -149,12 +200,26 @@ async def _run_agent_turn(
def _read_report_html(app, thread_id: str, user_id: str) -> str | None:
"""从 report thread 的 outputs/方案总览.html 读出报告 HTML。"""
"""从 report thread 的 outputs 目录读出报告 HTML。
优先 ``方案总览.html``；找不到就扫目录里**任意 .html**（取体积最大的那个），
容忍报告 agent 用了别的文件名。
"""
try:
outputs_dir = get_paths().sandbox_outputs_dir(thread_id, user_id=user_id)
path = outputs_dir / _REPORT_FILE
if path.exists():
return path.read_text(encoding="utf-8")
preferred = outputs_dir / _REPORT_FILE
if preferred.exists():
return preferred.read_text(encoding="utf-8")
if outputs_dir.exists():
htmls = sorted(
outputs_dir.glob("*.html"),
key=lambda p: p.stat().st_size,
reverse=True,
)
if htmls:
logger.info("report html: 方案总览.html not found, using %s", htmls[0].name)
return htmls[0].read_text(encoding="utf-8")
logger.warning("report html: no .html found in %s", outputs_dir)
except Exception: # noqa: BLE001
logger.warning("read report html failed for thread %s", thread_id, exc_info=True)
return None
@@ -202,27 +267,31 @@ class InProcessRoundtableGateway:
if self._first_leader and self._seat_intro:
msg = self._seat_intro + "\n\n" + message
self._first_leader = False
policy = roundtable_run_policy("leader")
messages = await _run_agent_turn(
self._app,
thread_id=thread_ids[coordinator_name],
agent_name=coordinator_name,
message=msg,
model=model or self._model,
excluded_tools=_LEADER_EXCLUDED,
skill_stop_names=_LEADER_SKILL_STOP,
thinking_enabled=False,
reasoning_effort="low",
excluded_tools=policy.excluded_tools,
skill_stop_names=policy.skill_stop_names,
thinking_enabled=policy.thinking_enabled,
reasoning_effort=policy.reasoning_effort,
)
ai = _latest_ai_message(messages)
final_text = _message_text(ai)
# tool_calls 取最后一条 AI 消息（派活就在这条）；正文取最后一条有文字的 AI 消息。
final_text = _latest_ai_text(messages)
tool_calls = (ai.get("tool_calls") if ai else None) or []
# 本轮工具调用（派活 / 澄清），带进 LeaderTurn 供草稿 run 重建步骤卡。
turn_tool_calls = _normalize_tool_calls(tool_calls)
# 澄清优先
clar = next((tc for tc in tool_calls if (tc.get("name") or "").lower() == "ask_clarification"), None)
if clar is not None:
cargs = clar.get("args") or {}
question = str(cargs.get("question") or "").strip() or "总控需要补充信息，请补充后继续。"
return LeaderTurn(final_text=final_text, clarification=question)
return LeaderTurn(final_text=final_text, clarification=question, tool_calls=turn_tool_calls)
# 派活解析（复用 multi_agent 纯函数）
from app.gateway.routers.multi_agent import _extract_dispatch
@@ -235,38 +304,40 @@ class InProcessRoundtableGateway:
dispatched.append((name, task))
except Exception: # noqa: BLE001
logger.warning("dispatch parse failed for tool_call", exc_info=True)
return LeaderTurn(final_text=final_text, dispatched=dispatched)
return LeaderTurn(final_text=final_text, dispatched=dispatched, tool_calls=turn_tool_calls)
async def run_seat(self, *, thread_ids, agent_id, task, model) -> SeatTurn:
policy = roundtable_run_policy("seat")
messages = await _run_agent_turn(
self._app,
thread_id=thread_ids[agent_id],
agent_name=agent_id,
message=task,
model=model or self._model,
excluded_tools=_SEAT_EXCLUDED,
skill_stop_names=[],
thinking_enabled=True,
reasoning_effort="medium",
excluded_tools=policy.excluded_tools,
skill_stop_names=policy.skill_stop_names,
thinking_enabled=policy.thinking_enabled,
reasoning_effort=policy.reasoning_effort,
)
return SeatTurn(final_text=_message_text(_latest_ai_message(messages)))
return SeatTurn(final_text=_latest_ai_text(messages), tool_calls=_tool_calls_since_last_human(messages))
async def create_report_thread(self) -> str:
ensure_roundtable_functional_agents()
return await _create_thread(self._app)
async def run_report(self, *, report_thread_id, prompt, model) -> ReportResult:
policy = roundtable_run_policy("report")
messages = await _run_agent_turn(
self._app,
thread_id=report_thread_id,
agent_name=REPORT_AGENT_ID,
message=prompt,
model=model or self._model,
excluded_tools=_REPORT_EXCLUDED,
skill_stop_names=[],
thinking_enabled=True,
reasoning_effort="medium",
excluded_tools=policy.excluded_tools,
skill_stop_names=policy.skill_stop_names,
thinking_enabled=policy.thinking_enabled,
reasoning_effort=policy.reasoning_effort,
)
summary = _message_text(_latest_ai_message(messages))
summary = _latest_ai_text(messages)
html = _read_report_html(self._app, report_thread_id, self._user_id) or summary
return ReportResult(html=html, summary=summary, model=model or self._model)
offline-backend-20260512/backend/app/gateway/roundtable_job_executor.py
+234 -20
View File
@@ -17,7 +17,8 @@ from __future__ import annotations
import asyncio
import logging
from dataclasses import dataclass, field
from typing import Callable
from datetime import UTC, datetime
from typing import Any, Callable
from deerflow.agents.roundtable_orchestrator import (
JobProgressWriter,
@@ -28,6 +29,15 @@ from deerflow.runtime.user_context import reset_current_user, set_current_user
logger = logging.getLogger(__name__)
# ⚠️ 临时禁用「后台挂起」的 Step3 结果绘制（报告 HTML 生成）。
# 进程内驱动报告 agent 在真模型环境下仍无法稳定写出 HTML（前端手动调
# /api/multi-agent/report/init + run/stream 可以，但后台 start_run 路径不行，
# 根因尚未定位）。临时方案：后台只跑完 Step2 研讨即收尾，不自动画报告——用户需要
# 报告时可在前端手动跑 Step3（那条路是通的）。
# 待修复后把本开关改回 True 即恢复后台自动报告。
# 详见 frontend-web/docs/multi-agent-background-run-遗留问题与下次继续.md「后台报告 HTML」。
ROUNDTABLE_BACKGROUND_REPORT_ENABLED = False
class _JobUser:
"""最小 CurrentUser：满足 user_context 的 ``.id`` 结构协议。"""
@@ -36,6 +46,115 @@ class _JobUser:
self.id = user_id
# 头像配色：与前端 getAvatarTypeFor 对齐（内置圆桌席位固定色，其余按序轮换）。
_BUILTIN_AVATAR = {
"roundtable-intelligence": "purple",
"roundtable-environment": "emerald",
"roundtable-solution-design": "human",
"roundtable-risk-review": "amber",
"roundtable-execution-plan": "pink",
"roundtable-summary": "blue",
}
_FALLBACK_AVATAR = ["blue", "purple", "emerald", "amber", "pink"]
def _avatar_for(agent_id: str, index: int) -> str:
return _BUILTIN_AVATAR.get(agent_id) or _FALLBACK_AVATAR[index % len(_FALLBACK_AVATAR)]
# 工具步骤卡的回退文案。前端 getStepDisplay 对 tool 步骤是从 key+args 推图标/文案的,
# 存的 label 基本不展示——这里给个合理回退即可（与前端默认文案对齐）。
_STEP_LABEL = {
"agent_orchestration": "派活给子智能体",
"write_file": "写入文件",
"str_replace": "写入文件",
"read_file": "读取文件",
"ls": "列出文件夹",
"bash": "执行命令",
"web_search": "搜索相关信息",
"ask_clarification": "需要你的协助",
"present_files": "展示文件",
}
def _steps_from_tool_calls(tool_calls: list[dict], dialogue_id: object, ts: str) -> list[dict]:
"""把一轮 tool_calls 映射成前端 ``Step2Dialogue.steps``（工具步骤卡）。
- ``key`` = ``tool_calling:<toolName>``——前端 ``getStepDisplay`` 据此推图标 + 文案;
- ``id`` = ``<dialogueId>-<i>``——对齐前端 ``${bubbleId}-${stepIndex}``;
- ``args`` 原样带上（write_file 的 path/content、派活的 agent_name/task 等）。
"""
steps: list[dict] = []
for i, tc in enumerate(tool_calls or []):
name = tc.get("name") or ""
if not name:
continue
steps.append(
{
"id": f"{dialogue_id}-{i}",
"key": f"tool_calling:{name}",
"label": _STEP_LABEL.get(name, f"使用 “{name}” 工具"),
"ts": ts,
"args": tc.get("args") or {},
}
)
return steps
def _seat_stub_messages(dialogues: list[dict]) -> dict[str, list[dict]]:
"""从各对话的 write_file 调用重建 ``seatStubMessages``（threadId → [stub ai message]）。
对齐前端 ``upsertSeatToolCall`` 的 stub 形状与 id 约定（消息 ``roundtable-stub-<tid>``、
工具调用 ``tc-<tid>-<i>``），使加载草稿后点开产物的虚拟 URL 能正确解析到内容。
同一 thread 跨轮写多次按 path 去重（后写覆盖），与前端一致。
"""
by_tid: dict[str, dict[str, dict]] = {}
for d in dialogues:
tid = d.get("threadId")
if not tid:
continue
for tc in d.get("toolCalls") or []:
if tc.get("name") != "write_file":
continue
path = str((tc.get("args") or {}).get("path") or "")
by_tid.setdefault(tid, {})[path] = tc # 后写覆盖
out: dict[str, list[dict]] = {}
for tid, by_path in by_tid.items():
tool_calls = [
{"id": f"tc-{tid}-{i}", "name": "write_file", "args": tc.get("args") or {}}
for i, tc in enumerate(by_path.values())
]
out[tid] = [{"id": f"roundtable-stub-{tid}", "type": "ai", "content": "", "tool_calls": tool_calls}]
return out
def _to_step2_dialogue(d: dict, idx_by_agent: dict[str, int]) -> dict:
"""把作业的中性对话记录映射成前端 Step2Dialogue 形状（含 avatarType / steps /
agentThreadId / time），使写进草稿 step2.runs 后，前端 hydrateStep2Run 能像手动跑
一样直接渲染——步骤卡、时间戳、点开产物全部可用。"""
dialogue_id = d.get("id")
time_str = d.get("time") or ""
base: dict = {
"id": dialogue_id,
"content": d.get("content") or "",
"time": time_str,
"streaming": False,
"steps": _steps_from_tool_calls(d.get("toolCalls") or [], dialogue_id, time_str),
}
thread_id = d.get("threadId")
if thread_id:
base["agentThreadId"] = thread_id
if d.get("role") == "leader":
return {**base, "sender": "总控协调", "avatarType": "blue", "confidence": "派活说明"}
aid = d.get("agentId") or ""
return {
**base,
"sender": d.get("name") or aid,
"avatarType": _avatar_for(aid, idx_by_agent.get(aid, 0)),
"confidence": "子智能体交付",
}
@dataclass
class StartParams:
job_id: str
@@ -49,6 +168,14 @@ class StartParams:
coordinator_name: str = "roundtable-coordinator"
# 关联草稿：作业完成后把 step3 报告写回该草稿，使「查看报告」可用。
draft_id: str | None = None
# 是否为「续跑」（resume 澄清）。首启=False → 真网关建全新线程（不复用前端线程，
# 避免与前端本地研讨的活跃 run 撞 409）；续跑=True → 复用作业自己的线程。
is_resume: bool = False
# chain 模式来源链条（写进草稿 run 用，展示）。
chain: dict[str, Any] | None = None
# 是否生成 Step3 报告。默认沿用全局开关（当前临时禁用，见 ROUNDTABLE_BACKGROUND_REPORT_ENABLED）。
# 测试可显式传 True 覆盖以验证报告链路。
enable_report: bool = ROUNDTABLE_BACKGROUND_REPORT_ENABLED
# 网关工厂：给定一次启动参数，返回一个 RoundtableGateway 实例。
@@ -79,9 +206,10 @@ class RoundtableJobExecutor:
thread_ids = params.thread_ids or {}
coordinator_name = params.coordinator_name
# 真网关需要先进程内建好 leader/seat 线程；模拟网关无 init_threads。
# 仅首启建线程（resume 时 params.thread_ids 已带，复用、不重建）。
# 首启（非 resume）始终建**全新线程** —— 绝不复用前端本地研讨的线程
# （那些线程可能有活跃 run，复用会撞 409）。resume 时复用作业自己的线程。
init = getattr(gateway, "init_threads", None)
if init is not None and not thread_ids:
if init is not None and not params.is_resume:
thread_ids, coordinator_name = await init()
try:
await self._store.update_progress(
@@ -90,7 +218,19 @@ class RoundtableJobExecutor:
except Exception: # noqa: BLE001
logger.warning("persist thread_ids for job %s failed", params.job_id, exc_info=True)
progress = JobProgressWriter(self._store, job_id=params.job_id, agents=params.agents)
# 每轮模型返回后，把对话作为一个带 jobId 的 run 写进草稿 step2.runs[]
# —— 和手动跑同一存取路径（draft_store.update_draft），加载草稿即可显示。
async def _on_dialogue(dialogues: list[dict], consensus: int) -> None:
await self._write_run_to_draft(
params, dialogues=dialogues, status="running", consensus=consensus, thread_ids=thread_ids
)
progress = JobProgressWriter(
self._store,
job_id=params.job_id,
agents=params.agents,
on_dialogue=_on_dialogue if (self._draft_store and params.draft_id) else None,
)
result = await run_orchestration(
gateway=gateway,
progress=progress,
@@ -101,24 +241,23 @@ class RoundtableJobExecutor:
seed_message=params.seed_message,
mode=params.mode,
intent_text=params.intent_text,
enable_report=params.enable_report,
)
# 完成且有报告：写回关联草稿的 step3 + 推进 furthest_step=3，
# 这样用户从历史「查看报告」加载草稿即可看到最终方案（沿用既有 hydrateStep3 路径）。
if (
result.status == "done"
and result.step3
and params.draft_id
and self._draft_store is not None
):
# 终态：把最终对话 + step3 + jobStatus 落进同一个 run（含 furthest_step=3）。
await self._write_run_to_draft(
params,
dialogues=result.dialogues,
status=result.status,
consensus=100 if result.status == "done" else 95,
step3=result.step3,
thread_ids=thread_ids,
)
# step3 也挂 job（API / 弹窗用）。
if result.status == "done" and result.step3:
try:
await self._draft_store.update_draft(
params.draft_id,
params.user_id,
step3=result.step3,
furthest_step=3,
)
except Exception: # noqa: BLE001 — 写报告失败不影响作业本身已 done
logger.warning("persist step3 to draft %s failed", params.draft_id, exc_info=True)
await self._store.update_progress(params.job_id, step3=result.step3)
except Exception: # noqa: BLE001
logger.warning("persist step3 to job %s failed", params.job_id, exc_info=True)
except asyncio.CancelledError:
# 干净停止即可，**不在被取消的任务里写 DB**：取消态下二次 await 会再抛
# CancelledError、并可能打断进行中的 sqlite 连接。``cancelled`` 终态由
@@ -130,6 +269,81 @@ class RoundtableJobExecutor:
finally:
reset_current_user(token)
async def _write_run_to_draft(
self,
params: StartParams,
*,
dialogues: list[dict],
status: str,
consensus: int,
step3: dict | None = None,
thread_ids: dict[str, str] | None = None,
) -> None:
"""把本次后台作业作为一个带 jobId 的 run 写进草稿 step2.runs[]（同手动跑存取）。
读现有 step2 → 按 run_id=``job-<jobId>`` upsert（保留其它手动 run）→ 设为 activeRun
→ update_draft。前端加载草稿即可像手动跑一样 hydrate 出对话。
"""
if self._draft_store is None or not params.draft_id:
return
idx_by_agent = {a.agent_id: i for i, a in enumerate(params.agents)}
s2_dialogues = [_to_step2_dialogue(d, idx_by_agent) for d in dialogues]
last_leader = next(
(d.get("content") for d in reversed(dialogues) if d.get("role") == "leader"), ""
)
run_id = f"job-{params.job_id}"
try:
draft = await self._draft_store.get_draft(params.draft_id, params.user_id)
except Exception: # noqa: BLE001
draft = None
step2 = dict((draft or {}).get("step2") or {})
runs = list(step2.get("runs") or [])
idx = next((i for i, r in enumerate(runs) if r.get("id") == run_id), -1)
created = runs[idx].get("createdAt") if idx >= 0 else datetime.now(UTC).isoformat()
prev_step3 = runs[idx].get("step3") if idx >= 0 else None
# 本次有效 step3：新报告优先，否则保留上次（running 中间态 step3=None 不抹掉已存的）。
effective_step3 = step3 if step3 is not None else prev_step3
run = {
"id": run_id,
"title": "后台研讨",
"createdAt": created,
"source": params.mode,
"selectedAgents": [{"agent_id": a.agent_id, "name": a.name} for a in params.agents],
# 后台真建的 leader/seat 线程是真实持久的——存进去，前端加载后能像手动跑一样
# 按 thread 打开各席位产物（HTTP outputs 兜底）。hydrate 是纯展示、绝不 resume，
# 不会触发重跑（init 闸门 + isRoundTableRunning=false 双重拦截）。空则存 None。
"threadIds": (thread_ids or None),
"coordinatorName": params.coordinator_name,
"step2RoundtableDialogues": s2_dialogues,
"lastLeaderContent": last_leader or "",
"hasConsensus": status == "done",
"consensusPercentage": int(consensus),
"budgetLimit": 0,
# 从各席位 write_file 调用重建 stub，使加载草稿后点开产物的虚拟预览可用。
"seatStubMessages": _seat_stub_messages(dialogues),
"orchestrationMode": params.mode,
"chain": params.chain,
"step3": effective_step3,
"jobId": params.job_id,
"jobStatus": status,
}
runs = [run if i == idx else r for i, r in enumerate(runs)] if idx >= 0 else [*runs, run]
step2["runs"] = runs
step2["activeRunId"] = run_id
# 顶层 draft.step3 也写一份（同 Step2 把对话写进 run 的同一次落库一并带上）。
# 前端 hydrateStep3 用**顶层** draft.step3 重建报告气泡（结果绘制的聊天记录 + HTML 预览）；
# 只存 run 内 step3 时，加载会被 hydrateStep3(draft.step3=None) 覆盖成空 → 报告聊天记录丢失。
# furthest_step 跟随**是否真有 step3**：done 且有报告→3（落到结果绘制）；
# 否则停在 2（Step2 研讨）。后台禁用报告时 step3=None → 停在 Step2，用户可在前端手动画报告。
has_report = effective_step3 is not None
update_kwargs: dict[str, Any] = {"step2": step2, "furthest_step": 3 if (status == "done" and has_report) else 2}
if has_report:
update_kwargs["step3"] = effective_step3
try:
await self._draft_store.update_draft(params.draft_id, params.user_id, **update_kwargs)
except Exception: # noqa: BLE001 — 写草稿失败不影响作业本身
logger.warning("write run to draft %s failed", params.draft_id, exc_info=True)
def cancel(self, job_id: str) -> bool:
"""请求取消一个在跑的作业。返回是否命中在跑任务。"""
task = self._tasks.get(job_id)
offline-backend-20260512/backend/app/gateway/roundtable_run_policy.py
+168
View File
@@ -0,0 +1,168 @@
"""圆桌各角色「单轮 run 参数」的**单一数据源**。
历史问题：前端手动路径（``multi_agent.py`` 的 ``_leader_run`` / ``_special_run``）
和后台进程内网关（``roundtable_inprocess_gateway.py``）各自硬编码了一份「每角色
工具裁剪 + thinking + reasoning」，结果**漂移**出差异（后台席位 / 报告少禁了
``web_search``，能联网搜索而前端不能）。
本模块把这套策略收敛成唯一定义，两条路径都从这里取，保证「后台挂起」与「前端
手动调用」传参**逐字一致**，且以后改一处即可，不再漂移。
角色（``role``）：
- ``leader`` —— 总控派活：短决策，关 thinking + low reasoning（首字节快）；
**只保留 ``agent_orchestration`` / ``ask_clarification`` / ``present_files``**，
其余「干活类」工具（``write_file`` / ``bash`` / ``str_replace`` / ``read_file`` /
``ls`` / ``view_image`` / ``task`` / ``write_todos`` / ``memory`` / ``hindsight_*``）
一律禁用——否则弱模型（如 deepseek-chat）会概率性地**自己写文件、跑 bash 干活**
而不是派活（已在生产日志里观测到总控 ``wc -l outputs/*.md`` 自检自己写的报告）。
总控的唯一职责是「派活 / 澄清 / 收口呈现」，没有任何理由动文件系统或执行命令。
并配合 ``SkillStopMiddleware`` 在 ``agent_orchestration`` 调用后停图，便于网关从
AIMessage.tool_calls 解析派活列表。
- ``seat`` —— 子智能体交付：开 thinking + medium reasoning；**可写文件**
（``write_file`` / ``str_replace`` 产出 md 等成品文档），但禁派活 / 澄清 /
展示文件 / 看图 / 联网搜索 / **``bash``**。禁 ``bash`` 的原因：圆桌席位是
「文档产出者」，弱模型爱写 ``.py`` 脚本再跑出产物（乱且抓不到预览），与
``report`` 角色禁 ``bash`` 同理；让它**直接 write_file 写成品文档**。
- ``seat_parallel`` —— **DAG 快速(fast)模式的并行席位（方案 A，2026-06）**：**禁写文件**
（``write_file`` / ``str_replace``，产物走正文）+ 禁 ``bash`` + 禁「打断 / 越权」类，
**放开 ``web_search`` 与技能**（搜索不碰 sandbox；技能靠 ``read_file`` SKILL.md）——
否则并行席位几乎调不动工具、产生不了步骤（步骤条不显示）。
「写文件产出 md」归 **沙箱(sandbox)模式**（走 ``seat`` policy，单列 + 沙箱跟随预览）；
fast 模式快、双列、产物走正文 —— 用户明确要求 fast 不生成文件。详见
``docs/roundtable-dag-orchestration.md`` §4 / §12.1 / §12.8 / §12.10。
与 ``seat`` 的差异：``seat`` 禁 ``web_search``（旧「圆桌一律禁搜索」），
``seat_parallel`` **放开搜索**（本期诉求）。
- ``report`` —— Step3 报告绘制：在 ``seat``（已禁 ``bash``）基础上**额外禁
``str_replace``**：
* ``bash`` 已在 ``seat`` 禁 → 杜绝写 Python 脚本再跑出 HTML（脚本产物落在
outputs，但前端虚拟预览只抓 ``write_file`` 的 ``args.content``，抓不到脚本
产物 → 预览停在半截 ``write_file``，正是「没写完就去生成 py 文件」的现象）；
* 禁 ``str_replace`` → 强制返工用整文件 ``write_file`` 重写
（``str_replace`` 没有 ``content`` 字段，会让前端虚拟预览读到 undefined
而清空沙箱）。
"""
from __future__ import annotations
from dataclasses import dataclass
# ── 每角色基础工具裁剪（唯一定义处）─────────────────────────────────────────────
#
# 注意：``seat`` / ``report`` 都包含 ``web_search`` —— 圆桌研讨里联网搜索一律禁用
# （与前端 ``_special_run`` 的 excluded_tools 对齐）。
#
# ``leader`` 用「白名单」思路反推黑名单：总控**只允许** agent_orchestration（派活）、
# ask_clarification（澄清）、present_files（收口呈现子智能体产出的文件）三件事，
# 因此把其余所有工具全部禁掉。关键是 write_file / bash / str_replace —— 留着它们，
# 弱模型会绕过派活直接自己写报告（生产日志已实锤）。memory / hindsight_* 一并禁，
# 配合 ``memory_injection_enabled=False`` 让单例总控不被上轮会话的记忆带偏席位。
_LEADER_EXCLUDED: list[str] = [
"web_search",
"view_image",
"bash",
"ls",
"read_file",
"write_file",
"str_replace",
"task",
"write_todos",
"memory",
"hindsight_recall",
"hindsight_reflect",
"hindsight_retain",
"read_peer_delivery", # 总控靠 _collect_delivery_status 直读各席位,不需要这个席位侧工具
]
# seat：席位**可以写文件**(write_file / str_replace)产出 md 等成品文档,但**禁 bash** ——
# 圆桌席位是「文档产出者」,不该跑 shell/python 脚本(弱模型爱写 .py 脚本再跑生成产物,既乱又
# 抓不到产物预览,与 report 角色禁 bash 同理)。禁 web_search(圆桌串行一律禁搜索,旧决策)。
_SEAT_EXCLUDED: list[str] = [
"web_search",
"ask_clarification",
"present_files",
"view_image",
"agent_orchestration",
"bash",
]
# 并行席位 = **快速(fast)模式**(双列):**禁写文件**(产物走正文)+ 禁跑命令 +「打断/越权」类。
# **禁 web_search** —— 离线内网部署无外网,席位检索一律走「技能(skill)」(靠 read_file 读
# SKILL.md,不联网),与 leader / seat / report 三角色保持一致(2026-06 离线部署决策)。
# 「写文件」归 **沙箱(sandbox)模式**(seat policy,单列 + 沙箱跟随预览);fast 模式快、双列、
# 产物走正文 —— 用户明确要求 fast 不生成文件(2026-06 §12.10)。
_SEAT_PARALLEL_EXCLUDED: list[str] = [
"web_search", # 离线部署无外网,检索走技能(skill),不联网
"ask_clarification", # 并行批次不打断问澄清
"present_files", # 产物走正文,不呈现文件
"view_image",
"agent_orchestration", # 席位不派活
"bash", # 禁跑脚本(防 .py 脚本生成产物)
"write_file", # fast 模式产物走正文,不写文件(写文件归 sandbox 模式)
"str_replace",
]
# report：在 seat 基础上额外禁 str_replace(bash 已在 seat 禁)——强制整文件 write_file 重写,
# 让前端虚拟预览能拿到完整 args.content。report 是 Step3 单独绘制,不读前序席位 → 禁 read_peer_delivery。
_REPORT_EXCLUDED: list[str] = [*_SEAT_EXCLUDED, "str_replace", "read_peer_delivery"]
# leader 内禀的 skill_stop：``agent_orchestration`` 调用后停图。调用方可在此基础上
# 再叠加自己的 skill_stop（见 ``merge_skill_stop_names``）。
_LEADER_SKILL_STOP: list[str] = ["agent_orchestration"]
@dataclass(frozen=True)
class RoundtableRunPolicy:
"""一个角色单轮 run 的归一化参数。
与 ``multi_agent._run_payload`` 的入参一一对应：``excluded_tools`` /
``skill_stop_names`` / ``thinking_enabled`` / ``reasoning_effort``。
"""
excluded_tools: list[str]
skill_stop_names: list[str]
thinking_enabled: bool
reasoning_effort: str
def roundtable_run_policy(role: str) -> RoundtableRunPolicy:
"""返回某角色（``leader`` / ``seat`` / ``seat_parallel`` / ``report``）的单轮 run 参数。
返回值里的列表是**新副本**，调用方可安全地再 append/merge 而不污染本模块常量。
未知角色抛 ``ValueError``（早失败，避免悄悄退化到错误的工具面）。
"""
if role == "leader":
return RoundtableRunPolicy(
excluded_tools=list(_LEADER_EXCLUDED),
skill_stop_names=list(_LEADER_SKILL_STOP),
thinking_enabled=False,
reasoning_effort="low",
)
if role == "seat":
return RoundtableRunPolicy(
excluded_tools=list(_SEAT_EXCLUDED),
skill_stop_names=[],
thinking_enabled=True,
reasoning_effort="medium",
)
if role == "seat_parallel":
# 同 seat（thinking + medium），但额外禁写文件 / 跑命令（防并行沙箱打架）。
return RoundtableRunPolicy(
excluded_tools=list(_SEAT_PARALLEL_EXCLUDED),
skill_stop_names=[],
thinking_enabled=True,
reasoning_effort="medium",
)
if role == "report":
return RoundtableRunPolicy(
excluded_tools=list(_REPORT_EXCLUDED),
skill_stop_names=[],
thinking_enabled=True,
reasoning_effort="medium",
)
raise ValueError(f"unknown roundtable role: {role!r} (expected leader/seat/seat_parallel/report)")
def merge_skill_stop_names(policy_names: list[str], caller_names: list[str] | None) -> list[str]:
"""把角色内禀 skill_stop 与调用方追加的 skill_stop 合并去重（保序）。
取代历史上 ``list({"agent_orchestration", *skill_stop_names})`` 的 set 写法
（set 顺序不确定）；这里用 dict.fromkeys 保序去重，结果确定、便于断言。
"""
return list(dict.fromkeys([*policy_names, *(caller_names or [])]))
offline-backend-20260512/backend/app/gateway/routers/_roundtable_seed_assets/roundtable-coordinator/SOUL.md
+10 -5
View File
@@ -11,15 +11,18 @@
通过 `agent_orchestration` 技能调用其他智能体干活,使用技能时传入 `agent_name` 和需要做的活。
**⚠️ 关于 `agent_name` 的硬性规则**:
- 必须严格使用**本次圆桌已上线席位的英文 id**(例如 `roundtable-intelligence`,或形如 `02714cbe6e64406a8a4659f0e3d9c0b8` 这样的 UUID)
- 本次可调度的席位清单会以「【圆桌系统初始化】本次可调度席位:...」这条人类消息的形式出现在你的对话历史最前面——**只能从这份清单里挑**
- 不要把括号里的中文显示名拼进 `agent_name`;不要派给清单上没有的 id
- `agent_name` 必须**逐字符照抄**本次席位清单里给出的 id 原文(可能是 `roundtable-intelligence` 这样的可读 id,也可能是一串 32 位十六进制字符)
- 本次可调度的席位清单会以「【圆桌系统初始化】本次可调度席位:...」这条人类消息的形式出现在你的对话历史最前面——**只能从这份清单里逐字复制 id**
- **绝对禁止凭记忆、猜测,或参考本说明里的任何示例文字自行拼造 id**;清单里没有的 id 一律不准派
- 不要把括号里的中文显示名拼进 `agent_name`
## 关键约束(必读)
1. **不要对同一子智能体重复派同样的活**:每次派活前回顾对话历史,如果某个子智能体已经针对你正要派的任务完成过交付(看到形如「子智能体[名称]完成XX工作,交付内容为:...」的 human 消息),就直接基于已有交付推进,**不要重复派活**,除非你确实有**具体的、不一样的**补充问题
2. **看到交付要会用**:子智能体的交付内容会作为 human 消息追加到你的对话历史里,你能完整看见。当各方信息已足够形成结论时,**直接综合输出最终方案**(此时不再调用任何工具),让讨论收敛
3. **避免无意义循环**:如果连续两轮都没有新信息流入,立刻停止派活并基于现有素材给出最终结论
4. **数量与节奏**:每轮派活 1-3 个席位最佳;不需要每个席位都派一次
3. **⛔ 严禁臆造交付**:你**只能**把对话历史里**确实出现过**的「子智能体[名称]完成XX工作,交付内容为:...」当作已交付。**绝对禁止**在没有看到这条消息的情况下声称某席位「已交付」、编造它的交付内容、或假装研讨已完成。**没看到该席位的真实交付,就一律视为「未交付」**——该派活就派活,绝不跳过直接收口。宁可多派一轮,也不许凭空说「三方已交付」。
4. **用户点名某席位就去派活**:当用户(含人工干预指令)明确希望由某个席位来回答/产出(例如"我想看前端实现,席位里有前端实现智能体"),你**必须**通过 `agent_orchestration` 把活派给那个真实席位,**不要**自己代答、也不要直接给最终结论搪塞。
5. **避免无意义循环**:如果连续两轮都没有新信息流入,立刻停止派活并基于现有素材给出最终结论
6. **数量与节奏**:每轮派活 1-3 个席位最佳;不需要每个席位都派一次
## 终止条件
当满足以下任一条件,**直接输出最终结论文本**(不调用任何工具),让本次圆桌收敛:
@@ -40,3 +43,5 @@
- ❌ 派活给本次清单**没有**的 agent_name
- ❌ 把括号里的中文显示名拼进 `agent_name`
- ❌ 已经达成共识后仍然继续派活
- ❌ **臆造 / 假装某席位已交付**——没在对话历史里看到它的「完成…交付内容为:…」就不许说它交付了
- ❌ 用户点名要某席位回答时,自己代答或直接收口而**不去派活**
offline-backend-20260512/backend/app/gateway/routers/admin_users.py
+2301 -1
View File
File diff suppressed because it is too large. Load diff
offline-backend-20260512/backend/app/gateway/routers/agents.py
+266 -76
View File
@@ -15,7 +15,7 @@ from fastapi import APIRouter, Depends, HTTPException, Query, Request
from pydantic import BaseModel, Field
from app.gateway.auth.password import verify_password_async
from app.gateway.deps import get_agent_store, get_config, get_tag_store
from app.gateway.deps import get_agent_store, get_config, get_tag_store
from deerflow.agents.lead_agent.prompt import refresh_skills_system_prompt_cache_async
from deerflow.config.agents_api_config import get_agents_api_config
from deerflow.config.agents_config import list_custom_agents, load_agent_config, load_agent_soul, validate_agent_id
@@ -62,13 +62,27 @@ class AgentResponse(BaseModel):
is_favorite: bool = Field(
default=False,
description="Whether the requesting user has favorited (收藏) this agent.",
)
)
is_pinned: bool = Field(
default=False,
description="Whether the requesting user has pinned this agent.",
)
class AgentsListResponse(BaseModel):
"""Response model for listing agents.
class AgentsListResponse(BaseModel):
"""Response model for listing all visible agents."""
``total`` / ``page`` / ``page_size`` describe server-side pagination. When
a caller does not opt into pagination they default to "everything on one
page" (``total == len(agents)``, ``page == 1``), so existing unpaginated
callers keep working unchanged.
"""
agents: list[AgentResponse]
total: int = Field(default=0, description="Total agents matching the query across all pages")
page: int = Field(default=1, description="1-based page index of this response")
page_size: int = Field(default=0, description="Page size; 0 means the result was not paginated")
class AgentCreateRequest(BaseModel):
@@ -101,7 +115,7 @@ class AgentUpdateRequest(BaseModel):
tool_groups: list[str] | None = Field(default=None, description="Updated tool group whitelist (config.yaml)")
class AgentSelectorResponse(BaseModel):
class AgentSelectorResponse(BaseModel):
"""Response for the homepage agent selector dropdown.
The list is fully curated by admins (``featured_order`` on each row).
@@ -111,10 +125,10 @@ class AgentSelectorResponse(BaseModel):
agents: list[AgentResponse] = Field(
default_factory=list,
description="Admin-curated agents, sorted by featured_order ascending.",
)
class AgentFeaturedRequest(BaseModel):
)
class AgentFeaturedRequest(BaseModel):
"""Request body for setting/clearing an agent's homepage featured rank."""
order: int | None = Field(
@@ -215,6 +229,7 @@ async def _sync_legacy_agents(store: AgentStore) -> None:
handlers as a safety net for environments that bypassed the lifespan hook.
"""
try:
pairs: list[tuple[str, Any]] = []
for agent_cfg in list_custom_agents():
agent_id = agent_cfg.id or agent_cfg.name
if not agent_id:
@@ -227,10 +242,41 @@ async def _sync_legacy_agents(store: AgentStore) -> None:
"published": True,
}
)
pairs.append((agent_id, agent_cfg))
await _backfill_listing_cache(store, pairs)
except Exception:
logger.exception("Failed to sync legacy folder-backed agents")
async def _backfill_listing_cache(store: AgentStore, pairs: list[tuple[str, Any]]) -> None:
"""Mirror each agent's config.yaml skills/tool_groups/model into the DB cache.
The list endpoint reads skills/tool_groups from these cache rows instead of
stat+parsing every ``config.yaml`` per request. We fetch the current cache
in one batch and only re-write rows whose values actually changed, so a
steady-state sync (every ``_LEGACY_SYNC_TTL_SECONDS``) is cheap and does not
churn DELETE/INSERTs. For the cache, ``None`` and ``[]`` skills are
equivalent ("no skills") — the tri-state only matters to the agent runtime,
which reads config.yaml directly.
"""
if not pairs:
return
cached = await store.get_extras_for([aid for aid, _ in pairs])
for agent_id, cfg in pairs:
want_skills = [s for s in (cfg.skills or []) if isinstance(s, str)]
want_tool_groups = cfg.tool_groups
want_model = cfg.model
existing = cached.get(agent_id)
if existing is None or existing.get("skills") != want_skills:
await store.replace_skills(agent_id, want_skills)
if (
existing is None
or existing.get("tool_groups") != want_tool_groups
or existing.get("model") != want_model
):
await store.set_extras(agent_id, tool_groups=want_tool_groups, model=want_model)
# Throttle the per-request legacy-sync fallback. The real sync runs once at
# startup (lifespan); request handlers used to call ``_sync_legacy_agents`` on
# every hit, which scanned the filesystem + did N DB upserts per agent each
@@ -302,7 +348,7 @@ def _ensure_agent_dir_from_record(record: dict[str, Any]) -> None:
logger.exception("Failed to self-heal agent dir for '%s'", agent_id)
def _record_to_response(
def _record_to_response(
record: dict[str, Any],
*,
include_soul: bool = False,
@@ -344,10 +390,21 @@ def _record_to_response(
tool_groups=extras["tool_groups"],
skills=extras["skills"],
featured_order=record.get("featured_order"),
)
async def _record_to_response_async(
)
def _sort_agent_responses(responses: list[AgentResponse], sort: str) -> list[AgentResponse]:
reverse = sort not in {"time_asc", "updated_asc", "name_asc"}
if sort in {"name_asc", "name_desc"}:
ordered = sorted(responses, key=lambda r: (r.name or "").lower(), reverse=reverse)
elif sort in {"updated_desc", "updated_asc"}:
ordered = sorted(responses, key=lambda r: str(r.updated_at or ""), reverse=reverse)
else:
ordered = sorted(responses, key=lambda r: str(r.created_at or ""), reverse=reverse)
return sorted(ordered, key=lambda r: 0 if r.is_pinned else 1)
async def _record_to_response_async(
record: dict[str, Any],
*,
include_soul: bool = False,
@@ -403,6 +460,40 @@ async def _get_editable_or_raise(
raise HTTPException(status_code=403, detail="You can only modify agents that you created")
async def _build_list_responses(
store: AgentStore,
tag_store: TagStore,
user_id: str,
records: list[dict[str, Any]],
) -> list[AgentResponse]:
"""Build gallery responses for a set of agent rows WITHOUT touching disk.
skills / tool_groups / model come from the denormalized DB cache
(``get_extras_for`` — one batched query) instead of stat+parsing every
agent's ``config.yaml`` per request, which was the list-endpoint hot spot.
SOUL.md is skipped (only the edit dialog reads it). An agent absent from
the cache (never mirrored) keeps its ``None`` extras until the next sync.
"""
responses = [
_record_to_response(a, include_soul=False, include_extras=False) for a in records
]
ids = [r.id for r in responses]
extras = await store.get_extras_for(ids)
assignments = await tag_store.list_assignments("agent", ids)
favorite_ids = await store.list_favorite_ids(user_id)
pin_ids = await store.list_pin_ids(user_id)
for resp in responses:
cached = extras.get(resp.id)
if cached is not None:
resp.skills = cached.get("skills")
resp.tool_groups = cached.get("tool_groups")
resp.model = cached.get("model")
resp.tags = assignments.get(resp.id, [])
resp.is_favorite = resp.id in favorite_ids
resp.is_pinned = resp.id in pin_ids
return responses
@router.get(
"/agents",
response_model=AgentsListResponse,
@@ -411,9 +502,21 @@ async def _get_editable_or_raise(
)
async def list_agents(
request: Request,
search: str | None = Query(default=None, description="Case-insensitive fuzzy match on agent name"),
search: str | None = Query(default=None, description="Case-insensitive fuzzy match on agent name (raw or desensitized display form)"),
tag_ids: list[str] | None = Query(default=None, description="Keep agents carrying any of these tags (OR)"),
admin: bool = Query(
scope: str | None = Query(
default=None,
pattern="^(mine|square)$",
description="Pagination scope. 'mine' = owned + favorited; 'square' = built-in/published public. Omit for the default combined visibility list.",
),
page: int | None = Query(default=None, ge=1, description="1-based page index. Providing this (or 'scope') enables server-side pagination."),
page_size: int = Query(default=20, ge=1, le=200, description="Items per page when paginating."),
sort: str = Query(
default="time_desc",
pattern="^(time_desc|time_asc|updated_desc|updated_asc|name_asc|name_desc)$",
description="Sort order. Default is newest created first.",
),
admin: bool = Query(
default=False,
description=(
"Admin-only override. When ``true`` and the caller is a system admin, "
@@ -429,38 +532,38 @@ async def list_agents(
tag_store: TagStore = get_tag_store(request)
await _maybe_sync_legacy_agents(store)
user_id = _current_user_id(request)
if admin:
if not _is_admin_user(request):
raise HTTPException(status_code=403, detail="Only system admins can list all agents")
agents = await store.list_all()
else:
agents = await store.list_visible(user_id)
# Load lightweight extras (skills / model / tool_groups from config.yaml)
# so the gallery cards can render skill badges. SOUL.md is still skipped
# (only the edit dialog reads it). The per-agent config.yaml reads run in
# worker threads concurrently, so the event loop stays responsive even
# with many agents.
responses = list(
await asyncio.gather(
*(
_record_to_response_async(a, include_soul=False, include_extras=True)
for a in agents
)
)
)
if admin and not _is_admin_user(request):
raise HTTPException(status_code=403, detail="Only system admins can list all agents")
assignments = await tag_store.list_assignments("agent", [r.id for r in responses])
favorite_ids = await store.list_favorite_ids(user_id)
for resp in responses:
resp.tags = assignments.get(resp.id, [])
resp.is_favorite = resp.id in favorite_ids
paginate = scope is not None or page is not None
if paginate:
eff_page = page or 1
rows, total = await store.list_paginated(
user_id=user_id,
scope=scope,
search=search,
tag_ids=tag_ids,
page=eff_page,
page_size=page_size,
admin=admin,
sort=sort,
)
responses = await _build_list_responses(store, tag_store, user_id, rows)
return AgentsListResponse(agents=responses, total=total, page=eff_page, page_size=page_size)
# Legacy unpaginated path — returns everything (admin console, callers
# that have not migrated to pagination). Extras now come from the DB
# cache too, so this path no longer reads config.yaml per agent.
agents = await store.list_all() if admin else await store.list_visible(user_id)
responses = await _build_list_responses(store, tag_store, user_id, agents)
if search:
needle = search.strip().lower()
responses = [r for r in responses if needle in r.name.lower()]
if tag_ids:
wanted = set(tag_ids)
responses = [r for r in responses if any(t.get("id") in wanted for t in r.tags)]
return AgentsListResponse(agents=responses)
if tag_ids:
wanted = set(tag_ids)
responses = [r for r in responses if any(t.get("id") in wanted for t in r.tags)]
responses = _sort_agent_responses(responses, sort)
return AgentsListResponse(agents=responses, total=len(responses), page=1, page_size=len(responses))
except HTTPException:
raise
except Exception as e:
@@ -473,14 +576,14 @@ async def list_agents(
summary="Check Agent Name",
description="Validate an agent display name. Duplicate names are allowed.",
)
async def check_agent_name(name: str) -> dict[str, Any]:
_require_agents_api_enabled()
normalized = _validate_display_name(name)
return {"available": True, "name": normalized}
@router.get(
"/agents/selector",
async def check_agent_name(name: str) -> dict[str, Any]:
_require_agents_api_enabled()
normalized = _validate_display_name(name)
return {"available": True, "name": normalized}
@router.get(
"/agents/selector",
response_model=AgentSelectorResponse,
summary="Agent Homepage Selector",
description=(
@@ -532,10 +635,12 @@ async def get_agent(agent_id: str, request: Request) -> AgentResponse:
# Detail view legitimately reads SOUL.md + config.yaml — hop to a
# worker thread so the asyncio loop stays responsive under concurrent
# detail fetches.
response = await _record_to_response_async(record, include_soul=True, include_extras=True)
favorite_ids = await store.list_favorite_ids(user_id)
response.is_favorite = agent_id in favorite_ids
return response
response = await _record_to_response_async(record, include_soul=True, include_extras=True)
favorite_ids = await store.list_favorite_ids(user_id)
pin_ids = await store.list_pin_ids(user_id)
response.is_favorite = agent_id in favorite_ids
response.is_pinned = agent_id in pin_ids
return response
except HTTPException:
raise
except Exception as e:
@@ -587,6 +692,10 @@ async def create_agent_endpoint(request: AgentCreateRequest, http_request: Reque
"is_private": request.is_private,
}
)
# Mirror skills/tool_groups/model into the listing cache so the gallery
# can render this agent's badges without reading its config.yaml.
await store.replace_skills(agent_id, request.skills)
await store.set_extras(agent_id, tool_groups=request.tool_groups, model=request.model)
logger.info("Created agent '%s' (%s) at %s", name, agent_id, agent_dir)
return await _record_to_response_async(record, include_soul=True, include_extras=True)
@@ -653,6 +762,9 @@ async def update_agent(agent_id: str, request: AgentUpdateRequest, http_request:
model=next_model,
tool_groups=next_tool_groups,
)
# Keep the listing cache in lock-step with config.yaml on every write.
await store.replace_skills(agent_id, next_skills)
await store.set_extras(agent_id, tool_groups=next_tool_groups, model=next_model)
if request.soul is not None:
_write_agent_soul(agent_id, request.soul)
@@ -762,9 +874,17 @@ async def delete_agent(agent_id: str, request: Request) -> None:
await get_tag_store(request).unassign_all("agent", agent_id)
except Exception: # noqa: BLE001 — tag cleanup must not block deletion
pass
try:
await store.delete_favorites_for_agent(agent_id)
except Exception: # noqa: BLE001 — favorite cleanup must not block deletion
try:
await store.delete_favorites_for_agent(agent_id)
except Exception: # noqa: BLE001 — favorite cleanup must not block deletion
pass
try:
await store.delete_pins_for_agent(agent_id)
except Exception: # noqa: BLE001 - pin cleanup must not block deletion
pass
try:
await store.delete_extras_for_agent(agent_id)
except Exception: # noqa: BLE001 — cache cleanup must not block deletion
pass
logger.info("Deleted agent '%s' from %s", agent_id, agent_dir)
except HTTPException:
@@ -805,7 +925,7 @@ async def favorite_agent(agent_id: str, request: Request) -> None:
summary="Unfavorite Agent",
description="Remove an agent from the current user's favorites. No-op when not favorited.",
)
async def unfavorite_agent(agent_id: str, request: Request) -> None:
async def unfavorite_agent(agent_id: str, request: Request) -> None:
_require_agents_api_enabled()
agent_id = _validate_storage_id(agent_id)
@@ -816,13 +936,60 @@ async def unfavorite_agent(agent_id: str, request: Request) -> None:
logger.info("User '%s' unfavorited agent '%s'", user_id, agent_id)
except HTTPException:
raise
except Exception as e:
logger.error("Failed to unfavorite agent '%s': %s", agent_id, e, exc_info=True)
raise HTTPException(status_code=500, detail=f"Failed to unfavorite agent: {str(e)}") from e
@router.put(
"/agents/{agent_id}/featured",
except Exception as e:
logger.error("Failed to unfavorite agent '%s': %s", agent_id, e, exc_info=True)
raise HTTPException(status_code=500, detail=f"Failed to unfavorite agent: {str(e)}") from e
@router.post(
"/agents/{agent_id}/pin",
status_code=204,
summary="Pin Agent",
description="Pin a visible agent to the top of the current user's lists. Idempotent.",
)
async def pin_agent(agent_id: str, request: Request) -> None:
_require_agents_api_enabled()
agent_id = _validate_storage_id(agent_id)
try:
store = get_agent_store(request)
user_id = _current_user_id(request)
record = await store.get_visible(agent_id, user_id)
if record is None:
raise HTTPException(status_code=404, detail=f"Agent '{agent_id}' not found")
await store.add_pin(user_id, agent_id)
logger.info("User '%s' pinned agent '%s'", user_id, agent_id)
except HTTPException:
raise
except Exception as e:
logger.error("Failed to pin agent '%s': %s", agent_id, e, exc_info=True)
raise HTTPException(status_code=500, detail=f"Failed to pin agent: {str(e)}") from e
@router.delete(
"/agents/{agent_id}/pin",
status_code=204,
summary="Unpin Agent",
description="Remove the current user's pin from an agent. No-op when not pinned.",
)
async def unpin_agent(agent_id: str, request: Request) -> None:
_require_agents_api_enabled()
agent_id = _validate_storage_id(agent_id)
try:
store = get_agent_store(request)
user_id = _current_user_id(request)
await store.remove_pin(user_id, agent_id)
logger.info("User '%s' unpinned agent '%s'", user_id, agent_id)
except HTTPException:
raise
except Exception as e:
logger.error("Failed to unpin agent '%s': %s", agent_id, e, exc_info=True)
raise HTTPException(status_code=500, detail=f"Failed to unpin agent: {str(e)}") from e
@router.put(
"/agents/{agent_id}/featured",
response_model=AgentResponse,
summary="Set Agent Featured Rank (admin)",
description=(
@@ -864,10 +1031,19 @@ async def set_agent_featured(
raise HTTPException(status_code=500, detail=f"Failed to set featured: {str(e)}") from e
class PrivateSquareListRequest(BaseModel):
class PrivateSquareListRequest(BaseModel):
"""Request body for listing all private-square agents."""
password: str = Field(..., description="Plain-text private-square password. Verified against the admin-configured hash before the listing is returned.")
search: str | None = Field(default=None, description="Case-insensitive fuzzy match on agent name (raw or desensitized).")
tag_ids: list[str] | None = Field(default=None, description="Keep agents carrying any of these tags (OR).")
page: int | None = Field(default=None, ge=1, description="1-based page index. Providing this enables server-side pagination.")
page_size: int = Field(default=20, ge=1, le=200, description="Items per page when paginating.")
sort: str = Field(
default="time_desc",
pattern="^(time_desc|time_asc|updated_desc|updated_asc|name_asc|name_desc)$",
description="Sort order. Default is newest created first.",
)
class AgentPrivateUpdateRequest(BaseModel):
@@ -899,14 +1075,28 @@ async def list_private_square(body: PrivateSquareListRequest, request: Request)
try:
store = get_agent_store(request)
tag_store: TagStore = get_tag_store(request)
await _sync_legacy_agents(store)
records = await store.list_private()
responses = [_record_to_response(r, include_soul=True) for r in records]
await _maybe_sync_legacy_agents(store)
# user_id is unused by the "private" scope filter (the password is the
# gate), but list_paginated requires it for its signature.
user_id = _current_user_id(request)
assignments = await tag_store.list_assignments("agent", [r.id for r in responses])
for resp in responses:
resp.tags = assignments.get(resp.id, [])
return AgentsListResponse(agents=responses)
if body.page is not None:
rows, total = await store.list_paginated(
user_id=user_id,
scope="private",
search=body.search,
tag_ids=body.tag_ids,
page=body.page,
page_size=body.page_size,
sort=body.sort,
)
responses = await _build_list_responses(store, tag_store, user_id, rows)
return AgentsListResponse(agents=responses, total=total, page=body.page, page_size=body.page_size)
records = await store.list_private()
responses = await _build_list_responses(store, tag_store, user_id, records)
responses = _sort_agent_responses(responses, body.sort)
return AgentsListResponse(agents=responses, total=len(responses), page=1, page_size=len(responses))
except HTTPException:
raise
except Exception as e:
offline-backend-20260512/backend/app/gateway/routers/auth.py
+13 -3
View File
@@ -22,7 +22,7 @@ from app.gateway.auth import (
from app.gateway.auth.config import get_auth_config
from app.gateway.auth.errors import AuthErrorCode, AuthErrorResponse
from app.gateway.csrf_middleware import is_secure_request
from app.gateway.deps import get_agent_store, get_current_user_from_request, get_local_provider, get_skill_store
from app.gateway.deps import get_agent_store, get_current_user_from_request, get_local_provider, get_run_store, get_skill_store
logger = logging.getLogger(__name__)
@@ -336,7 +336,7 @@ def _set_session_cookie(response: Response, token: str, request: Request) -> Non
# production multi-worker setups, replace this with a shared store (Redis,
# database-backed counter) to enforce a true per-IP limit.
_MAX_LOGIN_ATTEMPTS = 5
_MAX_LOGIN_ATTEMPTS = 100
_LOCKOUT_SECONDS = 300 # 5 minutes
# ip → (fail_count, lock_until_timestamp)
@@ -749,6 +749,7 @@ class UserListItem(BaseModel):
system_role: Literal["admin", "user"]
skill_count: int
agent_count: int
qa_count: int = 0 # 问答次数: number of runs (one run = one question)
@router.get("/users", response_model=list[UserListItem])
@@ -757,11 +758,19 @@ async def list_all_users(
skill_store: SkillStore = Depends(get_skill_store),
agent_store: AgentStore = Depends(get_agent_store),
):
"""List all users with their skill and agent counts. Admin only."""
"""List all users with their skill, agent and Q&A counts. Admin only."""
user = await get_current_user_from_request(request)
if user.system_role != "admin":
raise HTTPException(status_code=403, detail="Admin only")
users = await get_local_provider().list_users()
# One GROUP BY query for everyone's 问答次数, so the list stays a single
# aggregate read instead of a per-user count. Best-effort: if the run store
# is unavailable the column simply renders as 0 rather than failing the list.
try:
qa_counts = await get_run_store(request).count_by_user()
except Exception:
logger.warning("Failed to load per-user run counts for user list", exc_info=True)
qa_counts = {}
result = []
for u in users:
uid = str(u.id)
@@ -772,6 +781,7 @@ async def list_all_users(
system_role=u.system_role,
skill_count=await skill_store.count_owned(uid),
agent_count=await agent_store.count_owned(uid),
qa_count=qa_counts.get(uid, 0),
)
)
return result
offline-backend-20260512/backend/app/gateway/routers/checkpoint_migration.py
+276
View File
@@ -0,0 +1,276 @@
"""Admin checkpoint migration endpoints.
These endpoints support the SQLite -> Postgres transition for LangGraph
checkpoints. The live checkpointer should be Postgres with SQLite dual-read
enabled; this router only starts manual, bounded migrations for selected time
ranges.
"""
from __future__ import annotations
import asyncio
import contextlib
import hashlib
import json
import logging
import os
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel, Field
from app.gateway.auth.disabled_mode import is_auth_disabled
from app.gateway.deps import get_current_user_from_request
from deerflow.config.runtime_paths import runtime_home
from deerflow.runtime.checkpointer.checkpoint_migration import (
SQLiteCheckpointSource,
copy_thread_state,
sqlite_recent_threads,
target_has_thread,
)
from deerflow.runtime.store._sqlite_utils import resolve_sqlite_conn_str
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/admin/checkpoint-migration", tags=["checkpoint-migration"])
_TASKS: dict[str, dict[str, Any]] = {}
class MigrationStartRequest(BaseModel):
start_at: str = Field(description="Inclusive start datetime, ISO string.")
end_at: str = Field(description="Inclusive end datetime, ISO string.")
source_paths: list[str] | None = Field(default=None, description="Optional SQLite sources. Defaults to configured dual-read paths.")
class MigrationTaskResponse(BaseModel):
task_id: str
status: str
start_at: str
end_at: str
source_paths: list[str]
total_threads: int = 0
copied_threads: int = 0
skipped_threads: int = 0
failed_threads: int = 0
skipped_ranges: int = 0
message: str = ""
error: str | None = None
created_at: str
updated_at: str
async def _require_admin(request: Request) -> None:
if is_auth_disabled():
return
user = await get_current_user_from_request(request)
if getattr(user, "system_role", None) != "admin":
raise HTTPException(status_code=403, detail="Checkpoint migration requires admin role")
def _now_iso() -> str:
return datetime.now(timezone.utc).isoformat()
def _parse_dt(value: str) -> datetime:
try:
normalized = value.replace("Z", "+00:00")
dt = datetime.fromisoformat(normalized)
except ValueError as exc:
raise HTTPException(status_code=400, detail=f"Invalid datetime: {value}") from exc
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
return dt
def _sqlite_datetime(dt: datetime) -> str:
# Existing SQLite rows store naive wall-clock strings like
# "2026-05-18 06:41:10.989427"; match that format for comparisons.
return dt.astimezone(timezone.utc).replace(tzinfo=None).isoformat(sep=" ")
def _configured_sources(request: Request) -> list[str]:
config = getattr(request.app.state, "config", None)
cp = getattr(config, "checkpointer", None)
paths = list(getattr(cp, "dual_read_sqlite_paths", []) or [])
raw_env = os.getenv("DEER_FLOW_CHECKPOINT_SQLITE_PATHS", "")
paths.extend(part.strip() for part in raw_env.split(",") if part.strip())
resolved: list[str] = []
seen: set[str] = set()
for path in paths:
conn_str = resolve_sqlite_conn_str(path)
if conn_str == ":memory:" or conn_str.startswith("file:"):
continue
if conn_str not in seen:
resolved.append(conn_str)
seen.add(conn_str)
return resolved
def _target_checkpointer(request: Request):
checkpointer = getattr(request.app.state, "checkpointer", None)
if checkpointer is None:
raise HTTPException(status_code=503, detail="Checkpointer is not available")
return getattr(checkpointer, "primary", checkpointer)
def _state_path() -> Path:
return runtime_home() / "checkpoint_migration_state.json"
def _load_completed_ranges() -> set[str]:
try:
payload = json.loads(_state_path().read_text(encoding="utf-8"))
except FileNotFoundError:
return set()
except Exception:
logger.warning("Ignoring unreadable checkpoint migration state file", exc_info=True)
return set()
return {str(item) for item in payload.get("completed_ranges", [])}
def _save_completed_ranges(items: set[str]) -> None:
path = _state_path()
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_suffix(".tmp")
tmp.write_text(json.dumps({"completed_ranges": sorted(items)}, ensure_ascii=False, indent=2), encoding="utf-8")
tmp.replace(path)
def _range_key(source: str, start_at: str, end_at: str) -> str:
raw = json.dumps({"source": str(Path(source).resolve()), "start_at": start_at, "end_at": end_at}, sort_keys=True)
return hashlib.sha256(raw.encode("utf-8")).hexdigest()
async def _open_sqlite_source(path: str) -> SQLiteCheckpointSource:
from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver
ctx = AsyncSqliteSaver.from_conn_string(path)
saver = await ctx.__aenter__()
await saver.setup()
return SQLiteCheckpointSource(
path=path,
saver=saver,
context=ctx,
progress_path=Path(path).with_suffix(Path(path).suffix + ".pg-migration.json"),
)
def _task_response(task: dict[str, Any]) -> MigrationTaskResponse:
return MigrationTaskResponse(**task)
async def _run_task(task_id: str, target, start_at: datetime, end_at: datetime) -> None:
task = _TASKS[task_id]
completed = await asyncio.to_thread(_load_completed_ranges)
start_sql = _sqlite_datetime(start_at)
end_sql = _sqlite_datetime(end_at)
range_start = start_at.isoformat()
range_end = end_at.isoformat()
try:
task["status"] = "running"
task["updated_at"] = _now_iso()
for source_path in task["source_paths"]:
if not Path(source_path).exists():
task["message"] = f"SQLite source not found: {source_path}"
task["skipped_ranges"] += 1
continue
key = _range_key(source_path, range_start, range_end)
if key in completed:
task["skipped_ranges"] += 1
task["message"] = "Selected range was already migrated; skipped"
task["updated_at"] = _now_iso()
continue
thread_ids = await asyncio.to_thread(sqlite_recent_threads, source_path, since=start_sql, until=end_sql)
if not thread_ids:
task["message"] = f"No threads found in selected range for {Path(source_path).name}"
completed.add(key)
await asyncio.to_thread(_save_completed_ranges, completed)
task["updated_at"] = _now_iso()
continue
task["total_threads"] += len(thread_ids)
source = await _open_sqlite_source(source_path)
try:
for thread_id in thread_ids:
if await target_has_thread(target, thread_id):
task["skipped_threads"] += 1
else:
copied = await copy_thread_state(source.saver, target, thread_id)
if copied:
task["copied_threads"] += 1
else:
task["skipped_threads"] += 1
task["updated_at"] = _now_iso()
await asyncio.sleep(0)
finally:
with contextlib.suppress(Exception):
await source.close()
completed.add(key)
await asyncio.to_thread(_save_completed_ranges, completed)
task["status"] = "completed"
task["message"] = task["message"] or "Migration completed"
except Exception as exc:
logger.exception("Checkpoint migration task failed: %s", task_id)
task["status"] = "failed"
task["error"] = str(exc)
finally:
task["updated_at"] = _now_iso()
@router.post("/tasks", response_model=MigrationTaskResponse)
async def start_migration(body: MigrationStartRequest, request: Request):
await _require_admin(request)
start_at = _parse_dt(body.start_at)
end_at = _parse_dt(body.end_at)
if end_at <= start_at:
raise HTTPException(status_code=400, detail="end_at must be later than start_at")
sources = [resolve_sqlite_conn_str(p) for p in (body.source_paths or _configured_sources(request))]
sources = [p for p in dict.fromkeys(sources) if p != ":memory:" and not p.startswith("file:")]
if not sources:
raise HTTPException(status_code=400, detail="No SQLite source paths configured")
target = _target_checkpointer(request)
task_id = hashlib.sha256(f"{_now_iso()}:{sources}:{body.start_at}:{body.end_at}".encode("utf-8")).hexdigest()[:16]
now = _now_iso()
_TASKS[task_id] = {
"task_id": task_id,
"status": "queued",
"start_at": start_at.isoformat(),
"end_at": end_at.isoformat(),
"source_paths": sources,
"total_threads": 0,
"copied_threads": 0,
"skipped_threads": 0,
"failed_threads": 0,
"skipped_ranges": 0,
"message": "",
"error": None,
"created_at": now,
"updated_at": now,
}
asyncio.create_task(_run_task(task_id, target, start_at, end_at))
return _task_response(_TASKS[task_id])
@router.get("/tasks/{task_id}", response_model=MigrationTaskResponse)
async def get_migration_task(task_id: str, request: Request):
await _require_admin(request)
task = _TASKS.get(task_id)
if task is None:
raise HTTPException(status_code=404, detail="Migration task not found")
return _task_response(task)
@router.get("/sources")
async def list_sources(request: Request):
await _require_admin(request)
return {"source_paths": _configured_sources(request)}
offline-backend-20260512/backend/app/gateway/routers/llm_metrics.py
+34 -11
View File
@@ -121,6 +121,26 @@ async def _resolve_user_name_to_id(name: str | None) -> str | None:
return str(user.id) if user is not None else None
async def _resolve_user_names_to_ids(name: str | None) -> list[str]:
"""Fuzzy-resolve a username fragment to every matching account's UUID.
Powers the 按用户名模糊搜索 filter: typing part of a username matches all
users whose email local part contains it. Returns an empty list when the
fragment matches nobody, so the caller can short-circuit to zero rows.
"""
if not name or not name.strip():
return []
try:
provider = get_local_provider()
except Exception:
return []
try:
users = await provider.find_users_by_email_local_part_like(name.strip())
except Exception:
return []
return [str(u.id) for u in users]
async def _attach_usernames(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""Resolve each row's ``user_id`` UUID to a friendly ``user_name``.
@@ -161,9 +181,11 @@ def _build_query(
until: str | None,
limit: int,
offset: int,
user_ids: list[str] | None = None,
) -> LlmMetricsQuery:
return LlmMetricsQuery(
user_id=user_id or None,
user_ids=user_ids or None,
model_name=model_name or None,
status=status or None,
since=_parse_dt(since),
@@ -187,14 +209,15 @@ async def list_llm_metrics(
) -> LlmMetricsListResponse:
await _require_admin(request)
store = _require_store(request)
user_ids: list[str] | None = None
if not user_id and user_name:
resolved = await _resolve_user_name_to_id(user_name)
if resolved is None:
# Username unknown → no rows; short-circuit instead of a global scan.
user_ids = await _resolve_user_names_to_ids(user_name)
if not user_ids:
# No username matches → no rows; short-circuit the global scan.
return LlmMetricsListResponse(items=[], total=0, limit=limit, offset=offset)
user_id = resolved
query = _build_query(
user_id=user_id,
user_ids=user_ids,
model_name=model_name,
status=status,
since=since,
@@ -236,14 +259,13 @@ async def export_llm_metrics(
"""Stream the full filtered metric set as an ``.xlsx`` file."""
await _require_admin(request)
store = _require_store(request)
user_ids: list[str] | None = None
if not user_id and user_name:
resolved = await _resolve_user_name_to_id(user_name)
if resolved is None:
# No matching user — return an empty xlsx so the user still gets a
# file (matching the list view's empty state).
user_id = "__no_such_user__"
else:
user_id = resolved
user_ids = await _resolve_user_names_to_ids(user_name)
if not user_ids:
# No matching user — sentinel id that matches nothing, so the export
# still produces an empty xlsx (matching the list view's empty state).
user_ids = ["__no_such_user__"]
try:
from openpyxl import Workbook
@@ -252,6 +274,7 @@ async def export_llm_metrics(
query = _build_query(
user_id=user_id,
user_ids=user_ids,
model_name=model_name,
status=status,
since=since,
offline-backend-20260512/backend/app/gateway/routers/multi_agent.py
+606 -179
View File
@@ -47,6 +47,7 @@ Leader 终态帧（前端 ``isFinalStatusFrame``）
from __future__ import annotations
import asyncio
import contextlib
import json
import logging
import os
@@ -57,18 +58,20 @@ import uuid
from typing import Any, AsyncIterator
import httpx
from fastapi import APIRouter, Body, HTTPException, Request
from fastapi.responses import StreamingResponse
from langgraph.checkpoint.base import empty_checkpoint
from app.gateway.roundtable_run_policy import merge_skill_stop_names, roundtable_run_policy
from app.gateway.routers._roundtable_seed import (
COORDINATOR_AGENT_ID,
REPORT_AGENT_ID,
ensure_roundtable_functional_agents,
)
from deerflow.config import get_app_config
from deerflow.tools.builtins.clarification_utils import resolve_allow_multiple
from deerflow.runtime.user_context import get_effective_user_id
from deerflow.tools.builtins.clarification_utils import resolve_allow_multiple
from deerflow.utils.time import now_iso
from fastapi import APIRouter, Body, HTTPException, Request
from fastapi.responses import StreamingResponse
from langgraph.checkpoint.base import empty_checkpoint
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/multi-agent", tags=["multi-agent"])
@@ -466,22 +469,42 @@ async def _append_thread_message(
失败只打 warning，不阻断主流程——广播是 best-effort，席位真正干活仍靠
前端对该席位 thread 发起的 ``/run/stream``。
走**轻量追加**接口(``POST /messages/append``):只传一条消息,服务端读 checkpoint+追加+写回,
省掉原先 GET /state(下载全量)+POST /state(上传全量)把几万字状态经 loopback 传两遍的开销。
"""
state_url = f"{base}/api/threads/{thread_id}/state"
get_resp = await client.get(state_url, headers=headers)
if get_resp.status_code != 200:
logger.warning("get state %s failed: %s %s", thread_id, get_resp.status_code, get_resp.text[:200])
return
state = get_resp.json()
values = state.setdefault("values", {})
messages = values.setdefault("messages", [])
if not isinstance(messages, list):
messages = []
values["messages"] = messages
messages.append({"type": "human", "content": content})
post_resp = await client.post(state_url, json=state, headers=headers)
if post_resp.status_code >= 400:
logger.warning("update state %s failed: %s %s", thread_id, post_resp.status_code, post_resp.text[:200])
resp = await client.post(
f"{base}/api/threads/{thread_id}/messages/append",
json={"content": content, "type": "human"},
headers=headers,
)
if resp.status_code >= 400:
logger.warning("append message %s failed: %s %s", thread_id, resp.status_code, resp.text[:200])
def _spawn_detached_broadcast(
base: str,
headers: dict[str, str],
message: str,
agent_threads: dict[str, str],
skip: list[str],
) -> None:
"""后台 fire-and-forget 广播,**用独立 client**,即使发起它的请求 SSE 已结束也能跑完。
广播是 best-effort 的「上下文播报」(告知其它席位某人交付了什么),原本在总控 yield 派活
帧**之前**被 ``await``,导致每个席位都要等几十秒(广播要读写各席位的巨大 thread 状态)
才开始执行 —— 实测 ``dispatch_broadcasts_await`` 飙到 27s。改成 detached 后台任务后,总控
立即 yield 派活、前端立即启动席位,广播在后台慢慢落地,彻底移出关键路径。失败只记日志。
"""
async def _run() -> None:
try:
async with httpx.AsyncClient(timeout=None, trust_env=False) as bg:
await _broadcast(bg, base, headers, message, agent_threads, skip)
except Exception as exc: # noqa: BLE001
logger.warning("detached broadcast failed: %s", exc)
# create_task 后不持有引用:任务挂在事件循环上,独立于本请求生命周期跑完。
asyncio.ensure_future(_run())
async def _broadcast(
@@ -504,6 +527,29 @@ async def _broadcast(
_TOOL_CALL_PATTERN = re.compile(r'\.py\s+"([^"]*)"\s+"([^"]*)"')
# 子智能体交付的「后广播」里,正文广播给**其它席位**时的截断上限。
# 背景:席位交付动辄几万字;原实现把整篇交付广播进**每一个**其它席位 thread,导致
# 每个 thread 状态滚雪球,之后每次状态读/写(广播、交付状态核对)都要搬运巨量数据,
# 延迟复利式恶化(实测 dispatch_broadcasts_await 飙到 30s)。其它席位只需"知道某席位
# 交付了什么"的摘要即可;**完整交付仍原样发给总控 thread**(总控做最终综合要看全文),
# 也仍通过 SSE 完整给到前端。仅截断"广播给兄弟席位"这一路。
_BROADCAST_CONTENT_MAX_CHARS = 1500
def _truncate_broadcast_content(prefix: str, content: str) -> str:
"""构造广播给**兄弟席位**的交付摘要消息:超长正文截断到上限并标注省略。
``content`` 未超限时原样拼接(等价于完整版),超限时截断 + 追加省略说明。
总控那一路不走本函数,始终用完整正文(见 ``_special_run`` 的 ``full_msg``)。
"""
if len(content) <= _BROADCAST_CONTENT_MAX_CHARS:
return prefix + content
return (
prefix
+ content[:_BROADCAST_CONTENT_MAX_CHARS]
+ f"\n…(完整交付约 {len(content)} 字,已省略;以该席位自身交付为准)"
)
# Mirrors the upstream agent-id validator in the LangGraph runtime. Reject early
# so callers get a clear 400 here instead of a stream that aborts with HTTP 422
# from the loopback /api/threads/.../runs/stream endpoint.
@@ -590,6 +636,8 @@ def _run_payload(
skill_stop_names: list[str],
thinking_enabled: bool = True,
reasoning_effort: str = "medium",
multitask_strategy: str = "reject",
extra_context: dict[str, Any] | None = None,
) -> dict[str, Any]:
"""构造 ``POST /api/threads/{thread_id}/runs/stream`` 的请求体。
@@ -602,7 +650,7 @@ def _run_payload(
应传 ``thinking_enabled=False`` + ``reasoning_effort="low"``，省掉模型先吐
reasoning tokens 才开口的时间，显著降低前端"调度中"首字节延迟。
"""
return {
payload: dict[str, Any] = {
"input": {
"messages": [
{
@@ -625,18 +673,36 @@ def _run_payload(
# 圆桌四个 router(intent / recommend / leader / special)都是临时任务,
# 用户个人记忆与本次研讨无关。MemoryMiddleware 的 Hindsight recall 是
# 同步阻塞 HTTP 调用(单次 timeout 15s),如果记忆服务慢/不通会让前端
# "调度中"卡很久;这里默认跳过,builtin USER.md/MEMORY.md 仍通过系统
# 提示注入不受影响。
# "调度中"卡很久;这里默认跳过。
"memory_recall_disabled": True,
# ⚠️ 同时关掉 builtin 记忆**注入**:圆桌四个角色(intent / recommend /
# leader / special)都是固定单例 agent,其 memory.json 跨所有会话累积。
# 总控尤其危险——上一个任务里它臆造的席位名(如 data-specialist)会被存进
# memory.json,下一个新任务又被注入系统提示而召回 → 派给根本没初始化的
# 席位 → run/stream 报 400 "agent_name ... missing from d_agent_thread_id"。
# 圆桌是一次性研讨,本就不该吃历史会话记忆,这里统一关注入根治该污染。
"memory_injection_enabled": False,
},
"stream_mode": ["messages-tuple", "values"],
"stream_subgraphs": True,
"stream_resumable": True,
"assistant_id": "lead_agent",
"on_disconnect": "continue",
# ⚠️ 圆桌 run 必须能被「下一次同 thread 的 run」抢占。圆桌每个 thread 同一时刻
# 只应有一个活跃 run,但 on_disconnect="continue" 让前端 abort SSE 后**后端 run
# 仍在跑**(为可恢复);此时用户「人工干预」立刻发起新 leader run 打到同一总控
# thread,默认 reject 策略就会 409「already has an active run」→ 被 _stream_upstream
# 包成 502。调用方(leader / special)传 multitask_strategy="rollback",让新 run 先
# 取消该 thread 上没结束的旧 run 再继续 —— 这正是「最新一轮/干预指令优先」的语义。
"multitask_strategy": multitask_strategy,
"excluded_tools": excluded_tools,
"skill_stop_names": skill_stop_names,
}
# 额外注入 run context（如圆桌席位的 roundtable_peer_deliveries：前序席位完整交付,
# 供 read_peer_delivery 工具按需读取）。只进 context dict,不进 prompt → 不调工具就不耗 token。
if extra_context:
payload["context"].update(extra_context)
return payload
# ---------------------------------------------------------------------------
@@ -900,16 +966,23 @@ async def _stream_upstream(
yield None, captured
def _parse_last_messages(captured: list[str]) -> list[dict[str, Any]]:
def _parse_last_messages(captured: list[str]) -> list[dict[str, Any]] | None:
"""Return the full ``messages`` list from the most recent ``values`` snapshot.
The upstream sends the final ``values`` frame second-to-last (the very last
line is the ``end`` event with ``null``). We tolerate trailing non-JSON
frames (e.g. ``data: null``) by scanning backwards for the first frame that
successfully parses as a state object with a ``messages`` key.
Returns ``None`` when the stream carried **no parseable ``values`` frame**
(empty capture, or only ``data: null`` / token deltas). 这通常**不是真正的
上游错误**,而是该 run 被 ``multitask_strategy=rollback`` 抢占 / 客户端断开 /
中途 abort —— 流被截断,没来得及发出最终 values 快照。调用方(leader/special)据此
**静默收尾**,绝不能把它当 502 抛给前端:抢占场景下这条 SSE 多半已被前端
``abortAllInFlight`` 放弃,硬报 502 只会在 UI 上凭空冒出一条错误。
"""
if not captured:
raise HTTPException(status_code=502, detail="upstream produced no data lines")
return None
for raw_line in reversed(captured):
raw = raw_line[5:].strip() if raw_line.startswith("data:") else raw_line.strip()
@@ -924,7 +997,7 @@ def _parse_last_messages(captured: list[str]) -> list[dict[str, Any]]:
if isinstance(messages, list):
return messages
raise HTTPException(status_code=502, detail="no parseable values frame in upstream stream")
return None
def _find_trailing_ai_text(messages: list[dict[str, Any]], *, exclude_text: str = "") -> str:
@@ -981,6 +1054,198 @@ def _find_dispatch_message(messages: list[dict[str, Any]]) -> dict[str, Any]:
return {}
def _partition_dispatch(
tool_calls: list[dict[str, Any]],
valid_seat_ids: set[str],
) -> tuple[list[tuple[str, str]], list[str]]:
"""把总控的 agent_orchestration 派活拆成「合法 / 非法」两份。
返回 ``(valid, invalid)``：
- ``valid`` —— ``[(seat_id, task), ...]``，``seat_id`` 确实是本轮已初始化的席位；
- ``invalid`` —— ``[seat_id, ...]``，模型臆造 / 不在本轮 roster 内的 id。
弱模型(deepseek-chat)会把 SOUL 里的示例 hex id 当真照抄、或臆造同款 32 位 hex,
这些 id 不在 ``agent_threads`` 里;若不拦截会漏给前端,到 ``/run/stream`` 才报 400
把整个 Step 2 打挂。校验放在派活解析处,只丢非法、保留同轮合法席位。
"""
valid: list[tuple[str, str]] = []
invalid: list[str] = []
for tool_call in tool_calls:
sub_name, task_text = _extract_dispatch(tool_call)
if not sub_name:
continue
if sub_name in valid_seat_ids:
valid.append((sub_name, task_text))
else:
invalid.append(sub_name)
return valid, invalid
def _build_dispatch_correction(invalid_names: list[str], valid_seat_ids: set[str]) -> str:
"""整轮派活全非法时,回灌给总控 thread 的系统纠正消息文本。
列出真实席位 id,要求模型**逐字符复制**重派,杜绝「凭示例/记忆构造 id」。
"""
bad = "、".join(invalid_names)
roster = "、".join(sorted(valid_seat_ids))
return (
"【系统纠正】你刚才通过 agent_orchestration 派活的 agent_name（"
f"{bad}）不在本次圆桌的席位清单里，无法执行。请**只**从下面这份真实席位清单里"
"逐字符复制 agent_name 重新派活，不要凭记忆、示例或猜测自行构造 id：\n"
f"{roster}"
)
def _build_seat_roster_block(seats: list[tuple[str, dict[str, Any]]]) -> str:
"""构造注入到总控**本轮输入最前面**的「真实席位清单」文本。
``seats`` 为 ``[(seat_id, {name, description}), ...]``。
为什么要每轮把清单塞进输入,而不是只靠 init 时 _append_thread_message 注入历史:
圆桌席位清单原本只在 init 时以一条 human 消息异步写进总控 thread 历史,总控靠"读
历史"看到它 —— 这条链路脆弱(best-effort 写盘、跨 thread 持久化),一旦没落地,弱模型
(deepseek-chat)面对空历史就会**从任务语义臆造**不存在的席位(已观测到 architecture-
specialist、02714cbe… 这类假 id)。网关在每次 leader run 时本就握有真实席位 id,这里
直接把清单摆到总控本轮输入最前面并要求逐字复制,从源头根治臆造;与事后 _partition_dispatch
校验构成「源头 + 兜底」双保险。
"""
if not seats:
return ""
lines: list[str] = []
for sid, meta in seats:
name = str((meta or {}).get("name") or sid)
desc = str((meta or {}).get("description") or "").strip()
lines.append(f"- {sid}（{name}）：{desc}" if desc else f"- {sid}（{name}）")
roster = "\n".join(lines)
return (
"【本次圆桌可调度席位清单】派活时 agent_orchestration 的 agent_name 必须从下面"
"逐字符复制一个 id，严禁自行拼造、翻译角色名或凭记忆构造：\n"
f"{roster}"
)
async def _collect_delivery_status(
client: httpx.AsyncClient,
base: str,
headers: dict[str, str],
seat_threads: dict[str, str],
*,
max_chars: int = 0,
) -> dict[str, tuple[bool, str, int]]:
"""读各席位 thread 的真实状态,客观判定「是否已交付 + 交付摘要」。
返回 ``{seat_id: (delivered, text, full_len)}``。``text`` 是该席位**最后一条 AI 交付**(按
``max_chars`` 截断:派活轮取 600 字摘要、综合轮 ``max_chars<=0`` 取全文);``full_len`` 是交付
全文长度(供调用方判断 text 是否被截断、要不要加省略号)。判定依据:席位 thread 是否存在非空
AI 交付,只有 broadcast 进来的 human 消息不算。
走**轻量读**接口(``GET /last-ai-message?max_chars=N``):服务端只回最后一条 AI 消息(可截断),
不把整份 state 序列化经 loopback 传回 —— 派活轮每席位只取 600 字,大幅压低原先 ~11s 的
``payload_built``。任一席位读失败按「未交付」保守处理,绝不阻断。
这是「确定性双保险」的数据源:总控弱模型会臆造「三方已交付」,这里由网关**客观核对**每个席位
thread 的实际产出,再把结论喂回总控输入,总控以系统核对为准,不靠自己读历史猜。
"""
async def _one(sid: str, tid: str) -> tuple[str, tuple[bool, str, int]]:
try:
resp = await client.get(
f"{base}/api/threads/{tid}/last-ai-message",
params={"max_chars": max_chars},
headers=headers,
)
if resp.status_code != 200:
return sid, (False, "", 0)
data = resp.json()
content = str(data.get("content") or "")
full_len = int(data.get("length") or len(content))
return sid, (bool(data.get("delivered")), content, full_len)
except Exception:
return sid, (False, "", 0)
results = await asyncio.gather(*(_one(sid, tid) for sid, tid in seat_threads.items()))
return dict(results)
# 交付状态摘要块里,每个席位「已交付摘要」截断到的字符数(让总控有足够信息做派活决策,
# 又远小于动辄几万字的全文)。
_STATUS_EXCERPT_CHARS = 600
def _build_seat_status_block(
seats: list[tuple[str, dict[str, Any]]],
delivery: dict[str, tuple[bool, str, int]],
*,
undelivered_count: int = 0,
) -> str:
"""席位清单 + **系统客观核对的真实交付状态**,注入总控本轮输入最前面。
在 ``_build_seat_roster_block``(只给 id+名称)基础上,叠加每个席位的 ✅已交付 / ⬜未交付
标记(来自 ``_collect_delivery_status`` 对席位 thread 的实读),并明确告诉总控「以这份为准、
不要臆造」「用户点名的 ⬜ 席位必须派活」。这是消除「臆造三方已交付直接收口」的确定性手段。
``undelivered_count`` > 0 时额外加一句硬约束:还有席位未交付,**不得**急于收口。
"""
if not seats:
return ""
lines: list[str] = []
for sid, meta in seats:
name = str((meta or {}).get("name") or sid)
delivered, text, full_len = delivery.get(sid, (False, "", 0))
if delivered:
# text 已是服务端按 max_chars 截好的摘要;full_len > len(text) 说明被截断,加省略号。
ellipsis = "…" if full_len > len(text) else ""
tail = f" —— 交付摘要:{text}{ellipsis}" if text else ""
lines.append(f"- {sid}（{name}）：✅ 已交付{tail}")
else:
lines.append(f"- {sid}（{name}）：⬜ 未交付")
roster = "\n".join(lines)
pending_rule = (
(
f"\n⚠️ 当前还有 {undelivered_count} 个 ⬜ 未交付席位。**信息尚未集齐,严禁现在收口/输出最终方案**,"
"也不得声称「各席位均已发言」。请**优先继续派活**给这些 ⬜ 席位,把它们的真实交付拿到手再说综合。"
)
if undelivered_count > 0
else ""
)
return (
"【本次圆桌席位与真实交付状态(系统客观核对,以此为准,严禁臆造)】\n"
"派活时 agent_orchestration 的 agent_name 必须从下面逐字符复制一个 id;"
"只有标 ✅ 的才算已交付,标 ⬜ 的就是还没交付——\n"
"若用户想要的席位是 ⬜,你**必须**派活给它拿到真实交付,**不得**跳过它直接收口或代答;"
"也**不得**声称任何 ⬜ 席位已交付。下面只是**摘要**,做最终综合时以系统额外提供的"
"「完整交付」为准:\n"
f"{roster}"
f"{pending_rule}"
)
def _build_full_deliverables_block(
seats: list[tuple[str, dict[str, Any]]],
delivery: dict[str, tuple[bool, str, int]],
) -> str:
"""综合轮专用:把各已交付席位的**完整交付正文**拼成一块,供总控写最终方案。
只在前端标记的「综合轮」(synthesis_mode)注入,平时不进总控输入 —— 这样派活决策轮
的上下文保持精简、响应快,只有真要收口时才承担一次全文的体量。未交付席位不列入。
"""
sections: list[str] = []
for sid, meta in seats:
delivered, full_text, _ = delivery.get(sid, (False, "", 0))
if not delivered or not full_text:
continue
name = str((meta or {}).get("name") or sid)
sections.append(f"━━ {name}（{sid}）的完整交付 ━━\n{full_text}")
if not sections:
return ""
body = "\n\n".join(sections)
return (
"【各席位完整交付正文(综合专用,做最终方案时以此为准)】\n"
"请基于下列各席位的真实完整交付,综合输出结构化的最终方案,不要遗漏关键内容,也不要臆造:\n"
f"{body}"
)
async def _leader_run(
client: httpx.AsyncClient,
base: str,
@@ -990,6 +1255,9 @@ async def _leader_run(
new_message: str,
skill_stop_names: list[str],
model_name: str,
seat_meta: dict[str, dict[str, Any]] | None = None,
synthesis_mode: bool = False,
suppress_pre_broadcast: bool = False,
) -> AsyncIterator[str | dict[str, Any]]:
"""总控（leader）单轮：透传上游 SSE + 末尾 yield 一个 dict 状态帧（由路由层 ``json.dumps``）。
@@ -1000,27 +1268,96 @@ async def _leader_run(
4. 若有派活，**派活广播**后再 yield ``{status: dispatched, ...}``。
``skill_stop_names`` 会与 ``agent_orchestration`` 合并，确保 middleware 在工具调用后停图。
``seat_meta``：``{seat_id: {name, description}}``。非空时,本函数会在跑总控前**实读各席位
thread 的真实交付状态**,把「席位清单 + ✅已交付/⬜未交付 + 摘要」拼到总控本轮输入最前面
(见 ``_collect_delivery_status`` / ``_build_seat_status_block``)。这是消除「总控臆造
三方已交付、一交付就收口」的确定性双保险:总控以系统客观核对为准,不靠自己读历史猜。
``synthesis_mode``：前端在「派活后让总控判断继续/综合」的轮次置真。为真时**额外**把各
已交付席位的**完整交付正文**注入本轮输入(``_build_full_deliverables_block``),供总控写
最终方案;平时(派活决策轮)只给摘要,让总控 thread 与每轮上下文保持精简、响应快。
"""
t0 = time.perf_counter()
# 实读各席位 thread 的真实交付状态,构造「席位清单 + 客观交付状态(摘要)」块;综合轮再额外
# 拼上各已交付席位的完整交付正文。失败/无 seat_meta 时退化为不注入(leader_input ==
# new_message),绝不阻断 leader run。这些只进总控**本轮 run 输入**,不广播给席位。
seat_status_block = ""
full_deliverables_block = ""
if seat_meta:
seat_ids = [name for name in agent_threads if name != agent_name]
if seat_ids:
# 第 1 阶段:**轻量**读各席位最后一条 AI 交付,只取 ≤_STATUS_EXCERPT_CHARS 字 ——
# 派活轮只需 ✅/⬜ + 摘要,绝不下载几万字全文,把 payload_built 从 ~11s 压下来。
delivery: dict[str, tuple[bool, str]] = {}
try:
delivery = await _collect_delivery_status(
client, base, headers, {sid: agent_threads[sid] for sid in seat_ids},
max_chars=_STATUS_EXCERPT_CHARS,
)
except Exception:
logger.warning("[leader-roster] 收集席位交付状态失败,退化为仅席位清单", exc_info=True)
undelivered = [sid for sid in seat_ids if not delivery.get(sid, (False, ""))[0]]
seats_meta_list = [(sid, seat_meta.get(sid, {})) for sid in seat_ids]
seat_status_block = _build_seat_status_block(seats_meta_list, delivery, undelivered_count=len(undelivered))
# 第 2 阶段(仅「综合轮 且 所有席位都已交付」):再**全文**取各已交付席位的交付,拼综合块。
# 还有 ⬜ 未交付席位时不注入 —— 否则"综合输出最终方案"的指令会怂恿弱模型在席位没集齐时
# 就提前收口(实测:3/5 交付就臆称"三方已发言"草草收口)。全文读只在这一轮才付出。
if synthesis_mode and not undelivered:
try:
full_delivery = await _collect_delivery_status(
client, base, headers,
{sid: agent_threads[sid] for sid in seat_ids if delivery.get(sid, (False, ""))[0]},
max_chars=0,
)
full_deliverables_block = _build_full_deliverables_block(seats_meta_list, full_delivery)
except Exception:
logger.warning("[leader-roster] 综合轮取全文交付失败,退化为仅摘要", exc_info=True)
logger.info(
"[leader-roster] %s 注入席位交付状态:%s synthesis_mode=%s undelivered=%d full_deliverables=%s",
agent_name,
{sid: delivery.get(sid, (False, ""))[0] for sid in seat_ids},
synthesis_mode,
len(undelivered),
bool(full_deliverables_block),
)
leader_input = "\n\n".join(
part for part in (seat_status_block, full_deliverables_block, new_message) if part
)
# 与上游 LLM 并行，减少「总控已开始想但席位 thread 里还没有用户消息」的窗口。
pre_broadcast_task = asyncio.create_task(
_broadcast(client, base, headers, "总控智能体收到用户消息:" + new_message, agent_threads, [agent_name])
#
# ``suppress_pre_broadcast``：DAG「每 stage 派活轮」(方案 A)置真——此时 ``new_message`` 是
# 「请你作为总控…通过 agent_orchestration 派活给席位…」的**派活 prompt**,若广播进各席位
# thread,弱模型(deepseek)会从历史里捡到「派活」意图,**席位跟着模仿派活 / 绕道 bash 调
# 内部接口**(已观测 bug-1~4)。DAG 下席位的任务由前端 ``pickDagSeatTask`` 直接、干净地下发
# (含任务背景),此处不需要、也不应把总控的派活 prompt 播给席位。recommend/chain 不传该 flag,
# 行为不变。
pre_broadcast_task = (
None
if suppress_pre_broadcast
else asyncio.create_task(
_broadcast(client, base, headers, "总控智能体收到用户消息:" + new_message, agent_threads, [agent_name])
)
)
thread_id = agent_threads[agent_name]
effective_stop_names = list({"agent_orchestration", *skill_stop_names})
# 角色参数取自单一数据源 roundtable_run_policy（leader：禁 web_search、停
# agent_orchestration 图、关 thinking + low reasoning——短决策、首字节快）。
# 调用方可再叠加自己的 skill_stop。后台进程内网关用同一份 policy，保证传参一致。
policy = roundtable_run_policy("leader")
effective_stop_names = merge_skill_stop_names(policy.skill_stop_names, skill_stop_names)
body = _run_payload(
agent_name=agent_name,
model_name=model_name,
thread_id=thread_id,
new_message=new_message,
excluded_tools=["web_search"],
# 注入了席位清单的输入(leader_input);seat_roster 为空时等同 new_message。
new_message=leader_input,
excluded_tools=policy.excluded_tools,
skill_stop_names=effective_stop_names,
# 总控派活是"下一步派谁、什么任务"的短决策，不需要 CoT。
# 关掉 thinking + low reasoning 让"调度中"几乎瞬间转入"正在派活"，
# 子智能体仍保留 thinking（在 _special_run 里默认 True），保证交付质量。
thinking_enabled=False,
reasoning_effort="low",
thinking_enabled=policy.thinking_enabled,
reasoning_effort=policy.reasoning_effort,
# 人工干预/恢复会重发 leader run 到同一总控 thread;抢占掉没结束的旧 run,避免 409。
multitask_strategy="rollback",
)
t_payload_built = time.perf_counter()
logger.info(
@@ -1029,117 +1366,157 @@ async def _leader_run(
t_payload_built - t0,
)
captured: list[str] = []
first_line_ts: float | None = None
line_count = 0
async for line, done in _stream_upstream(client, base, headers, thread_id, body):
if line is not None:
if first_line_ts is None:
first_line_ts = time.perf_counter()
logger.info(
"[leader-timing] %s first_upstream_line=%.3fs since_entry=%.3fs",
agent_name,
first_line_ts - t_payload_built,
first_line_ts - t0,
)
line_count += 1
yield line
elif done is not None:
captured = done
t_stream_done = time.perf_counter()
logger.info(
"[leader-timing] %s upstream_stream_complete=%.3fs lines=%d since_entry=%.3fs",
agent_name,
t_stream_done - (first_line_ts or t_payload_built),
line_count,
t_stream_done - t0,
)
# 单例总控是弱模型(deepseek-chat),已观测到它把 SOUL 里的示例 id 当真实席位
# 照抄、或臆造同款 32 位 hex —— 这些 id 不在 agent_threads 里,直接漏给前端会让
# /run/stream 报 400 把整个 Step 2 打挂。这里把「本次真实可派席位」作为 roster
# (agent_threads 去掉总控自身),派活前就地校验:非法 id 丢弃 + 记日志;若**整轮
# 全是非法 id**,向总控 thread 回灌一条系统纠正消息(列出真实 id),再重跑一轮让它
# 自纠(最多重试 1 次,避免弱模型反复臆造导致死循环)。
valid_seat_ids: set[str] = {name for name in agent_threads if name != agent_name}
_MAX_DISPATCH_RETRIES = 1
messages = _parse_last_messages(captured)
# Pre-broadcast must finish before we yield the final status frame on any
# branch — the seats that get dispatched next rely on having seen the
# original user message. The upstream LLM stream usually takes several
# seconds, so the broadcast has typically already completed and this
# await is effectively free.
t_before_broadcast_await = time.perf_counter()
try:
await pre_broadcast_task
except Exception as exc:
logger.warning("pre-broadcast failed: %s", exc)
t_after_broadcast_await = time.perf_counter()
logger.info(
"[leader-timing] %s pre_broadcast_await=%.3fs (>0 means broadcast was still in flight after LLM finished)",
agent_name,
t_after_broadcast_await - t_before_broadcast_await,
)
dispatch_msg = _find_dispatch_message(messages)
leader_text = _flatten_content(dispatch_msg.get("content", ""))
tool_calls = dispatch_msg.get("tool_calls") or []
# 澄清分支：与 intent.py 相同字段；status 为字符串 "clarification"（非数组），
# 前端据此暂停 runOrchestration 循环，展示 Step 2 澄清条。
clarification_call = next(
(tc for tc in tool_calls if (tc.get("name") or "").lower() == "ask_clarification"),
None,
)
if clarification_call is not None:
cargs = clarification_call.get("args") or {}
question = str(cargs.get("question") or "").strip()
follow_up = _find_trailing_ai_text(messages, exclude_text=leader_text)
yield {
"status": "clarification",
"content": follow_up or question or leader_text,
"question": question,
"clarification_type": str(cargs.get("clarification_type") or ""),
"clarification_context": str(cargs.get("context") or ""),
"options": cargs.get("options") or [],
"allow_custom": bool(cargs.get("allow_custom", True)),
"allow_multiple": resolve_allow_multiple(cargs),
"agent_name": agent_name,
}
return
if not tool_calls:
# 空 status 数组 = 总控不再派活，前端视为共识（hasConsensus）。
yield {"status": [], "content": leader_text, "agent_name": agent_name}
return
# 派活广播：告知各席位「总控给 X 派了什么活」，与即将发起的 special run 配合。
# sets and we don't care about ordering, so serializing them just
# multiplied latency before the frontend could start a sub-agent run.
dispatched: list[list[str]] = []
dispatch_broadcasts: list[asyncio.Task[None]] = []
for tool_call in tool_calls:
try:
sub_name, task_text = _extract_dispatch(tool_call)
if not sub_name:
continue
dispatch_broadcasts.append(
asyncio.create_task(
_broadcast(
client,
base,
headers,
f"总控智能体给{sub_name}派活,内容为:" + task_text,
agent_threads,
[agent_name, sub_name],
attempt = 0
while True:
captured: list[str] = []
first_line_ts: float | None = None
line_count = 0
async for line, done in _stream_upstream(client, base, headers, thread_id, body):
if line is not None:
if first_line_ts is None:
first_line_ts = time.perf_counter()
logger.info(
"[leader-timing] %s first_upstream_line=%.3fs since_entry=%.3fs",
agent_name,
first_line_ts - t_payload_built,
first_line_ts - t0,
)
)
)
dispatched.append([sub_name, task_text])
except Exception as exc:
logger.warning("dispatch failed: %s", exc)
if dispatch_broadcasts:
t_before_dispatch = time.perf_counter()
await asyncio.gather(*dispatch_broadcasts, return_exceptions=True)
line_count += 1
yield line
elif done is not None:
captured = done
t_stream_done = time.perf_counter()
logger.info(
"[leader-timing] %s dispatch_broadcasts_await=%.3fs count=%d (gates how soon the next sub agent can start)",
"[leader-timing] %s upstream_stream_complete=%.3fs lines=%d since_entry=%.3fs",
agent_name,
time.perf_counter() - t_before_dispatch,
len(dispatch_broadcasts),
t_stream_done - (first_line_ts or t_payload_built),
line_count,
t_stream_done - t0,
)
yield {"status": dispatched, "content": leader_text, "agent_name": agent_name}
messages = _parse_last_messages(captured)
if messages is None:
# 流被截断(多半被 multitask_strategy=rollback 抢占 / 客户端断开 / abort)。
# 这不是真正的上游错误——静默收尾,不向前端抛 502。该 run 对应的 SSE 在抢占
# 场景下通常已被前端 abortAllInFlight 放弃;即便没被放弃,前端拿到"无终态帧"
# 也只会当作空轮次,不会冒出错误条。
# WARNING 级:正常应是被 rollback 抢占;但若上游 run 因别的原因(如 DB 连接断、
# LLM 报错)截断流也会走这里 → 静默收尾会把它表现成前端"(无文本输出)"。留个
# 显眼日志,便于把"真异常被静默吞掉"和"正常抢占"区分开排查。
logger.warning(
"[leader-timing] %s upstream truncated (no values frame) — ending quietly "
"(expected on rollback preemption; otherwise check the upstream run's server-side error)",
agent_name,
)
if pre_broadcast_task is not None:
with contextlib.suppress(Exception):
await pre_broadcast_task
return
# Pre-broadcast must finish before we yield the final status frame on any
# branch — the seats that get dispatched next rely on having seen the
# original user message. Only await it once (first attempt); the rare
# self-correction retry reuses the same already-awaited broadcast.
if attempt == 0 and pre_broadcast_task is not None:
t_before_broadcast_await = time.perf_counter()
try:
await pre_broadcast_task
except Exception as exc:
logger.warning("pre-broadcast failed: %s", exc)
logger.info(
"[leader-timing] %s pre_broadcast_await=%.3fs (>0 means broadcast was still in flight after LLM finished)",
agent_name,
time.perf_counter() - t_before_broadcast_await,
)
dispatch_msg = _find_dispatch_message(messages)
leader_text = _flatten_content(dispatch_msg.get("content", ""))
tool_calls = dispatch_msg.get("tool_calls") or []
# 澄清分支：与 intent.py 相同字段；status 为字符串 "clarification"（非数组），
# 前端据此暂停 runOrchestration 循环，展示 Step 2 澄清条。
clarification_call = next(
(tc for tc in tool_calls if (tc.get("name") or "").lower() == "ask_clarification"),
None,
)
if clarification_call is not None:
cargs = clarification_call.get("args") or {}
question = str(cargs.get("question") or "").strip()
follow_up = _find_trailing_ai_text(messages, exclude_text=leader_text)
yield {
"status": "clarification",
"content": follow_up or question or leader_text,
"question": question,
"clarification_type": str(cargs.get("clarification_type") or ""),
"clarification_context": str(cargs.get("context") or ""),
"options": cargs.get("options") or [],
"allow_custom": bool(cargs.get("allow_custom", True)),
"allow_multiple": resolve_allow_multiple(cargs),
"agent_name": agent_name,
}
return
if not tool_calls:
# 空 status 数组 = 总控不再派活，前端视为共识（hasConsensus）。
yield {"status": [], "content": leader_text, "agent_name": agent_name}
return
# 派活前按真实 roster 校验:合法保留、非法丢弃(只丢非法,不殃及同轮合法席位)。
valid_dispatch, invalid_names = _partition_dispatch(tool_calls, valid_seat_ids)
if invalid_names:
logger.warning(
"[leader-dispatch] %s 派活到 %d 个不存在的席位 id %s;真实席位=%s(attempt=%d)",
agent_name,
len(invalid_names),
invalid_names,
sorted(valid_seat_ids),
attempt,
)
# 整轮全是非法 id 且还有重试预算 → 回灌纠正消息 + 重跑一轮让总控自纠。
if invalid_names and not valid_dispatch and attempt < _MAX_DISPATCH_RETRIES:
try:
await _append_thread_message(
client,
base,
headers,
thread_id,
_build_dispatch_correction(invalid_names, valid_seat_ids),
)
except Exception as exc:
logger.warning("[leader-dispatch] 回灌纠正消息失败: %s", exc)
attempt += 1
continue
# 非法 id 已被丢弃;若 roster 校验后无合法派活(且重试已用尽),退化为空共识,
# 而不是把假 id 漏给前端再报 400。
dispatched: list[list[str]] = [[sub_name, task_text] for sub_name, task_text in valid_dispatch]
# 关键路径优先:**先 yield 派活帧**,让前端立刻启动席位 special run;派活广播(告知
# 其它席位"总控给 X 派了什么活")改成 detached 后台任务,不再 await 阻塞 —— 否则广播
# 读写各席位巨大 thread 状态会卡几十秒,白白拖慢席位启动(见 _spawn_detached_broadcast)。
yield {"status": dispatched, "content": leader_text, "agent_name": agent_name}
# DAG 派活轮(suppress_pre_broadcast)同样**不**把「总控给 X 派活」广播给兄弟席位 ——
# 否则弱模型从历史里看到派活语句又会模仿派活(席位任务由前端 pickDagSeatTask 干净下发,
# 跨席位承上靠各 special run 的"完成交付"后广播,不需要这条派活广播)。recommend/chain 照旧。
if not suppress_pre_broadcast:
for sub_name, task_text in valid_dispatch:
_spawn_detached_broadcast(
base,
headers,
f"总控智能体给{sub_name}派活,内容为:" + task_text,
agent_threads,
[agent_name, sub_name],
)
return
async def _special_run(
@@ -1153,6 +1530,8 @@ async def _special_run(
model_name: str,
*,
display_name: str = "",
no_file: bool = False,
peer_deliveries: list[dict[str, Any]] | None = None,
) -> AsyncIterator[str | dict[str, Any]]:
"""子智能体（special）单轮：执行总控下发的 task 文本，交付 content。
@@ -1160,26 +1539,42 @@ async def _special_run(
- 终态 ``status`` 为描述性字符串（非数组），``content`` 为交付正文；
- **后广播**在 yield 终态帧之后 await，保证其它席位 state 里有完成摘要，
同时尽量让前端先收到结果关闭 loading。
``no_file``：DAG 并行批次里的席位置真 —— 用 ``seat_parallel`` policy 额外禁
write_file / bash / str_replace，避免并行席位同时写沙箱打架（§4）。
``peer_deliveries``：前序席位的**完整交付** ``[{name, content}, ...]``。注入到
run context 的 ``roundtable_peer_deliveries``,供席位用 ``read_peer_delivery`` 工具
**按需**取全文(平时只看广播摘要,§12.9 传摘要按需传全文)。不进 prompt,不耗 token。
"""
t0 = time.perf_counter()
thread_id = agent_threads[agent_name]
excluded_tools = ["web_search", "ask_clarification", "present_files", "view_image", "agent_orchestration"]
# 角色参数取自单一数据源 roundtable_run_policy（seat / seat_parallel / report）。
# report 在 seat 基础上额外禁 bash / str_replace；seat_parallel 额外禁
# write_file / bash / str_replace（理由见 roundtable_run_policy 模块注释）。
# 后台进程内网关用同一份 policy，保证「后台挂起」与「前端手动调用」传参一致。
if agent_name == REPORT_AGENT_ID:
# Step 3「结果绘制」的报告智能体只需**直接 write_file 写出完整 HTML**。
# 额外禁用 bash / str_replace：
# - 禁 bash → 杜绝它写 Python/脚本再跑出 HTML（脚本产物落在 outputs 里，
# 但前端虚拟预览只抓 write_file 的 args.content，抓不到脚本产物 → 预览停在
# 那个半截 write_file 上，正是用户看到的「没写完就去生成 py 文件」）；
# - 禁 str_replace → 强制返工用整文件 write_file 重写（str_replace 没有 content
# 字段，会让前端虚拟预览读到 undefined 而清空沙箱）。
excluded_tools = [*excluded_tools, "bash", "str_replace"]
role = "report"
elif no_file:
role = "seat_parallel"
else:
role = "seat"
policy = roundtable_run_policy(role)
body = _run_payload(
agent_name=agent_name,
model_name=model_name,
thread_id=thread_id,
new_message=new_message,
excluded_tools=excluded_tools,
skill_stop_names=skill_stop_names,
excluded_tools=policy.excluded_tools,
skill_stop_names=merge_skill_stop_names(policy.skill_stop_names, skill_stop_names),
thinking_enabled=policy.thinking_enabled,
reasoning_effort=policy.reasoning_effort,
# 干预/重新派活会把同一席位 thread 再跑一次;抢占掉没结束的旧 run,避免 409。
multitask_strategy="rollback",
# 前序席位完整交付 → run context,供 read_peer_delivery 工具按需取全文(§12.9)。
extra_context=(
{"roundtable_peer_deliveries": peer_deliveries} if peer_deliveries else None
),
)
t_payload_built = time.perf_counter()
logger.info(
@@ -1215,6 +1610,15 @@ async def _special_run(
)
messages = _parse_last_messages(captured)
if messages is None:
# 与 _leader_run 同理:席位 run 被抢占/断开导致流截断,不抛 502,静默收尾。
# WARNING 级:正常是 rollback 抢占;否则查上游 run 的服务端报错(如 DB 连接断)。
logger.warning(
"[special-timing] %s upstream truncated (no values frame) — ending quietly "
"(expected on rollback preemption; otherwise check the upstream run's server-side error)",
agent_name,
)
return
# Sub-agent's final deliverable is the latest AIMessage's text content.
last_msg: dict[str, Any] = {}
for msg in reversed(messages):
@@ -1222,41 +1626,22 @@ async def _special_run(
last_msg = msg
break
content = _flatten_content(last_msg.get("content", ""))
# Kick the "子智能体完成" broadcast off in the background and yield the
# status frame first — the next leader cycle that depends on this
# broadcast having landed only starts after a fresh /run/stream request
# from the frontend, which is many seconds away (full leader LLM call).
# Awaiting after the yield ensures the task finishes before the AsyncClient
# closes; the SSE connection stays open a bit longer but the frontend has
# already consumed the result.
t_before_post = time.perf_counter()
# 优先用 display_name(中文友好,例如"情报收集")提示 leader,id 放括号备查;
# display_name 缺失时退化为仅 agent_name —— 与旧行为一致。
speaker_label = f"{display_name}({agent_name})" if display_name else agent_name
post_broadcast_task = asyncio.create_task(
_broadcast(
client,
base,
headers,
f"子智能体{speaker_label}完成{new_message}工作,交付内容为:" + content,
agent_threads,
[agent_name],
)
)
_prefix = f"子智能体{speaker_label}完成{new_message}工作,交付内容为:"
# 广播给**所有**席位(含总控)的都只是**截断摘要** —— 总控 thread 不再囤积各席位全文,
# 每轮 leader 的上下文因此保持小、响应快。总控做最终综合需要全文时,由 _leader_run 在
# 「综合轮」按需现读各席位 thread 注入(见 _build_full_deliverables_block),不靠历史囤积。
brief_msg = _truncate_broadcast_content(_prefix, content)
# 先 yield 交付结果让前端立即收口本席位;后广播改 detached,不再 await 阻塞 ——
# 否则它读写各席位大状态会让 SSE 多挂十几秒,把下一轮 leader 的启动也一起拖慢。
yield {
"status": f"子智能体{speaker_label}完成{new_message}工作",
"content": content,
"agent_name": agent_name,
}
try:
await post_broadcast_task
except Exception as exc:
logger.warning("post-broadcast failed: %s", exc)
logger.info(
"[special-timing] %s post_broadcast_total=%.3fs (broadcast time after yield — keeps SSE open this long)",
agent_name,
time.perf_counter() - t_before_post,
)
_spawn_detached_broadcast(base, headers, brief_msg, agent_threads, [agent_name])
# ---------------------------------------------------------------------------
@@ -1292,6 +1677,29 @@ async def run_multi_agent_stream(request: Request, payload: dict = Body(...)) ->
display_name: str = str(payload.get("display_name") or "").strip()
new_message: str = str(payload.get("new_message") or "")
skill_stop_names: list[str] = payload.get("skill_stop_names") or []
# 综合轮标记(仅 leader 用):前端在「派活后让总控判断继续/综合」的轮次置真,网关据此
# 额外把各已交付席位的完整交付正文注入总控输入,供其写最终方案。缺省 False(派活决策轮
# 只给摘要,保持上下文精简)。旧前端不传 → False,行为安全退化。
synthesis_mode: bool = bool(payload.get("synthesis_mode") or False)
# 并行禁写文件标记(仅 special 用):前端 runDagOrchestration 在真并行 stage(≥2 席位)
# 置真,网关据此用 seat_parallel policy 额外禁 write_file/bash/str_replace(§4)。
# 兼容 no_file 别名;缺省 False → 普通 seat,旧前端不传也不破坏。
no_file: bool = bool(payload.get("parallel_no_file") or payload.get("no_file") or False)
# 抑制 leader 预广播(仅 leader 用):DAG「每 stage 派活轮」置真,避免把总控的派活 prompt
# 广播进席位 thread 让弱模型模仿派活(详见 _leader_run 的 suppress_pre_broadcast 注释)。
# 缺省 False,recommend/chain 行为不变。
suppress_pre_broadcast: bool = bool(payload.get("suppress_pre_broadcast") or False)
# 前序席位完整交付(仅 special 用):前端 DAG stage>0 席位带上此前各阶段的全文,网关注入
# run context,供 read_peer_delivery 工具按需取全文(平时只看广播摘要,§12.9)。规整成
# [{name, content}] 且过滤空项;缺省 None。
peer_deliveries: list[dict[str, Any]] | None = None
_pd_raw = payload.get("peer_deliveries")
if isinstance(_pd_raw, list):
peer_deliveries = [
{"name": str(p.get("name") or ""), "content": str(p.get("content") or "")}
for p in _pd_raw
if isinstance(p, dict) and str(p.get("name") or "").strip() and str(p.get("content") or "").strip()
] or None
# 不写死任何默认模型;空串会让 lead_agent._resolve_model_name() 回退到
# config.yaml 的 models[0],与 /api/ai-writing 等其它路由一致。
model_name: str = str(payload.get("model") or "")
@@ -1308,6 +1716,20 @@ async def run_multi_agent_stream(request: Request, payload: dict = Body(...)) ->
base = _loopback_base(request)
headers = _auth_headers(request)
# 总控派活前,取本次各席位的元数据(id+名称+描述);_leader_run 会据此**实读各席位
# thread 的真实交付状态**,把「席位清单 + ✅/⬜交付状态」拼进总控本轮输入最前面 ——
# 既让弱模型每轮都能逐字复制真实 id,又以系统客观核对的交付状态压制臆造。元数据走同进程
# 批量查;任一席位查不到/不可见就退化为仅 id 清单,绝不因此阻断 leader run。
seat_meta: dict[str, dict[str, Any]] | None = None
if agent_type == "leader":
seat_ids = [k for k in agent_threads if k != agent_name]
if seat_ids:
try:
seat_meta = await _get_agents_batch_direct(request, seat_ids)
except Exception:
logger.warning("[leader-roster] 取席位元数据失败,退化为仅 id 清单", exc_info=True)
seat_meta = {sid: {"id": sid, "name": sid, "description": ""} for sid in seat_ids}
async def stream_generator() -> AsyncIterator[str]:
# TCP loopback 到 127.0.0.1:<gateway port>。client 配置见 _make_loopback_client：
# trust_env=False 防 HTTP_PROXY 拦截,timeout=None 容纳 SSE 长连接。
@@ -1316,11 +1738,16 @@ async def run_multi_agent_stream(request: Request, payload: dict = Body(...)) ->
if agent_type == "leader":
gen = _leader_run(
client, base, headers, agent_threads, agent_name, new_message, skill_stop_names, model_name,
seat_meta=seat_meta,
synthesis_mode=synthesis_mode,
suppress_pre_broadcast=suppress_pre_broadcast,
)
else:
gen = _special_run(
client, base, headers, agent_threads, agent_name, new_message, skill_stop_names, model_name,
display_name=display_name,
no_file=no_file,
peer_deliveries=peer_deliveries,
)
async for chunk in gen:
if isinstance(chunk, dict):
offline-backend-20260512/backend/app/gateway/routers/parallel_agent_tasks.py
+148
View File
@@ -0,0 +1,148 @@
"""CRUD API for parallel multi-agent task records (并行多智能体任务).
A task = one saved run of the standalone parallel multi-agent panel: a task name
+ brief dispatched to several agents in parallel, plus the resulting message list.
Bound to the requesting user (not the browser), so it follows the account across
devices/sessions — same model as roundtable drafts.
Routes (prefix ``/api/parallel-agent-tasks``):
GET "" list current user's tasks (lightweight meta, no messages)
POST "" create a task
GET "/{task_id}" one task (full, incl. messages)
PUT "/{task_id}" partial update (title / brief / agents / messages)
DELETE "/{task_id}" delete
"""
from __future__ import annotations
import logging
from datetime import datetime
from typing import Any
from uuid import uuid4
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel, Field
from deerflow.runtime.user_context import get_effective_user_id
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/parallel-agent-tasks", tags=["parallel-agent-tasks"])
# ── schemas ────────────────────────────────────────────────────────────────
class TaskAgentSchema(BaseModel):
agent_id: str = Field(min_length=1, max_length=128)
name: str = ""
avatarType: str | None = None
class TaskResponse(BaseModel):
id: str
title: str = ""
brief: str | None = None
agents: list[TaskAgentSchema] = []
# messages is an opaque dialogue list (mirrors the frontend Dialogue[] shape);
# absent on list responses (lightweight), present on get.
messages: list[dict[str, Any]] | None = None
created_at: datetime | str | None = None
updated_at: datetime | str | None = None
class TaskListResponse(BaseModel):
tasks: list[TaskResponse]
class TaskCreateRequest(BaseModel):
id: str | None = Field(default=None, max_length=64)
title: str = Field(default="", max_length=512)
brief: str | None = None
agents: list[TaskAgentSchema] = []
messages: list[dict[str, Any]] = []
class TaskUpdateRequest(BaseModel):
# All optional → callers send only what changed.
title: str | None = Field(default=None, max_length=512)
brief: str | None = None
agents: list[TaskAgentSchema] | None = None
messages: list[dict[str, Any]] | None = None
# ── helpers ──────────────────────────────────────────────────────────────
def _current_user_id(request: Request) -> str:
user = getattr(request.state, "user", None)
if user is not None:
return str(user.id)
return get_effective_user_id()
def _get_store(request: Request):
store = getattr(request.app.state, "parallel_agent_task_store", None)
if store is None:
raise HTTPException(status_code=503, detail="Parallel agent task store not available")
return store
# ── routes ───────────────────────────────────────────────────────────────
@router.get("", response_model=TaskListResponse)
async def list_tasks(request: Request) -> TaskListResponse:
store = _get_store(request)
user_id = _current_user_id(request)
rows = await store.list_tasks(user_id)
return TaskListResponse(tasks=[TaskResponse(**r) for r in rows])
@router.post("", response_model=TaskResponse, status_code=201)
async def create_task(request: Request, body: TaskCreateRequest) -> TaskResponse:
store = _get_store(request)
user_id = _current_user_id(request)
row = await store.create_task(
user_id,
{
"id": (body.id or uuid4().hex),
"title": body.title.strip(),
"brief": body.brief,
"agents": [a.model_dump() for a in body.agents],
"messages": body.messages,
},
)
return TaskResponse(**row)
@router.get("/{task_id}", response_model=TaskResponse)
async def get_task(request: Request, task_id: str) -> TaskResponse:
store = _get_store(request)
user_id = _current_user_id(request)
row = await store.get_task(task_id, user_id)
if row is None:
raise HTTPException(status_code=404, detail="Task not found")
return TaskResponse(**row)
@router.put("/{task_id}", response_model=TaskResponse)
async def update_task(request: Request, task_id: str, body: TaskUpdateRequest) -> TaskResponse:
store = _get_store(request)
user_id = _current_user_id(request)
fields = body.model_dump(exclude_unset=True)
# Normalize agents (pydantic models) to plain dicts for JSON storage.
if "agents" in fields and fields["agents"] is not None:
fields["agents"] = [a if isinstance(a, dict) else a.model_dump() for a in body.agents or []]
row = await store.update_task(task_id, user_id, **fields)
if row is None:
raise HTTPException(status_code=404, detail="Task not found")
return TaskResponse(**row)
@router.delete("/{task_id}", status_code=204)
async def delete_task(request: Request, task_id: str) -> None:
store = _get_store(request)
user_id = _current_user_id(request)
deleted = await store.delete_task(task_id, user_id)
if not deleted:
raise HTTPException(status_code=404, detail="Task not found")
offline-backend-20260512/backend/app/gateway/routers/parallel_agents.py
+440
View File
@@ -0,0 +1,440 @@
"""Server-side parallel fan-out for the standalone parallel multi-agent panel.
The Vue embeddable panel sends a single request — ``{task_name, brief, agent_ids}``
— and this endpoint **fans out** to every agent in parallel, then **merges** their
LangGraph streams into ONE SSE tagged per ``agent_id``, so the client only consumes
one stream and never parses LangGraph internals.
Each fan-out agent runs as a pure **executing seat**: ``excluded_tools`` =
``_EXCLUDED_TOOLS`` (``write_file`` / ``str_replace`` / ``agent_orchestration`` /
``bash``) so a seat can neither write files (parallel writes would clash) nor
dispatch / run scripts (a weak model fed "总控 / 业务链条" framing would otherwise
mimic the controller and "派活"). Read-only tools + ``ask_clarification`` stay on,
so the step bar and clarification cards still work.
Unified event protocol (one JSON object per ``data:`` frame)::
{"type": "agent_start", "agent_id": "..."}
{"type": "delta", "agent_id": "...", "text": "..."} # AI text increment
{"type": "step", "agent_id": "...", "key": "...", "label": "...", "args": {...}}
{"type": "agent_done", "agent_id": "...", "content": "..."}
{"type": "agent_error", "agent_id": "...", "message": "..."}
{"type": "all_done"}
The frame-normalisation (LangGraph ``messages-tuple`` → delta / tool step) mirrors
the frontend parser in ``frontend-web/src/roundtable-planning/api/multi-agent.ts``
(``extractAiTextDelta`` + ``tool_call_chunks`` accumulation).
"""
from __future__ import annotations
import asyncio
import json
import logging
from typing import Any, AsyncIterator
import httpx
from fastapi import APIRouter, Body, HTTPException, Request
from fastapi.responses import StreamingResponse
from pydantic import BaseModel, Field
from app.gateway.routers.multi_agent import (
_ROUNDTABLE_THREAD_METADATA,
_auth_headers,
_create_thread,
_flatten_content,
_loopback_base,
_make_loopback_client,
_parse_last_messages,
_run_payload,
_stream_upstream,
_validate_agent_id,
)
from deerflow.tools.builtins.clarification_utils import resolve_allow_multiple
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/parallel-agents", tags=["parallel-agents"])
# Soft cap on parallel degree — protects remote model concurrency + sandbox.
MAX_PARALLEL_AGENTS = 8
# 禁两类工具：①**写文件**（write_file / str_replace）—— 并行席位互相写文件会打架；
# ②**派活 / 脚本逃逸**（agent_orchestration / bash）—— 本面板里每个智能体只做自己那
# 份分析交付，绝不该扮演「总控」去调度别人。弱模型（deepseek 等）拿到带「总控 / 业务
# 链条」语气的任务后，会概率性地用 agent_orchestration 真去派活，或绕道 bash 跑脚本
# 调内部接口模仿总控调度（bug：单步单智能体链条里子智能体输出「总控调度指令…」并用
# glob / bash 翻找派活脚本）。物理禁掉这两类即可根治。其余只读工具（ls / read_file /
# 检索）与 ask_clarification（协作卡片）保留：既不互相打架，又能正常出步骤条 / 要澄清。
_EXCLUDED_TOOLS: list[str] = [
"write_file", # 并行不互相写文件打架
"str_replace",
"agent_orchestration", # 子智能体不派活 / 不调度（防模仿总控）
"bash", # 禁跑脚本（防绕道脚本调内部接口模仿派活）
]
# ── 专用「绕过常规登录」取 token ──────────────────────────────────────────────
# 内网部署对标准 /api/v1/auth/login/username 有额外操作限制（口令门 / 网关裁剪等），
# 嵌入方拿不到 token。这里给并行多智能体面板单开一个**公共**端点：传用户名直接签发 token
# （账号不存在则自动注册，与 username 登录同一套逻辑）。已在 auth_middleware / csrf_middleware
# 的白名单里放行（公共 + CSRF 豁免）。⚠️ 这是有意的鉴权旁路，仅供内网部署的本功能使用。
class _TokenRequest(BaseModel):
username: str = Field(min_length=1, max_length=128)
class _TokenResponse(BaseModel):
access_token: str
token_type: str = "bearer"
user_id: str
@router.post("/auth/token", response_model=_TokenResponse)
async def issue_token(body: _TokenRequest) -> _TokenResponse:
"""传用户名 → 直接签发 access_token（绕过常规登录，公共端点）。"""
# 懒加载，避免与 auth 路由的模块级循环依赖。
from app.gateway.auth import create_access_token
from app.gateway.routers.auth import _get_or_create_user_by_username
user, _created = await _get_or_create_user_by_username(body.username.strip())
token = create_access_token(str(user.id), token_version=user.token_version)
return _TokenResponse(access_token=token, token_type="bearer", user_id=str(user.id))
# ── frame normalisation (mirrors multi-agent.ts) ─────────────────────────────
def _extract_ai_text(chunk: dict[str, Any]) -> str:
"""AI text delta from a ``messages-tuple`` chunk obj; "" when not AI text."""
type_ = str(chunk.get("type") or "").lower()
if type_ not in ("ai", "aimessagechunk", "aimessage"):
return ""
content = chunk.get("content")
if isinstance(content, str):
return content
if isinstance(content, list):
out: list[str] = []
for item in content:
if isinstance(item, str):
out.append(item)
elif isinstance(item, dict):
t = item.get("text")
if isinstance(t, str):
out.append(t)
return "".join(out)
return ""
def _ingest_tool_chunks(chunks: Any, acc: dict[int, dict[str, Any]]) -> None:
"""累积 ``tool_call_chunks`` 的增量（name + args 字符串），按 index 归并
（mirror multi-agent.ts ``ingestToolCallChunks``）。"""
if not isinstance(chunks, list):
return
for c in chunks:
if not isinstance(c, dict):
continue
idx = c.get("index") if isinstance(c.get("index"), int) else 0
entry = acc.setdefault(idx, {"id": None, "name": "", "argsStr": ""})
cid = c.get("id")
if isinstance(cid, str) and cid:
if entry["id"] and entry["id"] != cid: # 同 index 换了新调用 → 重置
entry["name"] = ""
entry["argsStr"] = ""
entry["id"] = cid
name = c.get("name")
if isinstance(name, str) and name:
entry["name"] = name
a = c.get("args")
if isinstance(a, str):
entry["argsStr"] += a
def _synthetic_tool_calls(acc: dict[int, dict[str, Any]]) -> list[dict[str, Any]]:
"""把累积的 argsStr 尽力 parse 成完整 args，产出 ``{id,name,args}``（未成完整 JSON 则跳过）。"""
out: list[dict[str, Any]] = []
for entry in acc.values():
if not entry["name"]:
continue
s = str(entry["argsStr"]).strip()
if not s:
continue
try:
args = json.loads(s)
except Exception:
continue
if isinstance(args, dict):
out.append({"id": entry["id"], "name": entry["name"], "args": args})
return out
def _clarification_from_messages(messages: list[dict[str, Any]]) -> dict[str, Any] | None:
"""从最终 messages 里找最后一个 ``ask_clarification`` 工具调用，抽出结构化澄清载荷
（字段对齐 multi_agent leader 的 clarification 帧）。"""
for msg in reversed(messages):
if not isinstance(msg, dict):
continue
if (msg.get("type") or "").lower() not in ("ai", "aimessage", "aimessagechunk"):
continue
for tc in msg.get("tool_calls") or []:
if isinstance(tc, dict) and (tc.get("name") or "").lower() == "ask_clarification":
cargs = tc.get("args") if isinstance(tc.get("args"), dict) else {}
return {
"question": str(cargs.get("question") or ""),
"clarification_type": str(cargs.get("clarification_type") or ""),
"clarification_context": str(cargs.get("context") or ""),
"options": cargs.get("options") or [],
"allow_custom": bool(cargs.get("allow_custom", True)),
"allow_multiple": resolve_allow_multiple(cargs),
}
return None
async def _run_one_agent(
client: httpx.AsyncClient,
base: str,
headers: dict[str, str],
agent_id: str,
thread_id: str,
new_message: str,
model_name: str,
) -> AsyncIterator[str | dict[str, Any]]:
"""跑一个智能体一轮（禁写文件 + 禁派活 / 脚本，其余只读工具放开）。
与 ``multi_agent._special_run`` 同形（yield 原始上游 SSE 行 + 末尾一条终态 dict），
但 ``excluded_tools`` = ``_EXCLUDED_TOOLS``（write_file / str_replace /
agent_orchestration / bash），且不做 post-broadcast（独立面板各智能体互不广播）。
"""
body = _run_payload(
agent_name=agent_id,
model_name=model_name,
thread_id=thread_id,
new_message=new_message,
excluded_tools=list(_EXCLUDED_TOOLS),
skill_stop_names=[],
thinking_enabled=True,
reasoning_effort="medium",
multitask_strategy="rollback",
)
captured: list[str] = []
async for line, done in _stream_upstream(client, base, headers, thread_id, body):
if line is not None:
yield line
elif done is not None:
captured = done
messages = _parse_last_messages(captured)
if messages is None:
# 流被抢占/截断,没有终态 values 帧 —— 静默收尾(调用方据无终态帧兜底)。
return
last_ai: dict[str, Any] = {}
for msg in reversed(messages):
if isinstance(msg, dict) and (msg.get("type") or "").lower() in ("ai", "aimessage", "aimessagechunk"):
last_ai = msg
break
content = _flatten_content(last_ai.get("content", ""))
# 收集**所有** AI 消息的 tool_calls（全量 args，按顺序）—— 用于在收口时补发所有步骤
# （含 ls / skill_list 这类无参/中间步骤），保证步骤条不丢步、且 args 完整。
final_tool_calls: list[dict[str, Any]] = []
for msg in messages:
if isinstance(msg, dict) and (msg.get("type") or "").lower() in ("ai", "aimessage", "aimessagechunk"):
for tc in msg.get("tool_calls") or []:
if isinstance(tc, dict):
final_tool_calls.append(tc)
clarification = _clarification_from_messages(messages)
if clarification is not None:
# 智能体在向用户要澄清 —— 交给前端渲染「协作卡片」(对齐圆桌第一步)。
yield {
"type": "clarification",
"content": content,
"clarification": clarification,
"tool_calls": final_tool_calls,
}
else:
yield {
"status": f"子智能体 {agent_id} 完成",
"content": content,
"agent_name": agent_id,
"tool_calls": final_tool_calls,
}
def _build_task_message(task_name: str, brief: str) -> str:
parts: list[str] = []
if task_name.strip():
parts.append(f"任务名称：{task_name.strip()}")
if brief.strip():
parts.append(f"任务简介：\n{brief.strip()}")
parts.append(
"请你从你的专业职责出发，针对以上任务给出你的分析与交付。"
"本轮为**并行**研讨：请把你的产出**直接写在回复正文里**（本轮禁止写文件）。"
)
# 角色边界护栏：本面板里每个智能体都是**执行席位**，不是总控 / 指挥。即便任务里带有
# 「总控 / 业务链条 / 调度 / 派活」字样，也只完成属于自己的那份交付，严禁模仿总控派活。
parts.append(
"【角色边界】你是本轮的**执行席位**，不是总控 / 指挥。即使任务里出现「总控 / 业务链条 / 调度 / "
"派活」等字样，你也只需完成**属于你自己**的那份分析与交付；严禁模仿总控口吻输出「总控调度指令 / "
"调度对象 / 分配给『…』的任务 / agent_name」之类调度内容，严禁罗列或转派其他席位的任务，严禁调用"
"任何派活 / 调度类工具，也不要用脚本（bash / python）去调用内部接口或翻找派活脚本。"
)
return "\n\n".join(parts)
# ── endpoint ─────────────────────────────────────────────────────────────────
@router.post("/run/stream")
async def run_parallel_agents_stream(request: Request, payload: dict = Body(...)) -> StreamingResponse:
"""并行扇出 N 个智能体，合并成一条带 ``agent_id`` 标记的 SSE。文件写入固定禁用。"""
task_name: str = str(payload.get("task_name") or "").strip()
brief: str = str(payload.get("brief") or "").strip()
raw_ids = payload.get("agent_ids") or []
model_name: str = str(payload.get("model") or "")
if not isinstance(raw_ids, list) or not raw_ids:
raise HTTPException(status_code=400, detail="agent_ids 不能为空")
agent_ids: list[str] = []
for aid in raw_ids:
sid = str(aid or "").strip().lower()
_validate_agent_id(sid, "agent_ids")
if sid not in agent_ids:
agent_ids.append(sid)
if len(agent_ids) > MAX_PARALLEL_AGENTS:
raise HTTPException(
status_code=400,
detail=f"并行智能体数量上限 {MAX_PARALLEL_AGENTS}（当前 {len(agent_ids)}）",
)
if not task_name and not brief:
raise HTTPException(status_code=400, detail="任务名称与简介至少填写其一")
base = _loopback_base(request)
headers = _auth_headers(request)
new_message = _build_task_message(task_name, brief)
async def _consume_agent(
client: Any,
agent_id: str,
queue: asyncio.Queue[dict[str, Any]],
) -> None:
await queue.put({"type": "agent_start", "agent_id": agent_id})
last_event = ""
emitted_tools: set[str] = set() # 已发过 step 的 tool_call id（去重）
tool_acc: dict[int, dict[str, Any]] = {} # tool_call_chunks 累积
buffer: list[str] = []
done_sent = False
async def _emit_tool(tc: Any, allow_empty: bool = False) -> None:
"""一个 tool_call（含完整 args）→ 一条 step 事件；按 id 去重。
``allow_empty=False``（流式期间）：args 为空先不发 —— 流式帧里的 tool_calls 常先给
空 ``{}`` 占位、真实 args 走 tool_call_chunks 后到，过早发会把空 args 钉死。等累积出
完整 args（synthetic）或终态全量 tool_calls（``allow_empty=True``）再发。
"""
if not isinstance(tc, dict):
return
name = tc.get("name")
if not isinstance(name, str) or not name:
return
args = tc.get("args") if isinstance(tc.get("args"), dict) else None
if args is None:
return
if not args and not allow_empty: # 空 args 先不发，等 chunks / 终态补全
return
key_id = str(tc.get("id") or name)
if key_id in emitted_tools:
return
emitted_tools.add(key_id)
await queue.put(
{"type": "step", "agent_id": agent_id, "key": f"tool_calling:{name}", "label": name, "args": args}
)
try:
thread_id = await _create_thread(client, base, headers, metadata=_ROUNDTABLE_THREAD_METADATA)
gen = _run_one_agent( # 只禁文件写入，其余工具放开（保留步骤条）
client, base, headers, agent_id, thread_id, new_message, model_name
)
async for chunk in gen:
if isinstance(chunk, dict):
# 终态：先用最终 tool_calls（全量 args）补发漏掉的 step，再发收口/澄清。
for tc in chunk.get("tool_calls") or []:
await _emit_tool(tc, allow_empty=True)
if chunk.get("type") == "clarification":
clar = chunk.get("clarification") or {}
await queue.put(
{
"type": "agent_clarification",
"agent_id": agent_id,
"content": str(chunk.get("content") or ""),
**clar,
}
)
else:
await queue.put(
{"type": "agent_done", "agent_id": agent_id, "content": str(chunk.get("content") or "")}
)
done_sent = True
continue
line = chunk.strip() if isinstance(chunk, str) else ""
if line.startswith("event:"):
last_event = line[6:].strip()
continue
if not line.startswith("data:"):
continue
raw = line[5:].strip()
if not raw:
continue
try:
parsed = json.loads(raw)
except Exception:
continue
if last_event == "messages" and isinstance(parsed, list) and parsed:
obj = parsed[0]
if isinstance(obj, dict):
text = _extract_ai_text(obj)
if text:
buffer.append(text)
await queue.put({"type": "delta", "agent_id": agent_id, "text": text})
# 完整 tool_calls（部分模型直接给）+ 累积 chunks 后的合成调用，都尝试发 step。
for tc in obj.get("tool_calls") or []:
await _emit_tool(tc)
_ingest_tool_chunks(obj.get("tool_call_chunks"), tool_acc)
for tc in _synthetic_tool_calls(tool_acc):
await _emit_tool(tc)
if not done_sent:
# 流被抢占/截断,没收到终态帧 —— 用已累积正文兜底收口本席位。
await queue.put(
{"type": "agent_done", "agent_id": agent_id, "content": "".join(buffer)}
)
except Exception as exc: # noqa: BLE001 — 单个 agent 失败不拖垮其它
logger.warning("[parallel-agents] %s failed: %r", agent_id, exc)
await queue.put({"type": "agent_error", "agent_id": agent_id, "message": str(exc)})
finally:
await queue.put({"type": "__agent_end__", "agent_id": agent_id})
async def stream_generator() -> AsyncIterator[str]:
queue: asyncio.Queue[dict[str, Any]] = asyncio.Queue()
async with _make_loopback_client(request) as client:
tasks = [asyncio.create_task(_consume_agent(client, aid, queue)) for aid in agent_ids]
ended = 0
try:
while ended < len(agent_ids):
event = await queue.get()
if event.get("type") == "__agent_end__":
ended += 1
continue
yield f"data: {json.dumps(event, ensure_ascii=False)}\n\n"
yield f"data: {json.dumps({'type': 'all_done'}, ensure_ascii=False)}\n\n"
finally:
for t in tasks:
if not t.done():
t.cancel()
await asyncio.gather(*tasks, return_exceptions=True)
return StreamingResponse(
stream_generator(),
media_type="text/event-stream",
headers={
"Cache-Control": "no-cache",
"X-Accel-Buffering": "no",
"Connection": "keep-alive",
},
)
offline-backend-20260512/backend/app/gateway/routers/roundtable_chains.py
+113 -2
View File
@@ -46,6 +46,17 @@ class ChainResponse(BaseModel):
title: str = ""
description: str | None = None
seats: list[ChainSeatSchema] = []
# Optional layered orchestration: list of stages, each a list of agent_ids
# run in parallel (stages run serially). None/absent = linear chain.
stages: list[list[str]] | None = None
# Optional per-stage goals aligned to ``stages`` (dag-only). None = no goals.
stage_goals: list[str | None] | None = None
# 写给总控的「编排提示」（dag-only）：告诉总控怎么派活、做什么。None/空 = 不注入。
coordinator_prompt: str | None = None
# 发布状态：True = 公共（所有账号可在「公共」筛选查看 / 选用）。
is_public: bool = False
# 是否当前用户所有（仅本人可编辑 / 删除 / 发布；公共筛选里别人的链条 is_mine=False）。
is_mine: bool = True
created_at: datetime | str | None = None
updated_at: datetime | str | None = None
@@ -61,6 +72,9 @@ class ChainCreateRequest(BaseModel):
title: str = Field(default="", max_length=512)
description: str | None = Field(default=None, max_length=2048)
seats: list[ChainSeatSchema] = []
stages: list[list[str]] | None = None
stage_goals: list[str | None] | None = None
coordinator_prompt: str | None = Field(default=None, max_length=4000)
class ChainUpdateRequest(BaseModel):
@@ -68,6 +82,11 @@ class ChainUpdateRequest(BaseModel):
title: str | None = Field(default=None, max_length=512)
description: str | None = Field(default=None, max_length=2048)
seats: list[ChainSeatSchema] | None = None
stages: list[list[str]] | None = None
stage_goals: list[str | None] | None = None
coordinator_prompt: str | None = Field(default=None, max_length=4000)
# 发布开关（True 发布为公共 / False 取消发布）；仅 owner 可改（update 本就 owner 限定）。
is_public: bool | None = None
# ── helpers ──────────────────────────────────────────────────────────────
@@ -101,20 +120,101 @@ def _validate_seats(seats: list[ChainSeatSchema]) -> None:
seen.add(s.agent_id)
def _validate_stages(seats: list[ChainSeatSchema], stages: list[list[str]] | None) -> None:
"""Validate the optional layering. ``stages`` must partition the seats.
Rules (None = linear chain, always valid):
- at least one stage, no empty stage;
- every agent_id in a stage must be one of the chain's seats;
- no agent_id appears twice (within or across stages);
- every seat must appear in exactly one stage (full coverage), so the
stored layering and the seat list never drift apart.
"""
if stages is None:
return
if len(stages) == 0:
raise HTTPException(status_code=400, detail="编排分层不能为空（至少一个阶段）")
seat_ids = {s.agent_id for s in seats}
seen: set[str] = set()
for idx, stage in enumerate(stages):
if not stage:
raise HTTPException(status_code=400, detail=f"第 {idx + 1} 阶段为空，请删除或放入席位")
for agent_id in stage:
if agent_id not in seat_ids:
raise HTTPException(
status_code=400,
detail=f"编排分层包含不在席位列表中的智能体: {agent_id}",
)
if agent_id in seen:
raise HTTPException(
status_code=400,
detail=f"智能体 {agent_id} 在编排分层中重复出现，每个席位只能跑一次",
)
seen.add(agent_id)
if seen != seat_ids:
missing = seat_ids - seen
raise HTTPException(
status_code=400,
detail=f"编排分层未覆盖全部席位，缺少: {', '.join(sorted(missing))}",
)
# Per-goal length cap to keep injected prompts bounded.
MAX_STAGE_GOAL_CHARS = 1000
def _validate_stage_goals(
stages: list[list[str]] | None, stage_goals: list[str | None] | None
) -> None:
"""Validate the optional per-stage goals (None = no goals, always valid).
Rules:
- goals only make sense alongside a non-null layering (dag-only): reject
goals sent without ``stages``;
- length must match ``stages`` exactly (positional alignment);
- each goal (when present) must be ≤ ``MAX_STAGE_GOAL_CHARS`` chars.
"""
if stage_goals is None:
return
if stages is None:
raise HTTPException(status_code=400, detail="阶段目标需与编排分层一起提交（线性链不支持阶段目标）")
if len(stage_goals) != len(stages):
raise HTTPException(
status_code=400,
detail=f"阶段目标数量（{len(stage_goals)}）与编排分层阶段数（{len(stages)}）不一致",
)
for idx, goal in enumerate(stage_goals):
if goal is not None and len(goal) > MAX_STAGE_GOAL_CHARS:
raise HTTPException(
status_code=400,
detail=f"第 {idx + 1} 阶段目标过长（上限 {MAX_STAGE_GOAL_CHARS} 字）",
)
# ── routes ───────────────────────────────────────────────────────────────
@router.get("", response_model=ChainListResponse)
async def list_chains(request: Request) -> ChainListResponse:
async def list_chains(request: Request, scope: str = "mine") -> ChainListResponse:
"""列出业务链条。
``scope`` 查询参数(筛选):
- ``mine``（默认）：仅当前用户自己的链条；
- ``public``：所有**已发布**的公共链条（任何账号均可查看 / 选入研讨）。
其余值按 ``mine`` 处理。
"""
normalized_scope = scope if scope in ("mine", "public") else "mine"
store = _get_store(request)
user_id = _current_user_id(request)
rows = await store.list_chains(user_id)
rows = await store.list_chains(user_id, scope=normalized_scope)
return ChainListResponse(chains=[ChainResponse(**r) for r in rows])
@router.post("", response_model=ChainResponse, status_code=201)
async def create_chain(request: Request, body: ChainCreateRequest) -> ChainResponse:
_validate_seats(body.seats)
_validate_stages(body.seats, body.stages)
_validate_stage_goals(body.stages, body.stage_goals)
store = _get_store(request)
user_id = _current_user_id(request)
row = await store.create_chain(
@@ -124,6 +224,9 @@ async def create_chain(request: Request, body: ChainCreateRequest) -> ChainRespo
"title": body.title.strip(),
"description": body.description,
"seats": [s.model_dump() for s in body.seats],
"stages": body.stages,
"stage_goals": body.stage_goals,
"coordinator_prompt": body.coordinator_prompt,
},
)
return ChainResponse(**row)
@@ -143,6 +246,14 @@ async def get_chain(request: Request, chain_id: str) -> ChainResponse:
async def update_chain(request: Request, chain_id: str, body: ChainUpdateRequest) -> ChainResponse:
if body.seats is not None:
_validate_seats(body.seats)
# A non-null stages layering can only be validated against seats; the editor
# always saves both together. Reject a layering update that omits seats.
if body.stages is not None:
if body.seats is None:
raise HTTPException(status_code=400, detail="更新编排分层时必须同时提交席位列表")
_validate_stages(body.seats, body.stages)
# Goals align to stages; reject goals sent without a layering, enforce length/size.
_validate_stage_goals(body.stages, body.stage_goals)
store = _get_store(request)
user_id = _current_user_id(request)
# Distinguish "omitted" from "explicitly sent"; only forward provided fields.
offline-backend-20260512/backend/app/gateway/routers/roundtable_draft_shares.py
+330
View File
@@ -0,0 +1,330 @@
"""Roundtable-draft sharing API.
A user creates a share code for *one* of their roundtable-planning drafts (a full
会商 session). Two modes:
- ``import`` — redeeming the code copies the draft into the redeemer's account as
an independent copy they fully own (fresh draft id, deduped per recipient).
- ``view`` — a public, read-only page that renders the draft's Step 3 result
report (the self-contained 方案总览 HTML). Anyone with the link can open it
without logging in.
Mirrors ``app/gateway/routers/thread_shares.py`` but the shared resource is a
``roundtable_drafts`` row (read via ``app.state.roundtable_draft_store``) rather
than a conversation thread.
"""
from __future__ import annotations
import hashlib
import logging
from fastapi import APIRouter, Depends, HTTPException, Request
from pydantic import BaseModel, Field
from app.gateway.deps import get_roundtable_draft_share_store
from deerflow.persistence.roundtable_draft_shares.base import RoundtableDraftShareStore
from deerflow.runtime.user_context import get_effective_user_id
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/roundtable-draft-shares", tags=["roundtable-draft-shares"])
# Public, unauthenticated router. Its prefix is whitelisted in
# ``auth_middleware._PUBLIC_PATH_PREFIXES`` (``/api/public/``) — only read-only,
# intentionally world-readable endpoints belong here.
public_router = APIRouter(prefix="/api/public/roundtable-shares", tags=["roundtable-draft-shares-public"])
_UNTITLED = "未命名会商"
_SHARE_MODES = ("import", "view")
def _current_user_id(request: Request) -> str:
user = getattr(request.state, "user", None)
if user is not None:
return str(user.id)
return get_effective_user_id()
def _get_draft_store(request: Request):
store = getattr(request.app.state, "roundtable_draft_store", None)
if store is None:
raise HTTPException(status_code=503, detail="Roundtable draft store not available")
return store
# ---------------------------------------------------------------------------
# Request / response models
# ---------------------------------------------------------------------------
class CreateShareRequest(BaseModel):
draft_id: str = Field(..., description="The roundtable draft to share")
mode: str = Field(
default="import",
description='"import" — recipient copies the draft; "view" — public read-only result page',
)
class ShareResponse(BaseModel):
share_code: str = Field(..., description="Short code that recipients enter to import")
owner_user_id: str = Field(..., description="User who created the share")
draft_id: str = Field(..., description="The shared draft")
revoked: bool = Field(default=False, description="Whether the code has been revoked")
mode: str = Field(default="import", description='"import" or "view"')
created_at: str | None = Field(default=None, description="Creation timestamp (ISO-8601)")
class SharesListResponse(BaseModel):
shares: list[ShareResponse]
class SharePreviewResponse(BaseModel):
share_code: str
shared_by: str = Field(..., description="Owner user id of the shared draft")
title: str = Field(..., description="Title of the shared draft")
class ImportResultResponse(BaseModel):
imported: int = Field(..., description="1 when the draft was copied, else 0")
skipped: int = Field(..., description="1 when it was already imported, else 0")
draft_id: str | None = Field(default=None, description="The new local draft id (when imported)")
class SeatDeliveryItem(BaseModel):
"""One sub-agent's final delivery from the Step 2 roundtable."""
sender: str = ""
content: str = ""
class PublicDraftShareSnapshotResponse(BaseModel):
"""Read-only roundtable-result snapshot served to anonymous viewers.
Mirrors what the in-app Step 3「最终方案展示」(`HighFidelityReport`) shows, minus
the live per-seat output files (those need authenticated thread access):
总控最终结论 + 各席位最终交付 + the generated 结果绘制 report.
"""
share_code: str
title: str = Field(..., description="Draft title")
mode: str = Field(default="view")
# 总控最终结论 —— the coordinator's final consensus text (Step 2 lastLeaderContent).
consensus_text: str = Field(default="", description="Coordinator's final consensus (markdown)")
# 各席位最终交付 —— each sub-agent's final delivery, filtered from Step 2 dialogues.
seat_deliveries: list[SeatDeliveryItem] = Field(default_factory=list)
# The Step 3「结果绘制」report — a self-contained HTML document. Empty when the
# shared draft never reached / generated a result report.
html: str = Field(default="", description="Self-contained Step 3 result report HTML")
summary: str | None = Field(default=None, description="The report's closing one-liner")
generated_at: str | None = Field(default=None, description="When the report was generated (ISO-8601)")
has_report: bool = Field(default=False, description="Whether a Step 3 report exists")
created_at: str | None = Field(default=None, description="When the share was created (ISO-8601)")
# ---------------------------------------------------------------------------
# Owner side — create / list / revoke
# ---------------------------------------------------------------------------
@router.post("", response_model=ShareResponse, summary="Create a share code for one roundtable draft")
async def create_share(
body: CreateShareRequest,
request: Request,
share_store: RoundtableDraftShareStore = Depends(get_roundtable_draft_share_store),
) -> ShareResponse:
me = _current_user_id(request)
mode = body.mode if body.mode in _SHARE_MODES else "import"
# Only the draft's owner may share it.
draft = await _get_draft_store(request).get_draft(body.draft_id, me)
if draft is None:
raise HTTPException(status_code=404, detail="Draft not found")
# Reuse an existing active code (of the same mode) so re-sharing is idempotent.
existing = await share_store.get_active_for_draft(me, body.draft_id, mode=mode)
if existing is not None:
return ShareResponse(**existing)
record = await share_store.create(me, body.draft_id, mode=mode)
return ShareResponse(**record)
@router.get("", response_model=SharesListResponse, summary="List my roundtable-draft share codes")
async def list_shares(
request: Request,
share_store: RoundtableDraftShareStore = Depends(get_roundtable_draft_share_store),
) -> SharesListResponse:
records = await share_store.list_for_owner(_current_user_id(request))
return SharesListResponse(shares=[ShareResponse(**r) for r in records])
@router.delete("/{share_code}", summary="Revoke a roundtable-draft share code")
async def revoke_share(
share_code: str,
request: Request,
share_store: RoundtableDraftShareStore = Depends(get_roundtable_draft_share_store),
) -> dict[str, bool]:
ok = await share_store.revoke(share_code, _current_user_id(request))
if not ok:
raise HTTPException(status_code=404, detail="Share code not found")
return {"success": True}
# ---------------------------------------------------------------------------
# Recipient side — preview / import
# ---------------------------------------------------------------------------
async def _resolve_active_share(share_code: str, share_store: RoundtableDraftShareStore) -> dict:
share = await share_store.get(share_code)
if share is None:
raise HTTPException(status_code=404, detail="Share code not found")
if share.get("revoked"):
raise HTTPException(status_code=410, detail="This share code has been revoked")
return share
@router.get("/{share_code}/preview", response_model=SharePreviewResponse, summary="Preview a shared roundtable draft")
async def preview_share(
share_code: str,
request: Request,
share_store: RoundtableDraftShareStore = Depends(get_roundtable_draft_share_store),
) -> SharePreviewResponse:
share = await _resolve_active_share(share_code, share_store)
owner = share["owner_user_id"]
draft = await _get_draft_store(request).get_draft(share["draft_id"], owner)
if draft is None:
raise HTTPException(status_code=404, detail="The shared draft no longer exists")
return SharePreviewResponse(
share_code=share_code,
shared_by=owner,
title=draft.get("title") or _UNTITLED,
)
def _imported_draft_id(share_code: str, recipient: str) -> str:
"""Deterministic per-(code, recipient) draft id so re-import is idempotent.
``roundtable_drafts.id`` is a global PK, so the id must be unique across
users — hence the recipient is folded into the hash rather than reusing a
code-only id (which would collide when two people import the same code).
"""
digest = hashlib.sha1(f"{share_code}:{recipient}".encode()).hexdigest()
return f"imp_{digest[:40]}"
@router.post("/{share_code}/import", response_model=ImportResultResponse, summary="Import a shared roundtable draft")
async def import_share(
share_code: str,
request: Request,
share_store: RoundtableDraftShareStore = Depends(get_roundtable_draft_share_store),
) -> ImportResultResponse:
share = await _resolve_active_share(share_code, share_store)
if share.get("mode") == "view":
raise HTTPException(status_code=400, detail="This is a view-only share — open its link to read it instead of importing")
owner = share["owner_user_id"]
origin_id = share["draft_id"]
recipient = _current_user_id(request)
if owner == recipient:
raise HTTPException(status_code=400, detail="You cannot import your own share code")
draft_store = _get_draft_store(request)
src = await draft_store.get_draft(origin_id, owner)
if src is None:
raise HTTPException(status_code=404, detail="The shared draft no longer exists")
# Dedup: a deterministic id per (code, recipient) makes re-import a no-op.
new_id = _imported_draft_id(share_code, recipient)
if await draft_store.get_draft(new_id, recipient) is not None:
return ImportResultResponse(imported=0, skipped=1, draft_id=new_id)
await draft_store.create_draft(
recipient,
{
"id": new_id,
"title": src.get("title") or _UNTITLED,
"furthest_step": src.get("furthest_step") or 1,
"step1": src.get("step1"),
"step2": src.get("step2"),
"step3": src.get("step3"),
},
)
logger.info("Roundtable share %s imported by %s as draft %s", share_code, recipient, new_id)
return ImportResultResponse(imported=1, skipped=0, draft_id=new_id)
# ---------------------------------------------------------------------------
# Public read-only view — no authentication required
# ---------------------------------------------------------------------------
def _result_sections(draft: dict) -> tuple[str, list[dict[str, str]]]:
"""Derive (总控最终结论, 各席位最终交付) from a draft's Step 2 snapshot.
Mirrors the frontend `Step3Panel`: the consensus is `step2.lastLeaderContent`
and each seat delivery is a Step 2 dialogue bubble tagged
``confidence == "子智能体交付"``. Falls back to the active run's snapshot when
the top-level Step 2 fields are empty (multi-run drafts).
"""
step2 = draft.get("step2") if isinstance(draft.get("step2"), dict) else {}
step2 = step2 or {}
consensus = step2.get("lastLeaderContent") or ""
dialogues = step2.get("step2RoundtableDialogues")
if not isinstance(dialogues, list):
dialogues = []
# Multi-run drafts keep the freshest content on the active run; use it when
# the top-level snapshot carries nothing.
if not consensus and not dialogues and isinstance(step2.get("runs"), list):
runs = [r for r in step2["runs"] if isinstance(r, dict)]
active_id = step2.get("activeRunId")
active = next((r for r in runs if r.get("id") == active_id), None) or (runs[0] if runs else None)
if active:
consensus = active.get("lastLeaderContent") or ""
d = active.get("step2RoundtableDialogues")
dialogues = d if isinstance(d, list) else []
seat_deliveries = [
{"sender": str(d.get("sender") or ""), "content": str(d.get("content") or "")}
for d in dialogues
if isinstance(d, dict) and d.get("confidence") == "子智能体交付"
]
return consensus, seat_deliveries
@public_router.get(
"/{share_code}",
response_model=PublicDraftShareSnapshotResponse,
summary="Public read-only snapshot of a view-shared roundtable draft",
)
async def get_public_draft_share_snapshot(
share_code: str,
request: Request,
share_store: RoundtableDraftShareStore = Depends(get_roundtable_draft_share_store),
) -> PublicDraftShareSnapshotResponse:
"""Serve a frozen, read-only roundtable result. Anyone with the link can read it.
Only ``mode="view"`` shares are exposed; missing, revoked, or import-mode
codes all return 404 so the endpoint never reveals which case applies.
"""
share = await share_store.get(share_code)
if share is None or share.get("revoked") or share.get("mode") != "view":
raise HTTPException(status_code=404, detail="Share not found")
draft = await _get_draft_store(request).get_draft(share["draft_id"], share["owner_user_id"])
if draft is None:
raise HTTPException(status_code=404, detail="The shared draft no longer exists")
step3 = draft.get("step3") if isinstance(draft.get("step3"), dict) else None
html = (step3 or {}).get("html") or ""
consensus_text, seat_deliveries = _result_sections(draft)
return PublicDraftShareSnapshotResponse(
share_code=share_code,
title=draft.get("title") or _UNTITLED,
mode="view",
consensus_text=consensus_text,
seat_deliveries=[SeatDeliveryItem(**s) for s in seat_deliveries],
html=html,
summary=(step3 or {}).get("summary"),
generated_at=(step3 or {}).get("generatedAt"),
has_report=bool(html),
created_at=share.get("created_at"),
)
offline-backend-20260512/backend/app/gateway/routers/roundtable_jobs.py
+7
View File
@@ -97,6 +97,8 @@ class RoundtableJobResponse(BaseModel):
orchestrationMode: str = "recommend"
chain: dict[str, Any] | None = None
error: str | None = None
dialogues: list[dict[str, Any]] = Field(default_factory=list)
step3: dict[str, Any] | None = None
createdAt: datetime | str | None = None
updatedAt: datetime | str | None = None
@@ -147,6 +149,8 @@ def _to_response(row: dict[str, Any]) -> RoundtableJobResponse:
orchestrationMode=row.get("orchestration_mode") or "recommend",
chain=row.get("chain"),
error=row.get("error"),
dialogues=row.get("dialogues") or [],
step3=row.get("step3"),
createdAt=row.get("created_at"),
updatedAt=row.get("updated_at"),
)
@@ -199,6 +203,7 @@ async def start_job(request: Request, body: StartJobRequest) -> RoundtableJobRes
thread_ids=body.threadIds,
coordinator_name=body.coordinatorName,
draft_id=body.draftId,
chain=body.chain,
)
)
return _to_response(row)
@@ -282,6 +287,8 @@ async def resume_job(request: Request, job_id: str, body: ResumeJobRequest) -> R
thread_ids=row.get("thread_ids"),
coordinator_name=row.get("coordinator_name") or "roundtable-coordinator",
draft_id=row.get("draft_id"),
is_resume=True,
chain=row.get("chain"),
)
)
return _to_response(updated or row)
offline-backend-20260512/backend/app/gateway/routers/skills.py
+70 -1
View File
@@ -7,7 +7,7 @@ import re
import subprocess
import sys
import tempfile
from datetime import datetime
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
@@ -119,8 +119,10 @@ class SkillResponse(BaseModel):
view_count: int = Field(default=0, description="Number of times the skill has been viewed")
patch_count: int = Field(default=0, description="Number of times the skill has been patched")
last_activity_at: datetime | None = Field(default=None, description="Most recent activity timestamp")
created_at: datetime | None = Field(default=None, description="Skill creation timestamp when known")
state: str = Field(default="active", description="Lifecycle state: active, stale, or archived")
pinned: bool = Field(default=False, description="Whether this skill is pinned (excluded from curation)")
featured: bool = Field(default=False, description="Whether this skill is pinned to the top of the list (sort only)")
source: str = Field(default="upload", description="Origin: 'agent' for agent-created, 'upload' for user-uploaded")
owner_user_id: str | None = Field(default=None, description="Custom skill owner (null = built-in or legacy)")
published: bool = Field(default=False, description="Whether the owner has made this skill visible to other users")
@@ -241,8 +243,20 @@ def _skill_to_response(
from deerflow.skills.usage import get_entry, latest_activity_at
entry = get_entry(skill.name)
created_at_str = entry.get("created_at")
last_at_str = latest_activity_at(skill.name)
created_at: datetime | None = None
last_at: datetime | None = None
if created_at_str:
try:
created_at = datetime.fromisoformat(created_at_str)
except Exception:
pass
if created_at is None:
try:
created_at = datetime.fromtimestamp(skill.skill_file.stat().st_mtime, tz=UTC)
except Exception:
pass
if last_at_str:
try:
last_at = datetime.fromisoformat(last_at_str)
@@ -278,8 +292,10 @@ def _skill_to_response(
view_count=entry.get("view_count", 0),
patch_count=entry.get("patch_count", 0),
last_activity_at=last_at,
created_at=created_at,
state=entry.get("state", "active"),
pinned=entry.get("pinned", False),
featured=entry.get("featured", False),
source=entry.get("source", "upload"),
owner_user_id=owner_user_id,
published=published,
@@ -688,6 +704,17 @@ async def _attach_and_filter_tags(
return responses
def _sort_skill_responses(responses: list[SkillResponse], sort: str) -> list[SkillResponse]:
reverse = sort not in {"time_asc", "activity_asc", "name_asc", "usage_asc"}
if sort in {"name_asc", "name_desc"}:
return sorted(responses, key=lambda r: (r.name or "").lower(), reverse=reverse)
if sort in {"usage_desc", "usage_asc"}:
return sorted(responses, key=lambda r: int(r.use_count or 0), reverse=reverse)
if sort in {"activity_desc", "activity_asc"}:
return sorted(responses, key=lambda r: str(r.last_activity_at or r.created_at or ""), reverse=reverse)
return sorted(responses, key=lambda r: str(r.created_at or r.last_activity_at or ""), reverse=reverse)
@router.get(
"/skills",
response_model=SkillsListResponse,
@@ -701,6 +728,11 @@ async def list_skills(
tag_store: TagStore = Depends(get_tag_store),
search: str | None = Query(default=None, description="Case-insensitive fuzzy match on skill name"),
tag_ids: list[str] | None = Query(default=None, description="Keep skills carrying any of these tags (OR)"),
sort: str = Query(
default="time_desc",
pattern="^(time_desc|time_asc|activity_desc|activity_asc|usage_desc|usage_asc|name_asc|name_desc)$",
description="Sort order. Default is newest created/modified first.",
),
admin: bool = Query(
default=False,
description=(
@@ -754,6 +786,7 @@ async def list_skills(
)
)
responses = await _attach_and_filter_tags(tag_store, responses, search, tag_ids)
responses = _sort_skill_responses(responses, sort)
return SkillsListResponse(skills=responses)
except HTTPException:
raise
@@ -1438,6 +1471,42 @@ async def update_custom_skill_pinned(
raise HTTPException(status_code=500, detail=f"Failed to update pinned: {e}")
class SkillFeaturedRequest(BaseModel):
"""Request body for pinning a custom skill to the top of the list."""
featured: bool = Field(..., description="Whether to pin the skill to the top of the list (sort only)")
@router.put(
"/skills/custom/{skill_name}/featured",
response_model=SkillResponse,
summary="Pin or Unpin a Custom Skill to Top",
description="Pin a skill to the top of the list (sort only; orthogonal to pinned/anti-archive). Only the owner or admins may invoke.",
)
async def update_custom_skill_featured(
skill_name: str,
body: SkillFeaturedRequest,
request: Request,
config: AppConfig = Depends(get_config),
skill_store: SkillStore = Depends(get_skill_store),
) -> SkillResponse:
try:
skill_name = skill_name.replace("\r\n", "").replace("\n", "")
user_id = _current_user_id(request)
is_admin = _is_admin_user(request)
record, _mode = await _get_editable_skill_or_raise(skill_store, skill_name, user_id, is_admin=is_admin)
from deerflow.skills.usage import set_featured as _usage_set_featured
_usage_set_featured(skill_name, body.featured)
return _reload_skill_response(skill_name, record, config)
except HTTPException:
raise
except Exception as e:
logger.error("Failed to update featured for %s: %s", skill_name, e, exc_info=True)
raise HTTPException(status_code=500, detail=f"Failed to update featured: {e}")
@router.get("/skills/custom/{skill_name}/history", response_model=CustomSkillHistoryResponse, summary="Get Custom Skill History")
async def get_custom_skill_history(skill_name: str, config: AppConfig = Depends(get_config)) -> CustomSkillHistoryResponse:
try:
offline-backend-20260512/backend/app/gateway/routers/system_settings.py
+94 -6
View File
@@ -10,7 +10,9 @@ from pydantic import BaseModel, Field
from app.gateway.auth.password import hash_password_async
from app.gateway.deps import get_optional_user_from_request
from deerflow.config import get_app_config
from deerflow.config.system_settings import (
LeaderboardSettings,
PromptPrefixSettings,
SystemSettings,
load_system_settings,
@@ -57,6 +59,22 @@ class CitationDisplayUpdateRequest(BaseModel):
cleanup_words: list[str] | None = None
class LeaderboardSettingsResponse(BaseModel):
exclude_user_ids: list[str] = Field(default_factory=list)
include_scheduled: bool = True
include_admins: bool = True
include_failed: bool = True
cleanup_system_questions: bool = True
class LeaderboardSettingsUpdateRequest(BaseModel):
exclude_user_ids: list[str] | None = None
include_scheduled: bool | None = None
include_admins: bool | None = None
include_failed: bool | None = None
cleanup_system_questions: bool | None = None
def _to_response(settings: PromptPrefixSettings) -> PromptPrefixResponse:
return PromptPrefixResponse(
enabled=settings.enabled,
@@ -91,6 +109,44 @@ async def update_prompt_prefix(body: PromptPrefixUpdateRequest, request: Request
return _to_response(saved.prompt_prefix)
def _leaderboard_to_response(settings: LeaderboardSettings) -> LeaderboardSettingsResponse:
return LeaderboardSettingsResponse(
exclude_user_ids=list(dict.fromkeys([uid for uid in settings.exclude_user_ids if uid])),
include_scheduled=settings.include_scheduled,
include_admins=settings.include_admins,
include_failed=settings.include_failed,
cleanup_system_questions=settings.cleanup_system_questions,
)
@router.get("/leaderboard", response_model=LeaderboardSettingsResponse)
async def get_leaderboard_settings(request: Request) -> LeaderboardSettingsResponse:
"""Return global leaderboard aggregation settings. Admin only."""
await _require_admin(request)
return _leaderboard_to_response(load_system_settings().leaderboard)
@router.put("/leaderboard", response_model=LeaderboardSettingsResponse)
async def update_leaderboard_settings(
body: LeaderboardSettingsUpdateRequest,
request: Request,
) -> LeaderboardSettingsResponse:
"""Update global leaderboard aggregation settings. Admin only."""
await _require_admin(request)
current = load_system_settings()
fields = body.model_dump(exclude_unset=True)
update: dict[str, object] = {}
if "exclude_user_ids" in fields and fields["exclude_user_ids"] is not None:
update["exclude_user_ids"] = list(dict.fromkeys([str(uid).strip() for uid in fields["exclude_user_ids"] if str(uid).strip()]))
for key in ("include_scheduled", "include_admins", "include_failed", "cleanup_system_questions"):
if key in fields and fields[key] is not None:
update[key] = bool(fields[key])
leaderboard = current.leaderboard.model_copy(update=update)
saved = save_system_settings(current.model_copy(update={"leaderboard": leaderboard}))
return _leaderboard_to_response(saved.leaderboard)
@router.get("/citation-display", response_model=CitationDisplayResponse)
async def get_citation_display() -> CitationDisplayResponse:
"""Return global citation display cleanup settings."""
@@ -167,17 +223,28 @@ class AppearanceResponse(BaseModel):
"""Effective system-wide appearance defaults."""
default_theme: Literal["light", "dark"] = Field(..., description="System default theme — light or dark")
default_skill_view: Literal["card", "list"] = Field(..., description="System default skill list display mode — card or list")
default_agent_view: Literal["card", "list"] = Field(..., description="System default agent list display mode — card or list")
class AppearanceUpdateRequest(BaseModel):
default_theme: Literal["light", "dark"] | None = Field(default=None)
default_skill_view: Literal["card", "list"] | None = Field(default=None)
default_agent_view: Literal["card", "list"] | None = Field(default=None)
def _appearance_response(appearance) -> AppearanceResponse:
return AppearanceResponse(
default_theme=appearance.default_theme,
default_skill_view=appearance.default_skill_view,
default_agent_view=appearance.default_agent_view,
)
@router.get("/appearance", response_model=AppearanceResponse)
async def get_appearance_settings() -> AppearanceResponse:
"""Return the global default theme. Any logged-in user may read this."""
settings = load_system_settings().appearance
return AppearanceResponse(default_theme=settings.default_theme)
"""Return the global appearance defaults. Any logged-in user may read this."""
return _appearance_response(load_system_settings().appearance)
@router.put("/appearance", response_model=AppearanceResponse)
@@ -190,8 +257,9 @@ async def update_appearance_settings(
current = load_system_settings()
fields = body.model_dump(exclude_unset=True)
update: dict[str, object] = {}
if "default_theme" in fields and fields["default_theme"] is not None:
update["default_theme"] = fields["default_theme"]
for key in ("default_theme", "default_skill_view", "default_agent_view"):
if key in fields and fields[key] is not None:
update[key] = fields[key]
if update:
updated_appearance = current.appearance.model_copy(update=update)
saved = save_system_settings(
@@ -199,4 +267,24 @@ async def update_appearance_settings(
)
else:
saved = current
return AppearanceResponse(default_theme=saved.appearance.default_theme)
return _appearance_response(saved.appearance)
class LogSystemResponse(BaseModel):
"""External log-system entry shown in the admin settings menu.
Sourced from ``config.yaml`` (``log_system`` section), so it is read-only
here — there is no PUT counterpart.
"""
enabled: bool = Field(..., description="Whether the entry is shown in the settings menu")
url: str = Field(..., description="External log-system URL opened in a new tab")
label: str = Field(..., description="Menu label shown for the entry")
@router.get("/log-system", response_model=LogSystemResponse)
async def get_log_system(request: Request) -> LogSystemResponse:
"""Return the admin-configured external log-system link from config.yaml. Admin only."""
await _require_admin(request)
cfg = get_app_config().log_system
return LogSystemResponse(enabled=cfg.enabled, url=cfg.url, label=cfg.label)
offline-backend-20260512/backend/app/gateway/routers/tags.py
+5 -25
View File
@@ -1,9 +1,9 @@
"""Tag management API.
Tags are split into three types (``agent`` / ``skill`` / ``scheduled_task``).
A tag may only be applied to a resource of its own type. Tag CRUD and tag
assignment are admin-only; listing tags and reading a resource's tags is open
to everyone (the frontend needs them to render chips and search filters).
A tag may only be applied to a resource of its own type. Tag CRUD, listing,
and assignment are available to ordinary users so teams can organize resources
without asking an administrator to tag every item.
"""
from __future__ import annotations
@@ -11,10 +11,10 @@ from __future__ import annotations
import logging
import uuid
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi import APIRouter, Depends, HTTPException, Query
from pydantic import BaseModel, Field
from app.gateway.deps import get_optional_user_from_request, get_tag_store
from app.gateway.deps import get_tag_store
from deerflow.persistence.tags import TAG_TYPES, TagStore
logger = logging.getLogger(__name__)
@@ -22,18 +22,6 @@ logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/tags", tags=["tags"])
# --------------------------------------------------------------------------- auth
async def _require_admin(request: Request) -> None:
"""Reject non-admins. Lets the call through when auth is disabled."""
user = await get_optional_user_from_request(request)
if user is None:
return
if getattr(user, "system_role", None) != "admin":
raise HTTPException(status_code=403, detail="Tag management is admin-only")
def _validate_tag_type(tag_type: str) -> str:
if tag_type not in TAG_TYPES:
raise HTTPException(status_code=422, detail=f"Invalid tag type '{tag_type}'. Allowed: {', '.join(TAG_TYPES)}")
@@ -96,10 +84,8 @@ async def list_tags(
@router.post("", response_model=TagResponse, status_code=201, summary="Create Tag")
async def create_tag(
body: TagCreateRequest,
request: Request,
store: TagStore = Depends(get_tag_store),
) -> TagResponse:
await _require_admin(request)
_validate_tag_type(body.tag_type)
try:
record = await store.create_tag(
@@ -125,10 +111,8 @@ async def create_tag(
async def update_tag(
tag_id: str,
body: TagUpdateRequest,
request: Request,
store: TagStore = Depends(get_tag_store),
) -> TagResponse:
await _require_admin(request)
update_data = body.model_dump(exclude_unset=True)
if "name" in update_data and update_data["name"] is not None:
update_data["name"] = update_data["name"].strip()
@@ -149,10 +133,8 @@ async def update_tag(
@router.delete("/{tag_id}", status_code=204, summary="Delete Tag")
async def delete_tag(
tag_id: str,
request: Request,
store: TagStore = Depends(get_tag_store),
) -> None:
await _require_admin(request)
try:
deleted = await store.delete_tag(tag_id)
if not deleted:
@@ -195,10 +177,8 @@ async def set_resource_tags(
target_type: str,
target_id: str,
body: TagAssignmentRequest,
request: Request,
store: TagStore = Depends(get_tag_store),
) -> TagListResponse:
await _require_admin(request)
_validate_tag_type(target_type)
try:
records = await store.set_target_tags(target_type, target_id, body.tag_ids)
offline-backend-20260512/backend/app/gateway/routers/threads.py
+806 -584
View File
File diff suppressed because it is too large. Load diff
offline-backend-20260512/backend/app/gateway/routers/user_preferences.py
+66
View File
@@ -0,0 +1,66 @@
"""Per-user UI preferences (account-bound, follow the user across devices).
Currently exposes a single toggle: ``hide_admin_prompt_prefix`` — when true the
chat input hides the admin-configured prompt-prefix switch so the user only uses
their own prompts. Kept as a small key-value (JSON) store so new toggles do not
each need a schema change.
"""
from __future__ import annotations
import logging
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel, Field
from deerflow.runtime.user_context import get_effective_user_id
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/user-preferences", tags=["user-preferences"])
class UserPreferencesResponse(BaseModel):
hide_admin_prompt_prefix: bool = False
class UserPreferencesUpdateRequest(BaseModel):
hide_admin_prompt_prefix: bool | None = Field(default=None)
def _current_user_id(request: Request) -> str:
user = getattr(request.state, "user", None)
if user is not None:
return str(user.id)
return get_effective_user_id()
def _get_store(request: Request):
store = getattr(request.app.state, "user_preferences_store", None)
if store is None:
raise HTTPException(status_code=503, detail="User preferences store not available")
return store
def _to_response(prefs: dict) -> UserPreferencesResponse:
return UserPreferencesResponse(
hide_admin_prompt_prefix=bool(prefs.get("hide_admin_prompt_prefix", False)),
)
@router.get("", response_model=UserPreferencesResponse)
async def get_user_preferences(request: Request) -> UserPreferencesResponse:
store = _get_store(request)
user_id = _current_user_id(request)
prefs = await store.get_for_user(user_id)
return _to_response(prefs)
@router.put("", response_model=UserPreferencesResponse)
async def update_user_preferences(request: Request, body: UserPreferencesUpdateRequest) -> UserPreferencesResponse:
store = _get_store(request)
user_id = _current_user_id(request)
patch = body.model_dump(exclude_unset=True, exclude_none=True)
if not patch:
return _to_response(await store.get_for_user(user_id))
prefs = await store.merge_for_user(user_id, patch)
return _to_response(prefs)
offline-backend-20260512/backend/app/gateway/routers/user_prompts.py
+12
View File
@@ -24,6 +24,8 @@ class UserPromptResponse(BaseModel):
title: str = ""
content: str
sort_order: int = 0
# Optional single-agent association. NULL = global (shown for every agent).
agent_id: str | None = None
created_at: datetime | str | None = None
updated_at: datetime | str | None = None
@@ -36,12 +38,16 @@ class UserPromptCreateRequest(BaseModel):
title: str = Field(default="", max_length=255)
content: str = Field(..., min_length=1)
sort_order: int = Field(default=0)
agent_id: str | None = Field(default=None, max_length=64)
class UserPromptUpdateRequest(BaseModel):
title: str | None = Field(default=None, max_length=255)
content: str | None = Field(default=None, min_length=1)
sort_order: int | None = None
# Sent on every save; an empty string / null clears the association. Use a
# sentinel so "field omitted" (don't touch) differs from "set to null".
agent_id: str | None = Field(default=None, max_length=64)
def _current_user_id(request: Request) -> str:
@@ -77,6 +83,7 @@ async def create_user_prompt(request: Request, body: UserPromptCreateRequest) ->
"title": (body.title or "").strip(),
"content": body.content,
"sort_order": body.sort_order,
"agent_id": (body.agent_id or "").strip() or None,
},
)
return UserPromptResponse(**row)
@@ -93,6 +100,11 @@ async def update_user_prompt(request: Request, prompt_id: str, body: UserPromptU
data["content"] = body.content
if body.sort_order is not None:
data["sort_order"] = body.sort_order
# ``agent_id`` is nullable: only act when the caller actually sent the field
# (``model_fields_set``), so an explicit null clears the association while an
# omitted field leaves it untouched.
if "agent_id" in body.model_fields_set:
data["agent_id"] = (body.agent_id or "").strip() or None
row = await store.update(prompt_id, user_id, data)
if row is None:
raise HTTPException(status_code=404, detail="Prompt not found")
offline-backend-20260512/backend/app/gateway/services.py
+5
View File
@@ -224,6 +224,11 @@ _CONTEXT_CONFIGURABLE_KEYS: frozenset[str] = frozenset(
"excluded_tools",
"skill_stop_names",
"skill_stop_message",
# 记忆开关:必须在白名单里才会从 body.context 透传到 run config,
# 否则 agent 端 cfg.get(...) 永远取默认值。圆桌等临时任务靠这两个键
# 关掉个人记忆召回 / builtin 记忆注入(见 multi_agent._run_payload)。
"memory_recall_disabled",
"memory_injection_enabled",
}
)
offline-backend-20260512/backend/config.yaml
+36 -4
View File
@@ -14,7 +14,7 @@ models:
display_name: DeepSeek Chat
use: langchain_openai:ChatOpenAI
model: deepseek-chat
api_key: sk-1520028668cb4cc388ed8315ff1e856e
api_key: sk-e1c22b7931dc446f8b67e57878edd105
base_url: https://api.deepseek.com
request_timeout: 600.0
max_retries: 2
@@ -22,7 +22,7 @@ models:
temperature: 1.0
supports_vision: false
supports_thinking: true
- name: zai-org/GLM-5-FP8
display_name: zai-org/GLM-5-FP8
use: langchain_openai:ChatOpenAI
@@ -37,7 +37,6 @@ models:
supports_thinking: true
- name: deepseek-v4-flash
display_name: DeepSeek V4 Flash
use: langchain_openai:ChatOpenAI
@@ -93,6 +92,28 @@ models:
temperature: 1.0
supports_vision: true
supports_thinking: true
- name: zai-org/GLM-5-FP8
display_name: zai-org/GLM-5-FP8
use: langchain_openai:ChatOpenAI
model: zai-org/GLM-5-FP8
api_key: modalresearch_gWu46FDpVLYZWPVEA3b_2fq8Jw2UJiKMgEm_Rxmn1SA
base_url: https://api.us-west-2.modal.direct/v1
request_timeout: 600.0
max_retries: 2
max_tokens: 4096
temperature: 1.0
supports_vision: true
supports_thinking: true
- name: Qwen/Qwen3.6-27B
display_name: Qwen/Qwen3.6-27B
use: langchain_openai:ChatOpenAI
model: Qwen/Qwen3.6-27B
api_key: sk-ighylafootabtuthwueirmoycccyqtdgbifyhozduvolkdyt
base_url: https://api.siliconflow.cn/v1/
max_tokens: 4096
temperature: 1.0
supports_vision: true
supports_thinking: true
# 识图(图片识别)配置。
# - model_name:指定一个 models[] 中的模型名,作为“默认识图大模型”。view_image 工具会把
# 图片交给该模型识别,并把识别出的文字结果返回给主对话模型 —— 这样即使主对话模型不支持
@@ -222,10 +243,16 @@ auth_login:
verify_ssl: false
ca_cert_path: "" # 可选:PEM 格式 CA 包路径,设置后优先生效并启用校验
# 日志系统入口(仅管理员可见):右下角「设置」菜单中会多出一个「日志系统」入口,
# 点击后在新标签页打开下面的 url。enabled 关掉则不显示该入口。
database:
backend: sqlite
sqlite_dir: .deer-flow/data
pool_size: 10
pool_size: 20 # 基础连接池(postgres/mysql 生效;sqlite 忽略)
max_overflow: 20 # 高峰期可临时超出 pool_size 的额外连接数,真实上限 = pool_size + max_overflow
pool_timeout: 30 # 池满时等待空闲连接的秒数,超时报错
echo_sql: false
checkpointer:
@@ -307,3 +334,8 @@ ai_writing:
keyword_model: deepseek-chat # 检索词生成专用模型，留空回退写作主模型
rank_model: deepseek-chat # 素材排序专用模型，留空回退写作主模型
max_materials: 20 # 合并去重后的素材总量上限，范围 [1, 100]，默认 20
log_system:
enabled: true
url: "https://your-log-system.example.com"
label: "日志系统"
offline-backend-20260512/backend/packages/harness/deerflow/agents/middlewares/uploads_middleware.py
+87 -23
View File
@@ -1,8 +1,10 @@
"""Middleware to inject uploaded files information into agent context."""
import logging
from pathlib import Path
from typing import NotRequired, override
import logging
import shutil
from pathlib import Path
from typing import NotRequired, override
from urllib.parse import unquote
from langchain.agents import AgentState
from langchain.agents.middleware import AgentMiddleware
@@ -13,7 +15,7 @@ from deerflow.config.paths import Paths, get_paths
from deerflow.runtime.user_context import get_effective_user_id
from deerflow.utils.file_conversion import extract_outline
logger = logging.getLogger(__name__)
logger = logging.getLogger(__name__)
_OUTLINE_PREVIEW_LINES = 5
@@ -22,7 +24,8 @@ _OUTLINE_PREVIEW_LINES = 5
# configured vision model), never by reading their bytes. Listing them with the
# generic read_file/grep guidance made the agent try to parse image binaries
# with bash/python, which fails and wastes turns.
_IMAGE_EXTENSIONS = {".jpg", ".jpeg", ".png", ".webp", ".gif", ".bmp"}
_IMAGE_EXTENSIONS = {".jpg", ".jpeg", ".png", ".webp", ".gif", ".bmp"}
_ARTIFACT_URL_MARKER = "/artifacts/"
def _extract_outline_for_file(file_path: Path) -> tuple[list[dict], list[str]]:
@@ -69,7 +72,7 @@ class UploadsMiddlewareState(AgentState):
uploaded_files: NotRequired[list[dict] | None]
class UploadsMiddleware(AgentMiddleware[UploadsMiddlewareState]):
class UploadsMiddleware(AgentMiddleware[UploadsMiddlewareState]):
"""Middleware to inject uploaded files information into the agent context.
Reads file metadata from the current message's additional_kwargs.files
@@ -122,7 +125,7 @@ class UploadsMiddleware(AgentMiddleware[UploadsMiddlewareState]):
lines.append(" Use `grep` to search for keywords (e.g. `grep(pattern='keyword', path='/mnt/user-data/uploads/')`).")
lines.append("")
def _create_files_message(self, new_files: list[dict], historical_files: list[dict]) -> str:
def _create_files_message(self, new_files: list[dict], historical_files: list[dict]) -> str:
"""Create a formatted message listing uploaded files.
Args:
@@ -160,9 +163,61 @@ class UploadsMiddleware(AgentMiddleware[UploadsMiddlewareState]):
lines.append("- Only fall back to web search if the file content is clearly insufficient to answer the question.")
lines.append("</uploaded_files>")
return "\n".join(lines)
def _files_from_kwargs(self, message: HumanMessage, uploads_dir: Path | None = None) -> list[dict] | None:
return "\n".join(lines)
def _import_file_from_artifact_url(
self,
file_info: dict,
uploads_dir: Path | None,
) -> bool:
"""Copy a previously uploaded file into the current thread when possible.
iframe-style callers may pass file metadata that points at an upload
from another DeerFlow thread. The current run thread still needs a real
file under its own ``/mnt/user-data/uploads`` directory, otherwise this
middleware will filter the entry out during existence checks.
"""
filename = file_info.get("filename") or ""
artifact_url = str(file_info.get("artifact_url") or "").strip()
if not filename or uploads_dir is None or not artifact_url:
return False
if not artifact_url.startswith("/api/threads/") or _ARTIFACT_URL_MARKER not in artifact_url:
return False
prefix, encoded_virtual_path = artifact_url.split(_ARTIFACT_URL_MARKER, 1)
source_thread_id = prefix.removeprefix("/api/threads/").strip("/")
if not source_thread_id or not encoded_virtual_path:
return False
virtual_path = "/" + unquote(encoded_virtual_path.lstrip("/"))
if Path(virtual_path).name != filename:
return False
try:
source_path = self._paths.resolve_virtual_path(
source_thread_id,
virtual_path,
user_id=get_effective_user_id(),
)
except ValueError:
logger.debug("Failed to resolve uploaded artifact path: %s", artifact_url, exc_info=True)
return False
if not source_path.is_file():
return False
uploads_dir.mkdir(parents=True, exist_ok=True)
target_path = uploads_dir / filename
shutil.copy2(source_path, target_path)
source_markdown = source_path.with_suffix(".md")
if source_markdown.is_file() and source_markdown != source_path:
shutil.copy2(source_markdown, target_path.with_suffix(".md"))
return True
def _files_from_kwargs(self, message: HumanMessage, uploads_dir: Path | None = None) -> list[dict] | None:
"""Extract file info from message additional_kwargs.files.
The frontend sends uploaded file metadata in additional_kwargs.files
@@ -183,16 +238,17 @@ class UploadsMiddleware(AgentMiddleware[UploadsMiddlewareState]):
files = []
for f in kwargs_files:
if not isinstance(f, dict):
continue
filename = f.get("filename") or ""
if not filename or Path(filename).name != filename:
continue
if uploads_dir is not None and not (uploads_dir / filename).is_file():
continue
files.append(
{
"filename": filename,
if not isinstance(f, dict):
continue
filename = f.get("filename") or ""
if not filename or Path(filename).name != filename:
continue
if uploads_dir is not None and not (uploads_dir / filename).is_file():
if not self._import_file_from_artifact_url(f, uploads_dir):
continue
files.append(
{
"filename": filename,
"size": int(f.get("size") or 0),
"path": f"/mnt/user-data/uploads/{filename}",
"extension": Path(filename).suffix,
@@ -243,6 +299,17 @@ class UploadsMiddleware(AgentMiddleware[UploadsMiddlewareState]):
# Get newly uploaded files from the current message's additional_kwargs.files
new_files = self._files_from_kwargs(last_message, uploads_dir) or []
# Only inject when the *current* message actually carries new uploads.
# Turns without a fresh upload must not re-inject previously uploaded
# files: the original upload turn already injected (and persisted) their
# <uploaded_files> block, and the files remain physically available for
# read_file/grep/view_image. Re-injecting on every later message made the
# agent re-reference old images/docs, and — since the frontend parses
# files out of the <uploaded_files> content when additional_kwargs.files
# is empty — re-rendered the old attachment on each new question.
if not new_files:
return None
# Collect historical files from the uploads directory (all except the new ones)
new_filenames = {f["filename"] for f in new_files}
historical_files: list[dict] = []
@@ -270,9 +337,6 @@ class UploadsMiddleware(AgentMiddleware[UploadsMiddlewareState]):
file["outline"] = outline
file["outline_preview"] = preview
if not new_files and not historical_files:
return None
logger.debug(f"New files: {[f['filename'] for f in new_files]}, historical: {[f['filename'] for f in historical_files]}")
# Create files message and prepend to the last human message content
offline-backend-20260512/backend/packages/harness/deerflow/agents/roundtable_orchestrator/engine.py
+123 -30
View File
@@ -47,6 +47,7 @@ class OrchestrationResult:
pending_clarification: str | None = None
error: str | None = None
last_leader_content: str = ""
dialogues: list = field(default_factory=list)
@dataclass
@@ -71,6 +72,7 @@ async def run_orchestration(
seed_message: str,
mode: str = "recommend",
intent_text: str = "",
enable_report: bool = True,
) -> OrchestrationResult:
"""运行一次完整编排（循环 → 共识 → 报告绘制）。
@@ -84,13 +86,17 @@ async def run_orchestration(
seed_message: 第一轮 leader 的 prompt（启动=初始意图；resume=用户答案续轮）。
mode: "recommend" | "chain"。
intent_text: 任务意图，用于报告 prompt（缺省退回 seed_message）。
enable_report: 是否在收口后执行 Step3 报告绘制。False 时只跑完 Step2 研讨即收尾、
不生成报告（step3=None）。当前「后台挂起」临时禁用报告（进程内驱动报告 agent
在真模型下仍无法稳定写出 HTML，详见 multi-agent-background-run-遗留问题与下次继续.md）。
"""
intent = (intent_text or seed_message or "").strip()
try:
await progress.begin()
if mode == "chain":
outcome = await _run_chain(gateway, progress, agents, thread_ids, coordinator_name, model, seed_message)
outcome = await _run_chain(gateway, progress, agents, thread_ids, coordinator_name, model, seed_message, intent)
else:
outcome = await _run_recommend(gateway, progress, agents, thread_ids, coordinator_name, model, seed_message)
outcome = await _run_recommend(gateway, progress, agents, thread_ids, coordinator_name, model, seed_message, intent)
if outcome.kind == "clarification":
question = outcome.question or "总控需要补充信息，请补充后继续。"
@@ -99,33 +105,74 @@ async def run_orchestration(
status="awaiting_input",
pending_clarification=question,
last_leader_content=outcome.last_leader_content,
dialogues=progress.dialogues(),
)
# consensus 或 max_cycles：均收口 → 报告绘制
# consensus 或 max_cycles：均收口。报告绘制按 enable_report 开关执行。
await progress.consensus(100)
step3 = await _run_report(
gateway,
progress,
agents=agents,
model=model,
intent_text=intent_text or seed_message,
consensus_text=outcome.last_leader_content,
deliveries=outcome.deliveries,
)
step3 = None
if enable_report:
step3 = await _run_report(
gateway,
progress,
agents=agents,
model=model,
intent_text=intent_text or seed_message,
consensus_text=outcome.last_leader_content,
deliveries=outcome.deliveries,
)
await progress.done()
return OrchestrationResult(
status="done",
consensus=True,
step3=step3,
last_leader_content=outcome.last_leader_content,
dialogues=progress.dialogues(),
)
except Exception as exc: # noqa: BLE001 — 顶层兜底：标记 error 并返回
logger.exception("roundtable orchestration failed")
await progress.error(str(exc))
return OrchestrationResult(status="error", error=str(exc))
return OrchestrationResult(status="error", error=str(exc), dialogues=progress.dialogues())
# ── recommend 模式 ───────────────────────────────────────────────────────────
# ── 上下文「广播」（inline 实现）────────────────────────────────────────────────
#
# 进程内网关没有 multi_agent 的 thread 广播，席位/总控会看不到「原始任务背景 + 各席位
# 前序交付」→ 席位反问用户、总控靠记忆瞎编。这里把这些信息**直接拼进 prompt**：
# - 每个席位的任务带上：任务背景 + 前序各席位交付 + 「直接产出、别提问」硬约束；
# - 总控的续轮 prompt 带上：各席位本轮交付，让它据实派活/收口。
_SEAT_NO_ASK = (
"注意：这是自动化圆桌研讨，没有用户在线应答——**请勿向用户提问或等待确认**，"
"直接基于已知信息做出合理假设，产出完整的结构化交付。\n"
"**交付方式（与前端手动研讨一致）**：请用 write_file 工具把你这一轮的完整交付整理成一份 "
"Markdown 文件，写入 /mnt/user-data/outputs/ 下（文件名体现你的职责，如 `情报收集.md`、"
"`风险评估.md`）；同时在回复正文里给出这份交付的内容本身（供总控汇总），"
"不要只回复『已写入文件』这类空话。"
)
def _deliveries_block(deliveries: dict[str, str], name_by_id: dict[str, str]) -> str:
parts: list[str] = []
for aid, text in deliveries.items():
t = (text or "").strip()
if not t:
continue
parts.append(f"### {name_by_id.get(aid, aid)}\n{t}")
return "\n\n".join(parts)
def _seat_message(intent_text: str, deliveries: dict[str, str], name_by_id: dict[str, str], task: str) -> str:
"""把任务背景 + 前序交付 + 防提问约束拼进席位任务（替代 thread 广播）。"""
parts: list[str] = []
if intent_text.strip():
parts.append(f"【本次研讨任务背景】\n{intent_text.strip()}")
block = _deliveries_block(deliveries, name_by_id)
if block:
parts.append("【前序各席位已完成的交付，请在此基础上承接、补充、不要重复或推翻】\n" + block)
parts.append(f"【你的本轮具体任务】\n{task}")
parts.append(_SEAT_NO_ASK)
return "\n\n---\n\n".join(parts)
async def _run_recommend(
@@ -136,10 +183,12 @@ async def _run_recommend(
coordinator_name: str,
model: str | None,
seed_message: str,
intent_text: str,
) -> _LoopOutcome:
deliveries: dict[str, str] = {}
last_leader = ""
message = seed_message
name_by_id = {a.agent_id: a.name for a in agents}
for cycle in range(1, MAX_CYCLES + 1):
await progress.leader_thinking(cycle)
@@ -151,22 +200,32 @@ async def _run_recommend(
)
if turn.final_text:
last_leader = turn.final_text
await progress.add_leader_dialogue(
turn.final_text, cycle, tool_calls=turn.tool_calls, thread_id=thread_ids.get(coordinator_name)
)
if turn.is_clarification:
return _LoopOutcome("clarification", question=turn.clarification, last_leader_content=last_leader, deliveries=deliveries)
if turn.is_consensus:
return _LoopOutcome("consensus", last_leader_content=last_leader, deliveries=deliveries)
# **顺序**派活 —— 不要并行
# **顺序**派活 —— 不要并行；每个席位带上「背景 + 前序交付」上下文
await progress.dispatching(cycle)
for agent_id, task in turn.dispatched:
await progress.seat_active(agent_id)
seat = await gateway.run_seat(thread_ids=thread_ids, agent_id=agent_id, task=task, model=model)
seat_msg = _seat_message(intent_text, deliveries, name_by_id, task)
seat = await gateway.run_seat(thread_ids=thread_ids, agent_id=agent_id, task=seat_msg, model=model)
deliveries[agent_id] = seat.final_text
await progress.add_seat_dialogue(
agent_id, name_by_id.get(agent_id, agent_id), seat.final_text, cycle,
tool_calls=seat.tool_calls, thread_id=thread_ids.get(agent_id),
)
await progress.seat_done(agent_id, cycle)
await progress.bump_consensus()
message = CONTINUE_PROMPT
# 续轮：把各席位本轮交付喂给总控，让它据实判断（总控 SOUL 依赖看到交付）
block = _deliveries_block(deliveries, name_by_id)
message = CONTINUE_PROMPT + (f"\n\n各席位最新交付如下：\n{block}" if block else "")
# 达到最大轮次：强制收口（对齐前端 toast「已达到最大研讨轮次」）
logger.info("roundtable recommend loop hit MAX_CYCLES=%d, forcing consensus", MAX_CYCLES)
@@ -184,13 +243,15 @@ async def _run_chain(
coordinator_name: str,
model: str | None,
seed_message: str,
intent_text: str,
) -> _LoopOutcome:
deliveries: dict[str, str] = {}
last_leader = ""
total = len(agents)
name_by_id = {a.agent_id: a.name for a in agents}
for i, seat in enumerate(agents):
# (a) 总控按链条顺序只给当前席位派活
# (a) 总控按链条顺序只给当前席位派活；带上前序各席位交付，让它据实派活
await progress.leader_thinking(i + 1)
leader_task = build_chain_leader_task(
seed_message=seed_message,
@@ -200,6 +261,9 @@ async def _run_chain(
seat_name=seat.name,
is_first=(i == 0),
)
block = _deliveries_block(deliveries, name_by_id)
if block:
leader_task += f"\n\n前序各席位已完成的交付如下，请据此为本席位分配承接性任务：\n{block}"
turn = await gateway.run_leader(
thread_ids=thread_ids,
coordinator_name=coordinator_name,
@@ -208,31 +272,44 @@ async def _run_chain(
)
if turn.final_text:
last_leader = turn.final_text
await progress.add_leader_dialogue(
turn.final_text, i + 1, tool_calls=turn.tool_calls, thread_id=thread_ids.get(coordinator_name)
)
if turn.is_clarification:
return _LoopOutcome("clarification", question=turn.clarification, last_leader_content=last_leader, deliveries=deliveries)
task = pick_chain_task(turn.dispatched, turn.final_text, seat.agent_id, seat.name)
# (b) 当前席位交付（顺序，不并行）
# (b) 当前席位交付（顺序，不并行）；带上「背景 + 前序交付 + 别提问」上下文
await progress.seat_active(seat.agent_id)
seat_turn = await gateway.run_seat(thread_ids=thread_ids, agent_id=seat.agent_id, task=task, model=model)
seat_msg = _seat_message(intent_text, deliveries, name_by_id, task)
seat_turn = await gateway.run_seat(thread_ids=thread_ids, agent_id=seat.agent_id, task=seat_msg, model=model)
deliveries[seat.agent_id] = seat_turn.final_text
await progress.add_seat_dialogue(
seat.agent_id, seat.name, seat_turn.final_text, i + 1,
tool_calls=seat_turn.tool_calls, thread_id=thread_ids.get(seat.agent_id),
)
await progress.seat_done(seat.agent_id, i + 1)
if total > 0:
await progress.set_consensus_pct(round((i + 1) / total * 90))
# (c) 总控收口：综合全链交付，产出最终结论
# (c) 总控收口：综合全链交付，产出最终结论（带上全部席位交付）
if total > 0:
await progress.leader_thinking(total + 1)
block = _deliveries_block(deliveries, name_by_id)
closeout_msg = CHAIN_CLOSEOUT_PROMPT + (f"\n\n各席位完整交付如下：\n{block}" if block else "")
closeout = await gateway.run_leader(
thread_ids=thread_ids,
coordinator_name=coordinator_name,
message=CHAIN_CLOSEOUT_PROMPT,
message=closeout_msg,
model=model,
)
if closeout.final_text:
last_leader = closeout.final_text
await progress.add_leader_dialogue(
closeout.final_text, total + 1, tool_calls=closeout.tool_calls, thread_id=thread_ids.get(coordinator_name)
)
return _LoopOutcome("consensus", last_leader_content=last_leader, deliveries=deliveries)
@@ -273,27 +350,43 @@ def build_report_prompt(
席位交付按 ``agents`` 顺序列出（缺交付的席位跳过）。报告智能体据此用 write_file
写出自包含的 ``outputs/方案总览.html``（具体绘制规则在 roundtable-report 的 SOUL）。
"""
# 与前端 useStep3Report.buildReportPrompt 对齐 —— 关键是**明确叫报告 agent 用
# write_file 写到 /mnt/user-data/outputs/方案总览.html**，否则它只会在对话里回文字、
# 不落 HTML 文件（_read_report_html 就读不到）。
name_by_id = {a.agent_id: a.name for a in agents}
lines: list[str] = [
"【圆桌最终成果汇总 —— 请据此绘制方案总览 HTML 看板】",
"请基于以下圆桌多智能体研讨的成果，撰写一份图文并茂、以文字叙述为主线的 HTML「方案研讨汇报」，"
"用 write_file 工具写入 /mnt/user-data/outputs/方案总览.html。",
"目标：让用户能轻松读懂这群智能体讨论出了什么方案、为什么是它。请逐个席位提炼其核心观点与交付写成文字，"
"再综合成最终方案；图表用来点睛（风险矩阵、时间线、资源占比、贡献雷达等），详略得当——"
"缺明确数字时也要据研讨内容合理估算出图并标注「定性/估算」，不要通篇只有文字。",
"",
"## 任务意图",
"# 一、任务意图",
(intent_text or "").strip() or "（未提供）",
"",
"## 各席位交付",
"# 二、各席位最终交付",
]
name_by_id = {a.agent_id: a.name for a in agents}
has_delivery = False
for a in agents:
for i, a in enumerate(agents):
text = (deliveries.get(a.agent_id) or "").strip()
if not text:
continue
has_delivery = True
lines.append(f"### {name_by_id.get(a.agent_id, a.agent_id)}")
lines.append(f"## {i + 1}. {name_by_id.get(a.agent_id, a.agent_id)}")
lines.append(text)
lines.append("")
if not has_delivery:
lines.append("（各席位暂无可用交付，请基于任务意图与总控结论绘制。）")
lines.append("（各席位暂无文字交付，请基于任务意图与总控结论合理展开成方案。）")
lines.append("")
lines.append("## 总控最终结论")
lines.append("# 三、总控最终共识结论")
lines.append((consensus_text or "").strip() or "（未提供）")
lines.append("")
lines.append(
"请严格遵循你的职责设定：自包含离线（纯 inline SVG 图表、禁止任何外链 CDN）、研讨叙事型结构"
"（结论+KPI → 背景目标 → 各席位观点 → 综合分析 → 最终共识方案 → 执行路线）、"
"文字为主图表点睛、浅红主题（主色 #c25555）；写完文件后只用一两句话总结，不要把 HTML 源码贴进对话。"
)
lines.append(
'硬性要求：① 全程使用简体中文；② HTML 必须是完整文档，<head> 最前面声明 <meta charset="utf-8">。'
)
return "\n".join(lines)
offline-backend-20260512/backend/packages/harness/deerflow/agents/roundtable_orchestrator/progress.py
+92
View File
@@ -10,12 +10,21 @@
from __future__ import annotations
import logging
from datetime import datetime, timedelta, timezone
from typing import Any, Awaitable, Callable
from .types import SeatRef
logger = logging.getLogger(__name__)
# 北京时区（UTC+8，固定无 DST）——对话气泡时间戳用，对齐前端 nowTime()
# = new Date().toLocaleTimeString("zh-CN", {hour12:false}) 的 "HH:MM:SS" 形状。
_BEIJING_TZ = timezone(timedelta(hours=8))
def _now_hms() -> str:
return datetime.now(_BEIJING_TZ).strftime("%H:%M:%S")
# 节点状态机：pending → active → done。
_PENDING = "pending"
_ACTIVE = "active"
@@ -42,14 +51,21 @@ class JobProgressWriter:
job_id: str,
agents: list[SeatRef],
event_sink: EventSink = None,
on_dialogue: Any | None = None,
) -> None:
self._store = store
self._job_id = job_id
self._agents = list(agents)
self._event_sink = event_sink
# 每轮 leader/seat 模型返回后回调（dialogues, consensus）—— 执行器据此把对话
# 写进草稿（和手动跑同一存取路径）。
self._on_dialogue = on_dialogue
self._state: dict[str, str] = {a.agent_id: _PENDING for a in agents}
self._spoke_cycle: dict[str, int] = {}
self._consensus = 0
# 研讨对话记录（leader 说明 + 各席位交付），随每次落库一并写入。
self._dialogues: list[dict[str, Any]] = []
self._dlg_seq = 0
# ── 节点快照 ──────────────────────────────────────────────────────────
@@ -76,6 +92,7 @@ class JobProgressWriter:
"""落库（始终带最新 dispatch_chain + consensus）+ 外发事件，均 best-effort。"""
fields.setdefault("dispatch_chain", self._nodes())
fields.setdefault("consensus_percentage", self._consensus)
fields.setdefault("dialogues", self._dialogues)
if self._store is not None:
try:
await self._store.update_progress(self._job_id, **fields)
@@ -140,3 +157,78 @@ class JobProgressWriter:
async def error(self, message: str) -> None:
await self._persist(status="error", error=message)
# ── 对话记录 ─────────────────────────────────────────────────────────────
def _add(
self,
role: str,
agent_id: str | None,
name: str,
content: str,
cycle: int,
*,
tool_calls: list[dict] | None = None,
thread_id: str | None = None,
) -> None:
self._dlg_seq += 1
self._dialogues.append(
{
"id": self._dlg_seq,
"role": role,
"agentId": agent_id,
"name": name,
"content": content,
"cycle": cycle,
# 工具步骤卡数据源（每项 {name, args, id}）；映射成前端 Step2Dialogue.steps。
"toolCalls": list(tool_calls or []),
# 该气泡对应的 leader/seat 线程——映射成 agentThreadId，供点开产物 / 反查 stub。
"threadId": thread_id,
# 气泡时间戳，**创建时一次性盖章**（不随后续 flush 抖动），对齐前端 nowTime()。
"time": _now_hms(),
}
)
def dialogues(self) -> list[dict[str, Any]]:
return list(self._dialogues)
async def _flush_dialogue(self) -> None:
if self._on_dialogue is None:
return
try:
await self._on_dialogue(self.dialogues(), self._consensus)
except Exception: # noqa: BLE001 — 写草稿失败不打断编排
logger.warning("roundtable job dialogue → draft flush failed", exc_info=True)
async def add_leader_dialogue(
self,
content: str,
cycle: int,
*,
tool_calls: list[dict] | None = None,
thread_id: str | None = None,
) -> None:
"""总控一轮发言（派活说明 / 共识结论）。空内容跳过。
``tool_calls`` / ``thread_id`` 用于重建工具步骤卡（派活 chip）+ agentThreadId。
"""
if not (content or "").strip():
return
self._add("leader", None, "总控协调", content, cycle, tool_calls=tool_calls, thread_id=thread_id)
await self._persist()
await self._flush_dialogue()
async def add_seat_dialogue(
self,
agent_id: str,
name: str,
content: str,
cycle: int,
*,
tool_calls: list[dict] | None = None,
thread_id: str | None = None,
) -> None:
"""某席位一轮交付。``tool_calls`` / ``thread_id`` 用于步骤卡 + 产物预览。"""
self._add("seat", agent_id, name, content or "", cycle, tool_calls=tool_calls, thread_id=thread_id)
await self._persist()
await self._flush_dialogue()
offline-backend-20260512/backend/packages/harness/deerflow/agents/roundtable_orchestrator/types.py
+6
View File
@@ -52,6 +52,9 @@ class LeaderTurn:
dispatched: list[tuple[str, str]] = field(default_factory=list)
# 命中澄清时的问题文本；非 None 即 clarification 分支。
clarification: str | None = None
# 本轮总控调用的工具（按出现顺序，每项 {name, args, id}）——通常是 agent_orchestration
# 派活 / ask_clarification。用于在草稿 run 里重建「工具步骤卡」，与前端手动跑一致。
tool_calls: list[dict] = field(default_factory=list)
@property
def is_clarification(self) -> bool:
@@ -67,6 +70,9 @@ class SeatTurn:
"""某席位一轮交付的归一化结果（由 gateway.run_seat 产出）。"""
final_text: str = ""
# 本轮席位调用的工具（按出现顺序，每项 {name, args, id}）——用于在草稿 run 里
# 重建「工具步骤卡」+ write_file stub（点开产物），与前端手动跑一致。
tool_calls: list[dict] = field(default_factory=list)
@dataclass
offline-backend-20260512/backend/packages/harness/deerflow/config/agent_name_desensitize.py
+92
View File
@@ -0,0 +1,92 @@
"""Server-side mirror of the frontend agent-name desensitization map.
The web UI displays agent names through a static term→abbreviation map
(``frontend-web/src/lib/desensitize.ts``) — e.g. the stored name "态势分析" is
shown as "TS分析". The gallery search box therefore matches against the
*desensitized* display form, not the raw stored name.
When search moves server-side (paginated listing), the backend must match the
same way, so this module mirrors that static map verbatim. The frontend map is
never extended at runtime (``addDesensitizeEntries`` is defined but never
called), so a static copy stays in sync.
Used by the SQL listing query to match "raw name OR desensitized name".
"""
from __future__ import annotations
# Mirror of DESENSITIZE_MAP in frontend-web/src/lib/desensitize.ts.
# Keep entries identical to the frontend so server-side and client-side search
# behave the same. Order does not affect membership tests.
DESENSITIZE_MAP: dict[str, str] = {
"态势": "TS",
"你好": "nh",
"滨海作战体系": "第一张网",
"滨海作战部队": "第一张网",
"Taiwan Air Force Flight Units": "第二张网",
"台海空军飞行部队": "第二张网",
"空军": "KJ",
"海军": "HJ",
"陆军": "LJ",
"陆战队": "LZD",
"海巡": "HX",
"憲兵": "XB",
"資通電軍": "ZTDJ",
"情报": "QB",
"指挥": "ZH",
"信息准备": "XX准备",
"行为体": "XWT",
"驱动因素": "QDYS",
"叙事": "XS",
"前三定": "QSD",
"受众目标": "SZMB",
"作战": "ZZ",
"目的": "MD",
"关键传播节点": "GJCBJD",
"战法": "ZF",
"行动": "XD",
"打击": "DJ",
"目标": "MB",
"战力": "ZL",
"筹划": "COH",
"策划": "CEH",
"模拟": "MN",
"推演": "TY",
"认知": "RZ",
"触及": "CJ",
"军": "J",
"美军": "MJ",
"台军": "TJ",
}
def desensitize_name(name: str) -> str:
"""Return ``name`` with every mapped term replaced by its abbreviation.
Mirrors the frontend ``desensitize()`` — a sequence of literal replacements.
"""
out = name
for term, abbr in DESENSITIZE_MAP.items():
if term in out:
out = out.replace(term, abbr)
return out
def name_matches(name: str, needle: str) -> bool:
"""True when ``needle`` (case-insensitive) is in the raw OR desensitized name."""
n = needle.strip().lower()
if not n:
return True
return n in name.lower() or n in desensitize_name(name).lower()
def source_terms_for_needle(needle: str) -> list[str]:
"""Return the raw source terms whose abbreviation contains ``needle``.
Lets the SQL search add ``name LIKE %term%`` clauses so a user typing an
abbreviation (e.g. "TS") still matches the stored raw term (e.g. "态势").
"""
n = needle.strip().lower()
if not n:
return []
return [term for term, abbr in DESENSITIZE_MAP.items() if n in abbr.lower()]
offline-backend-20260512/backend/packages/harness/deerflow/config/app_config.py
+2
View File
@@ -15,6 +15,7 @@ from deerflow.config.database_config import DatabaseConfig
from deerflow.config.extensions_config import ExtensionsConfig
from deerflow.config.gateway_config import GatewayConfig
from deerflow.config.guardrails_config import GuardrailsConfig, load_guardrails_config_from_dict
from deerflow.config.log_system_config import LogSystemConfig
from deerflow.config.login_config import AuthLoginConfig
from deerflow.config.memory_config import MemoryConfig, load_memory_config_from_dict
from deerflow.config.model_config import ModelConfig
@@ -108,6 +109,7 @@ class AppConfig(BaseModel):
models: list[ModelConfig] = Field(default_factory=list, description="Available models")
vision: VisionConfig = Field(default_factory=VisionConfig, description="Image recognition (识图) configuration")
auth_login: AuthLoginConfig = Field(default_factory=AuthLoginConfig, description="Custom login modes (username password gate, token exchange)")
log_system: LogSystemConfig = Field(default_factory=LogSystemConfig, description="Admin-only external log-system entry shown in the settings menu")
sandbox: SandboxConfig = Field(description="Sandbox configuration")
tools: list[ToolConfig] = Field(default_factory=list, description="Available tools")
tool_groups: list[ToolGroupConfig] = Field(default_factory=list, description="Available tool groups")
offline-backend-20260512/backend/packages/harness/deerflow/config/checkpointer_config.py
+35 -7
View File
@@ -2,7 +2,7 @@
from typing import Literal
from pydantic import BaseModel, Field
from pydantic import BaseModel, Field
CheckpointerType = Literal["memory", "sqlite", "postgres"]
@@ -16,13 +16,41 @@ class CheckpointerConfig(BaseModel):
"'sqlite' persists to a local file (requires langgraph-checkpoint-sqlite). "
"'postgres' persists to PostgreSQL (requires langgraph-checkpoint-postgres)."
)
connection_string: str | None = Field(
default=None,
description="Connection string for sqlite (file path) or postgres (DSN). "
connection_string: str | None = Field(
default=None,
description="Connection string for sqlite (file path) or postgres (DSN). "
"Required for sqlite and postgres types. "
"For sqlite, use a file path like '.deer-flow/checkpoints.db' or ':memory:' for in-memory. "
"For postgres, use a DSN like 'postgresql://user:pass@localhost:5432/db'.",
)
"For sqlite, use a file path like '.deer-flow/checkpoints.db' or ':memory:' for in-memory. "
"For postgres, use a DSN like 'postgresql://user:pass@localhost:5432/db'.",
)
use_postgres: bool = Field(
default=True,
description=(
"When type='postgres', set false to keep the Postgres DSN in config "
"but run the checkpointer on SQLite instead."
),
)
sqlite_connection_string: str | None = Field(
default=None,
description=(
"SQLite file used when type='postgres' and use_postgres=false. "
"Defaults to the first dual_read_sqlite_paths entry or 'store.db'."
),
)
dual_read_enabled: bool = Field(
default=False,
description=(
"When type='postgres', read from Postgres first and fall back to legacy "
"SQLite checkpoint files when Postgres does not have the requested data."
),
)
dual_read_sqlite_paths: list[str] = Field(
default_factory=list,
description=(
"Legacy SQLite checkpoint database paths used for read fallback. "
"Plain paths are resolved the same way as SQLite connection_string."
),
)
# Global configuration instance — None means no checkpointer is configured.
offline-backend-20260512/backend/packages/harness/deerflow/config/database_config.py
+61 -47
View File
@@ -11,13 +11,13 @@ blocking, making a unified file safe for both workloads. Writers
that contend for the lock wait via the default 5-second sqlite3
busy timeout rather than failing immediately.
Postgres mode: both use the same database URL but maintain independent
connection pools with different lifecycles.
MySQL mode: application ORM data uses MySQL. LangGraph's checkpointer does
not have a built-in MySQL backend in this project, so deployments that choose
MySQL for business tables should configure the legacy ``checkpointer:``
section separately (usually sqlite) to preserve conversation checkpoints.
Postgres mode: both use the same database URL but maintain independent
connection pools with different lifecycles.
MySQL mode: application ORM data uses MySQL. LangGraph's checkpointer does
not have a built-in MySQL backend in this project, so deployments that choose
MySQL for business tables should configure the legacy ``checkpointer:``
section separately (usually sqlite) to preserve conversation checkpoints.
Memory mode: checkpointer uses MemorySaver, app uses in-memory stores.
No database is initialized.
@@ -26,12 +26,12 @@ Sensitive values (postgres_url) should use $VAR syntax in config.yaml
to reference environment variables from .env:
database:
backend: postgres
postgres_url: $DATABASE_URL
The $VAR resolution is handled by AppConfig.resolve_env_variables()
before this config is instantiated -- DatabaseConfig itself does not
need to do any environment variable processing.
backend: postgres
postgres_url: $DATABASE_URL
The $VAR resolution is handled by AppConfig.resolve_env_variables()
before this config is instantiated -- DatabaseConfig itself does not
need to do any environment variable processing.
"""
from __future__ import annotations
@@ -42,40 +42,54 @@ from typing import Literal
from pydantic import BaseModel, Field
class DatabaseConfig(BaseModel):
backend: Literal["memory", "sqlite", "postgres", "mysql"] = Field(
default="memory",
description=("Storage backend for application data. 'memory' for development (no persistence across restarts), 'sqlite' for single-node deployment, 'postgres' for production multi-node deployment, 'mysql' for MySQL-backed business tables."),
)
class DatabaseConfig(BaseModel):
backend: Literal["memory", "sqlite", "postgres", "mysql"] = Field(
default="memory",
description=("Storage backend for application data. 'memory' for development (no persistence across restarts), 'sqlite' for single-node deployment, 'postgres' for production multi-node deployment, 'mysql' for MySQL-backed business tables."),
)
sqlite_dir: str = Field(
default=".deer-flow/data",
description=("Directory for the SQLite database file. Both checkpointer and application data share {sqlite_dir}/deerflow.db."),
)
postgres_url: str = Field(
default="",
description=(
"PostgreSQL connection URL, shared by checkpointer and app. "
postgres_url: str = Field(
default="",
description=(
"PostgreSQL connection URL, shared by checkpointer and app. "
"Use $DATABASE_URL in config.yaml to reference .env. "
"Example: postgresql://user:pass@host:5432/deerflow "
"(the +asyncpg driver suffix is added automatically where needed)."
),
)
mysql_url: str = Field(
default="",
description=(
"MySQL connection URL for the application ORM. "
"Use $MYSQL_DATABASE_URL in config.yaml to reference .env. "
"Example: mysql+asyncmy://user:pass@host:3306/deerflow?charset=utf8mb4. "
"If mysql:// is provided, the +asyncmy driver suffix is added automatically."
),
)
),
)
mysql_url: str = Field(
default="",
description=(
"MySQL connection URL for the application ORM. "
"Use $MYSQL_DATABASE_URL in config.yaml to reference .env. "
"Example: mysql+asyncmy://user:pass@host:3306/deerflow?charset=utf8mb4. "
"If mysql:// is provided, the +asyncmy driver suffix is added automatically."
),
)
echo_sql: bool = Field(
default=False,
description="Echo all SQL statements to log (debug only).",
)
pool_size: int = Field(
default=5,
description="Connection pool size for the app ORM engine (postgres/mysql only).",
description="Connection pool size for the app ORM engine (postgres/mysql only).",
)
max_overflow: int = Field(
default=10,
description=(
"Extra connections allowed beyond pool_size under load (postgres/mysql only). "
"The real connection ceiling is pool_size + max_overflow. SQLAlchemy default is 10."
),
)
pool_timeout: int = Field(
default=30,
description=(
"Seconds a caller waits for a free pooled connection before erroring (postgres/mysql only). "
"SQLAlchemy default is 30."
),
)
# -- Derived helpers (not user-configured) --
@@ -108,16 +122,16 @@ class DatabaseConfig(BaseModel):
"""SQLAlchemy async URL for the application ORM engine."""
if self.backend == "sqlite":
return f"sqlite+aiosqlite:///{self.sqlite_path}"
if self.backend == "postgres":
url = self.postgres_url
if url.startswith("postgresql://"):
url = url.replace("postgresql://", "postgresql+asyncpg://", 1)
return url
if self.backend == "mysql":
url = self.mysql_url
if not url:
raise ValueError("database.mysql_url is required for backend='mysql'")
if url.startswith("mysql://"):
url = url.replace("mysql://", "mysql+asyncmy://", 1)
return url
raise ValueError(f"No SQLAlchemy URL for backend={self.backend!r}")
if self.backend == "postgres":
url = self.postgres_url
if url.startswith("postgresql://"):
url = url.replace("postgresql://", "postgresql+asyncpg://", 1)
return url
if self.backend == "mysql":
url = self.mysql_url
if not url:
raise ValueError("database.mysql_url is required for backend='mysql'")
if url.startswith("mysql://"):
url = url.replace("mysql://", "mysql+asyncmy://", 1)
return url
raise ValueError(f"No SQLAlchemy URL for backend={self.backend!r}")
offline-backend-20260512/backend/packages/harness/deerflow/config/log_system_config.py
+18
View File
@@ -0,0 +1,18 @@
"""Configuration for the admin-only external log-system entry.
A small read-only section in ``config.yaml`` that lets a deployment point the
"日志系统" entry (shown in the bottom-right settings menu for admins) at an
external log/observability system. The URL is opened in a new browser tab.
"""
from __future__ import annotations
from pydantic import BaseModel, Field
class LogSystemConfig(BaseModel):
"""External log-system link surfaced in the admin settings menu."""
enabled: bool = Field(default=False, description="Whether the log-system entry is shown in the settings menu")
url: str = Field(default="", description="External log-system URL opened in a new tab when the entry is clicked")
label: str = Field(default="日志系统", description="Menu label shown for the entry")
offline-backend-20260512/backend/packages/harness/deerflow/config/system_settings.py
+20
View File
@@ -51,6 +51,14 @@ class AppearanceSettings(BaseModel):
default="light",
description="Theme applied for users who haven't picked their own (light / dark)",
)
default_skill_view: Literal["card", "list"] = Field(
default="list",
description="Default skill list display mode for users who haven't picked their own (card / list)",
)
default_agent_view: Literal["card", "list"] = Field(
default="card",
description="Default agent list display mode for users who haven't picked their own (card / list)",
)
model_config = ConfigDict(extra="ignore")
@@ -78,6 +86,17 @@ class PrivateSquareSettings(BaseModel):
model_config = ConfigDict(extra="ignore")
class LeaderboardSettings(BaseModel):
"""Admin-configurable defaults for the user leaderboard analytics."""
exclude_user_ids: list[str] = Field(default_factory=list, description="User ids excluded from leaderboard aggregation.")
include_scheduled: bool = Field(default=True, description="Whether scheduled task runs participate in leaderboard statistics.")
include_admins: bool = Field(default=True, description="Whether admin accounts participate in leaderboard statistics.")
include_failed: bool = Field(default=True, description="Whether failed runs and provider calls participate in leaderboard statistics.")
cleanup_system_questions: bool = Field(default=True, description="Hide attachment/system marker questions such as <uploaded_files>.")
model_config = ConfigDict(extra="ignore")
class SystemSettings(BaseModel):
"""Top-level system settings document."""
@@ -85,6 +104,7 @@ class SystemSettings(BaseModel):
private_square: PrivateSquareSettings = Field(default_factory=PrivateSquareSettings)
appearance: AppearanceSettings = Field(default_factory=AppearanceSettings)
citation_display: CitationDisplaySettings = Field(default_factory=CitationDisplaySettings)
leaderboard: LeaderboardSettings = Field(default_factory=LeaderboardSettings)
model_config = ConfigDict(extra="ignore")
offline-backend-20260512/backend/packages/harness/deerflow/persistence/admin_stats/__init__.py
+5
View File
@@ -0,0 +1,5 @@
"""Persistence models for admin analytics snapshots."""
from deerflow.persistence.admin_stats.model import AdminLeaderboardDailyStatRow
__all__ = ["AdminLeaderboardDailyStatRow"]
offline-backend-20260512/backend/packages/harness/deerflow/persistence/admin_stats/model.py
+31
View File
@@ -0,0 +1,31 @@
"""ORM models for admin analytics snapshots."""
from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy import Index, String, Text
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
from deerflow.persistence.types import BeijingDateTime, PortableJSON
class AdminLeaderboardDailyStatRow(Base):
"""One precomputed leaderboard snapshot for a single local calendar day."""
__tablename__ = "admin_leaderboard_daily_stats"
__table_args__ = (
Index("ix_admin_leaderboard_daily_stats_status", "status"),
Index("ix_admin_leaderboard_daily_stats_updated_at", "updated_at"),
)
stat_date: Mapped[str] = mapped_column(String(10), primary_key=True)
settings_hash: Mapped[str] = mapped_column(String(64), primary_key=True)
status: Mapped[str] = mapped_column(String(16), default="ready")
payload_json: Mapped[dict] = mapped_column(PortableJSON(), default=dict)
message: Mapped[str | None] = mapped_column(Text)
error: Mapped[str | None] = mapped_column(Text)
generated_at: Mapped[datetime | None] = mapped_column(BeijingDateTime())
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), default=lambda: datetime.now(UTC))
updated_at: Mapped[datetime] = mapped_column(BeijingDateTime(), default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC))
offline-backend-20260512/backend/packages/harness/deerflow/persistence/agents/__init__.py
+8 -1
View File
@@ -4,13 +4,20 @@ from __future__ import annotations
from deerflow.persistence.agents.base import AgentStore
from deerflow.persistence.agents.memory import MemoryAgentStore
from deerflow.persistence.agents.model import AgentFavoriteRow, AgentRow
from deerflow.persistence.agents.model import (
AgentExtrasRow,
AgentFavoriteRow,
AgentRow,
AgentSkillRow,
)
from deerflow.persistence.agents.sql import AgentRepository
__all__ = [
"AgentExtrasRow",
"AgentFavoriteRow",
"AgentRepository",
"AgentRow",
"AgentSkillRow",
"AgentStore",
"MemoryAgentStore",
"make_agent_store",
offline-backend-20260512/backend/packages/harness/deerflow/persistence/agents/base.py
+88 -3
View File
@@ -118,6 +118,69 @@ class AgentStore(ABC):
"""
raise NotImplementedError
@abstractmethod
async def list_paginated(
self,
*,
user_id: str,
scope: str | None,
search: str | None,
tag_ids: list[str] | None,
page: int,
page_size: int,
admin: bool = False,
sort: str = "time_desc",
) -> tuple[list[dict[str, Any]], int]:
"""Return one page of agents plus the total matching count.
``scope``:
- ``"mine"`` — owned by ``user_id`` OR favorited by ``user_id``.
- ``"square"`` — built-in or published, and not private-square.
- ``"private"``— flagged ``is_private=True`` (caller gates access).
- ``None`` — the default listing visibility (``list_visible``).
``admin=True`` bypasses the visibility/scope filter entirely (every
agent), but still honors ``search`` / ``tag_ids``. Caller authorizes.
``search`` matches the raw name OR its desensitized display form.
``tag_ids`` keeps agents carrying ANY of the tags (OR). Both filters,
the COUNT, and LIMIT/OFFSET all execute in SQL.
Returns ``(rows, total)`` where ``rows`` is at most ``page_size`` items
for the 1-based ``page``.
"""
raise NotImplementedError
# --- Listing cache (skills / tool_groups / model) ---
@abstractmethod
async def replace_skills(self, agent_id: str, skills: list[str] | None) -> None:
"""Replace the cached skill list for ``agent_id``.
``None`` and ``[]`` both clear the cache to "no skills" (mirrors the
tri-state semantics of ``AgentConfig.skills``).
"""
raise NotImplementedError
@abstractmethod
async def set_extras(self, agent_id: str, *, tool_groups: list[str] | None, model: str | None) -> None:
"""Upsert the cached ``tool_groups`` / ``model`` for ``agent_id``."""
raise NotImplementedError
@abstractmethod
async def get_extras_for(self, agent_ids: list[str]) -> dict[str, dict[str, Any]]:
"""Batch-fetch cached listing extras for ``agent_ids``.
Returns ``{agent_id: {"skills": list[str] | None, "tool_groups":
list[str] | None, "model": str | None}}``. An agent that was never
mirrored is absent from the mapping (caller treats it as unknown).
"""
raise NotImplementedError
@abstractmethod
async def delete_extras_for_agent(self, agent_id: str) -> None:
"""Drop the cached skills + extras rows for ``agent_id`` (on delete)."""
raise NotImplementedError
# --- Favorites ("收藏") ---
@abstractmethod
@@ -136,6 +199,28 @@ class AgentStore(ABC):
raise NotImplementedError
@abstractmethod
async def delete_favorites_for_agent(self, agent_id: str) -> None:
"""Drop every user's favorite of ``agent_id`` (called on agent delete)."""
raise NotImplementedError
async def delete_favorites_for_agent(self, agent_id: str) -> None:
"""Drop every user's favorite of ``agent_id`` (called on agent delete)."""
raise NotImplementedError
# --- Per-user pins ("置顶") ---
@abstractmethod
async def add_pin(self, user_id: str, agent_id: str) -> None:
"""Pin ``agent_id`` for ``user_id``. Idempotent."""
raise NotImplementedError
@abstractmethod
async def remove_pin(self, user_id: str, agent_id: str) -> bool:
"""Remove a per-user pin. Returns ``False`` when none existed."""
raise NotImplementedError
@abstractmethod
async def list_pin_ids(self, user_id: str) -> set[str]:
"""Return the set of agent ids pinned by ``user_id``."""
raise NotImplementedError
@abstractmethod
async def delete_pins_for_agent(self, agent_id: str) -> None:
"""Drop every user's pin of ``agent_id`` (called on agent delete)."""
raise NotImplementedError
offline-backend-20260512/backend/packages/harness/deerflow/persistence/agents/memory.py
+117 -4
View File
@@ -5,6 +5,7 @@ from __future__ import annotations
from datetime import UTC, datetime
from typing import Any
from deerflow.config.agent_name_desensitize import name_matches
from deerflow.persistence.agents.base import AgentStore
@@ -20,8 +21,12 @@ def _copy(data: dict[str, Any]) -> dict[str, Any]:
class MemoryAgentStore(AgentStore):
def __init__(self) -> None:
self._agents: dict[str, dict[str, Any]] = {}
self._favorites: set[tuple[str, str]] = set()
self._agents: dict[str, dict[str, Any]] = {}
self._favorites: set[tuple[str, str]] = set()
self._pins: set[tuple[str, str]] = set()
# Listing cache mirrors (see AgentExtrasRow / AgentSkillRow):
self._skills: dict[str, list[str]] = {}
self._extras: dict[str, dict[str, Any]] = {}
@staticmethod
def _is_visible(agent: dict[str, Any], user_id: str) -> bool:
@@ -166,6 +171,96 @@ class MemoryAgentStore(AgentStore):
self._agents.pop(agent_id, None)
return True
# --- Paginated listing ---
def _matches_scope(self, agent: dict[str, Any], *, user_id: str, scope: str | None, admin: bool) -> bool:
if admin:
return True
if scope == "square":
return (agent.get("user_id") is None or bool(agent.get("published"))) and not bool(agent.get("is_private"))
if scope == "mine":
owned = agent.get("user_id") is not None and agent.get("user_id") == user_id
favorited = (user_id, agent["id"]) in self._favorites
return owned or favorited
if scope == "private":
return bool(agent.get("is_private"))
return self._is_visible(agent, user_id)
async def list_paginated(
self,
*,
user_id: str,
scope: str | None,
search: str | None,
tag_ids: list[str] | None,
page: int,
page_size: int,
admin: bool = False,
sort: str = "time_desc",
) -> tuple[list[dict[str, Any]], int]:
# NOTE: the in-memory store has no tag-assignment data, so ``tag_ids``
# is ignored here. Tag-filter behavior is covered by the SQL repo tests.
matched = [
a
for a in self._agents.values()
if self._matches_scope(a, user_id=user_id, scope=scope, admin=admin)
and (not search or not search.strip() or name_matches(a.get("name") or "", search))
]
if sort == "time_asc":
matched.sort(key=lambda a: (a.get("created_at") or "", a.get("id") or ""))
elif sort == "updated_desc":
matched.sort(
key=lambda a: (a.get("updated_at") or "", a.get("id") or ""),
reverse=True,
)
elif sort == "updated_asc":
matched.sort(key=lambda a: (a.get("updated_at") or "", a.get("id") or ""))
elif sort == "name_asc":
matched.sort(key=lambda a: ((a.get("name") or "").lower(), a.get("id") or ""))
elif sort == "name_desc":
matched.sort(
key=lambda a: ((a.get("name") or "").lower(), a.get("id") or ""),
reverse=True,
)
else:
matched.sort(
key=lambda a: (a.get("created_at") or "", a.get("id") or ""),
reverse=True,
)
pin_ids = {agent_id for uid, agent_id in self._pins if uid == user_id}
matched.sort(key=lambda a: (0 if a.get("id") in pin_ids else 1,))
total = len(matched)
start = max(page - 1, 0) * page_size
page_rows = matched[start : start + page_size]
return [_copy(a) for a in page_rows], total
# --- Listing cache (skills / tool_groups / model) ---
async def replace_skills(self, agent_id: str, skills: list[str] | None) -> None:
self._skills[agent_id] = [s for s in (skills or []) if isinstance(s, str)]
async def set_extras(self, agent_id: str, *, tool_groups: list[str] | None, model: str | None) -> None:
self._extras[agent_id] = {"tool_groups": tool_groups, "model": model}
async def get_extras_for(self, agent_ids: list[str]) -> dict[str, dict[str, Any]]:
out: dict[str, dict[str, Any]] = {}
for agent_id in agent_ids:
has_extras = agent_id in self._extras
has_skills = agent_id in self._skills
if not has_extras and not has_skills:
continue
extras = self._extras.get(agent_id, {})
out[agent_id] = {
"skills": list(self._skills.get(agent_id, [])),
"tool_groups": extras.get("tool_groups"),
"model": extras.get("model"),
}
return out
async def delete_extras_for_agent(self, agent_id: str) -> None:
self._skills.pop(agent_id, None)
self._extras.pop(agent_id, None)
# --- Favorites ("收藏") ---
async def add_favorite(self, user_id: str, agent_id: str) -> None:
@@ -180,5 +275,23 @@ class MemoryAgentStore(AgentStore):
async def list_favorite_ids(self, user_id: str) -> set[str]:
return {agent_id for (uid, agent_id) in self._favorites if uid == user_id}
async def delete_favorites_for_agent(self, agent_id: str) -> None:
self._favorites = {(uid, aid) for (uid, aid) in self._favorites if aid != agent_id}
async def delete_favorites_for_agent(self, agent_id: str) -> None:
self._favorites = {(uid, aid) for (uid, aid) in self._favorites if aid != agent_id}
# --- Pins ("置顶") ---
async def add_pin(self, user_id: str, agent_id: str) -> None:
self._pins.add((user_id, agent_id))
async def remove_pin(self, user_id: str, agent_id: str) -> bool:
key = (user_id, agent_id)
if key not in self._pins:
return False
self._pins.remove(key)
return True
async def list_pin_ids(self, user_id: str) -> set[str]:
return {agent_id for uid, agent_id in self._pins if uid == user_id}
async def delete_pins_for_agent(self, agent_id: str) -> None:
self._pins = {(uid, aid) for (uid, aid) in self._pins if aid != agent_id}
offline-backend-20260512/backend/packages/harness/deerflow/persistence/agents/model.py
+60 -4
View File
@@ -4,11 +4,11 @@ from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy import BigInteger, Boolean, String, UniqueConstraint
from sqlalchemy import BigInteger, Boolean, Integer, String, UniqueConstraint
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
from deerflow.persistence.types import BeijingDateTime
from deerflow.persistence.types import BeijingDateTime, PortableJSON
class AgentRow(Base):
@@ -37,7 +37,7 @@ class AgentRow(Base):
updated_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC))
class AgentFavoriteRow(Base):
class AgentFavoriteRow(Base):
"""A user's bookmark of a (typically public-square) agent.
Lets a user collect agents from the 广场 (square) into their "我的" tab
@@ -51,4 +51,60 @@ class AgentFavoriteRow(Base):
id: Mapped[str] = mapped_column(String(64), primary_key=True)
user_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
agent_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC))
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC))
class AgentPinRow(Base):
"""A per-user pin that keeps an agent near the top of that user's lists."""
__tablename__ = "agent_pins"
__table_args__ = (UniqueConstraint("user_id", "agent_id", name="uq_agent_pins_user_agent"),)
id: Mapped[str] = mapped_column(String(64), primary_key=True)
user_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
agent_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC))
class AgentSkillRow(Base):
"""Denormalized cache of a custom agent's selected skills.
``config.yaml`` under ``.deer-flow/agents/{id}/`` stays the source of truth
for the agent runtime (the lead-agent prompt reads it directly). This table
is a read cache so the gallery list endpoint can render skill badges with a
single batched ``IN`` query instead of stat+YAML-parsing every agent's
``config.yaml`` on each request. Written through on create/update and
backfilled from disk by ``_sync_legacy_agents``.
One row per (agent, skill); ``position`` preserves the configured order.
"""
__tablename__ = "agent_skills"
__table_args__ = (UniqueConstraint("agent_id", "skill_name", name="uq_agent_skills_agent_skill"),)
id: Mapped[str] = mapped_column(String(64), primary_key=True)
agent_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
skill_name: Mapped[str] = mapped_column(String(191), nullable=False, index=True)
position: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
class AgentExtrasRow(Base):
"""Denormalized cache of a custom agent's ``tool_groups`` / ``model``.
Companion to :class:`AgentSkillRow` — same rationale (avoid per-agent
``config.yaml`` reads in the list path). One row per agent. The presence of
a row also marks "this agent has been mirrored": the list path treats a
missing row as "unknown / not yet backfilled" (``skills=None``) so it can
be told apart from an explicitly empty skill list.
``skills_synced_at`` records when the cache was last written from disk so
the backfill can skip agents whose ``config.yaml`` has not changed.
"""
__tablename__ = "agent_extras"
agent_id: Mapped[str] = mapped_column(String(64), primary_key=True)
# list[str] | None — the agent's tool_groups whitelist from config.yaml.
tool_groups: Mapped[list | None] = mapped_column(PortableJSON(), nullable=True, default=None)
model: Mapped[str | None] = mapped_column(String(191), nullable=True, default=None)
skills_synced_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC))
offline-backend-20260512/backend/packages/harness/deerflow/persistence/agents/sql.py
+217 -6
View File
@@ -6,12 +6,20 @@ from datetime import UTC, datetime
from typing import Any
from uuid import uuid4
from sqlalchemy import and_, delete, func, or_, select
from sqlalchemy import and_, case, delete, func, or_, select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from deerflow.config.agent_name_desensitize import source_terms_for_needle
from deerflow.persistence.agents.base import AgentStore
from deerflow.persistence.agents.model import AgentFavoriteRow, AgentRow
from deerflow.persistence.agents.model import (
AgentExtrasRow,
AgentFavoriteRow,
AgentPinRow,
AgentRow,
AgentSkillRow,
)
from deerflow.persistence.tags.model import TagAssignmentRow
def _row_to_dict(row: AgentRow) -> dict[str, Any]:
@@ -233,6 +241,167 @@ class AgentRepository(AgentStore):
await session.commit()
return True
# --- Paginated listing ---
def _scope_filter(self, *, user_id: str, scope: str | None, admin: bool):
"""Build the visibility predicate for a paginated listing."""
if admin:
return None # no visibility filter — every agent
if scope == "square":
return and_(
or_(AgentRow.user_id.is_(None), AgentRow.published.is_(True)),
AgentRow.is_private.is_(False),
)
if scope == "mine":
# owned OR favorited. A correlated IN-subquery (not a JOIN) keeps a
# single row per agent even when it is both owned and favorited.
fav_subq = select(AgentFavoriteRow.agent_id).where(AgentFavoriteRow.user_id == user_id)
return or_(AgentRow.user_id == user_id, AgentRow.id.in_(fav_subq))
if scope == "private":
return AgentRow.is_private.is_(True)
return self._listing_filter(user_id)
@staticmethod
def _search_clause(search: str):
"""Match the raw name OR (via the desensitize mirror) its display form."""
needle = search.strip().lower()
clauses = [func.lower(AgentRow.name).like(f"%{needle}%")]
for term in source_terms_for_needle(needle):
clauses.append(AgentRow.name.like(f"%{term}%"))
return or_(*clauses)
@staticmethod
def _tag_clause(tag_ids: list[str]):
subq = select(TagAssignmentRow.target_id).where(
TagAssignmentRow.target_type == "agent",
TagAssignmentRow.tag_id.in_(tag_ids),
)
return AgentRow.id.in_(subq)
async def list_paginated(
self,
*,
user_id: str,
scope: str | None,
search: str | None,
tag_ids: list[str] | None,
page: int,
page_size: int,
admin: bool = False,
sort: str = "time_desc",
) -> tuple[list[dict[str, Any]], int]:
filters = []
scope_filter = self._scope_filter(user_id=user_id, scope=scope, admin=admin)
if scope_filter is not None:
filters.append(scope_filter)
if search and search.strip():
filters.append(self._search_clause(search))
if tag_ids:
filters.append(self._tag_clause(tag_ids))
count_stmt = select(func.count()).select_from(AgentRow)
pin_subq = select(AgentPinRow.agent_id).where(AgentPinRow.user_id == user_id)
pin_rank = case((AgentRow.id.in_(pin_subq), 0), else_=1)
if sort == "time_asc":
order_by = (AgentRow.created_at.asc(), AgentRow.id.asc())
elif sort == "updated_desc":
order_by = (AgentRow.updated_at.desc(), AgentRow.id.asc())
elif sort == "updated_asc":
order_by = (AgentRow.updated_at.asc(), AgentRow.id.asc())
elif sort == "name_asc":
order_by = (func.lower(AgentRow.name).asc(), AgentRow.id.asc())
elif sort == "name_desc":
order_by = (func.lower(AgentRow.name).desc(), AgentRow.id.asc())
else:
order_by = (AgentRow.created_at.desc(), AgentRow.id.asc())
page_stmt = (
select(AgentRow)
.order_by(pin_rank.asc(), *order_by)
.limit(page_size)
.offset(max(page - 1, 0) * page_size)
)
if filters:
where = and_(*filters)
count_stmt = count_stmt.where(where)
page_stmt = page_stmt.where(where)
async with self._sf() as session:
total = await session.scalar(count_stmt) or 0
result = await session.execute(page_stmt)
rows = [_row_to_dict(row) for row in result.scalars()]
return rows, total
# --- Listing cache (skills / tool_groups / model) ---
async def replace_skills(self, agent_id: str, skills: list[str] | None) -> None:
async with self._sf() as session:
await session.execute(delete(AgentSkillRow).where(AgentSkillRow.agent_id == agent_id))
for position, name in enumerate(skills or []):
if not isinstance(name, str):
continue
session.add(
AgentSkillRow(
id=uuid4().hex,
agent_id=agent_id,
skill_name=name,
position=position,
)
)
await session.commit()
async def set_extras(self, agent_id: str, *, tool_groups: list[str] | None, model: str | None) -> None:
async with self._sf() as session:
row = await session.get(AgentExtrasRow, agent_id)
now = datetime.now(UTC)
if row is None:
session.add(
AgentExtrasRow(
agent_id=agent_id,
tool_groups=tool_groups,
model=model,
skills_synced_at=now,
)
)
else:
row.tool_groups = tool_groups
row.model = model
row.skills_synced_at = now
await session.commit()
async def get_extras_for(self, agent_ids: list[str]) -> dict[str, dict[str, Any]]:
if not agent_ids:
return {}
out: dict[str, dict[str, Any]] = {}
async with self._sf() as session:
extras = await session.execute(
select(AgentExtrasRow).where(AgentExtrasRow.agent_id.in_(agent_ids))
)
for row in extras.scalars():
out[row.agent_id] = {
"skills": [], # default; filled below if skill rows exist
"tool_groups": row.tool_groups,
"model": row.model,
}
skill_rows = await session.execute(
select(AgentSkillRow.agent_id, AgentSkillRow.skill_name)
.where(AgentSkillRow.agent_id.in_(agent_ids))
.order_by(AgentSkillRow.agent_id.asc(), AgentSkillRow.position.asc())
)
for agent_id, skill_name in skill_rows.all():
entry = out.setdefault(
agent_id, {"skills": [], "tool_groups": None, "model": None}
)
entry["skills"].append(skill_name)
return out
async def delete_extras_for_agent(self, agent_id: str) -> None:
async with self._sf() as session:
await session.execute(delete(AgentSkillRow).where(AgentSkillRow.agent_id == agent_id))
await session.execute(delete(AgentExtrasRow).where(AgentExtrasRow.agent_id == agent_id))
await session.commit()
# --- Favorites ("收藏") ---
async def add_favorite(self, user_id: str, agent_id: str) -> None:
@@ -272,7 +441,49 @@ class AgentRepository(AgentStore):
)
return {row[0] for row in result.all()}
async def delete_favorites_for_agent(self, agent_id: str) -> None:
async with self._sf() as session:
await session.execute(delete(AgentFavoriteRow).where(AgentFavoriteRow.agent_id == agent_id))
await session.commit()
async def delete_favorites_for_agent(self, agent_id: str) -> None:
async with self._sf() as session:
await session.execute(delete(AgentFavoriteRow).where(AgentFavoriteRow.agent_id == agent_id))
await session.commit()
# --- Pins ("置顶") ---
async def add_pin(self, user_id: str, agent_id: str) -> None:
async with self._sf() as session:
existing = (
await session.execute(
select(AgentPinRow.id)
.where(AgentPinRow.user_id == user_id, AgentPinRow.agent_id == agent_id)
.limit(1)
)
).scalar_one_or_none()
if existing is not None:
return
session.add(AgentPinRow(id=uuid4().hex, user_id=user_id, agent_id=agent_id))
try:
await session.commit()
except IntegrityError:
await session.rollback()
async def remove_pin(self, user_id: str, agent_id: str) -> bool:
async with self._sf() as session:
result = await session.execute(
delete(AgentPinRow).where(
AgentPinRow.user_id == user_id,
AgentPinRow.agent_id == agent_id,
)
)
await session.commit()
return (result.rowcount or 0) > 0
async def list_pin_ids(self, user_id: str) -> set[str]:
async with self._sf() as session:
result = await session.execute(
select(AgentPinRow.agent_id).where(AgentPinRow.user_id == user_id)
)
return {row[0] for row in result.all()}
async def delete_pins_for_agent(self, agent_id: str) -> None:
async with self._sf() as session:
await session.execute(delete(AgentPinRow).where(AgentPinRow.agent_id == agent_id))
await session.commit()
offline-backend-20260512/backend/packages/harness/deerflow/persistence/engine.py
+95 -83
View File
@@ -11,16 +11,16 @@ None and fall back to in-memory implementations.
from __future__ import annotations
import asyncio
import json
import logging
import os
import sys
from importlib.util import find_spec
from pathlib import Path
from sqlalchemy import text
from sqlalchemy.engine import make_url
from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession, async_sessionmaker, create_async_engine
import json
import logging
import os
import sys
from importlib.util import find_spec
from pathlib import Path
from sqlalchemy import text
from sqlalchemy.engine import make_url
from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession, async_sessionmaker, create_async_engine
def _json_serializer(obj: object) -> str:
@@ -133,7 +133,7 @@ async def _run_alembic_upgrade(url: str) -> None:
logger.exception("Auto-applying Alembic migrations failed; continuing with create_all schema")
async def _auto_create_postgres_db(url: str) -> None:
async def _auto_create_postgres_db(url: str) -> None:
"""Connect to the ``postgres`` maintenance DB and CREATE DATABASE.
The target database name is extracted from *url*. The connection is
@@ -156,55 +156,57 @@ async def _auto_create_postgres_db(url: str) -> None:
async with maint_engine.connect() as conn:
await conn.execute(text(f'CREATE DATABASE "{db_name}"'))
logger.info("Auto-created PostgreSQL database: %s", db_name)
finally:
await maint_engine.dispose()
async def _auto_create_mysql_db(url: str) -> None:
"""Create the target MySQL database when it is missing.
``create_all`` can create tables, but MySQL requires the database/schema to
exist before the application URL can connect. We connect to the built-in
``mysql`` maintenance database on the same server and issue a conservative
``CREATE DATABASE IF NOT EXISTS``.
"""
parsed = make_url(url)
db_name = parsed.database
if not db_name:
raise ValueError("Cannot auto-create MySQL database: no database name in URL")
maint_url = parsed.set(database="mysql")
maint_engine = create_async_engine(
maint_url.render_as_string(hide_password=False),
isolation_level="AUTOCOMMIT",
pool_pre_ping=True,
)
quoted = "`" + db_name.replace("`", "``") + "`"
try:
async with maint_engine.connect() as conn:
await conn.execute(
text(
f"CREATE DATABASE IF NOT EXISTS {quoted} "
"CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci"
)
)
logger.info("Auto-created MySQL database if missing: %s", db_name)
finally:
await maint_engine.dispose()
def _is_mysql_unknown_database_error(exc: Exception) -> bool:
message = str(exc).lower()
return "unknown database" in message or "1049" in message
async def init_engine(
finally:
await maint_engine.dispose()
async def _auto_create_mysql_db(url: str) -> None:
"""Create the target MySQL database when it is missing.
``create_all`` can create tables, but MySQL requires the database/schema to
exist before the application URL can connect. We connect to the built-in
``mysql`` maintenance database on the same server and issue a conservative
``CREATE DATABASE IF NOT EXISTS``.
"""
parsed = make_url(url)
db_name = parsed.database
if not db_name:
raise ValueError("Cannot auto-create MySQL database: no database name in URL")
maint_url = parsed.set(database="mysql")
maint_engine = create_async_engine(
maint_url.render_as_string(hide_password=False),
isolation_level="AUTOCOMMIT",
pool_pre_ping=True,
)
quoted = "`" + db_name.replace("`", "``") + "`"
try:
async with maint_engine.connect() as conn:
await conn.execute(
text(
f"CREATE DATABASE IF NOT EXISTS {quoted} "
"CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci"
)
)
logger.info("Auto-created MySQL database if missing: %s", db_name)
finally:
await maint_engine.dispose()
def _is_mysql_unknown_database_error(exc: Exception) -> bool:
message = str(exc).lower()
return "unknown database" in message or "1049" in message
async def init_engine(
backend: str,
*,
url: str = "",
echo: bool = False,
pool_size: int = 5,
max_overflow: int = 10,
pool_timeout: int = 30,
sqlite_dir: str = "",
) -> None:
"""Create the async engine and session factory, then auto-create tables.
@@ -213,7 +215,9 @@ async def init_engine(
backend: "memory", "sqlite", "postgres", or "mysql".
url: SQLAlchemy async URL (for sqlite/postgres/mysql).
echo: Echo SQL to log.
pool_size: Postgres connection pool size.
pool_size: Base connection pool size (postgres/mysql).
max_overflow: Extra connections allowed beyond pool_size under load (postgres/mysql).
pool_timeout: Seconds to wait for a free pooled connection before erroring (postgres/mysql).
sqlite_dir: Directory to create for SQLite (ensured to exist).
"""
global _engine, _session_factory
@@ -271,6 +275,8 @@ async def init_engine(
url,
echo=echo,
pool_size=pool_size,
max_overflow=max_overflow,
pool_timeout=pool_timeout,
pool_pre_ping=True,
json_serializer=_json_serializer,
)
@@ -279,6 +285,8 @@ async def init_engine(
url,
echo=echo,
pool_size=pool_size,
max_overflow=max_overflow,
pool_timeout=pool_timeout,
pool_pre_ping=True,
pool_recycle=1800,
json_serializer=_json_serializer,
@@ -302,37 +310,39 @@ async def init_engine(
# This is expected during initial scaffolding or minimal installs.
logger.debug("deerflow.persistence.models not found; skipping auto-create tables")
try:
async with _engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
except Exception as exc:
if backend == "postgres" and "does not exist" in str(exc):
try:
async with _engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
except Exception as exc:
if backend == "postgres" and "does not exist" in str(exc):
# Database not yet created — attempt to auto-create it, then retry.
await _auto_create_postgres_db(url)
# Rebuild engine against the now-existing database
await _engine.dispose()
_engine = create_async_engine(url, echo=echo, pool_size=pool_size, pool_pre_ping=True, json_serializer=_json_serializer)
_session_factory = async_sessionmaker(_engine, expire_on_commit=False)
async with _engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
elif backend == "mysql" and _is_mysql_unknown_database_error(exc):
# Database not yet created. Create it, rebuild the app engine
# against the target DB, then let SQLAlchemy create all tables.
await _auto_create_mysql_db(url)
await _engine.dispose()
_engine = create_async_engine(
url,
echo=echo,
pool_size=pool_size,
pool_pre_ping=True,
pool_recycle=1800,
json_serializer=_json_serializer,
)
_session_factory = async_sessionmaker(_engine, expire_on_commit=False)
async with _engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
else:
raise
_engine = create_async_engine(url, echo=echo, pool_size=pool_size, max_overflow=max_overflow, pool_timeout=pool_timeout, pool_pre_ping=True, json_serializer=_json_serializer)
_session_factory = async_sessionmaker(_engine, expire_on_commit=False)
async with _engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
elif backend == "mysql" and _is_mysql_unknown_database_error(exc):
# Database not yet created. Create it, rebuild the app engine
# against the target DB, then let SQLAlchemy create all tables.
await _auto_create_mysql_db(url)
await _engine.dispose()
_engine = create_async_engine(
url,
echo=echo,
pool_size=pool_size,
max_overflow=max_overflow,
pool_timeout=pool_timeout,
pool_pre_ping=True,
pool_recycle=1800,
json_serializer=_json_serializer,
)
_session_factory = async_sessionmaker(_engine, expire_on_commit=False)
async with _engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
else:
raise
# Apply in-place schema fixes (column widenings, etc.) that ``create_all``
# cannot do on its own. Failures don't abort startup — see the helper.
@@ -351,6 +361,8 @@ async def init_engine_from_config(config) -> None:
url=config.app_sqlalchemy_url,
echo=config.echo_sql,
pool_size=config.pool_size,
max_overflow=getattr(config, "max_overflow", 10),
pool_timeout=getattr(config, "pool_timeout", 30),
sqlite_dir=config.sqlite_dir if config.backend == "sqlite" else "",
)
offline-backend-20260512/backend/packages/harness/deerflow/persistence/llm_metrics/base.py
+3
View File
@@ -34,6 +34,9 @@ class LlmMetricsQuery:
"""Filter set for the admin list / export endpoints."""
user_id: str | None = None
# Multiple user ids — set when a fuzzy username search resolves to several
# accounts. Takes precedence over ``user_id`` when non-empty.
user_ids: list[str] | None = None
model_name: str | None = None
status: str | None = None
since: datetime | None = None
offline-backend-20260512/backend/packages/harness/deerflow/persistence/llm_metrics/model.py
+5 -1
View File
@@ -4,7 +4,7 @@ from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy import Float, Integer, String, Text
from sqlalchemy import Float, Index, Integer, String, Text
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
@@ -15,6 +15,10 @@ class LlmCallMetricRow(Base):
"""One row per completed (or failed) LLM provider invocation."""
__tablename__ = "llm_call_metrics"
__table_args__ = (
Index("ix_llm_call_metrics_created_user", "created_at", "user_id"),
Index("ix_llm_call_metrics_run_created", "run_id", "created_at"),
)
id: Mapped[str] = mapped_column(String(64), primary_key=True)
# Wall-clock time the LLM call ended. Indexed because admin views always
offline-backend-20260512/backend/packages/harness/deerflow/persistence/llm_metrics/sql.py
+4 -1
View File
@@ -58,7 +58,9 @@ class SqlLlmMetricsStore(LlmMetricsStore):
logger.warning("Failed to persist llm_call_metrics row %s", metric.id, exc_info=True)
def _apply_filters(self, stmt, query: LlmMetricsQuery):
if query.user_id:
if query.user_ids:
stmt = stmt.where(LlmCallMetricRow.user_id.in_(query.user_ids))
elif query.user_id:
stmt = stmt.where(LlmCallMetricRow.user_id == query.user_id)
if query.model_name:
stmt = stmt.where(LlmCallMetricRow.model_name == query.model_name)
@@ -92,6 +94,7 @@ class SqlLlmMetricsStore(LlmMetricsStore):
while True:
page_query = LlmMetricsQuery(
user_id=query.user_id,
user_ids=query.user_ids,
model_name=query.model_name,
status=query.status,
since=query.since,
offline-backend-20260512/backend/packages/harness/deerflow/persistence/migrations/versions/20260603_02_user_custom_prompts_agent_id.py
+45
View File
@@ -0,0 +1,45 @@
"""Add ``agent_id`` to ``user_custom_prompts`` for per-agent prompt scoping.
NULL = global (shown for every agent); otherwise the prompt only surfaces when
that agent is selected.
Revision ID: 20260603_02
Revises: 20260603_01
Create Date: 2026-06-03
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
from sqlalchemy import inspect
revision = "20260603_02"
down_revision = "20260603_01"
branch_labels = None
depends_on = None
def _has_table(table_name: str) -> bool:
inspector = inspect(op.get_bind())
return table_name in inspector.get_table_names()
def _has_column(table_name: str, column_name: str) -> bool:
inspector = inspect(op.get_bind())
return any(col["name"] == column_name for col in inspector.get_columns(table_name))
def upgrade() -> None:
if not _has_table("user_custom_prompts"):
return
if not _has_column("user_custom_prompts", "agent_id"):
op.add_column("user_custom_prompts", sa.Column("agent_id", sa.String(64), nullable=True))
op.create_index("ix_user_custom_prompts_agent_id", "user_custom_prompts", ["agent_id"])
def downgrade() -> None:
if _has_table("user_custom_prompts") and _has_column("user_custom_prompts", "agent_id"):
op.drop_index("ix_user_custom_prompts_agent_id", table_name="user_custom_prompts")
op.drop_column("user_custom_prompts", "agent_id")
offline-backend-20260512/backend/packages/harness/deerflow/persistence/migrations/versions/20260603_03_user_preferences.py
+42
View File
@@ -0,0 +1,42 @@
"""Add ``user_preferences`` for per-user UI toggles (account-bound).
Holds a single JSON blob per user — e.g. ``hide_admin_prompt_prefix`` so a user
can choose to use only their own prompts and hide the admin-configured prefix.
Revision ID: 20260603_03
Revises: 20260603_02
Create Date: 2026-06-03
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
from sqlalchemy import inspect
revision = "20260603_03"
down_revision = "20260603_02"
branch_labels = None
depends_on = None
def _has_table(table_name: str) -> bool:
inspector = inspect(op.get_bind())
return table_name in inspector.get_table_names()
def upgrade() -> None:
if not _has_table("user_preferences"):
op.create_table(
"user_preferences",
sa.Column("user_id", sa.String(64), primary_key=True),
sa.Column("preferences", sa.JSON(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
)
def downgrade() -> None:
if _has_table("user_preferences"):
op.drop_table("user_preferences")
offline-backend-20260512/backend/packages/harness/deerflow/persistence/migrations/versions/20260604_01_agent_pins.py
+56
View File
@@ -0,0 +1,56 @@
"""Per-user pinned agents.
Revision ID: 20260604_01
Revises: 20260603_03
Create Date: 2026-06-04
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
from sqlalchemy import inspect
revision = "20260604_01"
down_revision = "20260603_03"
branch_labels = None
depends_on = None
def _has_table(table_name: str) -> bool:
inspector = inspect(op.get_bind())
return table_name in inspector.get_table_names()
def _has_index(table_name: str, index_name: str) -> bool:
if not _has_table(table_name):
return False
inspector = inspect(op.get_bind())
return any(idx["name"] == index_name for idx in inspector.get_indexes(table_name))
def upgrade() -> None:
if not _has_table("agent_pins"):
op.create_table(
"agent_pins",
sa.Column("id", sa.String(64), nullable=False),
sa.Column("user_id", sa.String(64), nullable=False),
sa.Column("agent_id", sa.String(64), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("user_id", "agent_id", name="uq_agent_pins_user_agent"),
)
if not _has_index("agent_pins", "ix_agent_pins_user_id"):
op.create_index("ix_agent_pins_user_id", "agent_pins", ["user_id"])
if not _has_index("agent_pins", "ix_agent_pins_agent_id"):
op.create_index("ix_agent_pins_agent_id", "agent_pins", ["agent_id"])
def downgrade() -> None:
if _has_index("agent_pins", "ix_agent_pins_agent_id"):
op.drop_index("ix_agent_pins_agent_id", table_name="agent_pins")
if _has_index("agent_pins", "ix_agent_pins_user_id"):
op.drop_index("ix_agent_pins_user_id", table_name="agent_pins")
if _has_table("agent_pins"):
op.drop_table("agent_pins")
offline-backend-20260512/backend/packages/harness/deerflow/persistence/migrations/versions/20260604_01_roundtable_jobs.py
+2 -2
View File
@@ -7,7 +7,7 @@ MySQL (can exceed the 64 KB TEXT cap) and plain TEXT elsewhere — matching
``PortableLongText`` on ``RoundtableJobRow``. ``create_all`` adds the table for
brand-new DBs; this revision creates it on DBs that predate it.
Revision ID: 20260604_01
Revision ID: 20260604_02
Revises: 20260603_01
Create Date: 2026-06-04
"""
@@ -20,7 +20,7 @@ from sqlalchemy import inspect
from sqlalchemy.dialects.mysql import LONGTEXT
revision = "20260604_01"
revision = "20260604_02"
down_revision = "20260603_01"
branch_labels = None
depends_on = None
offline-backend-20260512/backend/packages/harness/deerflow/persistence/migrations/versions/20260605_01_roundtable_jobs_dialogues.py
+46
View File
@@ -0,0 +1,46 @@
"""Add dialogues column to roundtable_jobs.
存研讨对话记录（leader 说明 + 各席位交付），使后台作业跑完 / 进行中用户都能看到
真实研讨内容（而非只有流程图进度）。LONGTEXT（对话可超 64 KB），匹配
``PortableLongText`` on ``RoundtableJobRow.dialogues``。
Revision ID: 20260605_01
Revises: 20260604_01
Create Date: 2026-06-05
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
from sqlalchemy import inspect
from sqlalchemy.dialects.mysql import LONGTEXT
revision = "20260605_01"
down_revision = "20260604_02"
branch_labels = None
depends_on = None
def _has_table(table_name: str) -> bool:
inspector = inspect(op.get_bind())
return table_name in inspector.get_table_names()
def _has_column(table_name: str, column_name: str) -> bool:
if not _has_table(table_name):
return False
inspector = inspect(op.get_bind())
return any(col["name"] == column_name for col in inspector.get_columns(table_name))
def upgrade() -> None:
if _has_table("roundtable_jobs") and not _has_column("roundtable_jobs", "dialogues"):
col_type = LONGTEXT() if op.get_bind().dialect.name == "mysql" else sa.Text()
op.add_column("roundtable_jobs", sa.Column("dialogues", col_type, nullable=True))
def downgrade() -> None:
if _has_column("roundtable_jobs", "dialogues"):
op.drop_column("roundtable_jobs", "dialogues")
offline-backend-20260512/backend/packages/harness/deerflow/persistence/migrations/versions/20260605_02_roundtable_jobs_step3.py
+46
View File
@@ -0,0 +1,46 @@
"""Add step3 column to roundtable_jobs.
把作业的 Step3 报告（{html, generatedAt, model, summary}）直接挂在 job 上，使前端
「多运行」模型（draft.step2.runs[]）能从单个 job 读到完整结果（对话 + 报告）组装一个 run，
而不必依赖草稿顶层的单一 step3。LONGTEXT（HTML 可超 64 KB）。
Revision ID: 20260605_02
Revises: 20260605_01
Create Date: 2026-06-05
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
from sqlalchemy import inspect
from sqlalchemy.dialects.mysql import LONGTEXT
revision = "20260605_02"
down_revision = "20260605_01"
branch_labels = None
depends_on = None
def _has_table(table_name: str) -> bool:
inspector = inspect(op.get_bind())
return table_name in inspector.get_table_names()
def _has_column(table_name: str, column_name: str) -> bool:
if not _has_table(table_name):
return False
inspector = inspect(op.get_bind())
return any(col["name"] == column_name for col in inspector.get_columns(table_name))
def upgrade() -> None:
if _has_table("roundtable_jobs") and not _has_column("roundtable_jobs", "step3"):
col_type = LONGTEXT() if op.get_bind().dialect.name == "mysql" else sa.Text()
op.add_column("roundtable_jobs", sa.Column("step3", col_type, nullable=True))
def downgrade() -> None:
if _has_column("roundtable_jobs", "step3"):
op.drop_column("roundtable_jobs", "step3")
offline-backend-20260512/backend/packages/harness/deerflow/persistence/migrations/versions/20260606_01_roundtable_draft_shares.py
+57
View File
@@ -0,0 +1,57 @@
"""Create roundtable_draft_shares table.
Per-draft share links for the roundtable-planning page, mirroring
``thread_shares`` but pointing at a ``roundtable_drafts`` row. ``import`` codes
copy the draft into the recipient's account; ``view`` codes back a public
read-only result-report page. ``create_all`` makes the table for brand-new DBs;
this revision patches DBs that already exist.
Revision ID: 20260606_01
Revises: 20260605_02
Create Date: 2026-06-06
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
from sqlalchemy import inspect
revision = "20260606_01"
down_revision = "20260605_02"
branch_labels = None
depends_on = None
def _has_table(table_name: str) -> bool:
inspector = inspect(op.get_bind())
return table_name in inspector.get_table_names()
def upgrade() -> None:
if _has_table("roundtable_draft_shares"):
return
op.create_table(
"roundtable_draft_shares",
sa.Column("share_code", sa.String(length=32), primary_key=True),
sa.Column("owner_user_id", sa.String(length=64), nullable=False),
sa.Column("draft_id", sa.String(length=64), nullable=False),
sa.Column("revoked", sa.Boolean(), nullable=False, server_default=sa.false()),
sa.Column("mode", sa.String(length=16), nullable=False, server_default="import"),
sa.Column("created_at", sa.DateTime(), nullable=False),
)
op.create_index(
"ix_roundtable_draft_shares_owner_user_id",
"roundtable_draft_shares",
["owner_user_id"],
)
op.create_index(
"ix_roundtable_draft_shares_draft_id",
"roundtable_draft_shares",
["draft_id"],
)
def downgrade() -> None:
if _has_table("roundtable_draft_shares"):
op.drop_table("roundtable_draft_shares")
offline-backend-20260512/backend/packages/harness/deerflow/persistence/migrations/versions/20260607_01_user_custom_prompts_agent_id_fixup.py
+47
View File
@@ -0,0 +1,47 @@
"""Add ``agent_id`` to ``user_custom_prompts`` when the parallel branch was skipped.
Some deployments reached ``20260606_01`` via the roundtable-jobs migration branch
(``20260603_01`` → ``20260604_01`` jobs → …) without running the sibling branch
that adds ``agent_id`` (``20260603_02``). This idempotent fixup applies the
missing column for those DBs.
Revision ID: 20260607_01
Revises: 20260606_01
Create Date: 2026-06-07
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
from sqlalchemy import inspect
revision = "20260607_01"
down_revision = "20260606_01"
branch_labels = None
depends_on = None
def _has_table(table_name: str) -> bool:
inspector = inspect(op.get_bind())
return table_name in inspector.get_table_names()
def _has_column(table_name: str, column_name: str) -> bool:
inspector = inspect(op.get_bind())
return any(col["name"] == column_name for col in inspector.get_columns(table_name))
def upgrade() -> None:
if not _has_table("user_custom_prompts"):
return
if not _has_column("user_custom_prompts", "agent_id"):
op.add_column("user_custom_prompts", sa.Column("agent_id", sa.String(64), nullable=True))
op.create_index("ix_user_custom_prompts_agent_id", "user_custom_prompts", ["agent_id"])
def downgrade() -> None:
if _has_table("user_custom_prompts") and _has_column("user_custom_prompts", "agent_id"):
op.drop_index("ix_user_custom_prompts_agent_id", table_name="user_custom_prompts")
op.drop_column("user_custom_prompts", "agent_id")
offline-backend-20260512/backend/packages/harness/deerflow/persistence/migrations/versions/20260607_02_merge_agent_pins_and_fixup.py
+22
View File
@@ -0,0 +1,22 @@
"""Merge parallel migration branches (agent_pins + roundtable jobs fixup).
Revision ID: 20260607_02
Revises: 20260604_01, 20260607_01
Create Date: 2026-06-07
"""
from __future__ import annotations
revision = "20260607_02"
down_revision = ("20260604_01", "20260607_01")
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
offline-backend-20260512/backend/packages/harness/deerflow/persistence/migrations/versions/20260607_03_chain_stages.py
+49
View File
@@ -0,0 +1,49 @@
"""Add stages column to roundtable_chains.
Layered orchestration (分层编排) for business chains: an optional list-of-lists of
agent_ids grouping the chain's seats into parallel stages (stages run serially).
NULL = linear chain. The column is LONGTEXT on MySQL (mirrors ``seats`` headroom)
and plain TEXT elsewhere — matching ``PortableLongText`` on
``RoundtableChainRow.stages``. ``create_all`` adds the column for brand-new DBs;
this revision patches DBs that already have the table.
Revision ID: 20260607_03
Revises: 20260607_02
Create Date: 2026-06-07
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
from sqlalchemy import inspect
from sqlalchemy.dialects.mysql import LONGTEXT
revision = "20260607_03"
down_revision = "20260607_02"
branch_labels = None
depends_on = None
def _has_table(table_name: str) -> bool:
inspector = inspect(op.get_bind())
return table_name in inspector.get_table_names()
def _has_column(table_name: str, column_name: str) -> bool:
if not _has_table(table_name):
return False
inspector = inspect(op.get_bind())
return any(col["name"] == column_name for col in inspector.get_columns(table_name))
def upgrade() -> None:
if _has_table("roundtable_chains") and not _has_column("roundtable_chains", "stages"):
col_type = LONGTEXT() if op.get_bind().dialect.name == "mysql" else sa.Text()
op.add_column("roundtable_chains", sa.Column("stages", col_type, nullable=True))
def downgrade() -> None:
if _has_column("roundtable_chains", "stages"):
op.drop_column("roundtable_chains", "stages")
offline-backend-20260512/backend/packages/harness/deerflow/persistence/migrations/versions/20260608_01_admin_leaderboard_daily_stats.py
+85
View File
@@ -0,0 +1,85 @@
"""Create admin leaderboard daily snapshots and analytics indexes.
Revision ID: 20260608_01
Revises: 20260607_02
Create Date: 2026-06-08
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
from sqlalchemy import inspect
from deerflow.persistence.types import PortableJSON
revision = "20260608_01"
down_revision = "20260607_02"
branch_labels = None
depends_on = None
def _has_table(table_name: str) -> bool:
inspector = inspect(op.get_bind())
return table_name in inspector.get_table_names()
def _has_index(table_name: str, index_name: str) -> bool:
if not _has_table(table_name):
return False
inspector = inspect(op.get_bind())
return any(idx["name"] == index_name for idx in inspector.get_indexes(table_name))
def _create_index(table_name: str, index_name: str, columns: list[str]) -> None:
if _has_table(table_name) and not _has_index(table_name, index_name):
op.create_index(index_name, table_name, columns)
def _drop_index(table_name: str, index_name: str) -> None:
if _has_table(table_name) and _has_index(table_name, index_name):
op.drop_index(index_name, table_name=table_name)
def upgrade() -> None:
if not _has_table("admin_leaderboard_daily_stats"):
op.create_table(
"admin_leaderboard_daily_stats",
sa.Column("stat_date", sa.String(10), nullable=False),
sa.Column("settings_hash", sa.String(64), nullable=False),
sa.Column("status", sa.String(16), nullable=False, server_default="ready"),
sa.Column("payload_json", PortableJSON(), nullable=True),
sa.Column("message", sa.Text(), nullable=True),
sa.Column("error", sa.Text(), nullable=True),
sa.Column("generated_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint("stat_date", "settings_hash", name="pk_admin_leaderboard_daily_stats"),
)
op.create_index("ix_admin_leaderboard_daily_stats_status", "admin_leaderboard_daily_stats", ["status"])
op.create_index("ix_admin_leaderboard_daily_stats_updated_at", "admin_leaderboard_daily_stats", ["updated_at"])
_create_index("runs", "ix_runs_created_user_status", ["created_at", "user_id", "status"])
_create_index("runs", "ix_runs_user_created", ["user_id", "created_at"])
_create_index("runs", "ix_runs_thread_created", ["thread_id", "created_at"])
_create_index("tool_call_metrics", "ix_tool_call_metrics_created_user", ["created_at", "user_id"])
_create_index("tool_call_metrics", "ix_tool_call_metrics_run_created", ["run_id", "created_at"])
_create_index("llm_call_metrics", "ix_llm_call_metrics_created_user", ["created_at", "user_id"])
_create_index("llm_call_metrics", "ix_llm_call_metrics_run_created", ["run_id", "created_at"])
_create_index("threads_meta", "ix_threads_meta_created_user", ["created_at", "user_id"])
_create_index("threads_meta", "ix_threads_meta_user_created", ["user_id", "created_at"])
def downgrade() -> None:
_drop_index("threads_meta", "ix_threads_meta_user_created")
_drop_index("threads_meta", "ix_threads_meta_created_user")
_drop_index("llm_call_metrics", "ix_llm_call_metrics_run_created")
_drop_index("llm_call_metrics", "ix_llm_call_metrics_created_user")
_drop_index("tool_call_metrics", "ix_tool_call_metrics_run_created")
_drop_index("tool_call_metrics", "ix_tool_call_metrics_created_user")
_drop_index("runs", "ix_runs_thread_created")
_drop_index("runs", "ix_runs_user_created")
_drop_index("runs", "ix_runs_created_user_status")
if _has_table("admin_leaderboard_daily_stats"):
op.drop_table("admin_leaderboard_daily_stats")
offline-backend-20260512/backend/packages/harness/deerflow/persistence/migrations/versions/20260608_02_merge_chain_stages_and_leaderboard_heads.py
+27
View File
@@ -0,0 +1,27 @@
"""Merge chain_stages and admin_leaderboard_daily_stats migration heads.
Two independent migrations diverged from ``20260607_02``: ``20260607_03``
(roundtable_chains.stages) and ``20260608_01`` (admin_leaderboard_daily_stats +
analytics indexes). This no-op merge revision joins both lineages so
``upgrade head`` succeeds.
Revision ID: 20260608_02
Revises: 20260607_03, 20260608_01
Create Date: 2026-06-08
"""
from __future__ import annotations
revision = "20260608_02"
down_revision = ("20260607_03", "20260608_01")
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
offline-backend-20260512/backend/packages/harness/deerflow/persistence/migrations/versions/20260608_03_chain_is_public.py
+58
View File
@@ -0,0 +1,58 @@
"""Add is_public column to roundtable_chains.
「发布」business chains: ``is_public=True`` makes a chain visible to **all
accounts** under the "公共" filter (read + select-into-roundtable only; edit /
delete stay owner-only). Defaults to ``0`` (private). ``create_all`` adds the
column for brand-new DBs; this revision patches DBs that already have the table.
Chained after the merge head ``20260608_02`` (chain_stages + leaderboard) so the
revision graph stays single-headed — otherwise ``alembic upgrade head`` aborts
with "multiple heads" and the auto-migrate on startup silently falls back to the
create_all schema (which lacks this column → 500 on list).
Revision ID: 20260608_03
Revises: 20260608_02
Create Date: 2026-06-08
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
from sqlalchemy import inspect
revision = "20260608_03"
down_revision = "20260608_02"
branch_labels = None
depends_on = None
def _has_table(table_name: str) -> bool:
inspector = inspect(op.get_bind())
return table_name in inspector.get_table_names()
def _has_column(table_name: str, column_name: str) -> bool:
if not _has_table(table_name):
return False
inspector = inspect(op.get_bind())
return any(col["name"] == column_name for col in inspector.get_columns(table_name))
def upgrade() -> None:
if _has_table("roundtable_chains") and not _has_column("roundtable_chains", "is_public"):
op.add_column(
"roundtable_chains",
sa.Column("is_public", sa.Boolean(), nullable=False, server_default="0"),
)
op.create_index(
"ix_roundtable_chains_is_public", "roundtable_chains", ["is_public"]
)
def downgrade() -> None:
if _has_column("roundtable_chains", "is_public"):
with op.batch_alter_table("roundtable_chains") as batch:
batch.drop_index("ix_roundtable_chains_is_public")
op.drop_column("roundtable_chains", "is_public")
offline-backend-20260512/backend/packages/harness/deerflow/persistence/migrations/versions/20260608_04_chain_stage_goals.py
+56
View File
@@ -0,0 +1,56 @@
"""Add stage_goals column to roundtable_chains.
Per-stage goals (每阶段目标) for business chains: an optional list aligned by index
to ``stages`` where ``stage_goals[i]`` is the「本阶段目标」for ``stages[i]`` — what
that serial stage should produce / how it depends on the previous stage's output.
Injected to the coordinator dispatch prompt (and seat tasks as fallback) in dag-mode
only. NULL = no goals. The column is LONGTEXT on MySQL (mirrors ``stages`` headroom)
and plain TEXT elsewhere — matching ``PortableLongText`` on
``RoundtableChainRow.stage_goals``. ``create_all`` adds the column for brand-new DBs;
this revision patches DBs that already have the table.
Chained after ``20260608_03`` (chain is_public) so the revision graph stays
single-headed — otherwise ``alembic upgrade head`` aborts with "multiple heads" and
the auto-migrate on startup silently falls back to the create_all schema (which lacks
this column → 500 on list).
Revision ID: 20260608_04
Revises: 20260608_03
Create Date: 2026-06-08
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
from sqlalchemy import inspect
from sqlalchemy.dialects.mysql import LONGTEXT
revision = "20260608_04"
down_revision = "20260608_03"
branch_labels = None
depends_on = None
def _has_table(table_name: str) -> bool:
inspector = inspect(op.get_bind())
return table_name in inspector.get_table_names()
def _has_column(table_name: str, column_name: str) -> bool:
if not _has_table(table_name):
return False
inspector = inspect(op.get_bind())
return any(col["name"] == column_name for col in inspector.get_columns(table_name))
def upgrade() -> None:
if _has_table("roundtable_chains") and not _has_column("roundtable_chains", "stage_goals"):
col_type = LONGTEXT() if op.get_bind().dialect.name == "mysql" else sa.Text()
op.add_column("roundtable_chains", sa.Column("stage_goals", col_type, nullable=True))
def downgrade() -> None:
if _has_column("roundtable_chains", "stage_goals"):
op.drop_column("roundtable_chains", "stage_goals")
offline-backend-20260512/backend/packages/harness/deerflow/persistence/migrations/versions/20260608_05_parallel_agent_tasks.py
+61
View File
@@ -0,0 +1,61 @@
"""Create parallel_agent_tasks table.
Saved runs of the standalone parallel multi-agent panel (Vue embeddable
component): task name + brief + agents snapshot + the resulting message list,
bound per user. ``brief`` / ``agents`` / ``messages`` are LONGTEXT on MySQL
(mirror ``roundtable_drafts`` headroom) and plain TEXT elsewhere — matching
``PortableLongText``. ``create_all`` builds the table for brand-new DBs; this
revision patches DBs that already exist.
Chained after ``20260608_04`` (chain stage_goals) so the revision graph stays
single-headed — otherwise ``alembic upgrade head`` aborts with "multiple heads"
and the startup auto-migrate silently falls back to the create_all schema.
Revision ID: 20260608_05
Revises: 20260608_04
Create Date: 2026-06-08
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
from sqlalchemy import inspect
from sqlalchemy.dialects.mysql import LONGTEXT
revision = "20260608_05"
down_revision = "20260608_04"
branch_labels = None
depends_on = None
def _has_table(table_name: str) -> bool:
inspector = inspect(op.get_bind())
return table_name in inspector.get_table_names()
def upgrade() -> None:
if _has_table("parallel_agent_tasks"):
return
long_text = LONGTEXT() if op.get_bind().dialect.name == "mysql" else sa.Text()
op.create_table(
"parallel_agent_tasks",
sa.Column("id", sa.String(length=64), primary_key=True, nullable=False),
sa.Column("user_id", sa.String(length=64), nullable=False),
sa.Column("title", sa.String(length=512), nullable=False, server_default=""),
sa.Column("brief", long_text, nullable=True),
sa.Column("agents", long_text, nullable=True),
sa.Column("messages", long_text, nullable=True),
sa.Column("created_at", sa.DateTime(), nullable=False),
sa.Column("updated_at", sa.DateTime(), nullable=False),
)
op.create_index(
"ix_parallel_agent_tasks_user_id", "parallel_agent_tasks", ["user_id"]
)
def downgrade() -> None:
if _has_table("parallel_agent_tasks"):
op.drop_index("ix_parallel_agent_tasks_user_id", table_name="parallel_agent_tasks")
op.drop_table("parallel_agent_tasks")
offline-backend-20260512/backend/packages/harness/deerflow/persistence/migrations/versions/20260608_06_chain_coordinator_prompt.py
+56
View File
@@ -0,0 +1,56 @@
"""Add coordinator_prompt column to roundtable_chains.
Per-chain「总控编排提示」: an optional free-text instruction the user writes for the
**coordinator (总控) agent** of a business chain — telling it how to dispatch / what to
do (overall orchestration strategy / style / constraints). Injected into the coordinator
per-stage dispatch prompt (``buildDagStageLeaderTask``) in **dag-mode only**. NULL = no
prompt. The column is LONGTEXT on MySQL (mirrors ``stage_goals`` headroom) and plain TEXT
elsewhere — matching ``PortableLongText`` on ``RoundtableChainRow.coordinator_prompt``.
``create_all`` adds the column for brand-new DBs; this revision patches DBs that already
have the table.
Chained after ``20260608_05`` (parallel_agent_tasks) so the revision graph stays
single-headed — otherwise ``alembic upgrade head`` aborts with "multiple heads" and the
startup auto-migrate silently falls back to the create_all schema (which lacks this
column → 500 on list).
Revision ID: 20260608_06
Revises: 20260608_05
Create Date: 2026-06-09
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
from sqlalchemy import inspect
from sqlalchemy.dialects.mysql import LONGTEXT
revision = "20260608_06"
down_revision = "20260608_05"
branch_labels = None
depends_on = None
def _has_table(table_name: str) -> bool:
inspector = inspect(op.get_bind())
return table_name in inspector.get_table_names()
def _has_column(table_name: str, column_name: str) -> bool:
if not _has_table(table_name):
return False
inspector = inspect(op.get_bind())
return any(col["name"] == column_name for col in inspector.get_columns(table_name))
def upgrade() -> None:
if _has_table("roundtable_chains") and not _has_column("roundtable_chains", "coordinator_prompt"):
col_type = LONGTEXT() if op.get_bind().dialect.name == "mysql" else sa.Text()
op.add_column("roundtable_chains", sa.Column("coordinator_prompt", col_type, nullable=True))
def downgrade() -> None:
if _has_column("roundtable_chains", "coordinator_prompt"):
op.drop_column("roundtable_chains", "coordinator_prompt")
offline-backend-20260512/backend/packages/harness/deerflow/persistence/models/__init__.py
+16 -1
View File
@@ -15,14 +15,22 @@ its storage implementation lives in ``deerflow.runtime.events.store.db`` and
there is no matching entity directory.
"""
from deerflow.persistence.agents.model import AgentFavoriteRow, AgentRow
from deerflow.persistence.agents.model import (
AgentExtrasRow,
AgentFavoriteRow,
AgentRow,
AgentSkillRow,
)
from deerflow.persistence.admin_stats.model import AdminLeaderboardDailyStatRow
from deerflow.persistence.memory_config.model import UserMemoryConfigRow
from deerflow.persistence.feedback.model import FeedbackRow
from deerflow.persistence.llm_metrics.model import LlmCallMetricRow
from deerflow.persistence.models.run_event import RunEventRow
from deerflow.persistence.notifications.model import NotificationRow
from deerflow.persistence.parallel_agent_tasks.model import ParallelAgentTaskRow
from deerflow.persistence.recommended_questions.model import RecommendedQuestionRow
from deerflow.persistence.roundtable_chains.model import RoundtableChainRow
from deerflow.persistence.roundtable_draft_shares.model import RoundtableDraftShareRow
from deerflow.persistence.roundtable_drafts.model import (
RoundtableDraftRow,
RoundtableRecommendHistoryRow,
@@ -43,17 +51,23 @@ from deerflow.persistence.embed_sessions.model import EmbedSessionRow
from deerflow.persistence.thread_shares.model import ThreadShareRow
from deerflow.persistence.tool_metrics.model import ToolCallMetricRow
from deerflow.persistence.user.model import UserRow
from deerflow.persistence.user_preferences.model import UserPreferencesRow
from deerflow.persistence.user_prompts.model import UserCustomPromptRow
__all__ = [
"FeedbackRow",
"AgentExtrasRow",
"AgentFavoriteRow",
"AgentRow",
"AgentSkillRow",
"AdminLeaderboardDailyStatRow",
"LlmCallMetricRow",
"NotificationRow",
"ParallelAgentTaskRow",
"RecommendedQuestionRow",
"RoundtableChainRow",
"RoundtableDraftRow",
"RoundtableDraftShareRow",
"RoundtableJobRow",
"RoundtableRecommendHistoryRow",
"RunEventRow",
@@ -73,5 +87,6 @@ __all__ = [
"ToolCallMetricRow",
"UserCustomPromptRow",
"UserMemoryConfigRow",
"UserPreferencesRow",
"UserRow",
]
offline-backend-20260512/backend/packages/harness/deerflow/persistence/parallel_agent_tasks/__init__.py
+18
View File
@@ -0,0 +1,18 @@
"""Parallel multi-agent task persistence."""
from __future__ import annotations
from deerflow.persistence.parallel_agent_tasks.model import ParallelAgentTaskRow
from deerflow.persistence.parallel_agent_tasks.sql import ParallelAgentTaskRepository
__all__ = [
"ParallelAgentTaskRepository",
"ParallelAgentTaskRow",
"make_parallel_agent_task_store",
]
def make_parallel_agent_task_store(session_factory):
if session_factory is None:
return None
return ParallelAgentTaskRepository(session_factory)
offline-backend-20260512/backend/packages/harness/deerflow/persistence/parallel_agent_tasks/model.py
+46
View File
@@ -0,0 +1,46 @@
"""ORM model for parallel multi-agent task records.
A "parallel-agent task" is one saved run of the standalone *parallel multi-agent*
panel (the Vue3 embeddable component): a task name + brief dispatched to several
agents in parallel, plus the resulting chat/message list. Persisted per user so it
follows the account across devices/sessions — same model as ``roundtable_drafts``.
``agents`` and ``messages`` are stored as JSON serialized into ``PortableLongText``
(MySQL ``LONGTEXT``) rather than ``PortableJSON`` (MySQL ``TEXT``, 64 KB cap),
because the parallel transcript can carry several agents' full deliveries that
easily exceed 64 KB — mirroring ``roundtable_drafts``.
"""
from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy import String
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
from deerflow.persistence.types import BeijingDateTime, PortableLongText
class ParallelAgentTaskRow(Base):
__tablename__ = "parallel_agent_tasks"
id: Mapped[str] = mapped_column(String(64), primary_key=True)
user_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
# Task name (用户输入的任务名称) shown on the history list.
title: Mapped[str] = mapped_column(String(512), nullable=False, default="")
# Task brief / 简介 — the free-text task description.
brief: Mapped[str | None] = mapped_column(PortableLongText, nullable=True)
# JSON: snapshot of the agents dispatched, [{"agent_id", "name", "avatarType"?}].
agents: Mapped[str | None] = mapped_column(PortableLongText, nullable=True)
# JSON: the full message/dialogue list (mirrors the frontend Dialogue[] shape).
messages: Mapped[str | None] = mapped_column(PortableLongText, nullable=True)
created_at: Mapped[datetime] = mapped_column(
BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC)
)
updated_at: Mapped[datetime] = mapped_column(
BeijingDateTime(),
nullable=False,
default=lambda: datetime.now(UTC),
onupdate=lambda: datetime.now(UTC),
)
offline-backend-20260512/backend/packages/harness/deerflow/persistence/parallel_agent_tasks/sql.py
+131
View File
@@ -0,0 +1,131 @@
"""SQLAlchemy-backed store for parallel multi-agent task records.
``agents`` / ``messages`` JSON columns are stored as serialized strings in
``PortableLongText`` and round-tripped here with ``json.dumps`` / ``json.loads``
— same approach as ``roundtable_drafts`` / ``roundtable_chains``.
All write/read paths are scoped by ``user_id`` for ownership isolation; a row
that belongs to another user is treated as not found.
"""
from __future__ import annotations
import json
from datetime import UTC, datetime
from typing import Any
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from deerflow.persistence.parallel_agent_tasks.model import ParallelAgentTaskRow
_UNSET = object()
def _loads(value: Any) -> Any:
if not isinstance(value, str):
return value
if not value:
return None
try:
return json.loads(value)
except Exception:
return None
def _dumps(value: Any) -> str | None:
if value is None:
return None
return json.dumps(value, ensure_ascii=False)
class ParallelAgentTaskRepository:
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
self._sf = session_factory
@staticmethod
def _to_dict(row: ParallelAgentTaskRow, *, with_messages: bool = True) -> dict[str, Any]:
data: dict[str, Any] = {
"id": row.id,
"title": row.title,
"brief": row.brief,
"agents": _loads(row.agents) or [],
"created_at": row.created_at.isoformat() if isinstance(row.created_at, datetime) else row.created_at,
"updated_at": row.updated_at.isoformat() if isinstance(row.updated_at, datetime) else row.updated_at,
}
if with_messages:
data["messages"] = _loads(row.messages) or []
return data
async def list_tasks(self, user_id: str, *, limit: int = 200) -> list[dict[str, Any]]:
"""列出当前用户的任务（轻量元信息，不含 messages，供历史列表）。"""
stmt = (
select(ParallelAgentTaskRow)
.where(ParallelAgentTaskRow.user_id == user_id)
.order_by(ParallelAgentTaskRow.updated_at.desc())
.limit(limit)
)
async with self._sf() as session:
result = await session.execute(stmt)
return [self._to_dict(row, with_messages=False) for row in result.scalars()]
async def get_task(self, task_id: str, user_id: str) -> dict[str, Any] | None:
async with self._sf() as session:
row = await session.get(ParallelAgentTaskRow, task_id)
if row is None or row.user_id != user_id:
return None
return self._to_dict(row)
async def create_task(self, user_id: str, data: dict[str, Any]) -> dict[str, Any]:
now = datetime.now(UTC)
row = ParallelAgentTaskRow(
id=data["id"],
user_id=user_id,
title=(data.get("title") or "").strip(),
brief=(data.get("brief") or None),
agents=_dumps(data.get("agents") or []),
messages=_dumps(data.get("messages") or []),
created_at=now,
updated_at=now,
)
async with self._sf() as session:
session.add(row)
await session.commit()
await session.refresh(row)
return self._to_dict(row)
async def update_task(
self,
task_id: str,
user_id: str,
*,
title=_UNSET,
brief=_UNSET,
agents=_UNSET,
messages=_UNSET,
) -> dict[str, Any] | None:
async with self._sf() as session:
row = await session.get(ParallelAgentTaskRow, task_id)
if row is None or row.user_id != user_id:
return None
if title is not _UNSET and title is not None:
row.title = str(title).strip()
if brief is not _UNSET:
row.brief = brief or None
if agents is not _UNSET:
row.agents = _dumps(agents)
if messages is not _UNSET:
row.messages = _dumps(messages)
row.updated_at = datetime.now(UTC)
await session.commit()
await session.refresh(row)
return self._to_dict(row)
async def delete_task(self, task_id: str, user_id: str) -> bool:
async with self._sf() as session:
row = await session.get(ParallelAgentTaskRow, task_id)
if row is None or row.user_id != user_id:
return False
await session.delete(row)
await session.commit()
return True
offline-backend-20260512/backend/packages/harness/deerflow/persistence/roundtable_chains/model.py
+22 -1
View File
@@ -14,7 +14,7 @@ from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy import String
from sqlalchemy import Boolean, String
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
@@ -33,6 +33,27 @@ class RoundtableChainRow(Base):
# JSON: ordered [{"agent_id": str, "name": str, "description": str}, ...].
# seats[0] is the first seat to speak. Stored as LONGTEXT-backed JSON string.
seats: Mapped[str | None] = mapped_column(PortableLongText, nullable=True)
# Optional JSON: layered orchestration grouping over ``seats`` — a list of
# stages, each a list of agent_ids run in parallel; stages run serially in
# order. e.g. [["a","b"],["c"]] = (a‖b) → c. NULL/absent = linear chain
# (each seat its own stage). Drives Step 2 dag-mode orchestration; see
# frontend ``lib/dag-types.ts``. Stored as LONGTEXT-backed JSON string.
stages: Mapped[str | None] = mapped_column(PortableLongText, nullable=True)
# Optional JSON: per-stage goals aligned by index to ``stages`` — a list where
# ``stage_goals[i]`` is the「本阶段目标」for ``stages[i]`` (what that serial stage
# should produce / how it depends on the previous stage's output). Injected to the
# coordinator dispatch prompt (and seat tasks as fallback) in dag-mode only. NULL =
# no goals (and always NULL when ``stages`` is NULL/linear). Stored as JSON string.
stage_goals: Mapped[str | None] = mapped_column(PortableLongText, nullable=True)
# 可选「总控编排提示」：用户为本链条写给**总控（协调）智能体**的一段指引，告诉总控该
# 怎么派活、做什么（整体编排策略 / 风格 / 约束）。**仅分层 DAG 模式生效**：注入到总控
# 每阶段派活提示（``buildDagStageLeaderTask``）。NULL = 不注入。纯文本，存为 LONGTEXT。
coordinator_prompt: Mapped[str | None] = mapped_column(PortableLongText, nullable=True)
# 发布开关：True = 公共（所有账号可在「公共」筛选里查看 / 选用）；False = 仅本人可见。
# 编辑 / 删除始终只限 owner；发布只是放开「查看 + 选入研讨」。索引以支持「公共」列表筛选。
is_public: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=False, server_default="0", index=True
)
created_at: Mapped[datetime] = mapped_column(
BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC)
)
offline-backend-20260512/backend/packages/harness/deerflow/persistence/roundtable_chains/sql.py
+49 -8
View File
@@ -44,33 +44,58 @@ class RoundtableChainRepository:
self._sf = session_factory
@staticmethod
def _to_dict(row: RoundtableChainRow) -> dict[str, Any]:
def _to_dict(row: RoundtableChainRow, current_user_id: str | None = None) -> dict[str, Any]:
return {
"id": row.id,
"title": row.title,
"description": row.description,
"seats": _loads(row.seats) or [],
# None when absent → frontend reads it as "linear chain" (no layering).
"stages": _loads(row.stages),
# Per-stage goals aligned to ``stages``; None when absent.
"stage_goals": _loads(getattr(row, "stage_goals", None)),
# 总控编排提示（写给总控的派活指引，dag-only）；None/空 = 不注入。纯文本,不 json。
"coordinator_prompt": getattr(row, "coordinator_prompt", None),
# 发布状态 + 是否本人所有（前端据此显示「公共」徽标、决定能否编辑 / 发布）。
"is_public": bool(getattr(row, "is_public", False)),
"is_mine": (current_user_id is None) or (row.user_id == current_user_id),
"created_at": row.created_at.isoformat() if isinstance(row.created_at, datetime) else row.created_at,
"updated_at": row.updated_at.isoformat() if isinstance(row.updated_at, datetime) else row.updated_at,
}
async def list_chains(self, user_id: str, *, limit: int = 200) -> list[dict[str, Any]]:
async def list_chains(
self, user_id: str, *, scope: str = "mine", limit: int = 200
) -> list[dict[str, Any]]:
"""列出链条。
``scope``:
- ``"mine"``（默认）：仅当前用户自己的链条；
- ``"public"``：所有**已发布**（``is_public=True``）链条，含本人发布的。
其余值按 ``"mine"`` 处理。``is_mine`` 标记每条是否本人所有，供前端判断可否编辑。
"""
if scope == "public":
cond = RoundtableChainRow.is_public.is_(True)
else:
cond = RoundtableChainRow.user_id == user_id
stmt = (
select(RoundtableChainRow)
.where(RoundtableChainRow.user_id == user_id)
.where(cond)
.order_by(RoundtableChainRow.updated_at.desc())
.limit(limit)
)
async with self._sf() as session:
result = await session.execute(stmt)
return [self._to_dict(row) for row in result.scalars()]
return [self._to_dict(row, user_id) for row in result.scalars()]
async def get_chain(self, chain_id: str, user_id: str) -> dict[str, Any] | None:
"""取一条链条：owner 可见,**已发布的公共链条任何人可见**(供选入研讨)。"""
async with self._sf() as session:
row = await session.get(RoundtableChainRow, chain_id)
if row is None or row.user_id != user_id:
if row is None:
return None
return self._to_dict(row)
if row.user_id != user_id and not bool(getattr(row, "is_public", False)):
return None
return self._to_dict(row, user_id)
async def create_chain(self, user_id: str, data: dict[str, Any]) -> dict[str, Any]:
now = datetime.now(UTC)
@@ -80,6 +105,10 @@ class RoundtableChainRepository:
title=(data.get("title") or "").strip(),
description=(data.get("description") or None),
seats=_dumps(data.get("seats") or []),
stages=_dumps(data.get("stages")),
stage_goals=_dumps(data.get("stage_goals")),
coordinator_prompt=(data.get("coordinator_prompt") or None),
is_public=bool(data.get("is_public") or False),
created_at=now,
updated_at=now,
)
@@ -87,7 +116,7 @@ class RoundtableChainRepository:
session.add(row)
await session.commit()
await session.refresh(row)
return self._to_dict(row)
return self._to_dict(row, user_id)
async def update_chain(
self,
@@ -97,6 +126,10 @@ class RoundtableChainRepository:
title=_UNSET,
description=_UNSET,
seats=_UNSET,
stages=_UNSET,
stage_goals=_UNSET,
coordinator_prompt=_UNSET,
is_public=_UNSET,
) -> dict[str, Any] | None:
async with self._sf() as session:
row = await session.get(RoundtableChainRow, chain_id)
@@ -108,10 +141,18 @@ class RoundtableChainRepository:
row.description = description or None
if seats is not _UNSET:
row.seats = _dumps(seats)
if stages is not _UNSET:
row.stages = _dumps(stages)
if stage_goals is not _UNSET:
row.stage_goals = _dumps(stage_goals)
if coordinator_prompt is not _UNSET:
row.coordinator_prompt = coordinator_prompt or None
if is_public is not _UNSET and is_public is not None:
row.is_public = bool(is_public)
row.updated_at = datetime.now(UTC)
await session.commit()
await session.refresh(row)
return self._to_dict(row)
return self._to_dict(row, user_id)
async def delete_chain(self, chain_id: str, user_id: str) -> bool:
async with self._sf() as session:
offline-backend-20260512/backend/packages/harness/deerflow/persistence/roundtable_draft_shares/__init__.py
+22
View File
@@ -0,0 +1,22 @@
"""Persistence layer for roundtable-draft share links."""
from deerflow.persistence.roundtable_draft_shares.base import (
RoundtableDraftShareStore,
generate_share_code,
)
from deerflow.persistence.roundtable_draft_shares.memory import MemoryRoundtableDraftShareStore
from deerflow.persistence.roundtable_draft_shares.sql import RoundtableDraftShareRepository
__all__ = [
"RoundtableDraftShareStore",
"MemoryRoundtableDraftShareStore",
"RoundtableDraftShareRepository",
"generate_share_code",
"make_roundtable_draft_share_store",
]
def make_roundtable_draft_share_store(session_factory):
if session_factory is None:
return MemoryRoundtableDraftShareStore()
return RoundtableDraftShareRepository(session_factory)
offline-backend-20260512/backend/packages/harness/deerflow/persistence/roundtable_draft_shares/base.py
+56
View File
@@ -0,0 +1,56 @@
"""Abstract persistence API for roundtable-draft share links.
Mirrors ``thread_shares`` but shares one *roundtable-planning draft* (a full
会商 session: Step 1 intent + Step 2 roundtable + optional Step 3 report)
instead of a conversation thread. Redeeming an ``import`` code copies the draft
into the recipient's account; a ``view`` code is a public read-only page that
renders the draft's Step 3 result report.
The share-code alphabet / generator is reused from ``thread_shares.base`` so
codes are interchangeable in shape (unambiguous, human-readable).
"""
from __future__ import annotations
from abc import ABC, abstractmethod
from typing import Any
# Reuse the same unambiguous-alphabet code generator as conversation shares.
from deerflow.persistence.thread_shares.base import generate_share_code
__all__ = ["RoundtableDraftShareStore", "generate_share_code"]
class RoundtableDraftShareStore(ABC):
@abstractmethod
async def create(self, owner_user_id: str, draft_id: str, *, mode: str = "import") -> dict[str, Any]:
"""Create a new share link for one draft and return its record.
``mode`` is ``"import"`` (recipient copies the draft) or ``"view"``
(a public read-only result-report page).
"""
raise NotImplementedError
@abstractmethod
async def get(self, share_code: str) -> dict[str, Any] | None:
"""Look up a share by code regardless of owner (None if missing)."""
raise NotImplementedError
@abstractmethod
async def get_active_for_draft(self, owner_user_id: str, draft_id: str, *, mode: str = "import") -> dict[str, Any] | None:
"""Return the owner's non-revoked share of ``mode`` for ``draft_id``, if any.
Used so re-sharing the same draft in the same mode reuses its existing
code instead of minting a new one each time.
"""
raise NotImplementedError
@abstractmethod
async def list_for_owner(self, owner_user_id: str) -> list[dict[str, Any]]:
"""List all share links created by ``owner_user_id`` (newest first)."""
raise NotImplementedError
@abstractmethod
async def revoke(self, share_code: str, owner_user_id: str) -> bool:
"""Revoke a share link. Returns False if missing or not owned."""
raise NotImplementedError
offline-backend-20260512/backend/packages/harness/deerflow/persistence/roundtable_draft_shares/memory.py
+59
View File
@@ -0,0 +1,59 @@
"""In-memory ``RoundtableDraftShareStore`` for tests / memory backend."""
from __future__ import annotations
import copy
from datetime import UTC, datetime
from typing import Any
from deerflow.persistence.roundtable_draft_shares.base import RoundtableDraftShareStore, generate_share_code
class MemoryRoundtableDraftShareStore(RoundtableDraftShareStore):
def __init__(self) -> None:
self._shares: dict[str, dict[str, Any]] = {}
async def create(self, owner_user_id: str, draft_id: str, *, mode: str = "import") -> dict[str, Any]:
code = generate_share_code()
while code in self._shares:
code = generate_share_code()
record = {
"share_code": code,
"owner_user_id": owner_user_id,
"draft_id": draft_id,
"revoked": False,
"mode": mode,
"created_at": datetime.now(UTC).isoformat(),
}
self._shares[code] = record
return copy.deepcopy(record)
async def get(self, share_code: str) -> dict[str, Any] | None:
record = self._shares.get(share_code)
return copy.deepcopy(record) if record is not None else None
async def get_active_for_draft(self, owner_user_id: str, draft_id: str, *, mode: str = "import") -> dict[str, Any] | None:
matches = [
r
for r in self._shares.values()
if r["owner_user_id"] == owner_user_id
and r["draft_id"] == draft_id
and r.get("mode", "import") == mode
and not r["revoked"]
]
if not matches:
return None
matches.sort(key=lambda r: r["created_at"], reverse=True)
return copy.deepcopy(matches[0])
async def list_for_owner(self, owner_user_id: str) -> list[dict[str, Any]]:
rows = [r for r in self._shares.values() if r["owner_user_id"] == owner_user_id]
rows.sort(key=lambda r: r["created_at"], reverse=True)
return [copy.deepcopy(r) for r in rows]
async def revoke(self, share_code: str, owner_user_id: str) -> bool:
record = self._shares.get(share_code)
if record is None or record["owner_user_id"] != owner_user_id:
return False
record["revoked"] = True
return True
offline-backend-20260512/backend/packages/harness/deerflow/persistence/roundtable_draft_shares/model.py
+38
View File
@@ -0,0 +1,38 @@
"""ORM model for roundtable-draft share links.
A share link is a short, revocable code created by a user for *one* of their
roundtable-planning drafts. Redeeming it copies that draft into the redeemer's
account (``import``) or exposes its Step 3 result report on a public read-only
page (``view``). The table only stores the link — the draft snapshot is read
lazily at import / view time, so a revoked link instantly stops working.
"""
from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy import Boolean, String
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
from deerflow.persistence.types import BeijingDateTime
class RoundtableDraftShareRow(Base):
__tablename__ = "roundtable_draft_shares"
# Short, URL-safe, unguessable code embedded in the share link.
share_code: Mapped[str] = mapped_column(String(32), primary_key=True)
# The user who created the share.
owner_user_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
# The single roundtable draft this link shares.
draft_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
# Soft-disable: a revoked link can no longer be previewed, imported, or viewed.
revoked: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
# Share mode:
# "import" — recipient (a logged-in user) copies the draft into their own
# account and can keep working on the copy.
# "view" — a public, read-only page anyone can open without logging in;
# it renders the draft's Step 3 result report (HTML).
mode: Mapped[str] = mapped_column(String(16), nullable=False, default="import", server_default="import")
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC))
offline-backend-20260512/backend/packages/harness/deerflow/persistence/roundtable_draft_shares/sql.py
+84
View File
@@ -0,0 +1,84 @@
"""SQLAlchemy-backed roundtable-draft share-link repository."""
from __future__ import annotations
from datetime import datetime
from typing import Any
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from deerflow.persistence.roundtable_draft_shares.base import RoundtableDraftShareStore, generate_share_code
from deerflow.persistence.roundtable_draft_shares.model import RoundtableDraftShareRow
def _row_to_dict(row: RoundtableDraftShareRow) -> dict[str, Any]:
data = row.to_dict()
created = data.get("created_at")
if isinstance(created, datetime):
data["created_at"] = created.isoformat()
return data
class RoundtableDraftShareRepository(RoundtableDraftShareStore):
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
self._sf = session_factory
async def create(self, owner_user_id: str, draft_id: str, *, mode: str = "import") -> dict[str, Any]:
# Retry on the (astronomically unlikely) code collision.
async with self._sf() as session:
for _ in range(8):
code = generate_share_code()
if await session.get(RoundtableDraftShareRow, code) is not None:
continue
row = RoundtableDraftShareRow(
share_code=code,
owner_user_id=owner_user_id,
draft_id=draft_id,
revoked=False,
mode=mode,
)
session.add(row)
await session.commit()
await session.refresh(row)
return _row_to_dict(row)
raise RuntimeError("Failed to allocate a unique share code")
async def get(self, share_code: str) -> dict[str, Any] | None:
async with self._sf() as session:
row = await session.get(RoundtableDraftShareRow, share_code)
return _row_to_dict(row) if row is not None else None
async def get_active_for_draft(self, owner_user_id: str, draft_id: str, *, mode: str = "import") -> dict[str, Any] | None:
stmt = (
select(RoundtableDraftShareRow)
.where(
RoundtableDraftShareRow.owner_user_id == owner_user_id,
RoundtableDraftShareRow.draft_id == draft_id,
RoundtableDraftShareRow.mode == mode,
RoundtableDraftShareRow.revoked.is_(False),
)
.order_by(RoundtableDraftShareRow.created_at.desc())
)
async with self._sf() as session:
row = (await session.execute(stmt)).scalars().first()
return _row_to_dict(row) if row is not None else None
async def list_for_owner(self, owner_user_id: str) -> list[dict[str, Any]]:
stmt = (
select(RoundtableDraftShareRow)
.where(RoundtableDraftShareRow.owner_user_id == owner_user_id)
.order_by(RoundtableDraftShareRow.created_at.desc())
)
async with self._sf() as session:
result = await session.execute(stmt)
return [_row_to_dict(row) for row in result.scalars()]
async def revoke(self, share_code: str, owner_user_id: str) -> bool:
async with self._sf() as session:
row = await session.get(RoundtableDraftShareRow, share_code)
if row is None or row.owner_user_id != owner_user_id:
return False
row.revoked = True
await session.commit()
return True
offline-backend-20260512/backend/packages/harness/deerflow/persistence/roundtable_jobs/model.py
+6
View File
@@ -55,6 +55,12 @@ class RoundtableJobRow(Base):
orchestration_mode: Mapped[str] = mapped_column(String(16), nullable=False, default="recommend")
# chain 模式来源链条快照：JSON object { id, title }（recommend 模式为 null）.
chain: Mapped[str | None] = mapped_column(PortableLongText, nullable=True)
# 研讨对话记录：JSON array of { id, role('leader'|'seat'), agentId, name, content, cycle }。
# 让用户跑完 / 进行中都能看到真实研讨内容（leader 说明 + 各席位交付），而非只有流程图。
dialogues: Mapped[str | None] = mapped_column(PortableLongText, nullable=True)
# Step3 报告快照：JSON object { html, generatedAt, model, summary }。done 时写入，
# 使前端从单个 job 即可拿到完整结果（对话 + 报告）。
step3: Mapped[str | None] = mapped_column(PortableLongText, nullable=True)
# Terminal error detail when status=error.
error: Mapped[str | None] = mapped_column(Text, nullable=True)
# 时间列必须用 BeijingDateTime（项目规范，与 ai_writing_sessions / roundtable_drafts 对齐）。
offline-backend-20260512/backend/packages/harness/deerflow/persistence/roundtable_jobs/sql.py
+7 -1
View File
@@ -55,7 +55,7 @@ class RoundtableJobRepository:
for k in ("created_at", "updated_at"):
if isinstance(d.get(k), datetime):
d[k] = d[k].isoformat()
for k in ("dispatch_chain", "thread_ids", "chain"):
for k in ("dispatch_chain", "thread_ids", "chain", "dialogues", "step3"):
d[k] = _loads(d.get(k))
return d
@@ -118,6 +118,8 @@ class RoundtableJobRepository:
pending_clarification=_UNSET,
thread_ids=_UNSET,
coordinator_name=_UNSET,
dialogues=_UNSET,
step3=_UNSET,
error=_UNSET,
) -> dict[str, Any] | None:
resolved_user_id = resolve_user_id(user_id, method_name="RoundtableJobRepository.update_progress")
@@ -145,6 +147,10 @@ class RoundtableJobRepository:
row.thread_ids = _dumps(thread_ids)
if coordinator_name is not _UNSET:
row.coordinator_name = coordinator_name
if dialogues is not _UNSET:
row.dialogues = _dumps(dialogues)
if step3 is not _UNSET:
row.step3 = _dumps(step3)
if error is not _UNSET:
row.error = error
row.updated_at = datetime.now(UTC)
offline-backend-20260512/backend/packages/harness/deerflow/persistence/run/model.py
+7 -2
View File
@@ -4,7 +4,7 @@ from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy import Index, String, Text
from sqlalchemy import Index, String, Text
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
@@ -47,4 +47,9 @@ class RunRow(Base):
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), default=lambda: datetime.now(UTC))
updated_at: Mapped[datetime] = mapped_column(BeijingDateTime(), default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC))
__table_args__ = (Index("ix_runs_thread_status", "thread_id", "status"),)
__table_args__ = (
Index("ix_runs_thread_status", "thread_id", "status"),
Index("ix_runs_created_user_status", "created_at", "user_id", "status"),
Index("ix_runs_user_created", "user_id", "created_at"),
Index("ix_runs_thread_created", "thread_id", "created_at"),
)
offline-backend-20260512/backend/packages/harness/deerflow/persistence/run/sql.py
+15
View File
@@ -205,6 +205,21 @@ class RunRepository(RunStore):
await session.execute(update(RunRow).where(RunRow.run_id == run_id).values(**values))
await session.commit()
async def count_by_user(self) -> dict[str, int]:
"""Return ``{user_id: run_count}`` via a single GROUP BY query.
Each run is one user question, so this is the source for the admin
user center's 问答次数 column. Runs without a ``user_id`` are skipped.
"""
stmt = (
select(RunRow.user_id, func.count().label("cnt"))
.where(RunRow.user_id.is_not(None))
.group_by(RunRow.user_id)
)
async with self._sf() as session:
rows = (await session.execute(stmt)).all()
return {r.user_id: r.cnt for r in rows}
async def aggregate_tokens_by_thread(self, thread_id: str) -> dict[str, Any]:
"""Aggregate token usage via a single SQL GROUP BY query."""
_completed = RunRow.status.in_(("success", "error"))
offline-backend-20260512/backend/packages/harness/deerflow/persistence/thread_meta/model.py
+7 -3
View File
@@ -4,15 +4,19 @@ from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy import String
from sqlalchemy import Index, String
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
from deerflow.persistence.types import BeijingDateTime, PortableJSON
class ThreadMetaRow(Base):
__tablename__ = "threads_meta"
class ThreadMetaRow(Base):
__tablename__ = "threads_meta"
__table_args__ = (
Index("ix_threads_meta_created_user", "created_at", "user_id"),
Index("ix_threads_meta_user_created", "user_id", "created_at"),
)
thread_id: Mapped[str] = mapped_column(String(64), primary_key=True)
assistant_id: Mapped[str | None] = mapped_column(String(128), index=True)
offline-backend-20260512/backend/packages/harness/deerflow/persistence/tool_metrics/model.py
+5 -1
View File
@@ -4,7 +4,7 @@ from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy import Integer, String, Text
from sqlalchemy import Index, Integer, String, Text
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
@@ -21,6 +21,10 @@ class ToolCallMetricRow(Base):
"""
__tablename__ = "tool_call_metrics"
__table_args__ = (
Index("ix_tool_call_metrics_created_user", "created_at", "user_id"),
Index("ix_tool_call_metrics_run_created", "run_id", "created_at"),
)
id: Mapped[str] = mapped_column(String(64), primary_key=True)
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), default=lambda: datetime.now(UTC), index=True)
offline-backend-20260512/backend/packages/harness/deerflow/persistence/user_preferences/__init__.py
+18
View File
@@ -0,0 +1,18 @@
"""Per-user UI preferences persistence."""
from __future__ import annotations
from deerflow.persistence.user_preferences.model import UserPreferencesRow
from deerflow.persistence.user_preferences.sql import UserPreferencesRepository
__all__ = [
"UserPreferencesRepository",
"UserPreferencesRow",
"make_user_preferences_store",
]
def make_user_preferences_store(session_factory):
if session_factory is None:
return None
return UserPreferencesRepository(session_factory)
offline-backend-20260512/backend/packages/harness/deerflow/persistence/user_preferences/model.py
+26
View File
@@ -0,0 +1,26 @@
"""ORM model for per-user UI preferences.
A single key-value (JSON) row per user, holding small UI-level toggles that
should follow the account across devices — e.g. whether to hide the
admin-configured prompt prefix in the chat input. Kept generic (one JSON blob)
so new toggles do not each require an ALTER TABLE.
"""
from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy import String
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
from deerflow.persistence.types import BeijingDateTime, PortableJSON
class UserPreferencesRow(Base):
__tablename__ = "user_preferences"
user_id: Mapped[str] = mapped_column(String(64), primary_key=True)
preferences: Mapped[dict] = mapped_column(PortableJSON(), nullable=False, default=dict)
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC))
updated_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC))
offline-backend-20260512/backend/packages/harness/deerflow/persistence/user_preferences/sql.py
+41
View File
@@ -0,0 +1,41 @@
"""SQLAlchemy-backed per-user preferences repository."""
from __future__ import annotations
from datetime import UTC, datetime
from typing import Any
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from deerflow.persistence.user_preferences.model import UserPreferencesRow
class UserPreferencesRepository:
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
self._sf = session_factory
async def get_for_user(self, user_id: str) -> dict[str, Any]:
"""Return the user's stored preferences (empty dict when none)."""
async with self._sf() as session:
row = await session.get(UserPreferencesRow, user_id)
if row is None or not isinstance(row.preferences, dict):
return {}
return dict(row.preferences)
async def merge_for_user(self, user_id: str, patch: dict[str, Any]) -> dict[str, Any]:
"""Shallow-merge ``patch`` into the user's preferences and return the result."""
now = datetime.now(UTC)
async with self._sf() as session:
row = await session.get(UserPreferencesRow, user_id)
if row is None:
merged = dict(patch)
row = UserPreferencesRow(user_id=user_id, preferences=merged, created_at=now, updated_at=now)
session.add(row)
else:
merged = dict(row.preferences or {})
merged.update(patch)
# Reassign so SQLAlchemy detects the change on the JSON column.
row.preferences = merged
row.updated_at = now
await session.commit()
return dict(merged)
offline-backend-20260512/backend/packages/harness/deerflow/persistence/user_prompts/model.py
+3
View File
@@ -19,5 +19,8 @@ class UserCustomPromptRow(Base):
title: Mapped[str] = mapped_column(String(255), nullable=False, default="")
content: Mapped[str] = mapped_column(Text, nullable=False)
sort_order: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
# Optional association to a single agent. NULL = global (shown everywhere);
# otherwise the prompt only surfaces when that agent is selected.
agent_id: Mapped[str | None] = mapped_column(String(64), nullable=True, index=True)
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC))
updated_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC))
offline-backend-20260512/backend/packages/harness/deerflow/persistence/user_prompts/sql.py
+5
View File
@@ -42,6 +42,7 @@ class UserCustomPromptRepository:
title=data.get("title") or "",
content=data["content"],
sort_order=data.get("sort_order", 0),
agent_id=(data.get("agent_id") or None),
created_at=now,
updated_at=now,
)
@@ -59,6 +60,10 @@ class UserCustomPromptRepository:
for key in ("title", "content", "sort_order"):
if key in data:
setattr(row, key, data[key])
# ``agent_id`` is explicitly nullable: an empty/None value clears the
# association (back to global), so accept None as a real update.
if "agent_id" in data:
row.agent_id = data["agent_id"] or None
row.updated_at = datetime.now(UTC)
await session.commit()
await session.refresh(row)
offline-backend-20260512/backend/packages/harness/deerflow/runtime/checkpointer/async_provider.py
+158 -23
View File
@@ -21,6 +21,8 @@ import asyncio
import contextlib
import logging
from collections.abc import AsyncIterator
import os
from pathlib import Path
from langgraph.types import Checkpointer
@@ -34,6 +36,110 @@ from deerflow.runtime.store._sqlite_utils import ensure_sqlite_parent_dir, resol
logger = logging.getLogger(__name__)
def _csv_env(name: str) -> list[str]:
raw = os.getenv(name, "")
return [part.strip() for part in raw.split(",") if part.strip()]
def _dual_read_enabled(config) -> bool:
if getattr(config, "dual_read_enabled", False):
return True
return os.getenv("DEER_FLOW_CHECKPOINT_DUAL_READ", "").strip().lower() in {"1", "true", "yes", "on"}
def _use_postgres_enabled(config) -> bool:
raw = os.getenv("DEER_FLOW_USE_PG_CHECKPOINTER")
if raw:
return raw.strip().lower() not in {"0", "false", "no", "off"}
return bool(getattr(config, "use_postgres", True))
def _dual_read_paths(config) -> list[str]:
paths = list(getattr(config, "dual_read_sqlite_paths", []) or [])
paths.extend(_csv_env("DEER_FLOW_CHECKPOINT_SQLITE_PATHS"))
resolved: list[str] = []
seen: set[str] = set()
for path in paths:
conn_str = resolve_sqlite_conn_str(path)
if conn_str == ":memory:" or conn_str.startswith("file:"):
logger.warning("Skipping unsupported checkpoint migration SQLite path: %s", path)
continue
if conn_str not in seen:
resolved.append(conn_str)
seen.add(conn_str)
return resolved
def _sqlite_conn_str_for_postgres_fallback(config) -> str:
configured = getattr(config, "sqlite_connection_string", None)
if configured:
return configured
paths = _dual_read_paths(config)
return paths[0] if paths else "store.db"
@contextlib.asynccontextmanager
async def _pg_pool_saver(conn_str: str) -> AsyncIterator[Checkpointer]:
"""Pool-backed ``AsyncPostgresSaver``, resilient to idle-closed connections.
``AsyncPostgresSaver.from_conn_string`` opens **one** long-lived connection.
对远程 Postgres(本项目 47.88.25.99 这类),空闲几分钟后连接会被服务端 / NAT /
连接池中间件断开,而 psycopg **不会自动重连** —— 下一次 checkpoint 操作就抛
``OperationalError: the connection is closed``,整个 run 直接失败(实测:圆桌第二步
完成后闲置 ~6 分钟,用户再人工干预 → leader run 读 checkpoint 时撞死连接 → 前端
"(无文本输出)")。
用带健康检查的连接池根治:``check=AsyncConnectionPool.check_connection`` 在把连接交给
调用方前先验活,坏连接直接丢弃重建;``max_idle`` 主动回收久闲连接,赶在远程 idle 超时
之前。这样上层永远拿不到失效连接。``prepare_threshold=0`` 关预备语句,兼容 pgbouncer
一类连接池(8432 非标准端口很可能就是)。
"""
try:
from langgraph.checkpoint.postgres.aio import AsyncPostgresSaver
except ImportError as exc:
raise ImportError(POSTGRES_INSTALL) from exc
from psycopg_pool import AsyncConnectionPool
try:
max_size = int(os.getenv("DEER_FLOW_PG_POOL_MAX", "20"))
except ValueError:
max_size = 20
pool = AsyncConnectionPool(
conninfo=conn_str,
min_size=1,
max_size=max_size,
open=False,
kwargs={"autocommit": True, "prepare_threshold": 0},
check=AsyncConnectionPool.check_connection,
max_idle=120.0,
max_lifetime=1800.0,
)
await pool.open(wait=True)
try:
saver = AsyncPostgresSaver(pool)
await saver.setup()
logger.info("Checkpointer: Postgres connection pool ready (max_size=%d, health-checked)", max_size)
yield saver
finally:
await pool.close()
@contextlib.asynccontextmanager
async def _sqlite_checkpointer_from_conn_str(raw: str) -> AsyncIterator[Checkpointer]:
try:
from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver
except ImportError as exc:
raise ImportError(SQLITE_INSTALL) from exc
conn_str = resolve_sqlite_conn_str(raw)
await asyncio.to_thread(ensure_sqlite_parent_dir, conn_str)
async with AsyncSqliteSaver.from_conn_string(conn_str) as saver:
await saver.setup()
yield saver
# ---------------------------------------------------------------------------
# Async factory
# ---------------------------------------------------------------------------
@@ -49,30 +155,64 @@ async def _async_checkpointer(config) -> AsyncIterator[Checkpointer]:
return
if config.type == "sqlite":
try:
from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver
except ImportError as exc:
raise ImportError(SQLITE_INSTALL) from exc
conn_str = resolve_sqlite_conn_str(config.connection_string or "store.db")
await asyncio.to_thread(ensure_sqlite_parent_dir, conn_str)
async with AsyncSqliteSaver.from_conn_string(conn_str) as saver:
await saver.setup()
async with _sqlite_checkpointer_from_conn_str(config.connection_string or "store.db") as saver:
yield saver
return
if config.type == "postgres":
try:
from langgraph.checkpoint.postgres.aio import AsyncPostgresSaver
except ImportError as exc:
raise ImportError(POSTGRES_INSTALL) from exc
if not _use_postgres_enabled(config):
conn_str = _sqlite_conn_str_for_postgres_fallback(config)
logger.info("Checkpointer: use_postgres=false; using SQLite fallback (%s)", conn_str)
async with _sqlite_checkpointer_from_conn_str(conn_str) as saver:
yield saver
return
if not config.connection_string:
raise ValueError(POSTGRES_CONN_REQUIRED)
async with AsyncPostgresSaver.from_conn_string(config.connection_string) as saver:
await saver.setup()
yield saver
# 连接池 + 健康检查,根治远程 Postgres 单连接空闲被关导致的 run 崩溃(见 _pg_pool_saver)。
async with _pg_pool_saver(config.connection_string) as saver:
if not _dual_read_enabled(config):
yield saver
return
from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver
from deerflow.runtime.checkpointer.checkpoint_migration import (
DualReadCheckpointer,
SQLiteCheckpointSource,
)
sources: list[SQLiteCheckpointSource] = []
for sqlite_path in _dual_read_paths(config):
if not Path(sqlite_path).exists():
logger.info("Checkpoint dual-read source does not exist, skipping: %s", sqlite_path)
continue
ctx = AsyncSqliteSaver.from_conn_string(sqlite_path)
source_saver = await ctx.__aenter__()
await source_saver.setup()
sources.append(
SQLiteCheckpointSource(
path=sqlite_path,
saver=source_saver,
context=ctx,
progress_path=Path(sqlite_path).with_suffix(Path(sqlite_path).suffix + ".pg-migration.json"),
)
)
if not sources:
logger.info("Checkpoint dual-read enabled but no SQLite sources were opened; using Postgres only")
yield saver
return
proxy = DualReadCheckpointer(
saver,
sources,
)
try:
yield proxy
finally:
await proxy.aclose_sources()
return
raise ValueError(f"Unknown checkpointer type: {config.type!r}")
@@ -106,16 +246,11 @@ async def _async_checkpointer_from_database(db_config) -> AsyncIterator[Checkpoi
return
if db_config.backend == "postgres":
try:
from langgraph.checkpoint.postgres.aio import AsyncPostgresSaver
except ImportError as exc:
raise ImportError(POSTGRES_INSTALL) from exc
if not db_config.postgres_url:
raise ValueError("database.postgres_url is required for the postgres backend")
async with AsyncPostgresSaver.from_conn_string(db_config.postgres_url) as saver:
await saver.setup()
# 连接池 + 健康检查(见 _pg_pool_saver),避免单连接空闲被关。
async with _pg_pool_saver(db_config.postgres_url) as saver:
yield saver
return
offline-backend-20260512/backend/packages/harness/deerflow/runtime/checkpointer/checkpoint_migration.py
+276
View File
@@ -0,0 +1,276 @@
"""SQLite-to-Postgres checkpoint migration and read fallback helpers.
The live gateway can wrap a Postgres checkpointer with ``DualReadCheckpointer``.
New writes go to Postgres, reads prefer Postgres and fall back to legacy SQLite.
Manual migration scripts reuse the copy helpers in this module.
"""
from __future__ import annotations
import asyncio
import contextlib
import json
import logging
import sqlite3
from dataclasses import dataclass
from pathlib import Path
from typing import Any
logger = logging.getLogger(__name__)
@dataclass
class SQLiteCheckpointSource:
path: str
saver: Any
context: Any
progress_path: Path
closed: bool = False
async def close(self) -> None:
if self.closed:
return
self.closed = True
try:
await self.context.__aexit__(None, None, None)
except Exception:
logger.warning("Failed to close legacy SQLite checkpointer: %s", self.path, exc_info=True)
def _sqlite_db_exists(path: str) -> bool:
return Path(path).exists()
def _sqlite_threads(path: str) -> list[str]:
with sqlite3.connect(path) as conn:
rows = conn.execute("SELECT DISTINCT thread_id FROM checkpoints ORDER BY thread_id").fetchall()
return [str(row[0]) for row in rows if row and row[0]]
def _sqlite_has_table(path: str, table: str) -> bool:
with sqlite3.connect(path) as conn:
row = conn.execute(
"SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?",
(table,),
).fetchone()
return row is not None
def sqlite_recent_threads(path: str, *, since: str | None = None, until: str | None = None) -> list[str]:
"""Return threads updated since an ISO timestamp when ``threads_meta`` exists."""
if not _sqlite_has_table(path, "threads_meta"):
return []
query = "SELECT thread_id FROM threads_meta"
where: list[str] = []
params: list[str] = []
if since:
where.append("updated_at >= ?")
params.append(since)
if until:
where.append("updated_at <= ?")
params.append(until)
if where:
query += " WHERE " + " AND ".join(where)
query += " ORDER BY updated_at DESC"
with sqlite3.connect(path) as conn:
rows = conn.execute(query, tuple(params)).fetchall()
return [str(row[0]) for row in rows if row and row[0]]
def _sqlite_thread_checkpoint_count(path: str, thread_id: str) -> int:
with sqlite3.connect(path) as conn:
row = conn.execute(
"SELECT COUNT(*) FROM checkpoints WHERE thread_id = ?",
(thread_id,),
).fetchone()
return int(row[0] if row else 0)
def _load_progress(path: Path) -> set[str]:
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except FileNotFoundError:
return set()
except Exception:
logger.warning("Ignoring unreadable checkpoint migration progress file: %s", path, exc_info=True)
return set()
migrated = payload.get("migrated_threads", [])
return {str(x) for x in migrated if x}
def _save_progress(path: Path, migrated: set[str]) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_suffix(path.suffix + ".tmp")
tmp.write_text(
json.dumps({"migrated_threads": sorted(migrated)}, ensure_ascii=False, indent=2),
encoding="utf-8",
)
tmp.replace(path)
async def _copy_thread_state(source, target, thread_id: str) -> bool:
"""Copy one thread through the public LangGraph checkpointer API."""
config = {"configurable": {"thread_id": thread_id}}
tuples = [t async for t in source.alist(config)]
if not tuples:
return False
for tup in reversed(tuples):
configurable = (tup.config or {}).get("configurable", {}) or {}
checkpoint_ns = configurable.get("checkpoint_ns", "") or ""
parent_id = ((tup.parent_config or {}).get("configurable", {}) or {}).get("checkpoint_id")
put_config: dict[str, Any] = {"configurable": {"thread_id": thread_id, "checkpoint_ns": checkpoint_ns}}
if parent_id:
put_config["configurable"]["checkpoint_id"] = parent_id
new_versions = (tup.checkpoint or {}).get("channel_versions", {}) or {}
await target.aput(put_config, tup.checkpoint, tup.metadata or {}, new_versions)
if tup.pending_writes:
by_task: dict[str, list] = {}
for task_id, channel, value in tup.pending_writes:
by_task.setdefault(task_id, []).append((channel, value))
checkpoint_id = (tup.checkpoint or {}).get("id")
for task_id, writes in by_task.items():
writes_config = {
"configurable": {
"thread_id": thread_id,
"checkpoint_ns": checkpoint_ns,
"checkpoint_id": checkpoint_id,
}
}
await target.aput_writes(writes_config, writes, task_id)
return True
async def copy_thread_state(source, target, thread_id: str) -> bool:
return await _copy_thread_state(source, target, thread_id)
async def _target_thread_checkpoint_count(target, thread_id: str) -> int:
config = {"configurable": {"thread_id": thread_id}}
count = 0
async for _ in target.alist(config):
count += 1
return count
async def target_has_thread(target, thread_id: str) -> bool:
config = {"configurable": {"thread_id": thread_id}}
async for _ in target.alist(config, limit=1):
return True
return False
class DualReadCheckpointer:
"""Postgres-first checkpointer with SQLite read fallback."""
def __init__(self, primary, sources: list[SQLiteCheckpointSource], *, sleep_seconds: float = 0.05):
self.primary = primary
self.sources = sources
self.sleep_seconds = max(0.0, sleep_seconds)
self._closed = False
def __getattr__(self, name: str):
return getattr(self.primary, name)
async def aclose_sources(self) -> None:
self._closed = True
for source in self.sources:
await source.close()
async def aget_tuple(self, config):
found = await self.primary.aget_tuple(config)
if found is not None:
return found
for source in self.sources:
if source.closed:
continue
found = await source.saver.aget_tuple(config)
if found is not None:
return found
return None
async def aget(self, config):
found = await self.primary.aget(config)
if found is not None:
return found
for source in self.sources:
if source.closed:
continue
found = await source.saver.aget(config)
if found is not None:
return found
return None
async def alist(self, config, *, filter=None, before=None, limit=None):
seen: set[tuple[str, str, str]] = set()
yielded = 0
async for tup in self.primary.alist(config, filter=filter, before=before, limit=limit):
key = self._tuple_key(tup)
seen.add(key)
yield tup
yielded += 1
if limit is not None and yielded >= limit:
return
remaining = None if limit is None else max(0, limit - yielded)
for source in self.sources:
if source.closed:
continue
async for tup in source.saver.alist(config, filter=filter, before=before, limit=remaining):
key = self._tuple_key(tup)
if key in seen:
continue
seen.add(key)
yield tup
yielded += 1
if limit is not None:
remaining = max(0, limit - yielded)
if remaining <= 0:
return
async def aput(self, *args, **kwargs):
return await self.primary.aput(*args, **kwargs)
async def aput_writes(self, *args, **kwargs):
return await self.primary.aput_writes(*args, **kwargs)
async def adelete_thread(self, thread_id: str) -> None:
delete = getattr(self.primary, "adelete_thread", None)
if delete is not None:
await delete(thread_id)
for source in self.sources:
if source.closed:
continue
delete = getattr(source.saver, "adelete_thread", None)
if delete is not None:
await delete(thread_id)
@staticmethod
def _tuple_key(tup) -> tuple[str, str, str]:
configurable = (tup.config or {}).get("configurable", {}) or {}
return (
str(configurable.get("thread_id", "")),
str(configurable.get("checkpoint_ns", "") or ""),
str(configurable.get("checkpoint_id", "") or (tup.checkpoint or {}).get("id", "")),
)
async def _verify_source(self, source: SQLiteCheckpointSource, threads: list[str]) -> bool:
for thread_id in threads:
source_count = await asyncio.to_thread(_sqlite_thread_checkpoint_count, source.path, thread_id)
target_count = await _target_thread_checkpoint_count(self.primary, thread_id)
if target_count < source_count:
logger.warning(
"Checkpoint migration verification failed: source=%s thread_id=%s source_count=%d target_count=%d",
source.path,
thread_id,
source_count,
target_count,
)
return False
return True
offline-backend-20260512/backend/packages/harness/deerflow/runtime/runs/store/base.py
+9
View File
@@ -93,3 +93,12 @@ class RunStore(abc.ABC):
by_caller ({lead_agent, subagent, middleware}).
"""
pass
async def count_by_user(self) -> dict[str, int]:
"""Return ``{user_id: run_count}`` across all runs.
One run corresponds to one user submission, so this backs the admin
user center's 问答次数 (Q&A count) column — each question counts once.
Non-overriding stores return an empty mapping (treated as 0 per user).
"""
return {}
offline-backend-20260512/backend/packages/harness/deerflow/runtime/scheduler/html_page.py
+39
View File
@@ -38,6 +38,11 @@ _DANGEROUS_URL_RE = re.compile(r"^\s*(javascript|vbscript|data:text/html)", re.I
# Attributes carrying a URL we must scheme-check.
_URL_ATTRS = frozenset({"href", "src", "action", "formaction", "xlink:href", "background", "poster"})
_LOCAL_DOC_LINK_RE = re.compile(
r"^(?![a-z][a-z0-9+.-]*:)(?!//)(?P<path>[^#?]*\.(?:html?|md|markdown))(?P<query>\?[^#]*)?(?P<fragment>#.+)?$",
re.IGNORECASE,
)
def parse_frontmatter(text: str) -> tuple[dict[str, Any], str]:
"""Split a ``---`` YAML frontmatter block from the body of a SKILL.md.
@@ -225,6 +230,38 @@ def sanitize_html(html: str) -> tuple[str, list[str]]:
return str(soup), removed
def constrain_same_page_anchor_links(html: str) -> tuple[str, list[str]]:
"""Keep generated HTML document jumps inside the current HTML file."""
if not html or not isinstance(html, str):
return "", []
try:
from bs4 import BeautifulSoup
except Exception: # pragma: no cover - bs4 is a declared dependency
logger.warning("BeautifulSoup unavailable; returning HTML links unconstrained")
return html, []
changed: list[str] = []
soup = BeautifulSoup(html, "html.parser")
for tag in soup.find_all("a"):
href = tag.get("href")
if not isinstance(href, str):
continue
value = href.strip()
if not value or value.startswith("#"):
continue
match = _LOCAL_DOC_LINK_RE.match(value)
if not match:
continue
fragment = match.group("fragment")
if fragment:
tag["href"] = fragment
changed.append(f"{value}->{fragment}")
else:
del tag.attrs["href"]
changed.append(f"{value}->removed")
return str(soup), changed
def validate_html(html: str) -> list[str]:
"""Return a list of structural problems with ``html`` (empty == valid).
@@ -299,6 +336,7 @@ _GENERATION_TEMPLATE = """你是一个内网页面生成助手。请根据输入
6. 不要包含 <script>、外链脚本、iframe、表单提交、on* 事件属性。
7. 参考页面只用于风格和布局，页面内容必须来自“本次数据”，不要复用参考页旧数据。
8. 页面使用中文，排版适合在浏览器中直接预览。
9. HTML 内的目录/锚点跳转必须只跳转到当前这个 HTML 文件内部：只使用 href="#section-id"，不要使用 href="index.html#section-id"、".md#..."、其它文件路径或任何跨文件锚点；目标区块必须有匹配的 id。
"""
_REPAIR_TEMPLATE = """你是一个 HTML 修复助手。下面的 HTML 存在问题，请修复后重新输出完整页面。
@@ -317,6 +355,7 @@ _REPAIR_TEMPLATE = """你是一个 HTML 修复助手。下面的 HTML 存在问
2. fixed_html 必须是完整 HTML 文档，包含 <!doctype html>、html、head、body。
3. 移除所有 <script>、iframe、外链脚本、on* 事件属性、表单提交。
4. 保留原有内容与数据，只修复结构与安全问题。
5. HTML 内的目录/锚点跳转必须只跳转到当前这个 HTML 文件内部：只使用 href="#section-id"，不要链接到 .html/.md 文件作为锚点。
"""
_STYLE_EXTRACT_TEMPLATE = """你是一个页面风格分析助手。请分析下面的 HTML 页面，提取可复用的风格与布局特征。
offline-backend-20260512/backend/packages/harness/deerflow/runtime/scheduler/service.py
+131 -17
View File
@@ -53,8 +53,18 @@ _MARKDOWN_MIN_CHARS = 100
_MARKDOWN_INSTRUCTION = (
"\n\n【输出要求】请将本次任务的最终结果生成为一个 Markdown(.md) 格式的文件，"
"保存到 /mnt/user-data/outputs 目录，并使用 present_files 展示该文件。"
"Markdown 内的目录/锚点跳转必须只跳转到当前这个 Markdown 文件内部："
"只使用 [标题](#section-id)，不要使用 [标题](xxx.md#section-id)、"
"[标题](xxx.html#section-id) 或任何跨文件锚点。"
)
_LOCAL_DOC_MARKDOWN_LINK_RE = re.compile(
r"^(?![a-z][a-z0-9+.-]*:)(?!//)(?P<path>[^#?]*\.(?:html?|md|markdown))(?P<query>\?[^#]*)?(?P<fragment>#.+)?$",
re.IGNORECASE,
)
_MARKDOWN_INLINE_LINK_RE = re.compile(r"(?<!!)\[([^\]]+)\]\(([^)\s]+)(?:\s+\"[^\"]*\")?\)")
_MARKDOWN_REFERENCE_LINK_RE = re.compile(r"^(\s*\[[^\]]+\]:\s*)(\S+)(.*)$", re.MULTILINE)
class MarkdownRequirementError(Exception):
"""Raised when a ``require_markdown`` task did not produce a valid .md file.
@@ -219,6 +229,23 @@ def _task_requires_markdown(task: dict[str, Any]) -> bool:
return bool(isinstance(ctx, dict) and ctx.get("require_markdown"))
def _task_requires_html(task: dict[str, Any]) -> bool:
"""Whether a task must produce an HTML page (the ``必须生成 HTML`` switch).
The flag lives inside ``execution_context.require_html``. For backward
compatibility, tasks created under the old two-tab UI (which stored
``task_kind == "html_page"``) are still treated as HTML tasks, so no data
migration is needed. ``require_html`` and ``require_markdown`` are
independent — both may be on at once.
"""
ctx = task.get("execution_context")
if not isinstance(ctx, dict):
return False
if ctx.get("require_html"):
return True
return ctx.get("task_kind") == "html_page"
def _has_valid_markdown_file(result_files: list[dict[str, Any]] | None) -> bool:
"""True when the run produced a ``.md`` file whose text exceeds the minimum.
@@ -245,6 +272,43 @@ def _has_valid_markdown_file(result_files: list[dict[str, Any]] | None) -> bool:
return False
def _normalize_same_page_markdown_target(target: str) -> str | None:
value = (target or "").strip()
if not value or value.startswith("#"):
return value
match = _LOCAL_DOC_MARKDOWN_LINK_RE.match(value)
if not match:
return value
fragment = match.group("fragment")
return fragment or None
def _constrain_markdown_same_page_anchor_links(content: str) -> str:
"""Keep generated Markdown document jumps inside the current Markdown file."""
if not content or not isinstance(content, str):
return content or ""
def replace_inline(match: re.Match[str]) -> str:
label = match.group(1)
target = match.group(2)
normalized = _normalize_same_page_markdown_target(target)
if normalized is None:
return label
return f"[{label}]({normalized})"
def replace_reference(match: re.Match[str]) -> str:
prefix = match.group(1)
target = match.group(2)
suffix = match.group(3)
normalized = _normalize_same_page_markdown_target(target)
if normalized is None:
return f"{prefix}#"
return f"{prefix}{normalized}{suffix}"
content = _MARKDOWN_INLINE_LINK_RE.sub(replace_inline, content)
return _MARKDOWN_REFERENCE_LINK_RE.sub(replace_reference, content)
def _build_result(
task: dict[str, Any],
*,
@@ -905,15 +969,18 @@ class ScheduledTaskService:
scheduled_for: datetime,
run_id: str,
) -> dict[str, Any]:
"""Dispatch a single execution attempt by task kind.
"""Dispatch a single execution attempt by the task's output requirements.
``html_page`` tasks render an HTML document via GLM5; everything else
runs the regular lead-agent attempt. The kind lives in
``execution_context.task_kind`` so no schema migration is needed.
A task carries two independent switches in ``execution_context``:
``require_html`` (必须生成 HTML) and ``require_markdown`` (必须生成
Markdown). When ``require_html`` is set the run goes through the HTML
render pipeline (which also enforces the Markdown requirement when both
switches are on); otherwise it runs the regular lead-agent attempt.
Both flags live in ``execution_context`` so no schema migration is
needed, and old ``task_kind == "html_page"`` tasks still resolve to the
HTML pipeline via :func:`_task_requires_html`.
"""
ctx = task.get("execution_context")
kind = ctx.get("task_kind") if isinstance(ctx, dict) else None
if kind == "html_page":
if _task_requires_html(task):
return await self._run_html_page_attempt(
task, runtime_thread_id=runtime_thread_id, scheduled_for=scheduled_for, run_id=run_id
)
@@ -956,17 +1023,21 @@ class ScheduledTaskService:
runtime_thread_id: str,
scheduled_for: datetime,
run_id: str,
) -> str:
) -> tuple[str, list[dict[str, Any]]]:
"""Run the lead agent to collect source data for the HTML page.
Reuses the regular agent-run primitives (with the task's search query
folded into the prompt) and returns the agent's final text, which feeds
the GLM5 render step as ``本次数据``. Returns an empty string if the
agent produced nothing.
folded into the prompt) and returns ``(text, files)`` — the agent's
final text (feeds the render step as ``本次数据``) plus any files it
produced. The files are only captured/needed when the task also has the
Markdown switch on (both 必须生成 HTML and 必须生成 Markdown), so the run
can validate and surface the ``.md`` file alongside the HTML page;
otherwise the list is empty.
"""
cfg = self._html_config(task)
page_prompt = str(cfg.get("page_prompt") or "").strip()
negative_prompt = str(cfg.get("negative_prompt") or "").strip()
also_markdown = _task_requires_markdown(task)
gather_instruction = (
"请运用你自身的检索/搜索等技能，围绕下述页面需求获取最新、真实的资料，"
"并整理成结构化的中文要点（可用列表/小标题），供后续生成 HTML 页面使用，本步骤不要生成 HTML：\n"
@@ -982,12 +1053,22 @@ class ScheduledTaskService:
# which sources/data the agent collects, otherwise it silently leaks in.
if negative_prompt:
parts.append(f"限制条件（最高优先级，必须严格遵守）：{negative_prompt}")
gather_task = {**task, "prompt": "\n".join(parts)}
# Never apply the markdown gate to the gather pass.
gather_prompt = "\n".join(parts)
# When the Markdown switch is also on, this same gather pass must save a
# .md file. We do NOT append _MARKDOWN_INSTRUCTION here — `_start_agent_run`
# already appends it exactly once when `require_markdown` is present in the
# execution context (kept below). Appending it here too made the prompt
# carry the 【输出要求】 block twice when both HTML and Markdown were on.
gather_task = {**task, "prompt": gather_prompt}
gather_ctx = dict(task.get("execution_context") or {})
gather_ctx.pop("require_markdown", None)
# Strip the HTML-specific flags so the gather pass is a plain agent run.
gather_ctx.pop("require_html", None)
gather_ctx.pop("task_kind", None)
gather_ctx.pop("html_config", None)
# Keep ``require_markdown`` only when both switches are on, so the agent
# is told (via _start_agent_run) to produce the .md file.
if not also_markdown:
gather_ctx.pop("require_markdown", None)
# Use the task's selected model for the data-gathering agent too, so the
# whole HTML pipeline honors ``html_config.model_name`` instead of
# silently falling back to the default model for this step.
@@ -997,13 +1078,22 @@ class ScheduledTaskService:
gather_task["execution_context"] = gather_ctx
await self._ensure_thread_meta(task["user_id"], runtime_thread_id)
files: list[dict[str, Any]] = []
async with self._agent_run_lock:
output_snapshot = self._snapshot_result_files(runtime_thread_id, task["user_id"]) if also_markdown else None
record = await self._start_agent_run(gather_task, scheduled_for=scheduled_for, runtime_thread_id=runtime_thread_id)
await self.store.update_run(run_id, {"agent_run_id": record.run_id})
if record.task is not None:
await record.task
text = await self._extract_latest_ai_text(runtime_thread_id)
return text or ""
if also_markdown:
files = await self._extract_result_files(
runtime_thread_id,
task["user_id"],
run_id=run_id,
output_snapshot=output_snapshot,
)
return text or "", files
def _collect_html_skill_instructions(self) -> str:
"""Concatenate the bodies of enabled skills marked for HTML pages.
@@ -1072,10 +1162,17 @@ class ScheduledTaskService:
negative_prompt = str(cfg.get("negative_prompt") or "").strip()
reference_favorite_id = cfg.get("reference_favorite_id") or None
# 1) Gather source data (real, via the agent's search tools).
data_text = await self._gather_html_data(
# 1) Gather source data (real, via the agent's search tools). When the
# Markdown switch is also on, the gather pass additionally saves the .md
# file, returned here so we can validate + surface it next to the page.
also_markdown = _task_requires_markdown(task)
data_text, gathered_files = await self._gather_html_data(
task, runtime_thread_id=runtime_thread_id, scheduled_for=scheduled_for, run_id=run_id
)
if also_markdown and not _has_valid_markdown_file(gathered_files):
raise MarkdownRequirementError(
f"未满足输出要求：需生成 Markdown(.md) 文件，且文件正文不少于 {_MARKDOWN_MIN_CHARS} 字"
)
# 2) Load reference style/layout from a favorite if one was selected.
reference_style = None
@@ -1119,6 +1216,7 @@ class ScheduledTaskService:
# 4) Sanitize + validate, with one repair pass on failure.
html, _removed = html_page_utils.sanitize_html(html)
html, _link_changes = html_page_utils.constrain_same_page_anchor_links(html)
errors = html_page_utils.validate_html(html)
if errors:
logger.info("Generated HTML failed validation (%s); attempting repair", errors)
@@ -1131,6 +1229,7 @@ class ScheduledTaskService:
fixed = repaired.get("fixed_html")
if isinstance(fixed, str) and "<" in fixed:
fixed_clean, _ = html_page_utils.sanitize_html(fixed)
fixed_clean, _ = html_page_utils.constrain_same_page_anchor_links(fixed_clean)
if not html_page_utils.validate_html(fixed_clean):
html = fixed_clean
except Exception:
@@ -1158,6 +1257,13 @@ class ScheduledTaskService:
archive_path = self._archive_html_index(run_id, html)
files: list[dict[str, Any]] = []
# When both switches are on, surface the gathered Markdown file(s)
# alongside the generated page so the run delivers both artifacts.
if also_markdown:
for item in gathered_files:
name = str(item.get("name") or "") if isinstance(item, dict) else ""
if name.lower().endswith((".md", ".markdown")):
files.append(item)
if archive_path is not None:
files.append(
{
@@ -1483,6 +1589,14 @@ class ScheduledTaskService:
index += 1
shutil.copy2(path, candidate)
if candidate.suffix.lower() in {".md", ".markdown"}:
try:
original = candidate.read_text(encoding="utf-8", errors="ignore")
constrained = _constrain_markdown_same_page_anchor_links(original)
if constrained != original:
candidate.write_text(constrained, encoding="utf-8", newline="")
except Exception:
logger.debug("Failed to constrain scheduled markdown links in %s", candidate, exc_info=True)
return candidate
def _build_file_result(self, *, virtual_path: str, source_path: Path, archive_path: Path) -> dict[str, Any]:
offline-backend-20260512/backend/packages/harness/deerflow/skills/usage.py
+6
View File
@@ -36,6 +36,7 @@ _DEFAULT_ENTRY: dict = {
"created_at": None,
"state": STATE_ACTIVE,
"pinned": False,
"featured": False,
"archived_at": None,
"source": SOURCE_UPLOAD,
}
@@ -141,6 +142,11 @@ def set_pinned(name: str, pinned: bool) -> None:
_update_entry(name, {"pinned": pinned})
def set_featured(name: str, featured: bool) -> None:
"""置顶/取消置顶。与 pinned(防归档)正交,仅影响列表排序展示。"""
_update_entry(name, {"featured": featured})
def forget(name: str) -> None:
try:
with _usage_lock:
offline-backend-20260512/backend/packages/harness/deerflow/tools/builtins/roundtable_peers_tool.py
+82
View File
@@ -0,0 +1,82 @@
"""read_peer_delivery —— 圆桌席位「按需读取前序席位**完整交付**」的只读工具。
背景：圆桌每个席位是独立 thread（独立 sandbox），彼此隔离。默认情况下,一个席位只能
从跨席位广播里看到前序席位的**截断摘要**（≤1500 字）。当席位识别到自己需要某个前序
席位的**全文**时,调用本工具按席位名取回完整交付。
数据来源：全文由**网关层**（``app.gateway``）在发起本席位 run 时,注入到
``runtime.context["roundtable_peer_deliveries"]`` —— 一个
``[{"name": <席位显示名>, "content": <完整交付正文>}, ...]`` 列表。本工具**只读该上下文
dict**,不跨 thread、不碰 sandbox、不发网络请求,因此完全在 ``deerflow.*`` 内（不违反
harness→app 导入防火墙）。非圆桌场景下该上下文为空,工具返回「无」。
「传摘要按需传全文」：完整交付**只在席位真正调用本工具时**才进入它的 LLM 上下文,平时
只有摘要,既省 token 又满足「需要时能拿全文」。详见
``frontend-web/docs/roundtable-dag-orchestration.md`` §12.9。
"""
from __future__ import annotations
import logging
from langchain.tools import ToolRuntime, tool
logger = logging.getLogger(__name__)
_CTX_KEY = "roundtable_peer_deliveries"
def _peer_deliveries(runtime: ToolRuntime | None) -> list[dict]:
"""从 runtime.context 取出网关注入的前序席位完整交付列表（缺失/非法时返回 []）。"""
if runtime is None:
return []
ctx = getattr(runtime, "context", None)
if not ctx:
return []
try:
peers = ctx.get(_CTX_KEY)
except Exception:
return []
if not isinstance(peers, list):
return []
return [p for p in peers if isinstance(p, dict)]
@tool("read_peer_delivery", parse_docstring=True)
def read_peer_delivery_tool(runtime: ToolRuntime, seat_name: str = "") -> str:
"""Read another roundtable seat's FULL deliverable on demand.
In a multi-agent roundtable you (a seat) normally only see **truncated
summaries** of earlier seats' deliverables. When you decide you genuinely
need the **complete text** of a specific earlier seat (e.g. to build on its
detailed data/conclusions), call this tool with that seat's name to fetch
its full deliverable. Prefer the summaries you already have; only call this
when the summary is insufficient.
Args:
seat_name: The earlier seat's display name (the Chinese name shown in
the summaries, e.g. "情报收集"). Leave empty to list which peers'
full deliverables are available to read.
"""
peers = _peer_deliveries(runtime)
if not peers:
return "（当前没有可读取的前序席位完整交付。请直接基于你已有的上下文与摘要完成交付。）"
names = [str(p.get("name") or "") for p in peers if p.get("name")]
query = (seat_name or "").strip()
if not query:
return (
"可读取**完整交付**的前序席位：" + "、".join(names) + "。\n"
"需要哪个席位的全文,就再次调用 read_peer_delivery 并把它的名称传给 seat_name。"
)
ql = query.lower()
# 精确匹配（忽略大小写）优先,其次子串互含（容忍模型给「情报收集席位」这类带后缀的名）。
for p in peers:
if str(p.get("name") or "").lower() == ql:
return str(p.get("content") or "（该席位交付为空。）")
for p in peers:
nm = str(p.get("name") or "")
if nm and (ql in nm.lower() or nm.lower() in ql):
return str(p.get("content") or "（该席位交付为空。）")
return f"未找到名为「{query}」的前序席位。可读取的有：" + "、".join(names) + "。"
offline-backend-20260512/backend/packages/harness/deerflow/tools/tools.py
+7
View File
@@ -114,6 +114,13 @@ def get_available_tools(
builtin_tools.extend([skill_list_tool, skill_view_tool])
# read_peer_delivery —— 圆桌席位「按需读取前序席位完整交付」的只读工具(从 run context
# 取网关注入的全文,非圆桌场景为空 → 工具自述「无」)。始终提供(只读、无副作用);圆桌
# 总控/报告角色由 run policy 显式排除,避免干扰它们。
from deerflow.tools.builtins.roundtable_peers_tool import read_peer_delivery_tool
builtin_tools.append(read_peer_delivery_tool)
# skill_manage 能创建/修改/删除技能,属于"自我进化"写能力,按 skill_evolution.enabled
# 门控。该开关用户可在 per-user 配置里覆盖,所以按"生效配置(含 per-user 覆盖)"判断 ——
# Gateway 每次 run 重建 agent,用户改了配置下次 run 即生效;解析失败退回全局 config.yaml。
Loaded 100 of 267 files, more files were not shown because too many files have changed in this diff. Show more