sync update from LAN repo (source dhk-1 @ c21a2a4a)

This commit is contained in:
gozeros committed 2026-11-10 14:04:00 +08:00
1 parent 264e2200d6
commit a1fa13a238
122 files changed
+8881 -623

No files matched your search

+27 -2
View File
@@ -1,5 +1,27 @@
# CLAUDE.md
## Model-independent skill invocation attribution (2026-11-07)
Never derive skill-use UI or counters from model narration or the optional tool
`description`. The canonical signals are a successful `skill_view`, a successful
`read_file` under `/mnt/skills/{public,custom}/<skill>/...`, and an executed
`bash` command containing such a skill path. `sandbox/tools.py` resolves names
through the active skill view so a directory/frontmatter-name mismatch remains
correct; both public and custom skills count.
`skills/usage.py::bump_use(..., event_key=run_id)` de-duplicates all file/script
signals to one use per skill per user run. It updates `.usage.json` and schedules
an absolute, monotonic update of the `skills` MySQL catalog projection on the
engine home loop. Keep this bridge inside `deerflow.*`; the harness must not
import `app.*`. The skill-card counter is intentionally independent of the
heavier full-tool audit switch. `skill_metrics_enabled` defaults on and mounts
`ToolMetricsMiddleware(skill_only=True)` while `tool_metrics_enabled` is off;
non-skill calls pass through without DB work, but attributed skill calls still
feed audit pages and leaderboards. Frontend labels are recovered by
`core/tools/skill-invocation.ts` from explicit skill-tool arguments or skill
paths. Regression coverage lives in `test_skill_usage_signals.py`,
`test_skill_view_fast_path.py`, and `skill-invocation.test.ts`.
## Cold agent construction (2026-10-30)
`runtime/runs/worker.py` dispatches opted-in synchronous factories through
@@ -549,7 +571,10 @@ handoff replaces the foreground lease with the durable job occupancy. There is
no per-user pool. Admission is serialized by a database gate row, and the cap
is enforced at acquire, init, every model-producing run, and background-job
start. Full pools return HTTP 409; an enabled gate with missing storage or a
missing session id fails closed. The switch applies on the next admission
missing session id fails closed. Foreground leases last ``lease_minutes``
(default 5, range 1–60) and are renewed by the browser heartbeat; changing the
setting applies on the next admission or heartbeat and never extends
``max_hold_minutes``. The switch applies on the next admission
without a restart. Position and eight-department roundtables are not counted.
Roundtable backend jobs use a database unique active-dedupe key plus a leased
@@ -837,7 +862,7 @@ FastAPI application on port 8001 with health checks at `GET|HEAD /health`, `/api
| **Task Deep-link Threads** (`/api/task-deeplink-threads`) | Dedicated shared history index for the `3qfx` / `rwfx` / `xdfx` single-agent task workspaces. `GET /?task_id=&go_path=&limit=` performs an indexed `(task_id, go_path)` lookup in `task_deeplink_threads` and joins `threads_meta` by `thread_id` for the current title/status/timestamps; LangGraph checkpoints remain the source of truth for messages. Thread metadata carrying `taskRecordId + taskGoPath` is dual-written atomically by both SQL and memory thread stores and deleted with the thread. `3qfx/rwfx` use the task id as `taskRecordId`; `xdfx` uses the original action id while retaining the resolved real task id in `metadata.taskId`. `thread_type=task_deeplink` keeps these rows out of general recent chats; explicit `taskId` searches still opt out of that exclusion for compatibility. Alembic `20261019_01` is schema-only and intentionally does **not** backfill legacy task threads. Frontend hook: `core/threads/use-task-deeplink-threads.ts`. Tests: `packages/harness/tests/test_task_deeplink_threads.py`, `packages/harness/tests/test_thread_meta_task_lookup.py`. |
| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; unexpected failures are logged server-side and return a generic 500 detail |
| **Threads search** (`POST /threads/search`, LangGraph-compat) | Conversation listing. **System threads are excluded by default**: rows whose `metadata.system` is truthy (scheduler runs, roundtable workers) or whose `thread_type` is a known system type (`task-analysis` seat runs included; older seat rows are also matched by `task_analysis_seat`) never crowd the user's list; a caller whose `metadata` filter explicitly references `thread_type`/`system` (e.g. studio notebooks) opts out of the exclusion. New `query` field = case-insensitive title (`display_name`) substring search, used by the chats page's server-side pagination. `POST /threads/count` takes the same request body (limit/offset ignored) and returns `{total}` under identical filtering/exclusion semantics ? powers the chats page's ??/??? display without fetching rows (`ThreadMetaStore.count`). `ThreadMetaStore.search` gained `exclude_system`/`query` params; JSON-blob filters batch-scan until the requested `limit`/`offset` window is full (so pagination over a heavily-filtered set never under-fills), with a `thread_id` tie-breaker on the `updated_at DESC` ordering for stable OFFSET paging. Tests: `tests/test_thread_meta_search.py` |
| **Artifacts** (`/api/threads/{id}/artifacts`, `/api/artifact-library`, `/api/strategy-project-artifacts/{id}/download`) | `GET /{path}` streams thread artifacts; active content types (`text/html`, `application/xhtml+xml`, `image/svg+xml`) are always forced as download attachments to reduce XSS risk; `?download=true` still forces download for other file types. `GET /api/threads/{id}/artifact-preview?path=` returns bounded text, using streaming statistics plus deterministic head/middle/tail samples above the database threshold. `GET /api/artifact-library` combines every current-user normal thread's authoritative `outputs/` files with completed strategy-selection syntheses projected from the owner-scoped `parallel_agent_tasks.messages` snapshot. Strategy items use `source_type=strategy`, carry `strategy_project_id` / `strategy_name` / `strategy_stale`, and explicitly distinguish `strategy_source_kind=strategy_selection` from an `analysis_project_mode=strategy` thread (`analysis_project`); virtual items also expose `download_url`, so clients never infer routing from sentinel thread/path values. Search/source filters cover both. `workflowStale=true` keeps the last completed historical synthesis visible and downloadable. The dedicated strategy download route revalidates current-user ownership and snapshot completion and serves Markdown without pretending the virtual item is a thread artifact. Uploads and image/audio/video media remain excluded. Library metadata is persisted in `artifact_library_items`; list/search/filter/count/pagination read only this user-scoped SQL index. A generation-fenced `artifact_library_sync_states` lease schedules low-concurrency filesystem/task reconciliation in the background on first/stale reads or `refresh=true`, keeps the last successful snapshot visible during refresh/failure, and reports `sync_status` / `syncing` / `indexed_at` / `sync_error` to the client. Files themselves remain authoritative on disk and are never copied into the database. |
| **Artifacts** (`/api/threads/{id}/artifacts`, `/api/artifact-library`, `/api/strategy-project-artifacts/{id}/download`) | `GET /{path}` streams thread artifacts; active content types (`text/html`, `application/xhtml+xml`, `image/svg+xml`) are always forced as download attachments to reduce XSS risk; `?download=true` still forces download for other file types. HTML/XHTML additionally accepts `offline_bundle=true`: it returns `{stem}-离线包.zip` with `index.html`, the referenced trusted Gateway runtime/image dependencies rewritten under `assets/`, a manifest/README and relevant third-party licenses; public view-share files support the same query. The builder never fetches arbitrary remote URLs, and records remaining Leaflet tile/user-supplied external URLs in the manifest. `GET /api/threads/{id}/artifact-preview?path=` returns bounded text, using streaming statistics plus deterministic head/middle/tail samples above the database threshold. `GET /api/artifact-library` combines every current-user normal thread's authoritative `outputs/` files with completed strategy-selection syntheses projected from the owner-scoped `parallel_agent_tasks.messages` snapshot. Strategy items use `source_type=strategy`, carry `strategy_project_id` / `strategy_name` / `strategy_stale`, and explicitly distinguish `strategy_source_kind=strategy_selection` from an `analysis_project_mode=strategy` thread (`analysis_project`); virtual items also expose `download_url`, so clients never infer routing from sentinel thread/path values. Search/source filters cover both. `workflowStale=true` keeps the last completed historical synthesis visible and downloadable. The dedicated strategy download route revalidates current-user ownership and snapshot completion and serves Markdown without pretending the virtual item is a thread artifact. Uploads and image/audio/video media remain excluded. Library metadata is persisted in `artifact_library_items`; list/search/filter/count/pagination read only this user-scoped SQL index. A generation-fenced `artifact_library_sync_states` lease schedules low-concurrency filesystem/task reconciliation in the background on first/stale reads or `refresh=true`, keeps the last successful snapshot visible during refresh/failure, and reports `sync_status` / `syncing` / `indexed_at` / `sync_error` to the client. Files themselves remain authoritative on disk and are never copied into the database. |
| **Word export** (`POST /api/writing/export/docx`) | Shared Markdown → DOCX generator used by conversation, artifact, scheduled-task, Deep Research, and AI-writing download surfaces. `app/gateway/word_export.py` builds OOXML with no Python conversion dependency: A4 portrait, 3.7/3.5/2.8/2.6 cm margins, real multilevel heading numbering (`一、` / `(一)` / `1.`), two-character first-line indent on heading levels 1–3, formal body/table styles, page numbers beginning at 1 with odd pages left-aligned and even pages right-aligned, and embedded `app/gateway/assets/fonts/方正小标宋简体.ttf`. Before numbering it removes complete manual heading prefixes (including `2.1` / `2.1.1`) and drops Markdown horizontal rules from Word output. The ordinary-Q&A Lead Agent can emit the matching Markdown heading forms and avoid unordered bullet lists when the admin enables `prompt_prefix.ordinary_qa_markdown_format_enabled` in `system_settings.json`; the switch defaults off, while writing setup, writing mode, notebook, scheduled runs, and SOUL-driven custom agents keep their independent prompts. The font loader validates its internal family, SHA-256, and OpenType embedding permission (`OS/2.fsType`) before export. Tests: `tests/test_word_export.py`, `tests/test_es_query_routing_prompt.py`. |
| **Deep Research** (`/api/deep-research`) | Durable, user-owned research sessions/jobs with replayable SSE events and hidden-thread artifacts. `basic`/`quick` use the compact runner; `detailed` creates bounded subtopics, searches/writes sections concurrently, and assembles a report; `deep` recursively searches follow-up questions under a deterministic breadth/depth query budget. `multi_agent` is a LangGraph 1.x editor/researcher/writer/reviewer graph: its initial plan emits `awaiting_input`, the recorder stores the interrupt and releases the lease, and `POST /jobs/{id}/resume` rebuilds the graph from the persisted approved/revised plan. A bounded `custom_outline` is frozen into the session; headings/list entries deterministically supply the detailed and multi-agent plan, and the shared writer receives it only as a guarded content constraint. The compact writer validates generated Markdown against that frozen selection: `fixed` requires every non-placeholder heading with the same level and order, while `adaptive` retries only when no core template heading survives. That correction shares the single existing retry budget (never more than two writer calls), and its deterministic fallback is outline-shaped as well. When `custom_outline` is empty (the workbench default — the user did not pick a report structure), chat collection first calls `POST /sessions/{id}/intent-outline` after the collector turn ends: the chosen model plans a Markdown skeleton from the topic plus collected materials, the outline is frozen into the session snapshot and shown in the「开始撰写」form for editing. If that step is skipped or fails, `basic`/`quick` still call `generate_intent_outline` at write time, emit `plan_created` (`source: intent`), then write through the same shared writer; generation failure is a recoverable `intent_outline_fallback` warning and does not abort the report. Completed reports support owner-scoped follow-up (`GET /sessions/{id}/messages`, `POST /sessions/{id}/chat`): context is only the current report, selected sources and recent messages; source-id citations are persisted, while `allow_new_research=true` is explicitly rejected. Owners may change that evidence set only after the job stops via `PATCH /sessions/{id}/sources/{source_id}`; this applies to future follow-ups rather than rewriting a completed report. Optional report illustrations are gated by `config.yaml -> deep_research.image_generation`: the Harness `OpenAICompatibleImageProvider` requires base64 output, the shared runner step writes only bounded `research-image-01..04.(png|jpg|webp)` artifacts and Markdown uses private `deep-research://` links. `GET /sessions/{id}/artifacts/{name}` checks session ownership before serving an image; an unavailable provider yields a warning but never fails the text report. Runners live in `packages/harness/deerflow/agents/deep_research/runners/`, may never import `app.*`, and use only the `AdapterBundle`. |
| **Gateway auth** | Owner-scoped Deep Research routes await the platform's asynchronous `get_current_user` resolver before they read or write persistence, ensuring each session and job operation receives a concrete user ID. Deep Research artifacts are always resolved from a whitelisted filename via `/mnt/user-data/outputs/<name>` before mapping to a host path, preserving the sandbox guard on native Windows and Linux hosts. |
+27 -7
View File
@@ -1,5 +1,18 @@
# DeerFlow Backend
## 技能调用识别与统计(2026-11-07)
技能调用的展示和“调用次数”不再依赖模型主动说明。后端以真实工具信号为准:
成功的 `skill_view`、对 `/mnt/skills/{public,custom}/<name>/...` 的文件读取,
以及 `bash` 直接执行该目录下的技能脚本,都会识别为对应技能调用。一次问答中同一技能即使先读
`SKILL.md`、再读参考文件并执行多个脚本,也只累计一次;不同问答分别累计。
计数会同时更新 `.usage.json` 和技能列表使用的数据库投影,因此内置技能与自定义技能都能在
页面刷新后看到最新调用次数。对话步骤也直接从 `skill_view` 参数或真实技能路径显示
“调用技能「名称」”;弱模型未填写工具描述、未在正文里自述技能名时仍可识别。
完整工具审计仍由 `runtime_performance.tool_metrics_enabled` 独立控制;默认开启的
`skill_metrics_enabled` 只把可归因到具体技能的低频调用写入统计表,供技能排行和调用统计使用。
## 首次问答初始化响应性(2026-10-30)
普通问答、开放问答接口和工作流智能体节点的模型 SDK 加载、客户端初始化及
@@ -1715,7 +1728,9 @@ Multi-agent roundtable step 2 has a separate room cap. Administrators change
it under Settings → 会商并发 (`roundtable_room_limit`, default on, 3 rooms)
without restarting. The count is the number of live roundtable executions
across every account (foreground leases/live LangGraph runs plus executing
background jobs), not the current login. Merely opening a tab or viewing chat
background jobs), not the current login. A foreground lease stays occupied for
``lease_minutes`` without a renewal (default 5 minutes, range 1–60) and is
refreshed by the page heartbeat. Merely opening a tab or viewing chat
history does not count, but two tabs that both run the same saved draft count
as two rooms. All seats and Step 3 work in one execution share one room;
foreground-to-background handoff keeps one room. A full pool keeps the user on
@@ -1815,11 +1830,12 @@ Beta(8,2) 先验抑制小样本极值;零样本返回 `score: null` 和
实际加载的 `config.yaml`(`DEER_FLOW_CONFIG_PATH` 指定文件优先)支持以下独立开关。
`skill_view_reuse_enabled` 和 `skill_view_cache_enabled` 在配置里不写时默认开启;
要恢复每次工具调用都全量刷新技能视图,把对应项写成 `false` 后重启。
模型计时与线程池开关默认关闭,工具统计写入也默认暂停:
模型计时与线程池开关默认关闭,普通工具审计默认暂停,低频技能统计默认保留:
```yaml
runtime_performance:
tool_metrics_enabled: false
skill_metrics_enabled: true
skill_view_reuse_enabled: true
skill_view_cache_enabled: true
model_timing_nonblocking_enabled: true
@@ -1827,13 +1843,17 @@ runtime_performance:
asyncio_thread_pool_size: 100
```
- `tool_metrics_enabled`:工具/技能调用统计写入开关,**缺省为 false(暂停)**,旧部署配置
没有此项时也暂停。主智能体和子智能体不再装载工具统计中间件,不创建统计写库任务、
不执行技能统计合并查询,从源头减少与登录共用业务数据库连接池的竞争。
- `tool_metrics_enabled`:普通工具审计写入开关,**缺省为 false(暂停)**,旧部署配置
没有此项时也暂停。主智能体和子智能体不会为普通工具创建统计写库任务,
从源头减少与登录共用业务数据库连接池的竞争。
正常工具调用、错误处理、问答消息/checkpoint 和模型调用统计继续运行;已有
`tool_call_metrics` 记录与查询/导出接口保留。暂停期间工具/技能审计明细及排行榜的
工具/技能调用计数不再新增,且不缓存或事后补录。恢复时设为 `true` 并重启后台。
`tool_call_metrics` 记录与查询/导出接口保留。暂停期间普通工具调用计数不再新增,
且不缓存或事后补录。恢复时设为 `true` 并重启后台。
部署新代码并重启后才对所有运行实例生效;这不消除历史统计查询或其他数据库操作的开销。
- `skill_metrics_enabled`:只记录从 `skill_view` 参数或
`/mnt/skills/{public,custom}/<name>/...` 工具路径确定出的技能调用,缺省为 `true`。
非技能工具直接透传,不执行技能合并查询或写库;因此普通工具审计暂停时,技能排行与
技能调用统计仍会更新。若连这部分低频写入也要暂停,显式设为 `false` 并重启。
- `skill_view_reuse_enabled`:一次沙箱工具调用共用一份已验证技能视图,路径检查、
本地/AIO 沙箱初始化、命令转换和输出脱敏不再反复刷新。下一次工具调用重新验证,
不跨问答永久沿用启停状态。
@@ -186,6 +186,15 @@ def current_user_id(request: Request) -> str:
return get_effective_user_id()
def _lease_seconds(settings: Any) -> int:
minutes = getattr(settings, "lease_minutes", 5)
try:
minutes = int(minutes)
except (TypeError, ValueError):
minutes = 5
return max(1, min(60, minutes)) * 60
def _store(request: Request):
return getattr(request.app.state, "roundtable_room_store", None)
@@ -232,6 +241,7 @@ async def admit_roundtable_room(
enabled=settings.enabled,
max_rooms=settings.max_rooms,
max_hold_minutes=getattr(settings, "max_hold_minutes", 30),
lease_seconds=_lease_seconds(settings),
extra_keys=extra_keys,
)
@@ -265,6 +275,7 @@ async def renew_roundtable_room(
enabled=settings.enabled,
max_rooms=settings.max_rooms,
max_hold_minutes=getattr(settings, "max_hold_minutes", 30),
lease_seconds=_lease_seconds(settings),
extra_keys=extra_keys,
)
@@ -425,6 +436,7 @@ async def enforce_roundtable_run(
enabled=settings.enabled,
max_rooms=settings.max_rooms,
max_hold_minutes=getattr(settings, "max_hold_minutes", 30),
lease_seconds=_lease_seconds(settings),
extra_keys=await backend_occupancy_keys(request),
)
if not decision.get("ok", False):
@@ -30,7 +30,7 @@ from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi.responses import StreamingResponse
from langchain_core.messages import HumanMessage, SystemMessage
from pydantic import BaseModel, Field
from sqlalchemy import and_, case, desc, func, not_, or_, select, text
from sqlalchemy import and_, case, desc, func, literal, not_, or_, select, text
from app.gateway.admin_query_window import (
ADMIN_QUERY_WINDOW_DAYS,
@@ -279,6 +279,41 @@ class AdminLeaderboardDimension(BaseModel):
last_seen_at: str = ""
class AdminLeaderboardDetailItem(BaseModel):
"""One row in the expanded skill, tool, or agent analytics drawer."""
name: str
count: int = 0
user_count: int = 0
total_tokens: int = 0
avg_duration_ms: int = 0
class AdminLeaderboardDetailSummary(BaseModel):
count: int = 0
item_count: int = 0
user_count: int = 0
total_tokens: int = 0
avg_duration_ms: int = 0
class AdminLeaderboardDetailResponse(BaseModel):
"""Fresh, drill-down analytics for a leaderboard dimension.
Unlike the dashboard's compact daily snapshots, this response is queried
directly. It lets an administrator inspect the entire matching dimension
(up to the explicit limit) and does not lose long-tail rows because they
were outside a daily top-N snapshot.
"""
kind: str
range_days: int
since: str
until: str
summary: AdminLeaderboardDetailSummary
items: list[AdminLeaderboardDetailItem] = Field(default_factory=list)
class AdminLeaderboardTrendPoint(BaseModel):
date: str
runs: int = 0
@@ -961,6 +996,125 @@ async def _query_run_dimensions(
]
async def _query_leaderboard_dimension_details(
session,
since: datetime,
until: datetime,
*,
kind: str,
limit: int,
exclude_user_ids: set[str],
include_scheduled: bool,
include_admins: bool,
include_failed: bool,
) -> tuple[
AdminLeaderboardDetailSummary,
list[AdminLeaderboardDetailItem],
]:
"""Load complete, fresh detail statistics for one leaderboard dimension.
The main leaderboard deliberately reads compact daily snapshots so it can
remain quick under load. The drawer is an intentional admin drill-down,
therefore it runs a bounded aggregate against the source metric tables and
returns exact unique-user counts for the selected interval.
"""
if kind in {"skill", "tool"}:
name_col = ToolCallMetricRow.skill_name if kind == "skill" else ToolCallMetricRow.tool_name
base_filters = [
_tool_time_filter(since, until),
name_col.is_not(None),
name_col != "",
]
def apply_scope(stmt):
return _apply_tool_scope(
stmt,
exclude_user_ids=exclude_user_ids,
include_admins=include_admins,
include_scheduled=include_scheduled,
)
count_col = func.count(ToolCallMetricRow.id)
user_count_col = func.count(func.distinct(ToolCallMetricRow.user_id))
total_tokens_col = literal(0)
avg_duration_col = func.coalesce(func.avg(ToolCallMetricRow.duration_ms), 0)
elif kind == "agent":
name_col = RunRow.assistant_id
base_filters = [
_run_time_filter(since, until),
name_col.is_not(None),
name_col != "",
]
def apply_scope(stmt):
return _apply_run_scope(
stmt,
user_column=RunRow.user_id,
run_id_column=RunRow.run_id,
exclude_user_ids=exclude_user_ids,
include_admins=include_admins,
include_failed=include_failed,
include_scheduled=include_scheduled,
)
count_col = func.count(RunRow.run_id)
user_count_col = func.count(func.distinct(RunRow.user_id))
total_tokens_col = func.coalesce(func.sum(RunRow.total_tokens), 0)
# Run rows have no reliable wall-clock duration. Keep this explicit at
# zero instead of inferring elapsed time from an asynchronously updated
# journal timestamp.
avg_duration_col = literal(0)
else:
raise ValueError(f"Unknown leaderboard detail kind: {kind}")
item_stmt = (
select(
name_col.label("name"),
count_col.label("count"),
user_count_col.label("user_count"),
total_tokens_col.label("total_tokens"),
avg_duration_col.label("avg_duration_ms"),
)
.where(*base_filters)
.group_by(name_col)
.order_by(count_col.desc(), name_col.asc())
.limit(limit)
)
item_stmt = apply_scope(item_stmt)
summary_stmt = select(
count_col.label("count"),
func.count(func.distinct(name_col)).label("item_count"),
user_count_col.label("user_count"),
total_tokens_col.label("total_tokens"),
avg_duration_col.label("avg_duration_ms"),
).where(*base_filters)
summary_stmt = apply_scope(summary_stmt)
item_rows = (await _execute_leaderboard_query(session, f"{kind}_detail.items", item_stmt)).all()
summary_row = (await _execute_leaderboard_query(session, f"{kind}_detail.summary", summary_stmt)).one()
summary = AdminLeaderboardDetailSummary(
count=int(summary_row[0] or 0),
item_count=int(summary_row[1] or 0),
user_count=int(summary_row[2] or 0),
total_tokens=int(summary_row[3] or 0),
avg_duration_ms=int(summary_row[4] or 0),
)
items = [
AdminLeaderboardDetailItem(
name=row[0] or "",
count=int(row[1] or 0),
user_count=int(row[2] or 0),
total_tokens=int(row[3] or 0),
avg_duration_ms=int(row[4] or 0),
)
for row in item_rows
]
return summary, items
async def _query_model_rankings(
session,
since: datetime,
@@ -1876,6 +2030,59 @@ async def get_user_leaderboard(
)
@router.get("/leaderboard/details", response_model=AdminLeaderboardDetailResponse)
async def get_user_leaderboard_details(
request: Request,
kind: str = Query(description="One of: skill, tool, agent", pattern="^(skill|tool|agent)$"),
days: int = Query(default=30, ge=1, le=ADMIN_QUERY_WINDOW_DAYS),
since: str | None = Query(default=None, description="Inclusive range start, ISO datetime"),
until: str | None = Query(default=None, description="Inclusive range end, ISO datetime"),
limit: int = Query(default=500, ge=1, le=1000),
) -> AdminLeaderboardDetailResponse:
"""Return fresh drill-down statistics for one leaderboard card.
This endpoint intentionally reads the source metric tables instead of the
asynchronous daily leaderboard snapshots. A drawer represents an explicit
admin investigation, where exact unique-user counts, all matching entries,
and complete detail rows are more useful than a compact cache.
"""
await _require_admin(request)
session_factory = get_session_factory()
if session_factory is None:
raise HTTPException(status_code=503, detail="Analytics requires database persistence")
settings = load_system_settings().leaderboard
since_dt, until_dt, range_days = resolve_admin_query_window(
days=days,
since=since,
until=until,
)
excluded, include_scheduled, include_admins, include_failed, _settings_hash = _resolve_leaderboard_snapshot_scope(settings)
async with session_factory() as session:
await _set_analytics_read_timeout(session)
summary, items = await _query_leaderboard_dimension_details(
session,
since_dt,
until_dt,
kind=kind,
limit=limit,
exclude_user_ids=excluded,
include_scheduled=include_scheduled,
include_admins=include_admins,
include_failed=include_failed,
)
return AdminLeaderboardDetailResponse(
kind=kind,
range_days=range_days,
since=_iso(since_dt),
until=_iso(until_dt),
summary=summary,
items=items,
)
@router.get("/leaderboard/snapshots", response_model=AdminLeaderboardSnapshotStatusResponse)
async def get_leaderboard_snapshot_status(
request: Request,
@@ -1,14 +1,16 @@
"""Application chrome name and logo.
"""Application chrome name, logo, and optional red-theme artwork.
Stored separately from ``generated_page_branding`` and the
``generated-page-branding/`` directory. An empty name or a missing logo leaves
the built-in title and static theme logos in place.
``generated-page-branding/`` directory. Missing custom images leave the
built-in theme artwork in place.
"""
from __future__ import annotations
import asyncio
from typing import Annotated
from uuid import uuid4
from xml.etree import ElementTree
from fastapi import APIRouter, File, Form, HTTPException, Request, UploadFile
from fastapi.responses import FileResponse
@@ -18,6 +20,8 @@ from app.gateway.routers.system_settings import _require_admin
from deerflow.config.runtime_paths import runtime_home
from deerflow.config.system_settings import (
AppIdentitySettings,
app_identity_light_red_header_background_path,
app_identity_light_red_nav_active_path,
app_identity_logo_path,
load_system_settings,
save_system_settings,
@@ -26,25 +30,48 @@ from deerflow.config.system_settings import (
router = APIRouter(prefix="/api/system-settings", tags=["system-settings"])
public_router = APIRouter(prefix="/api/public/app-identity", tags=["app-identity"])
_MAX_LOGO_BYTES = 10 * 1024 * 1024
_MAX_IMAGE_BYTES = 10 * 1024 * 1024
_PUBLIC_LOGO_PATH = "/api/public/app-identity/logo"
_PUBLIC_LIGHT_RED_HEADER_PATH = "/api/public/app-identity/light-red-header-background"
_PUBLIC_LIGHT_RED_ACTIVE_PATH = "/api/public/app-identity/light-red-nav-active"
class AppIdentityResponse(BaseModel):
system_name: str = ""
header_title_font_mode: str = "adaptive"
header_title_font_size: int = 30
logo_url: str | None = None
logo_version: str | None = None
has_logo: bool = False
light_red_header_background_url: str | None = None
light_red_header_background_version: str | None = None
has_light_red_header_background: bool = False
light_red_nav_active_url: str | None = None
light_red_nav_active_version: str | None = None
has_light_red_nav_active: bool = False
def _detect_logo(data: bytes) -> tuple[str, str]:
def _detect_image(
data: bytes,
label: str,
*,
allow_svg: bool = False,
) -> tuple[str, str]:
if data.startswith(b"\x89PNG\r\n\x1a\n"):
return ".png", "image/png"
if data.startswith(b"\xff\xd8\xff"):
return ".jpg", "image/jpeg"
if len(data) >= 12 and data[:4] == b"RIFF" and data[8:12] == b"WEBP":
return ".webp", "image/webp"
raise HTTPException(status_code=422, detail="Logo 仅支持 PNG、JPG 或 WebP 图片")
if allow_svg:
try:
root = ElementTree.fromstring(data)
except ElementTree.ParseError:
root = None
if root is not None and root.tag.rsplit("}", 1)[-1].lower() == "svg":
return ".svg", "image/svg+xml"
supported = "PNG、JPG、WebP 或 SVG" if allow_svg else "PNG、JPG 或 WebP"
raise HTTPException(status_code=422, detail=f"{label}仅支持 {supported} 图片")
def _normalize_system_name(value: str | None) -> str:
@@ -54,21 +81,61 @@ def _normalize_system_name(value: str | None) -> str:
return normalized
def _normalize_header_title_font_mode(value: str) -> str:
normalized = str(value).strip().lower()
if normalized not in {"adaptive", "fixed"}:
raise HTTPException(status_code=422, detail="系统名称字号模式仅支持自适应或固定大小")
return normalized
def _normalize_header_title_font_size(value: int) -> int:
normalized = int(value)
if normalized < 16 or normalized > 40:
raise HTTPException(status_code=422, detail="系统名称固定字号必须在 16px 到 40px 之间")
return normalized
def _response(settings: AppIdentitySettings) -> AppIdentityResponse:
path = app_identity_logo_path(settings)
version = settings.logo_version if path is not None else None
header_background_path = app_identity_light_red_header_background_path(settings)
header_background_version = (
settings.light_red_header_background_version
if header_background_path is not None
else None
)
nav_active_path = app_identity_light_red_nav_active_path(settings)
nav_active_version = (
settings.light_red_nav_active_version if nav_active_path is not None else None
)
return AppIdentityResponse(
system_name=settings.system_name,
header_title_font_mode=settings.header_title_font_mode,
header_title_font_size=settings.header_title_font_size,
logo_url=f"{_PUBLIC_LOGO_PATH}?v={version}" if version else None,
logo_version=version,
has_logo=path is not None,
light_red_header_background_url=(
f"{_PUBLIC_LIGHT_RED_HEADER_PATH}?v={header_background_version}"
if header_background_version
else None
),
light_red_header_background_version=header_background_version,
has_light_red_header_background=header_background_path is not None,
light_red_nav_active_url=(
f"{_PUBLIC_LIGHT_RED_ACTIVE_PATH}?v={nav_active_version}"
if nav_active_version
else None
),
light_red_nav_active_version=nav_active_version,
has_light_red_nav_active=nav_active_path is not None,
)
def _write_logo(data: bytes, suffix: str) -> str:
def _write_asset(data: bytes, suffix: str, prefix: str) -> str:
directory = runtime_home() / "app-identity"
directory.mkdir(parents=True, exist_ok=True)
filename = f"logo-{uuid4().hex}{suffix}"
filename = f"{prefix}-{uuid4().hex}{suffix}"
destination = directory / filename
temporary = directory / f".{filename}.tmp"
try:
@@ -79,9 +146,40 @@ def _write_logo(data: bytes, suffix: str) -> str:
return filename
async def _read_uploaded_image(
upload: UploadFile,
label: str,
*,
allow_svg: bool = False,
max_bytes: int | None = _MAX_IMAGE_BYTES,
) -> tuple[bytes, str, str]:
data = await upload.read(max_bytes + 1) if max_bytes is not None else await upload.read()
await upload.close()
if not data:
raise HTTPException(status_code=422, detail=f"{label}文件不能为空")
if max_bytes is not None and len(data) > max_bytes:
raise HTTPException(status_code=413, detail=f"{label}不能超过 10MB")
suffix, media_type = _detect_image(data, label, allow_svg=allow_svg)
return data, suffix, media_type
def _public_asset_headers(media_type: str | None) -> dict[str, str]:
headers = {"Cache-Control": "public, max-age=31536000, immutable"}
if media_type == "image/svg+xml":
headers.update(
{
"Content-Security-Policy": (
"sandbox; default-src 'none'; style-src 'unsafe-inline'; img-src data:"
),
"X-Content-Type-Options": "nosniff",
}
)
return headers
@router.get("/app-identity", response_model=AppIdentityResponse)
async def get_app_identity_settings() -> AppIdentityResponse:
"""Return the application name and uploaded logo. Any logged-in user may read it."""
"""Return the application identity settings readable by all logged-in users."""
return _response(load_system_settings().app_identity)
@@ -89,68 +187,160 @@ async def get_app_identity_settings() -> AppIdentityResponse:
@router.put("/app-identity", response_model=AppIdentityResponse)
async def update_app_identity_settings(
request: Request,
system_name: str = Form(default=""),
remove_logo: bool = Form(default=False),
logo: UploadFile | None = File(default=None),
system_name: Annotated[str, Form()] = "",
remove_logo: Annotated[bool, Form()] = False,
logo: Annotated[UploadFile | None, File()] = None,
header_title_font_mode: Annotated[str | None, Form()] = None,
header_title_font_size: Annotated[int | None, Form()] = None,
light_red_header_background: Annotated[UploadFile | None, File()] = None,
remove_light_red_header_background: Annotated[bool, Form()] = False,
light_red_nav_active: Annotated[UploadFile | None, File()] = None,
remove_light_red_nav_active: Annotated[bool, Form()] = False,
) -> AppIdentityResponse:
"""Save the application chrome name and optional logo. Admin only."""
"""Save the application chrome identity and optional red-theme artwork."""
await _require_admin(request)
if logo is not None and remove_logo:
raise HTTPException(status_code=422, detail="上传 Logo 时不能同时恢复静态 Logo")
if light_red_header_background is not None and remove_light_red_header_background:
raise HTTPException(status_code=422, detail="上传导航栏背景图时不能同时恢复默认图片")
if light_red_nav_active is not None and remove_light_red_nav_active:
raise HTTPException(status_code=422, detail="上传菜单选中样式图时不能同时恢复默认图片")
current = load_system_settings()
previous_logo = app_identity_logo_path(current.app_identity)
previous_header_background = app_identity_light_red_header_background_path(
current.app_identity
)
previous_nav_active = app_identity_light_red_nav_active_path(current.app_identity)
update: dict[str, object] = {"system_name": _normalize_system_name(system_name)}
new_logo_path = None
if logo is not None:
data = await logo.read(_MAX_LOGO_BYTES + 1)
await logo.close()
if not data:
raise HTTPException(status_code=422, detail="Logo 文件不能为空")
if len(data) > _MAX_LOGO_BYTES:
raise HTTPException(status_code=413, detail="Logo 文件不能超过 10MB")
suffix, media_type = _detect_logo(data)
filename = await asyncio.to_thread(_write_logo, data, suffix)
new_logo_path = runtime_home() / "app-identity" / filename
update.update(
{
"logo_filename": filename,
"logo_media_type": media_type,
"logo_version": uuid4().hex,
}
if header_title_font_mode is not None:
update["header_title_font_mode"] = _normalize_header_title_font_mode(header_title_font_mode)
if header_title_font_size is not None:
update["header_title_font_size"] = _normalize_header_title_font_size(header_title_font_size)
logo_asset = (
await _read_uploaded_image(logo, "Logo") if logo is not None else None
)
header_background_asset = (
await _read_uploaded_image(
light_red_header_background,
"导航栏背景图",
allow_svg=True,
max_bytes=None,
)
elif remove_logo:
update.update(
{
"logo_filename": None,
"logo_media_type": None,
"logo_version": None,
}
if light_red_header_background is not None
else None
)
nav_active_asset = (
await _read_uploaded_image(
light_red_nav_active,
"菜单选中样式图",
allow_svg=True,
max_bytes=None,
)
if light_red_nav_active is not None
else None
)
updated = current.app_identity.model_copy(update=update)
staged_paths = []
try:
if logo_asset is not None:
data, suffix, media_type = logo_asset
filename = await asyncio.to_thread(_write_asset, data, suffix, "logo")
staged_paths.append(runtime_home() / "app-identity" / filename)
update.update(
{
"logo_filename": filename,
"logo_media_type": media_type,
"logo_version": uuid4().hex,
}
)
elif remove_logo:
update.update(
{
"logo_filename": None,
"logo_media_type": None,
"logo_version": None,
}
)
if header_background_asset is not None:
data, suffix, media_type = header_background_asset
filename = await asyncio.to_thread(
_write_asset, data, suffix, "light-red-header"
)
staged_paths.append(runtime_home() / "app-identity" / filename)
update.update(
{
"light_red_header_background_filename": filename,
"light_red_header_background_media_type": media_type,
"light_red_header_background_version": uuid4().hex,
}
)
elif remove_light_red_header_background:
update.update(
{
"light_red_header_background_filename": None,
"light_red_header_background_media_type": None,
"light_red_header_background_version": None,
}
)
if nav_active_asset is not None:
data, suffix, media_type = nav_active_asset
filename = await asyncio.to_thread(
_write_asset, data, suffix, "light-red-active"
)
staged_paths.append(runtime_home() / "app-identity" / filename)
update.update(
{
"light_red_nav_active_filename": filename,
"light_red_nav_active_media_type": media_type,
"light_red_nav_active_version": uuid4().hex,
}
)
elif remove_light_red_nav_active:
update.update(
{
"light_red_nav_active_filename": None,
"light_red_nav_active_media_type": None,
"light_red_nav_active_version": None,
}
)
updated = current.app_identity.model_copy(update=update)
saved = save_system_settings(current.model_copy(update={"app_identity": updated}))
except Exception:
if new_logo_path is not None:
await asyncio.to_thread(new_logo_path.unlink, missing_ok=True)
for staged_path in staged_paths:
await asyncio.to_thread(staged_path.unlink, missing_ok=True)
raise
if previous_logo is not None and previous_logo != app_identity_logo_path(saved.app_identity):
await asyncio.to_thread(previous_logo.unlink, missing_ok=True)
previous_paths = (previous_logo, previous_header_background, previous_nav_active)
saved_paths = {
app_identity_logo_path(saved.app_identity),
app_identity_light_red_header_background_path(saved.app_identity),
app_identity_light_red_nav_active_path(saved.app_identity),
}
for previous_path in previous_paths:
if previous_path is not None and previous_path not in saved_paths:
await asyncio.to_thread(previous_path.unlink, missing_ok=True)
return _response(saved.app_identity)
@router.delete("/app-identity", response_model=AppIdentityResponse)
async def delete_app_identity_settings(request: Request) -> AppIdentityResponse:
"""Clear the application name and uploaded logo. Admin only."""
"""Clear the application identity and uploaded theme artwork. Admin only."""
await _require_admin(request)
current = load_system_settings()
previous_logo = app_identity_logo_path(current.app_identity)
previous_paths = (
app_identity_logo_path(current.app_identity),
app_identity_light_red_header_background_path(current.app_identity),
app_identity_light_red_nav_active_path(current.app_identity),
)
saved = save_system_settings(current.model_copy(update={"app_identity": AppIdentitySettings()}))
if previous_logo is not None:
await asyncio.to_thread(previous_logo.unlink, missing_ok=True)
for previous_path in previous_paths:
if previous_path is not None:
await asyncio.to_thread(previous_path.unlink, missing_ok=True)
return _response(saved.app_identity)
@@ -166,5 +356,35 @@ async def get_app_identity_logo() -> FileResponse:
path,
media_type=settings.logo_media_type or "image/png",
content_disposition_type="inline",
headers={"Cache-Control": "public, max-age=31536000, immutable"},
headers=_public_asset_headers(settings.logo_media_type),
)
@public_router.get("/light-red-header-background")
async def get_light_red_header_background() -> FileResponse:
settings = load_system_settings().app_identity
path = app_identity_light_red_header_background_path(settings)
if path is None:
raise HTTPException(status_code=404, detail="未上传红色导航栏背景图")
return FileResponse(
path,
media_type=settings.light_red_header_background_media_type or "image/png",
content_disposition_type="inline",
headers=_public_asset_headers(
settings.light_red_header_background_media_type
),
)
@public_router.get("/light-red-nav-active")
async def get_light_red_nav_active() -> FileResponse:
settings = load_system_settings().app_identity
path = app_identity_light_red_nav_active_path(settings)
if path is None:
raise HTTPException(status_code=404, detail="未上传红色菜单选中样式图")
return FileResponse(
path,
media_type=settings.light_red_nav_active_media_type or "image/png",
content_disposition_type="inline",
headers=_public_asset_headers(settings.light_red_nav_active_media_type),
)
@@ -1,7 +1,9 @@
import asyncio
import logging
import mimetypes
import os
import re
import tempfile
import zipfile
from datetime import UTC, datetime
from pathlib import Path
@@ -11,12 +13,16 @@ from urllib.parse import quote
from fastapi import APIRouter, HTTPException, Query, Request
from fastapi.responses import FileResponse, PlainTextResponse, Response
from pydantic import BaseModel, Field
from starlette.background import BackgroundTask
from app.gateway.authz import require_permission
from app.gateway.operation_timing import OperationTiming
from app.gateway.path_utils import aresolve_thread_virtual_path
from app.gateway.utils import sanitize_log_param
from deerflow.config.paths import get_paths
from deerflow.config.system_settings import generated_page_logo_path, load_system_settings
from deerflow.html_runtime.addresses import html_map_tile_url, html_runtime_base_url
from deerflow.html_runtime.offline_bundle import build_html_offline_bundle
from deerflow.runtime.thread_paths import aresolve_path_user_id
from deerflow.runtime.user_context import get_effective_user_id
from deerflow.utils.file_io import file_io
@@ -213,6 +219,44 @@ def _build_attachment_headers(filename: str, extra_headers: dict[str, str] | Non
return headers
def _remove_temporary_bundle(path: str) -> None:
try:
Path(path).unlink(missing_ok=True)
except OSError:
logger.warning("Failed to remove temporary HTML offline bundle: %s", path, exc_info=True)
async def build_html_offline_bundle_response(actual_path: Path) -> FileResponse:
"""Build and stream a portable ZIP for one generated HTML artifact."""
descriptor, temporary_name = tempfile.mkstemp(prefix="deerflow-html-offline-", suffix=".zip")
os.close(descriptor)
temporary_path = Path(temporary_name)
try:
branding = load_system_settings().generated_page_branding
logo_path = generated_page_logo_path(branding)
await file_io.run_until_complete(
build_html_offline_bundle,
actual_path,
temporary_path,
runtime_base_url=html_runtime_base_url(),
map_tile_url=html_map_tile_url(),
generated_logo_path=logo_path,
)
except BaseException:
temporary_path.unlink(missing_ok=True)
raise
download_name = f"{actual_path.stem}-离线包.zip"
return FileResponse(
temporary_path,
filename=download_name,
media_type="application/zip",
headers={"Cache-Control": "private, no-store"},
background=BackgroundTask(_remove_temporary_bundle, str(temporary_path)),
)
def is_text_file_by_content(path: Path, sample_size: int = 8192) -> bool:
"""Check if file is text by examining content for null bytes."""
try:
@@ -988,10 +1032,20 @@ async def preview_artifact(
@router.get(
"/threads/{thread_id}/artifacts/{path:path}",
summary="Get Artifact File",
description="Retrieve an artifact file generated by the AI agent. Text and binary files can be viewed inline, while active web content is always downloaded.",
description=(
"Retrieve an artifact file generated by the AI agent. Text and binary files can be viewed inline, "
"while active web content is always downloaded. HTML files can be exported with their DeerFlow "
"runtime dependencies as a portable ZIP."
),
)
@require_permission("threads", "read", owner_check=True)
async def get_artifact(thread_id: str, path: str, request: Request, download: bool = False) -> Response:
async def get_artifact(
thread_id: str,
path: str,
request: Request,
download: bool = False,
offline_bundle: bool = False,
) -> Response:
"""Get an artifact file by its path.
The endpoint automatically detects file types and returns appropriate content types.
@@ -1018,6 +1072,8 @@ async def get_artifact(thread_id: str, path: str, request: Request, download: bo
download (bool): If true, forces attachment download for file types that are
otherwise returned inline or as plain text. Active HTML/XHTML/SVG content
is always downloaded regardless of this flag.
offline_bundle (bool): If true for an HTML/XHTML file, returns a ZIP containing
``index.html`` plus the packaged DeerFlow JS/CSS/image runtime dependencies.
Example:
- Get text file inline: `/api/threads/abc123/artifacts/mnt/user-data/outputs/notes.txt`
@@ -1075,6 +1131,11 @@ async def get_artifact(thread_id: str, path: str, request: Request, download: bo
mime_type, _ = mimetypes.guess_type(actual_path)
if offline_bundle:
if actual_path.suffix.lower() not in {".htm", ".html", ".xhtml"}:
raise HTTPException(status_code=400, detail="Offline bundles are available for HTML files only")
return await build_html_offline_bundle_response(actual_path)
if download:
return FileResponse(path=actual_path, filename=actual_path.name, media_type=mime_type, headers=_build_attachment_headers(actual_path.name))
@@ -681,6 +681,9 @@ def _embed_allowed_weknora_ids(ctx: WeKnoraEmbedContext) -> tuple[str, ...]:
def _embed_context_payload(ctx: WeKnoraEmbedContext) -> dict[str, Any]:
# Keep this payload bounded: it is signed directly into the browser cookie.
# The visible knowledge-base list is hydrated only for the embedded page or
# its list API call, after the cookie has been verified.
return {
"mapping_id": ctx.mapping_id,
"weknora_id": ctx.weknora_id,
@@ -688,11 +691,79 @@ def _embed_context_payload(ctx: WeKnoraEmbedContext) -> dict[str, Any]:
"is_admin": ctx.is_admin,
"can_write": ctx.can_write,
"is_conversation_deposit": ctx.is_conversation_deposit,
"allowed_weknora_ids": list(_embed_allowed_weknora_ids(ctx)),
"exp": ctx.exp,
}
async def _with_visible_embed_knowledge_bases(
request: Request,
ctx: WeKnoraEmbedContext,
) -> WeKnoraEmbedContext:
"""Hydrate the iframe-only picker allowlist without putting it in a cookie."""
visible_rows = await _store(request).list_visible(
ctx.user_id,
scope="all",
# Mirror the regular picker. An administrator must not receive another
# user's private bases merely because the iframe uses an admin token.
is_admin=False,
)
return replace(
ctx,
allowed_weknora_ids=tuple(
dict.fromkeys(
value
for value in (
ctx.weknora_id,
*(str(item.get("weknora_id") or "") for item in visible_rows),
)
if value
)
),
)
def _embed_directory_time(value: Any) -> str | None:
if isinstance(value, datetime):
return value.isoformat()
if isinstance(value, str) and value.strip():
return value
return None
async def _local_embed_knowledge_base_directory(
request: Request,
ctx: WeKnoraEmbedContext,
) -> Response:
"""Serve the iframe picker from DeerFlow mappings, never WeKnora's full directory."""
rows = await _store(request).list_visible(
ctx.user_id,
scope="all",
# Match the normal picker: the WeKnora service-admin token must not
# make an administrator's private libraries discoverable to others.
is_admin=False,
)
items = [
{
"id": str(row.get("weknora_id") or ""),
"name": str(row.get("name") or ""),
"description": str(row.get("description") or ""),
"type": str(row.get("kb_type") or "document"),
"status": "active",
"created_at": _embed_directory_time(row.get("created_at")),
"updated_at": _embed_directory_time(row.get("updated_at")),
}
for row in rows
if str(row.get("weknora_id") or "")
]
response = _json_response(
{"success": True, "data": items, "total": len(items)},
)
response.headers["Cache-Control"] = "no-store"
return response
def _weknora_id_from_platform_path(path: str) -> str | None:
prefix = "/platform/knowledge-bases/"
if not path.startswith(prefix):
@@ -708,15 +779,13 @@ async def _switch_embed_context(
ctx: WeKnoraEmbedContext,
weknora_id: str,
) -> WeKnoraEmbedContext:
"""Resolve a permitted native-UI switch to the exact cmzs mapping.
"""Resolve a native-UI switch to the exact cmzs mapping.
The embedded WeKnora UI uses a service-admin token, so every target remains
constrained to an ID signed into this short-lived session and is checked
again against the cmzs visibility rules before its page can be proxied.
The embedded WeKnora UI uses a service-admin token, so the target must be
checked against the current cmzs visibility rules on every switch. This is
deliberately dynamic rather than a signed ID list in the browser cookie.
"""
if weknora_id not in _embed_allowed_weknora_ids(ctx):
raise HTTPException(status_code=403, detail="Knowledge base is outside the embedded session scope")
store = _store(request)
mapped = await store.get_by_weknora_id(weknora_id)
if mapped is None:
@@ -2807,13 +2876,6 @@ async def _prepare_weknora_embed_session(
) -> tuple[str, dict[str, Any]]:
_runtime_for_iframe_proxy(request)
row, user_id, is_admin = await _authorized_mapping(request, mapping_id, write=False)
visible_rows = await _store(request).list_visible(
user_id,
scope="all",
# The iframe must mirror the knowledge-base picker, not grant an
# administrator an unlisted user's private knowledge bases.
is_admin=False,
)
writable = None
if not _is_conversation_deposit_mapping(row):
writable = await _store(request).get_authorized(mapping_id, user_id, write=True, is_admin=is_admin)
@@ -2825,7 +2887,6 @@ async def _prepare_weknora_embed_session(
is_admin=is_admin,
can_write=writable is not None,
is_conversation_deposit=_is_conversation_deposit_mapping(row),
allowed_weknora_ids=tuple(dict.fromkeys(value for value in (weknora_id, *(str(item.get("weknora_id") or "") for item in visible_rows)) if value)),
exp=int(time.time()) + _WEKNORA_EMBED_TTL_SECONDS,
)
return weknora_id, _embed_context_payload(ctx)
@@ -2885,6 +2946,7 @@ async def proxy_weknora_platform_page(request: Request, proxied_path: str = "")
requested_weknora_id = _weknora_id_from_platform_path(requested)
if requested_weknora_id is not None:
selected_ctx = await _switch_embed_context(request, ctx, requested_weknora_id)
selected_ctx = await _with_visible_embed_knowledge_bases(request, selected_ctx)
response = await _proxy_weknora_request(
request,
target_base_url=_runtime_for_iframe_proxy(request).web_base_url,
@@ -2909,6 +2971,7 @@ async def proxy_weknora_platform_page(request: Request, proxied_path: str = "")
public_target = _with_public_prefix(_request_public_prefix(request), allowed_path)
return RedirectResponse(f"{public_target}?{target_query}", status_code=302)
runtime = _runtime_for_iframe_proxy(request)
ctx = await _with_visible_embed_knowledge_bases(request, ctx)
return await _proxy_weknora_request(
request,
target_base_url=runtime.web_base_url,
@@ -2958,6 +3021,8 @@ async def proxy_weknora_api(request: Request, proxied_path: str) -> Response:
if request.method.upper() == "OPTIONS":
return Response(status_code=204)
ctx = await _read_embed_context(request)
if request.method.upper() in {"GET", "HEAD"} and proxied_path.rstrip("/") == "knowledge-bases":
return await _local_embed_knowledge_base_directory(request, ctx)
runtime = _runtime_for_iframe_proxy(request)
return await _proxy_weknora_request(
request,
@@ -2982,6 +3047,8 @@ async def proxy_weknora_embed_api(request: Request, proxied_path: str) -> Respon
if request.method.upper() == "OPTIONS":
return Response(status_code=204)
ctx = await _read_embed_context(request)
if request.method.upper() in {"GET", "HEAD"} and proxied_path.rstrip("/") == "knowledge-bases":
return await _local_embed_knowledge_base_directory(request, ctx)
runtime = _runtime_for_iframe_proxy(request)
return await _proxy_weknora_request(
request,
@@ -12,7 +12,7 @@ import time
import zipfile
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
from typing import Any, Literal
from fastapi import APIRouter, Depends, File, HTTPException, Query, Request, Response, UploadFile
from fastapi.responses import FileResponse
@@ -35,6 +35,7 @@ from app.gateway.skill_catalog_cache import (
load_or_warm_skill_catalog_snapshot,
load_skill_catalog_projection,
refresh_skill_catalog_projection,
set_skill_catalog_enabled,
)
from deerflow.agents.lead_agent.prompt import refresh_skills_system_prompt_cache_async
from deerflow.config.app_config import AppConfig
@@ -369,6 +370,24 @@ class SkillRollbackRequest(BaseModel):
history_index: int = Field(default=-1, description="History entry index to restore from, defaulting to the latest change.")
class SkillVersionEntryResponse(BaseModel):
version: int
sha256: str
created_at: str
created_by: str | None = None
action: str
summary: str | None = None
based_on_version: int | None = None
active: bool = False
class SkillVersionsResponse(BaseModel):
enabled: bool
skill_name: str
active_version: int | None = None
versions: list[SkillVersionEntryResponse] = Field(default_factory=list)
def _skill_to_response(
skill: Skill,
ownership: dict[str, Any] | None = None,
@@ -1305,6 +1324,10 @@ async def install_skill_upload(
request: Request,
file: UploadFile = File(..., description="The .zip or .skill archive to install"),
overwrite: bool = False,
install_mode: Literal["iterate", "overwrite"] | None = Query(
default=None,
description="Same-name custom upload mode when versioning is enabled: iterate or overwrite.",
),
target_category: SkillCategory = SkillCategory.CUSTOM,
config: AppConfig = Depends(get_config),
skill_store: SkillStore = Depends(get_skill_store),
@@ -1339,9 +1362,31 @@ async def install_skill_upload(
target_category=target_category,
conflict_kind=validation.get("conflict_kind"),
)
if (overwrite and not can_install_as_is and not can_overwrite) or (not overwrite and not can_install_as_is):
requested_overwrite = overwrite or install_mode in {"iterate", "overwrite"}
if (requested_overwrite and not can_install_as_is and not can_overwrite) or (
not requested_overwrite and not can_install_as_is
):
raise HTTPException(status_code=409, detail=conflict_message)
if install_mode == "iterate":
from deerflow.skills.versioning import record_custom_skill_version, skill_versioning_enabled
if not skill_versioning_enabled():
raise HTTPException(status_code=409, detail="技能迭代功能未开启")
if target_category != SkillCategory.CUSTOM or not _owned_by_me or not can_overwrite:
raise HTTPException(status_code=409, detail="只有本人已有的同名自定义技能可以迭代")
overwrite = True
await asyncio.to_thread(
record_custom_skill_version,
preview_name,
storage=storage,
action="baseline",
author=user_id,
summary="同名上传迭代前的活动版本",
)
elif install_mode == "overwrite":
overwrite = True
result = await storage.ainstall_skill_from_archive(tmp_path, overwrite=overwrite, require_skill_suffix=False, target_category=target_category)
storage.invalidate_skills_cache()
try:
from deerflow.skills.usage import mark_uploaded
mark_uploaded(result["skill_name"])
@@ -1360,8 +1405,49 @@ async def install_skill_upload(
await skill_store.ensure_legacy(
{"name": result["skill_name"], "owner_user_id": user_id, "published": False}
)
try:
from deerflow.skills.versioning import (
record_custom_skill_version,
reset_custom_skill_versions,
skill_versioning_enabled,
)
if skill_versioning_enabled():
if result.get("overwritten") and install_mode == "iterate":
await asyncio.to_thread(
record_custom_skill_version,
result["skill_name"],
storage=storage,
action="upload_iteration",
author=user_id,
summary="上传同名技能并迭代为新版本",
)
elif result.get("overwritten"):
await asyncio.to_thread(
reset_custom_skill_versions,
result["skill_name"],
storage=storage,
author=user_id,
summary="上传同名技能并直接覆盖",
)
else:
await asyncio.to_thread(
record_custom_skill_version,
result["skill_name"],
storage=storage,
action="upload",
author=user_id,
summary="上传技能初始版本",
)
except Exception:
logger.exception("Failed to persist custom skill version after upload: %s", result["skill_name"])
# The package is already installed. Do not report the whole
# upload as failed (which would invite an unsafe duplicate
# retry); opening the version list lazily snapshots the current
# package and repairs the manifest.
await _refresh_catalog_snapshot(config)
await refresh_skills_system_prompt_cache_async()
await _sync_skill_view(result["skill_name"])
return SkillInstallResponse(**result)
except FileNotFoundError as e:
raise HTTPException(status_code=404, detail=str(e))
@@ -1603,6 +1689,7 @@ async def install_skill(
target_category=target_category,
require_skill_suffix=False,
)
storage.invalidate_skills_cache()
try:
from deerflow.skills.usage import mark_uploaded
mark_uploaded(result.skill_name)
@@ -1616,8 +1703,16 @@ async def install_skill(
await skill_store.ensure_legacy(
{"name": result.skill_name, "owner_user_id": user_id, "published": False}
)
await _record_custom_version(
result.skill_name,
storage=storage,
action="artifact_install",
author=user_id,
summary="从会话产物安装技能初始版本",
)
await _refresh_catalog_snapshot(config)
await refresh_skills_system_prompt_cache_async()
await _sync_skill_view(result.skill_name)
return result
except FileNotFoundError as e:
raise HTTPException(status_code=404, detail=str(e))
@@ -1730,6 +1825,14 @@ async def duplicate_skill(
mark_uploaded(new_name)
except Exception:
pass
storage.invalidate_skills_cache()
await _record_custom_version(
new_name,
storage=storage,
action="duplicate",
author=user_id,
summary=f"从技能 {source_name} 复制创建",
)
await _refresh_catalog_snapshot(config)
await refresh_skills_system_prompt_cache_async()
return SkillInstallResponse(
@@ -1784,6 +1887,13 @@ async def reconcile_custom_skills(
mark_uploaded(skill.name)
except Exception:
pass
await _record_custom_version(
skill.name,
storage=storage,
action="reconcile",
author=user_id,
summary="接管已有自定义技能并建立初始版本",
)
adopted.append(skill.name)
if adopted:
@@ -1869,8 +1979,16 @@ async def update_custom_skill(skill_name: str, request: CustomSkillUpdateRequest
scan = await scan_skill_content(request.content, executable=False, location=f"{skill_name}/{SKILL_MD_FILE}", app_config=config)
if scan.decision == "block":
raise HTTPException(status_code=400, detail=f"Security scan blocked the edit: {scan.reason}")
await _record_custom_version(
skill_name,
storage=storage,
action="baseline",
author=None,
summary="编辑前的活动版本",
)
prev_content = storage.read_custom_skill(skill_name)
storage.write_custom_skill(skill_name, SKILL_MD_FILE, request.content)
storage.invalidate_skills_cache()
await _refresh_catalog_snapshot(config)
storage.append_history(
skill_name,
@@ -1889,7 +2007,15 @@ async def update_custom_skill(skill_name: str, request: CustomSkillUpdateRequest
bump_patch(skill_name)
except Exception:
pass
await _record_custom_version(
skill_name,
storage=storage,
action="human_edit",
author=None,
summary="编辑 SKILL.md",
)
await refresh_skills_system_prompt_cache_async()
await _sync_skill_view(skill_name)
return await get_custom_skill(skill_name, config)
except HTTPException:
raise
@@ -1941,6 +2067,12 @@ async def delete_custom_skill(
await skill_store.delete_admin(skill_name)
else:
await skill_store.delete(skill_name, user_id)
try:
from deerflow.skills.versioning import delete_custom_skill_versions
await asyncio.to_thread(delete_custom_skill_versions, skill_name)
except Exception:
logger.warning("Failed to delete version history for %s", skill_name, exc_info=True)
await _refresh_catalog_snapshot(config)
try:
@@ -2279,6 +2411,108 @@ async def get_custom_skill_history(skill_name: str, config: AppConfig = Depends(
raise HTTPException(status_code=500, detail=f"Failed to read history: {str(e)}")
@router.get(
"/skills/custom/{skill_name}/versions",
response_model=SkillVersionsResponse,
summary="List Custom Skill Versions",
)
async def get_custom_skill_versions(
skill_name: str,
request: Request,
config: AppConfig = Depends(get_config),
skill_store: SkillStore = Depends(get_skill_store),
) -> SkillVersionsResponse:
"""List complete-package snapshots for one owned/admin-managed custom skill."""
try:
skill_name = SkillStorage.validate_skill_name(skill_name.replace("\r\n", "").replace("\n", ""))
await _get_editable_skill_or_raise(
skill_store,
skill_name,
_current_user_id(request),
is_admin=_is_admin_user(request),
)
storage = get_or_new_skill_storage(app_config=config)
if not storage.custom_skill_exists(skill_name):
raise HTTPException(status_code=404, detail=f"Custom skill '{skill_name}' not found")
from deerflow.skills.versioning import list_custom_skill_versions
payload = await asyncio.to_thread(
list_custom_skill_versions,
skill_name,
storage=storage,
author=_current_user_id(request),
)
return SkillVersionsResponse.model_validate(payload)
except HTTPException:
raise
except (FileNotFoundError, ValueError) as e:
raise HTTPException(status_code=404, detail=str(e))
except Exception as e:
logger.error("Failed to list versions for %s: %s", skill_name, e, exc_info=True)
raise HTTPException(status_code=500, detail=f"Failed to list skill versions: {e}")
@router.post(
"/skills/custom/{skill_name}/versions/{version}/activate",
response_model=SkillVersionsResponse,
summary="Activate Custom Skill Version",
)
async def activate_custom_skill_version_endpoint(
skill_name: str,
version: int,
request: Request,
config: AppConfig = Depends(get_config),
skill_store: SkillStore = Depends(get_skill_store),
) -> SkillVersionsResponse:
"""Switch the sole model-visible package without touching usage counters."""
try:
skill_name = SkillStorage.validate_skill_name(skill_name.replace("\r\n", "").replace("\n", ""))
await _get_editable_skill_or_raise(
skill_store,
skill_name,
_current_user_id(request),
is_admin=_is_admin_user(request),
)
storage = get_or_new_skill_storage(app_config=config)
if not storage.custom_skill_exists(skill_name):
raise HTTPException(status_code=404, detail=f"Custom skill '{skill_name}' not found")
from deerflow.skills.versioning import (
activate_custom_skill_version,
list_custom_skill_versions,
)
await asyncio.to_thread(
activate_custom_skill_version,
skill_name,
version,
storage=storage,
)
storage.invalidate_skills_cache()
await _refresh_catalog_snapshot(config)
await refresh_skills_system_prompt_cache_async()
await _sync_skill_view(skill_name)
payload = await asyncio.to_thread(
list_custom_skill_versions,
skill_name,
storage=storage,
author=_current_user_id(request),
)
return SkillVersionsResponse.model_validate(payload)
except HTTPException:
raise
except RuntimeError as e:
raise HTTPException(status_code=409, detail=str(e))
except FileNotFoundError as e:
raise HTTPException(status_code=404, detail=str(e))
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
except Exception as e:
logger.error("Failed to activate %s version %s: %s", skill_name, version, e, exc_info=True)
raise HTTPException(status_code=500, detail=f"Failed to activate skill version: {e}")
@router.post("/skills/custom/{skill_name}/rollback", response_model=CustomSkillContentResponse, summary="Rollback Custom Skill")
async def rollback_custom_skill(skill_name: str, request: SkillRollbackRequest, config: AppConfig = Depends(get_config)) -> CustomSkillContentResponse:
try:
@@ -2309,10 +2543,26 @@ async def rollback_custom_skill(skill_name: str, request: SkillRollbackRequest,
if scan.decision == "block":
storage.append_history(skill_name, history_entry)
raise HTTPException(status_code=400, detail=f"Rollback blocked by security scanner: {scan.reason}")
await _record_custom_version(
skill_name,
storage=storage,
action="baseline",
author=None,
summary="旧历史回滚前的活动版本",
)
storage.write_custom_skill(skill_name, SKILL_MD_FILE, target_content)
storage.invalidate_skills_cache()
await _refresh_catalog_snapshot(config)
storage.append_history(skill_name, history_entry)
await _record_custom_version(
skill_name,
storage=storage,
action="legacy_rollback",
author=None,
summary="从旧编辑历史恢复 SKILL.md",
)
await refresh_skills_system_prompt_cache_async()
await _sync_skill_view(skill_name)
return await get_custom_skill(skill_name, config)
except HTTPException:
raise
@@ -2775,6 +3025,28 @@ async def _sync_skill_view(name: str) -> None:
logger.warning("Failed to sync skill %s into the sandbox view", name, exc_info=True)
async def _record_custom_version(
name: str,
*,
storage: SkillStorage,
action: str,
author: str | None,
summary: str,
) -> None:
"""Capture one complete custom-skill package when versioning is enabled."""
from deerflow.skills.versioning import record_custom_skill_version
await asyncio.to_thread(
record_custom_skill_version,
name,
storage=storage,
action=action,
author=author,
summary=summary,
)
@router.put(
"/skills/{skill_name}/files/content",
response_model=SkillFileSaveResponse,
@@ -2826,7 +3098,18 @@ async def save_skill_file(
# SKILL.md edits must keep their frontmatter valid (name must match).
storage.validate_skill_markdown_content(skill.name, body.content)
if skill.category == SkillCategory.CUSTOM:
await _record_custom_version(
skill.name,
storage=storage,
action="baseline",
author=_current_user_id(request),
summary="文件编辑前的活动版本",
)
_write_skill_text_file_atomically(target, body.content)
if skill.category == SkillCategory.CUSTOM:
storage.invalidate_skills_cache()
try:
if skill.category == SkillCategory.CUSTOM:
@@ -2852,6 +3135,15 @@ async def save_skill_file(
pass
await refresh_skills_system_prompt_cache_async()
if skill.category == SkillCategory.CUSTOM:
await _record_custom_version(
skill.name,
storage=storage,
action="human_edit_file",
author=_current_user_id(request),
summary=f"编辑文件 {relative_posix}",
)
await _sync_skill_view(skill.name)
stat = target.stat()
@@ -2911,10 +3203,19 @@ async def delete_skill_file(
prev_content = target.read_text(encoding="utf-8")
except UnicodeDecodeError:
pass
if skill.category == SkillCategory.CUSTOM:
await _record_custom_version(
skill.name,
storage=get_or_new_skill_storage(app_config=config),
action="baseline",
author=_current_user_id(request),
summary="文件删除前的活动版本",
)
target.unlink()
if skill.category == SkillCategory.CUSTOM:
storage = get_or_new_skill_storage(app_config=config)
storage.invalidate_skills_cache()
try:
storage.append_history(
skill.name,
@@ -2928,6 +3229,13 @@ async def delete_skill_file(
)
except Exception as history_err: # noqa: BLE001
logger.warning("Failed to append deletion history for %s/%s: %s", skill.name, relative_posix, history_err)
await _record_custom_version(
skill.name,
storage=storage,
action="human_delete_file",
author=_current_user_id(request),
summary=f"删除文件 {relative_posix}",
)
else:
logger.info("Administrator deleted built-in skill file %s/%s", skill.name, relative_posix)
await _sync_skill_view(skill.name)
@@ -3010,6 +3318,11 @@ async def update_skill(
logger.info(f"Skills configuration updated and saved to: {config_path}")
await refresh_skills_system_prompt_cache_async()
await _sync_skill_view(skill_name)
# The list endpoint reads skills.enabled from the catalog projection.
# Write that one row before the client refetches. A missing row still
# needs the full snapshot so a newly discovered skill can appear.
if not await set_skill_catalog_enabled(skill_name, request.enabled):
await _refresh_catalog_snapshot(config)
skills = get_or_new_skill_storage(app_config=config).load_skills(enabled_only=False)
updated_skill = next((s for s in skills if s.name == skill_name), None)
@@ -16,15 +16,20 @@ from app.gateway.deps import get_current_user_from_request, get_optional_user_fr
from deerflow.config import get_app_config
from deerflow.config.system_settings import (
BUILTIN_THEME_IDS,
AIWritingPageDescription,
AppearanceRouteParam,
AuthenticationSettings,
CallMetricsSettings,
CompactShortcutButton,
ConversationContextSettings,
CustomThemePreset,
GeneralQaPageDescriptions,
HtmlMapSettings,
LeaderboardSettings,
ModelConcurrencySettings,
ModelPromptSuffix,
PageDescriptionsSettings,
PageHeaderDescription,
PluginDomainAgentMapping,
PluginManagementSettings,
PromptPrefixSettings,
@@ -33,6 +38,7 @@ from deerflow.config.system_settings import (
RetryKeepPartialSettings,
RoundtableRoomLimitSettings,
SkillCompressionSettings,
SkillVersioningSettings,
StrategyProjectSettings,
WordHeadingNumberingStyle,
WriteFileCompatibilitySettings,
@@ -208,6 +214,91 @@ class StrategyProjectSettingsUpdateRequest(BaseModel):
taiwan_strait_demo_enabled: bool | None = None
class PageDescriptionsResponse(BaseModel):
general_qa: GeneralQaPageDescriptions = Field(default_factory=GeneralQaPageDescriptions)
knowledge_base: PageHeaderDescription = Field(default_factory=PageHeaderDescription)
ai_writing: AIWritingPageDescription = Field(default_factory=AIWritingPageDescription)
strategy_projects: PageHeaderDescription = Field(default_factory=PageHeaderDescription)
deep_research: PageHeaderDescription = Field(default_factory=PageHeaderDescription)
gongwen: PageHeaderDescription = Field(default_factory=PageHeaderDescription)
light_app_management: PageHeaderDescription = Field(default_factory=PageHeaderDescription)
light_app_list: PageHeaderDescription = Field(default_factory=PageHeaderDescription)
agents: PageHeaderDescription = Field(default_factory=PageHeaderDescription)
business_chains: PageHeaderDescription = Field(default_factory=PageHeaderDescription)
skills: PageHeaderDescription = Field(default_factory=PageHeaderDescription)
knowledge_management: PageHeaderDescription = Field(default_factory=PageHeaderDescription)
class PageDescriptionsUpdateRequest(BaseModel):
general_qa: GeneralQaPageDescriptions | None = None
knowledge_base: PageHeaderDescription | None = None
ai_writing: AIWritingPageDescription | None = None
strategy_projects: PageHeaderDescription | None = None
deep_research: PageHeaderDescription | None = None
gongwen: PageHeaderDescription | None = None
light_app_management: PageHeaderDescription | None = None
light_app_list: PageHeaderDescription | None = None
agents: PageHeaderDescription | None = None
business_chains: PageHeaderDescription | None = None
skills: PageHeaderDescription | None = None
knowledge_management: PageHeaderDescription | None = None
def _page_descriptions_response(
settings: PageDescriptionsSettings,
) -> PageDescriptionsResponse:
return PageDescriptionsResponse(
general_qa=settings.general_qa,
knowledge_base=settings.knowledge_base,
ai_writing=settings.ai_writing,
strategy_projects=settings.strategy_projects,
deep_research=settings.deep_research,
gongwen=settings.gongwen,
light_app_management=settings.light_app_management,
light_app_list=settings.light_app_list,
agents=settings.agents,
business_chains=settings.business_chains,
skills=settings.skills,
knowledge_management=settings.knowledge_management,
)
@router.get("/page-descriptions", response_model=PageDescriptionsResponse)
async def get_page_descriptions() -> PageDescriptionsResponse:
return _page_descriptions_response(load_system_settings().page_descriptions)
@router.put("/page-descriptions", response_model=PageDescriptionsResponse)
async def update_page_descriptions(
body: PageDescriptionsUpdateRequest,
request: Request,
) -> PageDescriptionsResponse:
await _require_admin(request)
current = load_system_settings()
updated = PageDescriptionsSettings(
general_qa=body.general_qa or current.page_descriptions.general_qa,
knowledge_base=body.knowledge_base or current.page_descriptions.knowledge_base,
ai_writing=body.ai_writing or current.page_descriptions.ai_writing,
strategy_projects=body.strategy_projects or current.page_descriptions.strategy_projects,
deep_research=body.deep_research or current.page_descriptions.deep_research,
gongwen=body.gongwen or current.page_descriptions.gongwen,
light_app_management=(
body.light_app_management or current.page_descriptions.light_app_management
),
light_app_list=body.light_app_list or current.page_descriptions.light_app_list,
agents=body.agents or current.page_descriptions.agents,
business_chains=body.business_chains or current.page_descriptions.business_chains,
skills=body.skills or current.page_descriptions.skills,
knowledge_management=(
body.knowledge_management or current.page_descriptions.knowledge_management
),
)
saved = save_system_settings(
current.model_copy(update={"page_descriptions": updated}),
)
return _page_descriptions_response(saved.page_descriptions)
@router.get("/ordinary-knowledge-landing", response_model=OrdinaryKnowledgeLanding)
async def get_ordinary_knowledge_landing() -> OrdinaryKnowledgeLanding:
return OrdinaryKnowledgeLanding(mode=load_system_settings().ordinary_knowledge_landing)
@@ -491,6 +582,14 @@ class SkillCompressionUpdateRequest(BaseModel):
keep_index_in_prompt: bool | None = None
class SkillVersioningResponse(BaseModel):
enabled: bool = Field(..., description="Whether custom-skill version history is active")
class SkillVersioningUpdateRequest(BaseModel):
enabled: bool
class ModelConcurrencyResponse(BaseModel):
enabled: bool = Field(..., description="Whether per-model concurrency limiting is active")
default_limit: int = Field(..., description="Fallback max concurrent runs for models without an explicit override")
@@ -532,6 +631,17 @@ class WriteFileCompatibilityUpdateRequest(BaseModel):
enabled: bool
class CallMetricsResponse(BaseModel):
enabled: bool = Field(
default=True,
description="Whether new tool and skill calls are recorded for management statistics",
)
class CallMetricsUpdateRequest(BaseModel):
enabled: bool
class RetryKeepPartialResponse(BaseModel):
enabled: bool = Field(
default=True,
@@ -579,6 +689,7 @@ class RoundtableRoomLimitResponse(BaseModel):
enabled: bool = True
max_rooms: int = 3
max_hold_minutes: int = 30
lease_minutes: int = 5
max_cycles: int = 8
write_file_compatibility_enabled: bool = False
@@ -587,6 +698,7 @@ class RoundtableRoomLimitUpdateRequest(BaseModel):
enabled: bool | None = None
max_rooms: int | None = Field(default=None, ge=1, le=20)
max_hold_minutes: int | None = Field(default=None, ge=1, le=1440)
lease_minutes: int | None = Field(default=None, ge=1, le=60)
max_cycles: int | None = Field(default=None, ge=2, le=100)
write_file_compatibility_enabled: bool | None = None
@@ -667,6 +779,32 @@ async def update_write_file_compatibility_setting(
)
@router.get("/call-metrics", response_model=CallMetricsResponse)
async def get_call_metrics_setting(request: Request) -> CallMetricsResponse:
"""Return the administrator-controlled tool/skill metrics switch."""
await _require_admin(request)
return CallMetricsResponse(enabled=load_system_settings().call_metrics.enabled)
@router.put("/call-metrics", response_model=CallMetricsResponse)
async def update_call_metrics_setting(
body: CallMetricsUpdateRequest,
request: Request,
) -> CallMetricsResponse:
"""Enable or pause new tool/skill metric writes without a restart."""
await _require_admin(request)
current = load_system_settings()
saved = save_system_settings(
current.model_copy(
update={"call_metrics": CallMetricsSettings(enabled=body.enabled)}
)
)
logger.info("Tool and skill call metrics switch changed: enabled=%s", saved.call_metrics.enabled)
return CallMetricsResponse(enabled=saved.call_metrics.enabled)
@router.get("/retry-keep-partial", response_model=RetryKeepPartialResponse)
async def get_retry_keep_partial_setting() -> RetryKeepPartialResponse:
"""Return the retry-keeps-partial display switch (readable by chat clients)."""
@@ -734,6 +872,7 @@ def _roundtable_room_limit_response(settings: RoundtableRoomLimitSettings) -> Ro
enabled=settings.enabled,
max_rooms=settings.max_rooms,
max_hold_minutes=settings.max_hold_minutes,
lease_minutes=settings.lease_minutes,
max_cycles=settings.max_cycles,
write_file_compatibility_enabled=settings.write_file_compatibility_enabled,
)
@@ -760,6 +899,8 @@ async def update_roundtable_room_limit_setting(
updates["max_rooms"] = body.max_rooms
if body.max_hold_minutes is not None:
updates["max_hold_minutes"] = body.max_hold_minutes
if body.lease_minutes is not None:
updates["lease_minutes"] = body.lease_minutes
if body.max_cycles is not None:
updates["max_cycles"] = body.max_cycles
if body.write_file_compatibility_enabled is not None:
@@ -767,10 +908,11 @@ async def update_roundtable_room_limit_setting(
updated = current.roundtable_room_limit.model_copy(update=updates)
saved = save_system_settings(current.model_copy(update={"roundtable_room_limit": updated}))
logger.info(
"Roundtable runtime limits changed: enabled=%s max_rooms=%s max_hold_minutes=%s max_cycles=%s write_file_compatibility=%s",
"Roundtable runtime limits changed: enabled=%s max_rooms=%s max_hold_minutes=%s lease_minutes=%s max_cycles=%s write_file_compatibility=%s",
saved.roundtable_room_limit.enabled,
saved.roundtable_room_limit.max_rooms,
saved.roundtable_room_limit.max_hold_minutes,
saved.roundtable_room_limit.lease_minutes,
saved.roundtable_room_limit.max_cycles,
saved.roundtable_room_limit.write_file_compatibility_enabled,
)
@@ -917,6 +1059,42 @@ async def update_skill_compression(body: SkillCompressionUpdateRequest, request:
return _skill_compression_to_response(saved.skill_compression)
@router.get("/skill-versioning", response_model=SkillVersioningResponse)
async def get_skill_versioning() -> SkillVersioningResponse:
"""Return the custom-skill versioning gate for skill detail clients."""
return SkillVersioningResponse(enabled=load_system_settings().skill_versioning.enabled)
@router.put("/skill-versioning", response_model=SkillVersioningResponse)
async def update_skill_versioning(
body: SkillVersioningUpdateRequest,
request: Request,
) -> SkillVersioningResponse:
"""Enable/disable custom-skill version capture and switching. Admin only."""
await _require_admin(request)
current = load_system_settings()
previous = current.skill_versioning.enabled
updated = SkillVersioningSettings(enabled=body.enabled)
saved = save_system_settings(current.model_copy(update={"skill_versioning": updated}))
if body.enabled and not previous:
try:
from deerflow.skills.versioning import reconcile_custom_skill_versions
actor = await get_optional_user_from_request(request)
actor_id = str(getattr(actor, "id", "") or "") or None
await asyncio.to_thread(reconcile_custom_skill_versions, author=actor_id)
except Exception:
logger.exception("Failed to capture custom-skill baselines after enabling versioning")
# The switch remains enabled. Individual skills are bootstrapped
# lazily when their version list is first opened or next changed.
logger.info("Custom skill versioning switch changed: enabled=%s", saved.skill_versioning.enabled)
return SkillVersioningResponse(enabled=saved.skill_versioning.enabled)
@router.get("/model-concurrency", response_model=ModelConcurrencyResponse)
async def get_model_concurrency(request: Request) -> ModelConcurrencyResponse:
"""Return the per-model concurrency-limit configuration. Admin only."""
@@ -19,6 +19,7 @@ from fastapi.responses import FileResponse
from pydantic import BaseModel, Field
from app.gateway.deps import get_checkpointer, get_thread_share_store, get_thread_store
from app.gateway.routers.artifacts import build_html_offline_bundle_response
from app.gateway.thread_copy import copy_thread_state
from deerflow.config.paths import get_paths
from deerflow.persistence.thread_meta.base import ThreadMetaStore
@@ -316,6 +317,7 @@ def _public_share_disposition(mime_type: str) -> str:
async def get_public_share_file(
share_code: str,
path: str,
offline_bundle: bool = False,
share_store: ThreadShareStore = Depends(get_thread_share_store),
) -> FileResponse:
"""Serve a file referenced by a view-shared conversation.
@@ -342,6 +344,11 @@ async def get_public_share_file(
if not actual_path.exists() or not actual_path.is_file():
raise HTTPException(status_code=404, detail="File not found")
if offline_bundle:
if actual_path.suffix.lower() not in {".htm", ".html", ".xhtml"}:
raise HTTPException(status_code=400, detail="Offline bundles are available for HTML files only")
return await build_html_offline_bundle_response(actual_path)
mime_type = mimetypes.guess_type(actual_path.name)[0] or "application/octet-stream"
return FileResponse(
actual_path,
@@ -252,6 +252,29 @@ async def refresh_skill_catalog_projection(
return len(catalog)
async def set_skill_catalog_enabled(name: str, enabled: bool) -> bool:
"""Persist one skill's online flag in the catalog row the list API reads.
The extensions file is the runtime source of truth. The skills page reads
``skills.enabled`` instead, so a toggle that only rewrites the file looks
unchanged after the list refreshes. Updating this single row avoids a full
catalog rebuild, which rewrites every skill and can roll the flag back if
an unrelated row fails.
"""
session_factory = get_session_factory()
if session_factory is None or not name:
return False
async with session_factory() as session:
result = await session.execute(
update(SkillRow)
.where(SkillRow.name == name)
.values(enabled=bool(enabled), updated_at=datetime.now(UTC))
)
await session.commit()
return int(result.rowcount or 0) > 0
def _build_catalog_payload(skills: list[Any], usage_entries: dict[str, Any]) -> list[dict[str, Any]]:
payload: list[dict[str, Any]] = []
defaults = {
+7 -3
View File
@@ -2,9 +2,10 @@
config_version: 8
log_level: info
# 高并发运行时优化。技能视图两项不写也默认开启,要关掉才写成 false。修改后需重启后台。
# 高并发运行时优化。工具/技能统计与技能视图默认开启;若确有写库压力才改成 false。修改后需重启后台。
runtime_performance:
tool_metrics_enabled: false # 暂停工具/技能统计写库;不影响问答与工具执行。恢复统计时设 true 并重启
tool_metrics_enabled: true # 记录普通工具调用,供综合管理的工具排行/调用统计使用
skill_metrics_enabled: true # 记录可归因到具体技能的调用,供综合管理的技能排行/调用统计使用
skill_view_reuse_enabled: true # 默认开启;false 才关闭。单次工具调用复用一次技能视图
skill_view_cache_enabled: true # 默认开启;false 才关闭。校验版本后复用未变化的技能视图
model_timing_nonblocking_enabled: true # 模型计时日志独立队列,不争抢工具线程池
@@ -661,7 +662,10 @@ ai_writing:
# admin_password: "$WEKNORA_ADMIN_PASSWORD"
llmwiki:
weknora:
api_base_url: "http://119.254.86.251:8088/deerflow-api"
# The restored WeKnora deployment exposes its API at the site root.
# The historical /deerflow-api prefix is a frontend-only route there and
# turns API mutations (POST/OPTIONS) into nginx 405 responses.
api_base_url: "http://119.254.86.251:8088"
web_base_url: "http://119.254.86.251:8088"
admin_email: "lqq@qq.com"
admin_password: "1qaz@WSX"
@@ -217,9 +217,9 @@ or_(
---
---
## 十、已知限制
## 十、实现状态与已知限制
1. **`use_count` 永不增长**:`bump_use()` 已定义但全后端无调用点。"使用一个技能"的语义模糊(技能是 prompt 知识,不是工具),尚未接线。`view_count`(`skill_view` 触发)与 `patch_count`(回滚触发)是正常工作的。
1. **`use_count` 按问答轮次统计**:成功 `skill_view`、读取 `/mnt/skills/{public,custom}/<name>/...` 文件或通过 `bash` 执行该目录脚本,都视为真实调用;同一 run 内同一技能只加一次,不依赖模型自述。计数同步写入 `.usage.json` 与技能列表数据库投影。
2. **`.usage.json` 全局单文件**:不按用户分文件;因技能名全局唯一,一个技能只有一份 state/计数,这是设计取舍。
3. **enabled 开关**:`extensions_config.json` 的技能 `enabled` 是全局的(非 per-user),前端开关已移除,归档功能覆盖了"对 Agent 隐藏技能"的需求。
4. **历史物理归档**:早期 curator 曾把技能目录物理移到 `custom/.archive/`,这些遗留目录不会出现在新的归档抽屉里(列表 API 扫不到);新流程已统一为软归档。
@@ -42,6 +42,15 @@
},
"intent-clarification-policy-copy": {
"enabled": true
},
"multi-seat-delivery-gating": {
"enabled": true
},
"stage-handoff-chat-delivery": {
"enabled": true
},
"leaflet-tile-zoom-guard": {
"enabled": true
}
}
}
@@ -264,17 +264,17 @@ def _get_enabled_skills():
def _get_enabled_skills_for_config(app_config: AppConfig | None = None) -> list[Skill]:
"""Return enabled skills using the caller's config source.
Resolve storage from the caller's config and reuse the parsed catalogue only
when its root matches. A cold cache or a different root falls back to that
storage, keeping request-scoped skill directories isolated.
Resolve storage from the caller's config and reuse the parsed catalogue only
when its root matches. A cold cache or a different root falls back to that
storage, keeping request-scoped skill directories isolated.
"""
if app_config is None:
return _get_enabled_skills()
storage = get_or_new_skill_storage(app_config=app_config)
warm = get_warm_enabled_skills(storage)
if warm is not None:
return warm
return _apply_always_on(_drop_archived_skills(list(storage.load_skills(enabled_only=True))))
storage = get_or_new_skill_storage(app_config=app_config)
warm = get_warm_enabled_skills(storage)
if warm is not None:
return warm
return _apply_always_on(_drop_archived_skills(list(storage.load_skills(enabled_only=True))))
def _skill_mutability_label(category: SkillCategory | str) -> str:
@@ -617,44 +617,44 @@ task(description="Oracle Cloud analysis", prompt="...", subagent_type="general-p
</subagent_system>"""
_ORDINARY_QA_HEADING_RULES = {
"hybrid_decimal": """- 一级章节使用 `## 一、标题`,二级章节使用 `### 1.1 标题`,三级章节使用 `#### 1.1.1 标题`。
- 二级章节按 `1.1`、`1.2`、`2.1` 的形式编号,三级及更深层级继续使用完整多级阿拉伯数字。
- 确需更深层级时,四级章节使用 `##### 1.1.1.1 标题`,五级章节使用 `###### 1.1.1.1.1 标题`。""",
"official_chinese": """- 一级章节使用 `## 一、标题`,二级章节使用 `### (一) 标题`,三级章节使用 `#### 1. 标题`。
- 确需更深层级时,四级章节使用 `##### (1) 标题`,五级章节使用 `###### 1) 标题`。""",
"decimal": """- 一级章节使用 `## 1 标题`,二级章节使用 `### 1.1 标题`,三级章节使用 `#### 1.1.1 标题`。
- 确需更深层级时,继续使用 `##### 1.1.1.1 标题`、`###### 1.1.1.1.1 标题`。""",
"chapter_section": """- 一级章节使用 `## 第一章 标题`,二级章节使用 `### 第一节 标题`,三级章节使用 `#### 一、标题`。
- 确需更深层级时,四级章节使用 `##### (一) 标题`,五级章节使用 `###### 1. 标题`。""",
"none": """- 各级章节只写标题文字,不手写编号:`## 一级标题`、`### 二级标题`、`#### 三级标题`。
- 确需更深层级时使用 `##### 四级标题`、`###### 五级标题`,仍不添加编号。""",
}
def _word_heading_numbering_style() -> str:
"""Return the configured Word heading-numbering preset with a safe default."""
try:
return load_system_settings().prompt_prefix.word_heading_numbering_style
except Exception:
logger.debug("Failed to read the Word heading-numbering style; using default", exc_info=True)
return "hybrid_decimal"
def _build_ordinary_qa_markdown_format_section() -> str:
rules = _ORDINARY_QA_HEADING_RULES.get(
_word_heading_numbering_style(),
_ORDINARY_QA_HEADING_RULES["hybrid_decimal"],
)
return f"""
<ordinary_qa_markdown_format>
当普通问答需要输出带章节结构的较长内容时,必须采用以下 Markdown 规范,以便下载为 Word 后得到正确的自动编号和公文标题格式:
- 文档主标题使用 `# 主标题`;主标题只写标题文字,不添加章节编号。
{rules}
- 默认禁止使用以 `-`、`*`、`+` 或 `•` 开头的无序圆点列表;并列内容优先改写为完整自然段,确需逐项表达时使用有序列表。只有用户明确要求无序列表时才使用。
- 简短问答不必为了套用格式而强行添加标题;本规范只约束实际使用的标题和列表。
</ordinary_qa_markdown_format>
"""
_ORDINARY_QA_HEADING_RULES = {
"hybrid_decimal": """- 一级章节使用 `## 一、标题`,二级章节使用 `### 1.1 标题`,三级章节使用 `#### 1.1.1 标题`。
- 二级章节按 `1.1`、`1.2`、`2.1` 的形式编号,三级及更深层级继续使用完整多级阿拉伯数字。
- 确需更深层级时,四级章节使用 `##### 1.1.1.1 标题`,五级章节使用 `###### 1.1.1.1.1 标题`。""",
"official_chinese": """- 一级章节使用 `## 一、标题`,二级章节使用 `### (一) 标题`,三级章节使用 `#### 1. 标题`。
- 确需更深层级时,四级章节使用 `##### (1) 标题`,五级章节使用 `###### 1) 标题`。""",
"decimal": """- 一级章节使用 `## 1 标题`,二级章节使用 `### 1.1 标题`,三级章节使用 `#### 1.1.1 标题`。
- 确需更深层级时,继续使用 `##### 1.1.1.1 标题`、`###### 1.1.1.1.1 标题`。""",
"chapter_section": """- 一级章节使用 `## 第一章 标题`,二级章节使用 `### 第一节 标题`,三级章节使用 `#### 一、标题`。
- 确需更深层级时,四级章节使用 `##### (一) 标题`,五级章节使用 `###### 1. 标题`。""",
"none": """- 各级章节只写标题文字,不手写编号:`## 一级标题`、`### 二级标题`、`#### 三级标题`。
- 确需更深层级时使用 `##### 四级标题`、`###### 五级标题`,仍不添加编号。""",
}
def _word_heading_numbering_style() -> str:
"""Return the configured Word heading-numbering preset with a safe default."""
try:
return load_system_settings().prompt_prefix.word_heading_numbering_style
except Exception:
logger.debug("Failed to read the Word heading-numbering style; using default", exc_info=True)
return "hybrid_decimal"
def _build_ordinary_qa_markdown_format_section() -> str:
rules = _ORDINARY_QA_HEADING_RULES.get(
_word_heading_numbering_style(),
_ORDINARY_QA_HEADING_RULES["hybrid_decimal"],
)
return f"""
<ordinary_qa_markdown_format>
当普通问答需要输出带章节结构的较长内容时,必须采用以下 Markdown 规范,以便下载为 Word 后得到正确的自动编号和公文标题格式:
- 文档主标题使用 `# 主标题`;主标题只写标题文字,不添加章节编号。
{rules}
- 默认禁止使用以 `-`、`*`、`+` 或 `•` 开头的无序圆点列表;并列内容优先改写为完整自然段,确需逐项表达时使用有序列表。只有用户明确要求无序列表时才使用。
- 简短问答不必为了套用格式而强行添加标题;本规范只约束实际使用的标题和列表。
</ordinary_qa_markdown_format>
"""
_ORDINARY_QA_VISUALIZATION_SECTION = """
@@ -1261,13 +1261,13 @@ RAG/reference display mode is enabled.
return prompt
def _get_memory_context(
agent_name: str | None = None,
*,
app_config: AppConfig | None = None,
injection_enabled: bool = True,
include_legacy_knowledge_routing: bool = True,
) -> str:
def _get_memory_context(
agent_name: str | None = None,
*,
app_config: AppConfig | None = None,
injection_enabled: bool = True,
include_legacy_knowledge_routing: bool = True,
) -> str:
"""组装注入系统提示词的记忆上下文 (Memory V2)。
通过 :class:`MemoryManager` 收集各 provider 的记忆块。P1 期只有 builtin
@@ -1278,10 +1278,10 @@ def _get_memory_context(
agent_name: agent 标识;``None`` 时按 "default" 处理。
app_config: 显式应用配置 (本函数实际按 per-user 覆盖解析生效配置)。
injection_enabled: 运行时开关;``False`` 时跳过注入直接返回空串。
include_legacy_knowledge_routing: 是否注入 ``/mnt/knowledge/routing.md``。
智能普通知识库检索已有数据库授权范围和专用检索工具时必须关闭,
避免旧文件路由把“这个库”错误引向沙箱目录;其它记忆 provider
仍按原配置保留。
include_legacy_knowledge_routing: 是否注入 ``/mnt/knowledge/routing.md``。
智能普通知识库检索已有数据库授权范围和专用检索工具时必须关闭,
避免旧文件路由把“这个库”错误引向沙箱目录;其它记忆 provider
仍按原配置保留。
Returns:
包在 ``<memory>`` 标签里的记忆上下文;未启用或为空时返回空串。
@@ -1306,11 +1306,11 @@ def _get_memory_context(
)
# 只取静态块 (builtin 的 frozen snapshot + Hindsight 的模式说明)。
# Hindsight 的动态召回需要当前 query,走 MemoryMiddleware.before_model。
context_content = manager.build_system_prompt(
excluded_provider_names=(
None if include_legacy_knowledge_routing else {"knowledge"}
)
)
context_content = manager.build_system_prompt(
excluded_provider_names=(
None if include_legacy_knowledge_routing else {"knowledge"}
)
)
if not context_content.strip():
return ""
@@ -1878,7 +1878,7 @@ def apply_prompt_template(
is_bootstrap: bool = False,
app_config: AppConfig | None = None,
memory_injection_enabled: bool = True,
include_legacy_knowledge_routing: bool = True,
include_legacy_knowledge_routing: bool = True,
rag_mode_enabled: bool = False,
is_notebook_mode: bool = False,
notebook_source_ids: list[int] | None = None,
@@ -1887,12 +1887,12 @@ def apply_prompt_template(
storage_agent_id = agent_id if agent_id is not None else agent_name
# Get memory context
memory_context = _get_memory_context(
storage_agent_id,
app_config=app_config,
injection_enabled=memory_injection_enabled,
include_legacy_knowledge_routing=include_legacy_knowledge_routing,
)
memory_context = _get_memory_context(
storage_agent_id,
app_config=app_config,
injection_enabled=memory_injection_enabled,
include_legacy_knowledge_routing=include_legacy_knowledge_routing,
)
# Include subagent section only if enabled (from runtime parameter)
n = max_concurrent_subagents
@@ -2004,8 +2004,8 @@ When a search, knowledge, skill, or MCP tool returns a result list:
if rag_mode_enabled
else ""
)
ordinary_qa_format_section = (
_build_ordinary_qa_markdown_format_section()
ordinary_qa_format_section = (
_build_ordinary_qa_markdown_format_section()
if _ordinary_qa_markdown_format_enabled()
and not is_bootstrap
and not writing_mode
@@ -136,13 +136,18 @@ def _build_runtime_middlewares(
# wrap_tool_call chain: its handler is the raw tool, so it observes the
# unconverted exception (before ToolErrorHandlingMiddleware turns it into
# an error ToolMessage) and the raw return value.
# Omit the recorder entirely while paused: no record allocation, skill-merge
# lookup, or fire-and-forget DB task is created by any tool invocation.
# Keep tool execution/error handling and LLM metrics independent of this flag.
if app_config.runtime_performance.tool_metrics_enabled:
from deerflow.agents.middlewares.tool_metrics_middleware import ToolMetricsMiddleware
middlewares.append(ToolMetricsMiddleware())
# Full auditing can stay paused under high concurrency while the low-volume
# skill-only recorder remains active. In skill-only mode non-skill calls are
# passed through without a merge query or fire-and-forget DB write.
performance = app_config.runtime_performance
if performance.tool_metrics_enabled or performance.skill_metrics_enabled:
from deerflow.agents.middlewares.tool_metrics_middleware import ToolMetricsMiddleware
middlewares.append(
ToolMetricsMiddleware(
skill_only=not performance.tool_metrics_enabled,
)
)
return middlewares
@@ -179,20 +184,20 @@ def build_subagent_runtime_middlewares(
model_name = app_config.models[0].name
model_config = app_config.get_model_config(model_name) if model_name else None
if model_config is not None and model_config.supports_vision:
from deerflow.agents.middlewares.view_image_middleware import ViewImageMiddleware
middlewares.append(ViewImageMiddleware())
# Must remain the innermost model-call wrapper: subagent-specific context,
# vision payloads, system prompts and tool schemas all need to be included
# in the final input-size decision.
from deerflow.agents.middlewares.context_window_guard_middleware import ContextWindowGuardMiddleware
middlewares.append(
ContextWindowGuardMiddleware(
max_input_tokens=model_config.max_input_tokens if model_config is not None else None,
)
)
return middlewares
if model_config is not None and model_config.supports_vision:
from deerflow.agents.middlewares.view_image_middleware import ViewImageMiddleware
middlewares.append(ViewImageMiddleware())
# Must remain the innermost model-call wrapper: subagent-specific context,
# vision payloads, system prompts and tool schemas all need to be included
# in the final input-size decision.
from deerflow.agents.middlewares.context_window_guard_middleware import ContextWindowGuardMiddleware
middlewares.append(
ContextWindowGuardMiddleware(
max_input_tokens=model_config.max_input_tokens if model_config is not None else None,
)
)
return middlewares
@@ -1,6 +1,7 @@
"""Middleware that records per-tool-call metrics for the admin audit page.
Every tool / skill invocation ends with one row in ``tool_call_metrics``:
While the administrator-controlled call-metrics switch is enabled, every tool
/ skill invocation ends with one row in ``tool_call_metrics``:
- Successes record the wall-clock duration.
- Failures record a coarse ``error_category`` plus a one-line message.
@@ -40,6 +41,7 @@ from langgraph.errors import GraphBubbleUp
from langgraph.prebuilt.tool_node import ToolCallRequest
from langgraph.types import Command
from deerflow.config.system_settings import get_call_metrics_settings
from deerflow.persistence.engine import fire_and_forget_db_write, get_session_factory
from deerflow.persistence.tool_metrics import ToolMetricRecord, make_tool_metrics_store
from deerflow.persistence.tool_metrics.base import ToolMetricsStore
@@ -188,11 +190,18 @@ def _evaluate_result(result: Any, tool_name: str | None) -> tuple[str, str | Non
class ToolMetricsMiddleware(AgentMiddleware[AgentState]):
"""Wrap each tool call to time it and persist a metric row."""
"""Time tool calls, optionally persisting only attributable skill calls."""
def __init__(self, *, store: ToolMetricsStore | None = None, **kwargs: Any) -> None:
def __init__(
self,
*,
store: ToolMetricsStore | None = None,
skill_only: bool = False,
**kwargs: Any,
) -> None:
super().__init__(**kwargs)
self._store_override = store
self._skill_only = skill_only
# -- store / context ----------------------------------------------------
@@ -205,6 +214,12 @@ class ToolMetricsMiddleware(AgentMiddleware[AgentState]):
return None
return make_tool_metrics_store(sf)
@staticmethod
def _metrics_enabled() -> bool:
"""Read the administrator switch at call time so no restart is needed."""
return bool(get_call_metrics_settings().enabled)
@staticmethod
def _runtime_lookup(request: ToolCallRequest, keys: tuple[str, ...]) -> str | None:
runtime = getattr(request, "runtime", None)
@@ -280,7 +295,11 @@ class ToolMetricsMiddleware(AgentMiddleware[AgentState]):
handler: Callable[[ToolCallRequest], ToolMessage | Command],
) -> ToolMessage | Command:
record = self._build_base_record(request)
if is_internal_tool_name(record.tool_name):
if (
not self._metrics_enabled()
or is_internal_tool_name(record.tool_name)
or (self._skill_only and not record.skill_name)
):
return handler(request)
started = time.monotonic()
try:
@@ -303,7 +322,11 @@ class ToolMetricsMiddleware(AgentMiddleware[AgentState]):
handler: Callable[[ToolCallRequest], Awaitable[ToolMessage | Command]],
) -> ToolMessage | Command:
record = self._build_base_record(request)
if is_internal_tool_name(record.tool_name):
if (
not self._metrics_enabled()
or is_internal_tool_name(record.tool_name)
or (self._skill_only and not record.skill_name)
):
return await handler(request)
started = time.monotonic()
try:
@@ -482,13 +482,14 @@ def _load_and_cache_app_config(config_path: str | None = None) -> AppConfig:
)
performance = _app_config.runtime_performance
logger.info(
"RUNTIME_PERFORMANCE skill_view_reuse_enabled=%s skill_view_cache_enabled=%s model_timing_nonblocking_enabled=%s asyncio_thread_pool_enabled=%s asyncio_thread_pool_size=%d tool_metrics_enabled=%s",
"RUNTIME_PERFORMANCE skill_view_reuse_enabled=%s skill_view_cache_enabled=%s model_timing_nonblocking_enabled=%s asyncio_thread_pool_enabled=%s asyncio_thread_pool_size=%d tool_metrics_enabled=%s skill_metrics_enabled=%s",
performance.skill_view_reuse_enabled,
performance.skill_view_cache_enabled,
performance.model_timing_nonblocking_enabled,
performance.asyncio_thread_pool_enabled,
performance.asyncio_thread_pool_size,
performance.tool_metrics_enabled,
performance.skill_metrics_enabled,
)
return _app_config
@@ -2,9 +2,11 @@
Skill-view reuse and cache are on when those keys are absent. Set them to
false in config.yaml to restore a full refresh on every tool call. Model
timing and the Gateway thread pool stay off unless configured. If application
configuration cannot be loaded, every switch stays off. Tool-call metric writes
are paused by default; explicitly enable them to resume audit statistics.
timing and the Gateway thread pool stay off unless configured. Tool and skill
metrics are on by default so the management leaderboard has a complete source
of truth. Deployments that need to shed this best-effort audit write load can
explicitly pause either metric type in ``config.yaml``. If configuration
cannot be loaded, the fallback keeps every switch off.
"""
from pydantic import BaseModel, ConfigDict, Field
@@ -15,7 +17,14 @@ class RuntimePerformanceConfig(BaseModel):
skill_view_reuse_enabled: bool = Field(default=True, description="Reuse one verified skill view within each sandbox tool invocation. Set false to refresh on every call.")
skill_view_cache_enabled: bool = Field(default=True, description="Reuse unchanged skill views after validating source, target and availability versions. Set false to rebuild every call.")
tool_metrics_enabled: bool = Field(default=False, description="Persist tool/skill audit statistics to tool_call_metrics. Disabled by default to avoid competing with interactive database traffic; historical reads remain available.")
tool_metrics_enabled: bool = Field(
default=True,
description="Persist tool/skill audit statistics to tool_call_metrics. Enabled by default so management leaderboard tool rankings receive data; set false only to pause ordinary tool auditing.",
)
skill_metrics_enabled: bool = Field(
default=True,
description="Persist calls that can be attributed to a concrete skill, keeping skill statistics available even if ordinary tool auditing is paused.",
)
model_timing_nonblocking_enabled: bool = Field(default=False, description="Dispatch model timing logs through a dedicated bounded queue, outside the shared tool executor")
asyncio_thread_pool_enabled: bool = Field(default=False, description="Override the Gateway event loop's default executor at startup; disabled preserves Python defaults")
asyncio_thread_pool_size: int = Field(default=100, ge=1, le=1024, strict=True, description="Maximum Gateway executor threads when enabled, unless overridden by the documented environment variables")
@@ -24,7 +33,12 @@ class RuntimePerformanceConfig(BaseModel):
# Field defaults turn the skill-view fast path on. This fallback is only for
# callers that cannot read application configuration, and it keeps every
# switch off.
_DEFAULT = RuntimePerformanceConfig(skill_view_reuse_enabled=False, skill_view_cache_enabled=False)
_DEFAULT = RuntimePerformanceConfig(
skill_view_reuse_enabled=False,
skill_view_cache_enabled=False,
tool_metrics_enabled=False,
skill_metrics_enabled=False,
)
def get_runtime_performance_config() -> RuntimePerformanceConfig:
@@ -21,7 +21,7 @@ from pathlib import Path
from typing import Literal
from uuid import uuid4
from pydantic import BaseModel, ConfigDict, Field, field_validator
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
from deerflow.config.llmwiki_config import LlmWikiRuntimeOverride
from deerflow.config.runtime_paths import runtime_home
@@ -474,6 +474,20 @@ class SkillCompressionSettings(BaseModel):
model_config = ConfigDict(extra="ignore")
class SkillVersioningSettings(BaseModel):
"""Deployment-wide gate for custom-skill package version history.
The default is intentionally disabled so existing deployments retain the
legacy in-place editing behaviour until an administrator opts in.
"""
enabled: bool = Field(
default=False,
description="Whether custom-skill package versioning and switching are enabled",
)
model_config = ConfigDict(extra="ignore")
class AgentModuleSettings(BaseModel):
"""Admin-configured「按类型聚合」module → selected agent-tag ids.
@@ -668,6 +682,24 @@ class WriteFileCompatibilitySettings(BaseModel):
model_config = ConfigDict(extra="ignore")
class CallMetricsSettings(BaseModel):
"""Administrator-controlled recording for tool and skill call statistics.
Both rankings use the same ``tool_call_metrics`` audit stream. Keeping a
single switch prevents a partially enabled configuration where the
management page can show one ranking but not the other.
"""
enabled: bool = Field(
default=True,
description=(
"Whether new tool and skill calls are persisted for management "
"statistics. Existing records are retained when disabled."
),
)
model_config = ConfigDict(extra="ignore")
class RetryKeepPartialSettings(BaseModel):
"""Keep already-streamed partial answers visible across silent LLM retries.
@@ -717,6 +749,15 @@ class RoundtableRoomLimitSettings(BaseModel):
"but never extend this deadline."
),
)
lease_minutes: int = Field(
default=5,
ge=1,
le=60,
description=(
"How long a foreground step-2 lease stays occupied without a renewal. "
"The browser heartbeat renews it; this does not extend max_hold_minutes."
),
)
max_cycles: int = Field(
default=8,
ge=2,
@@ -769,7 +810,40 @@ class HtmlMapSettings(BaseModel):
model_config = ConfigDict(extra="ignore")
_GENERATED_PAGE_LOGO_FILE_RE = re.compile(r"^logo-[a-f0-9]{32}\.(?:png|jpg|webp)$")
class GeneratedPageBrandingSettings(BaseModel):
"""Optional generated-page logo settings retained for offline bundles."""
chrome_enabled: bool = True
header_layout: Literal["left", "center"] = "center"
system_name: str | None = Field(default=None, max_length=80)
footer_text: str | None = Field(default=None, max_length=160)
logo_filename: str | None = None
logo_mode: Literal["custom", "default", "none"] = "custom"
logo_media_type: Literal["image/png", "image/jpeg", "image/webp"] | None = None
logo_version: str | None = Field(default=None, max_length=64)
model_config = ConfigDict(extra="ignore")
@field_validator("logo_filename")
@classmethod
def _validate_logo_filename(cls, value: str | None) -> str | None:
if value is None:
return None
normalized = str(value).strip().lower()
if not _GENERATED_PAGE_LOGO_FILE_RE.fullmatch(normalized):
raise ValueError("Invalid generated-page logo filename")
return normalized
_APP_IDENTITY_LOGO_FILE_RE = re.compile(r"^logo-[a-f0-9]{32}\.(?:png|jpg|webp)$")
_APP_IDENTITY_LIGHT_RED_HEADER_FILE_RE = re.compile(
r"^light-red-header-[a-f0-9]{32}\.(?:png|jpg|webp|svg)$"
)
_APP_IDENTITY_LIGHT_RED_ACTIVE_FILE_RE = re.compile(
r"^light-red-active-[a-f0-9]{32}\.(?:png|jpg|webp|svg)$"
)
class AppIdentitySettings(BaseModel):
@@ -780,9 +854,21 @@ class AppIdentitySettings(BaseModel):
"""
system_name: str = Field(default="", max_length=80)
header_title_font_mode: Literal["adaptive", "fixed"] = "adaptive"
header_title_font_size: int = Field(default=30, ge=16, le=40)
logo_filename: str | None = None
logo_media_type: Literal["image/png", "image/jpeg", "image/webp"] | None = None
logo_version: str | None = None
light_red_header_background_filename: str | None = None
light_red_header_background_media_type: Literal[
"image/png", "image/jpeg", "image/webp", "image/svg+xml"
] | None = None
light_red_header_background_version: str | None = None
light_red_nav_active_filename: str | None = None
light_red_nav_active_media_type: Literal[
"image/png", "image/jpeg", "image/webp", "image/svg+xml"
] | None = None
light_red_nav_active_version: str | None = None
model_config = ConfigDict(extra="ignore")
@field_validator("system_name")
@@ -801,6 +887,26 @@ class AppIdentitySettings(BaseModel):
raise ValueError("Invalid application logo filename")
return normalized
@field_validator("light_red_header_background_filename")
@classmethod
def _validate_light_red_header_filename(cls, value: str | None) -> str | None:
if value is None:
return None
normalized = str(value).strip().lower()
if not _APP_IDENTITY_LIGHT_RED_HEADER_FILE_RE.fullmatch(normalized):
raise ValueError("Invalid light-red header background filename")
return normalized
@field_validator("light_red_nav_active_filename")
@classmethod
def _validate_light_red_active_filename(cls, value: str | None) -> str | None:
if value is None:
return None
normalized = str(value).strip().lower()
if not _APP_IDENTITY_LIGHT_RED_ACTIVE_FILE_RE.fullmatch(normalized):
raise ValueError("Invalid light-red active navigation filename")
return normalized
def app_identity_logo_path(
settings: AppIdentitySettings | None = None,
@@ -814,6 +920,42 @@ def app_identity_logo_path(
return path if path.is_file() else None
def app_identity_light_red_header_background_path(
settings: AppIdentitySettings | None = None,
) -> Path | None:
"""Resolve the uploaded light-red header background inside runtime storage."""
current = settings or load_system_settings().app_identity
if not current.light_red_header_background_filename:
return None
path = runtime_home() / "app-identity" / current.light_red_header_background_filename
return path if path.is_file() else None
def app_identity_light_red_nav_active_path(
settings: AppIdentitySettings | None = None,
) -> Path | None:
"""Resolve the uploaded light-red active-navigation image inside runtime storage."""
current = settings or load_system_settings().app_identity
if not current.light_red_nav_active_filename:
return None
path = runtime_home() / "app-identity" / current.light_red_nav_active_filename
return path if path.is_file() else None
def generated_page_logo_path(
settings: GeneratedPageBrandingSettings | None = None,
) -> Path | None:
"""Resolve the custom generated-page logo without accepting arbitrary paths."""
current = settings or load_system_settings().generated_page_branding
if current.logo_mode != "custom" or not current.logo_filename:
return None
path = runtime_home() / "generated-page-branding" / current.logo_filename
return path if path.is_file() else None
class HtmlRuntimeSettings(BaseModel):
"""Addresses written into newly generated HTML pages.
@@ -844,6 +986,108 @@ class StrategyProjectSettings(BaseModel):
model_config = ConfigDict(extra="ignore")
class GeneralQaPageDescription(BaseModel):
"""Administrator-provided copy for the ordinary Q&A landing page."""
welcome_title: str = Field(default="", max_length=80)
welcome_description: str = Field(default="", max_length=500)
input_placeholder: str = Field(default="", max_length=120)
model_config = ConfigDict(extra="ignore")
@field_validator("welcome_title", "welcome_description", "input_placeholder")
@classmethod
def _normalize_copy(cls, value: str | None) -> str:
return str(value or "").strip()
class GeneralQaPageDescriptions(BaseModel):
"""Separate ordinary-Q&A copy for the full and compact layouts."""
full: GeneralQaPageDescription = Field(default_factory=GeneralQaPageDescription)
compact: GeneralQaPageDescription = Field(default_factory=GeneralQaPageDescription)
model_config = ConfigDict(extra="ignore")
@model_validator(mode="before")
@classmethod
def _migrate_legacy_single_description(cls, value: object) -> object:
if isinstance(value, GeneralQaPageDescription):
return {"full": value, "compact": value}
if not isinstance(value, dict):
return value
if "full" in value or "compact" in value:
return value
legacy = {
key: value.get(key, "")
for key in ("welcome_title", "welcome_description", "input_placeholder")
}
return {"full": legacy, "compact": legacy}
class PageHeaderDescription(BaseModel):
"""Configurable title and explanatory copy for a product landing page."""
title: str = Field(default="", max_length=80)
description: str = Field(default="", max_length=500)
model_config = ConfigDict(extra="ignore")
@field_validator("title", "description")
@classmethod
def _normalize_copy(cls, value: str | None) -> str:
return str(value or "").strip()
class AIWritingGuideCard(BaseModel):
"""Editable text for one of the fixed AI-writing welcome cards."""
title: str = Field(default="", max_length=80)
description: str = Field(default="", max_length=240)
model_config = ConfigDict(extra="ignore")
@field_validator("title", "description")
@classmethod
def _normalize_copy(cls, value: str | None) -> str:
return str(value or "").strip()
def _empty_ai_writing_guide_cards() -> list[AIWritingGuideCard]:
return [AIWritingGuideCard() for _ in range(3)]
class AIWritingPageDescription(PageHeaderDescription):
"""Configurable copy for the classic AI-writing welcome page."""
guide_cards: list[AIWritingGuideCard] = Field(
default_factory=_empty_ai_writing_guide_cards,
max_length=3,
)
@model_validator(mode="before")
@classmethod
def _keep_three_guide_cards(cls, value: object) -> object:
if not isinstance(value, dict) or not isinstance(value.get("guide_cards"), list):
return value
cards = value["guide_cards"][:3]
return {**value, "guide_cards": [*cards, *({} for _ in range(3 - len(cards)))]}
class PageDescriptionsSettings(BaseModel):
"""Page-scoped, administrator-managed display copy."""
general_qa: GeneralQaPageDescriptions = Field(default_factory=GeneralQaPageDescriptions)
knowledge_base: PageHeaderDescription = Field(default_factory=PageHeaderDescription)
ai_writing: AIWritingPageDescription = Field(default_factory=AIWritingPageDescription)
strategy_projects: PageHeaderDescription = Field(default_factory=PageHeaderDescription)
deep_research: PageHeaderDescription = Field(default_factory=PageHeaderDescription)
gongwen: PageHeaderDescription = Field(default_factory=PageHeaderDescription)
light_app_management: PageHeaderDescription = Field(default_factory=PageHeaderDescription)
light_app_list: PageHeaderDescription = Field(default_factory=PageHeaderDescription)
agents: PageHeaderDescription = Field(default_factory=PageHeaderDescription)
business_chains: PageHeaderDescription = Field(default_factory=PageHeaderDescription)
skills: PageHeaderDescription = Field(default_factory=PageHeaderDescription)
knowledge_management: PageHeaderDescription = Field(default_factory=PageHeaderDescription)
model_config = ConfigDict(extra="ignore")
class ModelPromptSuffix(BaseModel):
"""Administrator-owned instructions appended to one configured model's system prompt."""
@@ -862,9 +1106,14 @@ class SystemSettings(BaseModel):
private_square: PrivateSquareSettings = Field(default_factory=PrivateSquareSettings)
publish_squares: PublishSquaresSettings = Field(default_factory=PublishSquaresSettings)
appearance: AppearanceSettings = Field(default_factory=AppearanceSettings)
generated_page_branding: GeneratedPageBrandingSettings = Field(
default_factory=GeneratedPageBrandingSettings
)
page_descriptions: PageDescriptionsSettings = Field(default_factory=PageDescriptionsSettings)
citation_display: CitationDisplaySettings = Field(default_factory=CitationDisplaySettings)
leaderboard: LeaderboardSettings = Field(default_factory=LeaderboardSettings)
skill_compression: SkillCompressionSettings = Field(default_factory=SkillCompressionSettings)
skill_versioning: SkillVersioningSettings = Field(default_factory=SkillVersioningSettings)
plugin_management: PluginManagementSettings = Field(default_factory=PluginManagementSettings)
agent_modules: AgentModuleSettings = Field(default_factory=AgentModuleSettings)
wx_common_agents: WxCommonAgentsSettings = Field(default_factory=WxCommonAgentsSettings)
@@ -875,6 +1124,7 @@ class SystemSettings(BaseModel):
write_file_compatibility: WriteFileCompatibilitySettings = Field(
default_factory=WriteFileCompatibilitySettings
)
call_metrics: CallMetricsSettings = Field(default_factory=CallMetricsSettings)
retry_keep_partial: RetryKeepPartialSettings = Field(
default_factory=RetryKeepPartialSettings
)
@@ -1007,6 +1257,19 @@ def get_write_file_compatibility_settings() -> WriteFileCompatibilitySettings:
return WriteFileCompatibilitySettings()
def get_call_metrics_settings() -> CallMetricsSettings:
"""Return the live call-metrics switch, defaulting to recording enabled."""
try:
return load_system_settings().call_metrics
except Exception:
logger.debug(
"Failed to read call metrics setting; keeping metrics enabled",
exc_info=True,
)
return CallMetricsSettings()
def get_roundtable_room_limit_settings() -> RoundtableRoomLimitSettings:
"""Return the admin-controlled roundtable room cap, defaulting to 3 enabled rooms."""
try:
@@ -0,0 +1,321 @@
"""Build portable ZIP bundles for generated HTML artifacts.
Generated pages intentionally reference the Gateway's vendored browser runtime
instead of public CDNs. A plain ``.html`` download therefore stops working
after it is moved away from the DeerFlow deployment. This module rewrites
those trusted Gateway URLs to relative paths and copies the corresponding
packaged assets into a ZIP. It never downloads arbitrary third-party URLs.
"""
from __future__ import annotations
import hashlib
import json
import re
import zipfile
from dataclasses import dataclass
from importlib.resources import files
from pathlib import Path, PurePosixPath
from deerflow.html_runtime.public_images import resolve_html_image_path, verify_html_image_token
_STATIC_ASSETS = {
"alpine.min.js",
"bootstrap.bundle.min.js",
"bootstrap.min.css",
"brand-logo-dark.png",
"echarts.min.js",
"g6.min.js",
"leaflet.css",
"leaflet.js",
"topbar-dark.svg",
"topbar-light.svg",
}
_GEO_MAP_ASSETS = {
"china-provinces.geojson",
"japan-prefectures.geojson",
"taiwan-counties.geojson",
"us-states.geojson",
}
_COGNITIVE_ASSET_SUFFIXES = {
".css",
".js",
".json",
".jpeg",
".jpg",
".mjs",
".png",
".svg",
".woff2",
}
_LICENSE_BY_ASSET = {
"alpine.min.js": "LICENSE.alpine.txt",
"bootstrap.bundle.min.js": "LICENSE.bootstrap.txt",
"bootstrap.min.css": "LICENSE.bootstrap.txt",
"echarts.min.js": "LICENSE.echarts.txt",
"g6.min.js": "LICENSE.g6.txt",
"leaflet.css": "LICENSE.leaflet.txt",
"leaflet.js": "LICENSE.leaflet.txt",
}
_RUNTIME_ARCHIVE_ROOT = PurePosixPath("assets/html-runtime")
_HTML_IMAGES_ARCHIVE_ROOT = PurePosixPath("assets/html-images")
_STANDARD_RUNTIME_BASE_RE = (
r"(?:https?://[^/\"'\s<>()]+)?"
r"(?:/[A-Za-z0-9._~!$&()*+,;=:@%+-]+)*/api/public/html-runtime"
)
_HTML_IMAGE_URL_RE = re.compile(
r"(?P<url>(?:https?://[^/\"'\s<>()]+)?"
r"(?:/[A-Za-z0-9._~!$&()*+,;=:@%+-]+)*/api/public/html-images/"
r"(?P<token>[^\"'\s<>()?#]+)(?:\?[^\"'\s<>()#]*)?)",
re.IGNORECASE,
)
_EXTERNAL_URL_RE = re.compile(r"https?://[^\s\"'<>]+", re.IGNORECASE)
@dataclass(frozen=True)
class OfflineBundleResult:
"""Summary of one completed offline bundle."""
bundled_assets: tuple[str, ...]
external_urls: tuple[str, ...]
def _runtime_base_pattern(runtime_base_url: str) -> str:
bases = [_STANDARD_RUNTIME_BASE_RE]
configured = str(runtime_base_url or "").strip().rstrip("/")
if configured:
bases.insert(0, re.escape(configured))
return "(?:" + "|".join(bases) + ")"
def _safe_runtime_source(runtime_dir: Path, relative_path: str) -> Path | None:
"""Resolve only assets that the public HTML-runtime router exposes."""
normalized = relative_path.strip().lstrip("/").replace("\\", "/")
pure_path = PurePosixPath(normalized)
if not normalized or pure_path.is_absolute() or ".." in pure_path.parts:
return None
allowed = False
if len(pure_path.parts) == 1 and normalized in _STATIC_ASSETS:
allowed = True
elif len(pure_path.parts) == 2 and pure_path.parts[0] == "maps" and pure_path.name in _GEO_MAP_ASSETS:
allowed = True
elif pure_path.parts and pure_path.parts[0] == "cognitive-situation-map":
allowed = pure_path.suffix.lower() in _COGNITIVE_ASSET_SUFFIXES
if not allowed:
return None
root = runtime_dir.resolve()
source = (root / Path(*pure_path.parts)).resolve()
try:
source.relative_to(root)
except ValueError:
return None
return source if source.is_file() else None
def _add_file(
archive_files: dict[str, Path],
archive_path: PurePosixPath,
source: Path,
) -> None:
archive_files[archive_path.as_posix()] = source
def _add_cognitive_runtime(archive_files: dict[str, Path], runtime_dir: Path) -> None:
source_root = runtime_dir / "cognitive-situation-map"
if not source_root.is_dir():
return
for source in source_root.rglob("*"):
if not source.is_file():
continue
is_license = source.name.lower().startswith("license")
if source.suffix.lower() not in _COGNITIVE_ASSET_SUFFIXES and not is_license:
continue
relative = source.relative_to(runtime_dir)
_add_file(archive_files, _RUNTIME_ARCHIVE_ROOT / PurePosixPath(relative.as_posix()), source)
def _generated_logo_destination(generated_logo_path: Path | None) -> PurePosixPath:
suffix = generated_logo_path.suffix.lower() if generated_logo_path is not None else ".png"
if suffix not in {".jpg", ".jpeg", ".png", ".webp"}:
suffix = ".png"
return _RUNTIME_ARCHIVE_ROOT / f"generated-page-logo{suffix}"
def _runtime_script(
runtime_dir: Path,
*,
map_tile_url: str,
include_geo_maps: bool,
) -> str:
config: dict[str, object] = {"geoMaps": {}, "mapTileUrl": map_tile_url}
if include_geo_maps:
config["geoMaps"] = {
Path(filename).stem: (_RUNTIME_ARCHIVE_ROOT / "maps" / filename).as_posix()
for filename in sorted(_GEO_MAP_ASSETS)
}
prefix = "window.DEERFLOW_HTML_RUNTIME_CONFIG=" + json.dumps(
config,
ensure_ascii=False,
separators=(",", ":"),
) + ";\n"
return prefix + (runtime_dir / "deerflow-runtime.js").read_text(encoding="utf-8")
def _readme(original_name: str, external_urls: list[str]) -> str:
external_note = (
"未打包的外部地址已记录在 manifest.json 的 external_urls 中;例如在线地图瓦片、"
"用户主动引用的第三方图片或接口仍需要网络。"
if external_urls
else "本页没有检测到仍需联网的 http(s) 地址。"
)
return (
"DeerFlow 网页离线包\n"
"====================\n\n"
f"原始文件:{original_name}\n"
"入口文件:index.html\n"
"依赖目录:assets/\n\n"
"部署方式:请保持目录结构不变,将解压后的全部文件一起放到任意静态网站目录。\n"
"若页面含 GeoJSON 地图等 fetch 请求,不建议直接双击 file:// 打开;可使用 Nginx、"
"Apache、IIS 或其他静态 HTTP 服务访问 index.html。\n\n"
f"{external_note}\n"
)
def build_html_offline_bundle(
html_path: Path,
destination: Path,
*,
runtime_base_url: str,
map_tile_url: str,
generated_logo_path: Path | None = None,
runtime_dir: Path | None = None,
) -> OfflineBundleResult:
"""Write a portable HTML ZIP to *destination* and return its manifest summary."""
html_path = Path(html_path)
destination = Path(destination)
runtime_dir = runtime_dir or Path(str(files("deerflow.html_runtime.assets")))
runtime_dir = Path(runtime_dir)
html = html_path.read_text(encoding="utf-8", errors="replace")
archive_files: dict[str, Path] = {}
generated_files: dict[str, str] = {}
bundled_assets: set[str] = set()
licenses: set[str] = set()
include_geo_maps = bool(re.search(r"data-df-geo\s*=", html, re.IGNORECASE))
uses_leaflet_map = bool(re.search(r"data-df-map\s*=", html, re.IGNORECASE))
base_pattern = _runtime_base_pattern(runtime_base_url)
cognitive_base_re = re.compile(
rf"(?P<url>{base_pattern}/cognitive-situation-map)(?=$|[\"'\s<>()/])",
re.IGNORECASE,
)
def replace_cognitive_base(match: re.Match[str]) -> str:
_add_cognitive_runtime(archive_files, runtime_dir)
bundled_assets.add((_RUNTIME_ARCHIVE_ROOT / "cognitive-situation-map").as_posix() + "/")
return (_RUNTIME_ARCHIVE_ROOT / "cognitive-situation-map").as_posix()
html = cognitive_base_re.sub(replace_cognitive_base, html)
runtime_url_re = re.compile(
rf"(?P<url>{base_pattern}/(?P<asset>[A-Za-z0-9._/-]+)"
r"(?:\?[^\"'\s<>()#]*)?)",
re.IGNORECASE,
)
def replace_runtime_url(match: re.Match[str]) -> str:
asset = match.group("asset").strip("/")
if asset == "generated-page-logo":
source = generated_logo_path if generated_logo_path and generated_logo_path.is_file() else runtime_dir / "brand-logo-dark.png"
if not source.is_file():
return match.group("url")
archive_path = _generated_logo_destination(source)
_add_file(archive_files, archive_path, source)
bundled_assets.add(archive_path.as_posix())
return archive_path.as_posix()
if asset == "deerflow-runtime.js":
source = runtime_dir / asset
if not source.is_file():
return match.group("url")
archive_path = _RUNTIME_ARCHIVE_ROOT / asset
generated_files[archive_path.as_posix()] = _runtime_script(
runtime_dir,
map_tile_url=map_tile_url,
include_geo_maps=include_geo_maps,
)
bundled_assets.add(archive_path.as_posix())
if include_geo_maps:
for filename in sorted(_GEO_MAP_ASSETS):
map_source = _safe_runtime_source(runtime_dir, f"maps/{filename}")
if map_source is not None:
map_archive_path = _RUNTIME_ARCHIVE_ROOT / "maps" / filename
_add_file(archive_files, map_archive_path, map_source)
bundled_assets.add(map_archive_path.as_posix())
return archive_path.as_posix()
source = _safe_runtime_source(runtime_dir, asset)
if source is None:
return match.group("url")
archive_path = _RUNTIME_ARCHIVE_ROOT / PurePosixPath(asset)
_add_file(archive_files, archive_path, source)
bundled_assets.add(archive_path.as_posix())
license_name = _LICENSE_BY_ASSET.get(asset)
if license_name and (runtime_dir / license_name).is_file():
licenses.add(license_name)
return archive_path.as_posix()
html = runtime_url_re.sub(replace_runtime_url, html)
def replace_html_image(match: re.Match[str]) -> str:
token = match.group("token")
try:
_thread_id, _kind, filename = verify_html_image_token(token)
source = resolve_html_image_path(_thread_id, _kind, filename)
except (FileNotFoundError, ValueError):
return match.group("url")
digest = hashlib.sha256(token.encode("utf-8")).hexdigest()[:12]
safe_name = Path(filename).name
archive_path = _HTML_IMAGES_ARCHIVE_ROOT / f"{digest}-{safe_name}"
_add_file(archive_files, archive_path, source)
bundled_assets.add(archive_path.as_posix())
return archive_path.as_posix()
html = _HTML_IMAGE_URL_RE.sub(replace_html_image, html)
external_url_set = {
match.group(0).rstrip("),.;")
for match in _EXTERNAL_URL_RE.finditer(html)
}
if uses_leaflet_map and str(map_tile_url).lower().startswith(("http://", "https://")):
external_url_set.add(str(map_tile_url))
external_urls = sorted(external_url_set)
manifest = {
"format": "deerflow-html-offline-bundle-v1",
"entry": "index.html",
"source_name": html_path.name,
"bundled_assets": sorted(bundled_assets),
"external_urls": external_urls,
}
destination.parent.mkdir(parents=True, exist_ok=True)
with zipfile.ZipFile(destination, "w", compression=zipfile.ZIP_DEFLATED, compresslevel=6) as archive:
archive.writestr("index.html", html.encode("utf-8"))
archive.writestr(
"manifest.json",
json.dumps(manifest, ensure_ascii=False, indent=2).encode("utf-8"),
)
archive.writestr("README.txt", _readme(html_path.name, external_urls).encode("utf-8"))
for archive_path, source in sorted(archive_files.items()):
archive.write(source, archive_path)
for archive_path, content in sorted(generated_files.items()):
archive.writestr(archive_path, content.encode("utf-8"))
for license_name in sorted(licenses):
archive.write(runtime_dir / license_name, f"THIRD_PARTY_LICENSES/{license_name}")
return OfflineBundleResult(
bundled_assets=tuple(sorted(bundled_assets)),
external_urls=tuple(external_urls),
)
@@ -36,7 +36,9 @@ from deerflow.persistence.roundtable_jobs.model import RoundtableJobRow
from deerflow.persistence.roundtable_jobs.sql import compute_active_dedupe_key
from deerflow.persistence.roundtable_rooms.model import RoundtableRoomLeaseRow
LEASE_SECONDS = 60
DEFAULT_LEASE_SECONDS = 5 * 60
MIN_LEASE_SECONDS = 60
MAX_LEASE_SECONDS = 60 * 60
JOB_LIVE_SECONDS = 15 * 60
OCCUPYING_JOB_STATUSES = ("queued", "running", "awaiting_input", "cancel_requested")
# A background job keeps running while Step 3 generates its artifacts, but the
@@ -80,6 +82,15 @@ def _job_is_live(
return updated is not None and updated >= live_after
def normalize_lease_seconds(value: int | None) -> int:
"""Clamp a foreground room lease to the supported 1–60 minute window."""
try:
seconds = int(value) if value is not None else DEFAULT_LEASE_SECONDS
except (TypeError, ValueError):
seconds = DEFAULT_LEASE_SECONDS
return max(MIN_LEASE_SECONDS, min(MAX_LEASE_SECONDS, seconds))
def _hard_deadline_reached(
created_at: datetime | None,
now: datetime,
@@ -104,6 +115,7 @@ class RoundtableRoomRepository:
enabled: bool,
max_rooms: int,
max_hold_minutes: int = 30,
lease_seconds: int = DEFAULT_LEASE_SECONDS,
extra_keys: set[str] | Mapping[str, str | None] | None = None,
) -> dict[str, Any]:
"""Atomically reserve/renew a room, or reject when the pool is full."""
@@ -119,6 +131,7 @@ class RoundtableRoomRepository:
enabled=enabled,
max_rooms=max_rooms,
max_hold_minutes=max_hold_minutes,
lease_seconds=lease_seconds,
extra_keys=extra_keys,
)
@@ -133,6 +146,7 @@ class RoundtableRoomRepository:
enabled: bool,
max_rooms: int,
max_hold_minutes: int = 30,
lease_seconds: int = DEFAULT_LEASE_SECONDS,
extra_keys: set[str] | Mapping[str, str | None] | None = None,
) -> dict[str, Any]:
"""Atomically admit a pre-resolved room key.
@@ -201,6 +215,7 @@ class RoundtableRoomRepository:
task_id=task_id,
holder_user_id=holder_user_id,
now=now,
lease_seconds=lease_seconds,
)
await session.commit()
return self._decision(
@@ -272,6 +287,7 @@ class RoundtableRoomRepository:
enabled: bool,
max_rooms: int,
max_hold_minutes: int = 30,
lease_seconds: int = DEFAULT_LEASE_SECONDS,
extra_keys: set[str] | Mapping[str, str | None] | None = None,
) -> dict[str, Any]:
"""Renew one live foreground lease without ever creating it.
@@ -298,6 +314,7 @@ class RoundtableRoomRepository:
enabled=enabled,
max_rooms=max_rooms,
max_hold_minutes=max_hold_minutes,
lease_seconds=lease_seconds,
extra_keys=extra_keys,
)
@@ -309,6 +326,7 @@ class RoundtableRoomRepository:
enabled: bool,
max_rooms: int,
max_hold_minutes: int = 30,
lease_seconds: int = DEFAULT_LEASE_SECONDS,
extra_keys: set[str] | Mapping[str, str | None] | None = None,
) -> dict[str, Any]:
"""Renew a resolved persisted key without creating a missing lease."""
@@ -354,7 +372,7 @@ class RoundtableRoomRepository:
message="会商连接已断开,请重新恢复研讨。" if enabled else "",
)
row.lease_until = now + timedelta(seconds=LEASE_SECONDS)
row.lease_until = now + timedelta(seconds=normalize_lease_seconds(lease_seconds))
row.updated_at = now
await session.commit()
return self._decision(
@@ -538,9 +556,10 @@ class RoundtableRoomRepository:
task_id: str | None,
holder_user_id: str | None,
now: datetime,
lease_seconds: int = DEFAULT_LEASE_SECONDS,
) -> None:
row = await session.get(RoundtableRoomLeaseRow, room_key)
lease_until = now + timedelta(seconds=LEASE_SECONDS)
lease_until = now + timedelta(seconds=normalize_lease_seconds(lease_seconds))
if row is None:
session.add(
RoundtableRoomLeaseRow(
@@ -314,43 +314,41 @@ def _resolve_skills_path(path: str) -> str:
return _join_path_preserving_style(skills_host, relative)
def _custom_skill_name_from_path(path: str) -> str | None:
"""从虚拟技能路径里解析出 custom 技能名,用于"被使用"埋点。
只识别 ``{skills_container}/custom/{name}/...`` 形式 —— public 技能不参与
策展,给它们埋点只会往 ``.usage.json`` 写无用条目。读技能目录下任意文件
(SKILL.md 或 references/templates/scripts 支持文件)都算一次该技能的使用。
解析不出技能名时返回 None。
"""
skills_container = _get_skills_container_path()
if not (path == skills_container or path.startswith(f"{skills_container}/")):
return None
relative = path[len(skills_container) :].lstrip("/")
parts = [segment for segment in relative.split("/") if segment]
if len(parts) >= 2 and parts[0] == "custom":
return parts[1]
return None
def _record_skill_read(virtual_path: str) -> None:
"""技能文件被 ``read_file`` 读取时记一次 use —— 技能"被使用"的真实信号。
技能正文不进 system prompt(prompt 里只有 name/description/location),Agent
按 Progressive Loading 用 ``read_file`` 加载 SKILL.md 才算真正用到该技能。
curator 的年龄流转据此判断闲置,没有这个埋点会把每天都在用的技能误判为陈旧。
best-effort:埋点失败绝不影响 ``read_file`` 本身。
"""
def _record_skill_paths(
virtual_paths: list[str],
runtime: ToolRuntime[ContextT, ThreadState] | None,
) -> None:
"""Record public/custom skills from real tool paths, independent of prose."""
try:
name = _custom_skill_name_from_path(virtual_path)
if name is None:
return
from deerflow.skills.usage import bump_use
from deerflow.sandbox.skill_view import skill_names_for_paths
from deerflow.skills.usage import bump_use, runtime_use_event_key
bump_use(name)
event_key = runtime_use_event_key(runtime)
for name in skill_names_for_paths(
virtual_paths,
container_path=_get_skills_container_path(),
):
bump_use(name, event_key=event_key)
except Exception:
pass
def _record_skill_read(
virtual_path: str,
runtime: ToolRuntime[ContextT, ThreadState] | None = None,
) -> None:
"""Count a successful skill file read as a logical skill invocation."""
_record_skill_paths([virtual_path], runtime)
def _record_skill_command(
command: str,
runtime: ToolRuntime[ContextT, ThreadState] | None = None,
) -> None:
"""Count skills whose scripts are executed directly by ``bash``."""
_record_skill_paths(_skill_paths_in_text(command), runtime)
def _is_acp_workspace_path(path: str) -> bool:
"""Check if a path is under the ACP workspace virtual path."""
return path == _ACP_WORKSPACE_VIRTUAL_PATH or path.startswith(f"{_ACP_WORKSPACE_VIRTUAL_PATH}/")
@@ -1445,6 +1443,7 @@ def bash_tool(runtime: ToolRuntime[ContextT, ThreadState], description: str, com
command: The bash command to execute. Always use absolute paths for files and directories.
"""
try:
requested_command = command
_prepare_command_skill_capability(command)
_validate_enabled_skill_paths_in_command(command)
sandbox = ensure_sandbox_initialized(runtime)
@@ -1457,6 +1456,7 @@ def bash_tool(runtime: ToolRuntime[ContextT, ThreadState], description: str, com
command = replace_virtual_paths_in_command(command, thread_data)
command = _apply_cwd_prefix(command, thread_data)
output = sandbox.execute_command(command)
_record_skill_command(requested_command, runtime)
try:
from deerflow.config.app_config import get_app_config
@@ -1473,7 +1473,9 @@ def bash_tool(runtime: ToolRuntime[ContextT, ThreadState], description: str, com
max_chars = sandbox_cfg.bash_output_max_chars if sandbox_cfg else 20000
except Exception:
max_chars = 20000
return _truncate_bash_output(sandbox.execute_command(command), max_chars)
output = sandbox.execute_command(command)
_record_skill_command(requested_command, runtime)
return _truncate_bash_output(output, max_chars)
except SandboxError as e:
return f"Error: {e}"
except PermissionError as e:
@@ -1751,7 +1753,7 @@ def read_file_tool(
max_result_chars=max_chars,
hard_max_result_chars=hard_max,
)
_record_skill_read(requested_path)
_record_skill_read(requested_path, runtime)
return inspected["rendered"]
except OSError:
pass
@@ -1767,7 +1769,7 @@ def read_file_tool(
)
# 读成功即记一次技能使用 —— requested_path 仍是虚拟路径(本地/容器沙箱都一样),
# 非技能路径会被 _record_skill_read 自行忽略。
_record_skill_read(requested_path)
_record_skill_read(requested_path, runtime)
if not content and not truncated:
return "(empty)"
if truncated:
@@ -7,6 +7,7 @@ import logging
import os
import tempfile
import threading
from collections import OrderedDict
from datetime import UTC, datetime
from pathlib import Path
@@ -19,6 +20,14 @@ _USAGE_FILENAME = ".usage.json"
# 单进程跑,故进程内锁即可保证状态流转不丢。
_usage_lock = threading.Lock()
# One user run can first load SKILL.md and then execute several helper scripts.
# Those are implementation details of one logical skill invocation, so retain a
# small process-local LRU keyed by (run, skill) and increment the public counter
# only once. A run stays on one worker; persistence across restarts is neither
# needed nor desirable here.
_MAX_RECENT_USE_EVENTS = 8192
_recent_use_events: OrderedDict[tuple[str, str], None] = OrderedDict()
STATE_ACTIVE = "active"
STATE_STALE = "stale"
STATE_ARCHIVED = "archived"
@@ -121,8 +130,110 @@ def bump_view(name: str) -> None:
_update_entry(name, lambda e: {"view_count": e.get("view_count", 0) + 1, "last_viewed_at": _now_iso()})
def bump_use(name: str) -> None:
_update_entry(name, lambda e: {"use_count": e.get("use_count", 0) + 1, "last_used_at": _now_iso()})
async def _increment_use_projection(name: str, use_count: int, used_at: datetime) -> None:
"""Keep the MySQL skill-list projection current without importing ``app``."""
try:
from sqlalchemy import case, update
from deerflow.persistence.engine import get_session_factory
from deerflow.persistence.skills.model import SkillRow
session_factory = get_session_factory()
if session_factory is None:
return
async with session_factory() as session:
await session.execute(
update(SkillRow)
.where(SkillRow.name == name)
.values(
# Fire-and-forget writes may finish out of order. Never let
# an older absolute sidecar count overwrite a newer value.
use_count=case((SkillRow.use_count < use_count, use_count), else_=SkillRow.use_count),
last_activity_at=case(
(SkillRow.last_activity_at.is_(None), used_at),
(SkillRow.last_activity_at < used_at, used_at),
else_=SkillRow.last_activity_at,
),
updated_at=case(
(SkillRow.updated_at < used_at, used_at),
else_=SkillRow.updated_at,
),
)
)
await session.commit()
except Exception:
logger.debug("Failed to update skill usage projection for %r", name, exc_info=True)
def _schedule_use_projection_increment(name: str, use_count: int, used_at: datetime) -> None:
"""Best-effort bridge from synchronous sandbox tools to the DB home loop."""
try:
from deerflow.persistence.engine import fire_and_forget_db_write
fire_and_forget_db_write(_increment_use_projection(name, use_count, used_at))
except Exception:
logger.debug("Failed to schedule skill usage projection for %r", name, exc_info=True)
def runtime_use_event_key(runtime: Any | None) -> str | None:
"""Return the stable user-run identifier carried by ``ToolRuntime``."""
if runtime is None:
return None
context = getattr(runtime, "context", None)
if isinstance(context, dict):
for key in ("run_id", "langgraph_run_id"):
value = context.get(key)
if value is not None and str(value).strip():
return str(value).strip()
config = getattr(runtime, "config", None)
if isinstance(config, dict):
for bucket_name in ("configurable", "metadata"):
bucket = config.get(bucket_name)
if not isinstance(bucket, dict):
continue
for key in ("run_id", "langgraph_run_id"):
value = bucket.get(key)
if value is not None and str(value).strip():
return str(value).strip()
return None
def bump_use(name: str, *, event_key: str | None = None) -> bool:
"""Count one logical use, optionally de-duplicated by user run.
Returns ``True`` only when the counter changed. This makes callers and tests
able to distinguish a new invocation from a second file/script access in
the same run.
"""
normalized_name = str(name or "").strip()
if not normalized_name:
return False
normalized_event = str(event_key or "").strip() or None
dedupe_key = (normalized_event, normalized_name) if normalized_event else None
used_at = datetime.now(UTC)
new_count = 0
try:
with _usage_lock:
if dedupe_key is not None and dedupe_key in _recent_use_events:
_recent_use_events.move_to_end(dedupe_key)
return False
data = _read_all()
entry = {**_DEFAULT_ENTRY, **data.get(normalized_name, {})}
new_count = max(0, int(entry.get("use_count") or 0)) + 1
entry.update({"use_count": new_count, "last_used_at": used_at.isoformat()})
data[normalized_name] = entry
_write_all(data)
if dedupe_key is not None:
_recent_use_events[dedupe_key] = None
_recent_use_events.move_to_end(dedupe_key)
while len(_recent_use_events) > _MAX_RECENT_USE_EVENTS:
_recent_use_events.popitem(last=False)
except Exception:
logger.debug("Failed to bump usage entry for %r", normalized_name, exc_info=True)
return False
_schedule_use_projection_increment(normalized_name, new_count, used_at)
return True
def bump_patch(name: str) -> None:
@@ -0,0 +1,397 @@
"""Filesystem-backed version history for custom skills.
Only the active package lives below ``skills/custom``. Historical snapshots
are kept under the writable DeerFlow runtime directory, outside the canonical
skills tree and therefore outside every model-facing discovery/mount path.
The feature is deliberately gated by ``system_settings.skill_versioning``.
When the switch is off, callers keep the legacy write behaviour and this
module performs no snapshot writes.
"""
from __future__ import annotations
import hashlib
import json
import os
import shutil
import tempfile
import threading
from collections.abc import Iterator
from contextlib import contextmanager
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
from uuid import uuid4
from deerflow.config.paths import get_paths
from deerflow.config.system_settings import load_system_settings
from deerflow.skills.storage import SkillStorage, get_or_new_skill_storage
from deerflow.skills.types import SKILL_MD_FILE, SkillCategory
_MANIFEST_FILE = "manifest.json"
_SCHEMA_VERSION = 1
_LOCKS_GUARD = threading.Lock()
_THREAD_LOCKS: dict[str, threading.RLock] = {}
def skill_versioning_enabled() -> bool:
"""Return the deployment-wide custom-skill versioning switch."""
try:
return bool(load_system_settings().skill_versioning.enabled)
except Exception:
return False
def _thread_lock(name: str) -> threading.RLock:
with _LOCKS_GUARD:
return _THREAD_LOCKS.setdefault(name, threading.RLock())
def _versions_root() -> Path:
return get_paths().base_dir / "skill-versions"
def _skill_root(name: str) -> Path:
return _versions_root() / SkillStorage.validate_skill_name(name)
def _manifest_path(name: str) -> Path:
return _skill_root(name) / _MANIFEST_FILE
def _version_dir(name: str, version: int) -> Path:
return _skill_root(name) / f"v{version:06d}"
@contextmanager
def _cross_process_lock(name: str) -> Iterator[None]:
"""Serialize one skill's snapshot/switch operations across workers."""
root = _skill_root(name)
root.mkdir(parents=True, exist_ok=True)
lock_path = root / ".lock"
with _thread_lock(name):
with lock_path.open("a+b") as handle:
if os.name == "nt": # pragma: no cover - exercised on Windows hosts
import msvcrt
if lock_path.stat().st_size == 0:
handle.write(b"0")
handle.flush()
handle.seek(0)
msvcrt.locking(handle.fileno(), msvcrt.LK_LOCK, 1)
else: # pragma: no cover - exercised in Linux CI/deployments
import fcntl
fcntl.flock(handle.fileno(), fcntl.LOCK_EX)
try:
yield
finally:
if os.name == "nt": # pragma: no cover
handle.seek(0)
msvcrt.locking(handle.fileno(), msvcrt.LK_UNLCK, 1)
else: # pragma: no cover
fcntl.flock(handle.fileno(), fcntl.LOCK_UN)
def _empty_manifest(name: str) -> dict[str, Any]:
return {
"schema_version": _SCHEMA_VERSION,
"skill_name": name,
"active_version": None,
"versions": [],
}
def _load_manifest(name: str) -> dict[str, Any]:
path = _manifest_path(name)
if not path.is_file():
return _empty_manifest(name)
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
raise ValueError(f"Skill version manifest for '{name}' is unreadable") from exc
if not isinstance(payload, dict) or payload.get("skill_name") != name:
raise ValueError(f"Skill version manifest for '{name}' is invalid")
versions = payload.get("versions")
if not isinstance(versions, list):
raise ValueError(f"Skill version manifest for '{name}' has invalid versions")
return payload
def _save_manifest(name: str, manifest: dict[str, Any]) -> None:
path = _manifest_path(name)
path.parent.mkdir(parents=True, exist_ok=True)
fd, tmp_name = tempfile.mkstemp(prefix=".manifest-", suffix=".tmp", dir=path.parent)
try:
with os.fdopen(fd, "w", encoding="utf-8") as handle:
json.dump(manifest, handle, ensure_ascii=False, indent=2, sort_keys=True)
handle.write("\n")
os.replace(tmp_name, path)
except Exception:
try:
os.unlink(tmp_name)
except OSError:
pass
raise
def _resolve_custom_dir(storage: SkillStorage, name: str) -> Path:
for skill in storage.load_skills(enabled_only=False):
if skill.name == name and skill.category == SkillCategory.CUSTOM:
return skill.skill_dir.resolve()
raise FileNotFoundError(f"Custom skill '{name}' not found")
def _tree_hash(root: Path) -> str:
digest = hashlib.sha256()
for path in sorted(root.rglob("*"), key=lambda item: item.relative_to(root).as_posix()):
relative = path.relative_to(root).as_posix()
if any(part in {"__pycache__", ".git"} for part in path.relative_to(root).parts):
continue
if path.is_symlink():
digest.update(b"L\0")
digest.update(relative.encode("utf-8"))
digest.update(b"\0")
digest.update(os.readlink(path).encode("utf-8", errors="surrogateescape"))
continue
if not path.is_file():
continue
digest.update(b"F\0")
digest.update(relative.encode("utf-8"))
digest.update(b"\0")
with path.open("rb") as handle:
while chunk := handle.read(1024 * 1024):
digest.update(chunk)
return digest.hexdigest()
def _copy_package(source: Path, destination: Path) -> None:
shutil.copytree(
source,
destination,
symlinks=True,
ignore=shutil.ignore_patterns("__pycache__", ".git"),
)
def record_custom_skill_version(
name: str,
*,
storage: SkillStorage | None = None,
action: str,
author: str | None = None,
summary: str | None = None,
) -> dict[str, Any] | None:
"""Snapshot the current custom package when it differs from active.
Returns the active/new version metadata. No-op (and no filesystem writes)
while the deployment switch is disabled.
"""
if not skill_versioning_enabled():
return None
name = SkillStorage.validate_skill_name(name)
storage = storage or get_or_new_skill_storage()
source = _resolve_custom_dir(storage, name)
current_hash = _tree_hash(source)
with _cross_process_lock(name):
manifest = _load_manifest(name)
active = next(
(item for item in manifest["versions"] if int(item.get("version") or 0) == int(manifest.get("active_version") or 0)),
None,
)
if active is not None and active.get("sha256") == current_hash:
return {**active, "active": True}
next_version = max([int(item.get("version") or 0) for item in manifest["versions"]] or [0]) + 1
destination = _version_dir(name, next_version)
staging = destination.parent / f".{destination.name}.staging-{uuid4().hex}"
_copy_package(source, staging)
try:
os.replace(staging, destination)
except Exception:
shutil.rmtree(staging, ignore_errors=True)
raise
now = datetime.now(UTC).isoformat()
entry = {
"version": next_version,
"sha256": current_hash,
"created_at": now,
"created_by": author,
"action": str(action or "update"),
"summary": str(summary or "").strip() or None,
"based_on_version": manifest.get("active_version"),
}
manifest["versions"].append(entry)
manifest["active_version"] = next_version
manifest["updated_at"] = now
_save_manifest(name, manifest)
return {**entry, "active": True}
def list_custom_skill_versions(
name: str,
*,
storage: SkillStorage | None = None,
author: str | None = None,
) -> dict[str, Any]:
"""Return newest-first metadata, lazily bootstrapping the current package."""
if not skill_versioning_enabled():
return {"enabled": False, "skill_name": name, "active_version": None, "versions": []}
storage = storage or get_or_new_skill_storage()
record_custom_skill_version(
name,
storage=storage,
action="baseline",
author=author,
summary="启用技能迭代时保存的当前版本",
)
with _cross_process_lock(name):
manifest = _load_manifest(name)
active_version = manifest.get("active_version")
versions = [{**item, "active": int(item.get("version") or 0) == int(active_version or 0)} for item in reversed(manifest["versions"])]
return {
"enabled": True,
"skill_name": name,
"active_version": active_version,
"versions": versions,
}
def activate_custom_skill_version(
name: str,
version: int,
*,
storage: SkillStorage | None = None,
) -> dict[str, Any]:
"""Materialize one historical snapshot as the sole model-visible package."""
if not skill_versioning_enabled():
raise RuntimeError("技能迭代功能未开启")
name = SkillStorage.validate_skill_name(name)
storage = storage or get_or_new_skill_storage()
target = _resolve_custom_dir(storage, name)
with _cross_process_lock(name):
manifest = _load_manifest(name)
entry = next(
(item for item in manifest["versions"] if int(item.get("version") or 0) == int(version)),
None,
)
if entry is None:
raise FileNotFoundError(f"Skill '{name}' version {version} not found")
source = _version_dir(name, int(version))
if not source.is_dir() or _tree_hash(source) != entry.get("sha256"):
raise ValueError(f"Skill '{name}' version {version} snapshot is missing or corrupted")
storage.validate_skill_markdown_content(
name,
(source / SKILL_MD_FILE).read_text(encoding="utf-8"),
)
if int(manifest.get("active_version") or 0) == int(version) and _tree_hash(target) == entry.get("sha256"):
return {**entry, "active": True}
parent = target.parent
staging = parent / f".{name}.version-stage-{uuid4().hex}"
backup = parent / f".{name}.version-backup-{uuid4().hex}"
_copy_package(source, staging)
moved_old = False
try:
os.replace(target, backup)
moved_old = True
os.replace(staging, target)
manifest["active_version"] = int(version)
manifest["updated_at"] = datetime.now(UTC).isoformat()
_save_manifest(name, manifest)
except Exception:
shutil.rmtree(staging, ignore_errors=True)
if moved_old:
shutil.rmtree(target, ignore_errors=True)
os.replace(backup, target)
raise
finally:
if backup.exists() and target.exists():
shutil.rmtree(backup, ignore_errors=True)
return {**entry, "active": True}
def delete_custom_skill_versions(name: str) -> None:
"""Remove stored snapshots after the custom skill itself is deleted."""
name = SkillStorage.validate_skill_name(name)
root = _skill_root(name)
if root.exists():
shutil.rmtree(root)
def reset_custom_skill_versions(
name: str,
*,
storage: SkillStorage | None = None,
author: str | None = None,
summary: str | None = None,
) -> dict[str, Any] | None:
"""Discard the previous chain and make the current package the new v1.
This is the explicit "direct overwrite" semantic offered beside
"iterate as a new version" for same-name uploads.
"""
if not skill_versioning_enabled():
return None
name = SkillStorage.validate_skill_name(name)
root = _skill_root(name)
with _cross_process_lock(name):
for child in list(root.iterdir()):
if child.name == ".lock":
continue
if child.is_dir():
shutil.rmtree(child)
else:
child.unlink(missing_ok=True)
return record_custom_skill_version(
name,
storage=storage,
action="overwrite",
author=author,
summary=summary or "直接覆盖后重置版本链",
)
def reconcile_custom_skill_versions(*, storage: SkillStorage | None = None, author: str | None = None) -> int:
"""Capture each current custom skill after the master switch is enabled."""
if not skill_versioning_enabled():
return 0
storage = storage or get_or_new_skill_storage()
captured = 0
for skill in storage.load_skills(enabled_only=False):
if skill.category != SkillCategory.CUSTOM:
continue
before = _load_manifest(skill.name).get("active_version") if _manifest_path(skill.name).exists() else None
result = record_custom_skill_version(
skill.name,
storage=storage,
action="feature_enabled",
author=author,
summary="技能迭代开关开启后捕获当前内容",
)
if result is not None and result.get("version") != before:
captured += 1
return captured
__all__ = [
"activate_custom_skill_version",
"delete_custom_skill_versions",
"list_custom_skill_versions",
"record_custom_skill_version",
"reconcile_custom_skill_versions",
"reset_custom_skill_versions",
"skill_versioning_enabled",
]
@@ -273,7 +273,16 @@ def skill_view_tool(name: str, runtime: ToolRuntime) -> str:
Args:
name: The skill name (hyphen-case, e.g. 'my-skill').
"""
return skill_view_impl(name, _resolve_active_agent_id(runtime), _resolve_extra_skills(runtime))
result = skill_view_impl(name, _resolve_active_agent_id(runtime), _resolve_extra_skills(runtime))
head = result.lstrip()[:200].lower()
if not ((head.startswith("skill '") and "not found" in head) or head.startswith("error reading skill")):
try:
from deerflow.skills.usage import bump_use, runtime_use_event_key
bump_use(name, event_key=runtime_use_event_key(runtime))
except Exception:
pass
return result
def _resolve_skills_container_base_path() -> str:
@@ -161,6 +161,26 @@ async def _publish_managed_skill(name: str, *, removed: bool = False) -> None:
await _sync_managed_skill_view(name)
async def _record_managed_skill_version(
storage: SkillStorage,
name: str,
*,
action: str,
author: str,
summary: str,
) -> None:
from deerflow.skills.versioning import record_custom_skill_version
await _to_thread(
record_custom_skill_version,
name,
storage=storage,
action=action,
author=author,
summary=summary,
)
async def _bump_patch_best_effort(name: str) -> None:
"""记一次技能维护活动(patch / edit)。
@@ -363,6 +383,14 @@ async def _skill_manage_impl(
# async stack — contextvars don't propagate into ``to_thread``.
if skill_store is not None:
await _ensure_skill_db_registered(skill_store, name, user_id)
skill_storage.invalidate_skills_cache()
await _record_managed_skill_version(
skill_storage,
name,
action="agent_create",
author=user_id,
summary="大模型创建技能",
)
await _publish_managed_skill(name)
return f"Created custom skill '{name}'."
@@ -378,6 +406,13 @@ async def _skill_manage_impl(
scan = await _scan_or_raise(content, executable=False, location=f"{name}/{SKILL_MD_FILE}")
skill_file = skill_storage.get_custom_skill_file(name)
prev_content = await _to_thread(skill_file.read_text, encoding="utf-8")
await _record_managed_skill_version(
skill_storage,
name,
action="baseline",
author=user_id,
summary="大模型编辑前的活动版本",
)
await _to_thread(skill_storage.write_custom_skill, name, SKILL_MD_FILE, content)
await _to_thread(
skill_storage.append_history,
@@ -385,6 +420,13 @@ async def _skill_manage_impl(
_history_record(action="edit", file_path=SKILL_MD_FILE, prev_content=prev_content, new_content=content, thread_id=thread_id, scanner=scan),
)
await _bump_patch_best_effort(name)
await _record_managed_skill_version(
skill_storage,
name,
action="agent_edit",
author=user_id,
summary="大模型编辑 SKILL.md",
)
await _publish_managed_skill(name)
return f"Updated custom skill '{name}'."
@@ -406,6 +448,13 @@ async def _skill_manage_impl(
new_content = prev_content.replace(find, replace, replacement_count)
await _to_thread(skill_storage.validate_skill_markdown_content, name, new_content)
scan = await _scan_or_raise(new_content, executable=False, location=f"{name}/{SKILL_MD_FILE}")
await _record_managed_skill_version(
skill_storage,
name,
action="baseline",
author=user_id,
summary="大模型修补前的活动版本",
)
await _to_thread(skill_storage.write_custom_skill, name, SKILL_MD_FILE, new_content)
await _to_thread(
skill_storage.append_history,
@@ -413,6 +462,13 @@ async def _skill_manage_impl(
_history_record(action="patch", file_path=SKILL_MD_FILE, prev_content=prev_content, new_content=new_content, thread_id=thread_id, scanner=scan),
)
await _bump_patch_best_effort(name)
await _record_managed_skill_version(
skill_storage,
name,
action="agent_patch",
author=user_id,
summary="大模型修补 SKILL.md",
)
await _publish_managed_skill(name)
return f"Patched custom skill '{name}' ({replacement_count} replacement(s) applied, {occurrences} match(es) found)."
@@ -453,6 +509,12 @@ async def _skill_manage_impl(
await skill_store.delete(name, user_id)
except Exception:
logger.debug("Failed to delete DB row for %r", name, exc_info=True)
try:
from deerflow.skills.versioning import delete_custom_skill_versions
await _to_thread(delete_custom_skill_versions, name)
except Exception:
logger.debug("Failed to delete version history for %r", name, exc_info=True)
agent_store = _default_agent_store()
if agent_store is not None:
try:
@@ -482,12 +544,26 @@ async def _skill_manage_impl(
prev_content = await _to_thread(target.read_text, encoding="utf-8") if exists else None
executable = "scripts/" in path or path.startswith("scripts/")
scan = await _scan_or_raise(content, executable=executable, location=f"{name}/{path}")
await _record_managed_skill_version(
skill_storage,
name,
action="baseline",
author=user_id,
summary="大模型写文件前的活动版本",
)
await _to_thread(skill_storage.write_custom_skill, name, path, content)
await _to_thread(
skill_storage.append_history,
name,
_history_record(action="write_file", file_path=path, prev_content=prev_content, new_content=content, thread_id=thread_id, scanner=scan),
)
await _record_managed_skill_version(
skill_storage,
name,
action="agent_write_file",
author=user_id,
summary=f"大模型写入文件 {path}",
)
await _sync_managed_skill_view(name)
return f"Wrote '{path}' for custom skill '{name}'."
@@ -502,12 +578,26 @@ async def _skill_manage_impl(
if not await _to_thread(target.exists):
raise FileNotFoundError(f"Supporting file '{path}' not found for skill '{name}'.")
prev_content = await _to_thread(target.read_text, encoding="utf-8")
await _record_managed_skill_version(
skill_storage,
name,
action="baseline",
author=user_id,
summary="大模型删文件前的活动版本",
)
await _to_thread(target.unlink)
await _to_thread(
skill_storage.append_history,
name,
_history_record(action="remove_file", file_path=path, prev_content=prev_content, new_content=None, thread_id=thread_id, scanner={"decision": "allow", "reason": "Deletion requested."}),
)
await _record_managed_skill_version(
skill_storage,
name,
action="agent_remove_file",
author=user_id,
summary=f"大模型删除文件 {path}",
)
await _sync_managed_skill_view(name)
return f"Removed '{path}' from custom skill '{name}'."
@@ -5,26 +5,28 @@ import pytest
from deerflow.config.app_config import AppConfig
_SKILL_VIEW_KEYS = ("skill_view_reuse_enabled", "skill_view_cache_enabled")
_OPT_IN_KEYS = ("model_timing_nonblocking_enabled", "asyncio_thread_pool_enabled", "tool_metrics_enabled")
_DEFAULT_ON_KEYS = (*_SKILL_VIEW_KEYS, "tool_metrics_enabled", "skill_metrics_enabled")
_OPT_IN_KEYS = ("model_timing_nonblocking_enabled", "asyncio_thread_pool_enabled")
def test_missing_runtime_performance_enables_skill_view_only():
def test_missing_runtime_performance_enables_skill_view_and_skill_metrics():
config = AppConfig(sandbox={"use": "deerflow.sandbox.local:LocalSandboxProvider"})
assert config.runtime_performance.model_dump() == {
"skill_view_reuse_enabled": True,
"skill_view_cache_enabled": True,
"tool_metrics_enabled": False,
"tool_metrics_enabled": True,
"skill_metrics_enabled": True,
"model_timing_nonblocking_enabled": False,
"asyncio_thread_pool_enabled": False,
"asyncio_thread_pool_size": 100,
}
@pytest.mark.parametrize("key", [*_SKILL_VIEW_KEYS, *_OPT_IN_KEYS])
@pytest.mark.parametrize("key", [*_DEFAULT_ON_KEYS, *_OPT_IN_KEYS])
def test_runtime_performance_switches_are_independent(tmp_path: Path, key: str):
# Skill-view switches default on, so independence is checked by turning
# one of them off. The opt-in switches default off and are turned on.
enabled = key not in _SKILL_VIEW_KEYS
# Skill-view and audit metrics default on, so independence is checked by
# turning one off. Opt-in switches default off and are turned on.
enabled = key not in _DEFAULT_ON_KEYS
config_path = tmp_path / "config.yaml"
config_path.write_text(
f"sandbox:\n use: deerflow.sandbox.local:LocalSandboxProvider\nruntime_performance:\n {key}: {'true' if enabled else 'false'}\n",
@@ -32,7 +34,7 @@ def test_runtime_performance_switches_are_independent(tmp_path: Path, key: str):
)
flags = AppConfig.from_file(str(config_path)).runtime_performance.model_dump()
assert flags[key] is enabled
for other in _SKILL_VIEW_KEYS:
for other in _DEFAULT_ON_KEYS:
if other != key:
assert flags[other] is True
for other in _OPT_IN_KEYS:
@@ -0,0 +1,95 @@
"""Regression coverage for model-independent skill usage attribution."""
from __future__ import annotations
from pathlib import Path
from types import SimpleNamespace
import pytest
from deerflow.sandbox import tools
from deerflow.skills import usage
from deerflow.tools.builtins import skill_tools
@pytest.fixture(autouse=True)
def _clear_recent_use_events():
usage._recent_use_events.clear()
yield
usage._recent_use_events.clear()
def test_bump_use_counts_a_skill_once_per_run_and_syncs_the_catalog(tmp_path: Path, monkeypatch):
usage_path = tmp_path / ".usage.json"
projected: list[tuple[str, int, object]] = []
monkeypatch.setattr(usage, "_usage_path", lambda: usage_path)
monkeypatch.setattr(
usage,
"_schedule_use_projection_increment",
lambda name, use_count, used_at: projected.append((name, use_count, used_at)),
)
assert usage.bump_use("page-generator", event_key="run-1") is True
assert usage.bump_use("page-generator", event_key="run-1") is False
assert usage.bump_use("page-generator", event_key="run-2") is True
entry = usage.get_entry("page-generator")
assert entry["use_count"] == 2
assert entry["last_used_at"]
assert [(name, count) for name, count, _used_at in projected] == [
("page-generator", 1),
("page-generator", 2),
]
def test_skill_file_and_script_paths_record_public_and_custom_skills(monkeypatch):
calls: list[tuple[str, str | None]] = []
monkeypatch.setattr(
usage,
"bump_use",
lambda name, *, event_key=None: calls.append((name, event_key)),
)
runtime = SimpleNamespace(context={"run_id": "run-weak-model"}, config={})
tools._record_skill_read("/mnt/skills/public/report-maker/SKILL.md", runtime)
tools._record_skill_command(
"python /mnt/skills/custom/data-cleaner/scripts/run.py --fast",
runtime,
)
assert calls == [
("report-maker", "run-weak-model"),
("data-cleaner", "run-weak-model"),
]
def test_skill_view_counts_as_use_without_model_narration(monkeypatch):
calls: list[tuple[str, str | None]] = []
runtime = SimpleNamespace(context={"run_id": "run-skill-view"}, config={})
monkeypatch.setattr(skill_tools, "skill_view_impl", lambda *_args, **_kwargs: "skill instructions")
monkeypatch.setattr(
usage,
"bump_use",
lambda name, *, event_key=None: calls.append((name, event_key)),
)
result = skill_tools.skill_view_tool.func(name="report-maker", runtime=runtime)
assert result == "skill instructions"
assert calls == [("report-maker", "run-skill-view")]
def test_failed_skill_view_does_not_count_as_use(monkeypatch):
calls: list[str] = []
runtime = SimpleNamespace(context={"run_id": "run-missing"}, config={})
monkeypatch.setattr(
skill_tools,
"skill_view_impl",
lambda *_args, **_kwargs: "Skill 'missing' not found.",
)
monkeypatch.setattr(usage, "bump_use", lambda name, **_kwargs: calls.append(name))
result = skill_tools.skill_view_tool.func(name="missing", runtime=runtime)
assert result == "Skill 'missing' not found."
assert calls == []
@@ -288,7 +288,7 @@ def _tool_sandbox(tmp_path: Path, view_root: Path, monkeypatch: pytest.MonkeyPat
monkeypatch.setattr(tools, "is_local_sandbox", lambda _runtime: True)
monkeypatch.setattr(tools, "is_host_bash_allowed", lambda: True)
monkeypatch.setattr(tools, "get_thread_data", lambda _runtime: {"workspace_path": str(tmp_path / "workspace"), "uploads_path": str(tmp_path / "uploads"), "outputs_path": str(tmp_path / "outputs")})
monkeypatch.setattr(tools, "_record_skill_read", lambda _path: None)
monkeypatch.setattr(tools, "_record_skill_read", lambda *_args: None)
monkeypatch.setattr(tools, "_get_skills_container_path", lambda: "/mnt/skills")
(tmp_path / "workspace").mkdir(exist_ok=True)
assert view_root.is_dir()
@@ -326,6 +326,30 @@ def test_bash_runs_the_updated_skill_script(cache_setup, monkeypatch, tmp_path):
assert (view.root / "public" / "online" / "scripts" / "run.py").read_text(encoding="utf-8") == "print('two')\n"
def test_bash_records_a_direct_skill_script_invocation(cache_setup, monkeypatch, tmp_path):
root, _config, storage = cache_setup
script = root / "public" / "online" / "scripts"
script.mkdir()
(script / "run.py").write_text("print('used')\n", encoding="utf-8")
view = skill_view.refresh_skill_sandbox_view(storage=storage)
_tool_sandbox(tmp_path, view.root, monkeypatch)
recorded: list[tuple[str, object]] = []
monkeypatch.setattr(
tools,
"_record_skill_command",
lambda command, runtime: recorded.append((command, runtime)),
)
result = tools.bash_tool.func(
runtime=None,
description="test",
command="python /mnt/skills/public/online/scripts/run.py",
)
assert result == "ok"
assert recorded == [("python /mnt/skills/public/online/scripts/run.py", None)]
def test_bash_rejects_an_offline_skill(cache_setup, monkeypatch, tmp_path):
root, config, storage = cache_setup
_skill(root, "offline")
@@ -50,3 +50,21 @@ def test_canonical_write_file_is_still_recorded(monkeypatch) -> None:
middleware.wrap_tool_call(_request("write_file"), lambda _: _result("write_file"))
flush.assert_called_once()
def test_call_metrics_switch_skips_new_metric_rows(monkeypatch) -> None:
middleware = ToolMetricsMiddleware()
flush = Mock()
monkeypatch.setattr(middleware, "_flush", flush)
monkeypatch.setattr(
"deerflow.agents.middlewares.tool_metrics_middleware.get_call_metrics_settings",
lambda: SimpleNamespace(enabled=False),
)
result = middleware.wrap_tool_call(
_request("write_file"),
lambda _: _result("write_file"),
)
assert result.name == "write_file"
flush.assert_not_called()
@@ -20,8 +20,10 @@ from deerflow.config.app_config import AppConfig
BUILDERS = [build_lead_runtime_middlewares, build_subagent_runtime_middlewares]
def _config(enabled=None):
def _config(enabled=None, *, skill_enabled=None):
options = {} if enabled is None else {"tool_metrics_enabled": enabled}
if skill_enabled is not None:
options["skill_metrics_enabled"] = skill_enabled
return AppConfig(
sandbox={"use": "deerflow.sandbox.local:LocalSandboxProvider"},
runtime_performance=options,
@@ -29,18 +31,29 @@ def _config(enabled=None):
@pytest.mark.parametrize("builder", BUILDERS)
@pytest.mark.parametrize("enabled", [None, False])
def test_paused_runtime_never_constructs_a_recorder(builder, enabled, monkeypatch):
def test_explicitly_paused_runtime_never_constructs_a_recorder(builder, monkeypatch):
constructor = Mock(side_effect=AssertionError("paused runtime constructed a DB recorder"))
monkeypatch.setattr(metrics, "ToolMetricsMiddleware", constructor)
middlewares = builder(app_config=_config(enabled))
middlewares = builder(app_config=_config(False, skill_enabled=False))
constructor.assert_not_called()
assert any(isinstance(m, ToolErrorHandlingMiddleware) for m in middlewares)
assert any(isinstance(m, LlmMetricsMiddleware) for m in middlewares)
@pytest.mark.parametrize("builder", BUILDERS)
def test_default_runtime_keeps_a_full_tool_and_skill_recorder(builder):
recorders = [
middleware
for middleware in builder(app_config=_config())
if isinstance(middleware, metrics.ToolMetricsMiddleware)
]
assert len(recorders) == 1
assert recorders[0]._skill_only is False
def _tool_wrappers(builder, enabled):
# Exercise the actual error/audit wrappers selected by both runtime builders.
return [m for m in builder(app_config=_config(enabled)) if isinstance(m, (ToolErrorHandlingMiddleware, metrics.ToolMetricsMiddleware))]
@@ -77,6 +90,30 @@ def test_sync_tool_execution_and_error_conversion_survive_pause(builder, enabled
assert flushed.call_args.args[0].status == result.status
@pytest.mark.parametrize("builder", BUILDERS)
def test_skill_only_mode_records_a_path_based_skill_call(builder, monkeypatch):
flushed = Mock()
monkeypatch.setattr(metrics.ToolMetricsMiddleware, "_flush", lambda _, record: flushed(record))
request = SimpleNamespace(
tool_call={
"id": "call-skill",
"name": "bash",
"args": {"command": "python /mnt/skills/public/report-maker/scripts/run.py"},
}
)
expected = ToolMessage(content="ok", tool_call_id="call-skill", name="bash")
def handler(_request):
return expected
for middleware in reversed(_tool_wrappers(builder, False)):
handler = partial(middleware.wrap_tool_call, handler=handler)
assert handler(request) is expected
assert flushed.call_count == 1
assert flushed.call_args.args[0].skill_name == "report-maker"
@pytest.mark.asyncio
@pytest.mark.parametrize("builder", BUILDERS)
@pytest.mark.parametrize("enabled", [False, True])
@@ -1,9 +1,16 @@
from __future__ import annotations
from datetime import UTC, datetime
import pytest
from sqlalchemy import select
from sqlalchemy.dialects import mysql
from app.gateway.routers.admin_users import _apply_run_scope, _apply_tool_scope
from app.gateway.routers.admin_users import (
_apply_run_scope,
_apply_tool_scope,
_query_leaderboard_dimension_details,
)
from deerflow.persistence.run.model import RunRow
from deerflow.persistence.tool_metrics.model import ToolCallMetricRow
@@ -41,3 +48,83 @@ def test_tool_aggregation_does_not_join_the_full_runs_table() -> None:
assert sql.count("NOT (EXISTS") == 2
assert "LEFT OUTER JOIN" not in sql
assert " NOT IN " not in sql
class _DetailResult:
def __init__(self, *, rows=None, row=None):
self._rows = rows or []
self._row = row
def all(self):
return self._rows
def one(self):
return self._row
class _DetailSession:
def __init__(self):
self.statements = []
self.results = [
_DetailResult(rows=[("web-search", 3, 2, 0, 125.5)]),
_DetailResult(row=(3, 1, 2, 0, 125.5)),
]
async def execute(self, statement):
self.statements.append(statement)
return self.results.pop(0)
@pytest.mark.asyncio
async def test_skill_detail_uses_metric_table_and_returns_drawer_aggregates() -> None:
session = _DetailSession()
summary, items = await _query_leaderboard_dimension_details(
session,
datetime(2026, 11, 1, tzinfo=UTC),
datetime(2026, 11, 7, tzinfo=UTC),
kind="skill",
limit=500,
exclude_user_ids=set(),
include_scheduled=False,
include_admins=True,
include_failed=True,
)
assert summary.count == 3
assert summary.item_count == 1
assert items[0].name == "web-search"
assert items[0].avg_duration_ms == 125
sql = "\n".join(_mysql_sql(statement) for statement in session.statements)
assert "TOOL_CALL_METRICS" in sql
assert "LEFT OUTER JOIN" not in sql
assert " NOT IN " not in sql
@pytest.mark.asyncio
async def test_agent_detail_returns_token_aggregates_from_runs() -> None:
session = _DetailSession()
session.results = [
_DetailResult(rows=[("lead_agent", 4, 2, 960, 0)]),
_DetailResult(row=(4, 1, 2, 960, 0)),
]
summary, items = await _query_leaderboard_dimension_details(
session,
datetime(2026, 11, 1, tzinfo=UTC),
datetime(2026, 11, 7, tzinfo=UTC),
kind="agent",
limit=500,
exclude_user_ids=set(),
include_scheduled=False,
include_admins=True,
include_failed=True,
)
assert summary.total_tokens == 960
assert items[0].avg_duration_ms == 0
sql = "\n".join(_mysql_sql(statement) for statement in session.statements)
assert "FROM RUNS" in sql
assert "SCHEDULED_TASK_RUNS" in sql
@@ -405,6 +405,23 @@ async def test_skill_catalog_projection_uses_compact_rows_without_long_fields(
assert rows[0]["description"] == "compact metadata"
assert "detail" not in rows[0]
assert "detail" not in {column.key for column in skill_catalog_cache._SKILL_LIST_COLUMNS}
offline = SimpleNamespace(
name="catalog-offline",
description="stays untouched",
license=None,
category="custom",
enabled=True,
always_on=False,
skill_file=skill_file,
)
assert await skill_catalog_cache.refresh_skill_catalog_projection(skills=[skill, offline]) == 2
assert await skill_catalog_cache.set_skill_catalog_enabled("catalog-demo", False) is True
assert await skill_catalog_cache.set_skill_catalog_enabled("missing-skill", False) is False
rows = await skill_catalog_cache.load_skill_catalog_projection(user_id="user-1", admin=True)
by_name = {row["name"]: row for row in rows or []}
assert by_name["catalog-demo"]["enabled"] is False
assert by_name["catalog-offline"]["enabled"] is True
await engine.dispose()
@@ -12,6 +12,7 @@ PNG = bytes.fromhex(
"89504e470d0a1a0a0000000d49484452000000010000000108060000001f15c4"
"890000000a49444154789c63000100000500010d0a2db40000000049454e44ae426082"
)
SVG = b'<svg xmlns="http://www.w3.org/2000/svg" width="1920" height="64"><rect width="1920" height="64" fill="#8f1d14"/></svg>'
def _store(monkeypatch: pytest.MonkeyPatch, tmp_path):
@@ -41,6 +42,10 @@ def _store(monkeypatch: pytest.MonkeyPatch, tmp_path):
def test_new_settings_leave_app_identity_empty() -> None:
assert SystemSettings().app_identity == AppIdentitySettings()
assert SystemSettings().app_identity.header_title_font_mode == "adaptive"
assert SystemSettings().app_identity.header_title_font_size == 30
assert SystemSettings().app_identity.light_red_header_background_filename is None
assert SystemSettings().app_identity.light_red_nav_active_filename is None
@pytest.mark.asyncio
@@ -84,6 +89,58 @@ async def test_admin_saves_name_without_replacing_the_static_logo(
assert current["settings"].app_identity.logo_filename is None
@pytest.mark.asyncio
async def test_admin_saves_header_title_font_settings(
monkeypatch: pytest.MonkeyPatch,
tmp_path,
) -> None:
current = _store(monkeypatch, tmp_path)
response = await app_identity_router.update_app_identity_settings(
SimpleNamespace(),
system_name="研判中心",
remove_logo=False,
logo=None,
header_title_font_mode="fixed",
header_title_font_size=34,
)
assert response.header_title_font_mode == "fixed"
assert response.header_title_font_size == 34
assert current["settings"].app_identity.header_title_font_mode == "fixed"
assert current["settings"].app_identity.header_title_font_size == 34
renamed = await app_identity_router.update_app_identity_settings(
SimpleNamespace(),
system_name="新研判中心",
remove_logo=False,
logo=None,
)
assert renamed.header_title_font_mode == "fixed"
assert renamed.header_title_font_size == 34
@pytest.mark.asyncio
async def test_header_title_fixed_font_size_is_bounded(
monkeypatch: pytest.MonkeyPatch,
tmp_path,
) -> None:
_store(monkeypatch, tmp_path)
with pytest.raises(HTTPException) as error:
await app_identity_router.update_app_identity_settings(
SimpleNamespace(),
system_name="研判中心",
remove_logo=False,
logo=None,
header_title_font_mode="fixed",
header_title_font_size=41,
)
assert error.value.status_code == 422
@pytest.mark.asyncio
async def test_uploaded_logo_is_served_separately_from_generated_pages(
monkeypatch: pytest.MonkeyPatch,
@@ -126,6 +183,91 @@ async def test_invalid_logo_is_rejected(monkeypatch: pytest.MonkeyPatch, tmp_pat
assert error.value.status_code == 422
@pytest.mark.asyncio
async def test_red_navigation_images_can_be_uploaded_and_served(
monkeypatch: pytest.MonkeyPatch,
tmp_path,
) -> None:
current = _store(monkeypatch, tmp_path)
saved = await app_identity_router.update_app_identity_settings(
SimpleNamespace(),
system_name="研判中心",
remove_logo=False,
logo=None,
light_red_header_background=UploadFile(
file=BytesIO(SVG),
filename="header.svg",
),
light_red_nav_active=UploadFile(
file=BytesIO(PNG),
filename="active.png",
),
)
assert saved.has_light_red_header_background is True
assert saved.has_light_red_nav_active is True
assert saved.light_red_header_background_url is not None
assert saved.light_red_header_background_url.startswith(
"/api/public/app-identity/light-red-header-background?v="
)
assert saved.light_red_nav_active_url is not None
assert saved.light_red_nav_active_url.startswith(
"/api/public/app-identity/light-red-nav-active?v="
)
identity = current["settings"].app_identity
header_file = tmp_path / "app-identity" / identity.light_red_header_background_filename
active_file = tmp_path / "app-identity" / identity.light_red_nav_active_filename
assert header_file.is_file()
assert header_file.suffix == ".svg"
assert active_file.is_file()
served_header = await app_identity_router.get_light_red_header_background()
served_active = await app_identity_router.get_light_red_nav_active()
assert served_header.media_type == "image/svg+xml"
assert served_header.headers["content-security-policy"].startswith("sandbox;")
assert served_active.media_type == "image/png"
@pytest.mark.asyncio
async def test_restoring_red_navigation_images_uses_built_in_defaults(
monkeypatch: pytest.MonkeyPatch,
tmp_path,
) -> None:
current = _store(monkeypatch, tmp_path)
await app_identity_router.update_app_identity_settings(
SimpleNamespace(),
system_name="研判中心",
remove_logo=False,
logo=None,
light_red_header_background=UploadFile(
file=BytesIO(PNG),
filename="header.png",
),
light_red_nav_active=UploadFile(
file=BytesIO(PNG),
filename="active.png",
),
)
identity = current["settings"].app_identity
header_file = tmp_path / "app-identity" / identity.light_red_header_background_filename
active_file = tmp_path / "app-identity" / identity.light_red_nav_active_filename
restored = await app_identity_router.update_app_identity_settings(
SimpleNamespace(),
system_name="研判中心",
remove_logo=False,
logo=None,
remove_light_red_header_background=True,
remove_light_red_nav_active=True,
)
assert restored.has_light_red_header_background is False
assert restored.has_light_red_nav_active is False
assert header_file.exists() is False
assert active_file.exists() is False
@pytest.mark.asyncio
async def test_clearing_app_identity_removes_only_its_logo(
monkeypatch: pytest.MonkeyPatch,
@@ -144,6 +286,8 @@ async def test_clearing_app_identity_removes_only_its_logo(
cleared = await app_identity_router.delete_app_identity_settings(SimpleNamespace())
assert cleared.system_name == ""
assert cleared.header_title_font_mode == "adaptive"
assert cleared.header_title_font_size == 30
assert cleared.has_logo is False
assert logo_file.exists() is False
with pytest.raises(HTTPException) as error:
@@ -1,11 +1,15 @@
from __future__ import annotations
import json
import zipfile
from pathlib import Path
from types import SimpleNamespace
import pytest
from app.gateway.routers import artifacts
from deerflow.html_runtime import offline_bundle
from deerflow.html_runtime.offline_bundle import build_html_offline_bundle
from deerflow.uploads.policy import default_upload_policy, set_runtime_upload_policy
@@ -91,3 +95,165 @@ async def test_artifact_preview_rejects_binary_content(tmp_path: Path, monkeypat
max_chars=1024,
)
assert raised.value.status_code == 415
@pytest.mark.asyncio
async def test_html_artifact_offline_bundle_uses_zip_export(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
source = tmp_path / "page.html"
source.write_text("<!doctype html><title>portable</title>", encoding="utf-8")
calls: list[Path] = []
async def resolve(_thread_id: str, _path: str) -> Path:
return source
async def bundle_response(path: Path):
calls.append(path)
return artifacts.Response(content=b"zip-content", media_type="application/zip")
monkeypatch.setattr(artifacts, "aresolve_thread_virtual_path", resolve)
monkeypatch.setattr(artifacts, "build_html_offline_bundle_response", bundle_response)
response = await artifacts.get_artifact.__wrapped__(
thread_id="thread-1",
path="mnt/user-data/outputs/page.html",
request=SimpleNamespace(),
download=True,
offline_bundle=True,
)
assert response.media_type == "application/zip"
assert calls == [source]
def _write_offline_fixture(path: Path, content: str | bytes) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
if isinstance(content, bytes):
path.write_bytes(content)
else:
path.write_text(content, encoding="utf-8")
def _offline_runtime_fixture(root: Path) -> Path:
runtime = root / "runtime"
_write_offline_fixture(runtime / "echarts.min.js", "window.echarts={};")
_write_offline_fixture(runtime / "deerflow-runtime.js", "window.__runtimeLoaded=true;")
_write_offline_fixture(runtime / "brand-logo-dark.png", b"fallback-logo")
_write_offline_fixture(runtime / "LICENSE.echarts.txt", "ECharts license")
for name in (
"china-provinces.geojson",
"japan-prefectures.geojson",
"taiwan-counties.geojson",
"us-states.geojson",
):
_write_offline_fixture(runtime / "maps" / name, '{"type":"FeatureCollection","features":[]}')
return runtime
def test_html_offline_bundle_rewrites_and_packages_gateway_runtime(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
runtime = _offline_runtime_fixture(tmp_path)
logo = tmp_path / "branding" / "logo.webp"
_write_offline_fixture(logo, b"configured-logo")
uploaded_image = tmp_path / "uploads" / "background.png"
_write_offline_fixture(uploaded_image, b"uploaded-image")
monkeypatch.setattr(
offline_bundle,
"verify_html_image_token",
lambda _token: ("thread-1", "uploads", "background.png"),
)
monkeypatch.setattr(
offline_bundle,
"resolve_html_image_path",
lambda _thread_id, _kind, _filename: uploaded_image,
)
source = tmp_path / "态势报告.html"
source.write_text(
"""<!doctype html><html><head>
<script src="http://gateway.local/prefix/api/public/html-runtime/echarts.min.js"></script>
<script src="http://gateway.local/prefix/api/public/html-runtime/deerflow-runtime.js"></script>
<style>.brand{background:url('http://gateway.local/prefix/api/public/html-runtime/generated-page-logo?v=2')}</style>
</head><body><script type="application/json" data-df-geo="map">{"map":"china-provinces"}</script>
<script type="application/json" data-df-map="tile-map">{"markers":[]}</script>
<img src="http://gateway.local/prefix/api/public/html-images/signed.token">
<a href="https://example.com/source">来源</a></body></html>""",
encoding="utf-8",
)
destination = tmp_path / "bundle.zip"
result = build_html_offline_bundle(
source,
destination,
runtime_base_url="http://gateway.local/prefix/api/public/html-runtime",
map_tile_url="https://tiles.example/{z}/{x}/{y}.png",
generated_logo_path=logo,
runtime_dir=runtime,
)
assert destination.is_file()
assert "assets/html-runtime/echarts.min.js" in result.bundled_assets
assert "https://example.com/source" in result.external_urls
assert "https://tiles.example/{z}/{x}/{y}.png" in result.external_urls
with zipfile.ZipFile(destination) as archive:
names = set(archive.namelist())
assert "index.html" in names
assert "README.txt" in names
assert "manifest.json" in names
assert "assets/html-runtime/echarts.min.js" in names
assert "assets/html-runtime/deerflow-runtime.js" in names
assert "assets/html-runtime/generated-page-logo.webp" in names
assert "assets/html-runtime/maps/china-provinces.geojson" in names
assert any(name.startswith("assets/html-images/") for name in names)
assert "THIRD_PARTY_LICENSES/LICENSE.echarts.txt" in names
html = archive.read("index.html").decode("utf-8")
assert "gateway.local" not in html
assert 'src="assets/html-runtime/echarts.min.js"' in html
assert "assets/html-runtime/generated-page-logo.webp" in html
assert "/api/public/html-images/" not in html
runtime_script = archive.read("assets/html-runtime/deerflow-runtime.js").decode("utf-8")
assert "DEERFLOW_HTML_RUNTIME_CONFIG" in runtime_script
assert "assets/html-runtime/maps/china-provinces.geojson" in runtime_script
assert "window.__runtimeLoaded=true" in runtime_script
manifest = json.loads(archive.read("manifest.json"))
assert manifest["entry"] == "index.html"
assert manifest["source_name"] == "态势报告.html"
assert manifest["external_urls"] == [
"https://example.com/source",
"https://tiles.example/{z}/{x}/{y}.png",
]
def test_html_offline_bundle_copies_cognitive_runtime_tree(tmp_path: Path) -> None:
runtime = _offline_runtime_fixture(tmp_path)
_write_offline_fixture(runtime / "cognitive-situation-map" / "app.js", "window.cognitive=true;")
_write_offline_fixture(runtime / "cognitive-situation-map" / "styles.css", "body{color:white}")
_write_offline_fixture(runtime / "cognitive-situation-map" / "assets" / "maps" / "taiwan.png", b"map")
source = tmp_path / "认知态势图.html"
source.write_text(
"""<script>window.__DF_SKILL_BASE__="http://gw/api/public/html-runtime/cognitive-situation-map";</script>
<link rel="stylesheet" href="http://gw/api/public/html-runtime/cognitive-situation-map/styles.css">
<script src="http://gw/api/public/html-runtime/cognitive-situation-map/app.js"></script>""",
encoding="utf-8",
)
destination = tmp_path / "cognitive.zip"
build_html_offline_bundle(
source,
destination,
runtime_base_url="http://gw/api/public/html-runtime",
map_tile_url="https://tiles.example/{z}/{x}/{y}.png",
runtime_dir=runtime,
)
with zipfile.ZipFile(destination) as archive:
html = archive.read("index.html").decode("utf-8")
assert "http://gw" not in html
assert "assets/html-runtime/cognitive-situation-map" in html
assert "assets/html-runtime/cognitive-situation-map/app.js" in archive.namelist()
assert "assets/html-runtime/cognitive-situation-map/assets/maps/taiwan.png" in archive.namelist()
@@ -355,10 +355,7 @@ async def test_weknora_session_endpoint_sets_cookie_and_returns_platform_path(mo
return {"id": "local-kb", "weknora_id": "remote-kb"}
async def list_visible(self, *_: object, **__: object) -> list[dict[str, str]]:
return [
{"id": "local-kb", "weknora_id": "remote-kb"},
{"id": "public-kb", "weknora_id": "other-public-kb"},
]
raise AssertionError("creating an iframe session must not load every visible knowledge base")
monkeypatch.setattr(llmwiki_router, "_runtime_for_iframe_proxy", lambda _: object())
monkeypatch.setattr(llmwiki_router, "_authorized_mapping", authorized_mapping)
@@ -387,7 +384,39 @@ async def test_weknora_session_endpoint_sets_cookie_and_returns_platform_path(mo
assert ctx.mapping_id == "local-kb"
assert ctx.weknora_id == "remote-kb"
assert ctx.can_write is True
assert ctx.allowed_weknora_ids == ("remote-kb", "other-public-kb")
assert ctx.allowed_weknora_ids == ()
assert len(raw) < 1024
@pytest.mark.asyncio
async def test_weknora_embed_picker_hydrates_visible_bases_server_side(monkeypatch: pytest.MonkeyPatch) -> None:
from app.gateway.routers import llmwiki as llmwiki_router
calls: list[dict[str, object]] = []
class Store:
async def list_visible(self, user_id: str, *, scope: str, is_admin: bool) -> list[dict[str, str]]:
calls.append({"user_id": user_id, "scope": scope, "is_admin": is_admin})
return [
{"id": "local-kb", "weknora_id": "remote-kb"},
{"id": "public-kb", "weknora_id": "other-public-kb"},
]
monkeypatch.setattr(llmwiki_router, "_store", lambda _: Store())
ctx = llmwiki_router.WeKnoraEmbedContext(
mapping_id="local-kb",
weknora_id="remote-kb",
user_id="user-a",
is_admin=False,
can_write=True,
exp=4_102_444_800,
)
hydrated = await llmwiki_router._with_visible_embed_knowledge_bases(_request("/platform/knowledge-bases/remote-kb"), ctx)
assert calls == [{"user_id": "user-a", "scope": "all", "is_admin": False}]
assert hydrated.allowed_weknora_ids == ("remote-kb", "other-public-kb")
assert "allowed_weknora_ids" not in llmwiki_router._embed_context_payload(hydrated)
@pytest.mark.asyncio
@@ -2079,6 +2108,9 @@ async def test_prefixed_weknora_proxy_serves_page_static_assets_and_api_without_
async def get_authorized(self, *_: object, **__: object) -> dict[str, str]:
return {"id": "local-kb", "weknora_id": "remote-kb"}
async def list_visible(self, *_: object, **__: object) -> list[dict[str, str]]:
return [{"id": "local-kb", "weknora_id": "remote-kb"}]
runtime = SimpleNamespace(
web_base_url="http://weknora-web.test",
api_base_url="http://weknora-api.test",
@@ -2207,9 +2239,28 @@ async def test_prefixed_weknora_proxy_serves_page_static_assets_and_api_without_
api = await client.get("/deerflow/api/llmwiki/weknora-embed/api/v1/auth/me")
assert api.status_code == 200
directory = await client.get("/deerflow/api/llmwiki/weknora-embed/api/v1/knowledge-bases")
assert directory.status_code == 200
assert directory.json() == {
"success": True,
"data": [
{
"id": "remote-kb",
"name": "",
"description": "",
"type": "document",
"status": "active",
"created_at": None,
"updated_at": None,
}
],
"total": 1,
}
api_request = next(request for request in upstream_requests if request[1].startswith(runtime.api_base_url))
assert api_request[2]["Authorization"] == "Bearer admin-token"
assert api_request[2]["X-Tenant-ID"] == "tenant-1"
assert not any(request[1].endswith("/api/v1/knowledge-bases") for request in upstream_requests)
def test_weknora_iframe_hides_current_outer_sidebar_shell() -> None:
@@ -2334,7 +2385,6 @@ async def test_weknora_iframe_switch_rechecks_selected_mapping_permissions(
is_admin=False,
can_write=True,
exp=4_102_444_800,
allowed_weknora_ids=("remote-kb", "other-visible-kb"),
)
selected = await llmwiki_router._switch_embed_context(
@@ -2382,12 +2432,16 @@ async def test_weknora_platform_proxy_reissues_cookie_for_native_kb_switch(
seen["target"] = weknora_id
return selected
async def hydrate_visible_bases(_: Request, ctx: object) -> object:
return ctx
async def proxy_request(*_: object, **kwargs: object) -> Response:
seen.update(kwargs)
return Response("ok")
monkeypatch.setattr(llmwiki_router, "_read_embed_context", read_context)
monkeypatch.setattr(llmwiki_router, "_switch_embed_context", switch_context)
monkeypatch.setattr(llmwiki_router, "_with_visible_embed_knowledge_bases", hydrate_visible_bases)
monkeypatch.setattr(
llmwiki_router,
"_runtime_for_iframe_proxy",
@@ -40,7 +40,7 @@ def test_load_returns_defaults_when_missing(tmp_runtime_home: Path) -> None:
assert settings.conversation_context.recent_turn_window_enabled is False
assert settings.conversation_context.max_conversation_rounds == 20
assert settings.write_file_compatibility.enabled is False
assert settings.conversation_context.recent_turn_window_enabled is False
assert settings.call_metrics.enabled is True
def test_save_then_load_roundtrips(tmp_runtime_home: Path) -> None:
@@ -325,6 +325,28 @@ async def test_admin_can_toggle_write_file_compatibility_without_restart(
assert ss.get_write_file_compatibility_settings().enabled is False
@pytest.mark.asyncio
async def test_admin_can_toggle_call_metrics_without_restart(
tmp_runtime_home: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
_patch_admin(monkeypatch)
disabled = await ss_router.update_call_metrics_setting(
ss_router.CallMetricsUpdateRequest(enabled=False),
SimpleNamespace(),
)
assert disabled.enabled is False
assert ss.get_call_metrics_settings().enabled is False
enabled = await ss_router.update_call_metrics_setting(
ss_router.CallMetricsUpdateRequest(enabled=True),
SimpleNamespace(),
)
assert enabled.enabled is True
assert ss.get_call_metrics_settings().enabled is True
def test_prepend_prefix_tags_first_human_without_touching_content() -> None:
graph_input = {"messages": [HumanMessage(content="评价一下中美贸易")]}
out = _prepend_prompt_prefix(graph_input, "你需要站在中方角度")
@@ -69,6 +69,7 @@ async def _admit(repo: RoundtableRoomRepository, draft_id: str, user_id: str, **
room_session_id=kwargs.get("room_session_id"),
enabled=kwargs.get("enabled", True),
max_rooms=kwargs.get("max_rooms", 3),
lease_seconds=kwargs.get("lease_seconds", 5 * 60),
extra_keys=kwargs.get("extra_keys"),
)
@@ -257,6 +258,44 @@ async def test_delayed_heartbeat_cannot_resurrect_a_released_room(rooms) -> None
assert next_room["ok"] is True
@pytest.mark.asyncio
async def test_foreground_lease_defaults_to_five_minutes_and_honors_override(rooms) -> None:
repo, factory = rooms
before = datetime.now(UTC)
admitted = await _admit(
repo,
"draft-lease",
"user-a",
room_session_id="lease-session",
)
assert admitted["ok"] is True
key = compute_room_session_key("lease-session")
assert key is not None
async with factory() as session:
row = await session.get(RoundtableRoomLeaseRow, key)
assert row is not None and row.lease_until is not None
until = row.lease_until if row.lease_until.tzinfo else row.lease_until.replace(tzinfo=UTC)
remaining = until - before
assert timedelta(minutes=4, seconds=50) <= remaining <= timedelta(minutes=5, seconds=5)
renewed = await repo.renew(
draft_id="draft-lease",
user_id="user-a",
task_id=None,
room_session_id="lease-session",
enabled=True,
max_rooms=3,
lease_seconds=120,
)
assert renewed["ok"] is True
async with factory() as session:
row = await session.get(RoundtableRoomLeaseRow, key)
assert row is not None and row.lease_until is not None
until = row.lease_until if row.lease_until.tzinfo else row.lease_until.replace(tzinfo=UTC)
renewed_for = until - datetime.now(UTC)
assert timedelta(seconds=90) <= renewed_for <= timedelta(seconds=125)
@pytest.mark.asyncio
async def test_stale_run_for_leased_draft_does_not_consume_a_second_room(rooms) -> None:
"""Run-manager fallback must not double-count the authoritative lease."""
@@ -744,6 +783,7 @@ async def test_admin_can_change_roundtable_room_limit_without_restart(
assert ss.get_roundtable_room_limit_settings().enabled is True
assert ss.get_roundtable_room_limit_settings().max_rooms == 3
assert ss.get_roundtable_room_limit_settings().max_hold_minutes == 30
assert ss.get_roundtable_room_limit_settings().lease_minutes == 5
assert ss.get_roundtable_room_limit_settings().max_cycles == 8
assert ss.get_roundtable_room_limit_settings().write_file_compatibility_enabled is False
@@ -752,6 +792,7 @@ async def test_admin_can_change_roundtable_room_limit_without_restart(
enabled=False,
max_rooms=5,
max_hold_minutes=45,
lease_minutes=8,
max_cycles=12,
),
SimpleNamespace(),
@@ -759,9 +800,11 @@ async def test_admin_can_change_roundtable_room_limit_without_restart(
assert saved.enabled is False
assert saved.max_rooms == 5
assert saved.max_hold_minutes == 45
assert saved.lease_minutes == 8
assert saved.max_cycles == 12
assert ss.load_system_settings().roundtable_room_limit.max_rooms == 5
assert ss.load_system_settings().roundtable_room_limit.max_hold_minutes == 45
assert ss.load_system_settings().roundtable_room_limit.lease_minutes == 8
assert ss.load_system_settings().roundtable_room_limit.max_cycles == 12
assert saved.write_file_compatibility_enabled is False
@@ -0,0 +1,190 @@
from __future__ import annotations
import io
from pathlib import Path
from types import SimpleNamespace
import pytest
from fastapi import UploadFile
from deerflow.skills.storage import LocalSkillStorage
from deerflow.skills.types import SkillCategory
def _skill_markdown(version: str) -> str:
return f"""---
name: demo-skill
description: Demo custom skill
---
# Demo {version}
"""
@pytest.fixture
def versioned_skill(tmp_path: Path, monkeypatch: pytest.MonkeyPatch):
from deerflow.skills import versioning
skills_root = tmp_path / "skills"
skill_dir = skills_root / "custom" / "demo-skill"
skill_dir.mkdir(parents=True)
(skill_dir / "SKILL.md").write_text(_skill_markdown("v1"), encoding="utf-8")
(skill_dir / "assets").mkdir()
(skill_dir / "assets" / "payload.txt").write_text("asset-v1", encoding="utf-8")
storage = LocalSkillStorage(host_path=str(skills_root))
enabled = SimpleNamespace(enabled=True)
monkeypatch.setattr(
versioning,
"load_system_settings",
lambda: SimpleNamespace(skill_versioning=enabled),
)
monkeypatch.setattr(
versioning,
"get_paths",
lambda: SimpleNamespace(base_dir=tmp_path / "runtime"),
)
return versioning, storage, skill_dir, enabled, tmp_path
def test_versioning_is_noop_while_switch_is_disabled(versioned_skill) -> None:
versioning, storage, _skill_dir, enabled, tmp_path = versioned_skill
enabled.enabled = False
assert versioning.record_custom_skill_version("demo-skill", storage=storage, action="upload") is None
assert not (tmp_path / "runtime" / "skill-versions").exists()
def test_complete_package_versions_switch_without_resetting_usage(versioned_skill, monkeypatch: pytest.MonkeyPatch) -> None:
versioning, storage, skill_dir, _enabled, tmp_path = versioned_skill
import deerflow.skills.storage as storage_package
from deerflow.skills import usage
monkeypatch.setattr(storage_package, "get_or_new_skill_storage", lambda: storage)
monkeypatch.setattr(usage, "_schedule_use_projection_increment", lambda *args: None)
first = versioning.record_custom_skill_version("demo-skill", storage=storage, action="upload")
assert first and first["version"] == 1
assert usage.bump_use("demo-skill") is True
assert usage.bump_use("demo-skill") is True
(skill_dir / "SKILL.md").write_text(_skill_markdown("v2"), encoding="utf-8")
(skill_dir / "assets" / "payload.txt").write_text("asset-v2", encoding="utf-8")
second = versioning.record_custom_skill_version("demo-skill", storage=storage, action="upload_iteration")
assert second and second["version"] == 2
listed = versioning.list_custom_skill_versions("demo-skill", storage=storage)
assert listed["active_version"] == 2
assert [item["version"] for item in listed["versions"]] == [2, 1]
versioning.activate_custom_skill_version("demo-skill", 1, storage=storage)
assert "Demo v1" in (skill_dir / "SKILL.md").read_text(encoding="utf-8")
assert (skill_dir / "assets" / "payload.txt").read_text(encoding="utf-8") == "asset-v1"
assert usage.get_entry("demo-skill")["use_count"] == 2
snapshot_root = tmp_path / "runtime" / "skill-versions" / "demo-skill"
assert snapshot_root.is_dir()
assert not snapshot_root.is_relative_to(skills_root := storage.get_skills_root_path())
assert [skill.name for skill in storage.load_skills(enabled_only=False)] == ["demo-skill"]
assert skills_root == tmp_path / "skills"
def test_direct_overwrite_resets_version_chain(versioned_skill) -> None:
versioning, storage, skill_dir, _enabled, _tmp_path = versioned_skill
versioning.record_custom_skill_version("demo-skill", storage=storage, action="upload")
(skill_dir / "SKILL.md").write_text(_skill_markdown("v2"), encoding="utf-8")
versioning.record_custom_skill_version("demo-skill", storage=storage, action="upload_iteration")
(skill_dir / "SKILL.md").write_text(_skill_markdown("replacement"), encoding="utf-8")
reset = versioning.reset_custom_skill_versions("demo-skill", storage=storage, summary="direct overwrite")
assert reset and reset["version"] == 1
listed = versioning.list_custom_skill_versions("demo-skill", storage=storage)
assert listed["active_version"] == 1
assert [item["version"] for item in listed["versions"]] == [1]
assert "replacement" in (skill_dir / "SKILL.md").read_text(encoding="utf-8")
@pytest.mark.asyncio
@pytest.mark.parametrize(
("install_mode", "expected_action"),
[("iterate", "upload_iteration"), ("overwrite", "reset")],
)
async def test_same_name_upload_supports_iteration_or_direct_overwrite(
monkeypatch: pytest.MonkeyPatch,
install_mode: str,
expected_action: str,
) -> None:
from app.gateway.routers import skills as skills_router
from deerflow.skills import usage, versioning
class FakeStorage:
def invalidate_skills_cache(self):
return None
def validate_skill_archive(self, *_args, **_kwargs):
return {
"skill_name": "demo-skill",
"conflict_kind": "custom",
}
async def ainstall_skill_from_archive(self, *_args, **kwargs):
assert kwargs["overwrite"] is True
assert kwargs["target_category"] == SkillCategory.CUSTOM
return {
"success": True,
"skill_name": "demo-skill",
"message": "installed",
"overwritten": True,
"category": "custom",
}
class FakeSkillStore:
async def ensure_legacy(self, payload):
assert payload["name"] == "demo-skill"
return payload
storage = FakeStorage()
calls: list[str] = []
async def preview_target(**_kwargs):
return False, True, True, False, False, "user-1", "owner", "conflict"
async def no_op(*_args, **_kwargs):
return None
monkeypatch.setattr(skills_router, "get_or_new_skill_storage", lambda **_kwargs: storage)
monkeypatch.setattr(skills_router, "_preview_install_target", preview_target)
monkeypatch.setattr(skills_router, "_current_user_id", lambda _request: "user-1")
monkeypatch.setattr(skills_router, "_is_admin_user", lambda _request: False)
monkeypatch.setattr(skills_router, "_refresh_catalog_snapshot", no_op)
monkeypatch.setattr(skills_router, "refresh_skills_system_prompt_cache_async", no_op)
monkeypatch.setattr(skills_router, "_sync_skill_view", no_op)
monkeypatch.setattr(usage, "mark_uploaded", lambda _name: None)
monkeypatch.setattr(versioning, "skill_versioning_enabled", lambda: True)
monkeypatch.setattr(
versioning,
"record_custom_skill_version",
lambda _name, **kwargs: calls.append(kwargs["action"]),
)
monkeypatch.setattr(
versioning,
"reset_custom_skill_versions",
lambda _name, **_kwargs: calls.append("reset"),
)
result = await skills_router.install_skill_upload(
request=SimpleNamespace(),
file=UploadFile(filename="demo.skill", file=io.BytesIO(b"PK\x03\x04archive")),
overwrite=False,
install_mode=install_mode,
target_category=SkillCategory.CUSTOM,
config=SimpleNamespace(),
skill_store=FakeSkillStore(),
)
assert result.overwritten is True
assert expected_action in calls
if install_mode == "iterate":
assert calls == ["baseline", "upload_iteration"]
else:
assert calls == ["reset"]
@@ -221,11 +221,22 @@ async def test_update_public_skill_is_admin_only_and_preserves_raw_config(
monkeypatch.setattr(skills_router, "get_or_new_skill_storage", lambda **_kwargs: storage)
refresh_calls: list[bool] = []
catalog_refresh_calls: list[object] = []
enabled_row_updates: list[tuple[str, bool]] = []
async def _refresh() -> None:
refresh_calls.append(True)
async def _refresh_catalog(config) -> None:
catalog_refresh_calls.append(config)
async def _set_enabled(name: str, enabled: bool) -> bool:
enabled_row_updates.append((name, enabled))
return True
monkeypatch.setattr(skills_router, "refresh_skills_system_prompt_cache_async", _refresh)
monkeypatch.setattr(skills_router, "refresh_skill_catalog_projection", _refresh_catalog)
monkeypatch.setattr(skills_router, "set_skill_catalog_enabled", _set_enabled)
replace_calls: list[tuple[object, object]] = []
real_replace = os.replace
@@ -246,6 +257,7 @@ async def test_update_public_skill_is_admin_only_and_preserves_raw_config(
assert forbidden.value.status_code == 403
assert config_path.read_bytes() == before
assert refresh_calls == []
assert enabled_row_updates == []
response = await skills_router.update_skill(
"demo-skill",
@@ -264,6 +276,8 @@ async def test_update_public_skill_is_admin_only_and_preserves_raw_config(
assert persisted["mcpServers"]["preserved"]["command"] == "$MCP_COMMAND"
assert get_extensions_config().skills["demo-skill"].enabled is False
assert refresh_calls == [True]
assert enabled_row_updates == [("demo-skill", False)]
assert catalog_refresh_calls == []
config_replaces = [item for item in replace_calls if os.fspath(item[1]) == os.fspath(config_path)]
assert len(config_replaces) == 1
@@ -282,6 +296,8 @@ async def test_update_public_skill_is_admin_only_and_preserves_raw_config(
assert persisted["skills"]["demo-skill"]["enabled"] is True
assert persisted["concurrentUnknownField"] == {"survives": True}
assert refresh_calls == [True, True]
assert enabled_row_updates == [("demo-skill", False), ("demo-skill", True)]
assert catalog_refresh_calls == []
config_replaces = [item for item in replace_calls if os.fspath(item[1]) == os.fspath(config_path)]
assert len(config_replaces) == 2
@@ -4,7 +4,15 @@ import pytest
from fastapi import HTTPException
from app.gateway.routers import system_settings as system_settings_router
from deerflow.config.system_settings import SystemSettings
from deerflow.config.system_settings import (
AIWritingGuideCard,
AIWritingPageDescription,
GeneralQaPageDescription,
GeneralQaPageDescriptions,
PageDescriptionsSettings,
PageHeaderDescription,
SystemSettings,
)
def test_strategy_project_demo_is_hidden_by_default() -> None:
@@ -15,6 +23,10 @@ def test_assistant_knowledge_is_hidden_by_default() -> None:
assert SystemSettings().knowledge_base_product_version == "v1"
def test_custom_skill_versioning_is_disabled_by_default() -> None:
assert SystemSettings().skill_versioning.enabled is False
@pytest.mark.asyncio
async def test_non_admin_cannot_update_strategy_project_settings(
monkeypatch: pytest.MonkeyPatch,
@@ -194,6 +206,263 @@ def test_retry_keep_partial_defaults_to_enabled() -> None:
assert SystemSettings().retry_keep_partial.enabled is True
def test_page_descriptions_default_to_product_copy() -> None:
descriptions = SystemSettings().page_descriptions
assert descriptions.general_qa.full == GeneralQaPageDescription()
assert descriptions.general_qa.compact == GeneralQaPageDescription()
assert descriptions.knowledge_base == PageHeaderDescription()
assert descriptions.ai_writing == AIWritingPageDescription()
assert len(descriptions.ai_writing.guide_cards) == 3
assert descriptions.strategy_projects == PageHeaderDescription()
assert descriptions.deep_research == PageHeaderDescription()
assert descriptions.gongwen == PageHeaderDescription()
assert descriptions.light_app_management == PageHeaderDescription()
assert descriptions.light_app_list == PageHeaderDescription()
assert descriptions.agents == PageHeaderDescription()
assert descriptions.business_chains == PageHeaderDescription()
assert descriptions.skills == PageHeaderDescription()
assert descriptions.knowledge_management == PageHeaderDescription()
def test_legacy_general_qa_description_is_shared_by_both_layouts() -> None:
descriptions = PageDescriptionsSettings.model_validate(
{
"general_qa": {
"welcome_title": " 原有问答欢迎语 ",
"welcome_description": " 原有说明 ",
"input_placeholder": " 原有提示 ",
}
}
)
expected = GeneralQaPageDescription(
welcome_title="原有问答欢迎语",
welcome_description="原有说明",
input_placeholder="原有提示",
)
assert descriptions.general_qa.full == expected
assert descriptions.general_qa.compact == expected
legacy_request = system_settings_router.PageDescriptionsUpdateRequest(
general_qa=GeneralQaPageDescription(welcome_title="旧版请求"),
)
assert legacy_request.general_qa == GeneralQaPageDescriptions(
full=GeneralQaPageDescription(welcome_title="旧版请求"),
compact=GeneralQaPageDescription(welcome_title="旧版请求"),
)
def test_ai_writing_guide_cards_are_padded_to_three_items() -> None:
descriptions = PageDescriptionsSettings.model_validate(
{
"ai_writing": {
"guide_cards": [
{"title": " 自定义卡片 ", "description": " 自定义说明 "},
]
}
}
)
assert descriptions.ai_writing.guide_cards == [
AIWritingGuideCard(title="自定义卡片", description="自定义说明"),
AIWritingGuideCard(),
AIWritingGuideCard(),
]
@pytest.mark.asyncio
async def test_non_admin_cannot_update_page_descriptions(
monkeypatch: pytest.MonkeyPatch,
) -> None:
async def ordinary_user(_request):
return SimpleNamespace(system_role="user")
monkeypatch.setattr(
system_settings_router,
"get_optional_user_from_request",
ordinary_user,
)
with pytest.raises(HTTPException) as error:
await system_settings_router.update_page_descriptions(
system_settings_router.PageDescriptionsUpdateRequest(
general_qa=GeneralQaPageDescriptions(
full=GeneralQaPageDescription(welcome_title="不可保存"),
),
knowledge_base=PageHeaderDescription(),
ai_writing=AIWritingPageDescription(),
),
SimpleNamespace(),
)
assert error.value.status_code == 403
@pytest.mark.asyncio
async def test_admin_can_update_page_descriptions(
monkeypatch: pytest.MonkeyPatch,
) -> None:
current = SystemSettings()
saved: list[SystemSettings] = []
async def admin_user(_request):
return SimpleNamespace(system_role="admin")
monkeypatch.setattr(
system_settings_router,
"get_optional_user_from_request",
admin_user,
)
monkeypatch.setattr(system_settings_router, "load_system_settings", lambda: current)
monkeypatch.setattr(
system_settings_router,
"save_system_settings",
lambda settings: saved.append(settings) or settings,
)
response = await system_settings_router.update_page_descriptions(
system_settings_router.PageDescriptionsUpdateRequest(
general_qa=GeneralQaPageDescriptions(
full=GeneralQaPageDescription(
welcome_title=" 欢迎使用工程开发智能问答助手 ",
welcome_description=" 请描述您希望完成的工程开发任务。 ",
input_placeholder=" 输入问题或任务描述 ",
),
compact=GeneralQaPageDescription(
welcome_title=" 简洁问答欢迎语 ",
),
),
knowledge_base=PageHeaderDescription(
title=" 资料库 ",
description=" 在这里整理和检索团队资料。 ",
),
ai_writing=AIWritingPageDescription(
title=" 写作工作台 ",
description=" 将想法逐步发展为文章。 ",
guide_cards=[
AIWritingGuideCard(
title=" 明确写作意图 ",
description=" 用一句话说明想写什么。 ",
),
AIWritingGuideCard(
title=" 对齐写作参数 ",
description=" 选择体裁与篇幅。 ",
),
AIWritingGuideCard(
title=" 生成并优化文章 ",
description=" 查看并完善草稿。 ",
),
],
),
strategy_projects=PageHeaderDescription(
title=" 项目工作台 ",
description=" 集中管理各类项目成果。 ",
),
deep_research=PageHeaderDescription(
title=" 开始研究 ",
description=" 收集资料并生成研究报告。 ",
),
gongwen=PageHeaderDescription(
title=" 公文工作台 ",
description=" 对话完成公文起草和校审。 ",
),
light_app_management=PageHeaderDescription(
title=" 轻应用管理 ",
description=" 登记和维护轻应用。 ",
),
light_app_list=PageHeaderDescription(
title=" 轻应用列表 ",
description=" 快速打开已登记的轻应用。 ",
),
agents=PageHeaderDescription(
title=" 智能体工作台 ",
description=" 管理可用智能体。 ",
),
business_chains=PageHeaderDescription(
title=" 业务链目录 ",
description=" 管理可复用的智能体编排。 ",
),
skills=PageHeaderDescription(
title=" 技能工作台 ",
description=" 浏览和维护技能。 ",
),
knowledge_management=PageHeaderDescription(
title=" 知识服务 ",
description=" 管理知识库与访问范围。 ",
),
),
SimpleNamespace(),
)
assert response.general_qa == GeneralQaPageDescriptions(
full=GeneralQaPageDescription(
welcome_title="欢迎使用工程开发智能问答助手",
welcome_description="请描述您希望完成的工程开发任务。",
input_placeholder="输入问题或任务描述",
),
compact=GeneralQaPageDescription(welcome_title="简洁问答欢迎语"),
)
assert response.knowledge_base == PageHeaderDescription(
title="资料库",
description="在这里整理和检索团队资料。",
)
assert response.ai_writing == AIWritingPageDescription(
title="写作工作台",
description="将想法逐步发展为文章。",
guide_cards=[
AIWritingGuideCard(
title="明确写作意图",
description="用一句话说明想写什么。",
),
AIWritingGuideCard(
title="对齐写作参数",
description="选择体裁与篇幅。",
),
AIWritingGuideCard(
title="生成并优化文章",
description="查看并完善草稿。",
),
],
)
assert response.strategy_projects == PageHeaderDescription(
title="项目工作台",
description="集中管理各类项目成果。",
)
assert response.deep_research == PageHeaderDescription(
title="开始研究",
description="收集资料并生成研究报告。",
)
assert response.gongwen == PageHeaderDescription(
title="公文工作台",
description="对话完成公文起草和校审。",
)
assert response.light_app_management == PageHeaderDescription(
title="轻应用管理",
description="登记和维护轻应用。",
)
assert response.light_app_list == PageHeaderDescription(
title="轻应用列表",
description="快速打开已登记的轻应用。",
)
assert response.agents == PageHeaderDescription(
title="智能体工作台",
description="管理可用智能体。",
)
assert response.business_chains == PageHeaderDescription(
title="业务链目录",
description="管理可复用的智能体编排。",
)
assert response.skills == PageHeaderDescription(
title="技能工作台",
description="浏览和维护技能。",
)
assert response.knowledge_management == PageHeaderDescription(
title="知识服务",
description="管理知识库与访问范围。",
)
assert saved[0].page_descriptions.general_qa == response.general_qa
@pytest.mark.asyncio
async def test_retry_keep_partial_get_reflects_saved_settings(
monkeypatch: pytest.MonkeyPatch,
@@ -266,3 +535,83 @@ async def test_admin_can_update_retry_keep_partial(
assert response.enabled is False
assert saved[0].retry_keep_partial.enabled is False
@pytest.mark.asyncio
async def test_skill_versioning_get_reflects_saved_setting(
monkeypatch: pytest.MonkeyPatch,
) -> None:
current = SystemSettings().model_copy(
update={
"skill_versioning": system_settings_router.SkillVersioningSettings(
enabled=True
)
}
)
monkeypatch.setattr(system_settings_router, "load_system_settings", lambda: current)
response = await system_settings_router.get_skill_versioning()
assert response.enabled is True
@pytest.mark.asyncio
async def test_non_admin_cannot_update_skill_versioning(
monkeypatch: pytest.MonkeyPatch,
) -> None:
async def ordinary_user(_request):
return SimpleNamespace(system_role="user")
monkeypatch.setattr(
system_settings_router,
"get_optional_user_from_request",
ordinary_user,
)
with pytest.raises(HTTPException) as error:
await system_settings_router.update_skill_versioning(
system_settings_router.SkillVersioningUpdateRequest(enabled=True),
SimpleNamespace(),
)
assert error.value.status_code == 403
@pytest.mark.asyncio
async def test_admin_can_enable_skill_versioning_and_capture_baselines(
monkeypatch: pytest.MonkeyPatch,
) -> None:
from deerflow.skills import versioning
current = SystemSettings()
saved: list[SystemSettings] = []
reconciled: list[str | None] = []
async def admin_user(_request):
return SimpleNamespace(system_role="admin", id="admin-1")
monkeypatch.setattr(
system_settings_router,
"get_optional_user_from_request",
admin_user,
)
monkeypatch.setattr(system_settings_router, "load_system_settings", lambda: current)
monkeypatch.setattr(
system_settings_router,
"save_system_settings",
lambda settings: saved.append(settings) or settings,
)
monkeypatch.setattr(
versioning,
"reconcile_custom_skill_versions",
lambda *, author=None: reconciled.append(author) or 0,
)
response = await system_settings_router.update_skill_versioning(
system_settings_router.SkillVersioningUpdateRequest(enabled=True),
SimpleNamespace(),
)
assert response.enabled is True
assert saved[0].skill_versioning.enabled is True
assert reconciled == ["admin-1"]