解决 artifact 预览组件冲突:mermaid 渲染(dhk-1) 与 markdown 锚点跳转(master) 取并集;package-lock.json 按 master 删除(项目用 pnpm)。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
董宏凯andClaude Opus 4.8 committed 2026-06-05 19:43:02 +08:00
commit b2132dd668
104 files changed
+8775 -12038

No files matched your search

@@ -263,12 +263,22 @@ Proxied through nginx: `/api/langgraph/*` → LangGraph, all other `/api/*` →
**HTML page tasks (`task_kind == "html_page"`)**: a second task kind that renders a full HTML document each run instead of a Markdown report. The kind + its config (`html_config`: `page_prompt`, `negative_prompt`, `reference_favorite_id`, `model_name`) live inside `execution_context` — no schema change to `scheduled_tasks`. Data gathering is delegated to the agent's own search skills (driven by `page_prompt`), so there is no explicit search-query/data-source field; the render model is user-selectable (`model_name`, default `zai-org/GLM-5-FP8`). `_run_task_attempt()` dispatches by kind; `_run_html_page_attempt()` runs the pipeline: gather real data via the lead agent (its search tools) → optionally load a reference favorite's reusable style/layout → render with the chosen model (thinking off) → fault-tolerant JSON parse → sanitize (BeautifulSoup, strips `<script>`/`<iframe>`/`on*`/`javascript:`) → structural validate → one GLM5 repair pass if invalid → **completeness gate**: if the HTML is still not a full valid document (missing doctype/html/head/body, empty body, or residual scripts) after the repair pass, raise `HtmlGenerationError` so the run is marked **failed** (a page only counts as successful when it is a complete, previewable HTML document) → best-effort Playwright screenshot (no-op when Playwright absent) → archive `index.html` → persist a `scheduled_html_pages` row → return `result` with `format: "html"` and the full HTML inline. Pure helpers live in `deerflow/runtime/scheduler/html_page.py`. The frontend previews the HTML in a sandboxed `iframe` (no `allow-scripts`) on the run-detail and public pages; a user can bookmark a page (style summary extracted via GLM5) and later pick it as a style reference. New tables `scheduled_html_pages` + `html_page_favorites` (`deerflow/persistence/html_pages/`, wired as `app.state.html_page_store`) are created by `Base.metadata.create_all` with no migration. Tests: `tests/test_html_page_tasks.py`.
**Unified output switches (`require_html` + `require_markdown`).** The task form no longer splits "普通任务 / HTML 页面" into separate kinds. `execution_context` now carries two **independent** booleans that may both be on: `require_html` (必须生成 HTML) and `require_markdown` (必须生成 Markdown). `_task_requires_html()` returns true for `require_html` **or** legacy `task_kind == "html_page"` (so old tasks need no data migration), and `_run_task_attempt()` dispatches to `_run_html_page_attempt` when it is true, else the regular agent attempt. When **both** switches are on, `_gather_html_data()` returns `(text, files)` — the gather agent run is additionally told to save a `.md` file — and `_run_html_page_attempt` validates it (`MarkdownRequirementError` on failure) and surfaces it alongside the generated page, so the run delivers both artifacts. The single main `prompt` doubles as the HTML page requirement (`html_config.page_prompt`); there is no separate page-prompt field. Tests: `tests/test_scheduled_task_require_html.py` + the markdown-coexistence cases in `tests/test_html_page_tasks.py`.
### Per-user Prompts & UI Preferences
**Custom prompts can be scoped to one agent.** `user_custom_prompts` gains a nullable `agent_id` column (migration `20260603_02`): `NULL` = global (the chip shows in every chat), otherwise the prompt only surfaces when that agent is selected. The visibility filter is applied in the frontend `CustomPromptBar` (it receives the current agent id from `ChatPage`'s selector or `AgentChatPage`'s `assistantId`); the backend (`/api/user-prompts` CRUD) just stores/returns/clears `agent_id` (update accepts an explicit `null` to clear). Tests: `tests/test_user_prompts_agent_id.py`.
**Per-user UI preferences** (`deerflow/persistence/user_preferences/`, wired as `app.state.user_preferences_store`; migration `20260603_03`, also auto-created by `create_all`): one JSON-blob row per user so new toggles need no schema change. Router `/api/user-preferences` (`GET`/`PUT`, account-scoped). Current key: `hide_admin_prompt_prefix` — when true the chat input hides the admin-configured prompt-prefix switch and never applies the prefix (the user uses only their own prompts; enforced in `ChatPage` for both `shouldApplyPrefix` and `promptPrefixProp`). Toggled from the "管理常用提示词" dialog. Tests: `tests/test_user_preferences.py`.
**HTML skills (tools the render step uses).** Two offline skills under `skills/public/` shape the generated HTML and are toggled in the skills settings page (enabled by default): `html-page-builder` (a self-contained inline-CSS design system — colors/cards/tables/KPI/responsive) and `html-charts` (inline-SVG charts that render in the no-`allow-scripts` preview iframe; optional local ECharts documented in its `assets/README.md`). A skill opts into HTML injection via `metadata.kind: html_page` in its SKILL.md frontmatter; `_collect_html_skill_instructions()` reads the bodies of **enabled** such skills and `build_generation_prompt(skill_instructions=…)` injects them, so the (raw-model) render step honors the user's skill selection. The data-gathering step and the render/repair/style passes all use the task's selected `model_name` (no silent fallback to the default model — and there is **no hardcoded GLM-5**; an unset selection falls back to config `models[0]`). To stop auxiliary LLM calls from leaking onto the default model, **`TitleMiddleware` is skipped when `is_scheduled_run` is set** (`_build_middlewares` in `lead_agent/agent.py`) — scheduled/HTML runs use throwaway threads whose titles are never shown, and `config.title.model_name` is usually unset (→ would call `models[0]`, not the chosen model). The completeness gate (`validate_html`) also requires a non-empty `<title>` and forbids external stylesheets, on top of the doctype/html/head/body + non-empty-body + no-script checks.
### Custom Agent Persistence
Custom agent metadata lives in the `agents` ORM table (`deerflow.persistence.agents`). `agents.id` is the stable runtime id and the filesystem directory name under `.deer-flow/agents/{id}/`; `agents.name` is display-only, can contain Chinese characters, and is not unique. `user_id IS NULL` marks built-in agents, user-owned agents are visible only to their owner unless `published = true`, and update/delete operations require ownership. The runtime accepts `agent_id` in context/configurable, and non-default `assistant_id` values are treated as agent ids after visibility checks.
**Listing cache + pagination** — to keep `GET /api/agents` fast, the gallery's per-card extras (skills + tool_groups) are denormalized into two cache tables instead of reading every agent's `config.yaml` on each request: `agent_skills` (relational, one row per (agent, skill) with `position`) and `agent_extras` (one row per agent: `tool_groups` JSON + `model` + `skills_synced_at`). `config.yaml` remains authoritative for the agent runtime; these tables are a read cache, write-through on create/update and backfilled from disk by `_sync_legacy_agents` (mtime-vs-`skills_synced_at` drift check). The list endpoint enriches via `AgentStore.get_extras_for([ids])` (two batched `IN` queries). Both tables are auto-created by `Base.metadata.create_all` (no migration needed). `GET /api/agents` supports **opt-in server-side pagination**: `scope` (`mine` = owned + favorited / `square` = built-in/published public), `page`, `page_size`, returning `{agents, total, page, page_size}`; omitting `scope`/`page` preserves the legacy "return everything" behavior. Name `search` matches the raw OR desensitized name (the frontend's static term→abbrev map is mirrored server-side in `deerflow/config/agent_name_desensitize.py`); `tag_ids` is an SQL OR-filter. `POST /api/agents/private-square/list` takes the same pagination params. Repository entry point: `AgentStore.list_paginated(...)`.
**Homepage selector (agent picker on the new-chat page)** — single admin-curated list backed by `agents.featured_order` (nullable BIGINT). NULL = not featured. Only `published = true` agents can be featured (router enforces on writes); any published agent is eligible regardless of ownership. Cleared with `order: null`. The list rendered to end users is `WHERE featured_order IS NOT NULL` + visibility-filter, sorted by `featured_order ASC`. Admins reorder by sending the desired ids to `PUT /api/agents/featured/order` (assigns indices 0..N-1). BIGINT is used rather than INT so the management UI can seed initial values from `Date.now()` (overflows INT on MySQL).
### Portable Column Types (`deerflow/persistence/types.py`)
@@ -511,6 +521,8 @@ Per-user runtime overrides (whitelisted fields) live at `{base_dir}/users/{user_
Both endpoints are whitelisted in `auth_middleware._PUBLIC_EXACT_PATHS`. The shared `_get_or_create_user_by_username()` helper (in `routers/auth.py`) backs both `/login/username` and `/login/token`.
- `log_system` - Admin-only external log-system entry (`deerflow/config/log_system_config.py`): `{enabled, url, label}`. Surfaced read-only by `GET /api/system-settings/log-system` (admin only); the frontend renders it as a "日志系统" item in the bottom-right settings menu (`workspace-nav-menu.tsx`) that opens `url` in a new tab. Disabled by default.
**`extensions_config.json`**:
- `mcpServers` - Map of server name → config (enabled, type, command, args, env, url, headers, oauth, description)
- `skills` - Map of skill name → state (enabled)
@@ -59,6 +59,7 @@ from app.gateway.routers import (
thread_shares,
threads,
uploads,
user_preferences,
user_prompts,
writing,
)
@@ -859,6 +860,9 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
app.include_router(user_prompts.router)
# Per-user UI preferences at /api/user-preferences
app.include_router(user_preferences.router)
# Roundtable-planning drafts + recommendation history at /api/roundtable-drafts
app.include_router(roundtable_drafts.router)
@@ -71,6 +71,14 @@ class LocalAuthProvider(AuthProvider):
"""
return await self._repo.find_user_by_email_local_part(local_part)
async def find_users_by_email_local_part_like(self, substring: str, *, limit: int = 500) -> list[User]:
"""Fuzzy-match users by a substring of their email local part.
Backs the admin LLM-metrics username filter — returns every account
whose username contains ``substring`` (case-insensitive).
"""
return await self._repo.find_users_by_email_local_part_like(substring, limit=limit)
async def create_user(self, email: str, password: str | None = None, system_role: str = "user", needs_setup: bool = False) -> User:
"""Create a new local user.
@@ -81,6 +81,24 @@ class UserRepository(ABC):
"""
raise NotImplementedError
async def find_users_by_email_local_part_like(self, substring: str, *, limit: int = 500) -> list["User"]:
"""Fuzzy-match users whose email local part *contains* ``substring``.
Backs the admin LLM-metrics username filter (按用户名模糊搜索). Unlike
:meth:`find_user_by_email_local_part` (exact prefix → single account),
this returns every matching account. Case-insensitive substring match
against the part before ``@``. Non-overriding repositories return an
empty list.
Args:
substring: Case-insensitive substring to match within the local part.
limit: Maximum number of users to return.
Returns:
Matching users (may be empty).
"""
return []
@abstractmethod
async def update_user(self, user: User) -> User:
"""Update an existing user.
@@ -109,6 +109,33 @@ class SQLiteUserRepository(UserRepository):
row = result.scalar_one_or_none()
return self._row_to_user(row) if row is not None else None
async def find_users_by_email_local_part_like(self, substring: str, *, limit: int = 500) -> list[User]:
term = (substring or "").strip().lower()
if not term:
return []
# Escape SQL LIKE wildcards so a literal ``_``/``%`` in the query does
# not turn into a wildcard. ``\`` is the chosen escape char.
escaped = term.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
pattern = f"%{escaped}%"
# Prefilter in SQL on the whole email (portable across sqlite/postgres/
# mysql), then keep only rows whose *local part* contains the term so a
# domain-only hit (e.g. searching "gmail") doesn't pull unrelated users.
stmt = (
select(UserRow)
.where(func.lower(UserRow.email).like(pattern, escape="\\"))
.order_by(UserRow.created_at.asc())
.limit(max(1, limit))
)
async with self._sf() as session:
result = await session.execute(stmt)
rows = result.scalars().all()
out: list[User] = []
for row in rows:
local = (row.email or "").split("@", 1)[0].lower()
if term in local:
out.append(self._row_to_user(row))
return out
async def update_user(self, user: User) -> User:
async with self._sf() as session:
row = await session.get(UserRow, str(user.id))
@@ -163,6 +163,10 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
app.state.user_prompt_store = make_user_prompt_store(sf)
from deerflow.persistence.user_preferences import make_user_preferences_store
app.state.user_preferences_store = make_user_preferences_store(sf)
from deerflow.persistence.roundtable_drafts import make_roundtable_draft_store
app.state.roundtable_draft_store = make_roundtable_draft_store(sf)
File diff suppressed because it is too large. Load diff
@@ -15,7 +15,7 @@ from fastapi import APIRouter, Depends, HTTPException, Query, Request
from pydantic import BaseModel, Field
from app.gateway.auth.password import verify_password_async
from app.gateway.deps import get_agent_store, get_config, get_tag_store
from app.gateway.deps import get_agent_store, get_config, get_tag_store
from deerflow.agents.lead_agent.prompt import refresh_skills_system_prompt_cache_async
from deerflow.config.agents_api_config import get_agents_api_config
from deerflow.config.agents_config import list_custom_agents, load_agent_config, load_agent_soul, validate_agent_id
@@ -62,13 +62,27 @@ class AgentResponse(BaseModel):
is_favorite: bool = Field(
default=False,
description="Whether the requesting user has favorited (收藏) this agent.",
)
)
is_pinned: bool = Field(
default=False,
description="Whether the requesting user has pinned this agent.",
)
class AgentsListResponse(BaseModel):
"""Response model for listing agents.
class AgentsListResponse(BaseModel):
"""Response model for listing all visible agents."""
``total`` / ``page`` / ``page_size`` describe server-side pagination. When
a caller does not opt into pagination they default to "everything on one
page" (``total == len(agents)``, ``page == 1``), so existing unpaginated
callers keep working unchanged.
"""
agents: list[AgentResponse]
total: int = Field(default=0, description="Total agents matching the query across all pages")
page: int = Field(default=1, description="1-based page index of this response")
page_size: int = Field(default=0, description="Page size; 0 means the result was not paginated")
class AgentCreateRequest(BaseModel):
@@ -101,7 +115,7 @@ class AgentUpdateRequest(BaseModel):
tool_groups: list[str] | None = Field(default=None, description="Updated tool group whitelist (config.yaml)")
class AgentSelectorResponse(BaseModel):
class AgentSelectorResponse(BaseModel):
"""Response for the homepage agent selector dropdown.
The list is fully curated by admins (``featured_order`` on each row).
@@ -111,10 +125,10 @@ class AgentSelectorResponse(BaseModel):
agents: list[AgentResponse] = Field(
default_factory=list,
description="Admin-curated agents, sorted by featured_order ascending.",
)
class AgentFeaturedRequest(BaseModel):
)
class AgentFeaturedRequest(BaseModel):
"""Request body for setting/clearing an agent's homepage featured rank."""
order: int | None = Field(
@@ -215,6 +229,7 @@ async def _sync_legacy_agents(store: AgentStore) -> None:
handlers as a safety net for environments that bypassed the lifespan hook.
"""
try:
pairs: list[tuple[str, Any]] = []
for agent_cfg in list_custom_agents():
agent_id = agent_cfg.id or agent_cfg.name
if not agent_id:
@@ -227,10 +242,41 @@ async def _sync_legacy_agents(store: AgentStore) -> None:
"published": True,
}
)
pairs.append((agent_id, agent_cfg))
await _backfill_listing_cache(store, pairs)
except Exception:
logger.exception("Failed to sync legacy folder-backed agents")
async def _backfill_listing_cache(store: AgentStore, pairs: list[tuple[str, Any]]) -> None:
"""Mirror each agent's config.yaml skills/tool_groups/model into the DB cache.
The list endpoint reads skills/tool_groups from these cache rows instead of
stat+parsing every ``config.yaml`` per request. We fetch the current cache
in one batch and only re-write rows whose values actually changed, so a
steady-state sync (every ``_LEGACY_SYNC_TTL_SECONDS``) is cheap and does not
churn DELETE/INSERTs. For the cache, ``None`` and ``[]`` skills are
equivalent ("no skills") — the tri-state only matters to the agent runtime,
which reads config.yaml directly.
"""
if not pairs:
return
cached = await store.get_extras_for([aid for aid, _ in pairs])
for agent_id, cfg in pairs:
want_skills = [s for s in (cfg.skills or []) if isinstance(s, str)]
want_tool_groups = cfg.tool_groups
want_model = cfg.model
existing = cached.get(agent_id)
if existing is None or existing.get("skills") != want_skills:
await store.replace_skills(agent_id, want_skills)
if (
existing is None
or existing.get("tool_groups") != want_tool_groups
or existing.get("model") != want_model
):
await store.set_extras(agent_id, tool_groups=want_tool_groups, model=want_model)
# Throttle the per-request legacy-sync fallback. The real sync runs once at
# startup (lifespan); request handlers used to call ``_sync_legacy_agents`` on
# every hit, which scanned the filesystem + did N DB upserts per agent each
@@ -302,7 +348,7 @@ def _ensure_agent_dir_from_record(record: dict[str, Any]) -> None:
logger.exception("Failed to self-heal agent dir for '%s'", agent_id)
def _record_to_response(
def _record_to_response(
record: dict[str, Any],
*,
include_soul: bool = False,
@@ -344,10 +390,21 @@ def _record_to_response(
tool_groups=extras["tool_groups"],
skills=extras["skills"],
featured_order=record.get("featured_order"),
)
async def _record_to_response_async(
)
def _sort_agent_responses(responses: list[AgentResponse], sort: str) -> list[AgentResponse]:
reverse = sort not in {"time_asc", "updated_asc", "name_asc"}
if sort in {"name_asc", "name_desc"}:
ordered = sorted(responses, key=lambda r: (r.name or "").lower(), reverse=reverse)
elif sort in {"updated_desc", "updated_asc"}:
ordered = sorted(responses, key=lambda r: str(r.updated_at or ""), reverse=reverse)
else:
ordered = sorted(responses, key=lambda r: str(r.created_at or ""), reverse=reverse)
return sorted(ordered, key=lambda r: 0 if r.is_pinned else 1)
async def _record_to_response_async(
record: dict[str, Any],
*,
include_soul: bool = False,
@@ -403,6 +460,40 @@ async def _get_editable_or_raise(
raise HTTPException(status_code=403, detail="You can only modify agents that you created")
async def _build_list_responses(
store: AgentStore,
tag_store: TagStore,
user_id: str,
records: list[dict[str, Any]],
) -> list[AgentResponse]:
"""Build gallery responses for a set of agent rows WITHOUT touching disk.
skills / tool_groups / model come from the denormalized DB cache
(``get_extras_for`` — one batched query) instead of stat+parsing every
agent's ``config.yaml`` per request, which was the list-endpoint hot spot.
SOUL.md is skipped (only the edit dialog reads it). An agent absent from
the cache (never mirrored) keeps its ``None`` extras until the next sync.
"""
responses = [
_record_to_response(a, include_soul=False, include_extras=False) for a in records
]
ids = [r.id for r in responses]
extras = await store.get_extras_for(ids)
assignments = await tag_store.list_assignments("agent", ids)
favorite_ids = await store.list_favorite_ids(user_id)
pin_ids = await store.list_pin_ids(user_id)
for resp in responses:
cached = extras.get(resp.id)
if cached is not None:
resp.skills = cached.get("skills")
resp.tool_groups = cached.get("tool_groups")
resp.model = cached.get("model")
resp.tags = assignments.get(resp.id, [])
resp.is_favorite = resp.id in favorite_ids
resp.is_pinned = resp.id in pin_ids
return responses
@router.get(
"/agents",
response_model=AgentsListResponse,
@@ -411,9 +502,21 @@ async def _get_editable_or_raise(
)
async def list_agents(
request: Request,
search: str | None = Query(default=None, description="Case-insensitive fuzzy match on agent name"),
search: str | None = Query(default=None, description="Case-insensitive fuzzy match on agent name (raw or desensitized display form)"),
tag_ids: list[str] | None = Query(default=None, description="Keep agents carrying any of these tags (OR)"),
admin: bool = Query(
scope: str | None = Query(
default=None,
pattern="^(mine|square)$",
description="Pagination scope. 'mine' = owned + favorited; 'square' = built-in/published public. Omit for the default combined visibility list.",
),
page: int | None = Query(default=None, ge=1, description="1-based page index. Providing this (or 'scope') enables server-side pagination."),
page_size: int = Query(default=20, ge=1, le=200, description="Items per page when paginating."),
sort: str = Query(
default="time_desc",
pattern="^(time_desc|time_asc|updated_desc|updated_asc|name_asc|name_desc)$",
description="Sort order. Default is newest created first.",
),
admin: bool = Query(
default=False,
description=(
"Admin-only override. When ``true`` and the caller is a system admin, "
@@ -429,38 +532,38 @@ async def list_agents(
tag_store: TagStore = get_tag_store(request)
await _maybe_sync_legacy_agents(store)
user_id = _current_user_id(request)
if admin:
if not _is_admin_user(request):
raise HTTPException(status_code=403, detail="Only system admins can list all agents")
agents = await store.list_all()
else:
agents = await store.list_visible(user_id)
# Load lightweight extras (skills / model / tool_groups from config.yaml)
# so the gallery cards can render skill badges. SOUL.md is still skipped
# (only the edit dialog reads it). The per-agent config.yaml reads run in
# worker threads concurrently, so the event loop stays responsive even
# with many agents.
responses = list(
await asyncio.gather(
*(
_record_to_response_async(a, include_soul=False, include_extras=True)
for a in agents
)
)
)
if admin and not _is_admin_user(request):
raise HTTPException(status_code=403, detail="Only system admins can list all agents")
assignments = await tag_store.list_assignments("agent", [r.id for r in responses])
favorite_ids = await store.list_favorite_ids(user_id)
for resp in responses:
resp.tags = assignments.get(resp.id, [])
resp.is_favorite = resp.id in favorite_ids
paginate = scope is not None or page is not None
if paginate:
eff_page = page or 1
rows, total = await store.list_paginated(
user_id=user_id,
scope=scope,
search=search,
tag_ids=tag_ids,
page=eff_page,
page_size=page_size,
admin=admin,
sort=sort,
)
responses = await _build_list_responses(store, tag_store, user_id, rows)
return AgentsListResponse(agents=responses, total=total, page=eff_page, page_size=page_size)
# Legacy unpaginated path — returns everything (admin console, callers
# that have not migrated to pagination). Extras now come from the DB
# cache too, so this path no longer reads config.yaml per agent.
agents = await store.list_all() if admin else await store.list_visible(user_id)
responses = await _build_list_responses(store, tag_store, user_id, agents)
if search:
needle = search.strip().lower()
responses = [r for r in responses if needle in r.name.lower()]
if tag_ids:
wanted = set(tag_ids)
responses = [r for r in responses if any(t.get("id") in wanted for t in r.tags)]
return AgentsListResponse(agents=responses)
if tag_ids:
wanted = set(tag_ids)
responses = [r for r in responses if any(t.get("id") in wanted for t in r.tags)]
responses = _sort_agent_responses(responses, sort)
return AgentsListResponse(agents=responses, total=len(responses), page=1, page_size=len(responses))
except HTTPException:
raise
except Exception as e:
@@ -473,14 +576,14 @@ async def list_agents(
summary="Check Agent Name",
description="Validate an agent display name. Duplicate names are allowed.",
)
async def check_agent_name(name: str) -> dict[str, Any]:
_require_agents_api_enabled()
normalized = _validate_display_name(name)
return {"available": True, "name": normalized}
@router.get(
"/agents/selector",
async def check_agent_name(name: str) -> dict[str, Any]:
_require_agents_api_enabled()
normalized = _validate_display_name(name)
return {"available": True, "name": normalized}
@router.get(
"/agents/selector",
response_model=AgentSelectorResponse,
summary="Agent Homepage Selector",
description=(
@@ -532,10 +635,12 @@ async def get_agent(agent_id: str, request: Request) -> AgentResponse:
# Detail view legitimately reads SOUL.md + config.yaml — hop to a
# worker thread so the asyncio loop stays responsive under concurrent
# detail fetches.
response = await _record_to_response_async(record, include_soul=True, include_extras=True)
favorite_ids = await store.list_favorite_ids(user_id)
response.is_favorite = agent_id in favorite_ids
return response
response = await _record_to_response_async(record, include_soul=True, include_extras=True)
favorite_ids = await store.list_favorite_ids(user_id)
pin_ids = await store.list_pin_ids(user_id)
response.is_favorite = agent_id in favorite_ids
response.is_pinned = agent_id in pin_ids
return response
except HTTPException:
raise
except Exception as e:
@@ -587,6 +692,10 @@ async def create_agent_endpoint(request: AgentCreateRequest, http_request: Reque
"is_private": request.is_private,
}
)
# Mirror skills/tool_groups/model into the listing cache so the gallery
# can render this agent's badges without reading its config.yaml.
await store.replace_skills(agent_id, request.skills)
await store.set_extras(agent_id, tool_groups=request.tool_groups, model=request.model)
logger.info("Created agent '%s' (%s) at %s", name, agent_id, agent_dir)
return await _record_to_response_async(record, include_soul=True, include_extras=True)
@@ -653,6 +762,9 @@ async def update_agent(agent_id: str, request: AgentUpdateRequest, http_request:
model=next_model,
tool_groups=next_tool_groups,
)
# Keep the listing cache in lock-step with config.yaml on every write.
await store.replace_skills(agent_id, next_skills)
await store.set_extras(agent_id, tool_groups=next_tool_groups, model=next_model)
if request.soul is not None:
_write_agent_soul(agent_id, request.soul)
@@ -762,9 +874,17 @@ async def delete_agent(agent_id: str, request: Request) -> None:
await get_tag_store(request).unassign_all("agent", agent_id)
except Exception: # noqa: BLE001 — tag cleanup must not block deletion
pass
try:
await store.delete_favorites_for_agent(agent_id)
except Exception: # noqa: BLE001 — favorite cleanup must not block deletion
try:
await store.delete_favorites_for_agent(agent_id)
except Exception: # noqa: BLE001 — favorite cleanup must not block deletion
pass
try:
await store.delete_pins_for_agent(agent_id)
except Exception: # noqa: BLE001 - pin cleanup must not block deletion
pass
try:
await store.delete_extras_for_agent(agent_id)
except Exception: # noqa: BLE001 — cache cleanup must not block deletion
pass
logger.info("Deleted agent '%s' from %s", agent_id, agent_dir)
except HTTPException:
@@ -805,7 +925,7 @@ async def favorite_agent(agent_id: str, request: Request) -> None:
summary="Unfavorite Agent",
description="Remove an agent from the current user's favorites. No-op when not favorited.",
)
async def unfavorite_agent(agent_id: str, request: Request) -> None:
async def unfavorite_agent(agent_id: str, request: Request) -> None:
_require_agents_api_enabled()
agent_id = _validate_storage_id(agent_id)
@@ -816,13 +936,60 @@ async def unfavorite_agent(agent_id: str, request: Request) -> None:
logger.info("User '%s' unfavorited agent '%s'", user_id, agent_id)
except HTTPException:
raise
except Exception as e:
logger.error("Failed to unfavorite agent '%s': %s", agent_id, e, exc_info=True)
raise HTTPException(status_code=500, detail=f"Failed to unfavorite agent: {str(e)}") from e
@router.put(
"/agents/{agent_id}/featured",
except Exception as e:
logger.error("Failed to unfavorite agent '%s': %s", agent_id, e, exc_info=True)
raise HTTPException(status_code=500, detail=f"Failed to unfavorite agent: {str(e)}") from e
@router.post(
"/agents/{agent_id}/pin",
status_code=204,
summary="Pin Agent",
description="Pin a visible agent to the top of the current user's lists. Idempotent.",
)
async def pin_agent(agent_id: str, request: Request) -> None:
_require_agents_api_enabled()
agent_id = _validate_storage_id(agent_id)
try:
store = get_agent_store(request)
user_id = _current_user_id(request)
record = await store.get_visible(agent_id, user_id)
if record is None:
raise HTTPException(status_code=404, detail=f"Agent '{agent_id}' not found")
await store.add_pin(user_id, agent_id)
logger.info("User '%s' pinned agent '%s'", user_id, agent_id)
except HTTPException:
raise
except Exception as e:
logger.error("Failed to pin agent '%s': %s", agent_id, e, exc_info=True)
raise HTTPException(status_code=500, detail=f"Failed to pin agent: {str(e)}") from e
@router.delete(
"/agents/{agent_id}/pin",
status_code=204,
summary="Unpin Agent",
description="Remove the current user's pin from an agent. No-op when not pinned.",
)
async def unpin_agent(agent_id: str, request: Request) -> None:
_require_agents_api_enabled()
agent_id = _validate_storage_id(agent_id)
try:
store = get_agent_store(request)
user_id = _current_user_id(request)
await store.remove_pin(user_id, agent_id)
logger.info("User '%s' unpinned agent '%s'", user_id, agent_id)
except HTTPException:
raise
except Exception as e:
logger.error("Failed to unpin agent '%s': %s", agent_id, e, exc_info=True)
raise HTTPException(status_code=500, detail=f"Failed to unpin agent: {str(e)}") from e
@router.put(
"/agents/{agent_id}/featured",
response_model=AgentResponse,
summary="Set Agent Featured Rank (admin)",
description=(
@@ -864,10 +1031,19 @@ async def set_agent_featured(
raise HTTPException(status_code=500, detail=f"Failed to set featured: {str(e)}") from e
class PrivateSquareListRequest(BaseModel):
class PrivateSquareListRequest(BaseModel):
"""Request body for listing all private-square agents."""
password: str = Field(..., description="Plain-text private-square password. Verified against the admin-configured hash before the listing is returned.")
search: str | None = Field(default=None, description="Case-insensitive fuzzy match on agent name (raw or desensitized).")
tag_ids: list[str] | None = Field(default=None, description="Keep agents carrying any of these tags (OR).")
page: int | None = Field(default=None, ge=1, description="1-based page index. Providing this enables server-side pagination.")
page_size: int = Field(default=20, ge=1, le=200, description="Items per page when paginating.")
sort: str = Field(
default="time_desc",
pattern="^(time_desc|time_asc|updated_desc|updated_asc|name_asc|name_desc)$",
description="Sort order. Default is newest created first.",
)
class AgentPrivateUpdateRequest(BaseModel):
@@ -899,14 +1075,28 @@ async def list_private_square(body: PrivateSquareListRequest, request: Request)
try:
store = get_agent_store(request)
tag_store: TagStore = get_tag_store(request)
await _sync_legacy_agents(store)
records = await store.list_private()
responses = [_record_to_response(r, include_soul=True) for r in records]
await _maybe_sync_legacy_agents(store)
# user_id is unused by the "private" scope filter (the password is the
# gate), but list_paginated requires it for its signature.
user_id = _current_user_id(request)
assignments = await tag_store.list_assignments("agent", [r.id for r in responses])
for resp in responses:
resp.tags = assignments.get(resp.id, [])
return AgentsListResponse(agents=responses)
if body.page is not None:
rows, total = await store.list_paginated(
user_id=user_id,
scope="private",
search=body.search,
tag_ids=body.tag_ids,
page=body.page,
page_size=body.page_size,
sort=body.sort,
)
responses = await _build_list_responses(store, tag_store, user_id, rows)
return AgentsListResponse(agents=responses, total=total, page=body.page, page_size=body.page_size)
records = await store.list_private()
responses = await _build_list_responses(store, tag_store, user_id, records)
responses = _sort_agent_responses(responses, body.sort)
return AgentsListResponse(agents=responses, total=len(responses), page=1, page_size=len(responses))
except HTTPException:
raise
except Exception as e:
@@ -22,7 +22,7 @@ from app.gateway.auth import (
from app.gateway.auth.config import get_auth_config
from app.gateway.auth.errors import AuthErrorCode, AuthErrorResponse
from app.gateway.csrf_middleware import is_secure_request
from app.gateway.deps import get_agent_store, get_current_user_from_request, get_local_provider, get_skill_store
from app.gateway.deps import get_agent_store, get_current_user_from_request, get_local_provider, get_run_store, get_skill_store
logger = logging.getLogger(__name__)
@@ -336,7 +336,7 @@ def _set_session_cookie(response: Response, token: str, request: Request) -> Non
# production multi-worker setups, replace this with a shared store (Redis,
# database-backed counter) to enforce a true per-IP limit.
_MAX_LOGIN_ATTEMPTS = 5
_MAX_LOGIN_ATTEMPTS = 100
_LOCKOUT_SECONDS = 300 # 5 minutes
# ip → (fail_count, lock_until_timestamp)
@@ -749,6 +749,7 @@ class UserListItem(BaseModel):
system_role: Literal["admin", "user"]
skill_count: int
agent_count: int
qa_count: int = 0 # 问答次数: number of runs (one run = one question)
@router.get("/users", response_model=list[UserListItem])
@@ -757,11 +758,19 @@ async def list_all_users(
skill_store: SkillStore = Depends(get_skill_store),
agent_store: AgentStore = Depends(get_agent_store),
):
"""List all users with their skill and agent counts. Admin only."""
"""List all users with their skill, agent and Q&A counts. Admin only."""
user = await get_current_user_from_request(request)
if user.system_role != "admin":
raise HTTPException(status_code=403, detail="Admin only")
users = await get_local_provider().list_users()
# One GROUP BY query for everyone's 问答次数, so the list stays a single
# aggregate read instead of a per-user count. Best-effort: if the run store
# is unavailable the column simply renders as 0 rather than failing the list.
try:
qa_counts = await get_run_store(request).count_by_user()
except Exception:
logger.warning("Failed to load per-user run counts for user list", exc_info=True)
qa_counts = {}
result = []
for u in users:
uid = str(u.id)
@@ -772,6 +781,7 @@ async def list_all_users(
system_role=u.system_role,
skill_count=await skill_store.count_owned(uid),
agent_count=await agent_store.count_owned(uid),
qa_count=qa_counts.get(uid, 0),
)
)
return result
@@ -121,6 +121,26 @@ async def _resolve_user_name_to_id(name: str | None) -> str | None:
return str(user.id) if user is not None else None
async def _resolve_user_names_to_ids(name: str | None) -> list[str]:
"""Fuzzy-resolve a username fragment to every matching account's UUID.
Powers the 按用户名模糊搜索 filter: typing part of a username matches all
users whose email local part contains it. Returns an empty list when the
fragment matches nobody, so the caller can short-circuit to zero rows.
"""
if not name or not name.strip():
return []
try:
provider = get_local_provider()
except Exception:
return []
try:
users = await provider.find_users_by_email_local_part_like(name.strip())
except Exception:
return []
return [str(u.id) for u in users]
async def _attach_usernames(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""Resolve each row's ``user_id`` UUID to a friendly ``user_name``.
@@ -161,9 +181,11 @@ def _build_query(
until: str | None,
limit: int,
offset: int,
user_ids: list[str] | None = None,
) -> LlmMetricsQuery:
return LlmMetricsQuery(
user_id=user_id or None,
user_ids=user_ids or None,
model_name=model_name or None,
status=status or None,
since=_parse_dt(since),
@@ -187,14 +209,15 @@ async def list_llm_metrics(
) -> LlmMetricsListResponse:
await _require_admin(request)
store = _require_store(request)
user_ids: list[str] | None = None
if not user_id and user_name:
resolved = await _resolve_user_name_to_id(user_name)
if resolved is None:
# Username unknown → no rows; short-circuit instead of a global scan.
user_ids = await _resolve_user_names_to_ids(user_name)
if not user_ids:
# No username matches → no rows; short-circuit the global scan.
return LlmMetricsListResponse(items=[], total=0, limit=limit, offset=offset)
user_id = resolved
query = _build_query(
user_id=user_id,
user_ids=user_ids,
model_name=model_name,
status=status,
since=since,
@@ -236,14 +259,13 @@ async def export_llm_metrics(
"""Stream the full filtered metric set as an ``.xlsx`` file."""
await _require_admin(request)
store = _require_store(request)
user_ids: list[str] | None = None
if not user_id and user_name:
resolved = await _resolve_user_name_to_id(user_name)
if resolved is None:
# No matching user — return an empty xlsx so the user still gets a
# file (matching the list view's empty state).
user_id = "__no_such_user__"
else:
user_id = resolved
user_ids = await _resolve_user_names_to_ids(user_name)
if not user_ids:
# No matching user — sentinel id that matches nothing, so the export
# still produces an empty xlsx (matching the list view's empty state).
user_ids = ["__no_such_user__"]
try:
from openpyxl import Workbook
@@ -252,6 +274,7 @@ async def export_llm_metrics(
query = _build_query(
user_id=user_id,
user_ids=user_ids,
model_name=model_name,
status=status,
since=since,
@@ -7,7 +7,7 @@ import re
import subprocess
import sys
import tempfile
from datetime import datetime
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
@@ -119,8 +119,10 @@ class SkillResponse(BaseModel):
view_count: int = Field(default=0, description="Number of times the skill has been viewed")
patch_count: int = Field(default=0, description="Number of times the skill has been patched")
last_activity_at: datetime | None = Field(default=None, description="Most recent activity timestamp")
created_at: datetime | None = Field(default=None, description="Skill creation timestamp when known")
state: str = Field(default="active", description="Lifecycle state: active, stale, or archived")
pinned: bool = Field(default=False, description="Whether this skill is pinned (excluded from curation)")
featured: bool = Field(default=False, description="Whether this skill is pinned to the top of the list (sort only)")
source: str = Field(default="upload", description="Origin: 'agent' for agent-created, 'upload' for user-uploaded")
owner_user_id: str | None = Field(default=None, description="Custom skill owner (null = built-in or legacy)")
published: bool = Field(default=False, description="Whether the owner has made this skill visible to other users")
@@ -241,8 +243,20 @@ def _skill_to_response(
from deerflow.skills.usage import get_entry, latest_activity_at
entry = get_entry(skill.name)
created_at_str = entry.get("created_at")
last_at_str = latest_activity_at(skill.name)
created_at: datetime | None = None
last_at: datetime | None = None
if created_at_str:
try:
created_at = datetime.fromisoformat(created_at_str)
except Exception:
pass
if created_at is None:
try:
created_at = datetime.fromtimestamp(skill.skill_file.stat().st_mtime, tz=UTC)
except Exception:
pass
if last_at_str:
try:
last_at = datetime.fromisoformat(last_at_str)
@@ -278,8 +292,10 @@ def _skill_to_response(
view_count=entry.get("view_count", 0),
patch_count=entry.get("patch_count", 0),
last_activity_at=last_at,
created_at=created_at,
state=entry.get("state", "active"),
pinned=entry.get("pinned", False),
featured=entry.get("featured", False),
source=entry.get("source", "upload"),
owner_user_id=owner_user_id,
published=published,
@@ -688,6 +704,17 @@ async def _attach_and_filter_tags(
return responses
def _sort_skill_responses(responses: list[SkillResponse], sort: str) -> list[SkillResponse]:
reverse = sort not in {"time_asc", "activity_asc", "name_asc", "usage_asc"}
if sort in {"name_asc", "name_desc"}:
return sorted(responses, key=lambda r: (r.name or "").lower(), reverse=reverse)
if sort in {"usage_desc", "usage_asc"}:
return sorted(responses, key=lambda r: int(r.use_count or 0), reverse=reverse)
if sort in {"activity_desc", "activity_asc"}:
return sorted(responses, key=lambda r: str(r.last_activity_at or r.created_at or ""), reverse=reverse)
return sorted(responses, key=lambda r: str(r.created_at or r.last_activity_at or ""), reverse=reverse)
@router.get(
"/skills",
response_model=SkillsListResponse,
@@ -701,6 +728,11 @@ async def list_skills(
tag_store: TagStore = Depends(get_tag_store),
search: str | None = Query(default=None, description="Case-insensitive fuzzy match on skill name"),
tag_ids: list[str] | None = Query(default=None, description="Keep skills carrying any of these tags (OR)"),
sort: str = Query(
default="time_desc",
pattern="^(time_desc|time_asc|activity_desc|activity_asc|usage_desc|usage_asc|name_asc|name_desc)$",
description="Sort order. Default is newest created/modified first.",
),
admin: bool = Query(
default=False,
description=(
@@ -754,6 +786,7 @@ async def list_skills(
)
)
responses = await _attach_and_filter_tags(tag_store, responses, search, tag_ids)
responses = _sort_skill_responses(responses, sort)
return SkillsListResponse(skills=responses)
except HTTPException:
raise
@@ -1438,6 +1471,42 @@ async def update_custom_skill_pinned(
raise HTTPException(status_code=500, detail=f"Failed to update pinned: {e}")
class SkillFeaturedRequest(BaseModel):
"""Request body for pinning a custom skill to the top of the list."""
featured: bool = Field(..., description="Whether to pin the skill to the top of the list (sort only)")
@router.put(
"/skills/custom/{skill_name}/featured",
response_model=SkillResponse,
summary="Pin or Unpin a Custom Skill to Top",
description="Pin a skill to the top of the list (sort only; orthogonal to pinned/anti-archive). Only the owner or admins may invoke.",
)
async def update_custom_skill_featured(
skill_name: str,
body: SkillFeaturedRequest,
request: Request,
config: AppConfig = Depends(get_config),
skill_store: SkillStore = Depends(get_skill_store),
) -> SkillResponse:
try:
skill_name = skill_name.replace("\r\n", "").replace("\n", "")
user_id = _current_user_id(request)
is_admin = _is_admin_user(request)
record, _mode = await _get_editable_skill_or_raise(skill_store, skill_name, user_id, is_admin=is_admin)
from deerflow.skills.usage import set_featured as _usage_set_featured
_usage_set_featured(skill_name, body.featured)
return _reload_skill_response(skill_name, record, config)
except HTTPException:
raise
except Exception as e:
logger.error("Failed to update featured for %s: %s", skill_name, e, exc_info=True)
raise HTTPException(status_code=500, detail=f"Failed to update featured: {e}")
@router.get("/skills/custom/{skill_name}/history", response_model=CustomSkillHistoryResponse, summary="Get Custom Skill History")
async def get_custom_skill_history(skill_name: str, config: AppConfig = Depends(get_config)) -> CustomSkillHistoryResponse:
try:
@@ -10,7 +10,9 @@ from pydantic import BaseModel, Field
from app.gateway.auth.password import hash_password_async
from app.gateway.deps import get_optional_user_from_request
from deerflow.config import get_app_config
from deerflow.config.system_settings import (
LeaderboardSettings,
PromptPrefixSettings,
SystemSettings,
load_system_settings,
@@ -57,6 +59,22 @@ class CitationDisplayUpdateRequest(BaseModel):
cleanup_words: list[str] | None = None
class LeaderboardSettingsResponse(BaseModel):
exclude_user_ids: list[str] = Field(default_factory=list)
include_scheduled: bool = True
include_admins: bool = True
include_failed: bool = True
cleanup_system_questions: bool = True
class LeaderboardSettingsUpdateRequest(BaseModel):
exclude_user_ids: list[str] | None = None
include_scheduled: bool | None = None
include_admins: bool | None = None
include_failed: bool | None = None
cleanup_system_questions: bool | None = None
def _to_response(settings: PromptPrefixSettings) -> PromptPrefixResponse:
return PromptPrefixResponse(
enabled=settings.enabled,
@@ -91,6 +109,44 @@ async def update_prompt_prefix(body: PromptPrefixUpdateRequest, request: Request
return _to_response(saved.prompt_prefix)
def _leaderboard_to_response(settings: LeaderboardSettings) -> LeaderboardSettingsResponse:
return LeaderboardSettingsResponse(
exclude_user_ids=list(dict.fromkeys([uid for uid in settings.exclude_user_ids if uid])),
include_scheduled=settings.include_scheduled,
include_admins=settings.include_admins,
include_failed=settings.include_failed,
cleanup_system_questions=settings.cleanup_system_questions,
)
@router.get("/leaderboard", response_model=LeaderboardSettingsResponse)
async def get_leaderboard_settings(request: Request) -> LeaderboardSettingsResponse:
"""Return global leaderboard aggregation settings. Admin only."""
await _require_admin(request)
return _leaderboard_to_response(load_system_settings().leaderboard)
@router.put("/leaderboard", response_model=LeaderboardSettingsResponse)
async def update_leaderboard_settings(
body: LeaderboardSettingsUpdateRequest,
request: Request,
) -> LeaderboardSettingsResponse:
"""Update global leaderboard aggregation settings. Admin only."""
await _require_admin(request)
current = load_system_settings()
fields = body.model_dump(exclude_unset=True)
update: dict[str, object] = {}
if "exclude_user_ids" in fields and fields["exclude_user_ids"] is not None:
update["exclude_user_ids"] = list(dict.fromkeys([str(uid).strip() for uid in fields["exclude_user_ids"] if str(uid).strip()]))
for key in ("include_scheduled", "include_admins", "include_failed", "cleanup_system_questions"):
if key in fields and fields[key] is not None:
update[key] = bool(fields[key])
leaderboard = current.leaderboard.model_copy(update=update)
saved = save_system_settings(current.model_copy(update={"leaderboard": leaderboard}))
return _leaderboard_to_response(saved.leaderboard)
@router.get("/citation-display", response_model=CitationDisplayResponse)
async def get_citation_display() -> CitationDisplayResponse:
"""Return global citation display cleanup settings."""
@@ -200,3 +256,23 @@ async def update_appearance_settings(
else:
saved = current
return AppearanceResponse(default_theme=saved.appearance.default_theme)
class LogSystemResponse(BaseModel):
"""External log-system entry shown in the admin settings menu.
Sourced from ``config.yaml`` (``log_system`` section), so it is read-only
here — there is no PUT counterpart.
"""
enabled: bool = Field(..., description="Whether the entry is shown in the settings menu")
url: str = Field(..., description="External log-system URL opened in a new tab")
label: str = Field(..., description="Menu label shown for the entry")
@router.get("/log-system", response_model=LogSystemResponse)
async def get_log_system(request: Request) -> LogSystemResponse:
"""Return the admin-configured external log-system link from config.yaml. Admin only."""
await _require_admin(request)
cfg = get_app_config().log_system
return LogSystemResponse(enabled=cfg.enabled, url=cfg.url, label=cfg.label)
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,66 @@
"""Per-user UI preferences (account-bound, follow the user across devices).
Currently exposes a single toggle: ``hide_admin_prompt_prefix`` — when true the
chat input hides the admin-configured prompt-prefix switch so the user only uses
their own prompts. Kept as a small key-value (JSON) store so new toggles do not
each need a schema change.
"""
from __future__ import annotations
import logging
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel, Field
from deerflow.runtime.user_context import get_effective_user_id
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/user-preferences", tags=["user-preferences"])
class UserPreferencesResponse(BaseModel):
hide_admin_prompt_prefix: bool = False
class UserPreferencesUpdateRequest(BaseModel):
hide_admin_prompt_prefix: bool | None = Field(default=None)
def _current_user_id(request: Request) -> str:
user = getattr(request.state, "user", None)
if user is not None:
return str(user.id)
return get_effective_user_id()
def _get_store(request: Request):
store = getattr(request.app.state, "user_preferences_store", None)
if store is None:
raise HTTPException(status_code=503, detail="User preferences store not available")
return store
def _to_response(prefs: dict) -> UserPreferencesResponse:
return UserPreferencesResponse(
hide_admin_prompt_prefix=bool(prefs.get("hide_admin_prompt_prefix", False)),
)
@router.get("", response_model=UserPreferencesResponse)
async def get_user_preferences(request: Request) -> UserPreferencesResponse:
store = _get_store(request)
user_id = _current_user_id(request)
prefs = await store.get_for_user(user_id)
return _to_response(prefs)
@router.put("", response_model=UserPreferencesResponse)
async def update_user_preferences(request: Request, body: UserPreferencesUpdateRequest) -> UserPreferencesResponse:
store = _get_store(request)
user_id = _current_user_id(request)
patch = body.model_dump(exclude_unset=True, exclude_none=True)
if not patch:
return _to_response(await store.get_for_user(user_id))
prefs = await store.merge_for_user(user_id, patch)
return _to_response(prefs)
@@ -24,6 +24,8 @@ class UserPromptResponse(BaseModel):
title: str = ""
content: str
sort_order: int = 0
# Optional single-agent association. NULL = global (shown for every agent).
agent_id: str | None = None
created_at: datetime | str | None = None
updated_at: datetime | str | None = None
@@ -36,12 +38,16 @@ class UserPromptCreateRequest(BaseModel):
title: str = Field(default="", max_length=255)
content: str = Field(..., min_length=1)
sort_order: int = Field(default=0)
agent_id: str | None = Field(default=None, max_length=64)
class UserPromptUpdateRequest(BaseModel):
title: str | None = Field(default=None, max_length=255)
content: str | None = Field(default=None, min_length=1)
sort_order: int | None = None
# Sent on every save; an empty string / null clears the association. Use a
# sentinel so "field omitted" (don't touch) differs from "set to null".
agent_id: str | None = Field(default=None, max_length=64)
def _current_user_id(request: Request) -> str:
@@ -77,6 +83,7 @@ async def create_user_prompt(request: Request, body: UserPromptCreateRequest) ->
"title": (body.title or "").strip(),
"content": body.content,
"sort_order": body.sort_order,
"agent_id": (body.agent_id or "").strip() or None,
},
)
return UserPromptResponse(**row)
@@ -93,6 +100,11 @@ async def update_user_prompt(request: Request, prompt_id: str, body: UserPromptU
data["content"] = body.content
if body.sort_order is not None:
data["sort_order"] = body.sort_order
# ``agent_id`` is nullable: only act when the caller actually sent the field
# (``model_fields_set``), so an explicit null clears the association while an
# omitted field leaves it untouched.
if "agent_id" in body.model_fields_set:
data["agent_id"] = (body.agent_id or "").strip() or None
row = await store.update(prompt_id, user_id, data)
if row is None:
raise HTTPException(status_code=404, detail="Prompt not found")
@@ -207,6 +207,10 @@ auth_login:
verify_ssl: false
ca_cert_path: "" # 可选:PEM 格式 CA 包路径,设置后优先生效并启用校验
# 日志系统入口(仅管理员可见):右下角「设置」菜单中会多出一个「日志系统」入口,
# 点击后在新标签页打开下面的 url。enabled 关掉则不显示该入口。
database:
backend: sqlite
sqlite_dir: .deer-flow/data
@@ -292,3 +296,8 @@ ai_writing:
keyword_model: deepseek-chat # 检索词生成专用模型,留空回退写作主模型
rank_model: deepseek-chat # 素材排序专用模型,留空回退写作主模型
max_materials: 20 # 合并去重后的素材总量上限,范围 [1, 100],默认 20
log_system:
enabled: true
url: "https://your-log-system.example.com"
label: "日志系统"
@@ -1,8 +1,10 @@
"""Middleware to inject uploaded files information into agent context."""
import logging
from pathlib import Path
from typing import NotRequired, override
import logging
import shutil
from pathlib import Path
from typing import NotRequired, override
from urllib.parse import unquote
from langchain.agents import AgentState
from langchain.agents.middleware import AgentMiddleware
@@ -13,7 +15,7 @@ from deerflow.config.paths import Paths, get_paths
from deerflow.runtime.user_context import get_effective_user_id
from deerflow.utils.file_conversion import extract_outline
logger = logging.getLogger(__name__)
logger = logging.getLogger(__name__)
_OUTLINE_PREVIEW_LINES = 5
@@ -22,7 +24,8 @@ _OUTLINE_PREVIEW_LINES = 5
# configured vision model), never by reading their bytes. Listing them with the
# generic read_file/grep guidance made the agent try to parse image binaries
# with bash/python, which fails and wastes turns.
_IMAGE_EXTENSIONS = {".jpg", ".jpeg", ".png", ".webp", ".gif", ".bmp"}
_IMAGE_EXTENSIONS = {".jpg", ".jpeg", ".png", ".webp", ".gif", ".bmp"}
_ARTIFACT_URL_MARKER = "/artifacts/"
def _extract_outline_for_file(file_path: Path) -> tuple[list[dict], list[str]]:
@@ -69,7 +72,7 @@ class UploadsMiddlewareState(AgentState):
uploaded_files: NotRequired[list[dict] | None]
class UploadsMiddleware(AgentMiddleware[UploadsMiddlewareState]):
class UploadsMiddleware(AgentMiddleware[UploadsMiddlewareState]):
"""Middleware to inject uploaded files information into the agent context.
Reads file metadata from the current message's additional_kwargs.files
@@ -122,7 +125,7 @@ class UploadsMiddleware(AgentMiddleware[UploadsMiddlewareState]):
lines.append(" Use `grep` to search for keywords (e.g. `grep(pattern='keyword', path='/mnt/user-data/uploads/')`).")
lines.append("")
def _create_files_message(self, new_files: list[dict], historical_files: list[dict]) -> str:
def _create_files_message(self, new_files: list[dict], historical_files: list[dict]) -> str:
"""Create a formatted message listing uploaded files.
Args:
@@ -160,9 +163,61 @@ class UploadsMiddleware(AgentMiddleware[UploadsMiddlewareState]):
lines.append("- Only fall back to web search if the file content is clearly insufficient to answer the question.")
lines.append("</uploaded_files>")
return "\n".join(lines)
def _files_from_kwargs(self, message: HumanMessage, uploads_dir: Path | None = None) -> list[dict] | None:
return "\n".join(lines)
def _import_file_from_artifact_url(
self,
file_info: dict,
uploads_dir: Path | None,
) -> bool:
"""Copy a previously uploaded file into the current thread when possible.
iframe-style callers may pass file metadata that points at an upload
from another DeerFlow thread. The current run thread still needs a real
file under its own ``/mnt/user-data/uploads`` directory, otherwise this
middleware will filter the entry out during existence checks.
"""
filename = file_info.get("filename") or ""
artifact_url = str(file_info.get("artifact_url") or "").strip()
if not filename or uploads_dir is None or not artifact_url:
return False
if not artifact_url.startswith("/api/threads/") or _ARTIFACT_URL_MARKER not in artifact_url:
return False
prefix, encoded_virtual_path = artifact_url.split(_ARTIFACT_URL_MARKER, 1)
source_thread_id = prefix.removeprefix("/api/threads/").strip("/")
if not source_thread_id or not encoded_virtual_path:
return False
virtual_path = "/" + unquote(encoded_virtual_path.lstrip("/"))
if Path(virtual_path).name != filename:
return False
try:
source_path = self._paths.resolve_virtual_path(
source_thread_id,
virtual_path,
user_id=get_effective_user_id(),
)
except ValueError:
logger.debug("Failed to resolve uploaded artifact path: %s", artifact_url, exc_info=True)
return False
if not source_path.is_file():
return False
uploads_dir.mkdir(parents=True, exist_ok=True)
target_path = uploads_dir / filename
shutil.copy2(source_path, target_path)
source_markdown = source_path.with_suffix(".md")
if source_markdown.is_file() and source_markdown != source_path:
shutil.copy2(source_markdown, target_path.with_suffix(".md"))
return True
def _files_from_kwargs(self, message: HumanMessage, uploads_dir: Path | None = None) -> list[dict] | None:
"""Extract file info from message additional_kwargs.files.
The frontend sends uploaded file metadata in additional_kwargs.files
@@ -183,16 +238,17 @@ class UploadsMiddleware(AgentMiddleware[UploadsMiddlewareState]):
files = []
for f in kwargs_files:
if not isinstance(f, dict):
continue
filename = f.get("filename") or ""
if not filename or Path(filename).name != filename:
continue
if uploads_dir is not None and not (uploads_dir / filename).is_file():
continue
files.append(
{
"filename": filename,
if not isinstance(f, dict):
continue
filename = f.get("filename") or ""
if not filename or Path(filename).name != filename:
continue
if uploads_dir is not None and not (uploads_dir / filename).is_file():
if not self._import_file_from_artifact_url(f, uploads_dir):
continue
files.append(
{
"filename": filename,
"size": int(f.get("size") or 0),
"path": f"/mnt/user-data/uploads/{filename}",
"extension": Path(filename).suffix,
@@ -243,6 +299,17 @@ class UploadsMiddleware(AgentMiddleware[UploadsMiddlewareState]):
# Get newly uploaded files from the current message's additional_kwargs.files
new_files = self._files_from_kwargs(last_message, uploads_dir) or []
# Only inject when the *current* message actually carries new uploads.
# Turns without a fresh upload must not re-inject previously uploaded
# files: the original upload turn already injected (and persisted) their
# <uploaded_files> block, and the files remain physically available for
# read_file/grep/view_image. Re-injecting on every later message made the
# agent re-reference old images/docs, and — since the frontend parses
# files out of the <uploaded_files> content when additional_kwargs.files
# is empty — re-rendered the old attachment on each new question.
if not new_files:
return None
# Collect historical files from the uploads directory (all except the new ones)
new_filenames = {f["filename"] for f in new_files}
historical_files: list[dict] = []
@@ -270,9 +337,6 @@ class UploadsMiddleware(AgentMiddleware[UploadsMiddlewareState]):
file["outline"] = outline
file["outline_preview"] = preview
if not new_files and not historical_files:
return None
logger.debug(f"New files: {[f['filename'] for f in new_files]}, historical: {[f['filename'] for f in historical_files]}")
# Create files message and prepend to the last human message content
@@ -0,0 +1,92 @@
"""Server-side mirror of the frontend agent-name desensitization map.
The web UI displays agent names through a static term→abbreviation map
(``frontend-web/src/lib/desensitize.ts``) — e.g. the stored name "态势分析" is
shown as "TS分析". The gallery search box therefore matches against the
*desensitized* display form, not the raw stored name.
When search moves server-side (paginated listing), the backend must match the
same way, so this module mirrors that static map verbatim. The frontend map is
never extended at runtime (``addDesensitizeEntries`` is defined but never
called), so a static copy stays in sync.
Used by the SQL listing query to match "raw name OR desensitized name".
"""
from __future__ import annotations
# Mirror of DESENSITIZE_MAP in frontend-web/src/lib/desensitize.ts.
# Keep entries identical to the frontend so server-side and client-side search
# behave the same. Order does not affect membership tests.
DESENSITIZE_MAP: dict[str, str] = {
"态势": "TS",
"你好": "nh",
"滨海作战体系": "第一张网",
"滨海作战部队": "第一张网",
"Taiwan Air Force Flight Units": "第二张网",
"台海空军飞行部队": "第二张网",
"空军": "KJ",
"海军": "HJ",
"陆军": "LJ",
"陆战队": "LZD",
"海巡": "HX",
"憲兵": "XB",
"資通電軍": "ZTDJ",
"情报": "QB",
"指挥": "ZH",
"信息准备": "XX准备",
"行为体": "XWT",
"驱动因素": "QDYS",
"叙事": "XS",
"前三定": "QSD",
"受众目标": "SZMB",
"作战": "ZZ",
"目的": "MD",
"关键传播节点": "GJCBJD",
"战法": "ZF",
"行动": "XD",
"打击": "DJ",
"目标": "MB",
"战力": "ZL",
"筹划": "COH",
"策划": "CEH",
"模拟": "MN",
"推演": "TY",
"认知": "RZ",
"触及": "CJ",
"军": "J",
"美军": "MJ",
"台军": "TJ",
}
def desensitize_name(name: str) -> str:
"""Return ``name`` with every mapped term replaced by its abbreviation.
Mirrors the frontend ``desensitize()`` — a sequence of literal replacements.
"""
out = name
for term, abbr in DESENSITIZE_MAP.items():
if term in out:
out = out.replace(term, abbr)
return out
def name_matches(name: str, needle: str) -> bool:
"""True when ``needle`` (case-insensitive) is in the raw OR desensitized name."""
n = needle.strip().lower()
if not n:
return True
return n in name.lower() or n in desensitize_name(name).lower()
def source_terms_for_needle(needle: str) -> list[str]:
"""Return the raw source terms whose abbreviation contains ``needle``.
Lets the SQL search add ``name LIKE %term%`` clauses so a user typing an
abbreviation (e.g. "TS") still matches the stored raw term (e.g. "态势").
"""
n = needle.strip().lower()
if not n:
return []
return [term for term, abbr in DESENSITIZE_MAP.items() if n in abbr.lower()]
@@ -15,6 +15,7 @@ from deerflow.config.database_config import DatabaseConfig
from deerflow.config.extensions_config import ExtensionsConfig
from deerflow.config.gateway_config import GatewayConfig
from deerflow.config.guardrails_config import GuardrailsConfig, load_guardrails_config_from_dict
from deerflow.config.log_system_config import LogSystemConfig
from deerflow.config.login_config import AuthLoginConfig
from deerflow.config.memory_config import MemoryConfig, load_memory_config_from_dict
from deerflow.config.model_config import ModelConfig
@@ -108,6 +109,7 @@ class AppConfig(BaseModel):
models: list[ModelConfig] = Field(default_factory=list, description="Available models")
vision: VisionConfig = Field(default_factory=VisionConfig, description="Image recognition (识图) configuration")
auth_login: AuthLoginConfig = Field(default_factory=AuthLoginConfig, description="Custom login modes (username password gate, token exchange)")
log_system: LogSystemConfig = Field(default_factory=LogSystemConfig, description="Admin-only external log-system entry shown in the settings menu")
sandbox: SandboxConfig = Field(description="Sandbox configuration")
tools: list[ToolConfig] = Field(default_factory=list, description="Available tools")
tool_groups: list[ToolGroupConfig] = Field(default_factory=list, description="Available tool groups")
@@ -0,0 +1,18 @@
"""Configuration for the admin-only external log-system entry.
A small read-only section in ``config.yaml`` that lets a deployment point the
"日志系统" entry (shown in the bottom-right settings menu for admins) at an
external log/observability system. The URL is opened in a new browser tab.
"""
from __future__ import annotations
from pydantic import BaseModel, Field
class LogSystemConfig(BaseModel):
"""External log-system link surfaced in the admin settings menu."""
enabled: bool = Field(default=False, description="Whether the log-system entry is shown in the settings menu")
url: str = Field(default="", description="External log-system URL opened in a new tab when the entry is clicked")
label: str = Field(default="日志系统", description="Menu label shown for the entry")
@@ -78,6 +78,17 @@ class PrivateSquareSettings(BaseModel):
model_config = ConfigDict(extra="ignore")
class LeaderboardSettings(BaseModel):
"""Admin-configurable defaults for the user leaderboard analytics."""
exclude_user_ids: list[str] = Field(default_factory=list, description="User ids excluded from leaderboard aggregation.")
include_scheduled: bool = Field(default=True, description="Whether scheduled task runs participate in leaderboard statistics.")
include_admins: bool = Field(default=True, description="Whether admin accounts participate in leaderboard statistics.")
include_failed: bool = Field(default=True, description="Whether failed runs and provider calls participate in leaderboard statistics.")
cleanup_system_questions: bool = Field(default=True, description="Hide attachment/system marker questions such as <uploaded_files>.")
model_config = ConfigDict(extra="ignore")
class SystemSettings(BaseModel):
"""Top-level system settings document."""
@@ -85,6 +96,7 @@ class SystemSettings(BaseModel):
private_square: PrivateSquareSettings = Field(default_factory=PrivateSquareSettings)
appearance: AppearanceSettings = Field(default_factory=AppearanceSettings)
citation_display: CitationDisplaySettings = Field(default_factory=CitationDisplaySettings)
leaderboard: LeaderboardSettings = Field(default_factory=LeaderboardSettings)
model_config = ConfigDict(extra="ignore")
@@ -4,13 +4,20 @@ from __future__ import annotations
from deerflow.persistence.agents.base import AgentStore
from deerflow.persistence.agents.memory import MemoryAgentStore
from deerflow.persistence.agents.model import AgentFavoriteRow, AgentRow
from deerflow.persistence.agents.model import (
AgentExtrasRow,
AgentFavoriteRow,
AgentRow,
AgentSkillRow,
)
from deerflow.persistence.agents.sql import AgentRepository
__all__ = [
"AgentExtrasRow",
"AgentFavoriteRow",
"AgentRepository",
"AgentRow",
"AgentSkillRow",
"AgentStore",
"MemoryAgentStore",
"make_agent_store",
@@ -118,6 +118,69 @@ class AgentStore(ABC):
"""
raise NotImplementedError
@abstractmethod
async def list_paginated(
self,
*,
user_id: str,
scope: str | None,
search: str | None,
tag_ids: list[str] | None,
page: int,
page_size: int,
admin: bool = False,
sort: str = "time_desc",
) -> tuple[list[dict[str, Any]], int]:
"""Return one page of agents plus the total matching count.
``scope``:
- ``"mine"`` — owned by ``user_id`` OR favorited by ``user_id``.
- ``"square"`` — built-in or published, and not private-square.
- ``"private"``— flagged ``is_private=True`` (caller gates access).
- ``None`` — the default listing visibility (``list_visible``).
``admin=True`` bypasses the visibility/scope filter entirely (every
agent), but still honors ``search`` / ``tag_ids``. Caller authorizes.
``search`` matches the raw name OR its desensitized display form.
``tag_ids`` keeps agents carrying ANY of the tags (OR). Both filters,
the COUNT, and LIMIT/OFFSET all execute in SQL.
Returns ``(rows, total)`` where ``rows`` is at most ``page_size`` items
for the 1-based ``page``.
"""
raise NotImplementedError
# --- Listing cache (skills / tool_groups / model) ---
@abstractmethod
async def replace_skills(self, agent_id: str, skills: list[str] | None) -> None:
"""Replace the cached skill list for ``agent_id``.
``None`` and ``[]`` both clear the cache to "no skills" (mirrors the
tri-state semantics of ``AgentConfig.skills``).
"""
raise NotImplementedError
@abstractmethod
async def set_extras(self, agent_id: str, *, tool_groups: list[str] | None, model: str | None) -> None:
"""Upsert the cached ``tool_groups`` / ``model`` for ``agent_id``."""
raise NotImplementedError
@abstractmethod
async def get_extras_for(self, agent_ids: list[str]) -> dict[str, dict[str, Any]]:
"""Batch-fetch cached listing extras for ``agent_ids``.
Returns ``{agent_id: {"skills": list[str] | None, "tool_groups":
list[str] | None, "model": str | None}}``. An agent that was never
mirrored is absent from the mapping (caller treats it as unknown).
"""
raise NotImplementedError
@abstractmethod
async def delete_extras_for_agent(self, agent_id: str) -> None:
"""Drop the cached skills + extras rows for ``agent_id`` (on delete)."""
raise NotImplementedError
# --- Favorites ("收藏") ---
@abstractmethod
@@ -136,6 +199,28 @@ class AgentStore(ABC):
raise NotImplementedError
@abstractmethod
async def delete_favorites_for_agent(self, agent_id: str) -> None:
"""Drop every user's favorite of ``agent_id`` (called on agent delete)."""
raise NotImplementedError
async def delete_favorites_for_agent(self, agent_id: str) -> None:
"""Drop every user's favorite of ``agent_id`` (called on agent delete)."""
raise NotImplementedError
# --- Per-user pins ("置顶") ---
@abstractmethod
async def add_pin(self, user_id: str, agent_id: str) -> None:
"""Pin ``agent_id`` for ``user_id``. Idempotent."""
raise NotImplementedError
@abstractmethod
async def remove_pin(self, user_id: str, agent_id: str) -> bool:
"""Remove a per-user pin. Returns ``False`` when none existed."""
raise NotImplementedError
@abstractmethod
async def list_pin_ids(self, user_id: str) -> set[str]:
"""Return the set of agent ids pinned by ``user_id``."""
raise NotImplementedError
@abstractmethod
async def delete_pins_for_agent(self, agent_id: str) -> None:
"""Drop every user's pin of ``agent_id`` (called on agent delete)."""
raise NotImplementedError
@@ -5,6 +5,7 @@ from __future__ import annotations
from datetime import UTC, datetime
from typing import Any
from deerflow.config.agent_name_desensitize import name_matches
from deerflow.persistence.agents.base import AgentStore
@@ -20,8 +21,12 @@ def _copy(data: dict[str, Any]) -> dict[str, Any]:
class MemoryAgentStore(AgentStore):
def __init__(self) -> None:
self._agents: dict[str, dict[str, Any]] = {}
self._favorites: set[tuple[str, str]] = set()
self._agents: dict[str, dict[str, Any]] = {}
self._favorites: set[tuple[str, str]] = set()
self._pins: set[tuple[str, str]] = set()
# Listing cache mirrors (see AgentExtrasRow / AgentSkillRow):
self._skills: dict[str, list[str]] = {}
self._extras: dict[str, dict[str, Any]] = {}
@staticmethod
def _is_visible(agent: dict[str, Any], user_id: str) -> bool:
@@ -166,6 +171,96 @@ class MemoryAgentStore(AgentStore):
self._agents.pop(agent_id, None)
return True
# --- Paginated listing ---
def _matches_scope(self, agent: dict[str, Any], *, user_id: str, scope: str | None, admin: bool) -> bool:
if admin:
return True
if scope == "square":
return (agent.get("user_id") is None or bool(agent.get("published"))) and not bool(agent.get("is_private"))
if scope == "mine":
owned = agent.get("user_id") is not None and agent.get("user_id") == user_id
favorited = (user_id, agent["id"]) in self._favorites
return owned or favorited
if scope == "private":
return bool(agent.get("is_private"))
return self._is_visible(agent, user_id)
async def list_paginated(
self,
*,
user_id: str,
scope: str | None,
search: str | None,
tag_ids: list[str] | None,
page: int,
page_size: int,
admin: bool = False,
sort: str = "time_desc",
) -> tuple[list[dict[str, Any]], int]:
# NOTE: the in-memory store has no tag-assignment data, so ``tag_ids``
# is ignored here. Tag-filter behavior is covered by the SQL repo tests.
matched = [
a
for a in self._agents.values()
if self._matches_scope(a, user_id=user_id, scope=scope, admin=admin)
and (not search or not search.strip() or name_matches(a.get("name") or "", search))
]
if sort == "time_asc":
matched.sort(key=lambda a: (a.get("created_at") or "", a.get("id") or ""))
elif sort == "updated_desc":
matched.sort(
key=lambda a: (a.get("updated_at") or "", a.get("id") or ""),
reverse=True,
)
elif sort == "updated_asc":
matched.sort(key=lambda a: (a.get("updated_at") or "", a.get("id") or ""))
elif sort == "name_asc":
matched.sort(key=lambda a: ((a.get("name") or "").lower(), a.get("id") or ""))
elif sort == "name_desc":
matched.sort(
key=lambda a: ((a.get("name") or "").lower(), a.get("id") or ""),
reverse=True,
)
else:
matched.sort(
key=lambda a: (a.get("created_at") or "", a.get("id") or ""),
reverse=True,
)
pin_ids = {agent_id for uid, agent_id in self._pins if uid == user_id}
matched.sort(key=lambda a: (0 if a.get("id") in pin_ids else 1,))
total = len(matched)
start = max(page - 1, 0) * page_size
page_rows = matched[start : start + page_size]
return [_copy(a) for a in page_rows], total
# --- Listing cache (skills / tool_groups / model) ---
async def replace_skills(self, agent_id: str, skills: list[str] | None) -> None:
self._skills[agent_id] = [s for s in (skills or []) if isinstance(s, str)]
async def set_extras(self, agent_id: str, *, tool_groups: list[str] | None, model: str | None) -> None:
self._extras[agent_id] = {"tool_groups": tool_groups, "model": model}
async def get_extras_for(self, agent_ids: list[str]) -> dict[str, dict[str, Any]]:
out: dict[str, dict[str, Any]] = {}
for agent_id in agent_ids:
has_extras = agent_id in self._extras
has_skills = agent_id in self._skills
if not has_extras and not has_skills:
continue
extras = self._extras.get(agent_id, {})
out[agent_id] = {
"skills": list(self._skills.get(agent_id, [])),
"tool_groups": extras.get("tool_groups"),
"model": extras.get("model"),
}
return out
async def delete_extras_for_agent(self, agent_id: str) -> None:
self._skills.pop(agent_id, None)
self._extras.pop(agent_id, None)
# --- Favorites ("收藏") ---
async def add_favorite(self, user_id: str, agent_id: str) -> None:
@@ -180,5 +275,23 @@ class MemoryAgentStore(AgentStore):
async def list_favorite_ids(self, user_id: str) -> set[str]:
return {agent_id for (uid, agent_id) in self._favorites if uid == user_id}
async def delete_favorites_for_agent(self, agent_id: str) -> None:
self._favorites = {(uid, aid) for (uid, aid) in self._favorites if aid != agent_id}
async def delete_favorites_for_agent(self, agent_id: str) -> None:
self._favorites = {(uid, aid) for (uid, aid) in self._favorites if aid != agent_id}
# --- Pins ("置顶") ---
async def add_pin(self, user_id: str, agent_id: str) -> None:
self._pins.add((user_id, agent_id))
async def remove_pin(self, user_id: str, agent_id: str) -> bool:
key = (user_id, agent_id)
if key not in self._pins:
return False
self._pins.remove(key)
return True
async def list_pin_ids(self, user_id: str) -> set[str]:
return {agent_id for uid, agent_id in self._pins if uid == user_id}
async def delete_pins_for_agent(self, agent_id: str) -> None:
self._pins = {(uid, aid) for (uid, aid) in self._pins if aid != agent_id}
@@ -4,11 +4,11 @@ from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy import BigInteger, Boolean, String, UniqueConstraint
from sqlalchemy import BigInteger, Boolean, Integer, String, UniqueConstraint
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
from deerflow.persistence.types import BeijingDateTime
from deerflow.persistence.types import BeijingDateTime, PortableJSON
class AgentRow(Base):
@@ -37,7 +37,7 @@ class AgentRow(Base):
updated_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC))
class AgentFavoriteRow(Base):
class AgentFavoriteRow(Base):
"""A user's bookmark of a (typically public-square) agent.
Lets a user collect agents from the 广场 (square) into their "我的" tab
@@ -51,4 +51,60 @@ class AgentFavoriteRow(Base):
id: Mapped[str] = mapped_column(String(64), primary_key=True)
user_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
agent_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC))
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC))
class AgentPinRow(Base):
"""A per-user pin that keeps an agent near the top of that user's lists."""
__tablename__ = "agent_pins"
__table_args__ = (UniqueConstraint("user_id", "agent_id", name="uq_agent_pins_user_agent"),)
id: Mapped[str] = mapped_column(String(64), primary_key=True)
user_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
agent_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC))
class AgentSkillRow(Base):
"""Denormalized cache of a custom agent's selected skills.
``config.yaml`` under ``.deer-flow/agents/{id}/`` stays the source of truth
for the agent runtime (the lead-agent prompt reads it directly). This table
is a read cache so the gallery list endpoint can render skill badges with a
single batched ``IN`` query instead of stat+YAML-parsing every agent's
``config.yaml`` on each request. Written through on create/update and
backfilled from disk by ``_sync_legacy_agents``.
One row per (agent, skill); ``position`` preserves the configured order.
"""
__tablename__ = "agent_skills"
__table_args__ = (UniqueConstraint("agent_id", "skill_name", name="uq_agent_skills_agent_skill"),)
id: Mapped[str] = mapped_column(String(64), primary_key=True)
agent_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
skill_name: Mapped[str] = mapped_column(String(191), nullable=False, index=True)
position: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
class AgentExtrasRow(Base):
"""Denormalized cache of a custom agent's ``tool_groups`` / ``model``.
Companion to :class:`AgentSkillRow` — same rationale (avoid per-agent
``config.yaml`` reads in the list path). One row per agent. The presence of
a row also marks "this agent has been mirrored": the list path treats a
missing row as "unknown / not yet backfilled" (``skills=None``) so it can
be told apart from an explicitly empty skill list.
``skills_synced_at`` records when the cache was last written from disk so
the backfill can skip agents whose ``config.yaml`` has not changed.
"""
__tablename__ = "agent_extras"
agent_id: Mapped[str] = mapped_column(String(64), primary_key=True)
# list[str] | None — the agent's tool_groups whitelist from config.yaml.
tool_groups: Mapped[list | None] = mapped_column(PortableJSON(), nullable=True, default=None)
model: Mapped[str | None] = mapped_column(String(191), nullable=True, default=None)
skills_synced_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC))
@@ -6,12 +6,20 @@ from datetime import UTC, datetime
from typing import Any
from uuid import uuid4
from sqlalchemy import and_, delete, func, or_, select
from sqlalchemy import and_, case, delete, func, or_, select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from deerflow.config.agent_name_desensitize import source_terms_for_needle
from deerflow.persistence.agents.base import AgentStore
from deerflow.persistence.agents.model import AgentFavoriteRow, AgentRow
from deerflow.persistence.agents.model import (
AgentExtrasRow,
AgentFavoriteRow,
AgentPinRow,
AgentRow,
AgentSkillRow,
)
from deerflow.persistence.tags.model import TagAssignmentRow
def _row_to_dict(row: AgentRow) -> dict[str, Any]:
@@ -233,6 +241,167 @@ class AgentRepository(AgentStore):
await session.commit()
return True
# --- Paginated listing ---
def _scope_filter(self, *, user_id: str, scope: str | None, admin: bool):
"""Build the visibility predicate for a paginated listing."""
if admin:
return None # no visibility filter — every agent
if scope == "square":
return and_(
or_(AgentRow.user_id.is_(None), AgentRow.published.is_(True)),
AgentRow.is_private.is_(False),
)
if scope == "mine":
# owned OR favorited. A correlated IN-subquery (not a JOIN) keeps a
# single row per agent even when it is both owned and favorited.
fav_subq = select(AgentFavoriteRow.agent_id).where(AgentFavoriteRow.user_id == user_id)
return or_(AgentRow.user_id == user_id, AgentRow.id.in_(fav_subq))
if scope == "private":
return AgentRow.is_private.is_(True)
return self._listing_filter(user_id)
@staticmethod
def _search_clause(search: str):
"""Match the raw name OR (via the desensitize mirror) its display form."""
needle = search.strip().lower()
clauses = [func.lower(AgentRow.name).like(f"%{needle}%")]
for term in source_terms_for_needle(needle):
clauses.append(AgentRow.name.like(f"%{term}%"))
return or_(*clauses)
@staticmethod
def _tag_clause(tag_ids: list[str]):
subq = select(TagAssignmentRow.target_id).where(
TagAssignmentRow.target_type == "agent",
TagAssignmentRow.tag_id.in_(tag_ids),
)
return AgentRow.id.in_(subq)
async def list_paginated(
self,
*,
user_id: str,
scope: str | None,
search: str | None,
tag_ids: list[str] | None,
page: int,
page_size: int,
admin: bool = False,
sort: str = "time_desc",
) -> tuple[list[dict[str, Any]], int]:
filters = []
scope_filter = self._scope_filter(user_id=user_id, scope=scope, admin=admin)
if scope_filter is not None:
filters.append(scope_filter)
if search and search.strip():
filters.append(self._search_clause(search))
if tag_ids:
filters.append(self._tag_clause(tag_ids))
count_stmt = select(func.count()).select_from(AgentRow)
pin_subq = select(AgentPinRow.agent_id).where(AgentPinRow.user_id == user_id)
pin_rank = case((AgentRow.id.in_(pin_subq), 0), else_=1)
if sort == "time_asc":
order_by = (AgentRow.created_at.asc(), AgentRow.id.asc())
elif sort == "updated_desc":
order_by = (AgentRow.updated_at.desc(), AgentRow.id.asc())
elif sort == "updated_asc":
order_by = (AgentRow.updated_at.asc(), AgentRow.id.asc())
elif sort == "name_asc":
order_by = (func.lower(AgentRow.name).asc(), AgentRow.id.asc())
elif sort == "name_desc":
order_by = (func.lower(AgentRow.name).desc(), AgentRow.id.asc())
else:
order_by = (AgentRow.created_at.desc(), AgentRow.id.asc())
page_stmt = (
select(AgentRow)
.order_by(pin_rank.asc(), *order_by)
.limit(page_size)
.offset(max(page - 1, 0) * page_size)
)
if filters:
where = and_(*filters)
count_stmt = count_stmt.where(where)
page_stmt = page_stmt.where(where)
async with self._sf() as session:
total = await session.scalar(count_stmt) or 0
result = await session.execute(page_stmt)
rows = [_row_to_dict(row) for row in result.scalars()]
return rows, total
# --- Listing cache (skills / tool_groups / model) ---
async def replace_skills(self, agent_id: str, skills: list[str] | None) -> None:
async with self._sf() as session:
await session.execute(delete(AgentSkillRow).where(AgentSkillRow.agent_id == agent_id))
for position, name in enumerate(skills or []):
if not isinstance(name, str):
continue
session.add(
AgentSkillRow(
id=uuid4().hex,
agent_id=agent_id,
skill_name=name,
position=position,
)
)
await session.commit()
async def set_extras(self, agent_id: str, *, tool_groups: list[str] | None, model: str | None) -> None:
async with self._sf() as session:
row = await session.get(AgentExtrasRow, agent_id)
now = datetime.now(UTC)
if row is None:
session.add(
AgentExtrasRow(
agent_id=agent_id,
tool_groups=tool_groups,
model=model,
skills_synced_at=now,
)
)
else:
row.tool_groups = tool_groups
row.model = model
row.skills_synced_at = now
await session.commit()
async def get_extras_for(self, agent_ids: list[str]) -> dict[str, dict[str, Any]]:
if not agent_ids:
return {}
out: dict[str, dict[str, Any]] = {}
async with self._sf() as session:
extras = await session.execute(
select(AgentExtrasRow).where(AgentExtrasRow.agent_id.in_(agent_ids))
)
for row in extras.scalars():
out[row.agent_id] = {
"skills": [], # default; filled below if skill rows exist
"tool_groups": row.tool_groups,
"model": row.model,
}
skill_rows = await session.execute(
select(AgentSkillRow.agent_id, AgentSkillRow.skill_name)
.where(AgentSkillRow.agent_id.in_(agent_ids))
.order_by(AgentSkillRow.agent_id.asc(), AgentSkillRow.position.asc())
)
for agent_id, skill_name in skill_rows.all():
entry = out.setdefault(
agent_id, {"skills": [], "tool_groups": None, "model": None}
)
entry["skills"].append(skill_name)
return out
async def delete_extras_for_agent(self, agent_id: str) -> None:
async with self._sf() as session:
await session.execute(delete(AgentSkillRow).where(AgentSkillRow.agent_id == agent_id))
await session.execute(delete(AgentExtrasRow).where(AgentExtrasRow.agent_id == agent_id))
await session.commit()
# --- Favorites ("收藏") ---
async def add_favorite(self, user_id: str, agent_id: str) -> None:
@@ -272,7 +441,49 @@ class AgentRepository(AgentStore):
)
return {row[0] for row in result.all()}
async def delete_favorites_for_agent(self, agent_id: str) -> None:
async with self._sf() as session:
await session.execute(delete(AgentFavoriteRow).where(AgentFavoriteRow.agent_id == agent_id))
await session.commit()
async def delete_favorites_for_agent(self, agent_id: str) -> None:
async with self._sf() as session:
await session.execute(delete(AgentFavoriteRow).where(AgentFavoriteRow.agent_id == agent_id))
await session.commit()
# --- Pins ("置顶") ---
async def add_pin(self, user_id: str, agent_id: str) -> None:
async with self._sf() as session:
existing = (
await session.execute(
select(AgentPinRow.id)
.where(AgentPinRow.user_id == user_id, AgentPinRow.agent_id == agent_id)
.limit(1)
)
).scalar_one_or_none()
if existing is not None:
return
session.add(AgentPinRow(id=uuid4().hex, user_id=user_id, agent_id=agent_id))
try:
await session.commit()
except IntegrityError:
await session.rollback()
async def remove_pin(self, user_id: str, agent_id: str) -> bool:
async with self._sf() as session:
result = await session.execute(
delete(AgentPinRow).where(
AgentPinRow.user_id == user_id,
AgentPinRow.agent_id == agent_id,
)
)
await session.commit()
return (result.rowcount or 0) > 0
async def list_pin_ids(self, user_id: str) -> set[str]:
async with self._sf() as session:
result = await session.execute(
select(AgentPinRow.agent_id).where(AgentPinRow.user_id == user_id)
)
return {row[0] for row in result.all()}
async def delete_pins_for_agent(self, agent_id: str) -> None:
async with self._sf() as session:
await session.execute(delete(AgentPinRow).where(AgentPinRow.agent_id == agent_id))
await session.commit()
@@ -34,6 +34,9 @@ class LlmMetricsQuery:
"""Filter set for the admin list / export endpoints."""
user_id: str | None = None
# Multiple user ids — set when a fuzzy username search resolves to several
# accounts. Takes precedence over ``user_id`` when non-empty.
user_ids: list[str] | None = None
model_name: str | None = None
status: str | None = None
since: datetime | None = None
@@ -58,7 +58,9 @@ class SqlLlmMetricsStore(LlmMetricsStore):
logger.warning("Failed to persist llm_call_metrics row %s", metric.id, exc_info=True)
def _apply_filters(self, stmt, query: LlmMetricsQuery):
if query.user_id:
if query.user_ids:
stmt = stmt.where(LlmCallMetricRow.user_id.in_(query.user_ids))
elif query.user_id:
stmt = stmt.where(LlmCallMetricRow.user_id == query.user_id)
if query.model_name:
stmt = stmt.where(LlmCallMetricRow.model_name == query.model_name)
@@ -92,6 +94,7 @@ class SqlLlmMetricsStore(LlmMetricsStore):
while True:
page_query = LlmMetricsQuery(
user_id=query.user_id,
user_ids=query.user_ids,
model_name=query.model_name,
status=query.status,
since=query.since,
@@ -0,0 +1,45 @@
"""Add ``agent_id`` to ``user_custom_prompts`` for per-agent prompt scoping.
NULL = global (shown for every agent); otherwise the prompt only surfaces when
that agent is selected.
Revision ID: 20260603_02
Revises: 20260603_01
Create Date: 2026-06-03
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
from sqlalchemy import inspect
revision = "20260603_02"
down_revision = "20260603_01"
branch_labels = None
depends_on = None
def _has_table(table_name: str) -> bool:
inspector = inspect(op.get_bind())
return table_name in inspector.get_table_names()
def _has_column(table_name: str, column_name: str) -> bool:
inspector = inspect(op.get_bind())
return any(col["name"] == column_name for col in inspector.get_columns(table_name))
def upgrade() -> None:
if not _has_table("user_custom_prompts"):
return
if not _has_column("user_custom_prompts", "agent_id"):
op.add_column("user_custom_prompts", sa.Column("agent_id", sa.String(64), nullable=True))
op.create_index("ix_user_custom_prompts_agent_id", "user_custom_prompts", ["agent_id"])
def downgrade() -> None:
if _has_table("user_custom_prompts") and _has_column("user_custom_prompts", "agent_id"):
op.drop_index("ix_user_custom_prompts_agent_id", table_name="user_custom_prompts")
op.drop_column("user_custom_prompts", "agent_id")
@@ -0,0 +1,42 @@
"""Add ``user_preferences`` for per-user UI toggles (account-bound).
Holds a single JSON blob per user — e.g. ``hide_admin_prompt_prefix`` so a user
can choose to use only their own prompts and hide the admin-configured prefix.
Revision ID: 20260603_03
Revises: 20260603_02
Create Date: 2026-06-03
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
from sqlalchemy import inspect
revision = "20260603_03"
down_revision = "20260603_02"
branch_labels = None
depends_on = None
def _has_table(table_name: str) -> bool:
inspector = inspect(op.get_bind())
return table_name in inspector.get_table_names()
def upgrade() -> None:
if not _has_table("user_preferences"):
op.create_table(
"user_preferences",
sa.Column("user_id", sa.String(64), primary_key=True),
sa.Column("preferences", sa.JSON(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
)
def downgrade() -> None:
if _has_table("user_preferences"):
op.drop_table("user_preferences")
@@ -0,0 +1,56 @@
"""Per-user pinned agents.
Revision ID: 20260604_01
Revises: 20260603_03
Create Date: 2026-06-04
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
from sqlalchemy import inspect
revision = "20260604_01"
down_revision = "20260603_03"
branch_labels = None
depends_on = None
def _has_table(table_name: str) -> bool:
inspector = inspect(op.get_bind())
return table_name in inspector.get_table_names()
def _has_index(table_name: str, index_name: str) -> bool:
if not _has_table(table_name):
return False
inspector = inspect(op.get_bind())
return any(idx["name"] == index_name for idx in inspector.get_indexes(table_name))
def upgrade() -> None:
if not _has_table("agent_pins"):
op.create_table(
"agent_pins",
sa.Column("id", sa.String(64), nullable=False),
sa.Column("user_id", sa.String(64), nullable=False),
sa.Column("agent_id", sa.String(64), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("user_id", "agent_id", name="uq_agent_pins_user_agent"),
)
if not _has_index("agent_pins", "ix_agent_pins_user_id"):
op.create_index("ix_agent_pins_user_id", "agent_pins", ["user_id"])
if not _has_index("agent_pins", "ix_agent_pins_agent_id"):
op.create_index("ix_agent_pins_agent_id", "agent_pins", ["agent_id"])
def downgrade() -> None:
if _has_index("agent_pins", "ix_agent_pins_agent_id"):
op.drop_index("ix_agent_pins_agent_id", table_name="agent_pins")
if _has_index("agent_pins", "ix_agent_pins_user_id"):
op.drop_index("ix_agent_pins_user_id", table_name="agent_pins")
if _has_table("agent_pins"):
op.drop_table("agent_pins")
@@ -15,7 +15,12 @@ its storage implementation lives in ``deerflow.runtime.events.store.db`` and
there is no matching entity directory.
"""
from deerflow.persistence.agents.model import AgentFavoriteRow, AgentRow
from deerflow.persistence.agents.model import (
AgentExtrasRow,
AgentFavoriteRow,
AgentRow,
AgentSkillRow,
)
from deerflow.persistence.memory_config.model import UserMemoryConfigRow
from deerflow.persistence.feedback.model import FeedbackRow
from deerflow.persistence.llm_metrics.model import LlmCallMetricRow
@@ -44,12 +49,15 @@ from deerflow.persistence.embed_sessions.model import EmbedSessionRow
from deerflow.persistence.thread_shares.model import ThreadShareRow
from deerflow.persistence.tool_metrics.model import ToolCallMetricRow
from deerflow.persistence.user.model import UserRow
from deerflow.persistence.user_preferences.model import UserPreferencesRow
from deerflow.persistence.user_prompts.model import UserCustomPromptRow
__all__ = [
"FeedbackRow",
"AgentExtrasRow",
"AgentFavoriteRow",
"AgentRow",
"AgentSkillRow",
"LlmCallMetricRow",
"NotificationRow",
"RecommendedQuestionRow",
@@ -75,5 +83,6 @@ __all__ = [
"ToolCallMetricRow",
"UserCustomPromptRow",
"UserMemoryConfigRow",
"UserPreferencesRow",
"UserRow",
]
@@ -205,6 +205,21 @@ class RunRepository(RunStore):
await session.execute(update(RunRow).where(RunRow.run_id == run_id).values(**values))
await session.commit()
async def count_by_user(self) -> dict[str, int]:
"""Return ``{user_id: run_count}`` via a single GROUP BY query.
Each run is one user question, so this is the source for the admin
user center's 问答次数 column. Runs without a ``user_id`` are skipped.
"""
stmt = (
select(RunRow.user_id, func.count().label("cnt"))
.where(RunRow.user_id.is_not(None))
.group_by(RunRow.user_id)
)
async with self._sf() as session:
rows = (await session.execute(stmt)).all()
return {r.user_id: r.cnt for r in rows}
async def aggregate_tokens_by_thread(self, thread_id: str) -> dict[str, Any]:
"""Aggregate token usage via a single SQL GROUP BY query."""
_completed = RunRow.status.in_(("success", "error"))
@@ -0,0 +1,18 @@
"""Per-user UI preferences persistence."""
from __future__ import annotations
from deerflow.persistence.user_preferences.model import UserPreferencesRow
from deerflow.persistence.user_preferences.sql import UserPreferencesRepository
__all__ = [
"UserPreferencesRepository",
"UserPreferencesRow",
"make_user_preferences_store",
]
def make_user_preferences_store(session_factory):
if session_factory is None:
return None
return UserPreferencesRepository(session_factory)
@@ -0,0 +1,26 @@
"""ORM model for per-user UI preferences.
A single key-value (JSON) row per user, holding small UI-level toggles that
should follow the account across devices — e.g. whether to hide the
admin-configured prompt prefix in the chat input. Kept generic (one JSON blob)
so new toggles do not each require an ALTER TABLE.
"""
from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy import String
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
from deerflow.persistence.types import BeijingDateTime, PortableJSON
class UserPreferencesRow(Base):
__tablename__ = "user_preferences"
user_id: Mapped[str] = mapped_column(String(64), primary_key=True)
preferences: Mapped[dict] = mapped_column(PortableJSON(), nullable=False, default=dict)
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC))
updated_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC))
@@ -0,0 +1,41 @@
"""SQLAlchemy-backed per-user preferences repository."""
from __future__ import annotations
from datetime import UTC, datetime
from typing import Any
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from deerflow.persistence.user_preferences.model import UserPreferencesRow
class UserPreferencesRepository:
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
self._sf = session_factory
async def get_for_user(self, user_id: str) -> dict[str, Any]:
"""Return the user's stored preferences (empty dict when none)."""
async with self._sf() as session:
row = await session.get(UserPreferencesRow, user_id)
if row is None or not isinstance(row.preferences, dict):
return {}
return dict(row.preferences)
async def merge_for_user(self, user_id: str, patch: dict[str, Any]) -> dict[str, Any]:
"""Shallow-merge ``patch`` into the user's preferences and return the result."""
now = datetime.now(UTC)
async with self._sf() as session:
row = await session.get(UserPreferencesRow, user_id)
if row is None:
merged = dict(patch)
row = UserPreferencesRow(user_id=user_id, preferences=merged, created_at=now, updated_at=now)
session.add(row)
else:
merged = dict(row.preferences or {})
merged.update(patch)
# Reassign so SQLAlchemy detects the change on the JSON column.
row.preferences = merged
row.updated_at = now
await session.commit()
return dict(merged)
@@ -19,5 +19,8 @@ class UserCustomPromptRow(Base):
title: Mapped[str] = mapped_column(String(255), nullable=False, default="")
content: Mapped[str] = mapped_column(Text, nullable=False)
sort_order: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
# Optional association to a single agent. NULL = global (shown everywhere);
# otherwise the prompt only surfaces when that agent is selected.
agent_id: Mapped[str | None] = mapped_column(String(64), nullable=True, index=True)
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC))
updated_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC))
@@ -42,6 +42,7 @@ class UserCustomPromptRepository:
title=data.get("title") or "",
content=data["content"],
sort_order=data.get("sort_order", 0),
agent_id=(data.get("agent_id") or None),
created_at=now,
updated_at=now,
)
@@ -59,6 +60,10 @@ class UserCustomPromptRepository:
for key in ("title", "content", "sort_order"):
if key in data:
setattr(row, key, data[key])
# ``agent_id`` is explicitly nullable: an empty/None value clears the
# association (back to global), so accept None as a real update.
if "agent_id" in data:
row.agent_id = data["agent_id"] or None
row.updated_at = datetime.now(UTC)
await session.commit()
await session.refresh(row)
@@ -93,3 +93,12 @@ class RunStore(abc.ABC):
by_caller ({lead_agent, subagent, middleware}).
"""
pass
async def count_by_user(self) -> dict[str, int]:
"""Return ``{user_id: run_count}`` across all runs.
One run corresponds to one user submission, so this backs the admin
user center's 问答次数 (Q&A count) column — each question counts once.
Non-overriding stores return an empty mapping (treated as 0 per user).
"""
return {}
@@ -38,6 +38,11 @@ _DANGEROUS_URL_RE = re.compile(r"^\s*(javascript|vbscript|data:text/html)", re.I
# Attributes carrying a URL we must scheme-check.
_URL_ATTRS = frozenset({"href", "src", "action", "formaction", "xlink:href", "background", "poster"})
_LOCAL_DOC_LINK_RE = re.compile(
r"^(?![a-z][a-z0-9+.-]*:)(?!//)(?P<path>[^#?]*\.(?:html?|md|markdown))(?P<query>\?[^#]*)?(?P<fragment>#.+)?$",
re.IGNORECASE,
)
def parse_frontmatter(text: str) -> tuple[dict[str, Any], str]:
"""Split a ``---`` YAML frontmatter block from the body of a SKILL.md.
@@ -225,6 +230,38 @@ def sanitize_html(html: str) -> tuple[str, list[str]]:
return str(soup), removed
def constrain_same_page_anchor_links(html: str) -> tuple[str, list[str]]:
"""Keep generated HTML document jumps inside the current HTML file."""
if not html or not isinstance(html, str):
return "", []
try:
from bs4 import BeautifulSoup
except Exception: # pragma: no cover - bs4 is a declared dependency
logger.warning("BeautifulSoup unavailable; returning HTML links unconstrained")
return html, []
changed: list[str] = []
soup = BeautifulSoup(html, "html.parser")
for tag in soup.find_all("a"):
href = tag.get("href")
if not isinstance(href, str):
continue
value = href.strip()
if not value or value.startswith("#"):
continue
match = _LOCAL_DOC_LINK_RE.match(value)
if not match:
continue
fragment = match.group("fragment")
if fragment:
tag["href"] = fragment
changed.append(f"{value}->{fragment}")
else:
del tag.attrs["href"]
changed.append(f"{value}->removed")
return str(soup), changed
def validate_html(html: str) -> list[str]:
"""Return a list of structural problems with ``html`` (empty == valid).
@@ -299,6 +336,7 @@ _GENERATION_TEMPLATE = """你是一个内网页面生成助手。请根据输入
6. 不要包含 <script>、外链脚本、iframe、表单提交、on* 事件属性。
7. 参考页面只用于风格和布局,页面内容必须来自“本次数据”,不要复用参考页旧数据。
8. 页面使用中文,排版适合在浏览器中直接预览。
9. HTML 内的目录/锚点跳转必须只跳转到当前这个 HTML 文件内部:只使用 href="#section-id",不要使用 href="index.html#section-id"、".md#..."、其它文件路径或任何跨文件锚点;目标区块必须有匹配的 id。
"""
_REPAIR_TEMPLATE = """你是一个 HTML 修复助手。下面的 HTML 存在问题,请修复后重新输出完整页面。
@@ -317,6 +355,7 @@ _REPAIR_TEMPLATE = """你是一个 HTML 修复助手。下面的 HTML 存在问
2. fixed_html 必须是完整 HTML 文档,包含 <!doctype html>、html、head、body。
3. 移除所有 <script>、iframe、外链脚本、on* 事件属性、表单提交。
4. 保留原有内容与数据,只修复结构与安全问题。
5. HTML 内的目录/锚点跳转必须只跳转到当前这个 HTML 文件内部:只使用 href="#section-id",不要链接到 .html/.md 文件作为锚点。
"""
_STYLE_EXTRACT_TEMPLATE = """你是一个页面风格分析助手。请分析下面的 HTML 页面,提取可复用的风格与布局特征。
@@ -53,8 +53,18 @@ _MARKDOWN_MIN_CHARS = 100
_MARKDOWN_INSTRUCTION = (
"\n\n【输出要求】请将本次任务的最终结果生成为一个 Markdown(.md) 格式的文件,"
"保存到 /mnt/user-data/outputs 目录,并使用 present_files 展示该文件。"
"Markdown 内的目录/锚点跳转必须只跳转到当前这个 Markdown 文件内部:"
"只使用 [标题](#section-id),不要使用 [标题](xxx.md#section-id)、"
"[标题](xxx.html#section-id) 或任何跨文件锚点。"
)
_LOCAL_DOC_MARKDOWN_LINK_RE = re.compile(
r"^(?![a-z][a-z0-9+.-]*:)(?!//)(?P<path>[^#?]*\.(?:html?|md|markdown))(?P<query>\?[^#]*)?(?P<fragment>#.+)?$",
re.IGNORECASE,
)
_MARKDOWN_INLINE_LINK_RE = re.compile(r"(?<!!)\[([^\]]+)\]\(([^)\s]+)(?:\s+\"[^\"]*\")?\)")
_MARKDOWN_REFERENCE_LINK_RE = re.compile(r"^(\s*\[[^\]]+\]:\s*)(\S+)(.*)$", re.MULTILINE)
class MarkdownRequirementError(Exception):
"""Raised when a ``require_markdown`` task did not produce a valid .md file.
@@ -219,6 +229,23 @@ def _task_requires_markdown(task: dict[str, Any]) -> bool:
return bool(isinstance(ctx, dict) and ctx.get("require_markdown"))
def _task_requires_html(task: dict[str, Any]) -> bool:
"""Whether a task must produce an HTML page (the ``必须生成 HTML`` switch).
The flag lives inside ``execution_context.require_html``. For backward
compatibility, tasks created under the old two-tab UI (which stored
``task_kind == "html_page"``) are still treated as HTML tasks, so no data
migration is needed. ``require_html`` and ``require_markdown`` are
independent — both may be on at once.
"""
ctx = task.get("execution_context")
if not isinstance(ctx, dict):
return False
if ctx.get("require_html"):
return True
return ctx.get("task_kind") == "html_page"
def _has_valid_markdown_file(result_files: list[dict[str, Any]] | None) -> bool:
"""True when the run produced a ``.md`` file whose text exceeds the minimum.
@@ -245,6 +272,43 @@ def _has_valid_markdown_file(result_files: list[dict[str, Any]] | None) -> bool:
return False
def _normalize_same_page_markdown_target(target: str) -> str | None:
value = (target or "").strip()
if not value or value.startswith("#"):
return value
match = _LOCAL_DOC_MARKDOWN_LINK_RE.match(value)
if not match:
return value
fragment = match.group("fragment")
return fragment or None
def _constrain_markdown_same_page_anchor_links(content: str) -> str:
"""Keep generated Markdown document jumps inside the current Markdown file."""
if not content or not isinstance(content, str):
return content or ""
def replace_inline(match: re.Match[str]) -> str:
label = match.group(1)
target = match.group(2)
normalized = _normalize_same_page_markdown_target(target)
if normalized is None:
return label
return f"[{label}]({normalized})"
def replace_reference(match: re.Match[str]) -> str:
prefix = match.group(1)
target = match.group(2)
suffix = match.group(3)
normalized = _normalize_same_page_markdown_target(target)
if normalized is None:
return f"{prefix}#"
return f"{prefix}{normalized}{suffix}"
content = _MARKDOWN_INLINE_LINK_RE.sub(replace_inline, content)
return _MARKDOWN_REFERENCE_LINK_RE.sub(replace_reference, content)
def _build_result(
task: dict[str, Any],
*,
@@ -905,15 +969,18 @@ class ScheduledTaskService:
scheduled_for: datetime,
run_id: str,
) -> dict[str, Any]:
"""Dispatch a single execution attempt by task kind.
"""Dispatch a single execution attempt by the task's output requirements.
``html_page`` tasks render an HTML document via GLM5; everything else
runs the regular lead-agent attempt. The kind lives in
``execution_context.task_kind`` so no schema migration is needed.
A task carries two independent switches in ``execution_context``:
``require_html`` (必须生成 HTML) and ``require_markdown`` (必须生成
Markdown). When ``require_html`` is set the run goes through the HTML
render pipeline (which also enforces the Markdown requirement when both
switches are on); otherwise it runs the regular lead-agent attempt.
Both flags live in ``execution_context`` so no schema migration is
needed, and old ``task_kind == "html_page"`` tasks still resolve to the
HTML pipeline via :func:`_task_requires_html`.
"""
ctx = task.get("execution_context")
kind = ctx.get("task_kind") if isinstance(ctx, dict) else None
if kind == "html_page":
if _task_requires_html(task):
return await self._run_html_page_attempt(
task, runtime_thread_id=runtime_thread_id, scheduled_for=scheduled_for, run_id=run_id
)
@@ -956,17 +1023,21 @@ class ScheduledTaskService:
runtime_thread_id: str,
scheduled_for: datetime,
run_id: str,
) -> str:
) -> tuple[str, list[dict[str, Any]]]:
"""Run the lead agent to collect source data for the HTML page.
Reuses the regular agent-run primitives (with the task's search query
folded into the prompt) and returns the agent's final text, which feeds
the GLM5 render step as ``本次数据``. Returns an empty string if the
agent produced nothing.
folded into the prompt) and returns ``(text, files)`` — the agent's
final text (feeds the render step as ``本次数据``) plus any files it
produced. The files are only captured/needed when the task also has the
Markdown switch on (both 必须生成 HTML and 必须生成 Markdown), so the run
can validate and surface the ``.md`` file alongside the HTML page;
otherwise the list is empty.
"""
cfg = self._html_config(task)
page_prompt = str(cfg.get("page_prompt") or "").strip()
negative_prompt = str(cfg.get("negative_prompt") or "").strip()
also_markdown = _task_requires_markdown(task)
gather_instruction = (
"请运用你自身的检索/搜索等技能,围绕下述页面需求获取最新、真实的资料,"
"并整理成结构化的中文要点(可用列表/小标题),供后续生成 HTML 页面使用,本步骤不要生成 HTML:\n"
@@ -982,12 +1053,22 @@ class ScheduledTaskService:
# which sources/data the agent collects, otherwise it silently leaks in.
if negative_prompt:
parts.append(f"限制条件(最高优先级,必须严格遵守):{negative_prompt}")
gather_task = {**task, "prompt": "\n".join(parts)}
# Never apply the markdown gate to the gather pass.
gather_prompt = "\n".join(parts)
# When the Markdown switch is also on, this same gather pass must save a
# .md file. We do NOT append _MARKDOWN_INSTRUCTION here — `_start_agent_run`
# already appends it exactly once when `require_markdown` is present in the
# execution context (kept below). Appending it here too made the prompt
# carry the 【输出要求】 block twice when both HTML and Markdown were on.
gather_task = {**task, "prompt": gather_prompt}
gather_ctx = dict(task.get("execution_context") or {})
gather_ctx.pop("require_markdown", None)
# Strip the HTML-specific flags so the gather pass is a plain agent run.
gather_ctx.pop("require_html", None)
gather_ctx.pop("task_kind", None)
gather_ctx.pop("html_config", None)
# Keep ``require_markdown`` only when both switches are on, so the agent
# is told (via _start_agent_run) to produce the .md file.
if not also_markdown:
gather_ctx.pop("require_markdown", None)
# Use the task's selected model for the data-gathering agent too, so the
# whole HTML pipeline honors ``html_config.model_name`` instead of
# silently falling back to the default model for this step.
@@ -997,13 +1078,22 @@ class ScheduledTaskService:
gather_task["execution_context"] = gather_ctx
await self._ensure_thread_meta(task["user_id"], runtime_thread_id)
files: list[dict[str, Any]] = []
async with self._agent_run_lock:
output_snapshot = self._snapshot_result_files(runtime_thread_id, task["user_id"]) if also_markdown else None
record = await self._start_agent_run(gather_task, scheduled_for=scheduled_for, runtime_thread_id=runtime_thread_id)
await self.store.update_run(run_id, {"agent_run_id": record.run_id})
if record.task is not None:
await record.task
text = await self._extract_latest_ai_text(runtime_thread_id)
return text or ""
if also_markdown:
files = await self._extract_result_files(
runtime_thread_id,
task["user_id"],
run_id=run_id,
output_snapshot=output_snapshot,
)
return text or "", files
def _collect_html_skill_instructions(self) -> str:
"""Concatenate the bodies of enabled skills marked for HTML pages.
@@ -1072,10 +1162,17 @@ class ScheduledTaskService:
negative_prompt = str(cfg.get("negative_prompt") or "").strip()
reference_favorite_id = cfg.get("reference_favorite_id") or None
# 1) Gather source data (real, via the agent's search tools).
data_text = await self._gather_html_data(
# 1) Gather source data (real, via the agent's search tools). When the
# Markdown switch is also on, the gather pass additionally saves the .md
# file, returned here so we can validate + surface it next to the page.
also_markdown = _task_requires_markdown(task)
data_text, gathered_files = await self._gather_html_data(
task, runtime_thread_id=runtime_thread_id, scheduled_for=scheduled_for, run_id=run_id
)
if also_markdown and not _has_valid_markdown_file(gathered_files):
raise MarkdownRequirementError(
f"未满足输出要求:需生成 Markdown(.md) 文件,且文件正文不少于 {_MARKDOWN_MIN_CHARS} 字"
)
# 2) Load reference style/layout from a favorite if one was selected.
reference_style = None
@@ -1119,6 +1216,7 @@ class ScheduledTaskService:
# 4) Sanitize + validate, with one repair pass on failure.
html, _removed = html_page_utils.sanitize_html(html)
html, _link_changes = html_page_utils.constrain_same_page_anchor_links(html)
errors = html_page_utils.validate_html(html)
if errors:
logger.info("Generated HTML failed validation (%s); attempting repair", errors)
@@ -1131,6 +1229,7 @@ class ScheduledTaskService:
fixed = repaired.get("fixed_html")
if isinstance(fixed, str) and "<" in fixed:
fixed_clean, _ = html_page_utils.sanitize_html(fixed)
fixed_clean, _ = html_page_utils.constrain_same_page_anchor_links(fixed_clean)
if not html_page_utils.validate_html(fixed_clean):
html = fixed_clean
except Exception:
@@ -1158,6 +1257,13 @@ class ScheduledTaskService:
archive_path = self._archive_html_index(run_id, html)
files: list[dict[str, Any]] = []
# When both switches are on, surface the gathered Markdown file(s)
# alongside the generated page so the run delivers both artifacts.
if also_markdown:
for item in gathered_files:
name = str(item.get("name") or "") if isinstance(item, dict) else ""
if name.lower().endswith((".md", ".markdown")):
files.append(item)
if archive_path is not None:
files.append(
{
@@ -1483,6 +1589,14 @@ class ScheduledTaskService:
index += 1
shutil.copy2(path, candidate)
if candidate.suffix.lower() in {".md", ".markdown"}:
try:
original = candidate.read_text(encoding="utf-8", errors="ignore")
constrained = _constrain_markdown_same_page_anchor_links(original)
if constrained != original:
candidate.write_text(constrained, encoding="utf-8", newline="")
except Exception:
logger.debug("Failed to constrain scheduled markdown links in %s", candidate, exc_info=True)
return candidate
def _build_file_result(self, *, virtual_path: str, source_path: Path, archive_path: Path) -> dict[str, Any]:
@@ -36,6 +36,7 @@ _DEFAULT_ENTRY: dict = {
"created_at": None,
"state": STATE_ACTIVE,
"pinned": False,
"featured": False,
"archived_at": None,
"source": SOURCE_UPLOAD,
}
@@ -141,6 +142,11 @@ def set_pinned(name: str, pinned: bool) -> None:
_update_entry(name, {"pinned": pinned})
def set_featured(name: str, featured: bool) -> None:
"""置顶/取消置顶。与 pinned(防归档)正交,仅影响列表排序展示。"""
_update_entry(name, {"featured": featured})
def forget(name: str) -> None:
try:
with _usage_lock:
@@ -3,9 +3,14 @@ from __future__ import annotations
import pytest
import yaml
from app.gateway.routers.agents import _load_config_extras, _write_agent_config
from app.gateway.routers.agents import (
_backfill_listing_cache,
_load_config_extras,
_write_agent_config,
)
from app.gateway.services import build_run_config
from deerflow.config import paths as paths_module
from deerflow.config.agents_config import AgentConfig
from deerflow.persistence.agents import MemoryAgentStore
@@ -198,3 +203,46 @@ def test_write_agent_config_omits_skills_when_none(isolated_paths) -> None:
assert "skills" not in raw
assert "model" not in raw
assert "tool_groups" not in raw
@pytest.mark.asyncio
async def test_backfill_listing_cache_mirrors_config_values() -> None:
store = MemoryAgentStore()
cfg = AgentConfig(id="a1", name="Agent", skills=["s1", "s2"], tool_groups=["g1"], model="m1")
await _backfill_listing_cache(store, [("a1", cfg)])
extras = await store.get_extras_for(["a1"])
assert extras["a1"]["skills"] == ["s1", "s2"]
assert extras["a1"]["tool_groups"] == ["g1"]
assert extras["a1"]["model"] == "m1"
@pytest.mark.asyncio
async def test_backfill_listing_cache_is_idempotent_and_skips_unchanged(monkeypatch) -> None:
store = MemoryAgentStore()
cfg = AgentConfig(id="a1", name="Agent", skills=["s1"], tool_groups=None, model=None)
await _backfill_listing_cache(store, [("a1", cfg)])
# On a second run with identical config, no re-write should occur.
calls = {"replace": 0, "set": 0}
orig_replace = store.replace_skills
orig_set = store.set_extras
async def _replace(agent_id, skills):
calls["replace"] += 1
await orig_replace(agent_id, skills)
async def _set(agent_id, *, tool_groups, model):
calls["set"] += 1
await orig_set(agent_id, tool_groups=tool_groups, model=model)
monkeypatch.setattr(store, "replace_skills", _replace)
monkeypatch.setattr(store, "set_extras", _set)
await _backfill_listing_cache(store, [("a1", cfg)])
assert calls == {"replace": 0, "set": 0}
# Changing the config triggers a re-write.
cfg2 = AgentConfig(id="a1", name="Agent", skills=["s1", "s2"], tool_groups=None, model=None)
await _backfill_listing_cache(store, [("a1", cfg2)])
assert calls["replace"] == 1
@@ -217,6 +217,29 @@ async def test_sql_store_filters_errors_only(sql_store):
assert only_errors[0]["error_type"] == "APITimeoutError"
@pytest.mark.asyncio
async def test_sql_store_filters_by_user_ids(sql_store):
"""``user_ids`` (set by a fuzzy username search) matches several accounts."""
base = datetime.now(UTC)
for uid in ("alice", "alicia", "bob"):
await sql_store.record(
LlmMetricRecord(
id=uuid.uuid4().hex,
created_at=base,
user_id=uid,
duration_ms=1,
status="success",
)
)
# Fuzzy "ali" → {alice, alicia}; bob must be excluded.
rows = await sql_store.list(LlmMetricsQuery(user_ids=["alice", "alicia"], limit=10))
assert {r["user_id"] for r in rows} == {"alice", "alicia"}
assert await sql_store.count(LlmMetricsQuery(user_ids=["alice", "alicia"])) == 2
# user_ids takes precedence over a single user_id.
rows = await sql_store.list(LlmMetricsQuery(user_id="bob", user_ids=["alice"], limit=10))
assert {r["user_id"] for r in rows} == {"alice"}
@pytest.mark.asyncio
async def test_sql_store_iter_all_paginates(sql_store):
base = datetime.now(UTC)
+33 -1
View File
@@ -137,6 +137,17 @@ title:
User: {user_msg}
Assistant: {assistant_msg}
# 上下文压缩(对话摘要):长对话接近上下文上限时,把较早的消息总结成一段摘要,
# 只保留最近若干条原始消息,避免撑爆模型上下文窗口。
# - trigger:触发条件,可配多个,任意一个先满足就压缩(OR 关系)。
# type=tokens 按 token 总数(最准,直接防溢出)
# type=messages 按消息条数(简单直观,作兜底保险)
# type=fraction 按模型上下文窗口的比例(如 0.8);自定义 base_url 模型可能探不到
# 准确窗口,故此处用写死的 token 阈值更稳妥。
# - keep:压缩后保留多少最近的上下文(同样支持 tokens / messages / fraction)。
# - trim_tokens_to_summarize:喂给摘要模型的消息最多裁到多少 token。
# - model_name:生成摘要用的模型(留空则用主对话模型)。
skills:
path: /app/skills
security_scan:
@@ -163,6 +174,9 @@ auth_login:
# 永远 401。此处 uat 域名按自签处理,先关校验;生产请置 true 或配 ca_cert_path。
verify_ssl: false
ca_cert_path: "" # 可选:PEM 格式 CA 包路径,设置后优先生效并启用校验
# 日志系统入口(仅管理员可见):右下角「设置」菜单中会多出一个「日志系统」入口,
# 点击后在新标签页打开下面的 url。enabled 关掉则不显示该入口。
database:
backend: mysql
mysql_url: $MYSQL_DATABASE_URL
@@ -241,4 +255,22 @@ memory:
scan_content: true # 注入/外渗模式扫描
block_invisible_unicode: true
# streaming_scrubber:预留字段,代码尚未实现 —— 待版本支持后取消注释。
# streaming_scrubber: true # SSE 流前清洗 <memory-context>
# streaming_scrubber: true # SSE 流前清洗 <memory-context>
log_system:
enabled: true
url: "https://your-log-system.example.com"
label: "日志系统"
summarization:
enabled: true
# model_name: deepseek-chat
trigger:
- type: tokens
value: 60000
- type: messages
value: 80
keep:
type: messages
value: 20
trim_tokens_to_summarize: 4000