Merge branch 'master' of http://192.168.0.200/knowledge-base-bj/deerflow-server into dhk-1
解决 artifact 预览组件冲突:mermaid 渲染(dhk-1) 与 markdown 锚点跳转(master) 取并集;package-lock.json 按 master 删除(项目用 pnpm)。 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
commit
b2132dd668
104 files changed
+8775
-12038
No files matched your search
@@ -263,12 +263,22 @@ Proxied through nginx: `/api/langgraph/*` → LangGraph, all other `/api/*` →
|
||||
|
||||
**HTML page tasks (`task_kind == "html_page"`)**: a second task kind that renders a full HTML document each run instead of a Markdown report. The kind + its config (`html_config`: `page_prompt`, `negative_prompt`, `reference_favorite_id`, `model_name`) live inside `execution_context` — no schema change to `scheduled_tasks`. Data gathering is delegated to the agent's own search skills (driven by `page_prompt`), so there is no explicit search-query/data-source field; the render model is user-selectable (`model_name`, default `zai-org/GLM-5-FP8`). `_run_task_attempt()` dispatches by kind; `_run_html_page_attempt()` runs the pipeline: gather real data via the lead agent (its search tools) → optionally load a reference favorite's reusable style/layout → render with the chosen model (thinking off) → fault-tolerant JSON parse → sanitize (BeautifulSoup, strips `<script>`/`<iframe>`/`on*`/`javascript:`) → structural validate → one GLM5 repair pass if invalid → **completeness gate**: if the HTML is still not a full valid document (missing doctype/html/head/body, empty body, or residual scripts) after the repair pass, raise `HtmlGenerationError` so the run is marked **failed** (a page only counts as successful when it is a complete, previewable HTML document) → best-effort Playwright screenshot (no-op when Playwright absent) → archive `index.html` → persist a `scheduled_html_pages` row → return `result` with `format: "html"` and the full HTML inline. Pure helpers live in `deerflow/runtime/scheduler/html_page.py`. The frontend previews the HTML in a sandboxed `iframe` (no `allow-scripts`) on the run-detail and public pages; a user can bookmark a page (style summary extracted via GLM5) and later pick it as a style reference. New tables `scheduled_html_pages` + `html_page_favorites` (`deerflow/persistence/html_pages/`, wired as `app.state.html_page_store`) are created by `Base.metadata.create_all` with no migration. Tests: `tests/test_html_page_tasks.py`.
|
||||
|
||||
**Unified output switches (`require_html` + `require_markdown`).** The task form no longer splits "普通任务 / HTML 页面" into separate kinds. `execution_context` now carries two **independent** booleans that may both be on: `require_html` (必须生成 HTML) and `require_markdown` (必须生成 Markdown). `_task_requires_html()` returns true for `require_html` **or** legacy `task_kind == "html_page"` (so old tasks need no data migration), and `_run_task_attempt()` dispatches to `_run_html_page_attempt` when it is true, else the regular agent attempt. When **both** switches are on, `_gather_html_data()` returns `(text, files)` — the gather agent run is additionally told to save a `.md` file — and `_run_html_page_attempt` validates it (`MarkdownRequirementError` on failure) and surfaces it alongside the generated page, so the run delivers both artifacts. The single main `prompt` doubles as the HTML page requirement (`html_config.page_prompt`); there is no separate page-prompt field. Tests: `tests/test_scheduled_task_require_html.py` + the markdown-coexistence cases in `tests/test_html_page_tasks.py`.
|
||||
|
||||
### Per-user Prompts & UI Preferences
|
||||
|
||||
**Custom prompts can be scoped to one agent.** `user_custom_prompts` gains a nullable `agent_id` column (migration `20260603_02`): `NULL` = global (the chip shows in every chat), otherwise the prompt only surfaces when that agent is selected. The visibility filter is applied in the frontend `CustomPromptBar` (it receives the current agent id from `ChatPage`'s selector or `AgentChatPage`'s `assistantId`); the backend (`/api/user-prompts` CRUD) just stores/returns/clears `agent_id` (update accepts an explicit `null` to clear). Tests: `tests/test_user_prompts_agent_id.py`.
|
||||
|
||||
**Per-user UI preferences** (`deerflow/persistence/user_preferences/`, wired as `app.state.user_preferences_store`; migration `20260603_03`, also auto-created by `create_all`): one JSON-blob row per user so new toggles need no schema change. Router `/api/user-preferences` (`GET`/`PUT`, account-scoped). Current key: `hide_admin_prompt_prefix` — when true the chat input hides the admin-configured prompt-prefix switch and never applies the prefix (the user uses only their own prompts; enforced in `ChatPage` for both `shouldApplyPrefix` and `promptPrefixProp`). Toggled from the "管理常用提示词" dialog. Tests: `tests/test_user_preferences.py`.
|
||||
|
||||
**HTML skills (tools the render step uses).** Two offline skills under `skills/public/` shape the generated HTML and are toggled in the skills settings page (enabled by default): `html-page-builder` (a self-contained inline-CSS design system — colors/cards/tables/KPI/responsive) and `html-charts` (inline-SVG charts that render in the no-`allow-scripts` preview iframe; optional local ECharts documented in its `assets/README.md`). A skill opts into HTML injection via `metadata.kind: html_page` in its SKILL.md frontmatter; `_collect_html_skill_instructions()` reads the bodies of **enabled** such skills and `build_generation_prompt(skill_instructions=…)` injects them, so the (raw-model) render step honors the user's skill selection. The data-gathering step and the render/repair/style passes all use the task's selected `model_name` (no silent fallback to the default model — and there is **no hardcoded GLM-5**; an unset selection falls back to config `models[0]`). To stop auxiliary LLM calls from leaking onto the default model, **`TitleMiddleware` is skipped when `is_scheduled_run` is set** (`_build_middlewares` in `lead_agent/agent.py`) — scheduled/HTML runs use throwaway threads whose titles are never shown, and `config.title.model_name` is usually unset (→ would call `models[0]`, not the chosen model). The completeness gate (`validate_html`) also requires a non-empty `<title>` and forbids external stylesheets, on top of the doctype/html/head/body + non-empty-body + no-script checks.
|
||||
|
||||
### Custom Agent Persistence
|
||||
|
||||
Custom agent metadata lives in the `agents` ORM table (`deerflow.persistence.agents`). `agents.id` is the stable runtime id and the filesystem directory name under `.deer-flow/agents/{id}/`; `agents.name` is display-only, can contain Chinese characters, and is not unique. `user_id IS NULL` marks built-in agents, user-owned agents are visible only to their owner unless `published = true`, and update/delete operations require ownership. The runtime accepts `agent_id` in context/configurable, and non-default `assistant_id` values are treated as agent ids after visibility checks.
|
||||
|
||||
**Listing cache + pagination** — to keep `GET /api/agents` fast, the gallery's per-card extras (skills + tool_groups) are denormalized into two cache tables instead of reading every agent's `config.yaml` on each request: `agent_skills` (relational, one row per (agent, skill) with `position`) and `agent_extras` (one row per agent: `tool_groups` JSON + `model` + `skills_synced_at`). `config.yaml` remains authoritative for the agent runtime; these tables are a read cache, write-through on create/update and backfilled from disk by `_sync_legacy_agents` (mtime-vs-`skills_synced_at` drift check). The list endpoint enriches via `AgentStore.get_extras_for([ids])` (two batched `IN` queries). Both tables are auto-created by `Base.metadata.create_all` (no migration needed). `GET /api/agents` supports **opt-in server-side pagination**: `scope` (`mine` = owned + favorited / `square` = built-in/published public), `page`, `page_size`, returning `{agents, total, page, page_size}`; omitting `scope`/`page` preserves the legacy "return everything" behavior. Name `search` matches the raw OR desensitized name (the frontend's static term→abbrev map is mirrored server-side in `deerflow/config/agent_name_desensitize.py`); `tag_ids` is an SQL OR-filter. `POST /api/agents/private-square/list` takes the same pagination params. Repository entry point: `AgentStore.list_paginated(...)`.
|
||||
|
||||
**Homepage selector (agent picker on the new-chat page)** — single admin-curated list backed by `agents.featured_order` (nullable BIGINT). NULL = not featured. Only `published = true` agents can be featured (router enforces on writes); any published agent is eligible regardless of ownership. Cleared with `order: null`. The list rendered to end users is `WHERE featured_order IS NOT NULL` + visibility-filter, sorted by `featured_order ASC`. Admins reorder by sending the desired ids to `PUT /api/agents/featured/order` (assigns indices 0..N-1). BIGINT is used rather than INT so the management UI can seed initial values from `Date.now()` (overflows INT on MySQL).
|
||||
|
||||
### Portable Column Types (`deerflow/persistence/types.py`)
|
||||
@@ -511,6 +521,8 @@ Per-user runtime overrides (whitelisted fields) live at `{base_dir}/users/{user_
|
||||
|
||||
Both endpoints are whitelisted in `auth_middleware._PUBLIC_EXACT_PATHS`. The shared `_get_or_create_user_by_username()` helper (in `routers/auth.py`) backs both `/login/username` and `/login/token`.
|
||||
|
||||
- `log_system` - Admin-only external log-system entry (`deerflow/config/log_system_config.py`): `{enabled, url, label}`. Surfaced read-only by `GET /api/system-settings/log-system` (admin only); the frontend renders it as a "日志系统" item in the bottom-right settings menu (`workspace-nav-menu.tsx`) that opens `url` in a new tab. Disabled by default.
|
||||
|
||||
**`extensions_config.json`**:
|
||||
- `mcpServers` - Map of server name → config (enabled, type, command, args, env, url, headers, oauth, description)
|
||||
- `skills` - Map of skill name → state (enabled)
|
||||
|
||||
@@ -59,6 +59,7 @@ from app.gateway.routers import (
|
||||
thread_shares,
|
||||
threads,
|
||||
uploads,
|
||||
user_preferences,
|
||||
user_prompts,
|
||||
writing,
|
||||
)
|
||||
@@ -859,6 +860,9 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
|
||||
|
||||
app.include_router(user_prompts.router)
|
||||
|
||||
# Per-user UI preferences at /api/user-preferences
|
||||
app.include_router(user_preferences.router)
|
||||
|
||||
# Roundtable-planning drafts + recommendation history at /api/roundtable-drafts
|
||||
app.include_router(roundtable_drafts.router)
|
||||
|
||||
|
||||
@@ -71,6 +71,14 @@ class LocalAuthProvider(AuthProvider):
|
||||
"""
|
||||
return await self._repo.find_user_by_email_local_part(local_part)
|
||||
|
||||
async def find_users_by_email_local_part_like(self, substring: str, *, limit: int = 500) -> list[User]:
|
||||
"""Fuzzy-match users by a substring of their email local part.
|
||||
|
||||
Backs the admin LLM-metrics username filter — returns every account
|
||||
whose username contains ``substring`` (case-insensitive).
|
||||
"""
|
||||
return await self._repo.find_users_by_email_local_part_like(substring, limit=limit)
|
||||
|
||||
async def create_user(self, email: str, password: str | None = None, system_role: str = "user", needs_setup: bool = False) -> User:
|
||||
"""Create a new local user.
|
||||
|
||||
|
||||
@@ -81,6 +81,24 @@ class UserRepository(ABC):
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
async def find_users_by_email_local_part_like(self, substring: str, *, limit: int = 500) -> list["User"]:
|
||||
"""Fuzzy-match users whose email local part *contains* ``substring``.
|
||||
|
||||
Backs the admin LLM-metrics username filter (按用户名模糊搜索). Unlike
|
||||
:meth:`find_user_by_email_local_part` (exact prefix → single account),
|
||||
this returns every matching account. Case-insensitive substring match
|
||||
against the part before ``@``. Non-overriding repositories return an
|
||||
empty list.
|
||||
|
||||
Args:
|
||||
substring: Case-insensitive substring to match within the local part.
|
||||
limit: Maximum number of users to return.
|
||||
|
||||
Returns:
|
||||
Matching users (may be empty).
|
||||
"""
|
||||
return []
|
||||
|
||||
@abstractmethod
|
||||
async def update_user(self, user: User) -> User:
|
||||
"""Update an existing user.
|
||||
|
||||
@@ -109,6 +109,33 @@ class SQLiteUserRepository(UserRepository):
|
||||
row = result.scalar_one_or_none()
|
||||
return self._row_to_user(row) if row is not None else None
|
||||
|
||||
async def find_users_by_email_local_part_like(self, substring: str, *, limit: int = 500) -> list[User]:
|
||||
term = (substring or "").strip().lower()
|
||||
if not term:
|
||||
return []
|
||||
# Escape SQL LIKE wildcards so a literal ``_``/``%`` in the query does
|
||||
# not turn into a wildcard. ``\`` is the chosen escape char.
|
||||
escaped = term.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
|
||||
pattern = f"%{escaped}%"
|
||||
# Prefilter in SQL on the whole email (portable across sqlite/postgres/
|
||||
# mysql), then keep only rows whose *local part* contains the term so a
|
||||
# domain-only hit (e.g. searching "gmail") doesn't pull unrelated users.
|
||||
stmt = (
|
||||
select(UserRow)
|
||||
.where(func.lower(UserRow.email).like(pattern, escape="\\"))
|
||||
.order_by(UserRow.created_at.asc())
|
||||
.limit(max(1, limit))
|
||||
)
|
||||
async with self._sf() as session:
|
||||
result = await session.execute(stmt)
|
||||
rows = result.scalars().all()
|
||||
out: list[User] = []
|
||||
for row in rows:
|
||||
local = (row.email or "").split("@", 1)[0].lower()
|
||||
if term in local:
|
||||
out.append(self._row_to_user(row))
|
||||
return out
|
||||
|
||||
async def update_user(self, user: User) -> User:
|
||||
async with self._sf() as session:
|
||||
row = await session.get(UserRow, str(user.id))
|
||||
|
||||
@@ -163,6 +163,10 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
|
||||
app.state.user_prompt_store = make_user_prompt_store(sf)
|
||||
|
||||
from deerflow.persistence.user_preferences import make_user_preferences_store
|
||||
|
||||
app.state.user_preferences_store = make_user_preferences_store(sf)
|
||||
|
||||
from deerflow.persistence.roundtable_drafts import make_roundtable_draft_store
|
||||
|
||||
app.state.roundtable_draft_store = make_roundtable_draft_store(sf)
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -15,7 +15,7 @@ from fastapi import APIRouter, Depends, HTTPException, Query, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.gateway.auth.password import verify_password_async
|
||||
from app.gateway.deps import get_agent_store, get_config, get_tag_store
|
||||
from app.gateway.deps import get_agent_store, get_config, get_tag_store
|
||||
from deerflow.agents.lead_agent.prompt import refresh_skills_system_prompt_cache_async
|
||||
from deerflow.config.agents_api_config import get_agents_api_config
|
||||
from deerflow.config.agents_config import list_custom_agents, load_agent_config, load_agent_soul, validate_agent_id
|
||||
@@ -62,13 +62,27 @@ class AgentResponse(BaseModel):
|
||||
is_favorite: bool = Field(
|
||||
default=False,
|
||||
description="Whether the requesting user has favorited (收藏) this agent.",
|
||||
)
|
||||
)
|
||||
|
||||
is_pinned: bool = Field(
|
||||
default=False,
|
||||
description="Whether the requesting user has pinned this agent.",
|
||||
)
|
||||
|
||||
|
||||
class AgentsListResponse(BaseModel):
|
||||
"""Response model for listing agents.
|
||||
|
||||
|
||||
class AgentsListResponse(BaseModel):
|
||||
"""Response model for listing all visible agents."""
|
||||
``total`` / ``page`` / ``page_size`` describe server-side pagination. When
|
||||
a caller does not opt into pagination they default to "everything on one
|
||||
page" (``total == len(agents)``, ``page == 1``), so existing unpaginated
|
||||
callers keep working unchanged.
|
||||
"""
|
||||
|
||||
agents: list[AgentResponse]
|
||||
total: int = Field(default=0, description="Total agents matching the query across all pages")
|
||||
page: int = Field(default=1, description="1-based page index of this response")
|
||||
page_size: int = Field(default=0, description="Page size; 0 means the result was not paginated")
|
||||
|
||||
|
||||
class AgentCreateRequest(BaseModel):
|
||||
@@ -101,7 +115,7 @@ class AgentUpdateRequest(BaseModel):
|
||||
tool_groups: list[str] | None = Field(default=None, description="Updated tool group whitelist (config.yaml)")
|
||||
|
||||
|
||||
class AgentSelectorResponse(BaseModel):
|
||||
class AgentSelectorResponse(BaseModel):
|
||||
"""Response for the homepage agent selector dropdown.
|
||||
|
||||
The list is fully curated by admins (``featured_order`` on each row).
|
||||
@@ -111,10 +125,10 @@ class AgentSelectorResponse(BaseModel):
|
||||
agents: list[AgentResponse] = Field(
|
||||
default_factory=list,
|
||||
description="Admin-curated agents, sorted by featured_order ascending.",
|
||||
)
|
||||
|
||||
|
||||
class AgentFeaturedRequest(BaseModel):
|
||||
)
|
||||
|
||||
|
||||
class AgentFeaturedRequest(BaseModel):
|
||||
"""Request body for setting/clearing an agent's homepage featured rank."""
|
||||
|
||||
order: int | None = Field(
|
||||
@@ -215,6 +229,7 @@ async def _sync_legacy_agents(store: AgentStore) -> None:
|
||||
handlers as a safety net for environments that bypassed the lifespan hook.
|
||||
"""
|
||||
try:
|
||||
pairs: list[tuple[str, Any]] = []
|
||||
for agent_cfg in list_custom_agents():
|
||||
agent_id = agent_cfg.id or agent_cfg.name
|
||||
if not agent_id:
|
||||
@@ -227,10 +242,41 @@ async def _sync_legacy_agents(store: AgentStore) -> None:
|
||||
"published": True,
|
||||
}
|
||||
)
|
||||
pairs.append((agent_id, agent_cfg))
|
||||
await _backfill_listing_cache(store, pairs)
|
||||
except Exception:
|
||||
logger.exception("Failed to sync legacy folder-backed agents")
|
||||
|
||||
|
||||
async def _backfill_listing_cache(store: AgentStore, pairs: list[tuple[str, Any]]) -> None:
|
||||
"""Mirror each agent's config.yaml skills/tool_groups/model into the DB cache.
|
||||
|
||||
The list endpoint reads skills/tool_groups from these cache rows instead of
|
||||
stat+parsing every ``config.yaml`` per request. We fetch the current cache
|
||||
in one batch and only re-write rows whose values actually changed, so a
|
||||
steady-state sync (every ``_LEGACY_SYNC_TTL_SECONDS``) is cheap and does not
|
||||
churn DELETE/INSERTs. For the cache, ``None`` and ``[]`` skills are
|
||||
equivalent ("no skills") — the tri-state only matters to the agent runtime,
|
||||
which reads config.yaml directly.
|
||||
"""
|
||||
if not pairs:
|
||||
return
|
||||
cached = await store.get_extras_for([aid for aid, _ in pairs])
|
||||
for agent_id, cfg in pairs:
|
||||
want_skills = [s for s in (cfg.skills or []) if isinstance(s, str)]
|
||||
want_tool_groups = cfg.tool_groups
|
||||
want_model = cfg.model
|
||||
existing = cached.get(agent_id)
|
||||
if existing is None or existing.get("skills") != want_skills:
|
||||
await store.replace_skills(agent_id, want_skills)
|
||||
if (
|
||||
existing is None
|
||||
or existing.get("tool_groups") != want_tool_groups
|
||||
or existing.get("model") != want_model
|
||||
):
|
||||
await store.set_extras(agent_id, tool_groups=want_tool_groups, model=want_model)
|
||||
|
||||
|
||||
# Throttle the per-request legacy-sync fallback. The real sync runs once at
|
||||
# startup (lifespan); request handlers used to call ``_sync_legacy_agents`` on
|
||||
# every hit, which scanned the filesystem + did N DB upserts per agent each
|
||||
@@ -302,7 +348,7 @@ def _ensure_agent_dir_from_record(record: dict[str, Any]) -> None:
|
||||
logger.exception("Failed to self-heal agent dir for '%s'", agent_id)
|
||||
|
||||
|
||||
def _record_to_response(
|
||||
def _record_to_response(
|
||||
record: dict[str, Any],
|
||||
*,
|
||||
include_soul: bool = False,
|
||||
@@ -344,10 +390,21 @@ def _record_to_response(
|
||||
tool_groups=extras["tool_groups"],
|
||||
skills=extras["skills"],
|
||||
featured_order=record.get("featured_order"),
|
||||
)
|
||||
|
||||
|
||||
async def _record_to_response_async(
|
||||
)
|
||||
|
||||
|
||||
def _sort_agent_responses(responses: list[AgentResponse], sort: str) -> list[AgentResponse]:
|
||||
reverse = sort not in {"time_asc", "updated_asc", "name_asc"}
|
||||
if sort in {"name_asc", "name_desc"}:
|
||||
ordered = sorted(responses, key=lambda r: (r.name or "").lower(), reverse=reverse)
|
||||
elif sort in {"updated_desc", "updated_asc"}:
|
||||
ordered = sorted(responses, key=lambda r: str(r.updated_at or ""), reverse=reverse)
|
||||
else:
|
||||
ordered = sorted(responses, key=lambda r: str(r.created_at or ""), reverse=reverse)
|
||||
return sorted(ordered, key=lambda r: 0 if r.is_pinned else 1)
|
||||
|
||||
|
||||
async def _record_to_response_async(
|
||||
record: dict[str, Any],
|
||||
*,
|
||||
include_soul: bool = False,
|
||||
@@ -403,6 +460,40 @@ async def _get_editable_or_raise(
|
||||
raise HTTPException(status_code=403, detail="You can only modify agents that you created")
|
||||
|
||||
|
||||
async def _build_list_responses(
|
||||
store: AgentStore,
|
||||
tag_store: TagStore,
|
||||
user_id: str,
|
||||
records: list[dict[str, Any]],
|
||||
) -> list[AgentResponse]:
|
||||
"""Build gallery responses for a set of agent rows WITHOUT touching disk.
|
||||
|
||||
skills / tool_groups / model come from the denormalized DB cache
|
||||
(``get_extras_for`` — one batched query) instead of stat+parsing every
|
||||
agent's ``config.yaml`` per request, which was the list-endpoint hot spot.
|
||||
SOUL.md is skipped (only the edit dialog reads it). An agent absent from
|
||||
the cache (never mirrored) keeps its ``None`` extras until the next sync.
|
||||
"""
|
||||
responses = [
|
||||
_record_to_response(a, include_soul=False, include_extras=False) for a in records
|
||||
]
|
||||
ids = [r.id for r in responses]
|
||||
extras = await store.get_extras_for(ids)
|
||||
assignments = await tag_store.list_assignments("agent", ids)
|
||||
favorite_ids = await store.list_favorite_ids(user_id)
|
||||
pin_ids = await store.list_pin_ids(user_id)
|
||||
for resp in responses:
|
||||
cached = extras.get(resp.id)
|
||||
if cached is not None:
|
||||
resp.skills = cached.get("skills")
|
||||
resp.tool_groups = cached.get("tool_groups")
|
||||
resp.model = cached.get("model")
|
||||
resp.tags = assignments.get(resp.id, [])
|
||||
resp.is_favorite = resp.id in favorite_ids
|
||||
resp.is_pinned = resp.id in pin_ids
|
||||
return responses
|
||||
|
||||
|
||||
@router.get(
|
||||
"/agents",
|
||||
response_model=AgentsListResponse,
|
||||
@@ -411,9 +502,21 @@ async def _get_editable_or_raise(
|
||||
)
|
||||
async def list_agents(
|
||||
request: Request,
|
||||
search: str | None = Query(default=None, description="Case-insensitive fuzzy match on agent name"),
|
||||
search: str | None = Query(default=None, description="Case-insensitive fuzzy match on agent name (raw or desensitized display form)"),
|
||||
tag_ids: list[str] | None = Query(default=None, description="Keep agents carrying any of these tags (OR)"),
|
||||
admin: bool = Query(
|
||||
scope: str | None = Query(
|
||||
default=None,
|
||||
pattern="^(mine|square)$",
|
||||
description="Pagination scope. 'mine' = owned + favorited; 'square' = built-in/published public. Omit for the default combined visibility list.",
|
||||
),
|
||||
page: int | None = Query(default=None, ge=1, description="1-based page index. Providing this (or 'scope') enables server-side pagination."),
|
||||
page_size: int = Query(default=20, ge=1, le=200, description="Items per page when paginating."),
|
||||
sort: str = Query(
|
||||
default="time_desc",
|
||||
pattern="^(time_desc|time_asc|updated_desc|updated_asc|name_asc|name_desc)$",
|
||||
description="Sort order. Default is newest created first.",
|
||||
),
|
||||
admin: bool = Query(
|
||||
default=False,
|
||||
description=(
|
||||
"Admin-only override. When ``true`` and the caller is a system admin, "
|
||||
@@ -429,38 +532,38 @@ async def list_agents(
|
||||
tag_store: TagStore = get_tag_store(request)
|
||||
await _maybe_sync_legacy_agents(store)
|
||||
user_id = _current_user_id(request)
|
||||
if admin:
|
||||
if not _is_admin_user(request):
|
||||
raise HTTPException(status_code=403, detail="Only system admins can list all agents")
|
||||
agents = await store.list_all()
|
||||
else:
|
||||
agents = await store.list_visible(user_id)
|
||||
# Load lightweight extras (skills / model / tool_groups from config.yaml)
|
||||
# so the gallery cards can render skill badges. SOUL.md is still skipped
|
||||
# (only the edit dialog reads it). The per-agent config.yaml reads run in
|
||||
# worker threads concurrently, so the event loop stays responsive even
|
||||
# with many agents.
|
||||
responses = list(
|
||||
await asyncio.gather(
|
||||
*(
|
||||
_record_to_response_async(a, include_soul=False, include_extras=True)
|
||||
for a in agents
|
||||
)
|
||||
)
|
||||
)
|
||||
if admin and not _is_admin_user(request):
|
||||
raise HTTPException(status_code=403, detail="Only system admins can list all agents")
|
||||
|
||||
assignments = await tag_store.list_assignments("agent", [r.id for r in responses])
|
||||
favorite_ids = await store.list_favorite_ids(user_id)
|
||||
for resp in responses:
|
||||
resp.tags = assignments.get(resp.id, [])
|
||||
resp.is_favorite = resp.id in favorite_ids
|
||||
paginate = scope is not None or page is not None
|
||||
if paginate:
|
||||
eff_page = page or 1
|
||||
rows, total = await store.list_paginated(
|
||||
user_id=user_id,
|
||||
scope=scope,
|
||||
search=search,
|
||||
tag_ids=tag_ids,
|
||||
page=eff_page,
|
||||
page_size=page_size,
|
||||
admin=admin,
|
||||
sort=sort,
|
||||
)
|
||||
responses = await _build_list_responses(store, tag_store, user_id, rows)
|
||||
return AgentsListResponse(agents=responses, total=total, page=eff_page, page_size=page_size)
|
||||
|
||||
# Legacy unpaginated path — returns everything (admin console, callers
|
||||
# that have not migrated to pagination). Extras now come from the DB
|
||||
# cache too, so this path no longer reads config.yaml per agent.
|
||||
agents = await store.list_all() if admin else await store.list_visible(user_id)
|
||||
responses = await _build_list_responses(store, tag_store, user_id, agents)
|
||||
if search:
|
||||
needle = search.strip().lower()
|
||||
responses = [r for r in responses if needle in r.name.lower()]
|
||||
if tag_ids:
|
||||
wanted = set(tag_ids)
|
||||
responses = [r for r in responses if any(t.get("id") in wanted for t in r.tags)]
|
||||
return AgentsListResponse(agents=responses)
|
||||
if tag_ids:
|
||||
wanted = set(tag_ids)
|
||||
responses = [r for r in responses if any(t.get("id") in wanted for t in r.tags)]
|
||||
responses = _sort_agent_responses(responses, sort)
|
||||
return AgentsListResponse(agents=responses, total=len(responses), page=1, page_size=len(responses))
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
@@ -473,14 +576,14 @@ async def list_agents(
|
||||
summary="Check Agent Name",
|
||||
description="Validate an agent display name. Duplicate names are allowed.",
|
||||
)
|
||||
async def check_agent_name(name: str) -> dict[str, Any]:
|
||||
_require_agents_api_enabled()
|
||||
normalized = _validate_display_name(name)
|
||||
return {"available": True, "name": normalized}
|
||||
|
||||
|
||||
@router.get(
|
||||
"/agents/selector",
|
||||
async def check_agent_name(name: str) -> dict[str, Any]:
|
||||
_require_agents_api_enabled()
|
||||
normalized = _validate_display_name(name)
|
||||
return {"available": True, "name": normalized}
|
||||
|
||||
|
||||
@router.get(
|
||||
"/agents/selector",
|
||||
response_model=AgentSelectorResponse,
|
||||
summary="Agent Homepage Selector",
|
||||
description=(
|
||||
@@ -532,10 +635,12 @@ async def get_agent(agent_id: str, request: Request) -> AgentResponse:
|
||||
# Detail view legitimately reads SOUL.md + config.yaml — hop to a
|
||||
# worker thread so the asyncio loop stays responsive under concurrent
|
||||
# detail fetches.
|
||||
response = await _record_to_response_async(record, include_soul=True, include_extras=True)
|
||||
favorite_ids = await store.list_favorite_ids(user_id)
|
||||
response.is_favorite = agent_id in favorite_ids
|
||||
return response
|
||||
response = await _record_to_response_async(record, include_soul=True, include_extras=True)
|
||||
favorite_ids = await store.list_favorite_ids(user_id)
|
||||
pin_ids = await store.list_pin_ids(user_id)
|
||||
response.is_favorite = agent_id in favorite_ids
|
||||
response.is_pinned = agent_id in pin_ids
|
||||
return response
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
@@ -587,6 +692,10 @@ async def create_agent_endpoint(request: AgentCreateRequest, http_request: Reque
|
||||
"is_private": request.is_private,
|
||||
}
|
||||
)
|
||||
# Mirror skills/tool_groups/model into the listing cache so the gallery
|
||||
# can render this agent's badges without reading its config.yaml.
|
||||
await store.replace_skills(agent_id, request.skills)
|
||||
await store.set_extras(agent_id, tool_groups=request.tool_groups, model=request.model)
|
||||
logger.info("Created agent '%s' (%s) at %s", name, agent_id, agent_dir)
|
||||
return await _record_to_response_async(record, include_soul=True, include_extras=True)
|
||||
|
||||
@@ -653,6 +762,9 @@ async def update_agent(agent_id: str, request: AgentUpdateRequest, http_request:
|
||||
model=next_model,
|
||||
tool_groups=next_tool_groups,
|
||||
)
|
||||
# Keep the listing cache in lock-step with config.yaml on every write.
|
||||
await store.replace_skills(agent_id, next_skills)
|
||||
await store.set_extras(agent_id, tool_groups=next_tool_groups, model=next_model)
|
||||
if request.soul is not None:
|
||||
_write_agent_soul(agent_id, request.soul)
|
||||
|
||||
@@ -762,9 +874,17 @@ async def delete_agent(agent_id: str, request: Request) -> None:
|
||||
await get_tag_store(request).unassign_all("agent", agent_id)
|
||||
except Exception: # noqa: BLE001 — tag cleanup must not block deletion
|
||||
pass
|
||||
try:
|
||||
await store.delete_favorites_for_agent(agent_id)
|
||||
except Exception: # noqa: BLE001 — favorite cleanup must not block deletion
|
||||
try:
|
||||
await store.delete_favorites_for_agent(agent_id)
|
||||
except Exception: # noqa: BLE001 — favorite cleanup must not block deletion
|
||||
pass
|
||||
try:
|
||||
await store.delete_pins_for_agent(agent_id)
|
||||
except Exception: # noqa: BLE001 - pin cleanup must not block deletion
|
||||
pass
|
||||
try:
|
||||
await store.delete_extras_for_agent(agent_id)
|
||||
except Exception: # noqa: BLE001 — cache cleanup must not block deletion
|
||||
pass
|
||||
logger.info("Deleted agent '%s' from %s", agent_id, agent_dir)
|
||||
except HTTPException:
|
||||
@@ -805,7 +925,7 @@ async def favorite_agent(agent_id: str, request: Request) -> None:
|
||||
summary="Unfavorite Agent",
|
||||
description="Remove an agent from the current user's favorites. No-op when not favorited.",
|
||||
)
|
||||
async def unfavorite_agent(agent_id: str, request: Request) -> None:
|
||||
async def unfavorite_agent(agent_id: str, request: Request) -> None:
|
||||
_require_agents_api_enabled()
|
||||
agent_id = _validate_storage_id(agent_id)
|
||||
|
||||
@@ -816,13 +936,60 @@ async def unfavorite_agent(agent_id: str, request: Request) -> None:
|
||||
logger.info("User '%s' unfavorited agent '%s'", user_id, agent_id)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error("Failed to unfavorite agent '%s': %s", agent_id, e, exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to unfavorite agent: {str(e)}") from e
|
||||
|
||||
|
||||
@router.put(
|
||||
"/agents/{agent_id}/featured",
|
||||
except Exception as e:
|
||||
logger.error("Failed to unfavorite agent '%s': %s", agent_id, e, exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to unfavorite agent: {str(e)}") from e
|
||||
|
||||
|
||||
@router.post(
|
||||
"/agents/{agent_id}/pin",
|
||||
status_code=204,
|
||||
summary="Pin Agent",
|
||||
description="Pin a visible agent to the top of the current user's lists. Idempotent.",
|
||||
)
|
||||
async def pin_agent(agent_id: str, request: Request) -> None:
|
||||
_require_agents_api_enabled()
|
||||
agent_id = _validate_storage_id(agent_id)
|
||||
|
||||
try:
|
||||
store = get_agent_store(request)
|
||||
user_id = _current_user_id(request)
|
||||
record = await store.get_visible(agent_id, user_id)
|
||||
if record is None:
|
||||
raise HTTPException(status_code=404, detail=f"Agent '{agent_id}' not found")
|
||||
await store.add_pin(user_id, agent_id)
|
||||
logger.info("User '%s' pinned agent '%s'", user_id, agent_id)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error("Failed to pin agent '%s': %s", agent_id, e, exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to pin agent: {str(e)}") from e
|
||||
|
||||
|
||||
@router.delete(
|
||||
"/agents/{agent_id}/pin",
|
||||
status_code=204,
|
||||
summary="Unpin Agent",
|
||||
description="Remove the current user's pin from an agent. No-op when not pinned.",
|
||||
)
|
||||
async def unpin_agent(agent_id: str, request: Request) -> None:
|
||||
_require_agents_api_enabled()
|
||||
agent_id = _validate_storage_id(agent_id)
|
||||
|
||||
try:
|
||||
store = get_agent_store(request)
|
||||
user_id = _current_user_id(request)
|
||||
await store.remove_pin(user_id, agent_id)
|
||||
logger.info("User '%s' unpinned agent '%s'", user_id, agent_id)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error("Failed to unpin agent '%s': %s", agent_id, e, exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to unpin agent: {str(e)}") from e
|
||||
|
||||
|
||||
@router.put(
|
||||
"/agents/{agent_id}/featured",
|
||||
response_model=AgentResponse,
|
||||
summary="Set Agent Featured Rank (admin)",
|
||||
description=(
|
||||
@@ -864,10 +1031,19 @@ async def set_agent_featured(
|
||||
raise HTTPException(status_code=500, detail=f"Failed to set featured: {str(e)}") from e
|
||||
|
||||
|
||||
class PrivateSquareListRequest(BaseModel):
|
||||
class PrivateSquareListRequest(BaseModel):
|
||||
"""Request body for listing all private-square agents."""
|
||||
|
||||
password: str = Field(..., description="Plain-text private-square password. Verified against the admin-configured hash before the listing is returned.")
|
||||
search: str | None = Field(default=None, description="Case-insensitive fuzzy match on agent name (raw or desensitized).")
|
||||
tag_ids: list[str] | None = Field(default=None, description="Keep agents carrying any of these tags (OR).")
|
||||
page: int | None = Field(default=None, ge=1, description="1-based page index. Providing this enables server-side pagination.")
|
||||
page_size: int = Field(default=20, ge=1, le=200, description="Items per page when paginating.")
|
||||
sort: str = Field(
|
||||
default="time_desc",
|
||||
pattern="^(time_desc|time_asc|updated_desc|updated_asc|name_asc|name_desc)$",
|
||||
description="Sort order. Default is newest created first.",
|
||||
)
|
||||
|
||||
|
||||
class AgentPrivateUpdateRequest(BaseModel):
|
||||
@@ -899,14 +1075,28 @@ async def list_private_square(body: PrivateSquareListRequest, request: Request)
|
||||
try:
|
||||
store = get_agent_store(request)
|
||||
tag_store: TagStore = get_tag_store(request)
|
||||
await _sync_legacy_agents(store)
|
||||
records = await store.list_private()
|
||||
responses = [_record_to_response(r, include_soul=True) for r in records]
|
||||
await _maybe_sync_legacy_agents(store)
|
||||
# user_id is unused by the "private" scope filter (the password is the
|
||||
# gate), but list_paginated requires it for its signature.
|
||||
user_id = _current_user_id(request)
|
||||
|
||||
assignments = await tag_store.list_assignments("agent", [r.id for r in responses])
|
||||
for resp in responses:
|
||||
resp.tags = assignments.get(resp.id, [])
|
||||
return AgentsListResponse(agents=responses)
|
||||
if body.page is not None:
|
||||
rows, total = await store.list_paginated(
|
||||
user_id=user_id,
|
||||
scope="private",
|
||||
search=body.search,
|
||||
tag_ids=body.tag_ids,
|
||||
page=body.page,
|
||||
page_size=body.page_size,
|
||||
sort=body.sort,
|
||||
)
|
||||
responses = await _build_list_responses(store, tag_store, user_id, rows)
|
||||
return AgentsListResponse(agents=responses, total=total, page=body.page, page_size=body.page_size)
|
||||
|
||||
records = await store.list_private()
|
||||
responses = await _build_list_responses(store, tag_store, user_id, records)
|
||||
responses = _sort_agent_responses(responses, body.sort)
|
||||
return AgentsListResponse(agents=responses, total=len(responses), page=1, page_size=len(responses))
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
|
||||
@@ -22,7 +22,7 @@ from app.gateway.auth import (
|
||||
from app.gateway.auth.config import get_auth_config
|
||||
from app.gateway.auth.errors import AuthErrorCode, AuthErrorResponse
|
||||
from app.gateway.csrf_middleware import is_secure_request
|
||||
from app.gateway.deps import get_agent_store, get_current_user_from_request, get_local_provider, get_skill_store
|
||||
from app.gateway.deps import get_agent_store, get_current_user_from_request, get_local_provider, get_run_store, get_skill_store
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -336,7 +336,7 @@ def _set_session_cookie(response: Response, token: str, request: Request) -> Non
|
||||
# production multi-worker setups, replace this with a shared store (Redis,
|
||||
# database-backed counter) to enforce a true per-IP limit.
|
||||
|
||||
_MAX_LOGIN_ATTEMPTS = 5
|
||||
_MAX_LOGIN_ATTEMPTS = 100
|
||||
_LOCKOUT_SECONDS = 300 # 5 minutes
|
||||
|
||||
# ip → (fail_count, lock_until_timestamp)
|
||||
@@ -749,6 +749,7 @@ class UserListItem(BaseModel):
|
||||
system_role: Literal["admin", "user"]
|
||||
skill_count: int
|
||||
agent_count: int
|
||||
qa_count: int = 0 # 问答次数: number of runs (one run = one question)
|
||||
|
||||
|
||||
@router.get("/users", response_model=list[UserListItem])
|
||||
@@ -757,11 +758,19 @@ async def list_all_users(
|
||||
skill_store: SkillStore = Depends(get_skill_store),
|
||||
agent_store: AgentStore = Depends(get_agent_store),
|
||||
):
|
||||
"""List all users with their skill and agent counts. Admin only."""
|
||||
"""List all users with their skill, agent and Q&A counts. Admin only."""
|
||||
user = await get_current_user_from_request(request)
|
||||
if user.system_role != "admin":
|
||||
raise HTTPException(status_code=403, detail="Admin only")
|
||||
users = await get_local_provider().list_users()
|
||||
# One GROUP BY query for everyone's 问答次数, so the list stays a single
|
||||
# aggregate read instead of a per-user count. Best-effort: if the run store
|
||||
# is unavailable the column simply renders as 0 rather than failing the list.
|
||||
try:
|
||||
qa_counts = await get_run_store(request).count_by_user()
|
||||
except Exception:
|
||||
logger.warning("Failed to load per-user run counts for user list", exc_info=True)
|
||||
qa_counts = {}
|
||||
result = []
|
||||
for u in users:
|
||||
uid = str(u.id)
|
||||
@@ -772,6 +781,7 @@ async def list_all_users(
|
||||
system_role=u.system_role,
|
||||
skill_count=await skill_store.count_owned(uid),
|
||||
agent_count=await agent_store.count_owned(uid),
|
||||
qa_count=qa_counts.get(uid, 0),
|
||||
)
|
||||
)
|
||||
return result
|
||||
|
||||
@@ -121,6 +121,26 @@ async def _resolve_user_name_to_id(name: str | None) -> str | None:
|
||||
return str(user.id) if user is not None else None
|
||||
|
||||
|
||||
async def _resolve_user_names_to_ids(name: str | None) -> list[str]:
|
||||
"""Fuzzy-resolve a username fragment to every matching account's UUID.
|
||||
|
||||
Powers the 按用户名模糊搜索 filter: typing part of a username matches all
|
||||
users whose email local part contains it. Returns an empty list when the
|
||||
fragment matches nobody, so the caller can short-circuit to zero rows.
|
||||
"""
|
||||
if not name or not name.strip():
|
||||
return []
|
||||
try:
|
||||
provider = get_local_provider()
|
||||
except Exception:
|
||||
return []
|
||||
try:
|
||||
users = await provider.find_users_by_email_local_part_like(name.strip())
|
||||
except Exception:
|
||||
return []
|
||||
return [str(u.id) for u in users]
|
||||
|
||||
|
||||
async def _attach_usernames(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
"""Resolve each row's ``user_id`` UUID to a friendly ``user_name``.
|
||||
|
||||
@@ -161,9 +181,11 @@ def _build_query(
|
||||
until: str | None,
|
||||
limit: int,
|
||||
offset: int,
|
||||
user_ids: list[str] | None = None,
|
||||
) -> LlmMetricsQuery:
|
||||
return LlmMetricsQuery(
|
||||
user_id=user_id or None,
|
||||
user_ids=user_ids or None,
|
||||
model_name=model_name or None,
|
||||
status=status or None,
|
||||
since=_parse_dt(since),
|
||||
@@ -187,14 +209,15 @@ async def list_llm_metrics(
|
||||
) -> LlmMetricsListResponse:
|
||||
await _require_admin(request)
|
||||
store = _require_store(request)
|
||||
user_ids: list[str] | None = None
|
||||
if not user_id and user_name:
|
||||
resolved = await _resolve_user_name_to_id(user_name)
|
||||
if resolved is None:
|
||||
# Username unknown → no rows; short-circuit instead of a global scan.
|
||||
user_ids = await _resolve_user_names_to_ids(user_name)
|
||||
if not user_ids:
|
||||
# No username matches → no rows; short-circuit the global scan.
|
||||
return LlmMetricsListResponse(items=[], total=0, limit=limit, offset=offset)
|
||||
user_id = resolved
|
||||
query = _build_query(
|
||||
user_id=user_id,
|
||||
user_ids=user_ids,
|
||||
model_name=model_name,
|
||||
status=status,
|
||||
since=since,
|
||||
@@ -236,14 +259,13 @@ async def export_llm_metrics(
|
||||
"""Stream the full filtered metric set as an ``.xlsx`` file."""
|
||||
await _require_admin(request)
|
||||
store = _require_store(request)
|
||||
user_ids: list[str] | None = None
|
||||
if not user_id and user_name:
|
||||
resolved = await _resolve_user_name_to_id(user_name)
|
||||
if resolved is None:
|
||||
# No matching user — return an empty xlsx so the user still gets a
|
||||
# file (matching the list view's empty state).
|
||||
user_id = "__no_such_user__"
|
||||
else:
|
||||
user_id = resolved
|
||||
user_ids = await _resolve_user_names_to_ids(user_name)
|
||||
if not user_ids:
|
||||
# No matching user — sentinel id that matches nothing, so the export
|
||||
# still produces an empty xlsx (matching the list view's empty state).
|
||||
user_ids = ["__no_such_user__"]
|
||||
|
||||
try:
|
||||
from openpyxl import Workbook
|
||||
@@ -252,6 +274,7 @@ async def export_llm_metrics(
|
||||
|
||||
query = _build_query(
|
||||
user_id=user_id,
|
||||
user_ids=user_ids,
|
||||
model_name=model_name,
|
||||
status=status,
|
||||
since=since,
|
||||
|
||||
@@ -7,7 +7,7 @@ import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from datetime import datetime
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
@@ -119,8 +119,10 @@ class SkillResponse(BaseModel):
|
||||
view_count: int = Field(default=0, description="Number of times the skill has been viewed")
|
||||
patch_count: int = Field(default=0, description="Number of times the skill has been patched")
|
||||
last_activity_at: datetime | None = Field(default=None, description="Most recent activity timestamp")
|
||||
created_at: datetime | None = Field(default=None, description="Skill creation timestamp when known")
|
||||
state: str = Field(default="active", description="Lifecycle state: active, stale, or archived")
|
||||
pinned: bool = Field(default=False, description="Whether this skill is pinned (excluded from curation)")
|
||||
featured: bool = Field(default=False, description="Whether this skill is pinned to the top of the list (sort only)")
|
||||
source: str = Field(default="upload", description="Origin: 'agent' for agent-created, 'upload' for user-uploaded")
|
||||
owner_user_id: str | None = Field(default=None, description="Custom skill owner (null = built-in or legacy)")
|
||||
published: bool = Field(default=False, description="Whether the owner has made this skill visible to other users")
|
||||
@@ -241,8 +243,20 @@ def _skill_to_response(
|
||||
from deerflow.skills.usage import get_entry, latest_activity_at
|
||||
|
||||
entry = get_entry(skill.name)
|
||||
created_at_str = entry.get("created_at")
|
||||
last_at_str = latest_activity_at(skill.name)
|
||||
created_at: datetime | None = None
|
||||
last_at: datetime | None = None
|
||||
if created_at_str:
|
||||
try:
|
||||
created_at = datetime.fromisoformat(created_at_str)
|
||||
except Exception:
|
||||
pass
|
||||
if created_at is None:
|
||||
try:
|
||||
created_at = datetime.fromtimestamp(skill.skill_file.stat().st_mtime, tz=UTC)
|
||||
except Exception:
|
||||
pass
|
||||
if last_at_str:
|
||||
try:
|
||||
last_at = datetime.fromisoformat(last_at_str)
|
||||
@@ -278,8 +292,10 @@ def _skill_to_response(
|
||||
view_count=entry.get("view_count", 0),
|
||||
patch_count=entry.get("patch_count", 0),
|
||||
last_activity_at=last_at,
|
||||
created_at=created_at,
|
||||
state=entry.get("state", "active"),
|
||||
pinned=entry.get("pinned", False),
|
||||
featured=entry.get("featured", False),
|
||||
source=entry.get("source", "upload"),
|
||||
owner_user_id=owner_user_id,
|
||||
published=published,
|
||||
@@ -688,6 +704,17 @@ async def _attach_and_filter_tags(
|
||||
return responses
|
||||
|
||||
|
||||
def _sort_skill_responses(responses: list[SkillResponse], sort: str) -> list[SkillResponse]:
|
||||
reverse = sort not in {"time_asc", "activity_asc", "name_asc", "usage_asc"}
|
||||
if sort in {"name_asc", "name_desc"}:
|
||||
return sorted(responses, key=lambda r: (r.name or "").lower(), reverse=reverse)
|
||||
if sort in {"usage_desc", "usage_asc"}:
|
||||
return sorted(responses, key=lambda r: int(r.use_count or 0), reverse=reverse)
|
||||
if sort in {"activity_desc", "activity_asc"}:
|
||||
return sorted(responses, key=lambda r: str(r.last_activity_at or r.created_at or ""), reverse=reverse)
|
||||
return sorted(responses, key=lambda r: str(r.created_at or r.last_activity_at or ""), reverse=reverse)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/skills",
|
||||
response_model=SkillsListResponse,
|
||||
@@ -701,6 +728,11 @@ async def list_skills(
|
||||
tag_store: TagStore = Depends(get_tag_store),
|
||||
search: str | None = Query(default=None, description="Case-insensitive fuzzy match on skill name"),
|
||||
tag_ids: list[str] | None = Query(default=None, description="Keep skills carrying any of these tags (OR)"),
|
||||
sort: str = Query(
|
||||
default="time_desc",
|
||||
pattern="^(time_desc|time_asc|activity_desc|activity_asc|usage_desc|usage_asc|name_asc|name_desc)$",
|
||||
description="Sort order. Default is newest created/modified first.",
|
||||
),
|
||||
admin: bool = Query(
|
||||
default=False,
|
||||
description=(
|
||||
@@ -754,6 +786,7 @@ async def list_skills(
|
||||
)
|
||||
)
|
||||
responses = await _attach_and_filter_tags(tag_store, responses, search, tag_ids)
|
||||
responses = _sort_skill_responses(responses, sort)
|
||||
return SkillsListResponse(skills=responses)
|
||||
except HTTPException:
|
||||
raise
|
||||
@@ -1438,6 +1471,42 @@ async def update_custom_skill_pinned(
|
||||
raise HTTPException(status_code=500, detail=f"Failed to update pinned: {e}")
|
||||
|
||||
|
||||
class SkillFeaturedRequest(BaseModel):
|
||||
"""Request body for pinning a custom skill to the top of the list."""
|
||||
|
||||
featured: bool = Field(..., description="Whether to pin the skill to the top of the list (sort only)")
|
||||
|
||||
|
||||
@router.put(
|
||||
"/skills/custom/{skill_name}/featured",
|
||||
response_model=SkillResponse,
|
||||
summary="Pin or Unpin a Custom Skill to Top",
|
||||
description="Pin a skill to the top of the list (sort only; orthogonal to pinned/anti-archive). Only the owner or admins may invoke.",
|
||||
)
|
||||
async def update_custom_skill_featured(
|
||||
skill_name: str,
|
||||
body: SkillFeaturedRequest,
|
||||
request: Request,
|
||||
config: AppConfig = Depends(get_config),
|
||||
skill_store: SkillStore = Depends(get_skill_store),
|
||||
) -> SkillResponse:
|
||||
try:
|
||||
skill_name = skill_name.replace("\r\n", "").replace("\n", "")
|
||||
user_id = _current_user_id(request)
|
||||
is_admin = _is_admin_user(request)
|
||||
record, _mode = await _get_editable_skill_or_raise(skill_store, skill_name, user_id, is_admin=is_admin)
|
||||
|
||||
from deerflow.skills.usage import set_featured as _usage_set_featured
|
||||
|
||||
_usage_set_featured(skill_name, body.featured)
|
||||
return _reload_skill_response(skill_name, record, config)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error("Failed to update featured for %s: %s", skill_name, e, exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to update featured: {e}")
|
||||
|
||||
|
||||
@router.get("/skills/custom/{skill_name}/history", response_model=CustomSkillHistoryResponse, summary="Get Custom Skill History")
|
||||
async def get_custom_skill_history(skill_name: str, config: AppConfig = Depends(get_config)) -> CustomSkillHistoryResponse:
|
||||
try:
|
||||
|
||||
@@ -10,7 +10,9 @@ from pydantic import BaseModel, Field
|
||||
|
||||
from app.gateway.auth.password import hash_password_async
|
||||
from app.gateway.deps import get_optional_user_from_request
|
||||
from deerflow.config import get_app_config
|
||||
from deerflow.config.system_settings import (
|
||||
LeaderboardSettings,
|
||||
PromptPrefixSettings,
|
||||
SystemSettings,
|
||||
load_system_settings,
|
||||
@@ -57,6 +59,22 @@ class CitationDisplayUpdateRequest(BaseModel):
|
||||
cleanup_words: list[str] | None = None
|
||||
|
||||
|
||||
class LeaderboardSettingsResponse(BaseModel):
|
||||
exclude_user_ids: list[str] = Field(default_factory=list)
|
||||
include_scheduled: bool = True
|
||||
include_admins: bool = True
|
||||
include_failed: bool = True
|
||||
cleanup_system_questions: bool = True
|
||||
|
||||
|
||||
class LeaderboardSettingsUpdateRequest(BaseModel):
|
||||
exclude_user_ids: list[str] | None = None
|
||||
include_scheduled: bool | None = None
|
||||
include_admins: bool | None = None
|
||||
include_failed: bool | None = None
|
||||
cleanup_system_questions: bool | None = None
|
||||
|
||||
|
||||
def _to_response(settings: PromptPrefixSettings) -> PromptPrefixResponse:
|
||||
return PromptPrefixResponse(
|
||||
enabled=settings.enabled,
|
||||
@@ -91,6 +109,44 @@ async def update_prompt_prefix(body: PromptPrefixUpdateRequest, request: Request
|
||||
return _to_response(saved.prompt_prefix)
|
||||
|
||||
|
||||
def _leaderboard_to_response(settings: LeaderboardSettings) -> LeaderboardSettingsResponse:
|
||||
return LeaderboardSettingsResponse(
|
||||
exclude_user_ids=list(dict.fromkeys([uid for uid in settings.exclude_user_ids if uid])),
|
||||
include_scheduled=settings.include_scheduled,
|
||||
include_admins=settings.include_admins,
|
||||
include_failed=settings.include_failed,
|
||||
cleanup_system_questions=settings.cleanup_system_questions,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/leaderboard", response_model=LeaderboardSettingsResponse)
|
||||
async def get_leaderboard_settings(request: Request) -> LeaderboardSettingsResponse:
|
||||
"""Return global leaderboard aggregation settings. Admin only."""
|
||||
await _require_admin(request)
|
||||
return _leaderboard_to_response(load_system_settings().leaderboard)
|
||||
|
||||
|
||||
@router.put("/leaderboard", response_model=LeaderboardSettingsResponse)
|
||||
async def update_leaderboard_settings(
|
||||
body: LeaderboardSettingsUpdateRequest,
|
||||
request: Request,
|
||||
) -> LeaderboardSettingsResponse:
|
||||
"""Update global leaderboard aggregation settings. Admin only."""
|
||||
await _require_admin(request)
|
||||
|
||||
current = load_system_settings()
|
||||
fields = body.model_dump(exclude_unset=True)
|
||||
update: dict[str, object] = {}
|
||||
if "exclude_user_ids" in fields and fields["exclude_user_ids"] is not None:
|
||||
update["exclude_user_ids"] = list(dict.fromkeys([str(uid).strip() for uid in fields["exclude_user_ids"] if str(uid).strip()]))
|
||||
for key in ("include_scheduled", "include_admins", "include_failed", "cleanup_system_questions"):
|
||||
if key in fields and fields[key] is not None:
|
||||
update[key] = bool(fields[key])
|
||||
leaderboard = current.leaderboard.model_copy(update=update)
|
||||
saved = save_system_settings(current.model_copy(update={"leaderboard": leaderboard}))
|
||||
return _leaderboard_to_response(saved.leaderboard)
|
||||
|
||||
|
||||
@router.get("/citation-display", response_model=CitationDisplayResponse)
|
||||
async def get_citation_display() -> CitationDisplayResponse:
|
||||
"""Return global citation display cleanup settings."""
|
||||
@@ -200,3 +256,23 @@ async def update_appearance_settings(
|
||||
else:
|
||||
saved = current
|
||||
return AppearanceResponse(default_theme=saved.appearance.default_theme)
|
||||
|
||||
|
||||
class LogSystemResponse(BaseModel):
|
||||
"""External log-system entry shown in the admin settings menu.
|
||||
|
||||
Sourced from ``config.yaml`` (``log_system`` section), so it is read-only
|
||||
here — there is no PUT counterpart.
|
||||
"""
|
||||
|
||||
enabled: bool = Field(..., description="Whether the entry is shown in the settings menu")
|
||||
url: str = Field(..., description="External log-system URL opened in a new tab")
|
||||
label: str = Field(..., description="Menu label shown for the entry")
|
||||
|
||||
|
||||
@router.get("/log-system", response_model=LogSystemResponse)
|
||||
async def get_log_system(request: Request) -> LogSystemResponse:
|
||||
"""Return the admin-configured external log-system link from config.yaml. Admin only."""
|
||||
await _require_admin(request)
|
||||
cfg = get_app_config().log_system
|
||||
return LogSystemResponse(enabled=cfg.enabled, url=cfg.url, label=cfg.label)
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,66 @@
|
||||
"""Per-user UI preferences (account-bound, follow the user across devices).
|
||||
|
||||
Currently exposes a single toggle: ``hide_admin_prompt_prefix`` — when true the
|
||||
chat input hides the admin-configured prompt-prefix switch so the user only uses
|
||||
their own prompts. Kept as a small key-value (JSON) store so new toggles do not
|
||||
each need a schema change.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/api/user-preferences", tags=["user-preferences"])
|
||||
|
||||
|
||||
class UserPreferencesResponse(BaseModel):
|
||||
hide_admin_prompt_prefix: bool = False
|
||||
|
||||
|
||||
class UserPreferencesUpdateRequest(BaseModel):
|
||||
hide_admin_prompt_prefix: bool | None = Field(default=None)
|
||||
|
||||
|
||||
def _current_user_id(request: Request) -> str:
|
||||
user = getattr(request.state, "user", None)
|
||||
if user is not None:
|
||||
return str(user.id)
|
||||
return get_effective_user_id()
|
||||
|
||||
|
||||
def _get_store(request: Request):
|
||||
store = getattr(request.app.state, "user_preferences_store", None)
|
||||
if store is None:
|
||||
raise HTTPException(status_code=503, detail="User preferences store not available")
|
||||
return store
|
||||
|
||||
|
||||
def _to_response(prefs: dict) -> UserPreferencesResponse:
|
||||
return UserPreferencesResponse(
|
||||
hide_admin_prompt_prefix=bool(prefs.get("hide_admin_prompt_prefix", False)),
|
||||
)
|
||||
|
||||
|
||||
@router.get("", response_model=UserPreferencesResponse)
|
||||
async def get_user_preferences(request: Request) -> UserPreferencesResponse:
|
||||
store = _get_store(request)
|
||||
user_id = _current_user_id(request)
|
||||
prefs = await store.get_for_user(user_id)
|
||||
return _to_response(prefs)
|
||||
|
||||
|
||||
@router.put("", response_model=UserPreferencesResponse)
|
||||
async def update_user_preferences(request: Request, body: UserPreferencesUpdateRequest) -> UserPreferencesResponse:
|
||||
store = _get_store(request)
|
||||
user_id = _current_user_id(request)
|
||||
patch = body.model_dump(exclude_unset=True, exclude_none=True)
|
||||
if not patch:
|
||||
return _to_response(await store.get_for_user(user_id))
|
||||
prefs = await store.merge_for_user(user_id, patch)
|
||||
return _to_response(prefs)
|
||||
@@ -24,6 +24,8 @@ class UserPromptResponse(BaseModel):
|
||||
title: str = ""
|
||||
content: str
|
||||
sort_order: int = 0
|
||||
# Optional single-agent association. NULL = global (shown for every agent).
|
||||
agent_id: str | None = None
|
||||
created_at: datetime | str | None = None
|
||||
updated_at: datetime | str | None = None
|
||||
|
||||
@@ -36,12 +38,16 @@ class UserPromptCreateRequest(BaseModel):
|
||||
title: str = Field(default="", max_length=255)
|
||||
content: str = Field(..., min_length=1)
|
||||
sort_order: int = Field(default=0)
|
||||
agent_id: str | None = Field(default=None, max_length=64)
|
||||
|
||||
|
||||
class UserPromptUpdateRequest(BaseModel):
|
||||
title: str | None = Field(default=None, max_length=255)
|
||||
content: str | None = Field(default=None, min_length=1)
|
||||
sort_order: int | None = None
|
||||
# Sent on every save; an empty string / null clears the association. Use a
|
||||
# sentinel so "field omitted" (don't touch) differs from "set to null".
|
||||
agent_id: str | None = Field(default=None, max_length=64)
|
||||
|
||||
|
||||
def _current_user_id(request: Request) -> str:
|
||||
@@ -77,6 +83,7 @@ async def create_user_prompt(request: Request, body: UserPromptCreateRequest) ->
|
||||
"title": (body.title or "").strip(),
|
||||
"content": body.content,
|
||||
"sort_order": body.sort_order,
|
||||
"agent_id": (body.agent_id or "").strip() or None,
|
||||
},
|
||||
)
|
||||
return UserPromptResponse(**row)
|
||||
@@ -93,6 +100,11 @@ async def update_user_prompt(request: Request, prompt_id: str, body: UserPromptU
|
||||
data["content"] = body.content
|
||||
if body.sort_order is not None:
|
||||
data["sort_order"] = body.sort_order
|
||||
# ``agent_id`` is nullable: only act when the caller actually sent the field
|
||||
# (``model_fields_set``), so an explicit null clears the association while an
|
||||
# omitted field leaves it untouched.
|
||||
if "agent_id" in body.model_fields_set:
|
||||
data["agent_id"] = (body.agent_id or "").strip() or None
|
||||
row = await store.update(prompt_id, user_id, data)
|
||||
if row is None:
|
||||
raise HTTPException(status_code=404, detail="Prompt not found")
|
||||
|
||||
@@ -207,6 +207,10 @@ auth_login:
|
||||
verify_ssl: false
|
||||
ca_cert_path: "" # 可选:PEM 格式 CA 包路径,设置后优先生效并启用校验
|
||||
|
||||
# 日志系统入口(仅管理员可见):右下角「设置」菜单中会多出一个「日志系统」入口,
|
||||
# 点击后在新标签页打开下面的 url。enabled 关掉则不显示该入口。
|
||||
|
||||
|
||||
database:
|
||||
backend: sqlite
|
||||
sqlite_dir: .deer-flow/data
|
||||
@@ -292,3 +296,8 @@ ai_writing:
|
||||
keyword_model: deepseek-chat # 检索词生成专用模型,留空回退写作主模型
|
||||
rank_model: deepseek-chat # 素材排序专用模型,留空回退写作主模型
|
||||
max_materials: 20 # 合并去重后的素材总量上限,范围 [1, 100],默认 20
|
||||
|
||||
log_system:
|
||||
enabled: true
|
||||
url: "https://your-log-system.example.com"
|
||||
label: "日志系统"
|
||||
+87
-23
@@ -1,8 +1,10 @@
|
||||
"""Middleware to inject uploaded files information into agent context."""
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import NotRequired, override
|
||||
import logging
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from typing import NotRequired, override
|
||||
from urllib.parse import unquote
|
||||
|
||||
from langchain.agents import AgentState
|
||||
from langchain.agents.middleware import AgentMiddleware
|
||||
@@ -13,7 +15,7 @@ from deerflow.config.paths import Paths, get_paths
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
from deerflow.utils.file_conversion import extract_outline
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
_OUTLINE_PREVIEW_LINES = 5
|
||||
@@ -22,7 +24,8 @@ _OUTLINE_PREVIEW_LINES = 5
|
||||
# configured vision model), never by reading their bytes. Listing them with the
|
||||
# generic read_file/grep guidance made the agent try to parse image binaries
|
||||
# with bash/python, which fails and wastes turns.
|
||||
_IMAGE_EXTENSIONS = {".jpg", ".jpeg", ".png", ".webp", ".gif", ".bmp"}
|
||||
_IMAGE_EXTENSIONS = {".jpg", ".jpeg", ".png", ".webp", ".gif", ".bmp"}
|
||||
_ARTIFACT_URL_MARKER = "/artifacts/"
|
||||
|
||||
|
||||
def _extract_outline_for_file(file_path: Path) -> tuple[list[dict], list[str]]:
|
||||
@@ -69,7 +72,7 @@ class UploadsMiddlewareState(AgentState):
|
||||
uploaded_files: NotRequired[list[dict] | None]
|
||||
|
||||
|
||||
class UploadsMiddleware(AgentMiddleware[UploadsMiddlewareState]):
|
||||
class UploadsMiddleware(AgentMiddleware[UploadsMiddlewareState]):
|
||||
"""Middleware to inject uploaded files information into the agent context.
|
||||
|
||||
Reads file metadata from the current message's additional_kwargs.files
|
||||
@@ -122,7 +125,7 @@ class UploadsMiddleware(AgentMiddleware[UploadsMiddlewareState]):
|
||||
lines.append(" Use `grep` to search for keywords (e.g. `grep(pattern='keyword', path='/mnt/user-data/uploads/')`).")
|
||||
lines.append("")
|
||||
|
||||
def _create_files_message(self, new_files: list[dict], historical_files: list[dict]) -> str:
|
||||
def _create_files_message(self, new_files: list[dict], historical_files: list[dict]) -> str:
|
||||
"""Create a formatted message listing uploaded files.
|
||||
|
||||
Args:
|
||||
@@ -160,9 +163,61 @@ class UploadsMiddleware(AgentMiddleware[UploadsMiddlewareState]):
|
||||
lines.append("- Only fall back to web search if the file content is clearly insufficient to answer the question.")
|
||||
lines.append("</uploaded_files>")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
def _files_from_kwargs(self, message: HumanMessage, uploads_dir: Path | None = None) -> list[dict] | None:
|
||||
return "\n".join(lines)
|
||||
|
||||
def _import_file_from_artifact_url(
|
||||
self,
|
||||
file_info: dict,
|
||||
uploads_dir: Path | None,
|
||||
) -> bool:
|
||||
"""Copy a previously uploaded file into the current thread when possible.
|
||||
|
||||
iframe-style callers may pass file metadata that points at an upload
|
||||
from another DeerFlow thread. The current run thread still needs a real
|
||||
file under its own ``/mnt/user-data/uploads`` directory, otherwise this
|
||||
middleware will filter the entry out during existence checks.
|
||||
"""
|
||||
filename = file_info.get("filename") or ""
|
||||
artifact_url = str(file_info.get("artifact_url") or "").strip()
|
||||
if not filename or uploads_dir is None or not artifact_url:
|
||||
return False
|
||||
|
||||
if not artifact_url.startswith("/api/threads/") or _ARTIFACT_URL_MARKER not in artifact_url:
|
||||
return False
|
||||
|
||||
prefix, encoded_virtual_path = artifact_url.split(_ARTIFACT_URL_MARKER, 1)
|
||||
source_thread_id = prefix.removeprefix("/api/threads/").strip("/")
|
||||
if not source_thread_id or not encoded_virtual_path:
|
||||
return False
|
||||
|
||||
virtual_path = "/" + unquote(encoded_virtual_path.lstrip("/"))
|
||||
if Path(virtual_path).name != filename:
|
||||
return False
|
||||
|
||||
try:
|
||||
source_path = self._paths.resolve_virtual_path(
|
||||
source_thread_id,
|
||||
virtual_path,
|
||||
user_id=get_effective_user_id(),
|
||||
)
|
||||
except ValueError:
|
||||
logger.debug("Failed to resolve uploaded artifact path: %s", artifact_url, exc_info=True)
|
||||
return False
|
||||
|
||||
if not source_path.is_file():
|
||||
return False
|
||||
|
||||
uploads_dir.mkdir(parents=True, exist_ok=True)
|
||||
target_path = uploads_dir / filename
|
||||
shutil.copy2(source_path, target_path)
|
||||
|
||||
source_markdown = source_path.with_suffix(".md")
|
||||
if source_markdown.is_file() and source_markdown != source_path:
|
||||
shutil.copy2(source_markdown, target_path.with_suffix(".md"))
|
||||
|
||||
return True
|
||||
|
||||
def _files_from_kwargs(self, message: HumanMessage, uploads_dir: Path | None = None) -> list[dict] | None:
|
||||
"""Extract file info from message additional_kwargs.files.
|
||||
|
||||
The frontend sends uploaded file metadata in additional_kwargs.files
|
||||
@@ -183,16 +238,17 @@ class UploadsMiddleware(AgentMiddleware[UploadsMiddlewareState]):
|
||||
|
||||
files = []
|
||||
for f in kwargs_files:
|
||||
if not isinstance(f, dict):
|
||||
continue
|
||||
filename = f.get("filename") or ""
|
||||
if not filename or Path(filename).name != filename:
|
||||
continue
|
||||
if uploads_dir is not None and not (uploads_dir / filename).is_file():
|
||||
continue
|
||||
files.append(
|
||||
{
|
||||
"filename": filename,
|
||||
if not isinstance(f, dict):
|
||||
continue
|
||||
filename = f.get("filename") or ""
|
||||
if not filename or Path(filename).name != filename:
|
||||
continue
|
||||
if uploads_dir is not None and not (uploads_dir / filename).is_file():
|
||||
if not self._import_file_from_artifact_url(f, uploads_dir):
|
||||
continue
|
||||
files.append(
|
||||
{
|
||||
"filename": filename,
|
||||
"size": int(f.get("size") or 0),
|
||||
"path": f"/mnt/user-data/uploads/{filename}",
|
||||
"extension": Path(filename).suffix,
|
||||
@@ -243,6 +299,17 @@ class UploadsMiddleware(AgentMiddleware[UploadsMiddlewareState]):
|
||||
# Get newly uploaded files from the current message's additional_kwargs.files
|
||||
new_files = self._files_from_kwargs(last_message, uploads_dir) or []
|
||||
|
||||
# Only inject when the *current* message actually carries new uploads.
|
||||
# Turns without a fresh upload must not re-inject previously uploaded
|
||||
# files: the original upload turn already injected (and persisted) their
|
||||
# <uploaded_files> block, and the files remain physically available for
|
||||
# read_file/grep/view_image. Re-injecting on every later message made the
|
||||
# agent re-reference old images/docs, and — since the frontend parses
|
||||
# files out of the <uploaded_files> content when additional_kwargs.files
|
||||
# is empty — re-rendered the old attachment on each new question.
|
||||
if not new_files:
|
||||
return None
|
||||
|
||||
# Collect historical files from the uploads directory (all except the new ones)
|
||||
new_filenames = {f["filename"] for f in new_files}
|
||||
historical_files: list[dict] = []
|
||||
@@ -270,9 +337,6 @@ class UploadsMiddleware(AgentMiddleware[UploadsMiddlewareState]):
|
||||
file["outline"] = outline
|
||||
file["outline_preview"] = preview
|
||||
|
||||
if not new_files and not historical_files:
|
||||
return None
|
||||
|
||||
logger.debug(f"New files: {[f['filename'] for f in new_files]}, historical: {[f['filename'] for f in historical_files]}")
|
||||
|
||||
# Create files message and prepend to the last human message content
|
||||
|
||||
+92
@@ -0,0 +1,92 @@
|
||||
"""Server-side mirror of the frontend agent-name desensitization map.
|
||||
|
||||
The web UI displays agent names through a static term→abbreviation map
|
||||
(``frontend-web/src/lib/desensitize.ts``) — e.g. the stored name "态势分析" is
|
||||
shown as "TS分析". The gallery search box therefore matches against the
|
||||
*desensitized* display form, not the raw stored name.
|
||||
|
||||
When search moves server-side (paginated listing), the backend must match the
|
||||
same way, so this module mirrors that static map verbatim. The frontend map is
|
||||
never extended at runtime (``addDesensitizeEntries`` is defined but never
|
||||
called), so a static copy stays in sync.
|
||||
|
||||
Used by the SQL listing query to match "raw name OR desensitized name".
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# Mirror of DESENSITIZE_MAP in frontend-web/src/lib/desensitize.ts.
|
||||
# Keep entries identical to the frontend so server-side and client-side search
|
||||
# behave the same. Order does not affect membership tests.
|
||||
DESENSITIZE_MAP: dict[str, str] = {
|
||||
"态势": "TS",
|
||||
"你好": "nh",
|
||||
"滨海作战体系": "第一张网",
|
||||
"滨海作战部队": "第一张网",
|
||||
"Taiwan Air Force Flight Units": "第二张网",
|
||||
"台海空军飞行部队": "第二张网",
|
||||
"空军": "KJ",
|
||||
"海军": "HJ",
|
||||
"陆军": "LJ",
|
||||
"陆战队": "LZD",
|
||||
"海巡": "HX",
|
||||
"憲兵": "XB",
|
||||
"資通電軍": "ZTDJ",
|
||||
"情报": "QB",
|
||||
"指挥": "ZH",
|
||||
"信息准备": "XX准备",
|
||||
"行为体": "XWT",
|
||||
"驱动因素": "QDYS",
|
||||
"叙事": "XS",
|
||||
"前三定": "QSD",
|
||||
"受众目标": "SZMB",
|
||||
"作战": "ZZ",
|
||||
"目的": "MD",
|
||||
"关键传播节点": "GJCBJD",
|
||||
"战法": "ZF",
|
||||
"行动": "XD",
|
||||
"打击": "DJ",
|
||||
"目标": "MB",
|
||||
"战力": "ZL",
|
||||
"筹划": "COH",
|
||||
"策划": "CEH",
|
||||
"模拟": "MN",
|
||||
"推演": "TY",
|
||||
"认知": "RZ",
|
||||
"触及": "CJ",
|
||||
"军": "J",
|
||||
"美军": "MJ",
|
||||
"台军": "TJ",
|
||||
}
|
||||
|
||||
|
||||
def desensitize_name(name: str) -> str:
|
||||
"""Return ``name`` with every mapped term replaced by its abbreviation.
|
||||
|
||||
Mirrors the frontend ``desensitize()`` — a sequence of literal replacements.
|
||||
"""
|
||||
out = name
|
||||
for term, abbr in DESENSITIZE_MAP.items():
|
||||
if term in out:
|
||||
out = out.replace(term, abbr)
|
||||
return out
|
||||
|
||||
|
||||
def name_matches(name: str, needle: str) -> bool:
|
||||
"""True when ``needle`` (case-insensitive) is in the raw OR desensitized name."""
|
||||
n = needle.strip().lower()
|
||||
if not n:
|
||||
return True
|
||||
return n in name.lower() or n in desensitize_name(name).lower()
|
||||
|
||||
|
||||
def source_terms_for_needle(needle: str) -> list[str]:
|
||||
"""Return the raw source terms whose abbreviation contains ``needle``.
|
||||
|
||||
Lets the SQL search add ``name LIKE %term%`` clauses so a user typing an
|
||||
abbreviation (e.g. "TS") still matches the stored raw term (e.g. "态势").
|
||||
"""
|
||||
n = needle.strip().lower()
|
||||
if not n:
|
||||
return []
|
||||
return [term for term, abbr in DESENSITIZE_MAP.items() if n in abbr.lower()]
|
||||
@@ -15,6 +15,7 @@ from deerflow.config.database_config import DatabaseConfig
|
||||
from deerflow.config.extensions_config import ExtensionsConfig
|
||||
from deerflow.config.gateway_config import GatewayConfig
|
||||
from deerflow.config.guardrails_config import GuardrailsConfig, load_guardrails_config_from_dict
|
||||
from deerflow.config.log_system_config import LogSystemConfig
|
||||
from deerflow.config.login_config import AuthLoginConfig
|
||||
from deerflow.config.memory_config import MemoryConfig, load_memory_config_from_dict
|
||||
from deerflow.config.model_config import ModelConfig
|
||||
@@ -108,6 +109,7 @@ class AppConfig(BaseModel):
|
||||
models: list[ModelConfig] = Field(default_factory=list, description="Available models")
|
||||
vision: VisionConfig = Field(default_factory=VisionConfig, description="Image recognition (识图) configuration")
|
||||
auth_login: AuthLoginConfig = Field(default_factory=AuthLoginConfig, description="Custom login modes (username password gate, token exchange)")
|
||||
log_system: LogSystemConfig = Field(default_factory=LogSystemConfig, description="Admin-only external log-system entry shown in the settings menu")
|
||||
sandbox: SandboxConfig = Field(description="Sandbox configuration")
|
||||
tools: list[ToolConfig] = Field(default_factory=list, description="Available tools")
|
||||
tool_groups: list[ToolGroupConfig] = Field(default_factory=list, description="Available tool groups")
|
||||
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
"""Configuration for the admin-only external log-system entry.
|
||||
|
||||
A small read-only section in ``config.yaml`` that lets a deployment point the
|
||||
"日志系统" entry (shown in the bottom-right settings menu for admins) at an
|
||||
external log/observability system. The URL is opened in a new browser tab.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class LogSystemConfig(BaseModel):
|
||||
"""External log-system link surfaced in the admin settings menu."""
|
||||
|
||||
enabled: bool = Field(default=False, description="Whether the log-system entry is shown in the settings menu")
|
||||
url: str = Field(default="", description="External log-system URL opened in a new tab when the entry is clicked")
|
||||
label: str = Field(default="日志系统", description="Menu label shown for the entry")
|
||||
@@ -78,6 +78,17 @@ class PrivateSquareSettings(BaseModel):
|
||||
model_config = ConfigDict(extra="ignore")
|
||||
|
||||
|
||||
class LeaderboardSettings(BaseModel):
|
||||
"""Admin-configurable defaults for the user leaderboard analytics."""
|
||||
|
||||
exclude_user_ids: list[str] = Field(default_factory=list, description="User ids excluded from leaderboard aggregation.")
|
||||
include_scheduled: bool = Field(default=True, description="Whether scheduled task runs participate in leaderboard statistics.")
|
||||
include_admins: bool = Field(default=True, description="Whether admin accounts participate in leaderboard statistics.")
|
||||
include_failed: bool = Field(default=True, description="Whether failed runs and provider calls participate in leaderboard statistics.")
|
||||
cleanup_system_questions: bool = Field(default=True, description="Hide attachment/system marker questions such as <uploaded_files>.")
|
||||
model_config = ConfigDict(extra="ignore")
|
||||
|
||||
|
||||
class SystemSettings(BaseModel):
|
||||
"""Top-level system settings document."""
|
||||
|
||||
@@ -85,6 +96,7 @@ class SystemSettings(BaseModel):
|
||||
private_square: PrivateSquareSettings = Field(default_factory=PrivateSquareSettings)
|
||||
appearance: AppearanceSettings = Field(default_factory=AppearanceSettings)
|
||||
citation_display: CitationDisplaySettings = Field(default_factory=CitationDisplaySettings)
|
||||
leaderboard: LeaderboardSettings = Field(default_factory=LeaderboardSettings)
|
||||
model_config = ConfigDict(extra="ignore")
|
||||
|
||||
|
||||
|
||||
+8
-1
@@ -4,13 +4,20 @@ from __future__ import annotations
|
||||
|
||||
from deerflow.persistence.agents.base import AgentStore
|
||||
from deerflow.persistence.agents.memory import MemoryAgentStore
|
||||
from deerflow.persistence.agents.model import AgentFavoriteRow, AgentRow
|
||||
from deerflow.persistence.agents.model import (
|
||||
AgentExtrasRow,
|
||||
AgentFavoriteRow,
|
||||
AgentRow,
|
||||
AgentSkillRow,
|
||||
)
|
||||
from deerflow.persistence.agents.sql import AgentRepository
|
||||
|
||||
__all__ = [
|
||||
"AgentExtrasRow",
|
||||
"AgentFavoriteRow",
|
||||
"AgentRepository",
|
||||
"AgentRow",
|
||||
"AgentSkillRow",
|
||||
"AgentStore",
|
||||
"MemoryAgentStore",
|
||||
"make_agent_store",
|
||||
|
||||
+88
-3
@@ -118,6 +118,69 @@ class AgentStore(ABC):
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def list_paginated(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
scope: str | None,
|
||||
search: str | None,
|
||||
tag_ids: list[str] | None,
|
||||
page: int,
|
||||
page_size: int,
|
||||
admin: bool = False,
|
||||
sort: str = "time_desc",
|
||||
) -> tuple[list[dict[str, Any]], int]:
|
||||
"""Return one page of agents plus the total matching count.
|
||||
|
||||
``scope``:
|
||||
- ``"mine"`` — owned by ``user_id`` OR favorited by ``user_id``.
|
||||
- ``"square"`` — built-in or published, and not private-square.
|
||||
- ``"private"``— flagged ``is_private=True`` (caller gates access).
|
||||
- ``None`` — the default listing visibility (``list_visible``).
|
||||
``admin=True`` bypasses the visibility/scope filter entirely (every
|
||||
agent), but still honors ``search`` / ``tag_ids``. Caller authorizes.
|
||||
|
||||
``search`` matches the raw name OR its desensitized display form.
|
||||
``tag_ids`` keeps agents carrying ANY of the tags (OR). Both filters,
|
||||
the COUNT, and LIMIT/OFFSET all execute in SQL.
|
||||
|
||||
Returns ``(rows, total)`` where ``rows`` is at most ``page_size`` items
|
||||
for the 1-based ``page``.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
# --- Listing cache (skills / tool_groups / model) ---
|
||||
|
||||
@abstractmethod
|
||||
async def replace_skills(self, agent_id: str, skills: list[str] | None) -> None:
|
||||
"""Replace the cached skill list for ``agent_id``.
|
||||
|
||||
``None`` and ``[]`` both clear the cache to "no skills" (mirrors the
|
||||
tri-state semantics of ``AgentConfig.skills``).
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def set_extras(self, agent_id: str, *, tool_groups: list[str] | None, model: str | None) -> None:
|
||||
"""Upsert the cached ``tool_groups`` / ``model`` for ``agent_id``."""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def get_extras_for(self, agent_ids: list[str]) -> dict[str, dict[str, Any]]:
|
||||
"""Batch-fetch cached listing extras for ``agent_ids``.
|
||||
|
||||
Returns ``{agent_id: {"skills": list[str] | None, "tool_groups":
|
||||
list[str] | None, "model": str | None}}``. An agent that was never
|
||||
mirrored is absent from the mapping (caller treats it as unknown).
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def delete_extras_for_agent(self, agent_id: str) -> None:
|
||||
"""Drop the cached skills + extras rows for ``agent_id`` (on delete)."""
|
||||
raise NotImplementedError
|
||||
|
||||
# --- Favorites ("收藏") ---
|
||||
|
||||
@abstractmethod
|
||||
@@ -136,6 +199,28 @@ class AgentStore(ABC):
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def delete_favorites_for_agent(self, agent_id: str) -> None:
|
||||
"""Drop every user's favorite of ``agent_id`` (called on agent delete)."""
|
||||
raise NotImplementedError
|
||||
async def delete_favorites_for_agent(self, agent_id: str) -> None:
|
||||
"""Drop every user's favorite of ``agent_id`` (called on agent delete)."""
|
||||
raise NotImplementedError
|
||||
|
||||
# --- Per-user pins ("置顶") ---
|
||||
|
||||
@abstractmethod
|
||||
async def add_pin(self, user_id: str, agent_id: str) -> None:
|
||||
"""Pin ``agent_id`` for ``user_id``. Idempotent."""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def remove_pin(self, user_id: str, agent_id: str) -> bool:
|
||||
"""Remove a per-user pin. Returns ``False`` when none existed."""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def list_pin_ids(self, user_id: str) -> set[str]:
|
||||
"""Return the set of agent ids pinned by ``user_id``."""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def delete_pins_for_agent(self, agent_id: str) -> None:
|
||||
"""Drop every user's pin of ``agent_id`` (called on agent delete)."""
|
||||
raise NotImplementedError
|
||||
+117
-4
@@ -5,6 +5,7 @@ from __future__ import annotations
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from deerflow.config.agent_name_desensitize import name_matches
|
||||
from deerflow.persistence.agents.base import AgentStore
|
||||
|
||||
|
||||
@@ -20,8 +21,12 @@ def _copy(data: dict[str, Any]) -> dict[str, Any]:
|
||||
|
||||
class MemoryAgentStore(AgentStore):
|
||||
def __init__(self) -> None:
|
||||
self._agents: dict[str, dict[str, Any]] = {}
|
||||
self._favorites: set[tuple[str, str]] = set()
|
||||
self._agents: dict[str, dict[str, Any]] = {}
|
||||
self._favorites: set[tuple[str, str]] = set()
|
||||
self._pins: set[tuple[str, str]] = set()
|
||||
# Listing cache mirrors (see AgentExtrasRow / AgentSkillRow):
|
||||
self._skills: dict[str, list[str]] = {}
|
||||
self._extras: dict[str, dict[str, Any]] = {}
|
||||
|
||||
@staticmethod
|
||||
def _is_visible(agent: dict[str, Any], user_id: str) -> bool:
|
||||
@@ -166,6 +171,96 @@ class MemoryAgentStore(AgentStore):
|
||||
self._agents.pop(agent_id, None)
|
||||
return True
|
||||
|
||||
# --- Paginated listing ---
|
||||
|
||||
def _matches_scope(self, agent: dict[str, Any], *, user_id: str, scope: str | None, admin: bool) -> bool:
|
||||
if admin:
|
||||
return True
|
||||
if scope == "square":
|
||||
return (agent.get("user_id") is None or bool(agent.get("published"))) and not bool(agent.get("is_private"))
|
||||
if scope == "mine":
|
||||
owned = agent.get("user_id") is not None and agent.get("user_id") == user_id
|
||||
favorited = (user_id, agent["id"]) in self._favorites
|
||||
return owned or favorited
|
||||
if scope == "private":
|
||||
return bool(agent.get("is_private"))
|
||||
return self._is_visible(agent, user_id)
|
||||
|
||||
async def list_paginated(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
scope: str | None,
|
||||
search: str | None,
|
||||
tag_ids: list[str] | None,
|
||||
page: int,
|
||||
page_size: int,
|
||||
admin: bool = False,
|
||||
sort: str = "time_desc",
|
||||
) -> tuple[list[dict[str, Any]], int]:
|
||||
# NOTE: the in-memory store has no tag-assignment data, so ``tag_ids``
|
||||
# is ignored here. Tag-filter behavior is covered by the SQL repo tests.
|
||||
matched = [
|
||||
a
|
||||
for a in self._agents.values()
|
||||
if self._matches_scope(a, user_id=user_id, scope=scope, admin=admin)
|
||||
and (not search or not search.strip() or name_matches(a.get("name") or "", search))
|
||||
]
|
||||
if sort == "time_asc":
|
||||
matched.sort(key=lambda a: (a.get("created_at") or "", a.get("id") or ""))
|
||||
elif sort == "updated_desc":
|
||||
matched.sort(
|
||||
key=lambda a: (a.get("updated_at") or "", a.get("id") or ""),
|
||||
reverse=True,
|
||||
)
|
||||
elif sort == "updated_asc":
|
||||
matched.sort(key=lambda a: (a.get("updated_at") or "", a.get("id") or ""))
|
||||
elif sort == "name_asc":
|
||||
matched.sort(key=lambda a: ((a.get("name") or "").lower(), a.get("id") or ""))
|
||||
elif sort == "name_desc":
|
||||
matched.sort(
|
||||
key=lambda a: ((a.get("name") or "").lower(), a.get("id") or ""),
|
||||
reverse=True,
|
||||
)
|
||||
else:
|
||||
matched.sort(
|
||||
key=lambda a: (a.get("created_at") or "", a.get("id") or ""),
|
||||
reverse=True,
|
||||
)
|
||||
pin_ids = {agent_id for uid, agent_id in self._pins if uid == user_id}
|
||||
matched.sort(key=lambda a: (0 if a.get("id") in pin_ids else 1,))
|
||||
total = len(matched)
|
||||
start = max(page - 1, 0) * page_size
|
||||
page_rows = matched[start : start + page_size]
|
||||
return [_copy(a) for a in page_rows], total
|
||||
|
||||
# --- Listing cache (skills / tool_groups / model) ---
|
||||
|
||||
async def replace_skills(self, agent_id: str, skills: list[str] | None) -> None:
|
||||
self._skills[agent_id] = [s for s in (skills or []) if isinstance(s, str)]
|
||||
|
||||
async def set_extras(self, agent_id: str, *, tool_groups: list[str] | None, model: str | None) -> None:
|
||||
self._extras[agent_id] = {"tool_groups": tool_groups, "model": model}
|
||||
|
||||
async def get_extras_for(self, agent_ids: list[str]) -> dict[str, dict[str, Any]]:
|
||||
out: dict[str, dict[str, Any]] = {}
|
||||
for agent_id in agent_ids:
|
||||
has_extras = agent_id in self._extras
|
||||
has_skills = agent_id in self._skills
|
||||
if not has_extras and not has_skills:
|
||||
continue
|
||||
extras = self._extras.get(agent_id, {})
|
||||
out[agent_id] = {
|
||||
"skills": list(self._skills.get(agent_id, [])),
|
||||
"tool_groups": extras.get("tool_groups"),
|
||||
"model": extras.get("model"),
|
||||
}
|
||||
return out
|
||||
|
||||
async def delete_extras_for_agent(self, agent_id: str) -> None:
|
||||
self._skills.pop(agent_id, None)
|
||||
self._extras.pop(agent_id, None)
|
||||
|
||||
# --- Favorites ("收藏") ---
|
||||
|
||||
async def add_favorite(self, user_id: str, agent_id: str) -> None:
|
||||
@@ -180,5 +275,23 @@ class MemoryAgentStore(AgentStore):
|
||||
async def list_favorite_ids(self, user_id: str) -> set[str]:
|
||||
return {agent_id for (uid, agent_id) in self._favorites if uid == user_id}
|
||||
|
||||
async def delete_favorites_for_agent(self, agent_id: str) -> None:
|
||||
self._favorites = {(uid, aid) for (uid, aid) in self._favorites if aid != agent_id}
|
||||
async def delete_favorites_for_agent(self, agent_id: str) -> None:
|
||||
self._favorites = {(uid, aid) for (uid, aid) in self._favorites if aid != agent_id}
|
||||
|
||||
# --- Pins ("置顶") ---
|
||||
|
||||
async def add_pin(self, user_id: str, agent_id: str) -> None:
|
||||
self._pins.add((user_id, agent_id))
|
||||
|
||||
async def remove_pin(self, user_id: str, agent_id: str) -> bool:
|
||||
key = (user_id, agent_id)
|
||||
if key not in self._pins:
|
||||
return False
|
||||
self._pins.remove(key)
|
||||
return True
|
||||
|
||||
async def list_pin_ids(self, user_id: str) -> set[str]:
|
||||
return {agent_id for uid, agent_id in self._pins if uid == user_id}
|
||||
|
||||
async def delete_pins_for_agent(self, agent_id: str) -> None:
|
||||
self._pins = {(uid, aid) for (uid, aid) in self._pins if aid != agent_id}
|
||||
+60
-4
@@ -4,11 +4,11 @@ from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from sqlalchemy import BigInteger, Boolean, String, UniqueConstraint
|
||||
from sqlalchemy import BigInteger, Boolean, Integer, String, UniqueConstraint
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from deerflow.persistence.base import Base
|
||||
from deerflow.persistence.types import BeijingDateTime
|
||||
from deerflow.persistence.types import BeijingDateTime, PortableJSON
|
||||
|
||||
|
||||
class AgentRow(Base):
|
||||
@@ -37,7 +37,7 @@ class AgentRow(Base):
|
||||
updated_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC))
|
||||
|
||||
|
||||
class AgentFavoriteRow(Base):
|
||||
class AgentFavoriteRow(Base):
|
||||
"""A user's bookmark of a (typically public-square) agent.
|
||||
|
||||
Lets a user collect agents from the 广场 (square) into their "我的" tab
|
||||
@@ -51,4 +51,60 @@ class AgentFavoriteRow(Base):
|
||||
id: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
user_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||
agent_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC))
|
||||
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC))
|
||||
|
||||
|
||||
class AgentPinRow(Base):
|
||||
"""A per-user pin that keeps an agent near the top of that user's lists."""
|
||||
|
||||
__tablename__ = "agent_pins"
|
||||
__table_args__ = (UniqueConstraint("user_id", "agent_id", name="uq_agent_pins_user_agent"),)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
user_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||
agent_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC))
|
||||
|
||||
|
||||
class AgentSkillRow(Base):
|
||||
"""Denormalized cache of a custom agent's selected skills.
|
||||
|
||||
``config.yaml`` under ``.deer-flow/agents/{id}/`` stays the source of truth
|
||||
for the agent runtime (the lead-agent prompt reads it directly). This table
|
||||
is a read cache so the gallery list endpoint can render skill badges with a
|
||||
single batched ``IN`` query instead of stat+YAML-parsing every agent's
|
||||
``config.yaml`` on each request. Written through on create/update and
|
||||
backfilled from disk by ``_sync_legacy_agents``.
|
||||
|
||||
One row per (agent, skill); ``position`` preserves the configured order.
|
||||
"""
|
||||
|
||||
__tablename__ = "agent_skills"
|
||||
__table_args__ = (UniqueConstraint("agent_id", "skill_name", name="uq_agent_skills_agent_skill"),)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
agent_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||
skill_name: Mapped[str] = mapped_column(String(191), nullable=False, index=True)
|
||||
position: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
|
||||
|
||||
class AgentExtrasRow(Base):
|
||||
"""Denormalized cache of a custom agent's ``tool_groups`` / ``model``.
|
||||
|
||||
Companion to :class:`AgentSkillRow` — same rationale (avoid per-agent
|
||||
``config.yaml`` reads in the list path). One row per agent. The presence of
|
||||
a row also marks "this agent has been mirrored": the list path treats a
|
||||
missing row as "unknown / not yet backfilled" (``skills=None``) so it can
|
||||
be told apart from an explicitly empty skill list.
|
||||
|
||||
``skills_synced_at`` records when the cache was last written from disk so
|
||||
the backfill can skip agents whose ``config.yaml`` has not changed.
|
||||
"""
|
||||
|
||||
__tablename__ = "agent_extras"
|
||||
|
||||
agent_id: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
# list[str] | None — the agent's tool_groups whitelist from config.yaml.
|
||||
tool_groups: Mapped[list | None] = mapped_column(PortableJSON(), nullable=True, default=None)
|
||||
model: Mapped[str | None] = mapped_column(String(191), nullable=True, default=None)
|
||||
skills_synced_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC))
|
||||
+217
-6
@@ -6,12 +6,20 @@ from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
|
||||
from sqlalchemy import and_, delete, func, or_, select
|
||||
from sqlalchemy import and_, case, delete, func, or_, select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
from deerflow.config.agent_name_desensitize import source_terms_for_needle
|
||||
from deerflow.persistence.agents.base import AgentStore
|
||||
from deerflow.persistence.agents.model import AgentFavoriteRow, AgentRow
|
||||
from deerflow.persistence.agents.model import (
|
||||
AgentExtrasRow,
|
||||
AgentFavoriteRow,
|
||||
AgentPinRow,
|
||||
AgentRow,
|
||||
AgentSkillRow,
|
||||
)
|
||||
from deerflow.persistence.tags.model import TagAssignmentRow
|
||||
|
||||
|
||||
def _row_to_dict(row: AgentRow) -> dict[str, Any]:
|
||||
@@ -233,6 +241,167 @@ class AgentRepository(AgentStore):
|
||||
await session.commit()
|
||||
return True
|
||||
|
||||
# --- Paginated listing ---
|
||||
|
||||
def _scope_filter(self, *, user_id: str, scope: str | None, admin: bool):
|
||||
"""Build the visibility predicate for a paginated listing."""
|
||||
if admin:
|
||||
return None # no visibility filter — every agent
|
||||
if scope == "square":
|
||||
return and_(
|
||||
or_(AgentRow.user_id.is_(None), AgentRow.published.is_(True)),
|
||||
AgentRow.is_private.is_(False),
|
||||
)
|
||||
if scope == "mine":
|
||||
# owned OR favorited. A correlated IN-subquery (not a JOIN) keeps a
|
||||
# single row per agent even when it is both owned and favorited.
|
||||
fav_subq = select(AgentFavoriteRow.agent_id).where(AgentFavoriteRow.user_id == user_id)
|
||||
return or_(AgentRow.user_id == user_id, AgentRow.id.in_(fav_subq))
|
||||
if scope == "private":
|
||||
return AgentRow.is_private.is_(True)
|
||||
return self._listing_filter(user_id)
|
||||
|
||||
@staticmethod
|
||||
def _search_clause(search: str):
|
||||
"""Match the raw name OR (via the desensitize mirror) its display form."""
|
||||
needle = search.strip().lower()
|
||||
clauses = [func.lower(AgentRow.name).like(f"%{needle}%")]
|
||||
for term in source_terms_for_needle(needle):
|
||||
clauses.append(AgentRow.name.like(f"%{term}%"))
|
||||
return or_(*clauses)
|
||||
|
||||
@staticmethod
|
||||
def _tag_clause(tag_ids: list[str]):
|
||||
subq = select(TagAssignmentRow.target_id).where(
|
||||
TagAssignmentRow.target_type == "agent",
|
||||
TagAssignmentRow.tag_id.in_(tag_ids),
|
||||
)
|
||||
return AgentRow.id.in_(subq)
|
||||
|
||||
async def list_paginated(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
scope: str | None,
|
||||
search: str | None,
|
||||
tag_ids: list[str] | None,
|
||||
page: int,
|
||||
page_size: int,
|
||||
admin: bool = False,
|
||||
sort: str = "time_desc",
|
||||
) -> tuple[list[dict[str, Any]], int]:
|
||||
filters = []
|
||||
scope_filter = self._scope_filter(user_id=user_id, scope=scope, admin=admin)
|
||||
if scope_filter is not None:
|
||||
filters.append(scope_filter)
|
||||
if search and search.strip():
|
||||
filters.append(self._search_clause(search))
|
||||
if tag_ids:
|
||||
filters.append(self._tag_clause(tag_ids))
|
||||
|
||||
count_stmt = select(func.count()).select_from(AgentRow)
|
||||
pin_subq = select(AgentPinRow.agent_id).where(AgentPinRow.user_id == user_id)
|
||||
pin_rank = case((AgentRow.id.in_(pin_subq), 0), else_=1)
|
||||
|
||||
if sort == "time_asc":
|
||||
order_by = (AgentRow.created_at.asc(), AgentRow.id.asc())
|
||||
elif sort == "updated_desc":
|
||||
order_by = (AgentRow.updated_at.desc(), AgentRow.id.asc())
|
||||
elif sort == "updated_asc":
|
||||
order_by = (AgentRow.updated_at.asc(), AgentRow.id.asc())
|
||||
elif sort == "name_asc":
|
||||
order_by = (func.lower(AgentRow.name).asc(), AgentRow.id.asc())
|
||||
elif sort == "name_desc":
|
||||
order_by = (func.lower(AgentRow.name).desc(), AgentRow.id.asc())
|
||||
else:
|
||||
order_by = (AgentRow.created_at.desc(), AgentRow.id.asc())
|
||||
|
||||
page_stmt = (
|
||||
select(AgentRow)
|
||||
.order_by(pin_rank.asc(), *order_by)
|
||||
.limit(page_size)
|
||||
.offset(max(page - 1, 0) * page_size)
|
||||
)
|
||||
if filters:
|
||||
where = and_(*filters)
|
||||
count_stmt = count_stmt.where(where)
|
||||
page_stmt = page_stmt.where(where)
|
||||
|
||||
async with self._sf() as session:
|
||||
total = await session.scalar(count_stmt) or 0
|
||||
result = await session.execute(page_stmt)
|
||||
rows = [_row_to_dict(row) for row in result.scalars()]
|
||||
return rows, total
|
||||
|
||||
# --- Listing cache (skills / tool_groups / model) ---
|
||||
|
||||
async def replace_skills(self, agent_id: str, skills: list[str] | None) -> None:
|
||||
async with self._sf() as session:
|
||||
await session.execute(delete(AgentSkillRow).where(AgentSkillRow.agent_id == agent_id))
|
||||
for position, name in enumerate(skills or []):
|
||||
if not isinstance(name, str):
|
||||
continue
|
||||
session.add(
|
||||
AgentSkillRow(
|
||||
id=uuid4().hex,
|
||||
agent_id=agent_id,
|
||||
skill_name=name,
|
||||
position=position,
|
||||
)
|
||||
)
|
||||
await session.commit()
|
||||
|
||||
async def set_extras(self, agent_id: str, *, tool_groups: list[str] | None, model: str | None) -> None:
|
||||
async with self._sf() as session:
|
||||
row = await session.get(AgentExtrasRow, agent_id)
|
||||
now = datetime.now(UTC)
|
||||
if row is None:
|
||||
session.add(
|
||||
AgentExtrasRow(
|
||||
agent_id=agent_id,
|
||||
tool_groups=tool_groups,
|
||||
model=model,
|
||||
skills_synced_at=now,
|
||||
)
|
||||
)
|
||||
else:
|
||||
row.tool_groups = tool_groups
|
||||
row.model = model
|
||||
row.skills_synced_at = now
|
||||
await session.commit()
|
||||
|
||||
async def get_extras_for(self, agent_ids: list[str]) -> dict[str, dict[str, Any]]:
|
||||
if not agent_ids:
|
||||
return {}
|
||||
out: dict[str, dict[str, Any]] = {}
|
||||
async with self._sf() as session:
|
||||
extras = await session.execute(
|
||||
select(AgentExtrasRow).where(AgentExtrasRow.agent_id.in_(agent_ids))
|
||||
)
|
||||
for row in extras.scalars():
|
||||
out[row.agent_id] = {
|
||||
"skills": [], # default; filled below if skill rows exist
|
||||
"tool_groups": row.tool_groups,
|
||||
"model": row.model,
|
||||
}
|
||||
skill_rows = await session.execute(
|
||||
select(AgentSkillRow.agent_id, AgentSkillRow.skill_name)
|
||||
.where(AgentSkillRow.agent_id.in_(agent_ids))
|
||||
.order_by(AgentSkillRow.agent_id.asc(), AgentSkillRow.position.asc())
|
||||
)
|
||||
for agent_id, skill_name in skill_rows.all():
|
||||
entry = out.setdefault(
|
||||
agent_id, {"skills": [], "tool_groups": None, "model": None}
|
||||
)
|
||||
entry["skills"].append(skill_name)
|
||||
return out
|
||||
|
||||
async def delete_extras_for_agent(self, agent_id: str) -> None:
|
||||
async with self._sf() as session:
|
||||
await session.execute(delete(AgentSkillRow).where(AgentSkillRow.agent_id == agent_id))
|
||||
await session.execute(delete(AgentExtrasRow).where(AgentExtrasRow.agent_id == agent_id))
|
||||
await session.commit()
|
||||
|
||||
# --- Favorites ("收藏") ---
|
||||
|
||||
async def add_favorite(self, user_id: str, agent_id: str) -> None:
|
||||
@@ -272,7 +441,49 @@ class AgentRepository(AgentStore):
|
||||
)
|
||||
return {row[0] for row in result.all()}
|
||||
|
||||
async def delete_favorites_for_agent(self, agent_id: str) -> None:
|
||||
async with self._sf() as session:
|
||||
await session.execute(delete(AgentFavoriteRow).where(AgentFavoriteRow.agent_id == agent_id))
|
||||
await session.commit()
|
||||
async def delete_favorites_for_agent(self, agent_id: str) -> None:
|
||||
async with self._sf() as session:
|
||||
await session.execute(delete(AgentFavoriteRow).where(AgentFavoriteRow.agent_id == agent_id))
|
||||
await session.commit()
|
||||
|
||||
# --- Pins ("置顶") ---
|
||||
|
||||
async def add_pin(self, user_id: str, agent_id: str) -> None:
|
||||
async with self._sf() as session:
|
||||
existing = (
|
||||
await session.execute(
|
||||
select(AgentPinRow.id)
|
||||
.where(AgentPinRow.user_id == user_id, AgentPinRow.agent_id == agent_id)
|
||||
.limit(1)
|
||||
)
|
||||
).scalar_one_or_none()
|
||||
if existing is not None:
|
||||
return
|
||||
session.add(AgentPinRow(id=uuid4().hex, user_id=user_id, agent_id=agent_id))
|
||||
try:
|
||||
await session.commit()
|
||||
except IntegrityError:
|
||||
await session.rollback()
|
||||
|
||||
async def remove_pin(self, user_id: str, agent_id: str) -> bool:
|
||||
async with self._sf() as session:
|
||||
result = await session.execute(
|
||||
delete(AgentPinRow).where(
|
||||
AgentPinRow.user_id == user_id,
|
||||
AgentPinRow.agent_id == agent_id,
|
||||
)
|
||||
)
|
||||
await session.commit()
|
||||
return (result.rowcount or 0) > 0
|
||||
|
||||
async def list_pin_ids(self, user_id: str) -> set[str]:
|
||||
async with self._sf() as session:
|
||||
result = await session.execute(
|
||||
select(AgentPinRow.agent_id).where(AgentPinRow.user_id == user_id)
|
||||
)
|
||||
return {row[0] for row in result.all()}
|
||||
|
||||
async def delete_pins_for_agent(self, agent_id: str) -> None:
|
||||
async with self._sf() as session:
|
||||
await session.execute(delete(AgentPinRow).where(AgentPinRow.agent_id == agent_id))
|
||||
await session.commit()
|
||||
+3
@@ -34,6 +34,9 @@ class LlmMetricsQuery:
|
||||
"""Filter set for the admin list / export endpoints."""
|
||||
|
||||
user_id: str | None = None
|
||||
# Multiple user ids — set when a fuzzy username search resolves to several
|
||||
# accounts. Takes precedence over ``user_id`` when non-empty.
|
||||
user_ids: list[str] | None = None
|
||||
model_name: str | None = None
|
||||
status: str | None = None
|
||||
since: datetime | None = None
|
||||
|
||||
+4
-1
@@ -58,7 +58,9 @@ class SqlLlmMetricsStore(LlmMetricsStore):
|
||||
logger.warning("Failed to persist llm_call_metrics row %s", metric.id, exc_info=True)
|
||||
|
||||
def _apply_filters(self, stmt, query: LlmMetricsQuery):
|
||||
if query.user_id:
|
||||
if query.user_ids:
|
||||
stmt = stmt.where(LlmCallMetricRow.user_id.in_(query.user_ids))
|
||||
elif query.user_id:
|
||||
stmt = stmt.where(LlmCallMetricRow.user_id == query.user_id)
|
||||
if query.model_name:
|
||||
stmt = stmt.where(LlmCallMetricRow.model_name == query.model_name)
|
||||
@@ -92,6 +94,7 @@ class SqlLlmMetricsStore(LlmMetricsStore):
|
||||
while True:
|
||||
page_query = LlmMetricsQuery(
|
||||
user_id=query.user_id,
|
||||
user_ids=query.user_ids,
|
||||
model_name=query.model_name,
|
||||
status=query.status,
|
||||
since=query.since,
|
||||
|
||||
+45
@@ -0,0 +1,45 @@
|
||||
"""Add ``agent_id`` to ``user_custom_prompts`` for per-agent prompt scoping.
|
||||
|
||||
NULL = global (shown for every agent); otherwise the prompt only surfaces when
|
||||
that agent is selected.
|
||||
|
||||
Revision ID: 20260603_02
|
||||
Revises: 20260603_01
|
||||
Create Date: 2026-06-03
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy import inspect
|
||||
|
||||
|
||||
revision = "20260603_02"
|
||||
down_revision = "20260603_01"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def _has_table(table_name: str) -> bool:
|
||||
inspector = inspect(op.get_bind())
|
||||
return table_name in inspector.get_table_names()
|
||||
|
||||
|
||||
def _has_column(table_name: str, column_name: str) -> bool:
|
||||
inspector = inspect(op.get_bind())
|
||||
return any(col["name"] == column_name for col in inspector.get_columns(table_name))
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
if not _has_table("user_custom_prompts"):
|
||||
return
|
||||
if not _has_column("user_custom_prompts", "agent_id"):
|
||||
op.add_column("user_custom_prompts", sa.Column("agent_id", sa.String(64), nullable=True))
|
||||
op.create_index("ix_user_custom_prompts_agent_id", "user_custom_prompts", ["agent_id"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
if _has_table("user_custom_prompts") and _has_column("user_custom_prompts", "agent_id"):
|
||||
op.drop_index("ix_user_custom_prompts_agent_id", table_name="user_custom_prompts")
|
||||
op.drop_column("user_custom_prompts", "agent_id")
|
||||
+42
@@ -0,0 +1,42 @@
|
||||
"""Add ``user_preferences`` for per-user UI toggles (account-bound).
|
||||
|
||||
Holds a single JSON blob per user — e.g. ``hide_admin_prompt_prefix`` so a user
|
||||
can choose to use only their own prompts and hide the admin-configured prefix.
|
||||
|
||||
Revision ID: 20260603_03
|
||||
Revises: 20260603_02
|
||||
Create Date: 2026-06-03
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy import inspect
|
||||
|
||||
|
||||
revision = "20260603_03"
|
||||
down_revision = "20260603_02"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def _has_table(table_name: str) -> bool:
|
||||
inspector = inspect(op.get_bind())
|
||||
return table_name in inspector.get_table_names()
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
if not _has_table("user_preferences"):
|
||||
op.create_table(
|
||||
"user_preferences",
|
||||
sa.Column("user_id", sa.String(64), primary_key=True),
|
||||
sa.Column("preferences", sa.JSON(), nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
if _has_table("user_preferences"):
|
||||
op.drop_table("user_preferences")
|
||||
+56
@@ -0,0 +1,56 @@
|
||||
"""Per-user pinned agents.
|
||||
|
||||
Revision ID: 20260604_01
|
||||
Revises: 20260603_03
|
||||
Create Date: 2026-06-04
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy import inspect
|
||||
|
||||
|
||||
revision = "20260604_01"
|
||||
down_revision = "20260603_03"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def _has_table(table_name: str) -> bool:
|
||||
inspector = inspect(op.get_bind())
|
||||
return table_name in inspector.get_table_names()
|
||||
|
||||
|
||||
def _has_index(table_name: str, index_name: str) -> bool:
|
||||
if not _has_table(table_name):
|
||||
return False
|
||||
inspector = inspect(op.get_bind())
|
||||
return any(idx["name"] == index_name for idx in inspector.get_indexes(table_name))
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
if not _has_table("agent_pins"):
|
||||
op.create_table(
|
||||
"agent_pins",
|
||||
sa.Column("id", sa.String(64), nullable=False),
|
||||
sa.Column("user_id", sa.String(64), nullable=False),
|
||||
sa.Column("agent_id", sa.String(64), nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("user_id", "agent_id", name="uq_agent_pins_user_agent"),
|
||||
)
|
||||
if not _has_index("agent_pins", "ix_agent_pins_user_id"):
|
||||
op.create_index("ix_agent_pins_user_id", "agent_pins", ["user_id"])
|
||||
if not _has_index("agent_pins", "ix_agent_pins_agent_id"):
|
||||
op.create_index("ix_agent_pins_agent_id", "agent_pins", ["agent_id"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
if _has_index("agent_pins", "ix_agent_pins_agent_id"):
|
||||
op.drop_index("ix_agent_pins_agent_id", table_name="agent_pins")
|
||||
if _has_index("agent_pins", "ix_agent_pins_user_id"):
|
||||
op.drop_index("ix_agent_pins_user_id", table_name="agent_pins")
|
||||
if _has_table("agent_pins"):
|
||||
op.drop_table("agent_pins")
|
||||
+10
-1
@@ -15,7 +15,12 @@ its storage implementation lives in ``deerflow.runtime.events.store.db`` and
|
||||
there is no matching entity directory.
|
||||
"""
|
||||
|
||||
from deerflow.persistence.agents.model import AgentFavoriteRow, AgentRow
|
||||
from deerflow.persistence.agents.model import (
|
||||
AgentExtrasRow,
|
||||
AgentFavoriteRow,
|
||||
AgentRow,
|
||||
AgentSkillRow,
|
||||
)
|
||||
from deerflow.persistence.memory_config.model import UserMemoryConfigRow
|
||||
from deerflow.persistence.feedback.model import FeedbackRow
|
||||
from deerflow.persistence.llm_metrics.model import LlmCallMetricRow
|
||||
@@ -44,12 +49,15 @@ from deerflow.persistence.embed_sessions.model import EmbedSessionRow
|
||||
from deerflow.persistence.thread_shares.model import ThreadShareRow
|
||||
from deerflow.persistence.tool_metrics.model import ToolCallMetricRow
|
||||
from deerflow.persistence.user.model import UserRow
|
||||
from deerflow.persistence.user_preferences.model import UserPreferencesRow
|
||||
from deerflow.persistence.user_prompts.model import UserCustomPromptRow
|
||||
|
||||
__all__ = [
|
||||
"FeedbackRow",
|
||||
"AgentExtrasRow",
|
||||
"AgentFavoriteRow",
|
||||
"AgentRow",
|
||||
"AgentSkillRow",
|
||||
"LlmCallMetricRow",
|
||||
"NotificationRow",
|
||||
"RecommendedQuestionRow",
|
||||
@@ -75,5 +83,6 @@ __all__ = [
|
||||
"ToolCallMetricRow",
|
||||
"UserCustomPromptRow",
|
||||
"UserMemoryConfigRow",
|
||||
"UserPreferencesRow",
|
||||
"UserRow",
|
||||
]
|
||||
@@ -205,6 +205,21 @@ class RunRepository(RunStore):
|
||||
await session.execute(update(RunRow).where(RunRow.run_id == run_id).values(**values))
|
||||
await session.commit()
|
||||
|
||||
async def count_by_user(self) -> dict[str, int]:
|
||||
"""Return ``{user_id: run_count}`` via a single GROUP BY query.
|
||||
|
||||
Each run is one user question, so this is the source for the admin
|
||||
user center's 问答次数 column. Runs without a ``user_id`` are skipped.
|
||||
"""
|
||||
stmt = (
|
||||
select(RunRow.user_id, func.count().label("cnt"))
|
||||
.where(RunRow.user_id.is_not(None))
|
||||
.group_by(RunRow.user_id)
|
||||
)
|
||||
async with self._sf() as session:
|
||||
rows = (await session.execute(stmt)).all()
|
||||
return {r.user_id: r.cnt for r in rows}
|
||||
|
||||
async def aggregate_tokens_by_thread(self, thread_id: str) -> dict[str, Any]:
|
||||
"""Aggregate token usage via a single SQL GROUP BY query."""
|
||||
_completed = RunRow.status.in_(("success", "error"))
|
||||
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
"""Per-user UI preferences persistence."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from deerflow.persistence.user_preferences.model import UserPreferencesRow
|
||||
from deerflow.persistence.user_preferences.sql import UserPreferencesRepository
|
||||
|
||||
__all__ = [
|
||||
"UserPreferencesRepository",
|
||||
"UserPreferencesRow",
|
||||
"make_user_preferences_store",
|
||||
]
|
||||
|
||||
|
||||
def make_user_preferences_store(session_factory):
|
||||
if session_factory is None:
|
||||
return None
|
||||
return UserPreferencesRepository(session_factory)
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
"""ORM model for per-user UI preferences.
|
||||
|
||||
A single key-value (JSON) row per user, holding small UI-level toggles that
|
||||
should follow the account across devices — e.g. whether to hide the
|
||||
admin-configured prompt prefix in the chat input. Kept generic (one JSON blob)
|
||||
so new toggles do not each require an ALTER TABLE.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from sqlalchemy import String
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from deerflow.persistence.base import Base
|
||||
from deerflow.persistence.types import BeijingDateTime, PortableJSON
|
||||
|
||||
|
||||
class UserPreferencesRow(Base):
|
||||
__tablename__ = "user_preferences"
|
||||
|
||||
user_id: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
preferences: Mapped[dict] = mapped_column(PortableJSON(), nullable=False, default=dict)
|
||||
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC))
|
||||
updated_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC))
|
||||
+41
@@ -0,0 +1,41 @@
|
||||
"""SQLAlchemy-backed per-user preferences repository."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
from deerflow.persistence.user_preferences.model import UserPreferencesRow
|
||||
|
||||
|
||||
class UserPreferencesRepository:
|
||||
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
|
||||
self._sf = session_factory
|
||||
|
||||
async def get_for_user(self, user_id: str) -> dict[str, Any]:
|
||||
"""Return the user's stored preferences (empty dict when none)."""
|
||||
async with self._sf() as session:
|
||||
row = await session.get(UserPreferencesRow, user_id)
|
||||
if row is None or not isinstance(row.preferences, dict):
|
||||
return {}
|
||||
return dict(row.preferences)
|
||||
|
||||
async def merge_for_user(self, user_id: str, patch: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Shallow-merge ``patch`` into the user's preferences and return the result."""
|
||||
now = datetime.now(UTC)
|
||||
async with self._sf() as session:
|
||||
row = await session.get(UserPreferencesRow, user_id)
|
||||
if row is None:
|
||||
merged = dict(patch)
|
||||
row = UserPreferencesRow(user_id=user_id, preferences=merged, created_at=now, updated_at=now)
|
||||
session.add(row)
|
||||
else:
|
||||
merged = dict(row.preferences or {})
|
||||
merged.update(patch)
|
||||
# Reassign so SQLAlchemy detects the change on the JSON column.
|
||||
row.preferences = merged
|
||||
row.updated_at = now
|
||||
await session.commit()
|
||||
return dict(merged)
|
||||
+3
@@ -19,5 +19,8 @@ class UserCustomPromptRow(Base):
|
||||
title: Mapped[str] = mapped_column(String(255), nullable=False, default="")
|
||||
content: Mapped[str] = mapped_column(Text, nullable=False)
|
||||
sort_order: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
# Optional association to a single agent. NULL = global (shown everywhere);
|
||||
# otherwise the prompt only surfaces when that agent is selected.
|
||||
agent_id: Mapped[str | None] = mapped_column(String(64), nullable=True, index=True)
|
||||
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC))
|
||||
updated_at: Mapped[datetime] = mapped_column(BeijingDateTime(), nullable=False, default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC))
|
||||
+5
@@ -42,6 +42,7 @@ class UserCustomPromptRepository:
|
||||
title=data.get("title") or "",
|
||||
content=data["content"],
|
||||
sort_order=data.get("sort_order", 0),
|
||||
agent_id=(data.get("agent_id") or None),
|
||||
created_at=now,
|
||||
updated_at=now,
|
||||
)
|
||||
@@ -59,6 +60,10 @@ class UserCustomPromptRepository:
|
||||
for key in ("title", "content", "sort_order"):
|
||||
if key in data:
|
||||
setattr(row, key, data[key])
|
||||
# ``agent_id`` is explicitly nullable: an empty/None value clears the
|
||||
# association (back to global), so accept None as a real update.
|
||||
if "agent_id" in data:
|
||||
row.agent_id = data["agent_id"] or None
|
||||
row.updated_at = datetime.now(UTC)
|
||||
await session.commit()
|
||||
await session.refresh(row)
|
||||
|
||||
@@ -93,3 +93,12 @@ class RunStore(abc.ABC):
|
||||
by_caller ({lead_agent, subagent, middleware}).
|
||||
"""
|
||||
pass
|
||||
|
||||
async def count_by_user(self) -> dict[str, int]:
|
||||
"""Return ``{user_id: run_count}`` across all runs.
|
||||
|
||||
One run corresponds to one user submission, so this backs the admin
|
||||
user center's 问答次数 (Q&A count) column — each question counts once.
|
||||
Non-overriding stores return an empty mapping (treated as 0 per user).
|
||||
"""
|
||||
return {}
|
||||
+39
@@ -38,6 +38,11 @@ _DANGEROUS_URL_RE = re.compile(r"^\s*(javascript|vbscript|data:text/html)", re.I
|
||||
# Attributes carrying a URL we must scheme-check.
|
||||
_URL_ATTRS = frozenset({"href", "src", "action", "formaction", "xlink:href", "background", "poster"})
|
||||
|
||||
_LOCAL_DOC_LINK_RE = re.compile(
|
||||
r"^(?![a-z][a-z0-9+.-]*:)(?!//)(?P<path>[^#?]*\.(?:html?|md|markdown))(?P<query>\?[^#]*)?(?P<fragment>#.+)?$",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
def parse_frontmatter(text: str) -> tuple[dict[str, Any], str]:
|
||||
"""Split a ``---`` YAML frontmatter block from the body of a SKILL.md.
|
||||
@@ -225,6 +230,38 @@ def sanitize_html(html: str) -> tuple[str, list[str]]:
|
||||
return str(soup), removed
|
||||
|
||||
|
||||
def constrain_same_page_anchor_links(html: str) -> tuple[str, list[str]]:
|
||||
"""Keep generated HTML document jumps inside the current HTML file."""
|
||||
if not html or not isinstance(html, str):
|
||||
return "", []
|
||||
try:
|
||||
from bs4 import BeautifulSoup
|
||||
except Exception: # pragma: no cover - bs4 is a declared dependency
|
||||
logger.warning("BeautifulSoup unavailable; returning HTML links unconstrained")
|
||||
return html, []
|
||||
|
||||
changed: list[str] = []
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
for tag in soup.find_all("a"):
|
||||
href = tag.get("href")
|
||||
if not isinstance(href, str):
|
||||
continue
|
||||
value = href.strip()
|
||||
if not value or value.startswith("#"):
|
||||
continue
|
||||
match = _LOCAL_DOC_LINK_RE.match(value)
|
||||
if not match:
|
||||
continue
|
||||
fragment = match.group("fragment")
|
||||
if fragment:
|
||||
tag["href"] = fragment
|
||||
changed.append(f"{value}->{fragment}")
|
||||
else:
|
||||
del tag.attrs["href"]
|
||||
changed.append(f"{value}->removed")
|
||||
return str(soup), changed
|
||||
|
||||
|
||||
def validate_html(html: str) -> list[str]:
|
||||
"""Return a list of structural problems with ``html`` (empty == valid).
|
||||
|
||||
@@ -299,6 +336,7 @@ _GENERATION_TEMPLATE = """你是一个内网页面生成助手。请根据输入
|
||||
6. 不要包含 <script>、外链脚本、iframe、表单提交、on* 事件属性。
|
||||
7. 参考页面只用于风格和布局,页面内容必须来自“本次数据”,不要复用参考页旧数据。
|
||||
8. 页面使用中文,排版适合在浏览器中直接预览。
|
||||
9. HTML 内的目录/锚点跳转必须只跳转到当前这个 HTML 文件内部:只使用 href="#section-id",不要使用 href="index.html#section-id"、".md#..."、其它文件路径或任何跨文件锚点;目标区块必须有匹配的 id。
|
||||
"""
|
||||
|
||||
_REPAIR_TEMPLATE = """你是一个 HTML 修复助手。下面的 HTML 存在问题,请修复后重新输出完整页面。
|
||||
@@ -317,6 +355,7 @@ _REPAIR_TEMPLATE = """你是一个 HTML 修复助手。下面的 HTML 存在问
|
||||
2. fixed_html 必须是完整 HTML 文档,包含 <!doctype html>、html、head、body。
|
||||
3. 移除所有 <script>、iframe、外链脚本、on* 事件属性、表单提交。
|
||||
4. 保留原有内容与数据,只修复结构与安全问题。
|
||||
5. HTML 内的目录/锚点跳转必须只跳转到当前这个 HTML 文件内部:只使用 href="#section-id",不要链接到 .html/.md 文件作为锚点。
|
||||
"""
|
||||
|
||||
_STYLE_EXTRACT_TEMPLATE = """你是一个页面风格分析助手。请分析下面的 HTML 页面,提取可复用的风格与布局特征。
|
||||
|
||||
+131
-17
@@ -53,8 +53,18 @@ _MARKDOWN_MIN_CHARS = 100
|
||||
_MARKDOWN_INSTRUCTION = (
|
||||
"\n\n【输出要求】请将本次任务的最终结果生成为一个 Markdown(.md) 格式的文件,"
|
||||
"保存到 /mnt/user-data/outputs 目录,并使用 present_files 展示该文件。"
|
||||
"Markdown 内的目录/锚点跳转必须只跳转到当前这个 Markdown 文件内部:"
|
||||
"只使用 [标题](#section-id),不要使用 [标题](xxx.md#section-id)、"
|
||||
"[标题](xxx.html#section-id) 或任何跨文件锚点。"
|
||||
)
|
||||
|
||||
_LOCAL_DOC_MARKDOWN_LINK_RE = re.compile(
|
||||
r"^(?![a-z][a-z0-9+.-]*:)(?!//)(?P<path>[^#?]*\.(?:html?|md|markdown))(?P<query>\?[^#]*)?(?P<fragment>#.+)?$",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_MARKDOWN_INLINE_LINK_RE = re.compile(r"(?<!!)\[([^\]]+)\]\(([^)\s]+)(?:\s+\"[^\"]*\")?\)")
|
||||
_MARKDOWN_REFERENCE_LINK_RE = re.compile(r"^(\s*\[[^\]]+\]:\s*)(\S+)(.*)$", re.MULTILINE)
|
||||
|
||||
|
||||
class MarkdownRequirementError(Exception):
|
||||
"""Raised when a ``require_markdown`` task did not produce a valid .md file.
|
||||
@@ -219,6 +229,23 @@ def _task_requires_markdown(task: dict[str, Any]) -> bool:
|
||||
return bool(isinstance(ctx, dict) and ctx.get("require_markdown"))
|
||||
|
||||
|
||||
def _task_requires_html(task: dict[str, Any]) -> bool:
|
||||
"""Whether a task must produce an HTML page (the ``必须生成 HTML`` switch).
|
||||
|
||||
The flag lives inside ``execution_context.require_html``. For backward
|
||||
compatibility, tasks created under the old two-tab UI (which stored
|
||||
``task_kind == "html_page"``) are still treated as HTML tasks, so no data
|
||||
migration is needed. ``require_html`` and ``require_markdown`` are
|
||||
independent — both may be on at once.
|
||||
"""
|
||||
ctx = task.get("execution_context")
|
||||
if not isinstance(ctx, dict):
|
||||
return False
|
||||
if ctx.get("require_html"):
|
||||
return True
|
||||
return ctx.get("task_kind") == "html_page"
|
||||
|
||||
|
||||
def _has_valid_markdown_file(result_files: list[dict[str, Any]] | None) -> bool:
|
||||
"""True when the run produced a ``.md`` file whose text exceeds the minimum.
|
||||
|
||||
@@ -245,6 +272,43 @@ def _has_valid_markdown_file(result_files: list[dict[str, Any]] | None) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def _normalize_same_page_markdown_target(target: str) -> str | None:
|
||||
value = (target or "").strip()
|
||||
if not value or value.startswith("#"):
|
||||
return value
|
||||
match = _LOCAL_DOC_MARKDOWN_LINK_RE.match(value)
|
||||
if not match:
|
||||
return value
|
||||
fragment = match.group("fragment")
|
||||
return fragment or None
|
||||
|
||||
|
||||
def _constrain_markdown_same_page_anchor_links(content: str) -> str:
|
||||
"""Keep generated Markdown document jumps inside the current Markdown file."""
|
||||
if not content or not isinstance(content, str):
|
||||
return content or ""
|
||||
|
||||
def replace_inline(match: re.Match[str]) -> str:
|
||||
label = match.group(1)
|
||||
target = match.group(2)
|
||||
normalized = _normalize_same_page_markdown_target(target)
|
||||
if normalized is None:
|
||||
return label
|
||||
return f"[{label}]({normalized})"
|
||||
|
||||
def replace_reference(match: re.Match[str]) -> str:
|
||||
prefix = match.group(1)
|
||||
target = match.group(2)
|
||||
suffix = match.group(3)
|
||||
normalized = _normalize_same_page_markdown_target(target)
|
||||
if normalized is None:
|
||||
return f"{prefix}#"
|
||||
return f"{prefix}{normalized}{suffix}"
|
||||
|
||||
content = _MARKDOWN_INLINE_LINK_RE.sub(replace_inline, content)
|
||||
return _MARKDOWN_REFERENCE_LINK_RE.sub(replace_reference, content)
|
||||
|
||||
|
||||
def _build_result(
|
||||
task: dict[str, Any],
|
||||
*,
|
||||
@@ -905,15 +969,18 @@ class ScheduledTaskService:
|
||||
scheduled_for: datetime,
|
||||
run_id: str,
|
||||
) -> dict[str, Any]:
|
||||
"""Dispatch a single execution attempt by task kind.
|
||||
"""Dispatch a single execution attempt by the task's output requirements.
|
||||
|
||||
``html_page`` tasks render an HTML document via GLM5; everything else
|
||||
runs the regular lead-agent attempt. The kind lives in
|
||||
``execution_context.task_kind`` so no schema migration is needed.
|
||||
A task carries two independent switches in ``execution_context``:
|
||||
``require_html`` (必须生成 HTML) and ``require_markdown`` (必须生成
|
||||
Markdown). When ``require_html`` is set the run goes through the HTML
|
||||
render pipeline (which also enforces the Markdown requirement when both
|
||||
switches are on); otherwise it runs the regular lead-agent attempt.
|
||||
Both flags live in ``execution_context`` so no schema migration is
|
||||
needed, and old ``task_kind == "html_page"`` tasks still resolve to the
|
||||
HTML pipeline via :func:`_task_requires_html`.
|
||||
"""
|
||||
ctx = task.get("execution_context")
|
||||
kind = ctx.get("task_kind") if isinstance(ctx, dict) else None
|
||||
if kind == "html_page":
|
||||
if _task_requires_html(task):
|
||||
return await self._run_html_page_attempt(
|
||||
task, runtime_thread_id=runtime_thread_id, scheduled_for=scheduled_for, run_id=run_id
|
||||
)
|
||||
@@ -956,17 +1023,21 @@ class ScheduledTaskService:
|
||||
runtime_thread_id: str,
|
||||
scheduled_for: datetime,
|
||||
run_id: str,
|
||||
) -> str:
|
||||
) -> tuple[str, list[dict[str, Any]]]:
|
||||
"""Run the lead agent to collect source data for the HTML page.
|
||||
|
||||
Reuses the regular agent-run primitives (with the task's search query
|
||||
folded into the prompt) and returns the agent's final text, which feeds
|
||||
the GLM5 render step as ``本次数据``. Returns an empty string if the
|
||||
agent produced nothing.
|
||||
folded into the prompt) and returns ``(text, files)`` — the agent's
|
||||
final text (feeds the render step as ``本次数据``) plus any files it
|
||||
produced. The files are only captured/needed when the task also has the
|
||||
Markdown switch on (both 必须生成 HTML and 必须生成 Markdown), so the run
|
||||
can validate and surface the ``.md`` file alongside the HTML page;
|
||||
otherwise the list is empty.
|
||||
"""
|
||||
cfg = self._html_config(task)
|
||||
page_prompt = str(cfg.get("page_prompt") or "").strip()
|
||||
negative_prompt = str(cfg.get("negative_prompt") or "").strip()
|
||||
also_markdown = _task_requires_markdown(task)
|
||||
gather_instruction = (
|
||||
"请运用你自身的检索/搜索等技能,围绕下述页面需求获取最新、真实的资料,"
|
||||
"并整理成结构化的中文要点(可用列表/小标题),供后续生成 HTML 页面使用,本步骤不要生成 HTML:\n"
|
||||
@@ -982,12 +1053,22 @@ class ScheduledTaskService:
|
||||
# which sources/data the agent collects, otherwise it silently leaks in.
|
||||
if negative_prompt:
|
||||
parts.append(f"限制条件(最高优先级,必须严格遵守):{negative_prompt}")
|
||||
gather_task = {**task, "prompt": "\n".join(parts)}
|
||||
# Never apply the markdown gate to the gather pass.
|
||||
gather_prompt = "\n".join(parts)
|
||||
# When the Markdown switch is also on, this same gather pass must save a
|
||||
# .md file. We do NOT append _MARKDOWN_INSTRUCTION here — `_start_agent_run`
|
||||
# already appends it exactly once when `require_markdown` is present in the
|
||||
# execution context (kept below). Appending it here too made the prompt
|
||||
# carry the 【输出要求】 block twice when both HTML and Markdown were on.
|
||||
gather_task = {**task, "prompt": gather_prompt}
|
||||
gather_ctx = dict(task.get("execution_context") or {})
|
||||
gather_ctx.pop("require_markdown", None)
|
||||
# Strip the HTML-specific flags so the gather pass is a plain agent run.
|
||||
gather_ctx.pop("require_html", None)
|
||||
gather_ctx.pop("task_kind", None)
|
||||
gather_ctx.pop("html_config", None)
|
||||
# Keep ``require_markdown`` only when both switches are on, so the agent
|
||||
# is told (via _start_agent_run) to produce the .md file.
|
||||
if not also_markdown:
|
||||
gather_ctx.pop("require_markdown", None)
|
||||
# Use the task's selected model for the data-gathering agent too, so the
|
||||
# whole HTML pipeline honors ``html_config.model_name`` instead of
|
||||
# silently falling back to the default model for this step.
|
||||
@@ -997,13 +1078,22 @@ class ScheduledTaskService:
|
||||
gather_task["execution_context"] = gather_ctx
|
||||
|
||||
await self._ensure_thread_meta(task["user_id"], runtime_thread_id)
|
||||
files: list[dict[str, Any]] = []
|
||||
async with self._agent_run_lock:
|
||||
output_snapshot = self._snapshot_result_files(runtime_thread_id, task["user_id"]) if also_markdown else None
|
||||
record = await self._start_agent_run(gather_task, scheduled_for=scheduled_for, runtime_thread_id=runtime_thread_id)
|
||||
await self.store.update_run(run_id, {"agent_run_id": record.run_id})
|
||||
if record.task is not None:
|
||||
await record.task
|
||||
text = await self._extract_latest_ai_text(runtime_thread_id)
|
||||
return text or ""
|
||||
if also_markdown:
|
||||
files = await self._extract_result_files(
|
||||
runtime_thread_id,
|
||||
task["user_id"],
|
||||
run_id=run_id,
|
||||
output_snapshot=output_snapshot,
|
||||
)
|
||||
return text or "", files
|
||||
|
||||
def _collect_html_skill_instructions(self) -> str:
|
||||
"""Concatenate the bodies of enabled skills marked for HTML pages.
|
||||
@@ -1072,10 +1162,17 @@ class ScheduledTaskService:
|
||||
negative_prompt = str(cfg.get("negative_prompt") or "").strip()
|
||||
reference_favorite_id = cfg.get("reference_favorite_id") or None
|
||||
|
||||
# 1) Gather source data (real, via the agent's search tools).
|
||||
data_text = await self._gather_html_data(
|
||||
# 1) Gather source data (real, via the agent's search tools). When the
|
||||
# Markdown switch is also on, the gather pass additionally saves the .md
|
||||
# file, returned here so we can validate + surface it next to the page.
|
||||
also_markdown = _task_requires_markdown(task)
|
||||
data_text, gathered_files = await self._gather_html_data(
|
||||
task, runtime_thread_id=runtime_thread_id, scheduled_for=scheduled_for, run_id=run_id
|
||||
)
|
||||
if also_markdown and not _has_valid_markdown_file(gathered_files):
|
||||
raise MarkdownRequirementError(
|
||||
f"未满足输出要求:需生成 Markdown(.md) 文件,且文件正文不少于 {_MARKDOWN_MIN_CHARS} 字"
|
||||
)
|
||||
|
||||
# 2) Load reference style/layout from a favorite if one was selected.
|
||||
reference_style = None
|
||||
@@ -1119,6 +1216,7 @@ class ScheduledTaskService:
|
||||
|
||||
# 4) Sanitize + validate, with one repair pass on failure.
|
||||
html, _removed = html_page_utils.sanitize_html(html)
|
||||
html, _link_changes = html_page_utils.constrain_same_page_anchor_links(html)
|
||||
errors = html_page_utils.validate_html(html)
|
||||
if errors:
|
||||
logger.info("Generated HTML failed validation (%s); attempting repair", errors)
|
||||
@@ -1131,6 +1229,7 @@ class ScheduledTaskService:
|
||||
fixed = repaired.get("fixed_html")
|
||||
if isinstance(fixed, str) and "<" in fixed:
|
||||
fixed_clean, _ = html_page_utils.sanitize_html(fixed)
|
||||
fixed_clean, _ = html_page_utils.constrain_same_page_anchor_links(fixed_clean)
|
||||
if not html_page_utils.validate_html(fixed_clean):
|
||||
html = fixed_clean
|
||||
except Exception:
|
||||
@@ -1158,6 +1257,13 @@ class ScheduledTaskService:
|
||||
archive_path = self._archive_html_index(run_id, html)
|
||||
|
||||
files: list[dict[str, Any]] = []
|
||||
# When both switches are on, surface the gathered Markdown file(s)
|
||||
# alongside the generated page so the run delivers both artifacts.
|
||||
if also_markdown:
|
||||
for item in gathered_files:
|
||||
name = str(item.get("name") or "") if isinstance(item, dict) else ""
|
||||
if name.lower().endswith((".md", ".markdown")):
|
||||
files.append(item)
|
||||
if archive_path is not None:
|
||||
files.append(
|
||||
{
|
||||
@@ -1483,6 +1589,14 @@ class ScheduledTaskService:
|
||||
index += 1
|
||||
|
||||
shutil.copy2(path, candidate)
|
||||
if candidate.suffix.lower() in {".md", ".markdown"}:
|
||||
try:
|
||||
original = candidate.read_text(encoding="utf-8", errors="ignore")
|
||||
constrained = _constrain_markdown_same_page_anchor_links(original)
|
||||
if constrained != original:
|
||||
candidate.write_text(constrained, encoding="utf-8", newline="")
|
||||
except Exception:
|
||||
logger.debug("Failed to constrain scheduled markdown links in %s", candidate, exc_info=True)
|
||||
return candidate
|
||||
|
||||
def _build_file_result(self, *, virtual_path: str, source_path: Path, archive_path: Path) -> dict[str, Any]:
|
||||
|
||||
@@ -36,6 +36,7 @@ _DEFAULT_ENTRY: dict = {
|
||||
"created_at": None,
|
||||
"state": STATE_ACTIVE,
|
||||
"pinned": False,
|
||||
"featured": False,
|
||||
"archived_at": None,
|
||||
"source": SOURCE_UPLOAD,
|
||||
}
|
||||
@@ -141,6 +142,11 @@ def set_pinned(name: str, pinned: bool) -> None:
|
||||
_update_entry(name, {"pinned": pinned})
|
||||
|
||||
|
||||
def set_featured(name: str, featured: bool) -> None:
|
||||
"""置顶/取消置顶。与 pinned(防归档)正交,仅影响列表排序展示。"""
|
||||
_update_entry(name, {"featured": featured})
|
||||
|
||||
|
||||
def forget(name: str) -> None:
|
||||
try:
|
||||
with _usage_lock:
|
||||
|
||||
@@ -3,9 +3,14 @@ from __future__ import annotations
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
from app.gateway.routers.agents import _load_config_extras, _write_agent_config
|
||||
from app.gateway.routers.agents import (
|
||||
_backfill_listing_cache,
|
||||
_load_config_extras,
|
||||
_write_agent_config,
|
||||
)
|
||||
from app.gateway.services import build_run_config
|
||||
from deerflow.config import paths as paths_module
|
||||
from deerflow.config.agents_config import AgentConfig
|
||||
from deerflow.persistence.agents import MemoryAgentStore
|
||||
|
||||
|
||||
@@ -198,3 +203,46 @@ def test_write_agent_config_omits_skills_when_none(isolated_paths) -> None:
|
||||
assert "skills" not in raw
|
||||
assert "model" not in raw
|
||||
assert "tool_groups" not in raw
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_backfill_listing_cache_mirrors_config_values() -> None:
|
||||
store = MemoryAgentStore()
|
||||
cfg = AgentConfig(id="a1", name="Agent", skills=["s1", "s2"], tool_groups=["g1"], model="m1")
|
||||
await _backfill_listing_cache(store, [("a1", cfg)])
|
||||
|
||||
extras = await store.get_extras_for(["a1"])
|
||||
assert extras["a1"]["skills"] == ["s1", "s2"]
|
||||
assert extras["a1"]["tool_groups"] == ["g1"]
|
||||
assert extras["a1"]["model"] == "m1"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_backfill_listing_cache_is_idempotent_and_skips_unchanged(monkeypatch) -> None:
|
||||
store = MemoryAgentStore()
|
||||
cfg = AgentConfig(id="a1", name="Agent", skills=["s1"], tool_groups=None, model=None)
|
||||
await _backfill_listing_cache(store, [("a1", cfg)])
|
||||
|
||||
# On a second run with identical config, no re-write should occur.
|
||||
calls = {"replace": 0, "set": 0}
|
||||
orig_replace = store.replace_skills
|
||||
orig_set = store.set_extras
|
||||
|
||||
async def _replace(agent_id, skills):
|
||||
calls["replace"] += 1
|
||||
await orig_replace(agent_id, skills)
|
||||
|
||||
async def _set(agent_id, *, tool_groups, model):
|
||||
calls["set"] += 1
|
||||
await orig_set(agent_id, tool_groups=tool_groups, model=model)
|
||||
|
||||
monkeypatch.setattr(store, "replace_skills", _replace)
|
||||
monkeypatch.setattr(store, "set_extras", _set)
|
||||
|
||||
await _backfill_listing_cache(store, [("a1", cfg)])
|
||||
assert calls == {"replace": 0, "set": 0}
|
||||
|
||||
# Changing the config triggers a re-write.
|
||||
cfg2 = AgentConfig(id="a1", name="Agent", skills=["s1", "s2"], tool_groups=None, model=None)
|
||||
await _backfill_listing_cache(store, [("a1", cfg2)])
|
||||
assert calls["replace"] == 1
|
||||
@@ -217,6 +217,29 @@ async def test_sql_store_filters_errors_only(sql_store):
|
||||
assert only_errors[0]["error_type"] == "APITimeoutError"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sql_store_filters_by_user_ids(sql_store):
|
||||
"""``user_ids`` (set by a fuzzy username search) matches several accounts."""
|
||||
base = datetime.now(UTC)
|
||||
for uid in ("alice", "alicia", "bob"):
|
||||
await sql_store.record(
|
||||
LlmMetricRecord(
|
||||
id=uuid.uuid4().hex,
|
||||
created_at=base,
|
||||
user_id=uid,
|
||||
duration_ms=1,
|
||||
status="success",
|
||||
)
|
||||
)
|
||||
# Fuzzy "ali" → {alice, alicia}; bob must be excluded.
|
||||
rows = await sql_store.list(LlmMetricsQuery(user_ids=["alice", "alicia"], limit=10))
|
||||
assert {r["user_id"] for r in rows} == {"alice", "alicia"}
|
||||
assert await sql_store.count(LlmMetricsQuery(user_ids=["alice", "alicia"])) == 2
|
||||
# user_ids takes precedence over a single user_id.
|
||||
rows = await sql_store.list(LlmMetricsQuery(user_id="bob", user_ids=["alice"], limit=10))
|
||||
assert {r["user_id"] for r in rows} == {"alice"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sql_store_iter_all_paginates(sql_store):
|
||||
base = datetime.now(UTC)
|
||||
|
||||
@@ -137,6 +137,17 @@ title:
|
||||
|
||||
User: {user_msg}
|
||||
Assistant: {assistant_msg}
|
||||
# 上下文压缩(对话摘要):长对话接近上下文上限时,把较早的消息总结成一段摘要,
|
||||
# 只保留最近若干条原始消息,避免撑爆模型上下文窗口。
|
||||
# - trigger:触发条件,可配多个,任意一个先满足就压缩(OR 关系)。
|
||||
# type=tokens 按 token 总数(最准,直接防溢出)
|
||||
# type=messages 按消息条数(简单直观,作兜底保险)
|
||||
# type=fraction 按模型上下文窗口的比例(如 0.8);自定义 base_url 模型可能探不到
|
||||
# 准确窗口,故此处用写死的 token 阈值更稳妥。
|
||||
# - keep:压缩后保留多少最近的上下文(同样支持 tokens / messages / fraction)。
|
||||
# - trim_tokens_to_summarize:喂给摘要模型的消息最多裁到多少 token。
|
||||
# - model_name:生成摘要用的模型(留空则用主对话模型)。
|
||||
|
||||
skills:
|
||||
path: /app/skills
|
||||
security_scan:
|
||||
@@ -163,6 +174,9 @@ auth_login:
|
||||
# 永远 401。此处 uat 域名按自签处理,先关校验;生产请置 true 或配 ca_cert_path。
|
||||
verify_ssl: false
|
||||
ca_cert_path: "" # 可选:PEM 格式 CA 包路径,设置后优先生效并启用校验
|
||||
# 日志系统入口(仅管理员可见):右下角「设置」菜单中会多出一个「日志系统」入口,
|
||||
# 点击后在新标签页打开下面的 url。enabled 关掉则不显示该入口。
|
||||
|
||||
database:
|
||||
backend: mysql
|
||||
mysql_url: $MYSQL_DATABASE_URL
|
||||
@@ -241,4 +255,22 @@ memory:
|
||||
scan_content: true # 注入/外渗模式扫描
|
||||
block_invisible_unicode: true
|
||||
# streaming_scrubber:预留字段,代码尚未实现 —— 待版本支持后取消注释。
|
||||
# streaming_scrubber: true # SSE 流前清洗 <memory-context>
|
||||
# streaming_scrubber: true # SSE 流前清洗 <memory-context>
|
||||
|
||||
log_system:
|
||||
enabled: true
|
||||
url: "https://your-log-system.example.com"
|
||||
label: "日志系统"
|
||||
|
||||
summarization:
|
||||
enabled: true
|
||||
# model_name: deepseek-chat
|
||||
trigger:
|
||||
- type: tokens
|
||||
value: 60000
|
||||
- type: messages
|
||||
value: 80
|
||||
keep:
|
||||
type: messages
|
||||
value: 20
|
||||
trim_tokens_to_summarize: 4000
|
||||
Reference in new issue
Block a user