Merge branch ji into master: 工具/技能调用统计 + 用户活动审计 + 标签多选筛选

This commit is contained in:
吉鑫升 committed 2026-05-19 09:28:44 +08:00
commit bd1efd8026
46 files changed
+3299 -315

No files matched your search

+4 -3
View File
@@ -163,6 +163,7 @@ Lead-agent middlewares are assembled in strict append order across `packages/har
6. **GuardrailMiddleware** - Pre-tool-call authorization via pluggable `GuardrailProvider` protocol (optional, if `guardrails.enabled` in config). Evaluates each tool call and returns error ToolMessage on deny. Three provider options: built-in `AllowlistProvider` (zero deps), OAP policy providers (e.g. `aport-agent-guardrails`), or custom providers. See [docs/GUARDRAILS.md](docs/GUARDRAILS.md) for setup, usage, and how to implement a provider.
7. **SandboxAuditMiddleware** - Audits sandboxed shell/file operations for security logging before tool execution continues
8. **ToolErrorHandlingMiddleware** - Converts tool exceptions into error `ToolMessage`s so the run can continue instead of aborting
8a. **ToolMetricsMiddleware** - Appended immediately after `ToolErrorHandlingMiddleware` in `_build_runtime_middlewares`, so it sits *deepest* in the `wrap_tool_call` chain. Records one `tool_call_metrics` row per tool/skill/MCP invocation — timing duration and bucketing failures into a coarse `error_category` (rate_limited / ip_blocked / auth_error / timeout / network_error / not_found / empty_result / tool_error). It catches both raised exceptions (before #8 converts them) and *self-handled* errors returned as error `ToolMessage`s (search tools that swallow HTTP 429/403). It also captures the targeted Agent Skill into `tool_call_metrics.skill_name`: the lead agent invokes a skill by `read_file`-ing its `/mnt/skills/<category>/<name>/SKILL.md` path (per the `<skill_system>` prompt block — there is no dedicated "use skill" tool), so `skill_name` is recovered by scanning every tool call's string arguments for the `skills/{public,custom}/{name}/` path shape (`_SKILL_PATH_RE`), plus the explicit `name` argument of `skill_view`/`skill_manage`/`skill_archive`. `skill_view`-specific success detection handles its swallowed failures (`Skill 'x' not found.` returned as a plain string rather than raised). Regression coverage: `tests/test_tool_metrics_middleware.py`. Writes are best-effort fire-and-forget. Surfaced by the admin `GET /api/admin/tool-metrics` router (which has a `kind=skill` filter and a per-skill `/summary` breakdown).
9. **PromptPrefixMiddleware** - Injects the admin-configured prompt prefix into the model request only. The prefix is tagged onto the first human message's `additional_kwargs['prompt_prefix']` by the Gateway (`app.gateway.services`) and prepended to content here at call time, so the persisted/displayed message stays prefix-free
10. **SummarizationMiddleware** - Context reduction when approaching token limits (optional, if enabled)
11. **TodoListMiddleware** - Task tracking with `write_todos` tool (optional, if plan_mode)
@@ -214,9 +215,9 @@ FastAPI application on port 8001 with health check at `GET /health`. Set `GATEWA
|--------|-----------|
| **Models** (`/api/models`) | `GET /` - list models; `GET /{name}` - model details |
| **MCP** (`/api/mcp`) | `GET /config` - get config; `PUT /config` - update config (saves to extensions_config.json) |
| **Skills** (`/api/skills`) | `GET /` - list skills (optional `search` fuzzy-name + `tag_id` filters; each item carries `tags` + owner-authored `detail`); `GET /{name}` - details; `PUT /{name}` - update enabled; `PUT /{name}/detail` - set the long-form `detail` text (custom skill: owner or admin; built-in/public: admin only — materializes a `skills` row on demand); `GET /{name}/content` - raw SKILL.md source, restricted to an admin or the publisher (custom-skill owner); other users get 403 and see only description + detail; `POST /install` - install from .skill archive (accepts standard optional frontmatter like `version`, `author`, `compatibility`) |
| **Skills** (`/api/skills`) | `GET /` - list skills (optional `search` fuzzy-name + repeatable `tag_ids` multi-select filter, OR-matched; each item carries `tags` + owner-authored `detail`); `GET /{name}` - details; `PUT /{name}` - update enabled; `PUT /{name}/detail` - set the long-form `detail` text (custom skill: owner or admin; built-in/public: admin only — materializes a `skills` row on demand); `GET /{name}/content` - raw SKILL.md source, restricted to an admin or the publisher (custom-skill owner); other users get 403 and see only description + detail; `POST /install` - install from .skill archive (accepts standard optional frontmatter like `version`, `author`, `compatibility`) |
| **Memory** (`/api/memory`) | `GET /` - memory data; `POST /reload` - force reload; `GET /config` - config; `GET /status` - config + data |
| **Agents** (`/api/agents`) | `GET /` - list built-in, owned, and published agents (optional `search` + `tag_id` filters; each item carries `tags`); `POST /` - create user-owned agent; `GET/PUT/DELETE /{id}` - read/update/delete by stable agent id; `GET /check` - validate non-empty display name (duplicates allowed) |
| **Agents** (`/api/agents`) | `GET /` - list built-in, owned, and published agents (optional `search` + repeatable `tag_ids` multi-select filter, OR-matched; each item carries `tags`); `POST /` - create user-owned agent; `GET/PUT/DELETE /{id}` - read/update/delete by stable agent id; `GET /check` - validate non-empty display name (duplicates allowed) |
| **Tags** (`/api/tags`) | `GET /` - list tags (optional `tag_type` filter; open to all); `POST /` - create tag (admin); `PUT /{id}` / `DELETE /{id}` - update/delete tag (admin); `GET /assignments/{target_type}/{target_id}` - tags on a resource; `PUT /assignments/{target_type}/{target_id}` - replace a resource's tags (admin). Tags are typed `agent`/`skill`/`scheduled_task` and may only tag resources of their own type. |
| **Uploads** (`/api/threads/{id}/uploads`) | `POST /` - upload files (auto-converts PDF/PPT/Excel/Word); `GET /list` - list; `DELETE /{filename}` - delete |
| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; unexpected failures are logged server-side and return a generic 500 detail |
@@ -226,7 +227,7 @@ FastAPI application on port 8001 with health check at `GET /health`. Set `GATEWA
| **Feedback** (`/api/threads/{id}/runs/{rid}/feedback`) | `PUT /` - upsert feedback; `DELETE /` - delete user feedback; `POST /` - create feedback; `GET /` - list feedback; `GET /stats` - aggregate stats; `DELETE /{fid}` - delete specific |
| **Runs** (`/api/runs`) | `POST /stream` - stateless run + SSE; `POST /wait` - stateless run + block; `GET /{rid}/messages` - paginated messages by run_id `{data, has_more}` (cursor: `after_seq`/`before_seq`); `GET /{rid}/feedback` - list feedback by run_id |
| **Thread Shares** (`/api/thread-shares`) | Per-conversation sharing with two modes (`thread_shares.mode`). **`import`**: `POST /` `{thread_id, mode}` - create (or reuse) a revocable share code; `GET /{code}/preview` + `POST /{code}/import` - copy that conversation into the caller's account (fresh `thread_id` via `app/gateway/thread_copy.py` + new `threads_meta` row + `user-data` copy; dedup on `metadata.shared_origin`). **`view`**: a public, read-only static page — see Public Shares below. Both modes: `GET /` - list my codes; `DELETE /{code}` - revoke. |
| **Public Shares** (`/api/public/shares`) | **Unauthenticated** (the `/api/public/` prefix is whitelisted in `auth_middleware._PUBLIC_PATH_PREFIXES`). `GET /{code}` - read-only conversation snapshot (title + serialized messages + artifacts) for a `view`-mode share; messages are read straight from the latest checkpoint. `GET /{code}/files/{path}` - serves an **image** referenced by that conversation (images only — non-image paths 404; SVG forced to download). Missing/revoked/non-`view` codes all return 404. |
| **Public Shares** (`/api/public/shares`) | **Unauthenticated** (the `/api/public/` prefix is whitelisted in `auth_middleware._PUBLIC_PATH_PREFIXES`). `GET /{code}` - read-only conversation snapshot (title + serialized messages + artifacts) for a `view`-mode share; messages are read straight from the latest checkpoint. `GET /{code}/files/{path}` - serves a file referenced by that conversation (images, Markdown, text, PDF, …) so the shared page's links are openable; path resolution is confined to the owner's thread directory. Active content (`text/html`, `application/xhtml+xml`, `image/svg+xml`) is forced to download (`_public_share_disposition`); everything else is served inline. Missing/revoked/non-`view` codes all return 404. |
| **Scheduled Tasks** (`/api/scheduled-tasks`) | Per-user cron-like agent tasks. `GET/POST /` - list/create owned tasks; `PATCH/DELETE /{id}` - update/delete (owner); `POST /{id}/{pause,resume,trigger}`; `GET /{id}/runs`, `GET /runs/{rid}`, run-file preview/edit/download; `GET /runs/{rid}/messages` - agent messages for the run's underlying agent run (used to inspect a stuck/running run; authorized via `_load_run`); `DELETE /runs/{rid}` - owner-scoped, with an admin fallback so admins can delete any user's run; publish/subscribe under `/{id}/{publish,unpublish,subscribe}` and `GET /published`. **Admin (cross-user):** `GET /admin/all` - every task; `PATCH/DELETE /admin/{id}` - edit/delete any task; `POST /admin/{id}/trigger`; `GET /admin/{id}/runs`. Admin routes require `system_role == "admin"`. |
Proxied through nginx: `/api/langgraph/*` → LangGraph, all other `/api/*` → Gateway.
@@ -23,6 +23,7 @@ from app.gateway.csrf_middleware import CSRFMiddleware
from app.gateway.deps import langgraph_runtime
from app.gateway.routers import (
ai_writing,
admin_users,
agents,
artifacts,
assistants_compat,
@@ -42,6 +43,7 @@ from app.gateway.routers import (
system_settings,
tags,
thread_runs,
tool_metrics,
thread_shares,
threads,
uploads,
@@ -256,7 +258,7 @@ async def _run_db_migrations() -> None:
logger.info("Persistence backend=memory — skipping Alembic migrations")
return
db_url = engine.url.render_as_string(hide_password=False).replace("%", "%%")
db_url = engine.url.render_as_string(hide_password=False).replace("%", "%%")
migrations_dir = str(Path(_persistence_pkg.__file__).parent / "migrations")
def _run_sync() -> None:
@@ -689,6 +691,12 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
# Admin LLM metrics audit + export
app.include_router(llm_metrics.router)
# Admin window into other users' threads / messages / memory
app.include_router(admin_users.router)
# Admin tool / skill call metrics audit + export
app.include_router(tool_metrics.router)
# Channels API is mounted at /api/channels
app.include_router(channels.router)
@@ -97,6 +97,10 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
app.state.llm_metrics_store = make_llm_metrics_store(sf)
from deerflow.persistence.tool_metrics import make_tool_metrics_store
app.state.tool_metrics_store = make_tool_metrics_store(sf)
from deerflow.persistence.agents import make_agent_store
app.state.agent_store = make_agent_store(sf)
@@ -202,6 +206,11 @@ def get_llm_metrics_store(request: Request):
return getattr(request.app.state, "llm_metrics_store", None)
def get_tool_metrics_store(request: Request):
"""Return the tool-call metrics store, or None when not configured."""
return getattr(request.app.state, "tool_metrics_store", None)
def get_agent_store(request: Request) -> AgentStore:
"""Return the custom-agent store."""
val = getattr(request.app.state, "agent_store", None)
@@ -0,0 +1,162 @@
"""Admin-only window into another user's activity.
Lets an administrator inspect any user's conversation history (threads + the
Q&A messages inside them) and their accumulated memory (global + per-agent
buckets). All endpoints require ``system_role == "admin"``; they are strictly
read-only.
The underlying stores normally auto-scope every query to the *calling* user
via a contextvar. These endpoints pass an explicit ``user_id`` so the admin
reads the *target* user's data instead — which is exactly what those store
methods' ``user_id`` parameter is designed for.
"""
from __future__ import annotations
import logging
from fastapi import APIRouter, HTTPException, Query, Request
from pydantic import BaseModel, Field
from app.gateway.deps import (
get_checkpointer,
get_current_user_from_request,
get_thread_store,
)
from deerflow.agents.memory.updater import get_memory_data
from deerflow.config.agents_config import AGENT_NAME_PATTERN
from deerflow.config.paths import get_paths
from deerflow.runtime import serialize_channel_values
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/admin/users", tags=["admin-users"])
async def _require_admin(request: Request) -> None:
"""Reject non-admin callers. In no-auth mode there is no user → allow."""
try:
user = await get_current_user_from_request(request)
except HTTPException:
raise
if user is None:
return
if getattr(user, "system_role", None) != "admin":
raise HTTPException(status_code=403, detail="Admin only")
# ── Threads ────────────────────────────────────────────────────────────────
class AdminThreadItem(BaseModel):
thread_id: str
display_name: str = ""
status: str = "idle"
created_at: str = ""
updated_at: str = ""
@router.get("/{user_id}/threads", response_model=list[AdminThreadItem])
async def list_user_threads(
user_id: str,
request: Request,
limit: int = Query(default=100, ge=1, le=500),
offset: int = Query(default=0, ge=0),
) -> list[AdminThreadItem]:
"""List one user's conversation threads, newest first."""
await _require_admin(request)
repo = get_thread_store(request)
rows = await repo.search(limit=limit, offset=offset, user_id=user_id)
return [
AdminThreadItem(
thread_id=r["thread_id"],
display_name=r.get("display_name") or "",
status=r.get("status", "idle"),
created_at=str(r.get("created_at") or ""),
updated_at=str(r.get("updated_at") or ""),
)
for r in rows
]
@router.get("/{user_id}/threads/{thread_id}/messages", response_model=list[dict])
async def list_user_thread_messages(
user_id: str,
thread_id: str,
request: Request,
limit: int = Query(default=400, ge=1, le=1000),
) -> list[dict]:
"""Return the conversation messages of one of the user's threads.
Read straight from the LangGraph checkpoint — the same source the chat UI
renders — rather than the optional ``run_events`` audit log, which may be
unconfigured (``run_events.backend=memory``) or empty for older threads.
"""
await _require_admin(request)
repo = get_thread_store(request)
# ``user_id=None`` bypasses the owner filter so the admin can fetch any row;
# we then assert the thread really belongs to the target user.
thread = await repo.get(thread_id, user_id=None)
if thread is None:
raise HTTPException(status_code=404, detail="Thread not found")
owner = thread.get("user_id")
if owner is not None and owner != user_id:
raise HTTPException(status_code=404, detail="Thread does not belong to this user")
checkpointer = get_checkpointer(request)
config = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
try:
checkpoint_tuple = await checkpointer.aget_tuple(config)
except Exception as exc: # noqa: BLE001 - surface a clean error
logger.error("Failed to read checkpoint for thread %s: %s", thread_id, exc, exc_info=True)
raise HTTPException(status_code=500, detail="Failed to read thread state") from exc
if checkpoint_tuple is None:
return []
checkpoint = getattr(checkpoint_tuple, "checkpoint", {}) or {}
channel_values = checkpoint.get("channel_values", {}) if isinstance(checkpoint, dict) else {}
values = serialize_channel_values(channel_values)
messages = values.get("messages", [])
if not isinstance(messages, list):
return []
# Keep the most recent ``limit`` messages — long threads stay bounded.
return messages[-limit:]
# ── Memory ─────────────────────────────────────────────────────────────────
class AdminMemoryBucket(BaseModel):
name: str = Field(..., description="Bucket name: 'global' or a custom-agent name")
data: dict = Field(default_factory=dict, description="Raw memory data for the bucket")
class AdminMemoryResponse(BaseModel):
user_id: str
buckets: list[AdminMemoryBucket] = Field(default_factory=list)
@router.get("/{user_id}/memory", response_model=AdminMemoryResponse)
async def get_user_memory(user_id: str, request: Request) -> AdminMemoryResponse:
"""Return every memory bucket (global + per-agent) for one user."""
await _require_admin(request)
buckets: list[AdminMemoryBucket] = [
AdminMemoryBucket(name="global", data=get_memory_data(None, user_id=user_id))
]
agents_root = get_paths().user_dir(user_id) / "agents"
if agents_root.is_dir():
for entry in sorted(agents_root.iterdir(), key=lambda p: p.name.lower()):
if not entry.is_dir() or not (entry / "memory.json").exists():
continue
if not AGENT_NAME_PATTERN.match(entry.name):
continue
try:
data = get_memory_data(entry.name, user_id=user_id)
except Exception:
logger.warning("Failed to read memory bucket %s for user", entry.name)
continue
buckets.append(AdminMemoryBucket(name=entry.name, data=data))
return AdminMemoryResponse(user_id=user_id, buckets=buckets)
@@ -244,7 +244,7 @@ async def _get_editable_or_raise(
async def list_agents(
request: Request,
search: str | None = Query(default=None, description="Case-insensitive fuzzy match on agent name"),
tag_id: str | None = Query(default=None, description="Keep only agents carrying this tag"),
tag_ids: list[str] | None = Query(default=None, description="Keep agents carrying any of these tags (OR)"),
) -> AgentsListResponse:
_require_agents_api_enabled()
@@ -262,8 +262,9 @@ async def list_agents(
if search:
needle = search.strip().lower()
responses = [r for r in responses if needle in r.name.lower()]
if tag_id:
responses = [r for r in responses if any(t.get("id") == tag_id for t in r.tags)]
if tag_ids:
wanted = set(tag_ids)
responses = [r for r in responses if any(t.get("id") in wanted for t in r.tags)]
return AgentsListResponse(agents=responses)
except HTTPException:
raise
@@ -197,7 +197,7 @@ async def get_scheduler_thread(request: Request) -> dict:
async def list_scheduled_tasks(
request: Request,
search: str | None = Query(default=None, description="Case-insensitive fuzzy match on task name"),
tag_id: str | None = Query(default=None, description="Keep only tasks carrying this tag"),
tag_ids: list[str] | None = Query(default=None, description="Keep tasks carrying any of these tags (OR)"),
) -> list[dict]:
service = get_scheduled_task_service()
user_id = await _user_id(request)
@@ -211,8 +211,9 @@ async def list_scheduled_tasks(
if search:
needle = search.strip().lower()
tasks = [t for t in tasks if needle in str(t.get("name", "")).lower()]
if tag_id:
tasks = [t for t in tasks if any(tg.get("id") == tag_id for tg in t.get("tags", []))]
if tag_ids:
wanted = set(tag_ids)
tasks = [t for t in tasks if any(tg.get("id") in wanted for tg in t.get("tags", []))]
return tasks
@@ -236,7 +236,7 @@ async def _attach_and_filter_tags(
tag_store: TagStore,
responses: list[SkillResponse],
search: str | None,
tag_id: str | None,
tag_ids: list[str] | None,
) -> list[SkillResponse]:
"""Bulk-attach tags to skill responses, then apply name/tag filters."""
assignments = await tag_store.list_assignments("skill", [r.name for r in responses])
@@ -245,8 +245,9 @@ async def _attach_and_filter_tags(
if search:
needle = search.strip().lower()
responses = [r for r in responses if needle in r.name.lower()]
if tag_id:
responses = [r for r in responses if any(t.get("id") == tag_id for t in r.tags)]
if tag_ids:
wanted = set(tag_ids)
responses = [r for r in responses if any(t.get("id") in wanted for t in r.tags)]
return responses
@@ -262,7 +263,7 @@ async def list_skills(
skill_store: SkillStore = Depends(get_skill_store),
tag_store: TagStore = Depends(get_tag_store),
search: str | None = Query(default=None, description="Case-insensitive fuzzy match on skill name"),
tag_id: str | None = Query(default=None, description="Keep only skills carrying this tag"),
tag_ids: list[str] | None = Query(default=None, description="Keep skills carrying any of these tags (OR)"),
) -> SkillsListResponse:
try:
user_id = _current_user_id(request)
@@ -281,7 +282,7 @@ async def list_skills(
if record is None:
continue
responses.append(_skill_to_response(skill, record))
responses = await _attach_and_filter_tags(tag_store, responses, search, tag_id)
responses = await _attach_and_filter_tags(tag_store, responses, search, tag_ids)
return SkillsListResponse(skills=responses)
except Exception as e:
logger.error(f"Failed to load skills: {e}", exc_info=True)
@@ -294,20 +294,38 @@ async def get_public_share_snapshot(
)
# Content types that can execute scripts in the app origin — always served as
# a download attachment so a shared file can never run as a top-level document.
_PUBLIC_SHARE_ACTIVE_TYPES = {"text/html", "application/xhtml+xml", "image/svg+xml"}
def _public_share_disposition(mime_type: str) -> str:
"""Pick a Content-Disposition for a public-share file.
Active content (HTML/XHTML/SVG) is forced to ``attachment`` so it cannot
execute as a top-level document; everything else (images, Markdown, text,
PDF, …) is served ``inline`` so viewers can open it directly.
"""
return "attachment" if mime_type in _PUBLIC_SHARE_ACTIVE_TYPES else "inline"
@public_router.get(
"/{share_code}/files/{path:path}",
summary="Public image attachment from a view-shared conversation",
summary="Public file attachment from a view-shared conversation",
)
async def get_public_share_file(
share_code: str,
path: str,
share_store: ThreadShareStore = Depends(get_thread_share_store),
) -> FileResponse:
"""Serve an image referenced by a view-shared conversation.
"""Serve a file referenced by a view-shared conversation.
Restricted to images only — the public share page renders inline images,
but the endpoint must not become a way to read arbitrary thread files.
Non-image paths return 404 just like a missing file.
A ``view`` share is a public read-only page, so the files its conversation
references (images, Markdown, text, PDF, …) must be openable. Path
resolution is confined to the owner's thread directory, so this cannot read
arbitrary files. Active content (HTML/XHTML/SVG) is forced to download so it
can never execute as a top-level document in the app origin; everything
else is served inline.
"""
share = await share_store.get(share_code)
if share is None or share.get("revoked") or share.get("mode") != "view":
@@ -324,17 +342,10 @@ async def get_public_share_file(
if not actual_path.exists() or not actual_path.is_file():
raise HTTPException(status_code=404, detail="File not found")
mime_type, _ = mimetypes.guess_type(actual_path.name)
if not mime_type or not mime_type.startswith("image/"):
# Only images are public; everything else is treated as not found.
raise HTTPException(status_code=404, detail="File not found")
# SVG can carry scripts — serve it as a download so it can never execute
# as a top-level document in the app origin. Raster images render inline.
is_svg = mime_type == "image/svg+xml"
mime_type = mimetypes.guess_type(actual_path.name)[0] or "application/octet-stream"
return FileResponse(
actual_path,
media_type=mime_type,
content_disposition_type="attachment" if is_svg else "inline",
content_disposition_type=_public_share_disposition(mime_type),
headers={"Cache-Control": "public, max-age=300"},
)
@@ -0,0 +1,380 @@
"""Admin-only tool / skill call metrics: list + Excel export.
Companion to ``llm_metrics`` — where that audits every LLM provider call, this
audits every *tool* call (built-in tools, skills, MCP tools, subagent calls),
including the failures: retrieval errors, blocked/rate-limited IPs, timeouts,
auth problems. Failures are bucketed into a coarse ``error_category`` so an
admin can scan for systemic issues.
"""
from __future__ import annotations
import logging
from datetime import datetime
from io import BytesIO
from typing import Any
from urllib.parse import quote
from fastapi import APIRouter, HTTPException, Query, Request
from fastapi.responses import StreamingResponse
from pydantic import BaseModel, Field
from app.gateway.deps import get_local_provider, get_optional_user_from_request, get_tool_metrics_store
from deerflow.persistence.tool_metrics.base import ToolMetricsQuery, ToolMetricsStore
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/admin/tool-metrics", tags=["tool-metrics"])
_VALID_CATEGORIES = (
"rate_limited",
"ip_blocked",
"auth_error",
"timeout",
"network_error",
"not_found",
"empty_result",
"tool_error",
)
def _kind_is_skill(kind: str | None) -> bool:
"""``kind=skill`` keeps only rows that target a specific Agent Skill."""
return kind == "skill"
class ToolMetricResponse(BaseModel):
id: str
created_at: str | None = None
user_id: str | None = None
user_name: str | None = Field(
default=None,
description="Local part (before @) of the user's email, resolved from user_id",
)
thread_id: str | None = None
run_id: str | None = None
agent_name: str | None = None
tool_name: str | None = None
skill_name: str | None = Field(
default=None,
description="The Agent Skill this call targets (skill tools only)",
)
duration_ms: int = 0
status: str = "success"
error_category: str | None = None
error_message: str | None = None
class ToolMetricsListResponse(BaseModel):
items: list[ToolMetricResponse]
total: int = Field(..., description="Total rows matching the filter, ignoring pagination")
limit: int
offset: int
async def _require_admin(request: Request) -> None:
user = await get_optional_user_from_request(request)
if user is None:
return
if getattr(user, "system_role", None) != "admin":
raise HTTPException(status_code=403, detail="Tool metrics admin view is admin-only")
def _require_store(request: Request) -> ToolMetricsStore:
store = get_tool_metrics_store(request)
if store is None:
raise HTTPException(status_code=503, detail="Tool metrics store is not configured")
return store
def _parse_dt(value: str | None) -> datetime | None:
if not value:
return None
try:
return datetime.fromisoformat(value.replace("Z", "+00:00"))
except ValueError as exc:
raise HTTPException(status_code=422, detail=f"Invalid datetime: {value!r} ({exc})") from exc
def _email_local_part(email: str | None) -> str | None:
if not email:
return None
at = email.find("@")
return email[:at] if at > 0 else email
async def _resolve_user_name_to_id(name: str | None) -> str | None:
if not name:
return None
try:
provider = get_local_provider()
user = await provider.find_user_by_email_local_part(name)
except Exception:
return None
return str(user.id) if user is not None else None
async def _attach_usernames(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""Resolve each row's ``user_id`` UUID to a friendly ``user_name``."""
if not rows:
return rows
user_ids = {row["user_id"] for row in rows if isinstance(row, dict) and row.get("user_id")}
if not user_ids:
return rows
try:
provider = get_local_provider()
except Exception:
provider = None
email_by_id: dict[str, str] = {}
if provider is not None:
for uid in user_ids:
try:
user = await provider.get_user(uid)
except Exception:
user = None
if user is not None and user.email:
email_by_id[uid] = user.email
for row in rows:
if isinstance(row, dict) and row.get("user_id"):
email = email_by_id.get(row["user_id"])
row["user_name"] = _email_local_part(email) or row["user_id"]
return rows
def _build_query(
*,
user_id: str | None,
tool_name: str | None,
skill_only: bool = False,
skill_name: str | None = None,
status: str | None,
error_category: str | None,
since: str | None,
until: str | None,
limit: int,
offset: int,
) -> ToolMetricsQuery:
return ToolMetricsQuery(
user_id=user_id or None,
tool_name=tool_name or None,
skill_only=skill_only,
skill_name=skill_name or None,
status=status or None,
error_category=error_category or None,
since=_parse_dt(since),
until=_parse_dt(until),
limit=limit,
offset=offset,
)
@router.get("", response_model=ToolMetricsListResponse)
async def list_tool_metrics(
request: Request,
user_id: str | None = Query(default=None),
user_name: str | None = Query(default=None, description="Username (email local part) to filter by"),
tool_name: str | None = Query(default=None),
kind: str | None = Query(default=None, description="Coarse call kind: 'skill' keeps only Agent-Skill invocations"),
skill_name: str | None = Query(default=None, description="Filter to one specific Agent Skill by name"),
status: str | None = Query(default=None, pattern="^(success|error)$"),
error_category: str | None = Query(default=None, description="Coarse failure bucket"),
since: str | None = Query(default=None, description="ISO 8601 datetime, inclusive lower bound"),
until: str | None = Query(default=None, description="ISO 8601 datetime, inclusive upper bound"),
limit: int = Query(default=50, ge=1, le=500),
offset: int = Query(default=0, ge=0),
) -> ToolMetricsListResponse:
await _require_admin(request)
store = _require_store(request)
if error_category and error_category not in _VALID_CATEGORIES:
raise HTTPException(status_code=422, detail=f"Unknown error_category: {error_category!r}")
if not user_id and user_name:
resolved = await _resolve_user_name_to_id(user_name)
if resolved is None:
return ToolMetricsListResponse(items=[], total=0, limit=limit, offset=offset)
user_id = resolved
query = _build_query(
user_id=user_id,
tool_name=tool_name,
skill_only=_kind_is_skill(kind),
skill_name=skill_name,
status=status,
error_category=error_category,
since=since,
until=until,
limit=limit,
offset=offset,
)
rows = await store.list(query)
total = await store.count(query)
await _attach_usernames(rows)
return ToolMetricsListResponse(
items=[ToolMetricResponse(**row) for row in rows],
total=total,
limit=limit,
offset=offset,
)
class SkillStat(BaseModel):
skill_name: str
total: int
success: int
error: int
class ToolMetricsSummaryResponse(BaseModel):
total: int
error_total: int
by_category: dict[str, int] = Field(default_factory=dict)
by_skill: list[SkillStat] = Field(
default_factory=list,
description="Per-skill invocation counts — answers 'was skill X called, and did it succeed'",
)
@router.get("/summary", response_model=ToolMetricsSummaryResponse)
async def tool_metrics_summary(
request: Request,
user_id: str | None = Query(default=None),
user_name: str | None = Query(default=None),
tool_name: str | None = Query(default=None),
kind: str | None = Query(default=None),
skill_name: str | None = Query(default=None),
since: str | None = Query(default=None),
until: str | None = Query(default=None),
) -> ToolMetricsSummaryResponse:
"""Aggregate counts: total, errors, per-category breakdown, and per-skill stats."""
await _require_admin(request)
store = _require_store(request)
if not user_id and user_name:
resolved = await _resolve_user_name_to_id(user_name)
if resolved is None:
return ToolMetricsSummaryResponse(total=0, error_total=0, by_category={}, by_skill=[])
user_id = resolved
skill_only = _kind_is_skill(kind)
def _q(*, status: str | None = None, error_category: str | None = None) -> ToolMetricsQuery:
return _build_query(
user_id=user_id,
tool_name=tool_name,
skill_only=skill_only,
skill_name=skill_name,
status=status,
error_category=error_category,
since=since,
until=until,
limit=1,
offset=0,
)
total = await store.count(_q())
error_total = await store.count(_q(status="error"))
by_category: dict[str, int] = {}
for category in _VALID_CATEGORIES:
count = await store.count(_q(status="error", error_category=category))
if count:
by_category[category] = count
by_skill = [SkillStat(**row) for row in await store.skill_breakdown(_q())]
return ToolMetricsSummaryResponse(
total=total, error_total=error_total, by_category=by_category, by_skill=by_skill
)
def _format_cell(value: Any) -> Any:
if value is None:
return ""
if isinstance(value, datetime):
return value.replace(tzinfo=None) if value.tzinfo else value
return value
@router.get("/export")
async def export_tool_metrics(
request: Request,
user_id: str | None = Query(default=None),
user_name: str | None = Query(default=None),
tool_name: str | None = Query(default=None),
kind: str | None = Query(default=None),
skill_name: str | None = Query(default=None),
status: str | None = Query(default=None, pattern="^(success|error)$"),
error_category: str | None = Query(default=None),
since: str | None = Query(default=None),
until: str | None = Query(default=None),
):
"""Stream the full filtered metric set as an ``.xlsx`` file."""
await _require_admin(request)
store = _require_store(request)
if not user_id and user_name:
resolved = await _resolve_user_name_to_id(user_name)
user_id = resolved if resolved is not None else "__no_such_user__"
try:
from openpyxl import Workbook
except ImportError as exc:
raise HTTPException(status_code=500, detail=f"openpyxl is not installed: {exc}") from exc
query = _build_query(
user_id=user_id,
tool_name=tool_name,
skill_only=_kind_is_skill(kind),
skill_name=skill_name,
status=status,
error_category=error_category,
since=since,
until=until,
limit=1,
offset=0,
)
workbook = Workbook(write_only=True)
sheet = workbook.create_sheet("tool_call_metrics")
columns = [
"id",
"created_at",
"status",
"error_category",
"user_name",
"user_id",
"thread_id",
"run_id",
"agent_name",
"tool_name",
"skill_name",
"duration_ms",
"error_message",
]
sheet.append(columns)
buffered: list[dict[str, Any]] = []
FLUSH_AT = 200
async def flush() -> int:
if not buffered:
return 0
await _attach_usernames(buffered)
for row in buffered:
sheet.append([_format_cell(row.get(col)) for col in columns])
n = len(buffered)
buffered.clear()
return n
count = 0
async for row in store.iter_all(query):
buffered.append(row)
if len(buffered) >= FLUSH_AT:
count += await flush()
count += await flush()
buffer = BytesIO()
workbook.save(buffer)
buffer.seek(0)
logger.info("Exported %d tool_call_metrics rows", count)
timestamp = datetime.utcnow().strftime("%Y%m%d-%H%M%S")
filename = f"tool_call_metrics-{timestamp}.xlsx"
headers = {
"Content-Disposition": f"attachment; filename=\"{filename}\"; filename*=UTF-8''{quote(filename)}",
}
return StreamingResponse(
buffer,
media_type="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
headers=headers,
)
@@ -131,6 +131,14 @@ def _build_runtime_middlewares(
middlewares.append(SandboxAuditMiddleware())
middlewares.append(ToolErrorHandlingMiddleware())
# Tool / skill call metrics — appended last so it sits *deepest* in the
# wrap_tool_call chain: its handler is the raw tool, so it observes the
# unconverted exception (before ToolErrorHandlingMiddleware turns it into
# an error ToolMessage) and the raw return value.
from deerflow.agents.middlewares.tool_metrics_middleware import ToolMetricsMiddleware
middlewares.append(ToolMetricsMiddleware())
return middlewares
@@ -0,0 +1,319 @@
"""Middleware that records per-tool-call metrics for the admin audit page.
Every tool / skill invocation ends with one row in ``tool_call_metrics``:
- Successes record the wall-clock duration.
- Failures record a coarse ``error_category`` plus a one-line message.
Two failure shapes are captured:
1. **Raised exceptions** — the tool blew up. We record, then re-raise so the
outer :class:`ToolErrorHandlingMiddleware` still converts it into an error
``ToolMessage`` for the model.
2. **Self-handled errors** — search/retrieval tools (Serper, Tavily, Jina, …)
catch their own HTTP 429/403/timeout and *return* an error payload instead
of raising. Those never reach an ``except`` block, so we also inspect the
returned ``ToolMessage``: an ``status == "error"`` message, or content that
looks like an error string, is bucketed too.
This middleware is appended **after** ``ToolErrorHandlingMiddleware`` so it
sits deepest in the ``wrap_tool_call`` chain — its handler is the raw tool, so
it observes the unconverted exception and the raw return value.
Writes go through a best-effort store: a DB outage logs a warning and the
agent runtime keeps running.
"""
from __future__ import annotations
import asyncio
import logging
import re
import time
import uuid
from collections.abc import Awaitable, Callable
from datetime import UTC, datetime
from typing import Any, override
from langchain.agents import AgentState
from langchain.agents.middleware import AgentMiddleware
from langchain_core.messages import ToolMessage
from langgraph.errors import GraphBubbleUp
from langgraph.prebuilt.tool_node import ToolCallRequest
from langgraph.types import Command
from deerflow.persistence.engine import get_session_factory
from deerflow.persistence.tool_metrics import ToolMetricRecord, make_tool_metrics_store
from deerflow.persistence.tool_metrics.base import ToolMetricsStore
from deerflow.runtime.user_context import get_effective_user_id
logger = logging.getLogger(__name__)
def _now() -> datetime:
return datetime.now(UTC)
def _trim(text: str, limit: int = 500) -> str:
text = (text or "").strip()
return text if len(text) <= limit else text[: limit - 3] + "..."
def _classify_error(text: str) -> str:
"""Bucket an error string into a coarse, admin-friendly category."""
t = (text or "").lower()
if any(k in t for k in ("429", "rate limit", "ratelimit", "too many request", "quota")):
return "rate_limited"
if any(k in t for k in ("403", "forbidden", "ip ban", "ip block", "blocked", "access denied", "banned")):
return "ip_blocked"
if any(k in t for k in ("401", "unauthorized", "api key", "apikey", "invalid key", "authentication", "credential")):
return "auth_error"
if any(k in t for k in ("timeout", "timed out", "deadline")):
return "timeout"
if any(k in t for k in ("connecterror", "connection error", "connection refused", "network", "dns", "unreachable", "ssl", "proxy")):
return "network_error"
if any(k in t for k in ("404", "not found")):
return "not_found"
if any(k in t for k in ("no results", "no result", "0 results", "empty result")):
return "empty_result"
return "tool_error"
def _content_text(content: Any) -> str:
"""Flatten a tool result's content (string or block list) into text."""
if content is None:
return ""
if isinstance(content, str):
return content
if isinstance(content, list):
parts: list[str] = []
for block in content:
if isinstance(block, str):
parts.append(block)
elif isinstance(block, dict):
text = block.get("text")
if isinstance(text, str):
parts.append(text)
return " ".join(parts)
return str(content)
def _looks_like_error(text: str) -> bool:
"""Heuristically detect a self-handled error payload returned as content.
Deliberately conservative — only an explicit ``Error:`` prefix or a JSON
object whose first key is ``error`` counts, so a normal result that merely
mentions the word "error" is not misclassified.
"""
head = (text or "").lstrip()[:64].lower()
return head.startswith("error:") or head.startswith('{"error"') or head.startswith("{'error'")
# Skill tools whose ``name`` argument names one specific Agent Skill directly.
_SKILL_NAME_ARG_TOOLS = {"skill_view", "skill_manage", "skill_archive"}
# Matches a skill directory embedded in any tool-argument path. Skills live
# under ``skills/{public,custom}/{name}/`` — both inside the sandbox
# (``/mnt/skills/...``) and on the host. The lead agent's system prompt tells
# it to load a skill by calling ``read_file`` on that SKILL.md path (it does
# NOT call a dedicated "use skill" tool), so the skill name has to be recovered
# from the path argument of whatever tool touched it (read_file / bash / ls /
# str_replace / write_file ...).
_SKILL_PATH_RE = re.compile(r"[\\/]skills[\\/](?:public|custom)[\\/]([^\\/]+)(?:[\\/]|$)")
def _skill_name_from_call(tool_name: str | None, tool_call: Any) -> str | None:
"""Recover the Agent Skill a tool call targets, or ``None``.
Two detection paths:
1. ``skill_view`` / ``skill_manage`` / ``skill_archive`` take an explicit
``name`` argument.
2. Every other tool references a skill by *path*. The canonical skill
invocation is ``read_file('/mnt/skills/<cat>/<name>/SKILL.md')``; helper
files and ``bash`` script runs hit the same directory. So we scan the
string arguments for the ``skills/{public,custom}/{name}/`` shape.
"""
if not isinstance(tool_call, dict):
return None
args = tool_call.get("args")
if not isinstance(args, dict):
return None
# Case 1 — explicit ``name`` argument on the skill tools.
if tool_name in _SKILL_NAME_ARG_TOOLS:
raw = args.get("name")
if isinstance(raw, str) and raw.strip():
return raw.strip()
# Case 2 — a skill path inside any string argument.
for value in args.values():
if isinstance(value, str):
match = _SKILL_PATH_RE.search(value)
if match:
return match.group(1)
return None
def _evaluate_skill_view(text: str) -> tuple[str, str | None, str | None]:
"""Classify a ``skill_view`` return.
``skill_view`` never raises — on failure it returns a plain string
(``Skill 'x' not found.`` / ``Error reading skill 'x': ...``); on success
it returns the SKILL.md content. So success/failure must be read off the
returned text, not an exception or ``status`` flag.
"""
head = (text or "").lstrip()
low = head[:200].lower()
if head.startswith("Skill '") and "not found" in low:
return "error", "not_found", _trim(text)
if low.startswith("error reading skill"):
return "error", _classify_error(text), _trim(text)
return "success", None, None
def _evaluate_result(result: Any, tool_name: str | None) -> tuple[str, str | None, str | None]:
"""Return ``(status, error_category, error_message)`` for a tool return."""
if isinstance(result, ToolMessage):
text = _content_text(result.content)
if getattr(result, "status", None) == "error":
return "error", _classify_error(text), _trim(text)
# skill_view swallows its own failures into a plain string return.
if tool_name == "skill_view":
return _evaluate_skill_view(text)
if _looks_like_error(text):
return "error", _classify_error(text), _trim(text)
return "success", None, None
# Command (control flow) or any other shape — treat as a successful call.
return "success", None, None
class ToolMetricsMiddleware(AgentMiddleware[AgentState]):
"""Wrap each tool call to time it and persist a metric row."""
def __init__(self, *, store: ToolMetricsStore | None = None, **kwargs: Any) -> None:
super().__init__(**kwargs)
self._store_override = store
# -- store / context ----------------------------------------------------
def _store(self) -> ToolMetricsStore | None:
if self._store_override is not None:
return self._store_override
try:
sf = get_session_factory()
except Exception:
return None
return make_tool_metrics_store(sf)
@staticmethod
def _runtime_lookup(request: ToolCallRequest, keys: tuple[str, ...]) -> str | None:
runtime = getattr(request, "runtime", None)
if runtime is None:
return None
context = getattr(runtime, "context", None)
if isinstance(context, dict):
for key in keys:
value = context.get(key)
if isinstance(value, str) and value:
return value
config = getattr(runtime, "config", None)
if isinstance(config, dict):
configurable = config.get("configurable") if isinstance(config.get("configurable"), dict) else {}
metadata = config.get("metadata") if isinstance(config.get("metadata"), dict) else {}
for bucket in (configurable, metadata):
for key in keys:
value = bucket.get(key)
if isinstance(value, str) and value:
return value
return None
def _build_base_record(self, request: ToolCallRequest) -> ToolMetricRecord:
tool_call = getattr(request, "tool_call", None) or {}
tool_name = tool_call.get("name") if isinstance(tool_call, dict) else None
tool_name = str(tool_name) if tool_name else None
return ToolMetricRecord(
id=uuid.uuid4().hex,
created_at=_now(),
user_id=get_effective_user_id(),
thread_id=self._runtime_lookup(request, ("thread_id",)),
run_id=self._runtime_lookup(request, ("run_id", "langgraph_run_id")),
agent_name=self._runtime_lookup(request, ("agent_name", "agent_id")),
skill_name=_skill_name_from_call(tool_name, tool_call),
tool_name=tool_name,
)
# -- finalize -----------------------------------------------------------
def _finalize_result(self, record: ToolMetricRecord, result: Any, started: float) -> None:
record.duration_ms = int(max(0.0, time.monotonic() - started) * 1000)
record.status, record.error_category, record.error_message = _evaluate_result(
result, record.tool_name
)
def _finalize_error(self, record: ToolMetricRecord, exc: BaseException, started: float) -> None:
record.duration_ms = int(max(0.0, time.monotonic() - started) * 1000)
record.status = "error"
name = exc.__class__.__name__
combined = f"{name}: {(str(exc) or name).strip()}"
record.error_category = _classify_error(combined)
record.error_message = _trim(combined)
def _flush(self, record: ToolMetricRecord) -> None:
store = self._store()
if store is None:
return
try:
try:
loop = asyncio.get_running_loop()
except RuntimeError:
loop = None
if loop is not None and loop.is_running():
# Fire-and-forget — never let metric IO block the tool path.
loop.create_task(store.record(record))
else:
asyncio.run(store.record(record))
except Exception:
logger.warning("ToolMetricsMiddleware: failed to flush metric %s", record.id, exc_info=True)
# -- wrap ---------------------------------------------------------------
@override
def wrap_tool_call(
self,
request: ToolCallRequest,
handler: Callable[[ToolCallRequest], ToolMessage | Command],
) -> ToolMessage | Command:
record = self._build_base_record(request)
started = time.monotonic()
try:
result = handler(request)
except GraphBubbleUp:
# LangGraph control flow — not a tool failure.
raise
except Exception as exc:
self._finalize_error(record, exc, started)
self._flush(record)
raise
self._finalize_result(record, result, started)
self._flush(record)
return result
@override
async def awrap_tool_call(
self,
request: ToolCallRequest,
handler: Callable[[ToolCallRequest], Awaitable[ToolMessage | Command]],
) -> ToolMessage | Command:
record = self._build_base_record(request)
started = time.monotonic()
try:
result = await handler(request)
except GraphBubbleUp:
raise
except Exception as exc:
self._finalize_error(record, exc, started)
self._flush(record)
raise
self._finalize_result(record, result, started)
self._flush(record)
return result
@@ -0,0 +1,58 @@
"""Create tool_call_metrics table.
Revision ID: 20260518_07
Revises: 20260518_06
Create Date: 2026-05-18
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
from sqlalchemy import inspect
revision = "20260518_07"
down_revision = "20260518_06"
branch_labels = None
depends_on = None
def _has_table(table_name: str) -> bool:
inspector = inspect(op.get_bind())
return table_name in inspector.get_table_names()
def upgrade() -> None:
if _has_table("tool_call_metrics"):
return
op.create_table(
"tool_call_metrics",
sa.Column("id", sa.String(64), primary_key=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("user_id", sa.String(64), nullable=True),
sa.Column("thread_id", sa.String(64), nullable=True),
sa.Column("run_id", sa.String(64), nullable=True),
sa.Column("agent_name", sa.String(128), nullable=True),
sa.Column("tool_name", sa.String(128), nullable=True),
sa.Column("skill_name", sa.String(128), nullable=True),
sa.Column("duration_ms", sa.Integer(), nullable=False, server_default="0"),
sa.Column("status", sa.String(16), nullable=False, server_default="success"),
sa.Column("error_category", sa.String(32), nullable=True),
sa.Column("error_message", sa.Text(), nullable=True),
)
op.create_index("ix_tool_call_metrics_created_at", "tool_call_metrics", ["created_at"])
op.create_index("ix_tool_call_metrics_user_id", "tool_call_metrics", ["user_id"])
op.create_index("ix_tool_call_metrics_thread_id", "tool_call_metrics", ["thread_id"])
op.create_index("ix_tool_call_metrics_run_id", "tool_call_metrics", ["run_id"])
op.create_index("ix_tool_call_metrics_agent_name", "tool_call_metrics", ["agent_name"])
op.create_index("ix_tool_call_metrics_tool_name", "tool_call_metrics", ["tool_name"])
op.create_index("ix_tool_call_metrics_skill_name", "tool_call_metrics", ["skill_name"])
op.create_index("ix_tool_call_metrics_status", "tool_call_metrics", ["status"])
op.create_index("ix_tool_call_metrics_error_category", "tool_call_metrics", ["error_category"])
def downgrade() -> None:
if _has_table("tool_call_metrics"):
op.drop_table("tool_call_metrics")
@@ -0,0 +1,62 @@
"""Add skill_name column to tool_call_metrics.
``tool_call_metrics`` is created by ``create_all`` at startup. On deployments
where the table was materialised *before* the ``skill_name`` column existed,
neither ``create_all`` (it never ALTERs) nor revision ``20260518_07`` (it skips
when the table already exists) adds the column — leaving MySQL/Postgres with a
schema that the ORM model expects a column it does not have. This revision
patches the existing table.
Revision ID: 20260519_01
Revises: 20260518_07
Create Date: 2026-05-19
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
from sqlalchemy import inspect
revision = "20260519_01"
down_revision = "20260518_07"
branch_labels = None
depends_on = None
def _has_table(table_name: str) -> bool:
inspector = inspect(op.get_bind())
return table_name in inspector.get_table_names()
def _has_column(table_name: str, column_name: str) -> bool:
if not _has_table(table_name):
return False
inspector = inspect(op.get_bind())
return any(col["name"] == column_name for col in inspector.get_columns(table_name))
def _has_index(table_name: str, index_name: str) -> bool:
if not _has_table(table_name):
return False
inspector = inspect(op.get_bind())
return any(idx["name"] == index_name for idx in inspector.get_indexes(table_name))
def upgrade() -> None:
# Brand-new DBs: the table is created with the column already present
# (by create_all / revision 20260518_07) — nothing to patch.
if not _has_table("tool_call_metrics"):
return
if not _has_column("tool_call_metrics", "skill_name"):
op.add_column("tool_call_metrics", sa.Column("skill_name", sa.String(128), nullable=True))
if not _has_index("tool_call_metrics", "ix_tool_call_metrics_skill_name"):
op.create_index("ix_tool_call_metrics_skill_name", "tool_call_metrics", ["skill_name"])
def downgrade() -> None:
if _has_index("tool_call_metrics", "ix_tool_call_metrics_skill_name"):
op.drop_index("ix_tool_call_metrics_skill_name", table_name="tool_call_metrics")
if _has_column("tool_call_metrics", "skill_name"):
op.drop_column("tool_call_metrics", "skill_name")
@@ -33,6 +33,7 @@ from deerflow.persistence.skills.model import SkillRow
from deerflow.persistence.tags.model import TagAssignmentRow, TagRow
from deerflow.persistence.thread_meta.model import ThreadMetaRow
from deerflow.persistence.thread_shares.model import ThreadShareRow
from deerflow.persistence.tool_metrics.model import ToolCallMetricRow
from deerflow.persistence.user.model import UserRow
__all__ = [
@@ -53,5 +54,6 @@ __all__ = [
"TagRow",
"ThreadMetaRow",
"ThreadShareRow",
"ToolCallMetricRow",
"UserRow",
]
@@ -0,0 +1,27 @@
"""Persistence layer for per-tool-call (skill-call) metrics.
Every tool invocation — built-in tools, skills, MCP tools, subagent calls —
writes one row recording duration and outcome. Failures are bucketed into a
coarse ``error_category`` (rate_limited / ip_blocked / auth_error / timeout /
network_error / not_found / empty_result / tool_error) so administrators can
spot retrieval outages, blocked IPs, and quota problems at a glance.
"""
from deerflow.persistence.tool_metrics.base import ToolMetricRecord, ToolMetricsStore
from deerflow.persistence.tool_metrics.model import ToolCallMetricRow
from deerflow.persistence.tool_metrics.sql import SqlToolMetricsStore
__all__ = [
"ToolCallMetricRow",
"ToolMetricRecord",
"ToolMetricsStore",
"SqlToolMetricsStore",
"make_tool_metrics_store",
]
def make_tool_metrics_store(session_factory) -> ToolMetricsStore | None:
"""Return a SQL-backed metrics store, or ``None`` when no DB is configured."""
if session_factory is None:
return None
return SqlToolMetricsStore(session_factory)
@@ -0,0 +1,84 @@
"""Abstract repository contract for tool / skill call metrics."""
from __future__ import annotations
from abc import ABC, abstractmethod
from dataclasses import dataclass
from datetime import datetime
from typing import Any
@dataclass
class ToolMetricRecord:
"""In-memory shape of a tool-call metric row before persistence."""
id: str
created_at: datetime
user_id: str | None = None
thread_id: str | None = None
run_id: str | None = None
agent_name: str | None = None
tool_name: str | None = None
# The specific Agent Skill this call targets, when the tool is a skill tool
# (``skill_view`` / ``skill_manage`` / ``skill_archive``) — taken from the
# call's ``name`` argument. Null for every non-skill tool call. This is
# what lets the admin see "skill X was invoked, and succeeded/failed".
skill_name: str | None = None
duration_ms: int = 0
status: str = "success"
# Coarse failure bucket: rate_limited / ip_blocked / auth_error / timeout /
# network_error / not_found / empty_result / tool_error. Null on success.
error_category: str | None = None
error_message: str | None = None
@dataclass
class ToolMetricsQuery:
"""Filter set for the admin list / export endpoints."""
user_id: str | None = None
tool_name: str | None = None
# When True, keep only rows that target a specific Agent Skill
# (``skill_name IS NOT NULL``) — the ``kind=skill`` filter.
skill_only: bool = False
# Filter to one specific skill by name.
skill_name: str | None = None
status: str | None = None
error_category: str | None = None
since: datetime | None = None
until: datetime | None = None
limit: int = 100
offset: int = 0
class ToolMetricsStore(ABC):
"""Persistence contract for the tool-metrics middleware + admin API."""
@abstractmethod
async def record(self, metric: ToolMetricRecord) -> None:
"""Persist one metric row. Must never raise on best-effort writes."""
raise NotImplementedError
@abstractmethod
async def list(self, query: ToolMetricsQuery) -> list[dict[str, Any]]:
"""Return the matching rows (newest first), paginated by ``query``."""
raise NotImplementedError
@abstractmethod
async def count(self, query: ToolMetricsQuery) -> int:
"""Return the unpaginated row count for ``query``."""
raise NotImplementedError
@abstractmethod
async def iter_all(self, query: ToolMetricsQuery):
"""Yield rows matching ``query`` without pagination — used by export."""
raise NotImplementedError
@abstractmethod
async def skill_breakdown(self, query: ToolMetricsQuery) -> list[dict[str, Any]]:
"""Per-skill aggregate: ``[{skill_name, total, success, error}, ...]``.
Only rows with a non-null ``skill_name`` are counted, so this answers
"which skills were invoked, and how often did each succeed/fail".
"""
raise NotImplementedError
@@ -0,0 +1,59 @@
"""ORM model for per-tool-call (skill-call) metrics."""
from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy import Integer, String, Text
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
from deerflow.persistence.types import BeijingDateTime
class ToolCallMetricRow(Base):
"""One row per completed (or failed) tool / skill invocation.
Unlike :class:`LlmCallMetricRow` this also captures *self-handled*
failures — search tools that swallow an HTTP 429/403 and return an error
payload instead of raising — so an admin can see "retrieval failed" /
"IP blocked" events that never surfaced as exceptions.
"""
__tablename__ = "tool_call_metrics"
id: Mapped[str] = mapped_column(String(64), primary_key=True)
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), default=lambda: datetime.now(UTC), index=True)
user_id: Mapped[str | None] = mapped_column(String(64), index=True)
thread_id: Mapped[str | None] = mapped_column(String(64), index=True)
run_id: Mapped[str | None] = mapped_column(String(64), index=True)
agent_name: Mapped[str | None] = mapped_column(String(128), index=True)
tool_name: Mapped[str | None] = mapped_column(String(128), index=True)
# The specific Agent Skill targeted by a skill tool call (from the call's
# ``name`` argument); null for non-skill tool calls.
skill_name: Mapped[str | None] = mapped_column(String(128), index=True)
duration_ms: Mapped[int] = mapped_column(Integer, default=0)
# "success" or "error".
status: Mapped[str] = mapped_column(String(16), default="success", index=True)
# Coarse failure bucket; null on success.
error_category: Mapped[str | None] = mapped_column(String(32), index=True)
error_message: Mapped[str | None] = mapped_column(Text)
def to_dict(self) -> dict:
return {
"id": self.id,
"created_at": self.created_at.isoformat() if isinstance(self.created_at, datetime) else self.created_at,
"user_id": self.user_id,
"thread_id": self.thread_id,
"run_id": self.run_id,
"agent_name": self.agent_name,
"tool_name": self.tool_name,
"skill_name": self.skill_name,
"duration_ms": self.duration_ms,
"status": self.status,
"error_category": self.error_category,
"error_message": self.error_message,
}
@@ -0,0 +1,138 @@
"""SQLAlchemy-backed tool metrics repository."""
from __future__ import annotations
import logging
from collections.abc import AsyncIterator
from datetime import UTC, datetime
from typing import Any
from sqlalchemy import desc, func, select
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from deerflow.persistence.tool_metrics.base import ToolMetricRecord, ToolMetricsQuery, ToolMetricsStore
from deerflow.persistence.tool_metrics.model import ToolCallMetricRow
logger = logging.getLogger(__name__)
def _normalize_created_at(value: datetime) -> datetime:
return value if value.tzinfo else value.replace(tzinfo=UTC)
class SqlToolMetricsStore(ToolMetricsStore):
"""SQLite/Postgres/MySQL-backed implementation; writes are fire-and-forget."""
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
self._sf = session_factory
async def record(self, metric: ToolMetricRecord) -> None:
row = ToolCallMetricRow(
id=metric.id,
created_at=_normalize_created_at(metric.created_at),
user_id=metric.user_id,
thread_id=metric.thread_id,
run_id=metric.run_id,
agent_name=metric.agent_name,
tool_name=metric.tool_name,
skill_name=metric.skill_name,
duration_ms=metric.duration_ms,
status=metric.status,
error_category=metric.error_category,
error_message=metric.error_message,
)
try:
async with self._sf() as session:
session.add(row)
await session.commit()
except Exception:
# Metrics are best-effort; never surface a write failure to the runtime.
logger.warning("Failed to persist tool_call_metrics row %s", metric.id, exc_info=True)
def _apply_filters(self, stmt, query: ToolMetricsQuery):
if query.user_id:
stmt = stmt.where(ToolCallMetricRow.user_id == query.user_id)
if query.tool_name:
stmt = stmt.where(ToolCallMetricRow.tool_name == query.tool_name)
if query.skill_only:
stmt = stmt.where(ToolCallMetricRow.skill_name.is_not(None))
if query.skill_name:
stmt = stmt.where(ToolCallMetricRow.skill_name == query.skill_name)
if query.status:
stmt = stmt.where(ToolCallMetricRow.status == query.status)
if query.error_category:
stmt = stmt.where(ToolCallMetricRow.error_category == query.error_category)
if query.since is not None:
stmt = stmt.where(ToolCallMetricRow.created_at >= _normalize_created_at(query.since))
if query.until is not None:
stmt = stmt.where(ToolCallMetricRow.created_at <= _normalize_created_at(query.until))
return stmt
async def list(self, query: ToolMetricsQuery) -> list[dict[str, Any]]:
stmt = select(ToolCallMetricRow)
stmt = self._apply_filters(stmt, query)
stmt = stmt.order_by(desc(ToolCallMetricRow.created_at)).limit(max(1, min(query.limit, 1000))).offset(max(0, query.offset))
async with self._sf() as session:
result = await session.execute(stmt)
return [row.to_dict() for row in result.scalars()]
async def count(self, query: ToolMetricsQuery) -> int:
stmt = select(func.count()).select_from(ToolCallMetricRow)
stmt = self._apply_filters(stmt, query)
async with self._sf() as session:
return (await session.scalar(stmt)) or 0
async def iter_all(self, query: ToolMetricsQuery) -> AsyncIterator[dict[str, Any]]:
page_size = 500
offset = 0
while True:
page = await self.list(
ToolMetricsQuery(
user_id=query.user_id,
tool_name=query.tool_name,
skill_only=query.skill_only,
skill_name=query.skill_name,
status=query.status,
error_category=query.error_category,
since=query.since,
until=query.until,
limit=page_size,
offset=offset,
)
)
if not page:
return
for item in page:
yield item
if len(page) < page_size:
return
offset += page_size
async def skill_breakdown(self, query: ToolMetricsQuery) -> list[dict[str, Any]]:
# Count success/error per skill. Force the skill-only filter so rows
# without a skill_name (plain tool calls) never leak into the result.
scoped = ToolMetricsQuery(
user_id=query.user_id,
skill_only=True,
skill_name=query.skill_name,
since=query.since,
until=query.until,
)
stmt = select(
ToolCallMetricRow.skill_name,
ToolCallMetricRow.status,
func.count().label("n"),
)
stmt = self._apply_filters(stmt, scoped).group_by(
ToolCallMetricRow.skill_name, ToolCallMetricRow.status
)
agg: dict[str, dict[str, int]] = {}
async with self._sf() as session:
for skill_name, status, n in (await session.execute(stmt)).all():
bucket = agg.setdefault(skill_name, {"total": 0, "success": 0, "error": 0})
bucket["total"] += n
bucket["success" if status == "success" else "error"] += n
return [
{"skill_name": name, **counts}
for name, counts in sorted(agg.items(), key=lambda kv: kv[1]["total"], reverse=True)
]
@@ -0,0 +1,264 @@
"""Drop the stale business tables from the SQLite file after the MySQL switch.
Context
-------
In SQLite mode the LangGraph checkpointer **and** the application share one
file: ``.deer-flow/data/deerflow.db``. After ``database.backend`` is switched
to ``mysql`` (and ``migrate_business_sqlite_to_mysql.py`` has copied the data
over), the business tables in ``deerflow.db`` are dead weight — the app now
reads them from MySQL — while the LangGraph checkpoint tables in the *same*
file are still actively used (``checkpointer.type`` stays ``sqlite``).
This script drops only the stale business tables and ``VACUUM``s to reclaim
the freed pages.
Safety design
-------------
* **Explicit DROP allowlist** — only the known business tables are dropped.
LangGraph tables (``checkpoints`` / ``writes`` / ``store`` /
``store_migrations``) and any unrecognised table are left untouched.
* **MySQL pre-check** — before dropping, it connects to MySQL and verifies
every business table exists there. A missing table aborts the run (the
migration looks incomplete). Skippable only via an explicit flag.
* **Dry-run by default** — nothing is changed unless ``--apply`` is passed.
* **Automatic backup** — ``--apply`` first copies ``deerflow.db`` to a
timestamped ``.bak`` file (disable with ``--no-backup``).
IMPORTANT: stop the backend first. ``DROP TABLE`` / ``VACUUM`` need an
exclusive lock on the SQLite file; with the server running they will fail
with "database is locked" (a safe failure — nothing is changed).
Usage
-----
# from backend/ , with the backend's interpreter (has the MySQL driver)
PYTHONPATH=. uv run python scripts/cleanup_sqlite_after_mysql_migration.py # dry-run
PYTHONPATH=. uv run python scripts/cleanup_sqlite_after_mysql_migration.py --apply # do it
"""
from __future__ import annotations
import argparse
import asyncio
import os
import re
import shutil
import sqlite3
import sys
from datetime import datetime
from pathlib import Path
# Business tables migrated to MySQL — identical to BUSINESS_TABLES in
# migrate_business_sqlite_to_mysql.py. These are the DROP allowlist.
BUSINESS_TABLES: list[str] = [
"users",
"threads_meta",
"runs",
"run_events",
"feedback",
"agents",
"skills",
"scheduled_tasks",
"scheduled_task_subscriptions",
"scheduled_task_runs",
"scheduled_task_delivery_profiles",
"scheduler_threads",
"llm_call_metrics",
"notifications",
"recommended_questions",
"tags",
"tag_assignments",
"thread_shares",
]
# Newer business tables that may exist depending on deployment age. Dropped if
# present; absence is fine. They are NOT checked against MySQL (a fresh MySQL
# install creates them via create_all / Alembic, but an older SQLite file may
# never have had them).
OPTIONAL_BUSINESS_TABLES: list[str] = [
"ai_writing_sessions",
"article_types",
"tool_call_metrics",
]
# SQLite-side Alembic version tracker. After the MySQL switch the SQLite file
# only serves the checkpointer, which does not use Alembic — so this row is
# stale. Dropped, but never MySQL-checked.
ALEMBIC_TABLE = "alembic_version"
# LangGraph infrastructure tables — these stay on SQLite and must NEVER be
# dropped. The checkpointer/store keep using them.
KEEP_TABLES: set[str] = {"checkpoints", "writes", "store", "store_migrations"}
def _repo_root_from_script() -> Path:
# scripts/ -> backend/ -> offline-backend-20260512/
return Path(__file__).resolve().parents[2]
def _load_mysql_url(explicit: str | None) -> str | None:
if explicit:
return explicit
env = os.environ.get("MYSQL_DATABASE_URL")
if env:
return env
# Fall back to the .env next to the backend directory.
for candidate in (
_repo_root_from_script() / ".env",
Path(__file__).resolve().parents[1] / ".env",
):
if candidate.is_file():
for line in candidate.read_text(encoding="utf-8").splitlines():
if line.startswith("MYSQL_DATABASE_URL="):
return line.split("=", 1)[1].strip().strip('"').strip("'")
return None
def _normalize_async_url(url: str) -> str:
"""Force the URL onto the asyncmy async driver SQLAlchemy understands."""
if url.startswith(("mysql+asyncmy://", "mysql+aiomysql://")):
return url
return re.sub(r"^mysql(\+\w+)?://", "mysql+asyncmy://", url)
def _default_sqlite_path() -> Path:
return _repo_root_from_script() / "backend" / ".deer-flow" / "data" / "deerflow.db"
async def _mysql_existing_tables(url: str) -> set[str]:
"""Return the set of table names present in the MySQL business database."""
from sqlalchemy import text
from sqlalchemy.engine import make_url
from sqlalchemy.ext.asyncio import create_async_engine
async_url = _normalize_async_url(url)
db_name = make_url(async_url).database
engine = create_async_engine(async_url)
try:
async with engine.connect() as conn:
rows = await conn.execute(
text("SELECT table_name FROM information_schema.tables WHERE table_schema = :s"),
{"s": db_name},
)
return {r[0] for r in rows}
finally:
await engine.dispose()
def _sqlite_tables(con: sqlite3.Connection) -> list[str]:
return [r[0] for r in con.execute("SELECT name FROM sqlite_master WHERE type='table' ORDER BY name")]
def _row_count(con: sqlite3.Connection, table: str) -> int:
return con.execute(f'SELECT COUNT(*) FROM "{table}"').fetchone()[0]
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--sqlite", default=str(_default_sqlite_path()), help="Path to deerflow.db")
parser.add_argument("--mysql-url", default=None, help="MySQL URL (default: $MYSQL_DATABASE_URL or .env)")
parser.add_argument("--apply", action="store_true", help="Actually drop tables + VACUUM (default: dry-run)")
parser.add_argument("--no-backup", action="store_true", help="Skip the pre-apply backup copy")
parser.add_argument("--skip-mysql-check", action="store_true", help="Do NOT verify tables exist in MySQL (unsafe)")
parser.add_argument("--drop-alembic", action="store_true", help="Also drop the stale SQLite alembic_version table")
args = parser.parse_args()
sqlite_path = Path(args.sqlite).resolve()
if not sqlite_path.is_file():
print(f"ERROR: SQLite file not found: {sqlite_path}")
return 1
size_before = sqlite_path.stat().st_size
print(f"SQLite file : {sqlite_path}")
print(f"Size : {size_before / 1024 / 1024:.1f} MB")
print(f"Mode : {'APPLY' if args.apply else 'DRY-RUN'}\n")
con = sqlite3.connect(sqlite_path)
try:
present = set(_sqlite_tables(con))
drop_business = [t for t in BUSINESS_TABLES if t in present]
drop_optional = [t for t in OPTIONAL_BUSINESS_TABLES if t in present]
to_drop = drop_business + drop_optional
if args.drop_alembic and ALEMBIC_TABLE in present:
to_drop.append(ALEMBIC_TABLE)
keep = sorted(present & KEEP_TABLES)
known = set(BUSINESS_TABLES) | set(OPTIONAL_BUSINESS_TABLES) | KEEP_TABLES | {ALEMBIC_TABLE}
unknown = sorted(present - known)
missing_business = [t for t in BUSINESS_TABLES if t not in present]
if missing_business:
print(f"NOTE: business tables already absent from SQLite: {', '.join(missing_business)}\n")
print("KEEP (LangGraph checkpointer / store - never dropped):")
for t in keep:
print(f" + {t:<34}{_row_count(con, t):>10} rows")
if unknown:
print("\nUNKNOWN (not recognised - left untouched, review manually):")
for t in unknown:
print(f" ? {t:<34}{_row_count(con, t):>10} rows")
print("\nDROP (stale business tables):")
for t in to_drop:
print(f" - {t:<34}{_row_count(con, t):>10} rows")
if not to_drop:
print(" (nothing to drop - already clean)")
return 0
# --- MySQL safety pre-check -------------------------------------
if args.skip_mysql_check:
print("\nWARNING: --skip-mysql-check set - NOT verifying the MySQL side.")
else:
mysql_url = _load_mysql_url(args.mysql_url)
if not mysql_url:
print("\nERROR: no MySQL URL ($MYSQL_DATABASE_URL / .env / --mysql-url).")
print(" Pass --skip-mysql-check to bypass (only if you are certain).")
return 1
print("\nVerifying business tables exist in MySQL ...")
try:
mysql_tables = asyncio.run(_mysql_existing_tables(mysql_url))
except Exception as exc: # noqa: BLE001
print(f"ERROR: could not query MySQL: {exc}")
print(" Run this on the backend host (MySQL access is host-restricted),")
print(" or pass --skip-mysql-check to bypass (only if you are certain).")
return 1
missing_in_mysql = [t for t in drop_business if t not in mysql_tables]
if missing_in_mysql:
print(f"ABORT: these business tables are MISSING in MySQL: {', '.join(missing_in_mysql)}")
print(" The migration looks incomplete - not dropping anything.")
return 1
print(f"OK - all {len(drop_business)} business tables present in MySQL.")
if not args.apply:
print("\nDry-run only. Re-run with --apply to drop the tables and VACUUM.")
return 0
# --- apply ------------------------------------------------------
if not args.no_backup:
backup = sqlite_path.with_name(
f"{sqlite_path.name}.bak-{datetime.now():%Y%m%d-%H%M%S}"
)
print(f"\nBacking up -> {backup}")
shutil.copy2(sqlite_path, backup)
print("Dropping tables ...")
for t in to_drop:
con.execute(f'DROP TABLE IF EXISTS "{t}"')
con.commit()
print("VACUUM (reclaiming space) ...")
con.isolation_level = None # VACUUM cannot run inside a transaction
con.execute("VACUUM")
finally:
con.close()
size_after = sqlite_path.stat().st_size
print(
f"\nDone. Size {size_before / 1024 / 1024:.1f} MB "
f"-> {size_after / 1024 / 1024:.1f} MB "
f"(reclaimed {(size_before - size_after) / 1024 / 1024:.1f} MB)"
)
return 0
if __name__ == "__main__":
sys.exit(main())