Merge branch ji into master: 工具/技能调用统计 + 用户活动审计 + 标签多选筛选
This commit is contained in:
commit
bd1efd8026
46 files changed
+3299
-315
No files matched your search
@@ -163,6 +163,7 @@ Lead-agent middlewares are assembled in strict append order across `packages/har
|
||||
6. **GuardrailMiddleware** - Pre-tool-call authorization via pluggable `GuardrailProvider` protocol (optional, if `guardrails.enabled` in config). Evaluates each tool call and returns error ToolMessage on deny. Three provider options: built-in `AllowlistProvider` (zero deps), OAP policy providers (e.g. `aport-agent-guardrails`), or custom providers. See [docs/GUARDRAILS.md](docs/GUARDRAILS.md) for setup, usage, and how to implement a provider.
|
||||
7. **SandboxAuditMiddleware** - Audits sandboxed shell/file operations for security logging before tool execution continues
|
||||
8. **ToolErrorHandlingMiddleware** - Converts tool exceptions into error `ToolMessage`s so the run can continue instead of aborting
|
||||
8a. **ToolMetricsMiddleware** - Appended immediately after `ToolErrorHandlingMiddleware` in `_build_runtime_middlewares`, so it sits *deepest* in the `wrap_tool_call` chain. Records one `tool_call_metrics` row per tool/skill/MCP invocation — timing duration and bucketing failures into a coarse `error_category` (rate_limited / ip_blocked / auth_error / timeout / network_error / not_found / empty_result / tool_error). It catches both raised exceptions (before #8 converts them) and *self-handled* errors returned as error `ToolMessage`s (search tools that swallow HTTP 429/403). It also captures the targeted Agent Skill into `tool_call_metrics.skill_name`: the lead agent invokes a skill by `read_file`-ing its `/mnt/skills/<category>/<name>/SKILL.md` path (per the `<skill_system>` prompt block — there is no dedicated "use skill" tool), so `skill_name` is recovered by scanning every tool call's string arguments for the `skills/{public,custom}/{name}/` path shape (`_SKILL_PATH_RE`), plus the explicit `name` argument of `skill_view`/`skill_manage`/`skill_archive`. `skill_view`-specific success detection handles its swallowed failures (`Skill 'x' not found.` returned as a plain string rather than raised). Regression coverage: `tests/test_tool_metrics_middleware.py`. Writes are best-effort fire-and-forget. Surfaced by the admin `GET /api/admin/tool-metrics` router (which has a `kind=skill` filter and a per-skill `/summary` breakdown).
|
||||
9. **PromptPrefixMiddleware** - Injects the admin-configured prompt prefix into the model request only. The prefix is tagged onto the first human message's `additional_kwargs['prompt_prefix']` by the Gateway (`app.gateway.services`) and prepended to content here at call time, so the persisted/displayed message stays prefix-free
|
||||
10. **SummarizationMiddleware** - Context reduction when approaching token limits (optional, if enabled)
|
||||
11. **TodoListMiddleware** - Task tracking with `write_todos` tool (optional, if plan_mode)
|
||||
@@ -214,9 +215,9 @@ FastAPI application on port 8001 with health check at `GET /health`. Set `GATEWA
|
||||
|--------|-----------|
|
||||
| **Models** (`/api/models`) | `GET /` - list models; `GET /{name}` - model details |
|
||||
| **MCP** (`/api/mcp`) | `GET /config` - get config; `PUT /config` - update config (saves to extensions_config.json) |
|
||||
| **Skills** (`/api/skills`) | `GET /` - list skills (optional `search` fuzzy-name + `tag_id` filters; each item carries `tags` + owner-authored `detail`); `GET /{name}` - details; `PUT /{name}` - update enabled; `PUT /{name}/detail` - set the long-form `detail` text (custom skill: owner or admin; built-in/public: admin only — materializes a `skills` row on demand); `GET /{name}/content` - raw SKILL.md source, restricted to an admin or the publisher (custom-skill owner); other users get 403 and see only description + detail; `POST /install` - install from .skill archive (accepts standard optional frontmatter like `version`, `author`, `compatibility`) |
|
||||
| **Skills** (`/api/skills`) | `GET /` - list skills (optional `search` fuzzy-name + repeatable `tag_ids` multi-select filter, OR-matched; each item carries `tags` + owner-authored `detail`); `GET /{name}` - details; `PUT /{name}` - update enabled; `PUT /{name}/detail` - set the long-form `detail` text (custom skill: owner or admin; built-in/public: admin only — materializes a `skills` row on demand); `GET /{name}/content` - raw SKILL.md source, restricted to an admin or the publisher (custom-skill owner); other users get 403 and see only description + detail; `POST /install` - install from .skill archive (accepts standard optional frontmatter like `version`, `author`, `compatibility`) |
|
||||
| **Memory** (`/api/memory`) | `GET /` - memory data; `POST /reload` - force reload; `GET /config` - config; `GET /status` - config + data |
|
||||
| **Agents** (`/api/agents`) | `GET /` - list built-in, owned, and published agents (optional `search` + `tag_id` filters; each item carries `tags`); `POST /` - create user-owned agent; `GET/PUT/DELETE /{id}` - read/update/delete by stable agent id; `GET /check` - validate non-empty display name (duplicates allowed) |
|
||||
| **Agents** (`/api/agents`) | `GET /` - list built-in, owned, and published agents (optional `search` + repeatable `tag_ids` multi-select filter, OR-matched; each item carries `tags`); `POST /` - create user-owned agent; `GET/PUT/DELETE /{id}` - read/update/delete by stable agent id; `GET /check` - validate non-empty display name (duplicates allowed) |
|
||||
| **Tags** (`/api/tags`) | `GET /` - list tags (optional `tag_type` filter; open to all); `POST /` - create tag (admin); `PUT /{id}` / `DELETE /{id}` - update/delete tag (admin); `GET /assignments/{target_type}/{target_id}` - tags on a resource; `PUT /assignments/{target_type}/{target_id}` - replace a resource's tags (admin). Tags are typed `agent`/`skill`/`scheduled_task` and may only tag resources of their own type. |
|
||||
| **Uploads** (`/api/threads/{id}/uploads`) | `POST /` - upload files (auto-converts PDF/PPT/Excel/Word); `GET /list` - list; `DELETE /{filename}` - delete |
|
||||
| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; unexpected failures are logged server-side and return a generic 500 detail |
|
||||
@@ -226,7 +227,7 @@ FastAPI application on port 8001 with health check at `GET /health`. Set `GATEWA
|
||||
| **Feedback** (`/api/threads/{id}/runs/{rid}/feedback`) | `PUT /` - upsert feedback; `DELETE /` - delete user feedback; `POST /` - create feedback; `GET /` - list feedback; `GET /stats` - aggregate stats; `DELETE /{fid}` - delete specific |
|
||||
| **Runs** (`/api/runs`) | `POST /stream` - stateless run + SSE; `POST /wait` - stateless run + block; `GET /{rid}/messages` - paginated messages by run_id `{data, has_more}` (cursor: `after_seq`/`before_seq`); `GET /{rid}/feedback` - list feedback by run_id |
|
||||
| **Thread Shares** (`/api/thread-shares`) | Per-conversation sharing with two modes (`thread_shares.mode`). **`import`**: `POST /` `{thread_id, mode}` - create (or reuse) a revocable share code; `GET /{code}/preview` + `POST /{code}/import` - copy that conversation into the caller's account (fresh `thread_id` via `app/gateway/thread_copy.py` + new `threads_meta` row + `user-data` copy; dedup on `metadata.shared_origin`). **`view`**: a public, read-only static page — see Public Shares below. Both modes: `GET /` - list my codes; `DELETE /{code}` - revoke. |
|
||||
| **Public Shares** (`/api/public/shares`) | **Unauthenticated** (the `/api/public/` prefix is whitelisted in `auth_middleware._PUBLIC_PATH_PREFIXES`). `GET /{code}` - read-only conversation snapshot (title + serialized messages + artifacts) for a `view`-mode share; messages are read straight from the latest checkpoint. `GET /{code}/files/{path}` - serves an **image** referenced by that conversation (images only — non-image paths 404; SVG forced to download). Missing/revoked/non-`view` codes all return 404. |
|
||||
| **Public Shares** (`/api/public/shares`) | **Unauthenticated** (the `/api/public/` prefix is whitelisted in `auth_middleware._PUBLIC_PATH_PREFIXES`). `GET /{code}` - read-only conversation snapshot (title + serialized messages + artifacts) for a `view`-mode share; messages are read straight from the latest checkpoint. `GET /{code}/files/{path}` - serves a file referenced by that conversation (images, Markdown, text, PDF, …) so the shared page's links are openable; path resolution is confined to the owner's thread directory. Active content (`text/html`, `application/xhtml+xml`, `image/svg+xml`) is forced to download (`_public_share_disposition`); everything else is served inline. Missing/revoked/non-`view` codes all return 404. |
|
||||
| **Scheduled Tasks** (`/api/scheduled-tasks`) | Per-user cron-like agent tasks. `GET/POST /` - list/create owned tasks; `PATCH/DELETE /{id}` - update/delete (owner); `POST /{id}/{pause,resume,trigger}`; `GET /{id}/runs`, `GET /runs/{rid}`, run-file preview/edit/download; `GET /runs/{rid}/messages` - agent messages for the run's underlying agent run (used to inspect a stuck/running run; authorized via `_load_run`); `DELETE /runs/{rid}` - owner-scoped, with an admin fallback so admins can delete any user's run; publish/subscribe under `/{id}/{publish,unpublish,subscribe}` and `GET /published`. **Admin (cross-user):** `GET /admin/all` - every task; `PATCH/DELETE /admin/{id}` - edit/delete any task; `POST /admin/{id}/trigger`; `GET /admin/{id}/runs`. Admin routes require `system_role == "admin"`. |
|
||||
|
||||
Proxied through nginx: `/api/langgraph/*` → LangGraph, all other `/api/*` → Gateway.
|
||||
|
||||
@@ -23,6 +23,7 @@ from app.gateway.csrf_middleware import CSRFMiddleware
|
||||
from app.gateway.deps import langgraph_runtime
|
||||
from app.gateway.routers import (
|
||||
ai_writing,
|
||||
admin_users,
|
||||
agents,
|
||||
artifacts,
|
||||
assistants_compat,
|
||||
@@ -42,6 +43,7 @@ from app.gateway.routers import (
|
||||
system_settings,
|
||||
tags,
|
||||
thread_runs,
|
||||
tool_metrics,
|
||||
thread_shares,
|
||||
threads,
|
||||
uploads,
|
||||
@@ -256,7 +258,7 @@ async def _run_db_migrations() -> None:
|
||||
logger.info("Persistence backend=memory — skipping Alembic migrations")
|
||||
return
|
||||
|
||||
db_url = engine.url.render_as_string(hide_password=False).replace("%", "%%")
|
||||
db_url = engine.url.render_as_string(hide_password=False).replace("%", "%%")
|
||||
migrations_dir = str(Path(_persistence_pkg.__file__).parent / "migrations")
|
||||
|
||||
def _run_sync() -> None:
|
||||
@@ -689,6 +691,12 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
|
||||
# Admin LLM metrics audit + export
|
||||
app.include_router(llm_metrics.router)
|
||||
|
||||
# Admin window into other users' threads / messages / memory
|
||||
app.include_router(admin_users.router)
|
||||
|
||||
# Admin tool / skill call metrics audit + export
|
||||
app.include_router(tool_metrics.router)
|
||||
|
||||
# Channels API is mounted at /api/channels
|
||||
app.include_router(channels.router)
|
||||
|
||||
|
||||
@@ -97,6 +97,10 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
|
||||
app.state.llm_metrics_store = make_llm_metrics_store(sf)
|
||||
|
||||
from deerflow.persistence.tool_metrics import make_tool_metrics_store
|
||||
|
||||
app.state.tool_metrics_store = make_tool_metrics_store(sf)
|
||||
|
||||
from deerflow.persistence.agents import make_agent_store
|
||||
|
||||
app.state.agent_store = make_agent_store(sf)
|
||||
@@ -202,6 +206,11 @@ def get_llm_metrics_store(request: Request):
|
||||
return getattr(request.app.state, "llm_metrics_store", None)
|
||||
|
||||
|
||||
def get_tool_metrics_store(request: Request):
|
||||
"""Return the tool-call metrics store, or None when not configured."""
|
||||
return getattr(request.app.state, "tool_metrics_store", None)
|
||||
|
||||
|
||||
def get_agent_store(request: Request) -> AgentStore:
|
||||
"""Return the custom-agent store."""
|
||||
val = getattr(request.app.state, "agent_store", None)
|
||||
|
||||
@@ -0,0 +1,162 @@
|
||||
"""Admin-only window into another user's activity.
|
||||
|
||||
Lets an administrator inspect any user's conversation history (threads + the
|
||||
Q&A messages inside them) and their accumulated memory (global + per-agent
|
||||
buckets). All endpoints require ``system_role == "admin"``; they are strictly
|
||||
read-only.
|
||||
|
||||
The underlying stores normally auto-scope every query to the *calling* user
|
||||
via a contextvar. These endpoints pass an explicit ``user_id`` so the admin
|
||||
reads the *target* user's data instead — which is exactly what those store
|
||||
methods' ``user_id`` parameter is designed for.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.gateway.deps import (
|
||||
get_checkpointer,
|
||||
get_current_user_from_request,
|
||||
get_thread_store,
|
||||
)
|
||||
from deerflow.agents.memory.updater import get_memory_data
|
||||
from deerflow.config.agents_config import AGENT_NAME_PATTERN
|
||||
from deerflow.config.paths import get_paths
|
||||
from deerflow.runtime import serialize_channel_values
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/api/admin/users", tags=["admin-users"])
|
||||
|
||||
|
||||
async def _require_admin(request: Request) -> None:
|
||||
"""Reject non-admin callers. In no-auth mode there is no user → allow."""
|
||||
try:
|
||||
user = await get_current_user_from_request(request)
|
||||
except HTTPException:
|
||||
raise
|
||||
if user is None:
|
||||
return
|
||||
if getattr(user, "system_role", None) != "admin":
|
||||
raise HTTPException(status_code=403, detail="Admin only")
|
||||
|
||||
|
||||
# ── Threads ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class AdminThreadItem(BaseModel):
|
||||
thread_id: str
|
||||
display_name: str = ""
|
||||
status: str = "idle"
|
||||
created_at: str = ""
|
||||
updated_at: str = ""
|
||||
|
||||
|
||||
@router.get("/{user_id}/threads", response_model=list[AdminThreadItem])
|
||||
async def list_user_threads(
|
||||
user_id: str,
|
||||
request: Request,
|
||||
limit: int = Query(default=100, ge=1, le=500),
|
||||
offset: int = Query(default=0, ge=0),
|
||||
) -> list[AdminThreadItem]:
|
||||
"""List one user's conversation threads, newest first."""
|
||||
await _require_admin(request)
|
||||
repo = get_thread_store(request)
|
||||
rows = await repo.search(limit=limit, offset=offset, user_id=user_id)
|
||||
return [
|
||||
AdminThreadItem(
|
||||
thread_id=r["thread_id"],
|
||||
display_name=r.get("display_name") or "",
|
||||
status=r.get("status", "idle"),
|
||||
created_at=str(r.get("created_at") or ""),
|
||||
updated_at=str(r.get("updated_at") or ""),
|
||||
)
|
||||
for r in rows
|
||||
]
|
||||
|
||||
|
||||
@router.get("/{user_id}/threads/{thread_id}/messages", response_model=list[dict])
|
||||
async def list_user_thread_messages(
|
||||
user_id: str,
|
||||
thread_id: str,
|
||||
request: Request,
|
||||
limit: int = Query(default=400, ge=1, le=1000),
|
||||
) -> list[dict]:
|
||||
"""Return the conversation messages of one of the user's threads.
|
||||
|
||||
Read straight from the LangGraph checkpoint — the same source the chat UI
|
||||
renders — rather than the optional ``run_events`` audit log, which may be
|
||||
unconfigured (``run_events.backend=memory``) or empty for older threads.
|
||||
"""
|
||||
await _require_admin(request)
|
||||
|
||||
repo = get_thread_store(request)
|
||||
# ``user_id=None`` bypasses the owner filter so the admin can fetch any row;
|
||||
# we then assert the thread really belongs to the target user.
|
||||
thread = await repo.get(thread_id, user_id=None)
|
||||
if thread is None:
|
||||
raise HTTPException(status_code=404, detail="Thread not found")
|
||||
owner = thread.get("user_id")
|
||||
if owner is not None and owner != user_id:
|
||||
raise HTTPException(status_code=404, detail="Thread does not belong to this user")
|
||||
|
||||
checkpointer = get_checkpointer(request)
|
||||
config = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
|
||||
try:
|
||||
checkpoint_tuple = await checkpointer.aget_tuple(config)
|
||||
except Exception as exc: # noqa: BLE001 - surface a clean error
|
||||
logger.error("Failed to read checkpoint for thread %s: %s", thread_id, exc, exc_info=True)
|
||||
raise HTTPException(status_code=500, detail="Failed to read thread state") from exc
|
||||
if checkpoint_tuple is None:
|
||||
return []
|
||||
|
||||
checkpoint = getattr(checkpoint_tuple, "checkpoint", {}) or {}
|
||||
channel_values = checkpoint.get("channel_values", {}) if isinstance(checkpoint, dict) else {}
|
||||
values = serialize_channel_values(channel_values)
|
||||
messages = values.get("messages", [])
|
||||
if not isinstance(messages, list):
|
||||
return []
|
||||
# Keep the most recent ``limit`` messages — long threads stay bounded.
|
||||
return messages[-limit:]
|
||||
|
||||
|
||||
# ── Memory ─────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class AdminMemoryBucket(BaseModel):
|
||||
name: str = Field(..., description="Bucket name: 'global' or a custom-agent name")
|
||||
data: dict = Field(default_factory=dict, description="Raw memory data for the bucket")
|
||||
|
||||
|
||||
class AdminMemoryResponse(BaseModel):
|
||||
user_id: str
|
||||
buckets: list[AdminMemoryBucket] = Field(default_factory=list)
|
||||
|
||||
|
||||
@router.get("/{user_id}/memory", response_model=AdminMemoryResponse)
|
||||
async def get_user_memory(user_id: str, request: Request) -> AdminMemoryResponse:
|
||||
"""Return every memory bucket (global + per-agent) for one user."""
|
||||
await _require_admin(request)
|
||||
|
||||
buckets: list[AdminMemoryBucket] = [
|
||||
AdminMemoryBucket(name="global", data=get_memory_data(None, user_id=user_id))
|
||||
]
|
||||
|
||||
agents_root = get_paths().user_dir(user_id) / "agents"
|
||||
if agents_root.is_dir():
|
||||
for entry in sorted(agents_root.iterdir(), key=lambda p: p.name.lower()):
|
||||
if not entry.is_dir() or not (entry / "memory.json").exists():
|
||||
continue
|
||||
if not AGENT_NAME_PATTERN.match(entry.name):
|
||||
continue
|
||||
try:
|
||||
data = get_memory_data(entry.name, user_id=user_id)
|
||||
except Exception:
|
||||
logger.warning("Failed to read memory bucket %s for user", entry.name)
|
||||
continue
|
||||
buckets.append(AdminMemoryBucket(name=entry.name, data=data))
|
||||
|
||||
return AdminMemoryResponse(user_id=user_id, buckets=buckets)
|
||||
@@ -244,7 +244,7 @@ async def _get_editable_or_raise(
|
||||
async def list_agents(
|
||||
request: Request,
|
||||
search: str | None = Query(default=None, description="Case-insensitive fuzzy match on agent name"),
|
||||
tag_id: str | None = Query(default=None, description="Keep only agents carrying this tag"),
|
||||
tag_ids: list[str] | None = Query(default=None, description="Keep agents carrying any of these tags (OR)"),
|
||||
) -> AgentsListResponse:
|
||||
_require_agents_api_enabled()
|
||||
|
||||
@@ -262,8 +262,9 @@ async def list_agents(
|
||||
if search:
|
||||
needle = search.strip().lower()
|
||||
responses = [r for r in responses if needle in r.name.lower()]
|
||||
if tag_id:
|
||||
responses = [r for r in responses if any(t.get("id") == tag_id for t in r.tags)]
|
||||
if tag_ids:
|
||||
wanted = set(tag_ids)
|
||||
responses = [r for r in responses if any(t.get("id") in wanted for t in r.tags)]
|
||||
return AgentsListResponse(agents=responses)
|
||||
except HTTPException:
|
||||
raise
|
||||
|
||||
@@ -197,7 +197,7 @@ async def get_scheduler_thread(request: Request) -> dict:
|
||||
async def list_scheduled_tasks(
|
||||
request: Request,
|
||||
search: str | None = Query(default=None, description="Case-insensitive fuzzy match on task name"),
|
||||
tag_id: str | None = Query(default=None, description="Keep only tasks carrying this tag"),
|
||||
tag_ids: list[str] | None = Query(default=None, description="Keep tasks carrying any of these tags (OR)"),
|
||||
) -> list[dict]:
|
||||
service = get_scheduled_task_service()
|
||||
user_id = await _user_id(request)
|
||||
@@ -211,8 +211,9 @@ async def list_scheduled_tasks(
|
||||
if search:
|
||||
needle = search.strip().lower()
|
||||
tasks = [t for t in tasks if needle in str(t.get("name", "")).lower()]
|
||||
if tag_id:
|
||||
tasks = [t for t in tasks if any(tg.get("id") == tag_id for tg in t.get("tags", []))]
|
||||
if tag_ids:
|
||||
wanted = set(tag_ids)
|
||||
tasks = [t for t in tasks if any(tg.get("id") in wanted for tg in t.get("tags", []))]
|
||||
return tasks
|
||||
|
||||
|
||||
|
||||
@@ -236,7 +236,7 @@ async def _attach_and_filter_tags(
|
||||
tag_store: TagStore,
|
||||
responses: list[SkillResponse],
|
||||
search: str | None,
|
||||
tag_id: str | None,
|
||||
tag_ids: list[str] | None,
|
||||
) -> list[SkillResponse]:
|
||||
"""Bulk-attach tags to skill responses, then apply name/tag filters."""
|
||||
assignments = await tag_store.list_assignments("skill", [r.name for r in responses])
|
||||
@@ -245,8 +245,9 @@ async def _attach_and_filter_tags(
|
||||
if search:
|
||||
needle = search.strip().lower()
|
||||
responses = [r for r in responses if needle in r.name.lower()]
|
||||
if tag_id:
|
||||
responses = [r for r in responses if any(t.get("id") == tag_id for t in r.tags)]
|
||||
if tag_ids:
|
||||
wanted = set(tag_ids)
|
||||
responses = [r for r in responses if any(t.get("id") in wanted for t in r.tags)]
|
||||
return responses
|
||||
|
||||
|
||||
@@ -262,7 +263,7 @@ async def list_skills(
|
||||
skill_store: SkillStore = Depends(get_skill_store),
|
||||
tag_store: TagStore = Depends(get_tag_store),
|
||||
search: str | None = Query(default=None, description="Case-insensitive fuzzy match on skill name"),
|
||||
tag_id: str | None = Query(default=None, description="Keep only skills carrying this tag"),
|
||||
tag_ids: list[str] | None = Query(default=None, description="Keep skills carrying any of these tags (OR)"),
|
||||
) -> SkillsListResponse:
|
||||
try:
|
||||
user_id = _current_user_id(request)
|
||||
@@ -281,7 +282,7 @@ async def list_skills(
|
||||
if record is None:
|
||||
continue
|
||||
responses.append(_skill_to_response(skill, record))
|
||||
responses = await _attach_and_filter_tags(tag_store, responses, search, tag_id)
|
||||
responses = await _attach_and_filter_tags(tag_store, responses, search, tag_ids)
|
||||
return SkillsListResponse(skills=responses)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to load skills: {e}", exc_info=True)
|
||||
|
||||
@@ -294,20 +294,38 @@ async def get_public_share_snapshot(
|
||||
)
|
||||
|
||||
|
||||
# Content types that can execute scripts in the app origin — always served as
|
||||
# a download attachment so a shared file can never run as a top-level document.
|
||||
_PUBLIC_SHARE_ACTIVE_TYPES = {"text/html", "application/xhtml+xml", "image/svg+xml"}
|
||||
|
||||
|
||||
def _public_share_disposition(mime_type: str) -> str:
|
||||
"""Pick a Content-Disposition for a public-share file.
|
||||
|
||||
Active content (HTML/XHTML/SVG) is forced to ``attachment`` so it cannot
|
||||
execute as a top-level document; everything else (images, Markdown, text,
|
||||
PDF, …) is served ``inline`` so viewers can open it directly.
|
||||
"""
|
||||
return "attachment" if mime_type in _PUBLIC_SHARE_ACTIVE_TYPES else "inline"
|
||||
|
||||
|
||||
@public_router.get(
|
||||
"/{share_code}/files/{path:path}",
|
||||
summary="Public image attachment from a view-shared conversation",
|
||||
summary="Public file attachment from a view-shared conversation",
|
||||
)
|
||||
async def get_public_share_file(
|
||||
share_code: str,
|
||||
path: str,
|
||||
share_store: ThreadShareStore = Depends(get_thread_share_store),
|
||||
) -> FileResponse:
|
||||
"""Serve an image referenced by a view-shared conversation.
|
||||
"""Serve a file referenced by a view-shared conversation.
|
||||
|
||||
Restricted to images only — the public share page renders inline images,
|
||||
but the endpoint must not become a way to read arbitrary thread files.
|
||||
Non-image paths return 404 just like a missing file.
|
||||
A ``view`` share is a public read-only page, so the files its conversation
|
||||
references (images, Markdown, text, PDF, …) must be openable. Path
|
||||
resolution is confined to the owner's thread directory, so this cannot read
|
||||
arbitrary files. Active content (HTML/XHTML/SVG) is forced to download so it
|
||||
can never execute as a top-level document in the app origin; everything
|
||||
else is served inline.
|
||||
"""
|
||||
share = await share_store.get(share_code)
|
||||
if share is None or share.get("revoked") or share.get("mode") != "view":
|
||||
@@ -324,17 +342,10 @@ async def get_public_share_file(
|
||||
if not actual_path.exists() or not actual_path.is_file():
|
||||
raise HTTPException(status_code=404, detail="File not found")
|
||||
|
||||
mime_type, _ = mimetypes.guess_type(actual_path.name)
|
||||
if not mime_type or not mime_type.startswith("image/"):
|
||||
# Only images are public; everything else is treated as not found.
|
||||
raise HTTPException(status_code=404, detail="File not found")
|
||||
|
||||
# SVG can carry scripts — serve it as a download so it can never execute
|
||||
# as a top-level document in the app origin. Raster images render inline.
|
||||
is_svg = mime_type == "image/svg+xml"
|
||||
mime_type = mimetypes.guess_type(actual_path.name)[0] or "application/octet-stream"
|
||||
return FileResponse(
|
||||
actual_path,
|
||||
media_type=mime_type,
|
||||
content_disposition_type="attachment" if is_svg else "inline",
|
||||
content_disposition_type=_public_share_disposition(mime_type),
|
||||
headers={"Cache-Control": "public, max-age=300"},
|
||||
)
|
||||
@@ -0,0 +1,380 @@
|
||||
"""Admin-only tool / skill call metrics: list + Excel export.
|
||||
|
||||
Companion to ``llm_metrics`` — where that audits every LLM provider call, this
|
||||
audits every *tool* call (built-in tools, skills, MCP tools, subagent calls),
|
||||
including the failures: retrieval errors, blocked/rate-limited IPs, timeouts,
|
||||
auth problems. Failures are bucketed into a coarse ``error_category`` so an
|
||||
admin can scan for systemic issues.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from io import BytesIO
|
||||
from typing import Any
|
||||
from urllib.parse import quote
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query, Request
|
||||
from fastapi.responses import StreamingResponse
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.gateway.deps import get_local_provider, get_optional_user_from_request, get_tool_metrics_store
|
||||
from deerflow.persistence.tool_metrics.base import ToolMetricsQuery, ToolMetricsStore
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/api/admin/tool-metrics", tags=["tool-metrics"])
|
||||
|
||||
_VALID_CATEGORIES = (
|
||||
"rate_limited",
|
||||
"ip_blocked",
|
||||
"auth_error",
|
||||
"timeout",
|
||||
"network_error",
|
||||
"not_found",
|
||||
"empty_result",
|
||||
"tool_error",
|
||||
)
|
||||
|
||||
def _kind_is_skill(kind: str | None) -> bool:
|
||||
"""``kind=skill`` keeps only rows that target a specific Agent Skill."""
|
||||
return kind == "skill"
|
||||
|
||||
|
||||
class ToolMetricResponse(BaseModel):
|
||||
id: str
|
||||
created_at: str | None = None
|
||||
user_id: str | None = None
|
||||
user_name: str | None = Field(
|
||||
default=None,
|
||||
description="Local part (before @) of the user's email, resolved from user_id",
|
||||
)
|
||||
thread_id: str | None = None
|
||||
run_id: str | None = None
|
||||
agent_name: str | None = None
|
||||
tool_name: str | None = None
|
||||
skill_name: str | None = Field(
|
||||
default=None,
|
||||
description="The Agent Skill this call targets (skill tools only)",
|
||||
)
|
||||
duration_ms: int = 0
|
||||
status: str = "success"
|
||||
error_category: str | None = None
|
||||
error_message: str | None = None
|
||||
|
||||
|
||||
class ToolMetricsListResponse(BaseModel):
|
||||
items: list[ToolMetricResponse]
|
||||
total: int = Field(..., description="Total rows matching the filter, ignoring pagination")
|
||||
limit: int
|
||||
offset: int
|
||||
|
||||
|
||||
async def _require_admin(request: Request) -> None:
|
||||
user = await get_optional_user_from_request(request)
|
||||
if user is None:
|
||||
return
|
||||
if getattr(user, "system_role", None) != "admin":
|
||||
raise HTTPException(status_code=403, detail="Tool metrics admin view is admin-only")
|
||||
|
||||
|
||||
def _require_store(request: Request) -> ToolMetricsStore:
|
||||
store = get_tool_metrics_store(request)
|
||||
if store is None:
|
||||
raise HTTPException(status_code=503, detail="Tool metrics store is not configured")
|
||||
return store
|
||||
|
||||
|
||||
def _parse_dt(value: str | None) -> datetime | None:
|
||||
if not value:
|
||||
return None
|
||||
try:
|
||||
return datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=422, detail=f"Invalid datetime: {value!r} ({exc})") from exc
|
||||
|
||||
|
||||
def _email_local_part(email: str | None) -> str | None:
|
||||
if not email:
|
||||
return None
|
||||
at = email.find("@")
|
||||
return email[:at] if at > 0 else email
|
||||
|
||||
|
||||
async def _resolve_user_name_to_id(name: str | None) -> str | None:
|
||||
if not name:
|
||||
return None
|
||||
try:
|
||||
provider = get_local_provider()
|
||||
user = await provider.find_user_by_email_local_part(name)
|
||||
except Exception:
|
||||
return None
|
||||
return str(user.id) if user is not None else None
|
||||
|
||||
|
||||
async def _attach_usernames(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
"""Resolve each row's ``user_id`` UUID to a friendly ``user_name``."""
|
||||
if not rows:
|
||||
return rows
|
||||
user_ids = {row["user_id"] for row in rows if isinstance(row, dict) and row.get("user_id")}
|
||||
if not user_ids:
|
||||
return rows
|
||||
try:
|
||||
provider = get_local_provider()
|
||||
except Exception:
|
||||
provider = None
|
||||
email_by_id: dict[str, str] = {}
|
||||
if provider is not None:
|
||||
for uid in user_ids:
|
||||
try:
|
||||
user = await provider.get_user(uid)
|
||||
except Exception:
|
||||
user = None
|
||||
if user is not None and user.email:
|
||||
email_by_id[uid] = user.email
|
||||
for row in rows:
|
||||
if isinstance(row, dict) and row.get("user_id"):
|
||||
email = email_by_id.get(row["user_id"])
|
||||
row["user_name"] = _email_local_part(email) or row["user_id"]
|
||||
return rows
|
||||
|
||||
|
||||
def _build_query(
|
||||
*,
|
||||
user_id: str | None,
|
||||
tool_name: str | None,
|
||||
skill_only: bool = False,
|
||||
skill_name: str | None = None,
|
||||
status: str | None,
|
||||
error_category: str | None,
|
||||
since: str | None,
|
||||
until: str | None,
|
||||
limit: int,
|
||||
offset: int,
|
||||
) -> ToolMetricsQuery:
|
||||
return ToolMetricsQuery(
|
||||
user_id=user_id or None,
|
||||
tool_name=tool_name or None,
|
||||
skill_only=skill_only,
|
||||
skill_name=skill_name or None,
|
||||
status=status or None,
|
||||
error_category=error_category or None,
|
||||
since=_parse_dt(since),
|
||||
until=_parse_dt(until),
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
)
|
||||
|
||||
|
||||
@router.get("", response_model=ToolMetricsListResponse)
|
||||
async def list_tool_metrics(
|
||||
request: Request,
|
||||
user_id: str | None = Query(default=None),
|
||||
user_name: str | None = Query(default=None, description="Username (email local part) to filter by"),
|
||||
tool_name: str | None = Query(default=None),
|
||||
kind: str | None = Query(default=None, description="Coarse call kind: 'skill' keeps only Agent-Skill invocations"),
|
||||
skill_name: str | None = Query(default=None, description="Filter to one specific Agent Skill by name"),
|
||||
status: str | None = Query(default=None, pattern="^(success|error)$"),
|
||||
error_category: str | None = Query(default=None, description="Coarse failure bucket"),
|
||||
since: str | None = Query(default=None, description="ISO 8601 datetime, inclusive lower bound"),
|
||||
until: str | None = Query(default=None, description="ISO 8601 datetime, inclusive upper bound"),
|
||||
limit: int = Query(default=50, ge=1, le=500),
|
||||
offset: int = Query(default=0, ge=0),
|
||||
) -> ToolMetricsListResponse:
|
||||
await _require_admin(request)
|
||||
store = _require_store(request)
|
||||
if error_category and error_category not in _VALID_CATEGORIES:
|
||||
raise HTTPException(status_code=422, detail=f"Unknown error_category: {error_category!r}")
|
||||
if not user_id and user_name:
|
||||
resolved = await _resolve_user_name_to_id(user_name)
|
||||
if resolved is None:
|
||||
return ToolMetricsListResponse(items=[], total=0, limit=limit, offset=offset)
|
||||
user_id = resolved
|
||||
query = _build_query(
|
||||
user_id=user_id,
|
||||
tool_name=tool_name,
|
||||
skill_only=_kind_is_skill(kind),
|
||||
skill_name=skill_name,
|
||||
status=status,
|
||||
error_category=error_category,
|
||||
since=since,
|
||||
until=until,
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
)
|
||||
rows = await store.list(query)
|
||||
total = await store.count(query)
|
||||
await _attach_usernames(rows)
|
||||
return ToolMetricsListResponse(
|
||||
items=[ToolMetricResponse(**row) for row in rows],
|
||||
total=total,
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
)
|
||||
|
||||
|
||||
class SkillStat(BaseModel):
|
||||
skill_name: str
|
||||
total: int
|
||||
success: int
|
||||
error: int
|
||||
|
||||
|
||||
class ToolMetricsSummaryResponse(BaseModel):
|
||||
total: int
|
||||
error_total: int
|
||||
by_category: dict[str, int] = Field(default_factory=dict)
|
||||
by_skill: list[SkillStat] = Field(
|
||||
default_factory=list,
|
||||
description="Per-skill invocation counts — answers 'was skill X called, and did it succeed'",
|
||||
)
|
||||
|
||||
|
||||
@router.get("/summary", response_model=ToolMetricsSummaryResponse)
|
||||
async def tool_metrics_summary(
|
||||
request: Request,
|
||||
user_id: str | None = Query(default=None),
|
||||
user_name: str | None = Query(default=None),
|
||||
tool_name: str | None = Query(default=None),
|
||||
kind: str | None = Query(default=None),
|
||||
skill_name: str | None = Query(default=None),
|
||||
since: str | None = Query(default=None),
|
||||
until: str | None = Query(default=None),
|
||||
) -> ToolMetricsSummaryResponse:
|
||||
"""Aggregate counts: total, errors, per-category breakdown, and per-skill stats."""
|
||||
await _require_admin(request)
|
||||
store = _require_store(request)
|
||||
if not user_id and user_name:
|
||||
resolved = await _resolve_user_name_to_id(user_name)
|
||||
if resolved is None:
|
||||
return ToolMetricsSummaryResponse(total=0, error_total=0, by_category={}, by_skill=[])
|
||||
user_id = resolved
|
||||
skill_only = _kind_is_skill(kind)
|
||||
|
||||
def _q(*, status: str | None = None, error_category: str | None = None) -> ToolMetricsQuery:
|
||||
return _build_query(
|
||||
user_id=user_id,
|
||||
tool_name=tool_name,
|
||||
skill_only=skill_only,
|
||||
skill_name=skill_name,
|
||||
status=status,
|
||||
error_category=error_category,
|
||||
since=since,
|
||||
until=until,
|
||||
limit=1,
|
||||
offset=0,
|
||||
)
|
||||
|
||||
total = await store.count(_q())
|
||||
error_total = await store.count(_q(status="error"))
|
||||
by_category: dict[str, int] = {}
|
||||
for category in _VALID_CATEGORIES:
|
||||
count = await store.count(_q(status="error", error_category=category))
|
||||
if count:
|
||||
by_category[category] = count
|
||||
by_skill = [SkillStat(**row) for row in await store.skill_breakdown(_q())]
|
||||
return ToolMetricsSummaryResponse(
|
||||
total=total, error_total=error_total, by_category=by_category, by_skill=by_skill
|
||||
)
|
||||
|
||||
|
||||
def _format_cell(value: Any) -> Any:
|
||||
if value is None:
|
||||
return ""
|
||||
if isinstance(value, datetime):
|
||||
return value.replace(tzinfo=None) if value.tzinfo else value
|
||||
return value
|
||||
|
||||
|
||||
@router.get("/export")
|
||||
async def export_tool_metrics(
|
||||
request: Request,
|
||||
user_id: str | None = Query(default=None),
|
||||
user_name: str | None = Query(default=None),
|
||||
tool_name: str | None = Query(default=None),
|
||||
kind: str | None = Query(default=None),
|
||||
skill_name: str | None = Query(default=None),
|
||||
status: str | None = Query(default=None, pattern="^(success|error)$"),
|
||||
error_category: str | None = Query(default=None),
|
||||
since: str | None = Query(default=None),
|
||||
until: str | None = Query(default=None),
|
||||
):
|
||||
"""Stream the full filtered metric set as an ``.xlsx`` file."""
|
||||
await _require_admin(request)
|
||||
store = _require_store(request)
|
||||
if not user_id and user_name:
|
||||
resolved = await _resolve_user_name_to_id(user_name)
|
||||
user_id = resolved if resolved is not None else "__no_such_user__"
|
||||
|
||||
try:
|
||||
from openpyxl import Workbook
|
||||
except ImportError as exc:
|
||||
raise HTTPException(status_code=500, detail=f"openpyxl is not installed: {exc}") from exc
|
||||
|
||||
query = _build_query(
|
||||
user_id=user_id,
|
||||
tool_name=tool_name,
|
||||
skill_only=_kind_is_skill(kind),
|
||||
skill_name=skill_name,
|
||||
status=status,
|
||||
error_category=error_category,
|
||||
since=since,
|
||||
until=until,
|
||||
limit=1,
|
||||
offset=0,
|
||||
)
|
||||
|
||||
workbook = Workbook(write_only=True)
|
||||
sheet = workbook.create_sheet("tool_call_metrics")
|
||||
columns = [
|
||||
"id",
|
||||
"created_at",
|
||||
"status",
|
||||
"error_category",
|
||||
"user_name",
|
||||
"user_id",
|
||||
"thread_id",
|
||||
"run_id",
|
||||
"agent_name",
|
||||
"tool_name",
|
||||
"skill_name",
|
||||
"duration_ms",
|
||||
"error_message",
|
||||
]
|
||||
sheet.append(columns)
|
||||
|
||||
buffered: list[dict[str, Any]] = []
|
||||
FLUSH_AT = 200
|
||||
|
||||
async def flush() -> int:
|
||||
if not buffered:
|
||||
return 0
|
||||
await _attach_usernames(buffered)
|
||||
for row in buffered:
|
||||
sheet.append([_format_cell(row.get(col)) for col in columns])
|
||||
n = len(buffered)
|
||||
buffered.clear()
|
||||
return n
|
||||
|
||||
count = 0
|
||||
async for row in store.iter_all(query):
|
||||
buffered.append(row)
|
||||
if len(buffered) >= FLUSH_AT:
|
||||
count += await flush()
|
||||
count += await flush()
|
||||
|
||||
buffer = BytesIO()
|
||||
workbook.save(buffer)
|
||||
buffer.seek(0)
|
||||
logger.info("Exported %d tool_call_metrics rows", count)
|
||||
|
||||
timestamp = datetime.utcnow().strftime("%Y%m%d-%H%M%S")
|
||||
filename = f"tool_call_metrics-{timestamp}.xlsx"
|
||||
headers = {
|
||||
"Content-Disposition": f"attachment; filename=\"{filename}\"; filename*=UTF-8''{quote(filename)}",
|
||||
}
|
||||
return StreamingResponse(
|
||||
buffer,
|
||||
media_type="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
|
||||
headers=headers,
|
||||
)
|
||||
+8
@@ -131,6 +131,14 @@ def _build_runtime_middlewares(
|
||||
|
||||
middlewares.append(SandboxAuditMiddleware())
|
||||
middlewares.append(ToolErrorHandlingMiddleware())
|
||||
|
||||
# Tool / skill call metrics — appended last so it sits *deepest* in the
|
||||
# wrap_tool_call chain: its handler is the raw tool, so it observes the
|
||||
# unconverted exception (before ToolErrorHandlingMiddleware turns it into
|
||||
# an error ToolMessage) and the raw return value.
|
||||
from deerflow.agents.middlewares.tool_metrics_middleware import ToolMetricsMiddleware
|
||||
|
||||
middlewares.append(ToolMetricsMiddleware())
|
||||
return middlewares
|
||||
|
||||
|
||||
|
||||
+319
@@ -0,0 +1,319 @@
|
||||
"""Middleware that records per-tool-call metrics for the admin audit page.
|
||||
|
||||
Every tool / skill invocation ends with one row in ``tool_call_metrics``:
|
||||
- Successes record the wall-clock duration.
|
||||
- Failures record a coarse ``error_category`` plus a one-line message.
|
||||
|
||||
Two failure shapes are captured:
|
||||
|
||||
1. **Raised exceptions** — the tool blew up. We record, then re-raise so the
|
||||
outer :class:`ToolErrorHandlingMiddleware` still converts it into an error
|
||||
``ToolMessage`` for the model.
|
||||
2. **Self-handled errors** — search/retrieval tools (Serper, Tavily, Jina, …)
|
||||
catch their own HTTP 429/403/timeout and *return* an error payload instead
|
||||
of raising. Those never reach an ``except`` block, so we also inspect the
|
||||
returned ``ToolMessage``: an ``status == "error"`` message, or content that
|
||||
looks like an error string, is bucketed too.
|
||||
|
||||
This middleware is appended **after** ``ToolErrorHandlingMiddleware`` so it
|
||||
sits deepest in the ``wrap_tool_call`` chain — its handler is the raw tool, so
|
||||
it observes the unconverted exception and the raw return value.
|
||||
|
||||
Writes go through a best-effort store: a DB outage logs a warning and the
|
||||
agent runtime keeps running.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
import uuid
|
||||
from collections.abc import Awaitable, Callable
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any, override
|
||||
|
||||
from langchain.agents import AgentState
|
||||
from langchain.agents.middleware import AgentMiddleware
|
||||
from langchain_core.messages import ToolMessage
|
||||
from langgraph.errors import GraphBubbleUp
|
||||
from langgraph.prebuilt.tool_node import ToolCallRequest
|
||||
from langgraph.types import Command
|
||||
|
||||
from deerflow.persistence.engine import get_session_factory
|
||||
from deerflow.persistence.tool_metrics import ToolMetricRecord, make_tool_metrics_store
|
||||
from deerflow.persistence.tool_metrics.base import ToolMetricsStore
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(UTC)
|
||||
|
||||
|
||||
def _trim(text: str, limit: int = 500) -> str:
|
||||
text = (text or "").strip()
|
||||
return text if len(text) <= limit else text[: limit - 3] + "..."
|
||||
|
||||
|
||||
def _classify_error(text: str) -> str:
|
||||
"""Bucket an error string into a coarse, admin-friendly category."""
|
||||
t = (text or "").lower()
|
||||
if any(k in t for k in ("429", "rate limit", "ratelimit", "too many request", "quota")):
|
||||
return "rate_limited"
|
||||
if any(k in t for k in ("403", "forbidden", "ip ban", "ip block", "blocked", "access denied", "banned")):
|
||||
return "ip_blocked"
|
||||
if any(k in t for k in ("401", "unauthorized", "api key", "apikey", "invalid key", "authentication", "credential")):
|
||||
return "auth_error"
|
||||
if any(k in t for k in ("timeout", "timed out", "deadline")):
|
||||
return "timeout"
|
||||
if any(k in t for k in ("connecterror", "connection error", "connection refused", "network", "dns", "unreachable", "ssl", "proxy")):
|
||||
return "network_error"
|
||||
if any(k in t for k in ("404", "not found")):
|
||||
return "not_found"
|
||||
if any(k in t for k in ("no results", "no result", "0 results", "empty result")):
|
||||
return "empty_result"
|
||||
return "tool_error"
|
||||
|
||||
|
||||
def _content_text(content: Any) -> str:
|
||||
"""Flatten a tool result's content (string or block list) into text."""
|
||||
if content is None:
|
||||
return ""
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
if isinstance(content, list):
|
||||
parts: list[str] = []
|
||||
for block in content:
|
||||
if isinstance(block, str):
|
||||
parts.append(block)
|
||||
elif isinstance(block, dict):
|
||||
text = block.get("text")
|
||||
if isinstance(text, str):
|
||||
parts.append(text)
|
||||
return " ".join(parts)
|
||||
return str(content)
|
||||
|
||||
|
||||
def _looks_like_error(text: str) -> bool:
|
||||
"""Heuristically detect a self-handled error payload returned as content.
|
||||
|
||||
Deliberately conservative — only an explicit ``Error:`` prefix or a JSON
|
||||
object whose first key is ``error`` counts, so a normal result that merely
|
||||
mentions the word "error" is not misclassified.
|
||||
"""
|
||||
head = (text or "").lstrip()[:64].lower()
|
||||
return head.startswith("error:") or head.startswith('{"error"') or head.startswith("{'error'")
|
||||
|
||||
|
||||
# Skill tools whose ``name`` argument names one specific Agent Skill directly.
|
||||
_SKILL_NAME_ARG_TOOLS = {"skill_view", "skill_manage", "skill_archive"}
|
||||
|
||||
# Matches a skill directory embedded in any tool-argument path. Skills live
|
||||
# under ``skills/{public,custom}/{name}/`` — both inside the sandbox
|
||||
# (``/mnt/skills/...``) and on the host. The lead agent's system prompt tells
|
||||
# it to load a skill by calling ``read_file`` on that SKILL.md path (it does
|
||||
# NOT call a dedicated "use skill" tool), so the skill name has to be recovered
|
||||
# from the path argument of whatever tool touched it (read_file / bash / ls /
|
||||
# str_replace / write_file ...).
|
||||
_SKILL_PATH_RE = re.compile(r"[\\/]skills[\\/](?:public|custom)[\\/]([^\\/]+)(?:[\\/]|$)")
|
||||
|
||||
|
||||
def _skill_name_from_call(tool_name: str | None, tool_call: Any) -> str | None:
|
||||
"""Recover the Agent Skill a tool call targets, or ``None``.
|
||||
|
||||
Two detection paths:
|
||||
|
||||
1. ``skill_view`` / ``skill_manage`` / ``skill_archive`` take an explicit
|
||||
``name`` argument.
|
||||
2. Every other tool references a skill by *path*. The canonical skill
|
||||
invocation is ``read_file('/mnt/skills/<cat>/<name>/SKILL.md')``; helper
|
||||
files and ``bash`` script runs hit the same directory. So we scan the
|
||||
string arguments for the ``skills/{public,custom}/{name}/`` shape.
|
||||
"""
|
||||
if not isinstance(tool_call, dict):
|
||||
return None
|
||||
args = tool_call.get("args")
|
||||
if not isinstance(args, dict):
|
||||
return None
|
||||
|
||||
# Case 1 — explicit ``name`` argument on the skill tools.
|
||||
if tool_name in _SKILL_NAME_ARG_TOOLS:
|
||||
raw = args.get("name")
|
||||
if isinstance(raw, str) and raw.strip():
|
||||
return raw.strip()
|
||||
|
||||
# Case 2 — a skill path inside any string argument.
|
||||
for value in args.values():
|
||||
if isinstance(value, str):
|
||||
match = _SKILL_PATH_RE.search(value)
|
||||
if match:
|
||||
return match.group(1)
|
||||
return None
|
||||
|
||||
|
||||
def _evaluate_skill_view(text: str) -> tuple[str, str | None, str | None]:
|
||||
"""Classify a ``skill_view`` return.
|
||||
|
||||
``skill_view`` never raises — on failure it returns a plain string
|
||||
(``Skill 'x' not found.`` / ``Error reading skill 'x': ...``); on success
|
||||
it returns the SKILL.md content. So success/failure must be read off the
|
||||
returned text, not an exception or ``status`` flag.
|
||||
"""
|
||||
head = (text or "").lstrip()
|
||||
low = head[:200].lower()
|
||||
if head.startswith("Skill '") and "not found" in low:
|
||||
return "error", "not_found", _trim(text)
|
||||
if low.startswith("error reading skill"):
|
||||
return "error", _classify_error(text), _trim(text)
|
||||
return "success", None, None
|
||||
|
||||
|
||||
def _evaluate_result(result: Any, tool_name: str | None) -> tuple[str, str | None, str | None]:
|
||||
"""Return ``(status, error_category, error_message)`` for a tool return."""
|
||||
if isinstance(result, ToolMessage):
|
||||
text = _content_text(result.content)
|
||||
if getattr(result, "status", None) == "error":
|
||||
return "error", _classify_error(text), _trim(text)
|
||||
# skill_view swallows its own failures into a plain string return.
|
||||
if tool_name == "skill_view":
|
||||
return _evaluate_skill_view(text)
|
||||
if _looks_like_error(text):
|
||||
return "error", _classify_error(text), _trim(text)
|
||||
return "success", None, None
|
||||
# Command (control flow) or any other shape — treat as a successful call.
|
||||
return "success", None, None
|
||||
|
||||
|
||||
class ToolMetricsMiddleware(AgentMiddleware[AgentState]):
|
||||
"""Wrap each tool call to time it and persist a metric row."""
|
||||
|
||||
def __init__(self, *, store: ToolMetricsStore | None = None, **kwargs: Any) -> None:
|
||||
super().__init__(**kwargs)
|
||||
self._store_override = store
|
||||
|
||||
# -- store / context ----------------------------------------------------
|
||||
|
||||
def _store(self) -> ToolMetricsStore | None:
|
||||
if self._store_override is not None:
|
||||
return self._store_override
|
||||
try:
|
||||
sf = get_session_factory()
|
||||
except Exception:
|
||||
return None
|
||||
return make_tool_metrics_store(sf)
|
||||
|
||||
@staticmethod
|
||||
def _runtime_lookup(request: ToolCallRequest, keys: tuple[str, ...]) -> str | None:
|
||||
runtime = getattr(request, "runtime", None)
|
||||
if runtime is None:
|
||||
return None
|
||||
context = getattr(runtime, "context", None)
|
||||
if isinstance(context, dict):
|
||||
for key in keys:
|
||||
value = context.get(key)
|
||||
if isinstance(value, str) and value:
|
||||
return value
|
||||
config = getattr(runtime, "config", None)
|
||||
if isinstance(config, dict):
|
||||
configurable = config.get("configurable") if isinstance(config.get("configurable"), dict) else {}
|
||||
metadata = config.get("metadata") if isinstance(config.get("metadata"), dict) else {}
|
||||
for bucket in (configurable, metadata):
|
||||
for key in keys:
|
||||
value = bucket.get(key)
|
||||
if isinstance(value, str) and value:
|
||||
return value
|
||||
return None
|
||||
|
||||
def _build_base_record(self, request: ToolCallRequest) -> ToolMetricRecord:
|
||||
tool_call = getattr(request, "tool_call", None) or {}
|
||||
tool_name = tool_call.get("name") if isinstance(tool_call, dict) else None
|
||||
tool_name = str(tool_name) if tool_name else None
|
||||
return ToolMetricRecord(
|
||||
id=uuid.uuid4().hex,
|
||||
created_at=_now(),
|
||||
user_id=get_effective_user_id(),
|
||||
thread_id=self._runtime_lookup(request, ("thread_id",)),
|
||||
run_id=self._runtime_lookup(request, ("run_id", "langgraph_run_id")),
|
||||
agent_name=self._runtime_lookup(request, ("agent_name", "agent_id")),
|
||||
skill_name=_skill_name_from_call(tool_name, tool_call),
|
||||
tool_name=tool_name,
|
||||
)
|
||||
|
||||
# -- finalize -----------------------------------------------------------
|
||||
|
||||
def _finalize_result(self, record: ToolMetricRecord, result: Any, started: float) -> None:
|
||||
record.duration_ms = int(max(0.0, time.monotonic() - started) * 1000)
|
||||
record.status, record.error_category, record.error_message = _evaluate_result(
|
||||
result, record.tool_name
|
||||
)
|
||||
|
||||
def _finalize_error(self, record: ToolMetricRecord, exc: BaseException, started: float) -> None:
|
||||
record.duration_ms = int(max(0.0, time.monotonic() - started) * 1000)
|
||||
record.status = "error"
|
||||
name = exc.__class__.__name__
|
||||
combined = f"{name}: {(str(exc) or name).strip()}"
|
||||
record.error_category = _classify_error(combined)
|
||||
record.error_message = _trim(combined)
|
||||
|
||||
def _flush(self, record: ToolMetricRecord) -> None:
|
||||
store = self._store()
|
||||
if store is None:
|
||||
return
|
||||
try:
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
except RuntimeError:
|
||||
loop = None
|
||||
if loop is not None and loop.is_running():
|
||||
# Fire-and-forget — never let metric IO block the tool path.
|
||||
loop.create_task(store.record(record))
|
||||
else:
|
||||
asyncio.run(store.record(record))
|
||||
except Exception:
|
||||
logger.warning("ToolMetricsMiddleware: failed to flush metric %s", record.id, exc_info=True)
|
||||
|
||||
# -- wrap ---------------------------------------------------------------
|
||||
|
||||
@override
|
||||
def wrap_tool_call(
|
||||
self,
|
||||
request: ToolCallRequest,
|
||||
handler: Callable[[ToolCallRequest], ToolMessage | Command],
|
||||
) -> ToolMessage | Command:
|
||||
record = self._build_base_record(request)
|
||||
started = time.monotonic()
|
||||
try:
|
||||
result = handler(request)
|
||||
except GraphBubbleUp:
|
||||
# LangGraph control flow — not a tool failure.
|
||||
raise
|
||||
except Exception as exc:
|
||||
self._finalize_error(record, exc, started)
|
||||
self._flush(record)
|
||||
raise
|
||||
self._finalize_result(record, result, started)
|
||||
self._flush(record)
|
||||
return result
|
||||
|
||||
@override
|
||||
async def awrap_tool_call(
|
||||
self,
|
||||
request: ToolCallRequest,
|
||||
handler: Callable[[ToolCallRequest], Awaitable[ToolMessage | Command]],
|
||||
) -> ToolMessage | Command:
|
||||
record = self._build_base_record(request)
|
||||
started = time.monotonic()
|
||||
try:
|
||||
result = await handler(request)
|
||||
except GraphBubbleUp:
|
||||
raise
|
||||
except Exception as exc:
|
||||
self._finalize_error(record, exc, started)
|
||||
self._flush(record)
|
||||
raise
|
||||
self._finalize_result(record, result, started)
|
||||
self._flush(record)
|
||||
return result
|
||||
+58
@@ -0,0 +1,58 @@
|
||||
"""Create tool_call_metrics table.
|
||||
|
||||
Revision ID: 20260518_07
|
||||
Revises: 20260518_06
|
||||
Create Date: 2026-05-18
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy import inspect
|
||||
|
||||
|
||||
revision = "20260518_07"
|
||||
down_revision = "20260518_06"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def _has_table(table_name: str) -> bool:
|
||||
inspector = inspect(op.get_bind())
|
||||
return table_name in inspector.get_table_names()
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
if _has_table("tool_call_metrics"):
|
||||
return
|
||||
|
||||
op.create_table(
|
||||
"tool_call_metrics",
|
||||
sa.Column("id", sa.String(64), primary_key=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("user_id", sa.String(64), nullable=True),
|
||||
sa.Column("thread_id", sa.String(64), nullable=True),
|
||||
sa.Column("run_id", sa.String(64), nullable=True),
|
||||
sa.Column("agent_name", sa.String(128), nullable=True),
|
||||
sa.Column("tool_name", sa.String(128), nullable=True),
|
||||
sa.Column("skill_name", sa.String(128), nullable=True),
|
||||
sa.Column("duration_ms", sa.Integer(), nullable=False, server_default="0"),
|
||||
sa.Column("status", sa.String(16), nullable=False, server_default="success"),
|
||||
sa.Column("error_category", sa.String(32), nullable=True),
|
||||
sa.Column("error_message", sa.Text(), nullable=True),
|
||||
)
|
||||
op.create_index("ix_tool_call_metrics_created_at", "tool_call_metrics", ["created_at"])
|
||||
op.create_index("ix_tool_call_metrics_user_id", "tool_call_metrics", ["user_id"])
|
||||
op.create_index("ix_tool_call_metrics_thread_id", "tool_call_metrics", ["thread_id"])
|
||||
op.create_index("ix_tool_call_metrics_run_id", "tool_call_metrics", ["run_id"])
|
||||
op.create_index("ix_tool_call_metrics_agent_name", "tool_call_metrics", ["agent_name"])
|
||||
op.create_index("ix_tool_call_metrics_tool_name", "tool_call_metrics", ["tool_name"])
|
||||
op.create_index("ix_tool_call_metrics_skill_name", "tool_call_metrics", ["skill_name"])
|
||||
op.create_index("ix_tool_call_metrics_status", "tool_call_metrics", ["status"])
|
||||
op.create_index("ix_tool_call_metrics_error_category", "tool_call_metrics", ["error_category"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
if _has_table("tool_call_metrics"):
|
||||
op.drop_table("tool_call_metrics")
|
||||
+62
@@ -0,0 +1,62 @@
|
||||
"""Add skill_name column to tool_call_metrics.
|
||||
|
||||
``tool_call_metrics`` is created by ``create_all`` at startup. On deployments
|
||||
where the table was materialised *before* the ``skill_name`` column existed,
|
||||
neither ``create_all`` (it never ALTERs) nor revision ``20260518_07`` (it skips
|
||||
when the table already exists) adds the column — leaving MySQL/Postgres with a
|
||||
schema that the ORM model expects a column it does not have. This revision
|
||||
patches the existing table.
|
||||
|
||||
Revision ID: 20260519_01
|
||||
Revises: 20260518_07
|
||||
Create Date: 2026-05-19
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy import inspect
|
||||
|
||||
|
||||
revision = "20260519_01"
|
||||
down_revision = "20260518_07"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def _has_table(table_name: str) -> bool:
|
||||
inspector = inspect(op.get_bind())
|
||||
return table_name in inspector.get_table_names()
|
||||
|
||||
|
||||
def _has_column(table_name: str, column_name: str) -> bool:
|
||||
if not _has_table(table_name):
|
||||
return False
|
||||
inspector = inspect(op.get_bind())
|
||||
return any(col["name"] == column_name for col in inspector.get_columns(table_name))
|
||||
|
||||
|
||||
def _has_index(table_name: str, index_name: str) -> bool:
|
||||
if not _has_table(table_name):
|
||||
return False
|
||||
inspector = inspect(op.get_bind())
|
||||
return any(idx["name"] == index_name for idx in inspector.get_indexes(table_name))
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# Brand-new DBs: the table is created with the column already present
|
||||
# (by create_all / revision 20260518_07) — nothing to patch.
|
||||
if not _has_table("tool_call_metrics"):
|
||||
return
|
||||
if not _has_column("tool_call_metrics", "skill_name"):
|
||||
op.add_column("tool_call_metrics", sa.Column("skill_name", sa.String(128), nullable=True))
|
||||
if not _has_index("tool_call_metrics", "ix_tool_call_metrics_skill_name"):
|
||||
op.create_index("ix_tool_call_metrics_skill_name", "tool_call_metrics", ["skill_name"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
if _has_index("tool_call_metrics", "ix_tool_call_metrics_skill_name"):
|
||||
op.drop_index("ix_tool_call_metrics_skill_name", table_name="tool_call_metrics")
|
||||
if _has_column("tool_call_metrics", "skill_name"):
|
||||
op.drop_column("tool_call_metrics", "skill_name")
|
||||
+2
@@ -33,6 +33,7 @@ from deerflow.persistence.skills.model import SkillRow
|
||||
from deerflow.persistence.tags.model import TagAssignmentRow, TagRow
|
||||
from deerflow.persistence.thread_meta.model import ThreadMetaRow
|
||||
from deerflow.persistence.thread_shares.model import ThreadShareRow
|
||||
from deerflow.persistence.tool_metrics.model import ToolCallMetricRow
|
||||
from deerflow.persistence.user.model import UserRow
|
||||
|
||||
__all__ = [
|
||||
@@ -53,5 +54,6 @@ __all__ = [
|
||||
"TagRow",
|
||||
"ThreadMetaRow",
|
||||
"ThreadShareRow",
|
||||
"ToolCallMetricRow",
|
||||
"UserRow",
|
||||
]
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
"""Persistence layer for per-tool-call (skill-call) metrics.
|
||||
|
||||
Every tool invocation — built-in tools, skills, MCP tools, subagent calls —
|
||||
writes one row recording duration and outcome. Failures are bucketed into a
|
||||
coarse ``error_category`` (rate_limited / ip_blocked / auth_error / timeout /
|
||||
network_error / not_found / empty_result / tool_error) so administrators can
|
||||
spot retrieval outages, blocked IPs, and quota problems at a glance.
|
||||
"""
|
||||
|
||||
from deerflow.persistence.tool_metrics.base import ToolMetricRecord, ToolMetricsStore
|
||||
from deerflow.persistence.tool_metrics.model import ToolCallMetricRow
|
||||
from deerflow.persistence.tool_metrics.sql import SqlToolMetricsStore
|
||||
|
||||
__all__ = [
|
||||
"ToolCallMetricRow",
|
||||
"ToolMetricRecord",
|
||||
"ToolMetricsStore",
|
||||
"SqlToolMetricsStore",
|
||||
"make_tool_metrics_store",
|
||||
]
|
||||
|
||||
|
||||
def make_tool_metrics_store(session_factory) -> ToolMetricsStore | None:
|
||||
"""Return a SQL-backed metrics store, or ``None`` when no DB is configured."""
|
||||
if session_factory is None:
|
||||
return None
|
||||
return SqlToolMetricsStore(session_factory)
|
||||
+84
@@ -0,0 +1,84 @@
|
||||
"""Abstract repository contract for tool / skill call metrics."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
|
||||
@dataclass
|
||||
class ToolMetricRecord:
|
||||
"""In-memory shape of a tool-call metric row before persistence."""
|
||||
|
||||
id: str
|
||||
created_at: datetime
|
||||
user_id: str | None = None
|
||||
thread_id: str | None = None
|
||||
run_id: str | None = None
|
||||
agent_name: str | None = None
|
||||
tool_name: str | None = None
|
||||
# The specific Agent Skill this call targets, when the tool is a skill tool
|
||||
# (``skill_view`` / ``skill_manage`` / ``skill_archive``) — taken from the
|
||||
# call's ``name`` argument. Null for every non-skill tool call. This is
|
||||
# what lets the admin see "skill X was invoked, and succeeded/failed".
|
||||
skill_name: str | None = None
|
||||
duration_ms: int = 0
|
||||
status: str = "success"
|
||||
# Coarse failure bucket: rate_limited / ip_blocked / auth_error / timeout /
|
||||
# network_error / not_found / empty_result / tool_error. Null on success.
|
||||
error_category: str | None = None
|
||||
error_message: str | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class ToolMetricsQuery:
|
||||
"""Filter set for the admin list / export endpoints."""
|
||||
|
||||
user_id: str | None = None
|
||||
tool_name: str | None = None
|
||||
# When True, keep only rows that target a specific Agent Skill
|
||||
# (``skill_name IS NOT NULL``) — the ``kind=skill`` filter.
|
||||
skill_only: bool = False
|
||||
# Filter to one specific skill by name.
|
||||
skill_name: str | None = None
|
||||
status: str | None = None
|
||||
error_category: str | None = None
|
||||
since: datetime | None = None
|
||||
until: datetime | None = None
|
||||
limit: int = 100
|
||||
offset: int = 0
|
||||
|
||||
|
||||
class ToolMetricsStore(ABC):
|
||||
"""Persistence contract for the tool-metrics middleware + admin API."""
|
||||
|
||||
@abstractmethod
|
||||
async def record(self, metric: ToolMetricRecord) -> None:
|
||||
"""Persist one metric row. Must never raise on best-effort writes."""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def list(self, query: ToolMetricsQuery) -> list[dict[str, Any]]:
|
||||
"""Return the matching rows (newest first), paginated by ``query``."""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def count(self, query: ToolMetricsQuery) -> int:
|
||||
"""Return the unpaginated row count for ``query``."""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def iter_all(self, query: ToolMetricsQuery):
|
||||
"""Yield rows matching ``query`` without pagination — used by export."""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def skill_breakdown(self, query: ToolMetricsQuery) -> list[dict[str, Any]]:
|
||||
"""Per-skill aggregate: ``[{skill_name, total, success, error}, ...]``.
|
||||
|
||||
Only rows with a non-null ``skill_name`` are counted, so this answers
|
||||
"which skills were invoked, and how often did each succeed/fail".
|
||||
"""
|
||||
raise NotImplementedError
|
||||
+59
@@ -0,0 +1,59 @@
|
||||
"""ORM model for per-tool-call (skill-call) metrics."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from sqlalchemy import Integer, String, Text
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from deerflow.persistence.base import Base
|
||||
from deerflow.persistence.types import BeijingDateTime
|
||||
|
||||
|
||||
class ToolCallMetricRow(Base):
|
||||
"""One row per completed (or failed) tool / skill invocation.
|
||||
|
||||
Unlike :class:`LlmCallMetricRow` this also captures *self-handled*
|
||||
failures — search tools that swallow an HTTP 429/403 and return an error
|
||||
payload instead of raising — so an admin can see "retrieval failed" /
|
||||
"IP blocked" events that never surfaced as exceptions.
|
||||
"""
|
||||
|
||||
__tablename__ = "tool_call_metrics"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), default=lambda: datetime.now(UTC), index=True)
|
||||
|
||||
user_id: Mapped[str | None] = mapped_column(String(64), index=True)
|
||||
thread_id: Mapped[str | None] = mapped_column(String(64), index=True)
|
||||
run_id: Mapped[str | None] = mapped_column(String(64), index=True)
|
||||
agent_name: Mapped[str | None] = mapped_column(String(128), index=True)
|
||||
tool_name: Mapped[str | None] = mapped_column(String(128), index=True)
|
||||
# The specific Agent Skill targeted by a skill tool call (from the call's
|
||||
# ``name`` argument); null for non-skill tool calls.
|
||||
skill_name: Mapped[str | None] = mapped_column(String(128), index=True)
|
||||
|
||||
duration_ms: Mapped[int] = mapped_column(Integer, default=0)
|
||||
|
||||
# "success" or "error".
|
||||
status: Mapped[str] = mapped_column(String(16), default="success", index=True)
|
||||
# Coarse failure bucket; null on success.
|
||||
error_category: Mapped[str | None] = mapped_column(String(32), index=True)
|
||||
error_message: Mapped[str | None] = mapped_column(Text)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"id": self.id,
|
||||
"created_at": self.created_at.isoformat() if isinstance(self.created_at, datetime) else self.created_at,
|
||||
"user_id": self.user_id,
|
||||
"thread_id": self.thread_id,
|
||||
"run_id": self.run_id,
|
||||
"agent_name": self.agent_name,
|
||||
"tool_name": self.tool_name,
|
||||
"skill_name": self.skill_name,
|
||||
"duration_ms": self.duration_ms,
|
||||
"status": self.status,
|
||||
"error_category": self.error_category,
|
||||
"error_message": self.error_message,
|
||||
}
|
||||
+138
@@ -0,0 +1,138 @@
|
||||
"""SQLAlchemy-backed tool metrics repository."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from collections.abc import AsyncIterator
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import desc, func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
from deerflow.persistence.tool_metrics.base import ToolMetricRecord, ToolMetricsQuery, ToolMetricsStore
|
||||
from deerflow.persistence.tool_metrics.model import ToolCallMetricRow
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _normalize_created_at(value: datetime) -> datetime:
|
||||
return value if value.tzinfo else value.replace(tzinfo=UTC)
|
||||
|
||||
|
||||
class SqlToolMetricsStore(ToolMetricsStore):
|
||||
"""SQLite/Postgres/MySQL-backed implementation; writes are fire-and-forget."""
|
||||
|
||||
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
|
||||
self._sf = session_factory
|
||||
|
||||
async def record(self, metric: ToolMetricRecord) -> None:
|
||||
row = ToolCallMetricRow(
|
||||
id=metric.id,
|
||||
created_at=_normalize_created_at(metric.created_at),
|
||||
user_id=metric.user_id,
|
||||
thread_id=metric.thread_id,
|
||||
run_id=metric.run_id,
|
||||
agent_name=metric.agent_name,
|
||||
tool_name=metric.tool_name,
|
||||
skill_name=metric.skill_name,
|
||||
duration_ms=metric.duration_ms,
|
||||
status=metric.status,
|
||||
error_category=metric.error_category,
|
||||
error_message=metric.error_message,
|
||||
)
|
||||
try:
|
||||
async with self._sf() as session:
|
||||
session.add(row)
|
||||
await session.commit()
|
||||
except Exception:
|
||||
# Metrics are best-effort; never surface a write failure to the runtime.
|
||||
logger.warning("Failed to persist tool_call_metrics row %s", metric.id, exc_info=True)
|
||||
|
||||
def _apply_filters(self, stmt, query: ToolMetricsQuery):
|
||||
if query.user_id:
|
||||
stmt = stmt.where(ToolCallMetricRow.user_id == query.user_id)
|
||||
if query.tool_name:
|
||||
stmt = stmt.where(ToolCallMetricRow.tool_name == query.tool_name)
|
||||
if query.skill_only:
|
||||
stmt = stmt.where(ToolCallMetricRow.skill_name.is_not(None))
|
||||
if query.skill_name:
|
||||
stmt = stmt.where(ToolCallMetricRow.skill_name == query.skill_name)
|
||||
if query.status:
|
||||
stmt = stmt.where(ToolCallMetricRow.status == query.status)
|
||||
if query.error_category:
|
||||
stmt = stmt.where(ToolCallMetricRow.error_category == query.error_category)
|
||||
if query.since is not None:
|
||||
stmt = stmt.where(ToolCallMetricRow.created_at >= _normalize_created_at(query.since))
|
||||
if query.until is not None:
|
||||
stmt = stmt.where(ToolCallMetricRow.created_at <= _normalize_created_at(query.until))
|
||||
return stmt
|
||||
|
||||
async def list(self, query: ToolMetricsQuery) -> list[dict[str, Any]]:
|
||||
stmt = select(ToolCallMetricRow)
|
||||
stmt = self._apply_filters(stmt, query)
|
||||
stmt = stmt.order_by(desc(ToolCallMetricRow.created_at)).limit(max(1, min(query.limit, 1000))).offset(max(0, query.offset))
|
||||
async with self._sf() as session:
|
||||
result = await session.execute(stmt)
|
||||
return [row.to_dict() for row in result.scalars()]
|
||||
|
||||
async def count(self, query: ToolMetricsQuery) -> int:
|
||||
stmt = select(func.count()).select_from(ToolCallMetricRow)
|
||||
stmt = self._apply_filters(stmt, query)
|
||||
async with self._sf() as session:
|
||||
return (await session.scalar(stmt)) or 0
|
||||
|
||||
async def iter_all(self, query: ToolMetricsQuery) -> AsyncIterator[dict[str, Any]]:
|
||||
page_size = 500
|
||||
offset = 0
|
||||
while True:
|
||||
page = await self.list(
|
||||
ToolMetricsQuery(
|
||||
user_id=query.user_id,
|
||||
tool_name=query.tool_name,
|
||||
skill_only=query.skill_only,
|
||||
skill_name=query.skill_name,
|
||||
status=query.status,
|
||||
error_category=query.error_category,
|
||||
since=query.since,
|
||||
until=query.until,
|
||||
limit=page_size,
|
||||
offset=offset,
|
||||
)
|
||||
)
|
||||
if not page:
|
||||
return
|
||||
for item in page:
|
||||
yield item
|
||||
if len(page) < page_size:
|
||||
return
|
||||
offset += page_size
|
||||
|
||||
async def skill_breakdown(self, query: ToolMetricsQuery) -> list[dict[str, Any]]:
|
||||
# Count success/error per skill. Force the skill-only filter so rows
|
||||
# without a skill_name (plain tool calls) never leak into the result.
|
||||
scoped = ToolMetricsQuery(
|
||||
user_id=query.user_id,
|
||||
skill_only=True,
|
||||
skill_name=query.skill_name,
|
||||
since=query.since,
|
||||
until=query.until,
|
||||
)
|
||||
stmt = select(
|
||||
ToolCallMetricRow.skill_name,
|
||||
ToolCallMetricRow.status,
|
||||
func.count().label("n"),
|
||||
)
|
||||
stmt = self._apply_filters(stmt, scoped).group_by(
|
||||
ToolCallMetricRow.skill_name, ToolCallMetricRow.status
|
||||
)
|
||||
agg: dict[str, dict[str, int]] = {}
|
||||
async with self._sf() as session:
|
||||
for skill_name, status, n in (await session.execute(stmt)).all():
|
||||
bucket = agg.setdefault(skill_name, {"total": 0, "success": 0, "error": 0})
|
||||
bucket["total"] += n
|
||||
bucket["success" if status == "success" else "error"] += n
|
||||
return [
|
||||
{"skill_name": name, **counts}
|
||||
for name, counts in sorted(agg.items(), key=lambda kv: kv[1]["total"], reverse=True)
|
||||
]
|
||||
@@ -0,0 +1,264 @@
|
||||
"""Drop the stale business tables from the SQLite file after the MySQL switch.
|
||||
|
||||
Context
|
||||
-------
|
||||
In SQLite mode the LangGraph checkpointer **and** the application share one
|
||||
file: ``.deer-flow/data/deerflow.db``. After ``database.backend`` is switched
|
||||
to ``mysql`` (and ``migrate_business_sqlite_to_mysql.py`` has copied the data
|
||||
over), the business tables in ``deerflow.db`` are dead weight — the app now
|
||||
reads them from MySQL — while the LangGraph checkpoint tables in the *same*
|
||||
file are still actively used (``checkpointer.type`` stays ``sqlite``).
|
||||
|
||||
This script drops only the stale business tables and ``VACUUM``s to reclaim
|
||||
the freed pages.
|
||||
|
||||
Safety design
|
||||
-------------
|
||||
* **Explicit DROP allowlist** — only the known business tables are dropped.
|
||||
LangGraph tables (``checkpoints`` / ``writes`` / ``store`` /
|
||||
``store_migrations``) and any unrecognised table are left untouched.
|
||||
* **MySQL pre-check** — before dropping, it connects to MySQL and verifies
|
||||
every business table exists there. A missing table aborts the run (the
|
||||
migration looks incomplete). Skippable only via an explicit flag.
|
||||
* **Dry-run by default** — nothing is changed unless ``--apply`` is passed.
|
||||
* **Automatic backup** — ``--apply`` first copies ``deerflow.db`` to a
|
||||
timestamped ``.bak`` file (disable with ``--no-backup``).
|
||||
|
||||
IMPORTANT: stop the backend first. ``DROP TABLE`` / ``VACUUM`` need an
|
||||
exclusive lock on the SQLite file; with the server running they will fail
|
||||
with "database is locked" (a safe failure — nothing is changed).
|
||||
|
||||
Usage
|
||||
-----
|
||||
# from backend/ , with the backend's interpreter (has the MySQL driver)
|
||||
PYTHONPATH=. uv run python scripts/cleanup_sqlite_after_mysql_migration.py # dry-run
|
||||
PYTHONPATH=. uv run python scripts/cleanup_sqlite_after_mysql_migration.py --apply # do it
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import sqlite3
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
# Business tables migrated to MySQL — identical to BUSINESS_TABLES in
|
||||
# migrate_business_sqlite_to_mysql.py. These are the DROP allowlist.
|
||||
BUSINESS_TABLES: list[str] = [
|
||||
"users",
|
||||
"threads_meta",
|
||||
"runs",
|
||||
"run_events",
|
||||
"feedback",
|
||||
"agents",
|
||||
"skills",
|
||||
"scheduled_tasks",
|
||||
"scheduled_task_subscriptions",
|
||||
"scheduled_task_runs",
|
||||
"scheduled_task_delivery_profiles",
|
||||
"scheduler_threads",
|
||||
"llm_call_metrics",
|
||||
"notifications",
|
||||
"recommended_questions",
|
||||
"tags",
|
||||
"tag_assignments",
|
||||
"thread_shares",
|
||||
]
|
||||
|
||||
# Newer business tables that may exist depending on deployment age. Dropped if
|
||||
# present; absence is fine. They are NOT checked against MySQL (a fresh MySQL
|
||||
# install creates them via create_all / Alembic, but an older SQLite file may
|
||||
# never have had them).
|
||||
OPTIONAL_BUSINESS_TABLES: list[str] = [
|
||||
"ai_writing_sessions",
|
||||
"article_types",
|
||||
"tool_call_metrics",
|
||||
]
|
||||
|
||||
# SQLite-side Alembic version tracker. After the MySQL switch the SQLite file
|
||||
# only serves the checkpointer, which does not use Alembic — so this row is
|
||||
# stale. Dropped, but never MySQL-checked.
|
||||
ALEMBIC_TABLE = "alembic_version"
|
||||
|
||||
# LangGraph infrastructure tables — these stay on SQLite and must NEVER be
|
||||
# dropped. The checkpointer/store keep using them.
|
||||
KEEP_TABLES: set[str] = {"checkpoints", "writes", "store", "store_migrations"}
|
||||
|
||||
|
||||
def _repo_root_from_script() -> Path:
|
||||
# scripts/ -> backend/ -> offline-backend-20260512/
|
||||
return Path(__file__).resolve().parents[2]
|
||||
|
||||
|
||||
def _load_mysql_url(explicit: str | None) -> str | None:
|
||||
if explicit:
|
||||
return explicit
|
||||
env = os.environ.get("MYSQL_DATABASE_URL")
|
||||
if env:
|
||||
return env
|
||||
# Fall back to the .env next to the backend directory.
|
||||
for candidate in (
|
||||
_repo_root_from_script() / ".env",
|
||||
Path(__file__).resolve().parents[1] / ".env",
|
||||
):
|
||||
if candidate.is_file():
|
||||
for line in candidate.read_text(encoding="utf-8").splitlines():
|
||||
if line.startswith("MYSQL_DATABASE_URL="):
|
||||
return line.split("=", 1)[1].strip().strip('"').strip("'")
|
||||
return None
|
||||
|
||||
|
||||
def _normalize_async_url(url: str) -> str:
|
||||
"""Force the URL onto the asyncmy async driver SQLAlchemy understands."""
|
||||
if url.startswith(("mysql+asyncmy://", "mysql+aiomysql://")):
|
||||
return url
|
||||
return re.sub(r"^mysql(\+\w+)?://", "mysql+asyncmy://", url)
|
||||
|
||||
|
||||
def _default_sqlite_path() -> Path:
|
||||
return _repo_root_from_script() / "backend" / ".deer-flow" / "data" / "deerflow.db"
|
||||
|
||||
|
||||
async def _mysql_existing_tables(url: str) -> set[str]:
|
||||
"""Return the set of table names present in the MySQL business database."""
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.engine import make_url
|
||||
from sqlalchemy.ext.asyncio import create_async_engine
|
||||
|
||||
async_url = _normalize_async_url(url)
|
||||
db_name = make_url(async_url).database
|
||||
engine = create_async_engine(async_url)
|
||||
try:
|
||||
async with engine.connect() as conn:
|
||||
rows = await conn.execute(
|
||||
text("SELECT table_name FROM information_schema.tables WHERE table_schema = :s"),
|
||||
{"s": db_name},
|
||||
)
|
||||
return {r[0] for r in rows}
|
||||
finally:
|
||||
await engine.dispose()
|
||||
|
||||
|
||||
def _sqlite_tables(con: sqlite3.Connection) -> list[str]:
|
||||
return [r[0] for r in con.execute("SELECT name FROM sqlite_master WHERE type='table' ORDER BY name")]
|
||||
|
||||
|
||||
def _row_count(con: sqlite3.Connection, table: str) -> int:
|
||||
return con.execute(f'SELECT COUNT(*) FROM "{table}"').fetchone()[0]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
parser.add_argument("--sqlite", default=str(_default_sqlite_path()), help="Path to deerflow.db")
|
||||
parser.add_argument("--mysql-url", default=None, help="MySQL URL (default: $MYSQL_DATABASE_URL or .env)")
|
||||
parser.add_argument("--apply", action="store_true", help="Actually drop tables + VACUUM (default: dry-run)")
|
||||
parser.add_argument("--no-backup", action="store_true", help="Skip the pre-apply backup copy")
|
||||
parser.add_argument("--skip-mysql-check", action="store_true", help="Do NOT verify tables exist in MySQL (unsafe)")
|
||||
parser.add_argument("--drop-alembic", action="store_true", help="Also drop the stale SQLite alembic_version table")
|
||||
args = parser.parse_args()
|
||||
|
||||
sqlite_path = Path(args.sqlite).resolve()
|
||||
if not sqlite_path.is_file():
|
||||
print(f"ERROR: SQLite file not found: {sqlite_path}")
|
||||
return 1
|
||||
|
||||
size_before = sqlite_path.stat().st_size
|
||||
print(f"SQLite file : {sqlite_path}")
|
||||
print(f"Size : {size_before / 1024 / 1024:.1f} MB")
|
||||
print(f"Mode : {'APPLY' if args.apply else 'DRY-RUN'}\n")
|
||||
|
||||
con = sqlite3.connect(sqlite_path)
|
||||
try:
|
||||
present = set(_sqlite_tables(con))
|
||||
|
||||
drop_business = [t for t in BUSINESS_TABLES if t in present]
|
||||
drop_optional = [t for t in OPTIONAL_BUSINESS_TABLES if t in present]
|
||||
to_drop = drop_business + drop_optional
|
||||
if args.drop_alembic and ALEMBIC_TABLE in present:
|
||||
to_drop.append(ALEMBIC_TABLE)
|
||||
|
||||
keep = sorted(present & KEEP_TABLES)
|
||||
known = set(BUSINESS_TABLES) | set(OPTIONAL_BUSINESS_TABLES) | KEEP_TABLES | {ALEMBIC_TABLE}
|
||||
unknown = sorted(present - known)
|
||||
|
||||
missing_business = [t for t in BUSINESS_TABLES if t not in present]
|
||||
if missing_business:
|
||||
print(f"NOTE: business tables already absent from SQLite: {', '.join(missing_business)}\n")
|
||||
|
||||
print("KEEP (LangGraph checkpointer / store - never dropped):")
|
||||
for t in keep:
|
||||
print(f" + {t:<34}{_row_count(con, t):>10} rows")
|
||||
if unknown:
|
||||
print("\nUNKNOWN (not recognised - left untouched, review manually):")
|
||||
for t in unknown:
|
||||
print(f" ? {t:<34}{_row_count(con, t):>10} rows")
|
||||
print("\nDROP (stale business tables):")
|
||||
for t in to_drop:
|
||||
print(f" - {t:<34}{_row_count(con, t):>10} rows")
|
||||
if not to_drop:
|
||||
print(" (nothing to drop - already clean)")
|
||||
return 0
|
||||
|
||||
# --- MySQL safety pre-check -------------------------------------
|
||||
if args.skip_mysql_check:
|
||||
print("\nWARNING: --skip-mysql-check set - NOT verifying the MySQL side.")
|
||||
else:
|
||||
mysql_url = _load_mysql_url(args.mysql_url)
|
||||
if not mysql_url:
|
||||
print("\nERROR: no MySQL URL ($MYSQL_DATABASE_URL / .env / --mysql-url).")
|
||||
print(" Pass --skip-mysql-check to bypass (only if you are certain).")
|
||||
return 1
|
||||
print("\nVerifying business tables exist in MySQL ...")
|
||||
try:
|
||||
mysql_tables = asyncio.run(_mysql_existing_tables(mysql_url))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
print(f"ERROR: could not query MySQL: {exc}")
|
||||
print(" Run this on the backend host (MySQL access is host-restricted),")
|
||||
print(" or pass --skip-mysql-check to bypass (only if you are certain).")
|
||||
return 1
|
||||
missing_in_mysql = [t for t in drop_business if t not in mysql_tables]
|
||||
if missing_in_mysql:
|
||||
print(f"ABORT: these business tables are MISSING in MySQL: {', '.join(missing_in_mysql)}")
|
||||
print(" The migration looks incomplete - not dropping anything.")
|
||||
return 1
|
||||
print(f"OK - all {len(drop_business)} business tables present in MySQL.")
|
||||
|
||||
if not args.apply:
|
||||
print("\nDry-run only. Re-run with --apply to drop the tables and VACUUM.")
|
||||
return 0
|
||||
|
||||
# --- apply ------------------------------------------------------
|
||||
if not args.no_backup:
|
||||
backup = sqlite_path.with_name(
|
||||
f"{sqlite_path.name}.bak-{datetime.now():%Y%m%d-%H%M%S}"
|
||||
)
|
||||
print(f"\nBacking up -> {backup}")
|
||||
shutil.copy2(sqlite_path, backup)
|
||||
|
||||
print("Dropping tables ...")
|
||||
for t in to_drop:
|
||||
con.execute(f'DROP TABLE IF EXISTS "{t}"')
|
||||
con.commit()
|
||||
|
||||
print("VACUUM (reclaiming space) ...")
|
||||
con.isolation_level = None # VACUUM cannot run inside a transaction
|
||||
con.execute("VACUUM")
|
||||
finally:
|
||||
con.close()
|
||||
|
||||
size_after = sqlite_path.stat().st_size
|
||||
print(
|
||||
f"\nDone. Size {size_before / 1024 / 1024:.1f} MB "
|
||||
f"-> {size_after / 1024 / 1024:.1f} MB "
|
||||
f"(reclaimed {(size_before - size_after) / 1024 / 1024:.1f} MB)"
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in new issue
Block a user