Merge branch 'ji' into master
实时并发监控 + 真正停止对话 + 409 中文化(管理员排行榜)
This commit is contained in:
commit
9ef8b1990e
17 files changed
+1488
-1
No files matched your search
@@ -86,6 +86,8 @@ The frontend (`frontend-web/`) is a React 19 + Vite SPA:
|
|||||||
- **敏感词管理 (Sensitive Word Management)**: configures the **展示层脱敏映射** (原始词 → 替换词). **Gated on the username `lqq`** (not the admin role) — `getAccountDisplayName().toLowerCase() === "lqq"` (username = email `@` prefix, else `user_id`, same as the profile page); the dropdown entry, the route guard, and the page-level guard all use this check. `src/lib/desensitize.ts` ships a frozen seed map `SEED_DESENSITIZE_MAP` (the deployment seed) plus a live mutable `DESENSITIZE_MAP` (initialized from the seed so `desensitize()` works before any fetch). `DesensitizeWordsLoader` (mounted in `App.tsx` `AppProviders`) fetches `/api/sensitive-words` on startup and, if non-empty, calls `replaceDesensitizeMap(...)` to make the backend config the source of truth (empty/failed → keep seed). Page `SensitiveWordsPage.tsx` (route `/page/strategy/admin/sensitive-words`, opened from the 「敏感词管理」 item in the **bottom-left settings dropdown** of `components/page-sidebar.tsx`, present in **both** full + collapsed sidebar modes) lists/adds/edits/启停/deletes words and saves the **whole set atomically** (replace-all); a 「导入种子词」 button merges any missing `SEED_DESENSITIZE_MAP` entries into the editor. Client + hooks in `strategy-components/api/sensitive-words.ts` (`wordsToMap` folds enabled rows into the runtime map). Backend: `/api/sensitive-words` (global/shared; open read, write allowed for admin **or** the `lqq` account), mirroring 菜单管理's store shape. **侧边栏 + 导航栏 menu labels are desensitized** at the data source: `useSidebarMenuItems`/`useActiveThirdLevel` (`core/page-layout/use-sidebar-menu.ts`) map every node's `label` through a desensitizer built **directly from the `/api/sensitive-words` query** (`makeDesensitizer(wordsToMap(words))`, falling back to the global seed `desensitize` while empty) — this sidesteps the `replaceDesensitizeMap` effect-timing race. Only `label` is rewritten; `path`/`id`/`icon` are untouched so routing/highlight are unaffected. All nav surfaces (`page-sidebar.tsx`, `page-sidebar-dark.tsx`, `page-sidebar-v1.tsx`, `strategy-components/Header.tsx`, `PageRoutes.tsx`) consume this hook and are covered; `MenuManagementPage` uses the raw tree so admins still edit real labels.
|
- **敏感词管理 (Sensitive Word Management)**: configures the **展示层脱敏映射** (原始词 → 替换词). **Gated on the username `lqq`** (not the admin role) — `getAccountDisplayName().toLowerCase() === "lqq"` (username = email `@` prefix, else `user_id`, same as the profile page); the dropdown entry, the route guard, and the page-level guard all use this check. `src/lib/desensitize.ts` ships a frozen seed map `SEED_DESENSITIZE_MAP` (the deployment seed) plus a live mutable `DESENSITIZE_MAP` (initialized from the seed so `desensitize()` works before any fetch). `DesensitizeWordsLoader` (mounted in `App.tsx` `AppProviders`) fetches `/api/sensitive-words` on startup and, if non-empty, calls `replaceDesensitizeMap(...)` to make the backend config the source of truth (empty/failed → keep seed). Page `SensitiveWordsPage.tsx` (route `/page/strategy/admin/sensitive-words`, opened from the 「敏感词管理」 item in the **bottom-left settings dropdown** of `components/page-sidebar.tsx`, present in **both** full + collapsed sidebar modes) lists/adds/edits/启停/deletes words and saves the **whole set atomically** (replace-all); a 「导入种子词」 button merges any missing `SEED_DESENSITIZE_MAP` entries into the editor. Client + hooks in `strategy-components/api/sensitive-words.ts` (`wordsToMap` folds enabled rows into the runtime map). Backend: `/api/sensitive-words` (global/shared; open read, write allowed for admin **or** the `lqq` account), mirroring 菜单管理's store shape. **侧边栏 + 导航栏 menu labels are desensitized** at the data source: `useSidebarMenuItems`/`useActiveThirdLevel` (`core/page-layout/use-sidebar-menu.ts`) map every node's `label` through a desensitizer built **directly from the `/api/sensitive-words` query** (`makeDesensitizer(wordsToMap(words))`, falling back to the global seed `desensitize` while empty) — this sidesteps the `replaceDesensitizeMap` effect-timing race. Only `label` is rewritten; `path`/`id`/`icon` are untouched so routing/highlight are unaffected. All nav surfaces (`page-sidebar.tsx`, `page-sidebar-dark.tsx`, `page-sidebar-v1.tsx`, `strategy-components/Header.tsx`, `PageRoutes.tsx`) consume this hook and are covered; `MenuManagementPage` uses the raw tree so admins still edit real labels.
|
||||||
- **技能地址管理 (Skill Address Management, admin-only)**: 扫描运行中的技能(`skills/{public,custom}` 下所有 `.md`/`.py`)里的 **http(s) URL 与 IP/IP:端口**,按唯一地址聚合全部出现位置(跨 .md/.py),支持**批量替换**(如把 UAT 地址一次性改成生产)。主体抽成可复用组件 `SkillAddressesPanel`(`pages/SkillAddressesPage.tsx` 导出,默认导出 `SkillAddressesPage` 仍保留旧路由 `/page/strategy/admin/skill-addresses` 作直链回退),**作为「技能地址管理」标签页嵌入管理员「技能管理」页 `pages/AdminSkillsPage.tsx`**(该页加 Tabs:「技能列表」+「技能地址管理」;入口在「设置和更多」下拉的「技能管理」`workspace-nav-menu.tsx` → `/page/workspace/admin/skills`,不再单列于 `components/page-sidebar.tsx` 左下角设置下拉):打开自动扫描并带**进度条**(扫描逐技能 / 应用逐文件 X/N,应对 150+ 技能),URL/IP 类型过滤 + 搜索 + 「片段批量套用」(如所有 `uat.4.cn` → `prod.4.cn`)、每地址「替换为」输入框(可展开看 `file:line` 片段)、内联**预览**、**应用**、**多关键词检索**(空格分隔全部命中),以及**修改记录**面板——每次应用生成一条记录(显示时间/改了哪些地址 from→to/文件数/处数),可**针对单次记录回退**(二次确认)。client + NDJSON 流式解析在 `strategy-components/api/skill-addresses.ts`。后端 `/api/skill-addresses`(**仅管理员**):所有文件 IO 走 `asyncio.to_thread` 不阻塞事件循环;`GET /scan`(+`/scan/stream`)、`POST /preview`、`POST /apply`(+`/apply/stream`)、`GET /backups`、`POST /rollback/{id}`;**子串安全替换**(基于 token span,不会误伤子串)、`expected_total` 乐观锁(409)、写前自动磁盘备份到 `{skills_root}/.address-edits/`(同时写一份 `{id}.manifest.json` 记录本次替换明细,供修改记录展示)、应用后刷新技能系统提示缓存。纯引擎 `deerflow/skills/address_scan.py`,router `app/gateway/routers/skill_addresses.py`,tests `tests/test_skill_addresses.py`。范围仅运行中技能,不动 `deploy/minimal/skills/`。
|
- **技能地址管理 (Skill Address Management, admin-only)**: 扫描运行中的技能(`skills/{public,custom}` 下所有 `.md`/`.py`)里的 **http(s) URL 与 IP/IP:端口**,按唯一地址聚合全部出现位置(跨 .md/.py),支持**批量替换**(如把 UAT 地址一次性改成生产)。主体抽成可复用组件 `SkillAddressesPanel`(`pages/SkillAddressesPage.tsx` 导出,默认导出 `SkillAddressesPage` 仍保留旧路由 `/page/strategy/admin/skill-addresses` 作直链回退),**作为「技能地址管理」标签页嵌入管理员「技能管理」页 `pages/AdminSkillsPage.tsx`**(该页加 Tabs:「技能列表」+「技能地址管理」;入口在「设置和更多」下拉的「技能管理」`workspace-nav-menu.tsx` → `/page/workspace/admin/skills`,不再单列于 `components/page-sidebar.tsx` 左下角设置下拉):打开自动扫描并带**进度条**(扫描逐技能 / 应用逐文件 X/N,应对 150+ 技能),URL/IP 类型过滤 + 搜索 + 「片段批量套用」(如所有 `uat.4.cn` → `prod.4.cn`)、每地址「替换为」输入框(可展开看 `file:line` 片段)、内联**预览**、**应用**、**多关键词检索**(空格分隔全部命中),以及**修改记录**面板——每次应用生成一条记录(显示时间/改了哪些地址 from→to/文件数/处数),可**针对单次记录回退**(二次确认)。client + NDJSON 流式解析在 `strategy-components/api/skill-addresses.ts`。后端 `/api/skill-addresses`(**仅管理员**):所有文件 IO 走 `asyncio.to_thread` 不阻塞事件循环;`GET /scan`(+`/scan/stream`)、`POST /preview`、`POST /apply`(+`/apply/stream`)、`GET /backups`、`POST /rollback/{id}`;**子串安全替换**(基于 token span,不会误伤子串)、`expected_total` 乐观锁(409)、写前自动磁盘备份到 `{skills_root}/.address-edits/`(同时写一份 `{id}.manifest.json` 记录本次替换明细,供修改记录展示)、应用后刷新技能系统提示缓存。纯引擎 `deerflow/skills/address_scan.py`,router `app/gateway/routers/skill_addresses.py`,tests `tests/test_skill_addresses.py`。范围仅运行中技能,不动 `deploy/minimal/skills/`。
|
||||||
- **按钮管理 (Task Button Management, admin-only)**: configures the **可配置跳转按钮** rendered in the left task panel of the task deep-link workspaces (rwfx / 3qfx / xdfx, see `TaskFxWorkspace`). Each button = `{business, label, linkType, target, appendTaskId, idParam, idKind, openMode, loginParams, appendAuth, authTokenParam, authNameParam, enabled, sortOrder}`; `linkType` is `url` (外链 `window.open`, optional append of `?{idParam}={id}` — **`idParam` is the editable query-param name**, e.g. `id`/`task_id`), `business` (内部切换到另一业务, reuses `resolveTaskDeeplinkTarget`, always task id), or `purpose` (弹出当前任务的「任务目的」表格 via the shared `TaskPurposeDialog`, no target — configurable for any page incl. xdfx). **`url` buttons support two independent, coexisting login-append mechanisms**: (1) **`loginParams` (追加登录参数, like 应用管理/Light App Center)** — a list of `{key, source, customValue}` resolved at click time: `runTaskButton` reads `userInfo` (via `readUserInfo` from `strategy-components/lib/light-app`, embed-isolated through `authStorage`) and appends `?{key}={value}` for each (`source` = a userInfo field such as `accessToken`/`yUserId`, or `other` → literal `customValue`); empty list = nothing appended. Reuses `LOGIN_SOURCE_OPTIONS` from light-app; stored as JSON column `task_buttons.login_params_json` (`PortableJSON`, migration `20260625_01`). (2) **`appendAuth` (追加登录态)** — appends `?{authTokenParam}={token1}&{authNameParam}={username}` (default param names `token`/`username`) so the跳转目标外部系统 can identify the current user — e.g. `…/action/sentiment?token=<JWT>&username=lizhaoxing`. **Gated on token login**: `token1` = the `?authToken=` used by token-exchange login; `username` = the raw login name the backend resolved from `getTokenInfo`. Both are stashed at login by `persistLoginCredentials` into `login.token1`/`login.username` (`core/auth/login-credentials.ts`, scoped via [[scoped-storage]] so嵌入/独立会话隔离 + 随独立会话迁移). Only token login writes them → a non-token-login session has no `token1`, so `runTaskButton` silently skips the auth append (the jump still works, just without those two params). The frontend reads `username` straight from the new `UsernameLoginResponse.username` field (the stored email local-part is lossy — lowercased/normalized — so not used), **not** by decoding the JWT. Stored as columns `task_buttons.{append_auth, auth_token_param, auth_name_param}` (migration `20260625_02`). The admin page renders both editors for `url` buttons (loginParams 虚线框 + 追加登录态 switch). **`idKind` (`task`/`action`) only matters for xdfx**: its deep-link `?taskId=` is an 行动id and the 任务id is resolved at login (`fetchTaskIdFromAction`) — `TaskContext` now also stashes the raw `actionId`, so a url button can append either (default `task`). Page `TaskButtonsPage.tsx` (route `/page/strategy/admin/task-buttons`, **admin-gated**, entry「按钮管理」in the bottom-left settings dropdown of `components/page-sidebar.tsx`, both full + collapsed modes) has per-business tabs, edits name/type/target/启停/排序/增删, saves the **whole set atomically** (replace-all), plus 「导入默认」 to merge code-seed defaults. **Seed/fallback** (like 敏感词管理): only when the whole table is empty (fresh deploy, never saved) do the workspace + page fall back to `defaultTaskButtons(cfg)` — rwfx's url buttons (编辑/上一步/下一步/3q详情) built from the **live** `/api/public/config/task-deeplink` plus cross-business switch buttons for **all three** businesses (each switches to the other two — rwfx→三情/行动, 3qfx→任务/行动, xdfx→任务/三情). Once saved, the store is the sole source of truth. **Bottom 快捷跳转 bar groups by prev/next**: `TaskJumpBar` renders only「上一步/下一步」-type buttons in the bottom bar — a `prev` group (id 含 `prev` 或文案「上一步」, 描边次要样式 + ←图标) rendered **左侧**, then a `next` group (id 含 `next` 或文案「下一步」, 实心主色) on the **右侧** (container `justify-end`); everything else goes to the right-edge floating menu. The default 上一步 button (`rwfx-prev`, `url`, target = `DASHBOARD_AGENT_ROUTE` `/page/strategy/dashboard-agent`, `appendTaskId` + `idParam=taskId` → `?taskId=任务id`, `openMode=self`) 默认指向**应用内**「大屏绘制」页,在 **both** 单智能体 (`TaskFxWorkspace`) 与 多智能体会商 (`RoundtablePlanningPage` 深链 rwfx-A) 都显示(共用 `TaskJumpBar`)。**`runTaskButton` 的 `url` 类型支持应用内路由**:target 以单个 `/` 开头时按内部路由处理——`self` 走 react-router 同窗导航(不刷新),`blank` 拼成当前源的 hash 路由新标签打开;外链行为不变。 The old hardcoded jump buttons were migrated out: `TaskFxWorkspace` no longer hardcodes 切换到三Line truncated
|
- **按钮管理 (Task Button Management, admin-only)**: configures the **可配置跳转按钮** rendered in the left task panel of the task deep-link workspaces (rwfx / 3qfx / xdfx, see `TaskFxWorkspace`). Each button = `{business, label, linkType, target, appendTaskId, idParam, idKind, openMode, loginParams, appendAuth, authTokenParam, authNameParam, enabled, sortOrder}`; `linkType` is `url` (外链 `window.open`, optional append of `?{idParam}={id}` — **`idParam` is the editable query-param name**, e.g. `id`/`task_id`), `business` (内部切换到另一业务, reuses `resolveTaskDeeplinkTarget`, always task id), or `purpose` (弹出当前任务的「任务目的」表格 via the shared `TaskPurposeDialog`, no target — configurable for any page incl. xdfx). **`url` buttons support two independent, coexisting login-append mechanisms**: (1) **`loginParams` (追加登录参数, like 应用管理/Light App Center)** — a list of `{key, source, customValue}` resolved at click time: `runTaskButton` reads `userInfo` (via `readUserInfo` from `strategy-components/lib/light-app`, embed-isolated through `authStorage`) and appends `?{key}={value}` for each (`source` = a userInfo field such as `accessToken`/`yUserId`, or `other` → literal `customValue`); empty list = nothing appended. Reuses `LOGIN_SOURCE_OPTIONS` from light-app; stored as JSON column `task_buttons.login_params_json` (`PortableJSON`, migration `20260625_01`). (2) **`appendAuth` (追加登录态)** — appends `?{authTokenParam}={token1}&{authNameParam}={username}` (default param names `token`/`username`) so the跳转目标外部系统 can identify the current user — e.g. `…/action/sentiment?token=<JWT>&username=lizhaoxing`. **Gated on token login**: `token1` = the `?authToken=` used by token-exchange login; `username` = the raw login name the backend resolved from `getTokenInfo`. Both are stashed at login by `persistLoginCredentials` into `login.token1`/`login.username` (`core/auth/login-credentials.ts`, scoped via [[scoped-storage]] so嵌入/独立会话隔离 + 随独立会话迁移). Only token login writes them → a non-token-login session has no `token1`, so `runTaskButton` silently skips the auth append (the jump still works, just without those two params). The frontend reads `username` straight from the new `UsernameLoginResponse.username` field (the stored email local-part is lossy — lowercased/normalized — so not used), **not** by decoding the JWT. Stored as columns `task_buttons.{append_auth, auth_token_param, auth_name_param}` (migration `20260625_02`). The admin page renders both editors for `url` buttons (loginParams 虚线框 + 追加登录态 switch). **`idKind` (`task`/`action`) only matters for xdfx**: its deep-link `?taskId=` is an 行动id and the 任务id is resolved at login (`fetchTaskIdFromAction`) — `TaskContext` now also stashes the raw `actionId`, so a url button can append either (default `task`). Page `TaskButtonsPage.tsx` (route `/page/strategy/admin/task-buttons`, **admin-gated**, entry「按钮管理」in the bottom-left settings dropdown of `components/page-sidebar.tsx`, both full + collapsed modes) has per-business tabs, edits name/type/target/启停/排序/增删, saves the **whole set atomically** (replace-all), plus 「导入默认」 to merge code-seed defaults. **Seed/fallback** (like 敏感词管理): only when the whole table is empty (fresh deploy, never saved) do the workspace + page fall back to `defaultTaskButtons(cfg)` — rwfx's url buttons (编辑/上一步/下一步/3q详情) built from the **live** `/api/public/config/task-deeplink` plus cross-business switch buttons for **all three** businesses (each switches to the other two — rwfx→三情/行动, 3qfx→任务/行动, xdfx→任务/三情). Once saved, the store is the sole source of truth. **Bottom 快捷跳转 bar groups by prev/next**: `TaskJumpBar` renders only「上一步/下一步」-type buttons in the bottom bar — a `prev` group (id 含 `prev` 或文案「上一步」, 描边次要样式 + ←图标) rendered **左侧**, then a `next` group (id 含 `next` 或文案「下一步」, 实心主色) on the **右侧** (container `justify-end`); everything else goes to the right-edge floating menu. The default 上一步 button (`rwfx-prev`, `url`, target = `DASHBOARD_AGENT_ROUTE` `/page/strategy/dashboard-agent`, `appendTaskId` + `idParam=taskId` → `?taskId=任务id`, `openMode=self`) 默认指向**应用内**「大屏绘制」页,在 **both** 单智能体 (`TaskFxWorkspace`) 与 多智能体会商 (`RoundtablePlanningPage` 深链 rwfx-A) 都显示(共用 `TaskJumpBar`)。**`runTaskButton` 的 `url` 类型支持应用内路由**:target 以单个 `/` 开头时按内部路由处理——`self` 走 react-router 同窗导航(不刷新),`blank` 拼成当前源的 hash 路由新标签打开;外链行为不变。 The old hardcoded jump buttons were migrated out: `TaskFxWorkspace` no longer hardcodes 切换到三Line truncated
|
||||||
|
- **实时并发监控 (Concurrency / System-Pressure Monitor, admin-only)**: a live panel mounted at the **top of the admin 排行榜 page** (`components/workspace/admin/concurrency-monitor.tsx` `ConcurrencyMonitorPanel`, rendered in `pages/AdminUserLeaderboardPage.tsx`) showing **当前并发的大模型调用数 + 列表(用户/智能体/类型/已运行时长)**, each row with a 「停止」button (and a 「全部停止」) that **truly aborts** that run's in-flight model stream via `RunManager.cancel` (`POST /api/admin/active-runs/{run_id}/cancel`) — to relieve pressure when 使用人数过多. Two line charts (minute/hour toggle): **并发量折线图** (peak/avg, `GET /api/admin/active-runs/concurrency`, fed by a lightweight background sampler into `concurrency_samples`) and **大模型出字速度折线图** (tokens/sec, `GET /api/admin/active-runs/token-speed`, aggregated from `llm_call_metrics`) so a dip pinpoints where the bottleneck is. A **「后台采样」Switch** (`system_settings.json → concurrency_monitor.enabled`, **default off / opt-in**, `GET|PUT /api/admin/active-runs/settings`) turns the background sampling on/off **at runtime, no restart** (the live concurrency count + 「停止」 work even while sampling is off — only the 并发量 history chart needs it on); everything is designed to **never impact production** (one tiny INSERT/15s, retention-purged, best-effort). Client/hooks: `core/admin/active-runs.ts` + `core/admin/hooks.ts` (`useActiveRuns`/`useConcurrencySeries`/`useTokenSpeedSeries`/`useCancel*`/`useMonitorSettings`). See the backend CLAUDE.md「Admin Concurrency Monitor」.
|
||||||
|
- **会话冲突 409 中文化**: when a user submits a new question while the previous turn is still streaming, the backend's `RunManager.create_or_reject` rejects with HTTP 409 (`services.py` now returns a **Chinese** detail「当前对话还在生成回答…」). `core/threads/hooks.ts` `onError` detects the conflict (`isActiveRunConflictError`: 409 / "already has an active run" / the Chinese text) and shows a friendly `toast.info` instead of a raw error.
|
||||||
- **State**: TanStack Query for server state, context providers (`SubtasksProvider`, `ArtifactsProvider`, `PromptInputProvider`) scoped to `WorkspaceRoutes`
|
- **State**: TanStack Query for server state, context providers (`SubtasksProvider`, `ArtifactsProvider`, `PromptInputProvider`) scoped to `WorkspaceRoutes`
|
||||||
- **LLM streaming**: `@langchain/langgraph-sdk` streams messages from the backend via SSE
|
- **LLM streaming**: `@langchain/langgraph-sdk` streams messages from the backend via SSE
|
||||||
- **UI**: Radix UI primitives + TailwindCSS v4 + shadcn-style components in `src/components/ui/`
|
- **UI**: Radix UI primitives + TailwindCSS v4 + shadcn-style components in `src/components/ui/`
|
||||||
|
|||||||
@@ -0,0 +1,472 @@
|
|||||||
|
"use client";
|
||||||
|
|
||||||
|
import {
|
||||||
|
Activity,
|
||||||
|
Gauge,
|
||||||
|
Loader2,
|
||||||
|
OctagonX,
|
||||||
|
RefreshCw,
|
||||||
|
Square,
|
||||||
|
Zap,
|
||||||
|
} from "lucide-react";
|
||||||
|
import { useMemo, useState } from "react";
|
||||||
|
import { toast } from "sonner";
|
||||||
|
|
||||||
|
import { Badge } from "@/components/ui/badge";
|
||||||
|
import { Button } from "@/components/ui/button";
|
||||||
|
import { Switch } from "@/components/ui/switch";
|
||||||
|
import type {
|
||||||
|
ActiveRunItem,
|
||||||
|
ConcurrencyPoint,
|
||||||
|
Granularity,
|
||||||
|
TokenSpeedPoint,
|
||||||
|
} from "@/core/admin/active-runs";
|
||||||
|
import {
|
||||||
|
useActiveRuns,
|
||||||
|
useCancelActiveRun,
|
||||||
|
useCancelAllActiveRuns,
|
||||||
|
useConcurrencySeries,
|
||||||
|
useMonitorSettings,
|
||||||
|
useTokenSpeedSeries,
|
||||||
|
useUpdateMonitorSettings,
|
||||||
|
} from "@/core/admin/hooks";
|
||||||
|
import { cn } from "@/lib/utils";
|
||||||
|
|
||||||
|
/** lookback window (hours) per granularity. */
|
||||||
|
const LOOKBACK: Record<Granularity, number> = { minute: 3, hour: 48 };
|
||||||
|
|
||||||
|
function fmtBucket(iso: string, granularity: Granularity): string {
|
||||||
|
const date = new Date(iso);
|
||||||
|
if (Number.isNaN(date.getTime())) return iso;
|
||||||
|
const mm = String(date.getMonth() + 1).padStart(2, "0");
|
||||||
|
const dd = String(date.getDate()).padStart(2, "0");
|
||||||
|
const hh = String(date.getHours()).padStart(2, "0");
|
||||||
|
const mi = String(date.getMinutes()).padStart(2, "0");
|
||||||
|
return granularity === "hour" ? `${mm}-${dd} ${hh}:00` : `${hh}:${mi}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function fmtElapsed(seconds: number): string {
|
||||||
|
if (seconds < 60) return `${seconds}s`;
|
||||||
|
const m = Math.floor(seconds / 60);
|
||||||
|
const s = seconds % 60;
|
||||||
|
if (m < 60) return `${m}m${s.toString().padStart(2, "0")}s`;
|
||||||
|
const h = Math.floor(m / 60);
|
||||||
|
return `${h}h${(m % 60).toString().padStart(2, "0")}m`;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface LineSeries {
|
||||||
|
name: string;
|
||||||
|
color: string;
|
||||||
|
values: (number | null)[];
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Minimal dependency-free SVG line chart (1–2 series + area under the first). */
|
||||||
|
function LineChart({
|
||||||
|
labels,
|
||||||
|
series,
|
||||||
|
unit,
|
||||||
|
emptyHint,
|
||||||
|
}: {
|
||||||
|
labels: string[];
|
||||||
|
series: LineSeries[];
|
||||||
|
unit: string;
|
||||||
|
emptyHint: string;
|
||||||
|
}) {
|
||||||
|
const width = 760;
|
||||||
|
const height = 220;
|
||||||
|
const pad = { top: 18, right: 16, bottom: 26, left: 40 };
|
||||||
|
const cw = width - pad.left - pad.right;
|
||||||
|
const ch = height - pad.top - pad.bottom;
|
||||||
|
|
||||||
|
const allValues = series.flatMap((s) => s.values.filter((v): v is number => v != null));
|
||||||
|
const max = Math.max(1, ...allValues);
|
||||||
|
const n = labels.length;
|
||||||
|
|
||||||
|
if (n === 0) {
|
||||||
|
return (
|
||||||
|
<div className="flex h-[220px] items-center justify-center text-sm text-[#5F6368] dark:text-slate-400">
|
||||||
|
{emptyHint}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
const xFor = (i: number) => pad.left + (n <= 1 ? cw / 2 : (i / (n - 1)) * cw);
|
||||||
|
const yFor = (v: number) => pad.top + ch - (v / max) * ch;
|
||||||
|
const grid = [0, 0.25, 0.5, 0.75, 1];
|
||||||
|
|
||||||
|
const linePath = (vals: (number | null)[]) => {
|
||||||
|
let d = "";
|
||||||
|
let started = false;
|
||||||
|
vals.forEach((v, i) => {
|
||||||
|
if (v == null) {
|
||||||
|
started = false;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
d += `${started ? "L" : "M"} ${xFor(i).toFixed(1)} ${yFor(v).toFixed(1)} `;
|
||||||
|
started = true;
|
||||||
|
});
|
||||||
|
return d.trim();
|
||||||
|
};
|
||||||
|
|
||||||
|
const first = series[0];
|
||||||
|
const areaPath = (() => {
|
||||||
|
if (!first) return "";
|
||||||
|
const line = linePath(first.values);
|
||||||
|
if (!line) return "";
|
||||||
|
return `${line} L ${xFor(n - 1).toFixed(1)} ${(pad.top + ch).toFixed(1)} L ${xFor(0).toFixed(1)} ${(pad.top + ch).toFixed(1)} Z`;
|
||||||
|
})();
|
||||||
|
|
||||||
|
const labelStep = Math.max(1, Math.ceil(n / 7));
|
||||||
|
|
||||||
|
return (
|
||||||
|
<svg viewBox={`0 0 ${width} ${height}`} className="w-full" role="img" aria-label="时序折线图">
|
||||||
|
<defs>
|
||||||
|
<linearGradient id="cm-area" x1="0" x2="0" y1="0" y2="1">
|
||||||
|
<stop offset="0%" stopColor={first?.color ?? "#C1121F"} stopOpacity="0.18" />
|
||||||
|
<stop offset="100%" stopColor={first?.color ?? "#C1121F"} stopOpacity="0.02" />
|
||||||
|
</linearGradient>
|
||||||
|
</defs>
|
||||||
|
{grid.map((r) => {
|
||||||
|
const y = pad.top + ch * r;
|
||||||
|
return (
|
||||||
|
<g key={r}>
|
||||||
|
<line
|
||||||
|
x1={pad.left}
|
||||||
|
x2={width - pad.right}
|
||||||
|
y1={y}
|
||||||
|
y2={y}
|
||||||
|
stroke="currentColor"
|
||||||
|
className="text-[#E5E7EB] dark:text-[#3F3F46]"
|
||||||
|
strokeWidth="1"
|
||||||
|
/>
|
||||||
|
<text x={6} y={y + 4} className="fill-[#9CA3AF] text-[10px]">
|
||||||
|
{Math.round(max * (1 - r))}
|
||||||
|
</text>
|
||||||
|
</g>
|
||||||
|
);
|
||||||
|
})}
|
||||||
|
{areaPath && <path d={areaPath} fill="url(#cm-area)" />}
|
||||||
|
{series.map((s) => (
|
||||||
|
<path
|
||||||
|
key={s.name}
|
||||||
|
d={linePath(s.values)}
|
||||||
|
fill="none"
|
||||||
|
stroke={s.color}
|
||||||
|
strokeWidth="2"
|
||||||
|
strokeLinecap="round"
|
||||||
|
strokeLinejoin="round"
|
||||||
|
/>
|
||||||
|
))}
|
||||||
|
{labels.map((label, i) => {
|
||||||
|
if (i !== 0 && i !== n - 1 && i % labelStep !== 0) return null;
|
||||||
|
return (
|
||||||
|
<text key={`${label}-${i}`} x={xFor(i)} y={height - 8} textAnchor="middle" className="fill-[#6B7280] text-[10px]">
|
||||||
|
{label}
|
||||||
|
</text>
|
||||||
|
);
|
||||||
|
})}
|
||||||
|
{/* hover dots with native tooltip */}
|
||||||
|
{series.map((s) =>
|
||||||
|
s.values.map((v, i) =>
|
||||||
|
v == null ? null : (
|
||||||
|
<circle key={`${s.name}-${i}`} cx={xFor(i)} cy={yFor(v)} r={2.5} fill={s.color}>
|
||||||
|
<title>{`${labels[i]} · ${s.name}: ${v}${unit}`}</title>
|
||||||
|
</circle>
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)}
|
||||||
|
</svg>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
function ChartCard({
|
||||||
|
icon: Icon,
|
||||||
|
title,
|
||||||
|
subtitle,
|
||||||
|
children,
|
||||||
|
}: {
|
||||||
|
icon: typeof Activity;
|
||||||
|
title: string;
|
||||||
|
subtitle: string;
|
||||||
|
children: React.ReactNode;
|
||||||
|
}) {
|
||||||
|
return (
|
||||||
|
<section className="rounded-lg border border-[#F1D6D6] bg-white p-4 shadow-sm dark:border-[#213D5E] dark:bg-[#02244C]">
|
||||||
|
<div className="mb-3 flex items-center gap-2">
|
||||||
|
<Icon className="size-4 text-[#C1121F] dark:text-[#7FACF2]" />
|
||||||
|
<h3 className="font-semibold text-[#231815] dark:text-white">{title}</h3>
|
||||||
|
<span className="ml-auto text-xs text-[#8B6B6B] dark:text-[#7FACF2]">{subtitle}</span>
|
||||||
|
</div>
|
||||||
|
{children}
|
||||||
|
</section>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
function ActiveRunRow({
|
||||||
|
run,
|
||||||
|
onStop,
|
||||||
|
stopping,
|
||||||
|
}: {
|
||||||
|
run: ActiveRunItem;
|
||||||
|
onStop: (runId: string) => void;
|
||||||
|
stopping: boolean;
|
||||||
|
}) {
|
||||||
|
return (
|
||||||
|
<div className="grid grid-cols-[minmax(0,1.4fr)_minmax(0,1.2fr)_72px_64px_64px_72px] items-center gap-3 border-b border-[#F1D6D6] px-3 py-2 text-sm last:border-b-0 dark:border-[#213D5E]">
|
||||||
|
<div className="min-w-0">
|
||||||
|
<div className="truncate font-medium text-[#202124] dark:text-white">{run.user_name || "未知用户"}</div>
|
||||||
|
<div className="truncate text-xs text-[#5F6368] dark:text-slate-400" title={run.thread_title}>
|
||||||
|
{run.thread_title || run.thread_id}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="min-w-0 truncate text-[#202124] dark:text-slate-100">{run.agent_name}</div>
|
||||||
|
<Badge variant="outline" className="justify-center">
|
||||||
|
{run.kind}
|
||||||
|
</Badge>
|
||||||
|
<span
|
||||||
|
className={cn(
|
||||||
|
"text-center text-xs font-medium",
|
||||||
|
run.status === "running" ? "text-[#0F766E] dark:text-teal-300" : "text-[#B45309] dark:text-amber-300",
|
||||||
|
)}
|
||||||
|
>
|
||||||
|
{run.status === "running" ? "运行中" : "排队中"}
|
||||||
|
</span>
|
||||||
|
<span className="text-center text-xs tabular-nums text-[#5F6368] dark:text-slate-400">
|
||||||
|
{fmtElapsed(run.elapsed_seconds)}
|
||||||
|
</span>
|
||||||
|
<Button
|
||||||
|
size="sm"
|
||||||
|
variant="outline"
|
||||||
|
className="h-7 border-[#C1121F]/40 text-[#C1121F] hover:bg-[#FFF1F1] dark:border-rose-400/40 dark:text-rose-200"
|
||||||
|
disabled={stopping}
|
||||||
|
onClick={() => onStop(run.run_id)}
|
||||||
|
>
|
||||||
|
{stopping ? <Loader2 className="size-3.5 animate-spin" /> : <Square className="size-3.5" />}
|
||||||
|
停止
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function ConcurrencyMonitorPanel() {
|
||||||
|
const [open, setOpen] = useState(true);
|
||||||
|
const [granularity, setGranularity] = useState<Granularity>("minute");
|
||||||
|
|
||||||
|
const { data: live, isLoading: liveLoading, refetch, isRefetching } = useActiveRuns(open);
|
||||||
|
const concurrency = useConcurrencySeries(granularity, LOOKBACK[granularity], open);
|
||||||
|
const tokenSpeed = useTokenSpeedSeries(granularity, LOOKBACK[granularity], open);
|
||||||
|
const { data: settings } = useMonitorSettings();
|
||||||
|
const updateSettings = useUpdateMonitorSettings();
|
||||||
|
const cancelOne = useCancelActiveRun();
|
||||||
|
const cancelAll = useCancelAllActiveRuns();
|
||||||
|
|
||||||
|
const [stoppingId, setStoppingId] = useState<string | null>(null);
|
||||||
|
|
||||||
|
const count = live?.count ?? 0;
|
||||||
|
const runs = live?.runs ?? [];
|
||||||
|
const samplerOn = settings?.enabled ?? false;
|
||||||
|
|
||||||
|
const concurrencyChart = useMemo(() => {
|
||||||
|
const points: ConcurrencyPoint[] = concurrency.data?.points ?? [];
|
||||||
|
return {
|
||||||
|
labels: points.map((p) => fmtBucket(p.bucket, granularity)),
|
||||||
|
series: [
|
||||||
|
{ name: "峰值并发", color: "#C1121F", values: points.map((p) => p.peak) },
|
||||||
|
{ name: "平均并发", color: "#F59E0B", values: points.map((p) => p.avg) },
|
||||||
|
] as LineSeries[],
|
||||||
|
};
|
||||||
|
}, [concurrency.data, granularity]);
|
||||||
|
|
||||||
|
const tokenChart = useMemo(() => {
|
||||||
|
const points: TokenSpeedPoint[] = tokenSpeed.data?.points ?? [];
|
||||||
|
return {
|
||||||
|
labels: points.map((p) => fmtBucket(p.bucket, granularity)),
|
||||||
|
series: [
|
||||||
|
{ name: "平均速度", color: "#2563EB", values: points.map((p) => p.avg_tokens_per_sec) },
|
||||||
|
{ name: "最慢", color: "#9333EA", values: points.map((p) => p.min_tokens_per_sec) },
|
||||||
|
] as LineSeries[],
|
||||||
|
};
|
||||||
|
}, [tokenSpeed.data, granularity]);
|
||||||
|
|
||||||
|
const handleStop = async (runId: string) => {
|
||||||
|
setStoppingId(runId);
|
||||||
|
try {
|
||||||
|
const res = await cancelOne.mutateAsync(runId);
|
||||||
|
toast[res.cancelled ? "success" : "info"](res.cancelled ? "已停止该对话调用" : "该调用已结束或不存在");
|
||||||
|
} catch (e) {
|
||||||
|
toast.error(e instanceof Error ? e.message : "停止失败");
|
||||||
|
} finally {
|
||||||
|
setStoppingId(null);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const handleStopAll = async () => {
|
||||||
|
if (runs.length === 0) return;
|
||||||
|
if (!window.confirm(`确定要停止当前全部 ${count} 个并发调用吗?这会中断这些用户正在生成的回答。`)) return;
|
||||||
|
try {
|
||||||
|
const res = await cancelAll.mutateAsync();
|
||||||
|
toast.success(`已停止 ${res.cancelled_count} 个调用`);
|
||||||
|
} catch (e) {
|
||||||
|
toast.error(e instanceof Error ? e.message : "全部停止失败");
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const handleToggleSampler = async (next: boolean) => {
|
||||||
|
try {
|
||||||
|
await updateSettings.mutateAsync(next);
|
||||||
|
toast.success(next ? "已开启并发采样" : "已关闭并发采样(折线图不再新增历史点)");
|
||||||
|
} catch (e) {
|
||||||
|
toast.error(e instanceof Error ? e.message : "切换失败");
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
return (
|
||||||
|
<section className="space-y-4 rounded-xl border border-[#F1D6D6] bg-[#FFF7F7] p-4 shadow-sm dark:border-[#213D5E] dark:bg-[#011226]">
|
||||||
|
{/* header */}
|
||||||
|
<div className="flex flex-wrap items-center gap-3">
|
||||||
|
<div className="flex items-center gap-2">
|
||||||
|
<span className="flex size-9 items-center justify-center rounded-lg bg-[#C1121F] text-white dark:bg-[#2563EB]">
|
||||||
|
<Activity className="size-5" />
|
||||||
|
</span>
|
||||||
|
<div>
|
||||||
|
<div className="font-semibold text-[#231815] dark:text-white">实时并发监控</div>
|
||||||
|
<div className="text-xs text-[#8B6B6B] dark:text-[#7FACF2]">
|
||||||
|
监控大模型并发压力,可直接停止任意对话调用
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="ml-auto flex flex-wrap items-center gap-3">
|
||||||
|
<div className="flex items-center gap-2 rounded-lg border border-[#F1D6D6] bg-white px-3 py-1.5 dark:border-[#213D5E] dark:bg-[#02244C]">
|
||||||
|
<span className="text-xs text-[#5F6368] dark:text-slate-300">当前并发</span>
|
||||||
|
<span
|
||||||
|
className={cn(
|
||||||
|
"text-2xl font-semibold tabular-nums",
|
||||||
|
count > 0 ? "text-[#C1121F] dark:text-[#7FACF2]" : "text-[#202124] dark:text-white",
|
||||||
|
)}
|
||||||
|
>
|
||||||
|
{count}
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
<div className="flex items-center gap-2 rounded-lg border border-[#F1D6D6] bg-white px-3 py-1.5 dark:border-[#213D5E] dark:bg-[#02244C]">
|
||||||
|
<span className="text-xs text-[#5F6368] dark:text-slate-300">后台采样</span>
|
||||||
|
<Switch checked={samplerOn} onCheckedChange={handleToggleSampler} disabled={updateSettings.isPending} />
|
||||||
|
</div>
|
||||||
|
<Button variant="outline" size="sm" onClick={() => refetch()} disabled={isRefetching}>
|
||||||
|
<RefreshCw className={cn("mr-1 size-4", isRefetching && "animate-spin")} />
|
||||||
|
刷新
|
||||||
|
</Button>
|
||||||
|
<Button
|
||||||
|
variant="outline"
|
||||||
|
size="sm"
|
||||||
|
className="border-[#C1121F]/50 text-[#C1121F] hover:bg-[#FFF1F1] dark:border-rose-400/40 dark:text-rose-200"
|
||||||
|
onClick={handleStopAll}
|
||||||
|
disabled={cancelAll.isPending || runs.length === 0}
|
||||||
|
>
|
||||||
|
<OctagonX className="mr-1 size-4" />
|
||||||
|
全部停止
|
||||||
|
</Button>
|
||||||
|
<Button variant="ghost" size="sm" onClick={() => setOpen((v) => !v)}>
|
||||||
|
{open ? "收起" : "展开"}
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{open && (
|
||||||
|
<>
|
||||||
|
{!samplerOn && (
|
||||||
|
<div className="rounded-lg border border-amber-300/60 bg-amber-50 px-3 py-2 text-xs text-amber-800 dark:border-amber-300/30 dark:bg-amber-300/10 dark:text-amber-100">
|
||||||
|
后台采样默认关闭:实时并发数与「停止」功能始终可用。需要并发量历史折线图时,打开右上角「后台采样」开关即可开始记录(对线上服务影响极小,可随时关闭)。
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{/* granularity toggle */}
|
||||||
|
<div className="flex items-center gap-2">
|
||||||
|
<span className="text-xs text-[#5F6368] dark:text-slate-300">统计粒度</span>
|
||||||
|
{(["minute", "hour"] as Granularity[]).map((g) => (
|
||||||
|
<Button
|
||||||
|
key={g}
|
||||||
|
size="sm"
|
||||||
|
variant={granularity === g ? "default" : "outline"}
|
||||||
|
className={cn("h-7", granularity === g && "bg-[#C1121F] hover:bg-[#A50F1A] dark:bg-[#2563EB]")}
|
||||||
|
onClick={() => setGranularity(g)}
|
||||||
|
>
|
||||||
|
{g === "minute" ? "按分钟" : "按小时"}
|
||||||
|
</Button>
|
||||||
|
))}
|
||||||
|
<span className="text-xs text-[#8B6B6B] dark:text-[#7FACF2]">
|
||||||
|
{granularity === "minute" ? "近 3 小时" : "近 48 小时"}
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* charts */}
|
||||||
|
<div className="grid gap-3 xl:grid-cols-2">
|
||||||
|
<ChartCard icon={Gauge} title="并发量(系统压力)" subtitle="峰值 / 平均并发调用数">
|
||||||
|
{concurrency.isLoading ? (
|
||||||
|
<div className="flex h-[220px] items-center justify-center text-sm text-[#5F6368] dark:text-slate-400">
|
||||||
|
<Loader2 className="mr-2 size-4 animate-spin" /> 加载中
|
||||||
|
</div>
|
||||||
|
) : (
|
||||||
|
<LineChart
|
||||||
|
labels={concurrencyChart.labels}
|
||||||
|
series={concurrencyChart.series}
|
||||||
|
unit=""
|
||||||
|
emptyHint="暂无并发采样数据(采样关闭或时间窗口内无记录)"
|
||||||
|
/>
|
||||||
|
)}
|
||||||
|
</ChartCard>
|
||||||
|
<ChartCard icon={Zap} title="大模型出字速度" subtitle="tokens/秒(越低说明模型越慢)">
|
||||||
|
{tokenSpeed.isLoading ? (
|
||||||
|
<div className="flex h-[220px] items-center justify-center text-sm text-[#5F6368] dark:text-slate-400">
|
||||||
|
<Loader2 className="mr-2 size-4 animate-spin" /> 加载中
|
||||||
|
</div>
|
||||||
|
) : (
|
||||||
|
<LineChart
|
||||||
|
labels={tokenChart.labels}
|
||||||
|
series={tokenChart.series}
|
||||||
|
unit=" t/s"
|
||||||
|
emptyHint="暂无大模型调用记录"
|
||||||
|
/>
|
||||||
|
)}
|
||||||
|
</ChartCard>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* active run list */}
|
||||||
|
<section className="rounded-lg border border-[#F1D6D6] bg-white dark:border-[#213D5E] dark:bg-[#02244C]">
|
||||||
|
<div className="flex items-center gap-2 border-b border-[#F1D6D6] px-3 py-2 dark:border-[#213D5E]">
|
||||||
|
<span className="font-semibold text-[#231815] dark:text-white">正在进行的调用</span>
|
||||||
|
<Badge variant="outline">{count}</Badge>
|
||||||
|
{liveLoading && <Loader2 className="size-3.5 animate-spin text-[#8B6B6B]" />}
|
||||||
|
</div>
|
||||||
|
{runs.length === 0 ? (
|
||||||
|
<div className="py-10 text-center text-sm text-[#5F6368] dark:text-slate-400">
|
||||||
|
当前没有正在进行的大模型调用
|
||||||
|
</div>
|
||||||
|
) : (
|
||||||
|
<div>
|
||||||
|
<div className="grid grid-cols-[minmax(0,1.4fr)_minmax(0,1.2fr)_72px_64px_64px_72px] gap-3 border-b border-[#F1D6D6] px-3 py-2 text-xs text-[#8B6B6B] dark:border-[#213D5E] dark:text-[#7FACF2]">
|
||||||
|
<span>用户 / 会话</span>
|
||||||
|
<span>智能体</span>
|
||||||
|
<span className="text-center">类型</span>
|
||||||
|
<span className="text-center">状态</span>
|
||||||
|
<span className="text-center">已运行</span>
|
||||||
|
<span className="text-center">操作</span>
|
||||||
|
</div>
|
||||||
|
{runs.map((run) => (
|
||||||
|
<ActiveRunRow
|
||||||
|
key={run.run_id}
|
||||||
|
run={run}
|
||||||
|
onStop={handleStop}
|
||||||
|
stopping={stoppingId === run.run_id}
|
||||||
|
/>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</section>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
</section>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export default ConcurrencyMonitorPanel;
|
||||||
@@ -0,0 +1,159 @@
|
|||||||
|
import { apiFetch } from "@/core/api";
|
||||||
|
import { getBackendBaseURL } from "@/core/config";
|
||||||
|
import { OptType, writeYLog } from "@/core/log/y-log";
|
||||||
|
|
||||||
|
/** One in-flight model run as seen by the admin concurrency monitor. */
|
||||||
|
export interface ActiveRunItem {
|
||||||
|
run_id: string;
|
||||||
|
thread_id: string;
|
||||||
|
status: string;
|
||||||
|
created_at: string;
|
||||||
|
elapsed_seconds: number;
|
||||||
|
user_id: string;
|
||||||
|
user_name: string;
|
||||||
|
agent_id: string;
|
||||||
|
agent_name: string;
|
||||||
|
thread_title: string;
|
||||||
|
kind: string;
|
||||||
|
multitask_strategy: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ActiveRunsResponse {
|
||||||
|
count: number;
|
||||||
|
server_time: string;
|
||||||
|
runs: ActiveRunItem[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ConcurrencyPoint {
|
||||||
|
bucket: string;
|
||||||
|
peak: number;
|
||||||
|
avg: number;
|
||||||
|
samples: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ConcurrencySeriesResponse {
|
||||||
|
granularity: string;
|
||||||
|
since: string;
|
||||||
|
until: string;
|
||||||
|
enabled: boolean;
|
||||||
|
points: ConcurrencyPoint[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface TokenSpeedPoint {
|
||||||
|
bucket: string;
|
||||||
|
avg_tokens_per_sec: number;
|
||||||
|
min_tokens_per_sec: number;
|
||||||
|
max_tokens_per_sec: number;
|
||||||
|
samples: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface TokenSpeedSeriesResponse {
|
||||||
|
granularity: string;
|
||||||
|
since: string;
|
||||||
|
until: string;
|
||||||
|
points: TokenSpeedPoint[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export type Granularity = "minute" | "hour";
|
||||||
|
|
||||||
|
const base = () => `${getBackendBaseURL()}/api/admin/active-runs`;
|
||||||
|
|
||||||
|
async function parse<T>(res: Response, what: string): Promise<T> {
|
||||||
|
if (!res.ok) {
|
||||||
|
const err = (await res.json().catch(() => ({}))) as { detail?: string };
|
||||||
|
throw new Error(err.detail ?? `${what} (HTTP ${res.status})`);
|
||||||
|
}
|
||||||
|
return res.json() as Promise<T>;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Live snapshot of all in-flight runs. Polled — intentionally NOT y-logged. */
|
||||||
|
export async function loadActiveRuns(): Promise<ActiveRunsResponse> {
|
||||||
|
return parse<ActiveRunsResponse>(await apiFetch(base()), "Failed to load active runs");
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Concurrency time-series (peak/avg per minute|hour). Polled — not y-logged. */
|
||||||
|
export async function loadConcurrencySeries(
|
||||||
|
granularity: Granularity,
|
||||||
|
hours?: number,
|
||||||
|
): Promise<ConcurrencySeriesResponse> {
|
||||||
|
const params = new URLSearchParams({ granularity });
|
||||||
|
if (hours) params.set("hours", String(hours));
|
||||||
|
return parse<ConcurrencySeriesResponse>(
|
||||||
|
await apiFetch(`${base()}/concurrency?${params}`),
|
||||||
|
"Failed to load concurrency series",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Token-speed time-series (tokens/sec per minute|hour). Polled — not y-logged. */
|
||||||
|
export async function loadTokenSpeedSeries(
|
||||||
|
granularity: Granularity,
|
||||||
|
hours?: number,
|
||||||
|
): Promise<TokenSpeedSeriesResponse> {
|
||||||
|
const params = new URLSearchParams({ granularity });
|
||||||
|
if (hours) params.set("hours", String(hours));
|
||||||
|
return parse<TokenSpeedSeriesResponse>(
|
||||||
|
await apiFetch(`${base()}/token-speed?${params}`),
|
||||||
|
"Failed to load token speed series",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Truly stop one in-flight run (cancels the live model stream). */
|
||||||
|
export async function cancelActiveRun(runId: string): Promise<{ run_id: string; cancelled: boolean }> {
|
||||||
|
const url = `${base()}/${encodeURIComponent(runId)}/cancel`;
|
||||||
|
const result = await parse<{ run_id: string; cancelled: boolean }>(
|
||||||
|
await apiFetch(url, { method: "POST" }),
|
||||||
|
"Failed to cancel run",
|
||||||
|
);
|
||||||
|
writeYLog({
|
||||||
|
optType: OptType.other,
|
||||||
|
optDetail: "CM助手-全量-并发监控-停止对话调用",
|
||||||
|
url,
|
||||||
|
reqParam: { runId },
|
||||||
|
returnParam: result,
|
||||||
|
});
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Stop every in-flight run at once. */
|
||||||
|
export async function cancelAllActiveRuns(): Promise<{ cancelled_count: number }> {
|
||||||
|
const url = `${base()}/cancel-all`;
|
||||||
|
const result = await parse<{ cancelled_count: number }>(
|
||||||
|
await apiFetch(url, { method: "POST" }),
|
||||||
|
"Failed to cancel all runs",
|
||||||
|
);
|
||||||
|
writeYLog({
|
||||||
|
optType: OptType.other,
|
||||||
|
optDetail: "CM助手-全量-并发监控-全部停止",
|
||||||
|
url,
|
||||||
|
returnParam: result,
|
||||||
|
});
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Read the background-sampler on/off switch. */
|
||||||
|
export async function getMonitorSettings(): Promise<{ enabled: boolean }> {
|
||||||
|
return parse<{ enabled: boolean }>(
|
||||||
|
await apiFetch(`${base()}/settings`),
|
||||||
|
"Failed to load monitor settings",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Toggle the background concurrency sampler (admin only, no restart). */
|
||||||
|
export async function updateMonitorSettings(enabled: boolean): Promise<{ enabled: boolean }> {
|
||||||
|
const url = `${base()}/settings`;
|
||||||
|
const result = await parse<{ enabled: boolean }>(
|
||||||
|
await apiFetch(url, {
|
||||||
|
method: "PUT",
|
||||||
|
headers: { "Content-Type": "application/json" },
|
||||||
|
body: JSON.stringify({ enabled }),
|
||||||
|
}),
|
||||||
|
"Failed to update monitor settings",
|
||||||
|
);
|
||||||
|
writeYLog({
|
||||||
|
optType: enabled ? OptType.enabled : OptType.disabled,
|
||||||
|
optDetail: `CM助手-全量-并发监控-${enabled ? "开启" : "关闭"}采样`,
|
||||||
|
url,
|
||||||
|
reqParam: { enabled },
|
||||||
|
});
|
||||||
|
return result;
|
||||||
|
}
|
||||||
@@ -17,6 +17,96 @@ import {
|
|||||||
updateAdminV1Fact,
|
updateAdminV1Fact,
|
||||||
updateAdminV1Section,
|
updateAdminV1Section,
|
||||||
} from "./api";
|
} from "./api";
|
||||||
|
import {
|
||||||
|
cancelActiveRun,
|
||||||
|
cancelAllActiveRuns,
|
||||||
|
getMonitorSettings,
|
||||||
|
loadActiveRuns,
|
||||||
|
loadConcurrencySeries,
|
||||||
|
loadTokenSpeedSeries,
|
||||||
|
updateMonitorSettings,
|
||||||
|
type Granularity,
|
||||||
|
} from "./active-runs";
|
||||||
|
|
||||||
|
// ── 实时并发监控 (admin) ─────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/** Live snapshot of in-flight runs. Polls every `intervalMs` while `enabled`. */
|
||||||
|
export function useActiveRuns(enabled = true, intervalMs = 4000) {
|
||||||
|
return useQuery({
|
||||||
|
queryKey: ["admin-active-runs", "live"],
|
||||||
|
queryFn: loadActiveRuns,
|
||||||
|
enabled,
|
||||||
|
refetchInterval: enabled ? intervalMs : false,
|
||||||
|
refetchOnWindowFocus: false,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 并发量时序 (peak/avg per minute|hour). */
|
||||||
|
export function useConcurrencySeries(
|
||||||
|
granularity: Granularity,
|
||||||
|
hours: number | undefined,
|
||||||
|
enabled = true,
|
||||||
|
intervalMs = 15000,
|
||||||
|
) {
|
||||||
|
return useQuery({
|
||||||
|
queryKey: ["admin-active-runs", "concurrency", granularity, hours ?? null],
|
||||||
|
queryFn: () => loadConcurrencySeries(granularity, hours),
|
||||||
|
enabled,
|
||||||
|
refetchInterval: enabled ? intervalMs : false,
|
||||||
|
refetchOnWindowFocus: false,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 大模型 token 出字速度时序 (tokens/sec per minute|hour). */
|
||||||
|
export function useTokenSpeedSeries(
|
||||||
|
granularity: Granularity,
|
||||||
|
hours: number | undefined,
|
||||||
|
enabled = true,
|
||||||
|
intervalMs = 15000,
|
||||||
|
) {
|
||||||
|
return useQuery({
|
||||||
|
queryKey: ["admin-active-runs", "token-speed", granularity, hours ?? null],
|
||||||
|
queryFn: () => loadTokenSpeedSeries(granularity, hours),
|
||||||
|
enabled,
|
||||||
|
refetchInterval: enabled ? intervalMs : false,
|
||||||
|
refetchOnWindowFocus: false,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function useCancelActiveRun() {
|
||||||
|
const qc = useQueryClient();
|
||||||
|
return useMutation({
|
||||||
|
mutationFn: (runId: string) => cancelActiveRun(runId),
|
||||||
|
onSuccess: () => qc.invalidateQueries({ queryKey: ["admin-active-runs", "live"] }),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function useCancelAllActiveRuns() {
|
||||||
|
const qc = useQueryClient();
|
||||||
|
return useMutation({
|
||||||
|
mutationFn: () => cancelAllActiveRuns(),
|
||||||
|
onSuccess: () => qc.invalidateQueries({ queryKey: ["admin-active-runs", "live"] }),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function useMonitorSettings(enabled = true) {
|
||||||
|
return useQuery({
|
||||||
|
queryKey: ["admin-active-runs", "settings"],
|
||||||
|
queryFn: getMonitorSettings,
|
||||||
|
enabled,
|
||||||
|
refetchOnWindowFocus: false,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function useUpdateMonitorSettings() {
|
||||||
|
const qc = useQueryClient();
|
||||||
|
return useMutation({
|
||||||
|
mutationFn: (nextEnabled: boolean) => updateMonitorSettings(nextEnabled),
|
||||||
|
onSuccess: (data) => {
|
||||||
|
qc.setQueryData(["admin-active-runs", "settings"], data);
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
export function useUserLeaderboard(
|
export function useUserLeaderboard(
|
||||||
range: {
|
range: {
|
||||||
|
|||||||
@@ -219,6 +219,19 @@ function isRunNotFoundError(message: string): boolean {
|
|||||||
return /run\s+[\w-]{8,}\s+not\s+found/i.test(message);
|
return /run\s+[\w-]{8,}\s+not\s+found/i.test(message);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 同一会话上一轮还没生成完,又提交了新问题 → 后端返回 409
|
||||||
|
* 「already has an active run」。匹配 HTTP 409、英文兜底文案,或后端已中文化
|
||||||
|
* 的提示,统一映射成一句友好的中文。
|
||||||
|
*/
|
||||||
|
function isActiveRunConflictError(message: string): boolean {
|
||||||
|
return (
|
||||||
|
/\b409\b/.test(message) ||
|
||||||
|
/already has an active run/i.test(message) ||
|
||||||
|
/还在生成回答/.test(message)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
function buildCurrentUserLabel(): string {
|
function buildCurrentUserLabel(): string {
|
||||||
const auth = getStoredAuth();
|
const auth = getStoredAuth();
|
||||||
if (!auth) {
|
if (!auth) {
|
||||||
@@ -555,6 +568,12 @@ export function useThreadStream({
|
|||||||
}
|
}
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
// 上一轮还没结束又发了新问题(后端 409):给一句明确的中文提示,而不是
|
||||||
|
// 把原始的英文/HTTP 错误抛给用户。
|
||||||
|
if (isActiveRunConflictError(message)) {
|
||||||
|
toast.info("当前对话还在生成回答,请等待本轮回复结束,或先点「停止」再发送新的问题。");
|
||||||
|
return;
|
||||||
|
}
|
||||||
toast.error(message);
|
toast.error(message);
|
||||||
},
|
},
|
||||||
onFinish(state) {
|
onFinish(state) {
|
||||||
|
|||||||
@@ -46,6 +46,7 @@ import { ScrollArea } from "@/components/ui/scroll-area";
|
|||||||
import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "@/components/ui/select";
|
import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "@/components/ui/select";
|
||||||
import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs";
|
import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs";
|
||||||
import { MarkdownContent } from "@/components/workspace/messages/markdown-content";
|
import { MarkdownContent } from "@/components/workspace/messages/markdown-content";
|
||||||
|
import { ConcurrencyMonitorPanel } from "@/components/workspace/admin/concurrency-monitor";
|
||||||
import type {
|
import type {
|
||||||
AdminLeaderboardDimension,
|
AdminLeaderboardDimension,
|
||||||
AdminLeaderboardOverview,
|
AdminLeaderboardOverview,
|
||||||
@@ -1389,6 +1390,8 @@ export default function AdminUserLeaderboardPage() {
|
|||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div className="space-y-5 p-6">
|
<div className="space-y-5 p-6">
|
||||||
|
<ConcurrencyMonitorPanel />
|
||||||
|
|
||||||
{isLoading && (
|
{isLoading && (
|
||||||
<div className="flex flex-col items-center justify-center gap-3 py-20 text-center text-sm text-[#5F6368] dark:text-slate-400">
|
<div className="flex flex-col items-center justify-center gap-3 py-20 text-center text-sm text-[#5F6368] dark:text-slate-400">
|
||||||
<div className="flex items-center gap-2">
|
<div className="flex items-center gap-2">
|
||||||
|
|||||||
@@ -297,6 +297,15 @@ Two on-demand/automatic refresh paths layer on top:
|
|||||||
- **Manual "????"** ? `POST /leaderboard/refresh-today` force-rebuilds *today's* snapshot **synchronously** (bounded by the same semaphore/timeout the scheduler uses), then returns the merged range. Lets an admin pull the latest real-time records into the stats without waiting for the TTL. Frontend: a "????" button on `AdminUserLeaderboardPage` ? `useRefreshTodayLeaderboard` (`core/admin/hooks.ts`), which primes + invalidates the `["admin-activity","leaderboard"]` query.
|
- **Manual "????"** ? `POST /leaderboard/refresh-today` force-rebuilds *today's* snapshot **synchronously** (bounded by the same semaphore/timeout the scheduler uses), then returns the merged range. Lets an admin pull the latest real-time records into the stats without waiting for the TTL. Frontend: a "????" button on `AdminUserLeaderboardPage` ? `useRefreshTodayLeaderboard` (`core/admin/hooks.ts`), which primes + invalidates the `["admin-activity","leaderboard"]` query.
|
||||||
- **Daily finalize of the previous day** ? once per day, on/after `ADMIN_LEADERBOARD_FINALIZE_HOUR` (local Beijing, default `1` = ?? 1 ?), the scheduler tick force-recomputes *yesterday* a single time so runs/tool-calls that only landed near midnight are fully captured. The decision is the pure `deerflow.persistence.admin_stats.finalize.should_finalize_previous_day` (status/generated_at/now/hour) ? the rebuild stamps `generated_at` to today, which self-debounces it for the rest of the day (no extra state file). Tests: `tests/test_leaderboard_finalize.py`.
|
- **Daily finalize of the previous day** ? once per day, on/after `ADMIN_LEADERBOARD_FINALIZE_HOUR` (local Beijing, default `1` = ?? 1 ?), the scheduler tick force-recomputes *yesterday* a single time so runs/tool-calls that only landed near midnight are fully captured. The decision is the pure `deerflow.persistence.admin_stats.finalize.should_finalize_previous_day` (status/generated_at/now/hour) ? the rebuild stamps `generated_at` to today, which self-debounces it for the rest of the day (no extra state file). Tests: `tests/test_leaderboard_finalize.py`.
|
||||||
|
|
||||||
|
### Admin Concurrency Monitor (实时并发监控 / 系统压力)
|
||||||
|
|
||||||
|
A live admin monitor embedded **at the top of `AdminUserLeaderboardPage`** (`components/workspace/admin/concurrency-monitor.tsx`, `ConcurrencyMonitorPanel`) that answers "当前有几个并发的大模型调用、是谁、能不能直接停掉" and charts system pressure over time. Built on the existing in-memory `RunManager` (no new run plumbing).
|
||||||
|
|
||||||
|
- **Live concurrency + 真正停止** — `RunManager.list_active()` returns every run whose status is `pending`/`running` (the registry keeps finished records around until `cleanup`, so it filters by status). `len(list_active())` is the current concurrent-call count. Router `app/gateway/routers/admin_active_runs.py` (`/api/admin/active-runs`, **admin only** via the shared `admin_users._require_admin`): `GET /` returns `{count, server_time, runs:[{run_id, thread_id, status, created_at, elapsed_seconds, user_id, user_name, agent_id, agent_name, thread_title, kind, multitask_strategy}]}` — each run best-effort enriched (batched DB) with the owning user's email (`threads_meta.user_id`→`users.email`) and the agent's display name (`agents.name`); `kind` is derived from thread metadata (圆桌会商 / 系统/后台 / 任务对话 / 对话). `POST /{run_id}/cancel` calls `RunManager.cancel(run_id, action="interrupt")`, which sets the abort event **and cancels the run's asyncio task** — i.e. it truly aborts the in-flight model stream, freeing capacity. `POST /cancel-all` stops every in-flight run.
|
||||||
|
- **并发量折线图 (按分钟/小时)** — a lightweight background **sampler** (`_concurrency_sampler_loop` in `app/gateway/app.py`, singleton file-locked, started in the lifespan) records `len(list_active())` every `CONCURRENCY_SAMPLE_INTERVAL_SECONDS` (default 15s) into the new `concurrency_samples` table (`deerflow.persistence.concurrency`, `ConcurrencySampleRow{sampled_at, active_count}`, auto-created by `create_all`, **no migration**; store wired as `app.state.concurrency_sample_store`). ~Hourly it purges rows older than `CONCURRENCY_SAMPLE_RETENTION_HOURS` (default 168h). `GET /concurrency?granularity=minute|hour&hours=N` aggregates samples into per-bucket **peak + avg** (`ConcurrencySampleStore.aggregate`, bucketed in Python so it's identical across sqlite/mysql/postgres; fetch capped at 60k rows). Default lookback: minute→3h, hour→48h.
|
||||||
|
- **大模型出字速度折线图 (tokens/sec, 按分钟/小时)** — `GET /token-speed?granularity=&hours=` aggregates the persisted `llm_call_metrics` (`status="success"`) into per-bucket avg/min/max tokens/sec (deriving tokens/sec from `output_tokens÷duration_ms` when the stored value is null). A dip pinpoints the model/provider as the bottleneck for that window — "知道具体原因出现在哪里".
|
||||||
|
- **On/off + 不影响线上** — the whole feature is designed to never impact production: sampling is one tiny INSERT, retention-purged, all paths best-effort/guarded, queries bounded. A **runtime switch** lives in `system_settings.json → concurrency_monitor.enabled` (`ConcurrencyMonitorSettings`, **default false / opt-in**, mtime-hot-reloaded): the background loop always runs (just sleeps) but only samples when an admin turns it on, and it re-reads the flag **every tick** so toggling it on/off **takes effect immediately without a restart** (the on-demand live snapshot + 停止 stay available regardless; only the historical chart needs sampling on). `GET|PUT /api/admin/active-runs/settings` reads/writes it (admin only); the panel's 「后台采样」 Switch drives it. Env `CONCURRENCY_SAMPLE_ENABLED=0` is a hard master kill (never starts the loop). Frontend client/hooks: `core/admin/active-runs.ts` + `core/admin/hooks.ts` (`useActiveRuns` polls 4s while the panel is open, `useConcurrencySeries`/`useTokenSpeedSeries` poll 15s, `useCancelActiveRun`/`useCancelAllActiveRuns`, `useMonitorSettings`/`useUpdateMonitorSettings`). Tests: `tests/test_concurrency_monitor.py`.
|
||||||
|
|
||||||
### Scheduled Task Runtime
|
### Scheduled Task Runtime
|
||||||
|
|
||||||
`ScheduledTaskService` (`deerflow/runtime/scheduler/service.py`) polls due tasks every `poll_interval_seconds` and executes each in its own throwaway thread, serialized by a shared `_agent_run_lock`.
|
`ScheduledTaskService` (`deerflow/runtime/scheduler/service.py`) polls due tasks every `poll_interval_seconds` and executes each in its own throwaway thread, serialized by a shared `_agent_run_lock`.
|
||||||
|
|||||||
@@ -27,6 +27,7 @@ from app.gateway.csrf_middleware import CSRFMiddleware
|
|||||||
from app.gateway.deps import langgraph_runtime
|
from app.gateway.deps import langgraph_runtime
|
||||||
from app.gateway.routers import (
|
from app.gateway.routers import (
|
||||||
ai_writing,
|
ai_writing,
|
||||||
|
admin_active_runs,
|
||||||
admin_users,
|
admin_users,
|
||||||
agents,
|
agents,
|
||||||
artifacts,
|
artifacts,
|
||||||
@@ -488,6 +489,18 @@ _LEADERBOARD_SNAPSHOT_STARTUP_DELAY_SECONDS = _env_float(
|
|||||||
"ADMIN_LEADERBOARD_BACKGROUND_STARTUP_DELAY_SECONDS",
|
"ADMIN_LEADERBOARD_BACKGROUND_STARTUP_DELAY_SECONDS",
|
||||||
60.0,
|
60.0,
|
||||||
)
|
)
|
||||||
|
# Concurrency (system-pressure) sampler: how often to snapshot the in-flight run
|
||||||
|
# count, and how long to keep samples (drives the admin 并发量 line chart, which
|
||||||
|
# aggregates per minute/hour). Sampling is in-memory cheap (one tiny INSERT).
|
||||||
|
_CONCURRENCY_SAMPLE_ENABLED = _env_bool("CONCURRENCY_SAMPLE_ENABLED", True)
|
||||||
|
_CONCURRENCY_SAMPLE_INTERVAL_SECONDS = _env_float(
|
||||||
|
"CONCURRENCY_SAMPLE_INTERVAL_SECONDS",
|
||||||
|
15.0,
|
||||||
|
)
|
||||||
|
_CONCURRENCY_SAMPLE_RETENTION_HOURS = _env_float(
|
||||||
|
"CONCURRENCY_SAMPLE_RETENTION_HOURS",
|
||||||
|
168.0, # 7 days
|
||||||
|
)
|
||||||
# Hour (local Beijing time, 0-23) after which the scheduler force-recomputes the
|
# Hour (local Beijing time, 0-23) after which the scheduler force-recomputes the
|
||||||
# *previous* day's snapshot once, so records that landed near midnight are fully
|
# *previous* day's snapshot once, so records that landed near midnight are fully
|
||||||
# captured. Default 1 ? "???? 1 ????????".
|
# captured. Default 1 ? "???? 1 ????????".
|
||||||
@@ -828,6 +841,58 @@ async def _leaderboard_snapshot_scheduler_loop() -> None:
|
|||||||
_release_singleton_lock(lock_handle)
|
_release_singleton_lock(lock_handle)
|
||||||
|
|
||||||
|
|
||||||
|
async def _concurrency_sampler_loop(app: FastAPI) -> None:
|
||||||
|
"""Periodically record the in-flight run count for the admin pressure chart.
|
||||||
|
|
||||||
|
Runs in the gateway process, singleton-locked so only one worker samples.
|
||||||
|
Each tick records ``len(run_manager.list_active())`` and, roughly hourly,
|
||||||
|
purges samples older than the retention window. Best-effort throughout — a
|
||||||
|
failed tick is logged and the loop continues.
|
||||||
|
"""
|
||||||
|
if not _CONCURRENCY_SAMPLE_ENABLED:
|
||||||
|
logger.info("Concurrency sampler disabled by CONCURRENCY_SAMPLE_ENABLED")
|
||||||
|
return
|
||||||
|
|
||||||
|
store = getattr(app.state, "concurrency_sample_store", None)
|
||||||
|
run_mgr = getattr(app.state, "run_manager", None)
|
||||||
|
if store is None or run_mgr is None:
|
||||||
|
logger.info("Concurrency sampler not started (no store / run manager — memory backend?)")
|
||||||
|
return
|
||||||
|
|
||||||
|
from deerflow.config.runtime_paths import runtime_home
|
||||||
|
|
||||||
|
lock_handle = _acquire_singleton_lock(runtime_home() / ".concurrency_sampler.lock")
|
||||||
|
if lock_handle is None:
|
||||||
|
logger.info("Concurrency sampler: another gateway worker holds the lock; this worker skips")
|
||||||
|
return
|
||||||
|
|
||||||
|
interval = max(1.0, _CONCURRENCY_SAMPLE_INTERVAL_SECONDS)
|
||||||
|
purge_every_ticks = max(1, int(3600 / interval)) # ~once per hour
|
||||||
|
logger.info("Concurrency sampler started: interval=%.1fs retention=%.1fh", interval, _CONCURRENCY_SAMPLE_RETENTION_HOURS)
|
||||||
|
ticks = 0
|
||||||
|
try:
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
# Runtime toggle (admin can turn sampling off without a restart).
|
||||||
|
# mtime-cached read, so this is cheap every tick.
|
||||||
|
from deerflow.config.system_settings import get_concurrency_monitor_settings
|
||||||
|
|
||||||
|
if get_concurrency_monitor_settings().enabled:
|
||||||
|
active = await run_mgr.list_active()
|
||||||
|
await store.record(len(active))
|
||||||
|
ticks += 1
|
||||||
|
if ticks % purge_every_ticks == 0:
|
||||||
|
from datetime import UTC, datetime, timedelta
|
||||||
|
|
||||||
|
cutoff = datetime.now(UTC) - timedelta(hours=_CONCURRENCY_SAMPLE_RETENTION_HOURS)
|
||||||
|
await store.purge_older_than(cutoff)
|
||||||
|
except Exception:
|
||||||
|
logger.exception("Concurrency sampler tick failed (non-fatal)")
|
||||||
|
await asyncio.sleep(interval)
|
||||||
|
finally:
|
||||||
|
_release_singleton_lock(lock_handle)
|
||||||
|
|
||||||
|
|
||||||
def _warn_if_multi_worker() -> None:
|
def _warn_if_multi_worker() -> None:
|
||||||
"""?? uvicorn / gunicorn ? worker ???? ERROR ???
|
"""?? uvicorn / gunicorn ? worker ???? ERROR ???
|
||||||
|
|
||||||
@@ -1044,6 +1109,13 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
|
|||||||
except Exception:
|
except Exception:
|
||||||
logger.exception("Leaderboard snapshot scheduler failed to start (non-fatal)")
|
logger.exception("Leaderboard snapshot scheduler failed to start (non-fatal)")
|
||||||
|
|
||||||
|
# Concurrency (system-pressure) sampler — lightweight, singleton-locked,
|
||||||
|
# runtime-toggleable via system_settings.concurrency_monitor.enabled.
|
||||||
|
try:
|
||||||
|
app.state.concurrency_sampler_task = asyncio.create_task(_concurrency_sampler_loop(app))
|
||||||
|
except Exception:
|
||||||
|
logger.exception("Concurrency sampler failed to start (non-fatal)")
|
||||||
|
|
||||||
# Start IM channel service if any channels are configured
|
# Start IM channel service if any channels are configured
|
||||||
try:
|
try:
|
||||||
from app.channels.service import start_channel_service
|
from app.channels.service import start_channel_service
|
||||||
@@ -1102,6 +1174,16 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
|
|||||||
except Exception:
|
except Exception:
|
||||||
logger.exception("Leaderboard snapshot scheduler shutdown error")
|
logger.exception("Leaderboard snapshot scheduler shutdown error")
|
||||||
|
|
||||||
|
concurrency_sampler_task = getattr(app.state, "concurrency_sampler_task", None)
|
||||||
|
if concurrency_sampler_task is not None:
|
||||||
|
concurrency_sampler_task.cancel()
|
||||||
|
try:
|
||||||
|
await concurrency_sampler_task
|
||||||
|
except asyncio.CancelledError:
|
||||||
|
pass
|
||||||
|
except Exception:
|
||||||
|
logger.exception("Concurrency sampler shutdown error")
|
||||||
|
|
||||||
# ?? 7 ????AI ?? graph ???? checkpointer / in-flight task?
|
# ?? 7 ????AI ?? graph ???? checkpointer / in-flight task?
|
||||||
# thread ??? LangGraph Server ? checkpointer ???lifespan ???
|
# thread ??? LangGraph Server ? checkpointer ???lifespan ???
|
||||||
# ``langgraph_runtime()`` ? contextmanager ???????????????
|
# ``langgraph_runtime()`` ? contextmanager ???????????????
|
||||||
@@ -1434,6 +1516,8 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
|
|||||||
app.include_router(admin_users.router)
|
app.include_router(admin_users.router)
|
||||||
app.include_router(admin_users.self_router)
|
app.include_router(admin_users.self_router)
|
||||||
|
|
||||||
|
app.include_router(admin_active_runs.router)
|
||||||
|
|
||||||
# Admin tool / skill call metrics audit + export
|
# Admin tool / skill call metrics audit + export
|
||||||
app.include_router(tool_metrics.router)
|
app.include_router(tool_metrics.router)
|
||||||
|
|
||||||
|
|||||||
@@ -116,6 +116,10 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
|
|||||||
|
|
||||||
app.state.llm_metrics_store = make_llm_metrics_store(sf)
|
app.state.llm_metrics_store = make_llm_metrics_store(sf)
|
||||||
|
|
||||||
|
from deerflow.persistence.concurrency import make_concurrency_sample_store
|
||||||
|
|
||||||
|
app.state.concurrency_sample_store = make_concurrency_sample_store(sf)
|
||||||
|
|
||||||
from deerflow.persistence.tool_metrics import make_tool_metrics_store
|
from deerflow.persistence.tool_metrics import make_tool_metrics_store
|
||||||
|
|
||||||
app.state.tool_metrics_store = make_tool_metrics_store(sf)
|
app.state.tool_metrics_store = make_tool_metrics_store(sf)
|
||||||
@@ -433,6 +437,11 @@ def get_llm_metrics_store(request: Request):
|
|||||||
return getattr(request.app.state, "llm_metrics_store", None)
|
return getattr(request.app.state, "llm_metrics_store", None)
|
||||||
|
|
||||||
|
|
||||||
|
def get_concurrency_sample_store(request: Request):
|
||||||
|
"""Return the concurrency-sample store, or None on the memory backend."""
|
||||||
|
return getattr(request.app.state, "concurrency_sample_store", None)
|
||||||
|
|
||||||
|
|
||||||
def get_tool_metrics_store(request: Request):
|
def get_tool_metrics_store(request: Request):
|
||||||
"""Return the tool-call metrics store, or None when not configured."""
|
"""Return the tool-call metrics store, or None when not configured."""
|
||||||
return getattr(request.app.state, "tool_metrics_store", None)
|
return getattr(request.app.state, "tool_metrics_store", None)
|
||||||
|
|||||||
@@ -0,0 +1,406 @@
|
|||||||
|
"""Admin-only live concurrency monitor for in-flight model runs.
|
||||||
|
|
||||||
|
Surfaces every *currently running* call into the agent runtime (interactive
|
||||||
|
chat, agent chat, roundtable seats, …) so an administrator can see how much
|
||||||
|
concurrent pressure the system is putting on the LLM and **truly stop** any of
|
||||||
|
them — ``RunManager.cancel`` aborts the run's asyncio task, which cancels the
|
||||||
|
in-flight model stream, immediately freeing that capacity.
|
||||||
|
|
||||||
|
All endpoints require ``system_role == "admin"`` (shared ``_require_admin``).
|
||||||
|
The data source is the in-memory :class:`RunManager` registry (``list_active``),
|
||||||
|
enriched best-effort with the owning user's email + the agent's display name
|
||||||
|
from the DB. Enrichment failures degrade gracefully (the run still shows, just
|
||||||
|
with rawer labels) — the monitor must never break because a lookup failed.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from datetime import UTC, datetime, timedelta
|
||||||
|
|
||||||
|
from fastapi import APIRouter, Query, Request
|
||||||
|
from pydantic import BaseModel
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
|
from app.gateway.deps import get_concurrency_sample_store, get_run_manager
|
||||||
|
from app.gateway.routers.admin_users import _require_admin
|
||||||
|
from deerflow.config.system_settings import load_system_settings, save_system_settings
|
||||||
|
from deerflow.persistence.agents.model import AgentRow
|
||||||
|
from deerflow.persistence.engine import get_session_factory
|
||||||
|
from deerflow.persistence.llm_metrics.model import LlmCallMetricRow
|
||||||
|
from deerflow.persistence.thread_meta.model import ThreadMetaRow
|
||||||
|
from deerflow.persistence.user.model import UserRow
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
router = APIRouter(prefix="/api/admin/active-runs", tags=["admin-active-runs"])
|
||||||
|
|
||||||
|
# Lookback windows are clamped so an aggregation never scans an unbounded range.
|
||||||
|
_MAX_LOOKBACK_HOURS = 24 * 14 # 14 days
|
||||||
|
|
||||||
|
# Per-granularity default lookback (hours): minute view = recent, hour view = wider.
|
||||||
|
_DEFAULT_LOOKBACK_HOURS = {"minute": 3, "hour": 48}
|
||||||
|
|
||||||
|
# assistant_id values that are not real custom-agent ids (the default lead agent).
|
||||||
|
_DEFAULT_ASSISTANT_IDS = {"", "default", "lead_agent", "agent"}
|
||||||
|
|
||||||
|
|
||||||
|
class ActiveRunItem(BaseModel):
|
||||||
|
run_id: str
|
||||||
|
thread_id: str
|
||||||
|
status: str
|
||||||
|
created_at: str = ""
|
||||||
|
elapsed_seconds: int = 0
|
||||||
|
user_id: str = ""
|
||||||
|
user_name: str = ""
|
||||||
|
agent_id: str = ""
|
||||||
|
agent_name: str = ""
|
||||||
|
thread_title: str = ""
|
||||||
|
kind: str = "对话"
|
||||||
|
multitask_strategy: str = "reject"
|
||||||
|
|
||||||
|
|
||||||
|
class ActiveRunsResponse(BaseModel):
|
||||||
|
count: int
|
||||||
|
server_time: str
|
||||||
|
runs: list[ActiveRunItem]
|
||||||
|
|
||||||
|
|
||||||
|
class CancelRunResponse(BaseModel):
|
||||||
|
run_id: str
|
||||||
|
cancelled: bool
|
||||||
|
|
||||||
|
|
||||||
|
class CancelAllResponse(BaseModel):
|
||||||
|
cancelled_count: int
|
||||||
|
|
||||||
|
|
||||||
|
class ConcurrencyPoint(BaseModel):
|
||||||
|
bucket: str
|
||||||
|
peak: int
|
||||||
|
avg: float
|
||||||
|
samples: int
|
||||||
|
|
||||||
|
|
||||||
|
class ConcurrencySeriesResponse(BaseModel):
|
||||||
|
granularity: str
|
||||||
|
since: str
|
||||||
|
until: str
|
||||||
|
enabled: bool
|
||||||
|
points: list[ConcurrencyPoint]
|
||||||
|
|
||||||
|
|
||||||
|
class TokenSpeedPoint(BaseModel):
|
||||||
|
bucket: str
|
||||||
|
avg_tokens_per_sec: float
|
||||||
|
min_tokens_per_sec: float
|
||||||
|
max_tokens_per_sec: float
|
||||||
|
samples: int
|
||||||
|
|
||||||
|
|
||||||
|
class TokenSpeedSeriesResponse(BaseModel):
|
||||||
|
granularity: str
|
||||||
|
since: str
|
||||||
|
until: str
|
||||||
|
points: list[TokenSpeedPoint]
|
||||||
|
|
||||||
|
|
||||||
|
class MonitorSettingsResponse(BaseModel):
|
||||||
|
enabled: bool
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_window(granularity: str, hours: float | None) -> tuple[str, datetime, datetime]:
|
||||||
|
"""Clamp the granularity + lookback window into (gran, since, until)."""
|
||||||
|
gran = "hour" if granularity == "hour" else "minute"
|
||||||
|
lookback = hours if hours and hours > 0 else _DEFAULT_LOOKBACK_HOURS[gran]
|
||||||
|
lookback = min(float(lookback), _MAX_LOOKBACK_HOURS)
|
||||||
|
until = datetime.now(UTC)
|
||||||
|
since = until - timedelta(hours=lookback)
|
||||||
|
return gran, since, until
|
||||||
|
|
||||||
|
|
||||||
|
def _bucket_start(dt: datetime, granularity: str) -> datetime:
|
||||||
|
if granularity == "hour":
|
||||||
|
return dt.replace(minute=0, second=0, microsecond=0)
|
||||||
|
return dt.replace(second=0, microsecond=0)
|
||||||
|
|
||||||
|
|
||||||
|
def _derive_tokens_per_sec(tps, output_tokens, total_tokens, duration_ms) -> float | None:
|
||||||
|
"""Use the stored tokens/sec, else derive from tokens ÷ duration."""
|
||||||
|
if tps not in (None, 0):
|
||||||
|
return float(tps)
|
||||||
|
numerator = output_tokens if output_tokens else total_tokens
|
||||||
|
if not numerator or not duration_ms or duration_ms <= 0:
|
||||||
|
return None
|
||||||
|
return round((float(numerator) / float(duration_ms)) * 1000.0, 3)
|
||||||
|
|
||||||
|
|
||||||
|
def _elapsed_seconds(created_at: str, now: datetime) -> int:
|
||||||
|
"""Seconds since the (UTC ISO) ``created_at`` stamp; 0 on any parse error."""
|
||||||
|
if not created_at:
|
||||||
|
return 0
|
||||||
|
try:
|
||||||
|
started = datetime.fromisoformat(created_at)
|
||||||
|
except (ValueError, TypeError):
|
||||||
|
return 0
|
||||||
|
if started.tzinfo is None:
|
||||||
|
started = started.replace(tzinfo=UTC)
|
||||||
|
delta = (now - started).total_seconds()
|
||||||
|
return max(0, int(delta))
|
||||||
|
|
||||||
|
|
||||||
|
def _kind_for(metadata: dict | None) -> str:
|
||||||
|
"""Coarse human label for the run's origin, from its thread metadata."""
|
||||||
|
meta = metadata or {}
|
||||||
|
thread_type = str(meta.get("thread_type") or "")
|
||||||
|
if thread_type == "roundtable":
|
||||||
|
return "圆桌会商"
|
||||||
|
if meta.get("system"):
|
||||||
|
return "系统/后台"
|
||||||
|
if meta.get("taskId"):
|
||||||
|
return "任务对话"
|
||||||
|
return "对话"
|
||||||
|
|
||||||
|
|
||||||
|
async def _enrich(records) -> dict:
|
||||||
|
"""Batch-resolve thread metadata, user emails and agent names.
|
||||||
|
|
||||||
|
Returns a dict of lookup maps; every query is wrapped so a failure just
|
||||||
|
yields empty maps (the runs still render with raw ids).
|
||||||
|
"""
|
||||||
|
thread_ids = {r.thread_id for r in records if r.thread_id}
|
||||||
|
assistant_ids = {
|
||||||
|
r.assistant_id
|
||||||
|
for r in records
|
||||||
|
if r.assistant_id and r.assistant_id not in _DEFAULT_ASSISTANT_IDS
|
||||||
|
}
|
||||||
|
|
||||||
|
thread_meta: dict[str, ThreadMetaRow] = {}
|
||||||
|
user_emails: dict[str, str] = {}
|
||||||
|
agent_names: dict[str, str] = {}
|
||||||
|
|
||||||
|
try:
|
||||||
|
session_factory = get_session_factory()
|
||||||
|
async with session_factory() as session:
|
||||||
|
if thread_ids:
|
||||||
|
rows = (
|
||||||
|
await session.execute(
|
||||||
|
select(ThreadMetaRow).where(ThreadMetaRow.thread_id.in_(thread_ids))
|
||||||
|
)
|
||||||
|
).scalars().all()
|
||||||
|
thread_meta = {row.thread_id: row for row in rows}
|
||||||
|
|
||||||
|
user_ids = {row.user_id for row in thread_meta.values() if row.user_id}
|
||||||
|
if user_ids:
|
||||||
|
rows = (
|
||||||
|
await session.execute(
|
||||||
|
select(UserRow.id, UserRow.email).where(UserRow.id.in_(user_ids))
|
||||||
|
)
|
||||||
|
).all()
|
||||||
|
user_emails = {r[0]: (r[1] or "") for r in rows}
|
||||||
|
|
||||||
|
if assistant_ids:
|
||||||
|
rows = (
|
||||||
|
await session.execute(
|
||||||
|
select(AgentRow.id, AgentRow.name).where(AgentRow.id.in_(assistant_ids))
|
||||||
|
)
|
||||||
|
).all()
|
||||||
|
agent_names = {r[0]: (r[1] or "") for r in rows}
|
||||||
|
except Exception: # pragma: no cover - enrichment is best-effort
|
||||||
|
logger.warning("Failed to enrich active runs (non-fatal)", exc_info=True)
|
||||||
|
|
||||||
|
return {"thread_meta": thread_meta, "user_emails": user_emails, "agent_names": agent_names}
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("", response_model=ActiveRunsResponse)
|
||||||
|
@router.get("/", response_model=ActiveRunsResponse)
|
||||||
|
async def list_active_runs(request: Request) -> ActiveRunsResponse:
|
||||||
|
"""List every in-flight run with who/what is driving it (admin only)."""
|
||||||
|
await _require_admin(request)
|
||||||
|
run_mgr = get_run_manager(request)
|
||||||
|
records = await run_mgr.list_active()
|
||||||
|
now = datetime.now(UTC)
|
||||||
|
maps = await _enrich(records)
|
||||||
|
thread_meta = maps["thread_meta"]
|
||||||
|
user_emails = maps["user_emails"]
|
||||||
|
agent_names = maps["agent_names"]
|
||||||
|
|
||||||
|
items: list[ActiveRunItem] = []
|
||||||
|
for r in records:
|
||||||
|
meta_row = thread_meta.get(r.thread_id)
|
||||||
|
user_id = (meta_row.user_id if meta_row else "") or ""
|
||||||
|
metadata = (meta_row.metadata_json if meta_row else None) or r.metadata or {}
|
||||||
|
assistant_id = r.assistant_id or ""
|
||||||
|
if assistant_id in _DEFAULT_ASSISTANT_IDS:
|
||||||
|
agent_name = "默认助手"
|
||||||
|
else:
|
||||||
|
agent_name = agent_names.get(assistant_id) or assistant_id
|
||||||
|
items.append(
|
||||||
|
ActiveRunItem(
|
||||||
|
run_id=r.run_id,
|
||||||
|
thread_id=r.thread_id,
|
||||||
|
status=r.status.value,
|
||||||
|
created_at=r.created_at,
|
||||||
|
elapsed_seconds=_elapsed_seconds(r.created_at, now),
|
||||||
|
user_id=user_id,
|
||||||
|
user_name=user_emails.get(user_id) or user_id or "未知用户",
|
||||||
|
agent_id=assistant_id,
|
||||||
|
agent_name=agent_name,
|
||||||
|
thread_title=(meta_row.display_name if meta_row else "") or "",
|
||||||
|
kind=_kind_for(metadata),
|
||||||
|
multitask_strategy=r.multitask_strategy,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
return ActiveRunsResponse(count=len(items), server_time=now.isoformat(), runs=items)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/{run_id}/cancel", response_model=CancelRunResponse)
|
||||||
|
async def cancel_active_run(
|
||||||
|
run_id: str,
|
||||||
|
request: Request,
|
||||||
|
action: str = Query(default="interrupt", description="interrupt 保留检查点 / rollback 回滚"),
|
||||||
|
) -> CancelRunResponse:
|
||||||
|
"""Truly stop one in-flight run — cancels its task and the live model stream."""
|
||||||
|
await _require_admin(request)
|
||||||
|
run_mgr = get_run_manager(request)
|
||||||
|
normalized = "rollback" if action == "rollback" else "interrupt"
|
||||||
|
cancelled = await run_mgr.cancel(run_id, action=normalized)
|
||||||
|
return CancelRunResponse(run_id=run_id, cancelled=cancelled)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/cancel-all", response_model=CancelAllResponse)
|
||||||
|
async def cancel_all_active_runs(request: Request) -> CancelAllResponse:
|
||||||
|
"""Stop every in-flight run at once (admin emergency relief valve)."""
|
||||||
|
await _require_admin(request)
|
||||||
|
run_mgr = get_run_manager(request)
|
||||||
|
records = await run_mgr.list_active()
|
||||||
|
cancelled = 0
|
||||||
|
for r in records:
|
||||||
|
if await run_mgr.cancel(r.run_id, action="interrupt"):
|
||||||
|
cancelled += 1
|
||||||
|
return CancelAllResponse(cancelled_count=cancelled)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/concurrency", response_model=ConcurrencySeriesResponse)
|
||||||
|
async def concurrency_series(
|
||||||
|
request: Request,
|
||||||
|
granularity: str = Query(default="minute", description="minute | hour"),
|
||||||
|
hours: float | None = Query(default=None, description="回看小时数(默认按粒度取)"),
|
||||||
|
) -> ConcurrencySeriesResponse:
|
||||||
|
"""Per-minute / per-hour 并发量 series (peak + avg) for the pressure chart."""
|
||||||
|
await _require_admin(request)
|
||||||
|
gran, since, until = _resolve_window(granularity, hours)
|
||||||
|
enabled = load_system_settings().concurrency_monitor.enabled
|
||||||
|
store = get_concurrency_sample_store(request)
|
||||||
|
points: list[ConcurrencyPoint] = []
|
||||||
|
if store is not None:
|
||||||
|
buckets = await store.aggregate(since=since, until=until, granularity=gran)
|
||||||
|
points = [
|
||||||
|
ConcurrencyPoint(bucket=b.bucket, peak=b.peak, avg=b.avg, samples=b.samples)
|
||||||
|
for b in buckets
|
||||||
|
]
|
||||||
|
return ConcurrencySeriesResponse(
|
||||||
|
granularity=gran,
|
||||||
|
since=since.isoformat(),
|
||||||
|
until=until.isoformat(),
|
||||||
|
enabled=enabled,
|
||||||
|
points=points,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/token-speed", response_model=TokenSpeedSeriesResponse)
|
||||||
|
async def token_speed_series(
|
||||||
|
request: Request,
|
||||||
|
granularity: str = Query(default="minute", description="minute | hour"),
|
||||||
|
hours: float | None = Query(default=None, description="回看小时数(默认按粒度取)"),
|
||||||
|
) -> TokenSpeedSeriesResponse:
|
||||||
|
"""Per-minute / per-hour 大模型出字速度 (tokens/sec) series.
|
||||||
|
|
||||||
|
Aggregated from the persisted LLM-call metrics: a dip in tokens/sec for a
|
||||||
|
bucket points at the model/provider being the bottleneck for that window.
|
||||||
|
"""
|
||||||
|
await _require_admin(request)
|
||||||
|
gran, since, until = _resolve_window(granularity, hours)
|
||||||
|
|
||||||
|
buckets: dict[datetime, dict[str, float]] = {}
|
||||||
|
try:
|
||||||
|
session_factory = get_session_factory()
|
||||||
|
if session_factory is not None:
|
||||||
|
async with session_factory() as session:
|
||||||
|
rows = (
|
||||||
|
await session.execute(
|
||||||
|
select(
|
||||||
|
LlmCallMetricRow.created_at,
|
||||||
|
LlmCallMetricRow.tokens_per_sec,
|
||||||
|
LlmCallMetricRow.output_tokens,
|
||||||
|
LlmCallMetricRow.total_tokens,
|
||||||
|
LlmCallMetricRow.duration_ms,
|
||||||
|
)
|
||||||
|
.where(
|
||||||
|
LlmCallMetricRow.created_at >= since,
|
||||||
|
LlmCallMetricRow.created_at <= until,
|
||||||
|
LlmCallMetricRow.status == "success",
|
||||||
|
)
|
||||||
|
.order_by(LlmCallMetricRow.created_at)
|
||||||
|
.limit(100_000)
|
||||||
|
)
|
||||||
|
).all()
|
||||||
|
for created_at, tps_raw, output_tokens, total_tokens, duration_ms in rows:
|
||||||
|
tps = _derive_tokens_per_sec(tps_raw, output_tokens, total_tokens, duration_ms)
|
||||||
|
if tps is None or tps <= 0 or created_at is None:
|
||||||
|
continue
|
||||||
|
if created_at.tzinfo is None:
|
||||||
|
created_at = created_at.replace(tzinfo=UTC)
|
||||||
|
key = _bucket_start(created_at, gran)
|
||||||
|
entry = buckets.get(key)
|
||||||
|
if entry is None:
|
||||||
|
buckets[key] = {"sum": tps, "min": tps, "max": tps, "count": 1}
|
||||||
|
else:
|
||||||
|
entry["sum"] += tps
|
||||||
|
entry["min"] = min(entry["min"], tps)
|
||||||
|
entry["max"] = max(entry["max"], tps)
|
||||||
|
entry["count"] += 1
|
||||||
|
except Exception: # pragma: no cover - chart must degrade gracefully
|
||||||
|
logger.warning("Failed to aggregate token-speed series (non-fatal)", exc_info=True)
|
||||||
|
|
||||||
|
points: list[TokenSpeedPoint] = []
|
||||||
|
for key in sorted(buckets):
|
||||||
|
entry = buckets[key]
|
||||||
|
count = max(1, int(entry["count"]))
|
||||||
|
points.append(
|
||||||
|
TokenSpeedPoint(
|
||||||
|
bucket=key.isoformat(),
|
||||||
|
avg_tokens_per_sec=round(entry["sum"] / count, 2),
|
||||||
|
min_tokens_per_sec=round(entry["min"], 2),
|
||||||
|
max_tokens_per_sec=round(entry["max"], 2),
|
||||||
|
samples=count,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return TokenSpeedSeriesResponse(
|
||||||
|
granularity=gran,
|
||||||
|
since=since.isoformat(),
|
||||||
|
until=until.isoformat(),
|
||||||
|
points=points,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/settings", response_model=MonitorSettingsResponse)
|
||||||
|
async def get_monitor_settings(request: Request) -> MonitorSettingsResponse:
|
||||||
|
"""Read the concurrency-monitor on/off switch (admin only)."""
|
||||||
|
await _require_admin(request)
|
||||||
|
return MonitorSettingsResponse(enabled=load_system_settings().concurrency_monitor.enabled)
|
||||||
|
|
||||||
|
|
||||||
|
@router.put("/settings", response_model=MonitorSettingsResponse)
|
||||||
|
async def update_monitor_settings(
|
||||||
|
request: Request,
|
||||||
|
body: MonitorSettingsResponse,
|
||||||
|
) -> MonitorSettingsResponse:
|
||||||
|
"""Toggle the background concurrency sampler on/off (admin only, no restart)."""
|
||||||
|
await _require_admin(request)
|
||||||
|
settings = load_system_settings()
|
||||||
|
updated = settings.model_copy(
|
||||||
|
update={"concurrency_monitor": settings.concurrency_monitor.model_copy(update={"enabled": bool(body.enabled)})}
|
||||||
|
)
|
||||||
|
save_system_settings(updated)
|
||||||
|
return MonitorSettingsResponse(enabled=updated.concurrency_monitor.enabled)
|
||||||
@@ -581,7 +581,12 @@ async def start_run(
|
|||||||
multitask_strategy=body.multitask_strategy,
|
multitask_strategy=body.multitask_strategy,
|
||||||
)
|
)
|
||||||
except ConflictError as exc:
|
except ConflictError as exc:
|
||||||
raise HTTPException(status_code=409, detail=str(exc)) from exc
|
# 同一会话上一轮回答还没结束,又提交了新问题 → 拒绝。给用户中文提示,
|
||||||
|
# 而不是把内部英文「Thread ... already has an active run」直接抛到前端。
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=409,
|
||||||
|
detail="当前对话还在生成回答,请等待本轮回复结束,或先点「停止」再发送新的问题。",
|
||||||
|
) from exc
|
||||||
except UnsupportedStrategyError as exc:
|
except UnsupportedStrategyError as exc:
|
||||||
raise HTTPException(status_code=501, detail=str(exc)) from exc
|
raise HTTPException(status_code=501, detail=str(exc)) from exc
|
||||||
t_post_create = time.perf_counter()
|
t_post_create = time.perf_counter()
|
||||||
|
|||||||
@@ -234,6 +234,21 @@ class LoginWhitelistSettings(BaseModel):
|
|||||||
model_config = ConfigDict(extra="ignore")
|
model_config = ConfigDict(extra="ignore")
|
||||||
|
|
||||||
|
|
||||||
|
class ConcurrencyMonitorSettings(BaseModel):
|
||||||
|
"""管理员「实时并发监控」开关(排行榜页的系统压力监控)。
|
||||||
|
|
||||||
|
``enabled`` 控制**后台并发采样器**是否持续把当前并发调用数写库(供并发量
|
||||||
|
折线图按分钟/小时聚合)。**默认关**(opt-in,由管理员在排行榜页手动开启),
|
||||||
|
以对线上服务零影响为先;开启后采样器才开始写库,关闭则立即停止(每 tick 都
|
||||||
|
重读该开关,免重启),实时快照查询与「停止」始终可用、零常驻成本。设计上即便
|
||||||
|
开启也对线上零影响:采样仅一条极小 INSERT、有保留期清理、聚合查询有行数上限、
|
||||||
|
全程 best-effort。
|
||||||
|
"""
|
||||||
|
|
||||||
|
enabled: bool = Field(default=False, description="是否启用后台并发采样(默认关;开→开始写库,关→停止写库、折线图不再新增历史点)")
|
||||||
|
model_config = ConfigDict(extra="ignore")
|
||||||
|
|
||||||
|
|
||||||
class SystemSettings(BaseModel):
|
class SystemSettings(BaseModel):
|
||||||
"""Top-level system settings document."""
|
"""Top-level system settings document."""
|
||||||
|
|
||||||
@@ -246,6 +261,7 @@ class SystemSettings(BaseModel):
|
|||||||
leaderboard: LeaderboardSettings = Field(default_factory=LeaderboardSettings)
|
leaderboard: LeaderboardSettings = Field(default_factory=LeaderboardSettings)
|
||||||
skill_compression: SkillCompressionSettings = Field(default_factory=SkillCompressionSettings)
|
skill_compression: SkillCompressionSettings = Field(default_factory=SkillCompressionSettings)
|
||||||
agent_modules: AgentModuleSettings = Field(default_factory=AgentModuleSettings)
|
agent_modules: AgentModuleSettings = Field(default_factory=AgentModuleSettings)
|
||||||
|
concurrency_monitor: ConcurrencyMonitorSettings = Field(default_factory=ConcurrencyMonitorSettings)
|
||||||
model_config = ConfigDict(extra="ignore")
|
model_config = ConfigDict(extra="ignore")
|
||||||
|
|
||||||
|
|
||||||
@@ -280,6 +296,15 @@ def get_skill_compression_settings() -> SkillCompressionSettings:
|
|||||||
return SkillCompressionSettings()
|
return SkillCompressionSettings()
|
||||||
|
|
||||||
|
|
||||||
|
def get_concurrency_monitor_settings() -> ConcurrencyMonitorSettings:
|
||||||
|
"""Convenience accessor for the concurrency-monitor block (safe defaults on error)."""
|
||||||
|
try:
|
||||||
|
return load_system_settings().concurrency_monitor
|
||||||
|
except Exception:
|
||||||
|
logger.debug("Failed to read concurrency monitor settings; using defaults", exc_info=True)
|
||||||
|
return ConcurrencyMonitorSettings()
|
||||||
|
|
||||||
|
|
||||||
def save_system_settings(settings: SystemSettings) -> SystemSettings:
|
def save_system_settings(settings: SystemSettings) -> SystemSettings:
|
||||||
"""Persist the settings document atomically, returning the saved value."""
|
"""Persist the settings document atomically, returning the saved value."""
|
||||||
path = _settings_path()
|
path = _settings_path()
|
||||||
|
|||||||
+15
@@ -0,0 +1,15 @@
|
|||||||
|
"""Concurrency (system-pressure) sampling persistence."""
|
||||||
|
|
||||||
|
from deerflow.persistence.concurrency.model import ConcurrencySampleRow
|
||||||
|
from deerflow.persistence.concurrency.store import (
|
||||||
|
ConcurrencyBucket,
|
||||||
|
ConcurrencySampleStore,
|
||||||
|
make_concurrency_sample_store,
|
||||||
|
)
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"ConcurrencyBucket",
|
||||||
|
"ConcurrencySampleRow",
|
||||||
|
"ConcurrencySampleStore",
|
||||||
|
"make_concurrency_sample_store",
|
||||||
|
]
|
||||||
+31
@@ -0,0 +1,31 @@
|
|||||||
|
"""ORM model for periodic concurrency (system-pressure) samples.
|
||||||
|
|
||||||
|
A background sampler writes one row every few seconds holding the number of
|
||||||
|
in-flight model runs at that instant. The admin monitor aggregates these into
|
||||||
|
per-minute / per-hour buckets to draw the 并发量 pressure line chart, even for
|
||||||
|
time windows before the admin opened the page.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
|
||||||
|
from sqlalchemy import Integer
|
||||||
|
from sqlalchemy.orm import Mapped, mapped_column
|
||||||
|
|
||||||
|
from deerflow.persistence.base import Base
|
||||||
|
from deerflow.persistence.types import BeijingDateTime
|
||||||
|
|
||||||
|
|
||||||
|
class ConcurrencySampleRow(Base):
|
||||||
|
"""One sampled snapshot of concurrent in-flight model runs."""
|
||||||
|
|
||||||
|
__tablename__ = "concurrency_samples"
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
|
||||||
|
# Wall-clock time the sample was taken (Beijing time on disk, tz-aware in Python).
|
||||||
|
sampled_at: Mapped[datetime] = mapped_column(
|
||||||
|
BeijingDateTime(), index=True, default=lambda: datetime.now(UTC)
|
||||||
|
)
|
||||||
|
# Number of runs whose status was pending/running at ``sampled_at``.
|
||||||
|
active_count: Mapped[int] = mapped_column(Integer, default=0)
|
||||||
+142
@@ -0,0 +1,142 @@
|
|||||||
|
"""Async store for concurrency samples + minute/hour aggregation."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from datetime import UTC, datetime, timedelta
|
||||||
|
|
||||||
|
from sqlalchemy import delete, select
|
||||||
|
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||||
|
|
||||||
|
from deerflow.persistence.concurrency.model import ConcurrencySampleRow
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
# How far back the aggregation will fetch raw rows before bucketing. A safety
|
||||||
|
# cap so a degenerate range never tries to load the whole table into memory.
|
||||||
|
_MAX_ROWS = 60_000
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ConcurrencyBucket:
|
||||||
|
"""One minute/hour bucket of the concurrency series."""
|
||||||
|
|
||||||
|
bucket: str # ISO start of the bucket
|
||||||
|
peak: int # max concurrent runs observed in the bucket
|
||||||
|
avg: float # mean concurrent runs in the bucket
|
||||||
|
samples: int # how many raw samples fell into the bucket
|
||||||
|
|
||||||
|
|
||||||
|
def _bucket_start(dt: datetime, granularity: str) -> datetime:
|
||||||
|
"""Truncate a timestamp to the start of its minute/hour bucket."""
|
||||||
|
if granularity == "hour":
|
||||||
|
return dt.replace(minute=0, second=0, microsecond=0)
|
||||||
|
return dt.replace(second=0, microsecond=0)
|
||||||
|
|
||||||
|
|
||||||
|
class ConcurrencySampleStore:
|
||||||
|
"""Persists periodic concurrency snapshots and aggregates them."""
|
||||||
|
|
||||||
|
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
|
||||||
|
self._sf = session_factory
|
||||||
|
|
||||||
|
async def record(self, active_count: int, *, sampled_at: datetime | None = None) -> None:
|
||||||
|
"""Insert one sample. Best-effort — never raises into the caller."""
|
||||||
|
try:
|
||||||
|
async with self._sf() as session:
|
||||||
|
session.add(
|
||||||
|
ConcurrencySampleRow(
|
||||||
|
active_count=max(0, int(active_count)),
|
||||||
|
sampled_at=sampled_at or datetime.now(UTC),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
await session.commit()
|
||||||
|
except Exception: # pragma: no cover - sampling must never break the app
|
||||||
|
logger.warning("Failed to record concurrency sample", exc_info=True)
|
||||||
|
|
||||||
|
async def aggregate(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
since: datetime,
|
||||||
|
until: datetime,
|
||||||
|
granularity: str = "minute",
|
||||||
|
) -> list[ConcurrencyBucket]:
|
||||||
|
"""Return per-minute / per-hour buckets of peak + average concurrency.
|
||||||
|
|
||||||
|
Bucketing is done in Python (not SQL ``date_trunc``/``strftime``) so the
|
||||||
|
logic is identical across the sqlite/mysql/postgres backends.
|
||||||
|
"""
|
||||||
|
gran = "hour" if granularity == "hour" else "minute"
|
||||||
|
try:
|
||||||
|
async with self._sf() as session:
|
||||||
|
rows = (
|
||||||
|
await session.execute(
|
||||||
|
select(ConcurrencySampleRow.sampled_at, ConcurrencySampleRow.active_count)
|
||||||
|
.where(
|
||||||
|
ConcurrencySampleRow.sampled_at >= since,
|
||||||
|
ConcurrencySampleRow.sampled_at <= until,
|
||||||
|
)
|
||||||
|
.order_by(ConcurrencySampleRow.sampled_at)
|
||||||
|
.limit(_MAX_ROWS)
|
||||||
|
)
|
||||||
|
).all()
|
||||||
|
except Exception:
|
||||||
|
logger.warning("Failed to aggregate concurrency samples", exc_info=True)
|
||||||
|
return []
|
||||||
|
|
||||||
|
buckets: dict[datetime, dict[str, float]] = {}
|
||||||
|
for sampled_at, active_count in rows:
|
||||||
|
if sampled_at is None:
|
||||||
|
continue
|
||||||
|
if sampled_at.tzinfo is None:
|
||||||
|
sampled_at = sampled_at.replace(tzinfo=UTC)
|
||||||
|
key = _bucket_start(sampled_at, gran)
|
||||||
|
value = int(active_count or 0)
|
||||||
|
entry = buckets.get(key)
|
||||||
|
if entry is None:
|
||||||
|
buckets[key] = {"peak": value, "sum": value, "count": 1}
|
||||||
|
else:
|
||||||
|
entry["peak"] = max(entry["peak"], value)
|
||||||
|
entry["sum"] += value
|
||||||
|
entry["count"] += 1
|
||||||
|
|
||||||
|
result: list[ConcurrencyBucket] = []
|
||||||
|
for key in sorted(buckets):
|
||||||
|
entry = buckets[key]
|
||||||
|
count = max(1, int(entry["count"]))
|
||||||
|
result.append(
|
||||||
|
ConcurrencyBucket(
|
||||||
|
bucket=key.isoformat(),
|
||||||
|
peak=int(entry["peak"]),
|
||||||
|
avg=round(entry["sum"] / count, 2),
|
||||||
|
samples=count,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return result
|
||||||
|
|
||||||
|
async def purge_older_than(self, cutoff: datetime) -> int:
|
||||||
|
"""Delete samples older than ``cutoff``. Returns rows removed."""
|
||||||
|
try:
|
||||||
|
async with self._sf() as session:
|
||||||
|
result = await session.execute(
|
||||||
|
delete(ConcurrencySampleRow).where(ConcurrencySampleRow.sampled_at < cutoff)
|
||||||
|
)
|
||||||
|
await session.commit()
|
||||||
|
return int(result.rowcount or 0)
|
||||||
|
except Exception: # pragma: no cover
|
||||||
|
logger.warning("Failed to purge concurrency samples", exc_info=True)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def make_concurrency_sample_store(
|
||||||
|
session_factory: async_sessionmaker[AsyncSession] | None,
|
||||||
|
) -> ConcurrencySampleStore | None:
|
||||||
|
"""Factory mirroring the other ``make_*_store`` helpers.
|
||||||
|
|
||||||
|
Returns ``None`` on the memory backend (no SQL session factory), in which
|
||||||
|
case the monitor simply reports a live snapshot with no historical chart.
|
||||||
|
"""
|
||||||
|
if session_factory is None:
|
||||||
|
return None
|
||||||
|
return ConcurrencySampleStore(session_factory)
|
||||||
+2
@@ -23,6 +23,7 @@ from deerflow.persistence.agents.model import (
|
|||||||
)
|
)
|
||||||
from deerflow.persistence.admin_stats.model import AdminLeaderboardDailyStatRow
|
from deerflow.persistence.admin_stats.model import AdminLeaderboardDailyStatRow
|
||||||
from deerflow.persistence.business_mapping.model import BusinessMappingRow
|
from deerflow.persistence.business_mapping.model import BusinessMappingRow
|
||||||
|
from deerflow.persistence.concurrency.model import ConcurrencySampleRow
|
||||||
from deerflow.persistence.dashboard_sessions.model import DashboardSessionRow
|
from deerflow.persistence.dashboard_sessions.model import DashboardSessionRow
|
||||||
from deerflow.persistence.memory_config.model import UserMemoryConfigRow
|
from deerflow.persistence.memory_config.model import UserMemoryConfigRow
|
||||||
from deerflow.persistence.feedback.model import FeedbackRow
|
from deerflow.persistence.feedback.model import FeedbackRow
|
||||||
@@ -85,6 +86,7 @@ __all__ = [
|
|||||||
"AgentSkillRow",
|
"AgentSkillRow",
|
||||||
"AdminLeaderboardDailyStatRow",
|
"AdminLeaderboardDailyStatRow",
|
||||||
"BusinessMappingRow",
|
"BusinessMappingRow",
|
||||||
|
"ConcurrencySampleRow",
|
||||||
"DashboardSessionRow",
|
"DashboardSessionRow",
|
||||||
"KnowledgeEmbeddingRow",
|
"KnowledgeEmbeddingRow",
|
||||||
"KnowledgeEntityRow",
|
"KnowledgeEntityRow",
|
||||||
|
|||||||
@@ -279,6 +279,20 @@ class RunManager:
|
|||||||
async with self._lock:
|
async with self._lock:
|
||||||
return any(r.thread_id == thread_id and r.status in (RunStatus.pending, RunStatus.running) for r in self._runs.values())
|
return any(r.thread_id == thread_id and r.status in (RunStatus.pending, RunStatus.running) for r in self._runs.values())
|
||||||
|
|
||||||
|
async def list_active(self) -> list[RunRecord]:
|
||||||
|
"""Return every in-flight run (status pending/running), oldest first.
|
||||||
|
|
||||||
|
Powers the admin concurrency monitor: each record represents one live
|
||||||
|
call into the model runtime (interactive chat, agent chat, roundtable
|
||||||
|
seat, …), so ``len(list_active())`` is the current concurrent-call count
|
||||||
|
and each record can be cancelled to relieve model pressure. Completed
|
||||||
|
records linger in ``_runs`` until :meth:`cleanup`, so we filter by status
|
||||||
|
rather than returning the whole registry.
|
||||||
|
"""
|
||||||
|
async with self._lock:
|
||||||
|
# Dict insertion order == creation order, so this is oldest-first.
|
||||||
|
return [r for r in self._runs.values() if r.status in (RunStatus.pending, RunStatus.running)]
|
||||||
|
|
||||||
async def cleanup(self, run_id: str, *, delay: float = 300) -> None:
|
async def cleanup(self, run_id: str, *, delay: float = 300) -> None:
|
||||||
"""Remove a run record after an optional delay."""
|
"""Remove a run record after an optional delay."""
|
||||||
if delay > 0:
|
if delay > 0:
|
||||||
|
|||||||
Reference in new issue
Block a user