Merge branch 'ji' into master
合并 ji 的功能：开放问答接口（/api/open/chat[/stream] 回传生成文件 + 临时不留存）、 技能地址管理、智能体详情「最近对话」等。 冲突 frontend StrategyRoutes.tsx 已解决：同时保留 admin/roundtable-diagnostics(master) 与 admin/skill-addresses(ji) 两个管理员路由。 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
吉鑫升andClaude Opus 4.8 committed
commit
fbec222eb4
38 files changed
+3365
-94
No files matched your search
@@ -243,6 +243,7 @@ FastAPI application on port 8001 with health check at `GET /health`. Set `GATEWA
|
||||
| **Thread Runs** (`/api/threads/{id}/runs`) | `POST /` - create background run; `POST /stream` - create + SSE stream; `POST /wait` - create + block; `GET /` - list runs; `GET /{rid}` - run details; `POST /{rid}/cancel` - cancel; `GET /{rid}/join` - join SSE; `GET /{rid}/messages` - paginated messages `{data, has_more}`; `GET /{rid}/events` - full event stream; `GET /../messages` - thread messages with feedback; `GET /../token-usage` - aggregate tokens |
|
||||
| **Feedback** (`/api/threads/{id}/runs/{rid}/feedback`) | `PUT /` - upsert feedback; `DELETE /` - delete user feedback; `POST /` - create feedback; `GET /` - list feedback; `GET /stats` - aggregate stats; `DELETE /{fid}` - delete specific |
|
||||
| **Runs** (`/api/runs`) | `POST /stream` - stateless run + SSE; `POST /wait` - stateless run + block; `GET /{rid}/messages` - paginated messages by run_id `{data, has_more}` (cursor: `after_seq`/`before_seq`); `GET /{rid}/feedback` - list feedback by run_id |
|
||||
| **Open Chat** (`/api/open/chat`) | **Unauthenticated** open Q&A entry for external systems (the `/api/open/` prefix is whitelisted in `auth_middleware._PUBLIC_PATH_PREFIXES` + CSRF-exempt in `csrf_middleware.should_check_csrf`). `POST /chat` `{message \| messages[], model_name?, agent_name?, agent_id?, thinking_enabled=false, strip_think=true, thread_id?, recursion_limit=100}` → resolves the agent by **Chinese display name** (`agent_name`, matched against each `.deer-flow/agents/*/config.yaml` `name` via `_resolve_agent_id`; explicit `agent_id` wins if given), builds the lead agent via `make_lead_agent` (so the chosen **agent's** SOUL/tools/skills all apply), `ainvoke`s it once (stateless — no checkpointer, no thread persistence; runs under the `"default"` user bucket), and returns **only the final** `{answer, model_name, agent_name, agent_id, thinking_enabled, thread_id}` (no intermediate tool-call/process). **`POST /chat/stream`** is the **SSE streaming variant** (same request body) for **long tasks** — the blocking `/chat` produces no downstream bytes for the whole run, so a front nginx `proxy_read_timeout` (default 60s) cuts it off with a **504**; the stream keeps the connection warm (a `: keepalive` SSE comment every `_HEARTBEAT_SECONDS=15`s via a background producer-task + queue, plus `event: delta` AI-text increments and `event: progress`), and ends with **`event: result`** carrying the same final `{answer, …}` JSON (or `event: error`). Clients only need to read `event: result`; intermediate events are ignorable. Both endpoints share `_build_run_context` (config/agent/thread). `GET /agents` lists `{name, id}` so callers know what `agent_name` to send. **Thinking off is reliable**: when `thinking_enabled=false` the route also sets `thinking_force_disabled=true` so the lead agent calls `create_chat_model(force_disable_thinking=True)`, which injects **three** OpenAI-compatible "thinking off" shapes into `extra_body` — top-level `enable_thinking=false` (**SiliconFlow / DashScope hybrid models** like `deepseek-ai/DeepSeek-V4-*` on `api.siliconflow.cn` — the knob the nested `chat_template_kwargs` shape does NOT reach), `chat_template_kwargs.enable_thinking=false` (vLLM/Qwen native), and `thinking.type="disabled"` (Zhipu GLM) — even for models that only declare `supports_thinking: true`. For the **web-UI path** (which sends `thinking_enabled=false` but NOT `force_disable`), SiliconFlow models need `when_thinking_disabled: {extra_body: {enable_thinking: false}}` in their `config.yaml` entry (added to `deepseek-ai/DeepSeek-V4-Flash`) for thinking-off to take effect. Sets `is_scheduled_run=true` internally to skip the auto-title LLM call. **Generated-file return (`include_files=true`, default on)**: an agent that produces its report as a file (writes to `/mnt/user-data/outputs/` + `present_files`) returns only a short summary in `answer`; to recover the real deliverable both endpoints now also return `files: [{name, virtual_path, encoding(text|base64), content}]` — `_collect_output_files` scans the run's `state['thread_data']['outputs_path']` (the open thread is stateless so artifacts can't be fetched by thread, but the endpoint runs server-side and the sandbox already wrote the files to disk), text decoded as utf-8 else base64, capped by `_MAX_OUTPUT_FILES=20` / `_MAX_OUTPUT_FILE_BYTES=5MB`. `/chat` puts `files` on the response; `/chat/stream` puts it in the `event: result` payload. A run with no answer **and** no files now 502s ("…文本答案或文件"). Router `app/gateway/routers/open_chat.py`; standalone caller `test_open_chat_api.py` (repo root, streaming by default); tests `tests/test_open_chat.py`. |
|
||||
| **Thread Shares** (`/api/thread-shares`) | Per-conversation sharing with two modes (`thread_shares.mode`). **`import`**: `POST /` `{thread_id, mode}` - create (or reuse) a revocable share code; `GET /{code}/preview` + `POST /{code}/import` - copy that conversation into the caller's account (fresh `thread_id` via `app/gateway/thread_copy.py` + new `threads_meta` row + `user-data` copy; dedup on `metadata.shared_origin`). **`view`**: a public, read-only static page ? see Public Shares below. Both modes: `GET /` - list my codes; `DELETE /{code}` - revoke. |
|
||||
| **Public Shares** (`/api/public/shares`) | **Unauthenticated** (the `/api/public/` prefix is whitelisted in `auth_middleware._PUBLIC_PATH_PREFIXES`). `GET /{code}` - read-only conversation snapshot (title + serialized messages + artifacts) for a `view`-mode share; messages are read straight from the latest checkpoint. `GET /{code}/files/{path}` - serves a file referenced by that conversation (images, Markdown, text, PDF, ?) so the shared page's links are openable; path resolution is confined to the owner's thread directory. Active content (`text/html`, `application/xhtml+xml`, `image/svg+xml`) is forced to download (`_public_share_disposition`); everything else is served inline. Missing/revoked/non-`view` codes all return 404. |
|
||||
| **Roundtable Drafts** (`/api/roundtable-drafts`) | **Strictly per-user** persistence for the roundtable-planning page (replaces the old browser `localStorage`). `GET /` - list drafts (lightweight meta — incl. `chain_title`, the business-chain name used this session, cheaply regex-extracted from the in-memory `step2` blob in `_draft_to_meta`/`_extract_chain_title`; null = no chain / 自由模式; surfaced on the 第一步「分析记录」history card + sidebar list instead of the old「第 X 步」); `POST /` - create; `GET/PUT/DELETE /{id}` - full read / partial update / delete (all owner-scoped). A draft = one full session (Step 1 intent + Step 2 roundtable), step snapshots stored as JSON in `PortableLongText`. Recommendation history bound to a draft: `GET /{id}/recommendations` - list newest-first; `POST /{id}/recommendations` - append one run (objective/status/rationale/picks/candidates). taskId 深链会商记录 are **not** here — they live in the separate `/api/roundtable-task-drafts` store (see below) so personal history never mixes with task records. Backed by `deerflow.persistence.roundtable_drafts` (tables `roundtable_drafts`, `roundtable_recommend_history`); store wired in `deps.py` as `app.state.roundtable_draft_store`. Tests: `tests/test_roundtable_task_drafts.py`. |
|
||||
@@ -393,7 +394,10 @@ query-time type, `recommended_question`.
|
||||
Sync only ever adds/removes `origin='position'` rows, so a user's own
|
||||
favorites/subscriptions are never clobbered. Default sets (no binding for a
|
||||
type): agents = built-ins (`user_id IS NULL`); skills = empty (square stays
|
||||
full, nothing added to 我的); scheduled tasks = empty; questions = unrestricted.
|
||||
full, nothing added to 我的); scheduled tasks = empty; questions = **untagged
|
||||
only** (默认岗位 / 无标签绑定 只看到未打标的题目；打标的题目只在绑定了对应标签的
|
||||
岗位展示 — the router interprets `resolve_visible → None` as "show only untagged",
|
||||
not "show everything").
|
||||
`origin`/`position_id` columns were added to `agent_favorites` and
|
||||
`scheduled_task_subscriptions` — see migration `20260616_01` +
|
||||
`scripts/sql/column_additions_mysql.sql`. The running app also auto-adds these
|
||||
@@ -415,8 +419,12 @@ unassigned users inherit, else `null`); powers the top-bar 岗位 label
|
||||
changes: `GET /api/recommended-questions` is now
|
||||
filtered by the caller's position (`?all=1` bypasses for admin management) and
|
||||
admin can tag questions via `/api/tags/assignments/recommended_question/{id}`.
|
||||
Each listed question now carries its assigned `tags` (bulk-attached via
|
||||
`tag_store.list_assignments`), and the list accepts a repeatable `tag_ids`
|
||||
**Default-position rule**: tags are bulk-attached to **all** questions first,
|
||||
then the visibility filter applies — a position binding recommended_question 标签
|
||||
sees only those tagged questions; a 默认岗位 / 无绑定 (`resolve_visible → None`)
|
||||
sees only the **untagged** questions (打标的题目不再泄漏到默认岗位). `?all=1`
|
||||
(admin management) skips this filter and returns every question with its tags.
|
||||
The list also accepts a repeatable `tag_ids`
|
||||
OR-filter — so the 推荐问题配置 page can display per-question 标签 and search by
|
||||
them (frontend `RecommendedQuestionsPage` + `TagSearchBar`). Tests:
|
||||
`tests/test_recommended_questions_tags.py`.
|
||||
@@ -520,6 +528,61 @@ Operators upgrading an existing DB run the whole file (`USE <db>;` then execute)
|
||||
- **Installation**: `POST /api/skills/install` extracts .skill ZIP archive to custom/ directory
|
||||
- **Chinese name (`name_zh`)**: a Chinese display name stored in the **`skills` DB table** (`SkillRow.name_zh`, `String(128)`, migration `20260609_01`) ? never written to the on-disk SKILL.md, so it works for built-in skills too. Threaded through `SkillStore` create/ensure_legacy/update/update_admin exactly like `detail`; surfaced on `SkillResponse.name_zh` from the ownership record in `_skill_to_response`. Authorization mirrors `detail`: custom skill ? owner or admin; built-in/public ? admin only (row materialized on demand via `ensure_legacy`). Endpoints (in `app/gateway/routers/skills.py`): `PUT /api/skills/{name}/name-zh` (set/clear), `POST /api/skills/{name}/name-zh/generate` (LLM generates one), `POST /api/skills/name-zh/generate-missing` (????: **admin-only** bulk-generate for every skill that lacks one, bounded-concurrency `asyncio.gather`). **Optional model selection**: both generate endpoints accept an optional JSON body `{model_name}` ? an explicit name wins, else the curator model setting, else the main model (`_resolve_skill_name_zh_model_name`). **Model-error surfacing**: `_generate_skill_name_zh` raises `SkillNameZhModelError` when the model itself fails (build/timeout/API error) ? distinct from the model returning an unusable name (? `None`). The single endpoint maps that to `502 ?????,?????????:?`; the bulk endpoint fails fast on a model-build error (502) and otherwise carries a representative `model_error` string in `SkillNameZhBulkResponse` so the UI can say "N ?????????". The frontend (`skill-settings-page.tsx` list + `skill-detail-dialog.tsx` editor, plus the agent skill pickers/badges in `NewAgentPage.tsx` + `agent-card.tsx`) shows `name_zh || name`, offers per-skill "AI ??" plus an **admin-only** header "???????" button, and a compact model dropdown (`components/workspace/model-name-select.tsx`, default = configured model) beside each action.
|
||||
|
||||
#### Skill Address Management (技能地址管理 — URL/IP 批量替换)
|
||||
|
||||
Admin-only maintenance tool to find every `http(s)://` URL and IPv4 (`ip[:port]`)
|
||||
address in the **running** skills (`skills/{public,custom}`, all `.md` + `.py`,
|
||||
recursing into `scripts/`/`references/`/`templates/`) and rewrite them in bulk —
|
||||
e.g. swapping a UAT endpoint for production across all skills at once. The same
|
||||
address commonly lives in both a skill's `SKILL.md` and its `scripts/*.py`, so
|
||||
results are **aggregated by unique address** with every occurrence (`file:line` +
|
||||
snippet) and replaced together.
|
||||
|
||||
- **Pure engine** — `deerflow/skills/address_scan.py` (harness, no `app.*`):
|
||||
`find_spans` / `scan_occurrences` (URL regex with trailing-punctuation strip;
|
||||
IPv4 with 0–255 octet check + optional port; an IP inside a URL is *not*
|
||||
separately surfaced), `apply_replacements` (**substring-safe**: re-detects exact
|
||||
token spans and only rewrites a span whose text equals a requested `from`, so
|
||||
`1.2.3.4` is never rewritten inside `11.2.3.40`), `infer_kind`,
|
||||
`is_valid_replacement`. Tested in `tests/test_skill_addresses.py`.
|
||||
- **Router** — `app/gateway/routers/skill_addresses.py` (`/api/skill-addresses`,
|
||||
**admin only**). **Availability at scale (150+ skills):** every filesystem
|
||||
walk/read/write runs via `asyncio.to_thread` so the event loop is never blocked.
|
||||
Routes: `GET /scan` (aggregated, one-shot) + `GET /scan/stream` (per-skill NDJSON
|
||||
progress `{type:start|progress|result}`); `POST /preview` (dry-run diff +
|
||||
validation warnings); `POST /apply` + `POST /apply/stream` (per-file NDJSON
|
||||
progress); `GET /backups`; `POST /rollback/{backup_id}`. **Apply safety:**
|
||||
optimistic lock via `expected_total` (409 if files changed since the scan),
|
||||
path-guard (`is_relative_to` the skills root), auto disk backup to
|
||||
`{skills_root}/.address-edits/{timestamp}/` (the `.`-prefixed dir is excluded
|
||||
from scans), then `refresh_skills_system_prompt_cache_async()` so edits take
|
||||
effect immediately. Registered in `app.py` after `sensitive_words.router`.
|
||||
**Per-edit history:** each apply also writes a sibling manifest
|
||||
`{skills_root}/.address-edits/{timestamp}.manifest.json` (best-effort) capturing
|
||||
`{created_at, replaced_total, replacements:[{from,to,kind}], changed_files}`; the
|
||||
sibling-not-child placement keeps it out of both the restore set and the file
|
||||
count. `GET /backups` returns each record enriched with `replaced_total` +
|
||||
`replacements` (from the manifest; falls back to dir mtime when absent), and
|
||||
`POST /rollback/{backup_id}` restores a single record — so the UI shows a full
|
||||
modification history and rolls back any one entry independently.
|
||||
- **Frontend** — the panel is the reusable `SkillAddressesPanel` (exported from
|
||||
`pages/SkillAddressesPage.tsx`; its default export keeps the standalone route
|
||||
`/page/strategy/admin/skill-addresses` as a direct-link fallback). It is
|
||||
**embedded as a「技能地址管理」tab inside the admin「技能管理」page**
|
||||
`pages/AdminSkillsPage.tsx` (Tabs: 技能列表 + 技能地址管理; reached from the
|
||||
「技能管理」item in the「设置和更多」dropdown `workspace-nav-menu.tsx` →
|
||||
`/page/workspace/admin/skills`) — it is no longer a separate entry in the
|
||||
`components/page-sidebar.tsx` settings dropdown. The panel:
|
||||
auto-scan on open with a **progress bar** (扫描逐技能 / 应用逐文件 X/N), URL/IP
|
||||
type filter, search, a「片段批量套用」helper (e.g. all `uat.4.cn` → `prod.4.cn`),
|
||||
per-address「替换为」input with expandable occurrences, inline 预览 panel, 应用,
|
||||
**multi-keyword search** (space-separated, all-must-hit), and a **修改记录**
|
||||
(modification history) panel — each apply is one record (time / which addresses
|
||||
changed from→to / file & replacement counts) with a per-record 回退此次
|
||||
(single-record rollback, with inline confirm). Client + NDJSON stream parsing in
|
||||
`strategy-components/api/skill-addresses.ts`. Scope is the running skills only;
|
||||
the `deploy/minimal/skills/` copy is not touched (sync separately).
|
||||
|
||||
#### Skill Compression (技能压缩)
|
||||
|
||||
When the skill catalogue grows large, injecting every enabled skill full-text
|
||||
@@ -615,6 +678,7 @@ scheduler runs once per deployment. Per run:
|
||||
|
||||
- `create_chat_model(name, thinking_enabled)` instantiates LLM from config via reflection
|
||||
- Supports `thinking_enabled` flag with per-model `when_thinking_enabled` overrides
|
||||
- `force_disable_thinking=True` (kwarg) hard-disables thinking even when the model config declares **no** `when_thinking_*`/`thinking` shape — many internal vLLM/Qwen models only set `supports_thinking: true`, so `thinking_enabled=False` alone injects nothing and they keep reasoning on. The flag injects both OpenAI-compatible disable shapes (`extra_body.chat_template_kwargs.enable_thinking=false` for vLLM/Qwen + `extra_body.thinking.type="disabled"` for Zhipu GLM), only for classes that accept `extra_body`. Mirrors `TitleMiddleware._force_disable_thinking`. The lead agent reads the runtime flag `thinking_force_disabled` and forwards it (used by `/api/open/chat`)
|
||||
- Supports vLLM-style thinking toggles via `when_thinking_enabled.extra_body.chat_template_kwargs.enable_thinking` for Qwen reasoning models, while normalizing legacy `thinking` configs for backward compatibility
|
||||
- Supports `supports_vision` flag for image understanding models
|
||||
- Config values starting with `$` resolved as environment variables
|
||||
|
||||
@@ -50,6 +50,7 @@ from app.gateway.routers import (
|
||||
models,
|
||||
multi_agent,
|
||||
notifications,
|
||||
open_chat,
|
||||
parallel_agent_tasks,
|
||||
parallel_agents,
|
||||
positions,
|
||||
@@ -69,6 +70,7 @@ from app.gateway.routers import (
|
||||
runs,
|
||||
scheduled_tasks,
|
||||
sensitive_words,
|
||||
skill_addresses,
|
||||
skills,
|
||||
suggestions,
|
||||
system_settings,
|
||||
@@ -1359,6 +1361,9 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
|
||||
# Sensitive-word map (敏感词管理 / 脱敏词) at /api/sensitive-words
|
||||
app.include_router(sensitive_words.router)
|
||||
|
||||
# Skill URL/IP batch replace (技能地址管理) at /api/skill-addresses
|
||||
app.include_router(skill_addresses.router)
|
||||
|
||||
# 大屏绘制智能体独立页面会话（按 taskId 存取，不分权）at /api/dashboard-sessions
|
||||
app.include_router(dashboard_sessions.router)
|
||||
|
||||
@@ -1456,6 +1461,9 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
|
||||
# Stateless Runs API (stream/wait without a pre-existing thread)
|
||||
app.include_router(runs.router)
|
||||
|
||||
# 开放问答接口 /api/open/chat —— 无需登录,指定模型 + 智能体做问答,可关闭思考模式
|
||||
app.include_router(open_chat.router)
|
||||
|
||||
# --- ?????????? /page/workspace/roundtable/planning?---
|
||||
# ??? app/gateway/routers/{multi_agent,intent,recommend}.py?
|
||||
# ??? frontend-web/docs/multi-agent-backend-dev.md
|
||||
|
||||
@@ -35,6 +35,9 @@ _PUBLIC_PATH_PREFIXES: tuple[str, ...] = (
|
||||
"/redoc",
|
||||
"/openapi.json",
|
||||
"/api/public/",
|
||||
# 开放问答接口:无需登录即可指定模型 + 智能体做问答(见 routers/open_chat.py)。
|
||||
# 以 "default" 用户桶运行,只暴露受控的问答能力。
|
||||
"/api/open/",
|
||||
)
|
||||
|
||||
# Exact auth paths that are public (login/register/status check).
|
||||
|
||||
@@ -52,6 +52,10 @@ def should_check_csrf(request: Request) -> bool:
|
||||
# PUT/POST returns 403 even though auth was bypassed earlier in the chain.
|
||||
if path.startswith("/api/public/"):
|
||||
return False
|
||||
# 开放问答接口同样是匿名、无会话的(见 auth_middleware._PUBLIC_PATH_PREFIXES),
|
||||
# 双提交 Cookie 既无 token 可比、也无会话可被伪造,豁免之。
|
||||
if path.startswith("/api/open/"):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,432 @@
|
||||
"""开放问答接口(无需登录)。
|
||||
|
||||
给外部系统一个最简单的 HTTP 入口来「调用本项目的智能体做问答」:
|
||||
|
||||
- 可指定 **模型**(``model_name``,对应 ``config.yaml`` 中 ``models[]`` 的 name)。
|
||||
- 可指定 **智能体**(``agent_id``,对应 ``.deer-flow/agents/<id>/``;留空 = 默认 lead agent)。
|
||||
- 可关闭 **大模型思考模式**(``thinking_enabled=false``,默认即关闭)。内网 vLLM/Qwen
|
||||
模型仅靠 ``thinking_enabled=False`` 往往关不掉思考(其 config 只声明
|
||||
``supports_thinking: true``),本接口在关闭思考时会把
|
||||
``thinking_force_disabled`` 一并下发,经
|
||||
``create_chat_model(force_disable_thinking=True)`` 强制注入
|
||||
``extra_body.chat_template_kwargs.enable_thinking=false``(及 GLM 的
|
||||
``thinking.type=disabled``)——与「生成标题」时关闭思考的做法一致。
|
||||
|
||||
该路由前缀 ``/api/open/`` 已在 ``auth_middleware`` 白名单里,**绕过登录鉴权**,
|
||||
以 ``"default"`` 用户桶运行,只暴露受控的问答能力(无文件/线程持久化)。
|
||||
|
||||
请求示例::
|
||||
|
||||
POST /api/open/chat
|
||||
{
|
||||
"message": "用一句话介绍一下你自己",
|
||||
"model_name": "deepseek-ai/DeepSeek-V4-Flash",
|
||||
"agent_id": null,
|
||||
"thinking_enabled": false
|
||||
}
|
||||
|
||||
返回::
|
||||
|
||||
{"answer": "...", "model_name": "...", "agent_id": null,
|
||||
"thinking_enabled": false, "thread_id": "open-...."}
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import shutil
|
||||
import uuid
|
||||
from collections.abc import AsyncIterator
|
||||
from pathlib import Path
|
||||
from typing import Any, Literal
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from fastapi.responses import StreamingResponse
|
||||
from langchain_core.messages import AIMessage, BaseMessage, HumanMessage, SystemMessage
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from deerflow.agents.lead_agent.agent import make_lead_agent
|
||||
from deerflow.config.agents_config import list_custom_agents, validate_agent_id
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/open", tags=["open"])
|
||||
|
||||
_THINK_TAG_RE = re.compile(r"<think>[\s\S]*?</think>", re.IGNORECASE)
|
||||
|
||||
|
||||
class OpenChatMessage(BaseModel):
|
||||
"""多轮历史中的一条消息。"""
|
||||
|
||||
role: Literal["user", "assistant", "system"] = "user"
|
||||
content: str
|
||||
|
||||
|
||||
class OpenChatRequest(BaseModel):
|
||||
"""开放问答请求体。
|
||||
|
||||
``message`` 与 ``messages`` 二选一:单轮直接传 ``message``;需要带上下文时传
|
||||
``messages``(完整历史,本接口本身无状态、不落库)。
|
||||
"""
|
||||
|
||||
message: str | None = Field(default=None, description="单轮问题文本;与 messages 二选一")
|
||||
messages: list[OpenChatMessage] | None = Field(default=None, description="多轮历史;与 message 二选一")
|
||||
model_name: str | None = Field(default=None, description="models[] 中的模型名;留空用默认模型")
|
||||
agent_name: str | None = Field(default=None, description="智能体名称(中文显示名);留空用默认 lead agent")
|
||||
agent_id: str | None = Field(default=None, description="智能体 id;一般不用,优先级高于 agent_name")
|
||||
thinking_enabled: bool = Field(default=False, description="是否开启思考;默认 False = 关闭思考(内网模型也会被强制关闭)")
|
||||
strip_think: bool = Field(default=True, description="关闭思考时,从答案里剥离残留的 <think>…</think> 块")
|
||||
thread_id: str | None = Field(default=None, description="可选;仅用于文件隔离,留空自动生成")
|
||||
recursion_limit: int = Field(default=100, ge=1, le=300, description="智能体单次问答的最大步数")
|
||||
include_files: bool = Field(default=True, description="是否随结果回传智能体在 outputs 目录生成的文件（名称+内容）")
|
||||
|
||||
|
||||
class OpenChatFile(BaseModel):
|
||||
"""智能体本次运行在 /mnt/user-data/outputs 下生成的一个文件。"""
|
||||
|
||||
name: str # outputs/ 下的相对路径，如 "报告.md"
|
||||
virtual_path: str # 沙箱虚拟路径，如 "/mnt/user-data/outputs/报告.md"
|
||||
encoding: Literal["text", "base64"] = "text" # text=可直接用的文本；base64=二进制需解码
|
||||
content: str
|
||||
|
||||
|
||||
class OpenChatResponse(BaseModel):
|
||||
answer: str
|
||||
model_name: str | None
|
||||
agent_name: str | None
|
||||
agent_id: str | None
|
||||
thinking_enabled: bool
|
||||
thread_id: str
|
||||
files: list[OpenChatFile] = Field(default_factory=list, description="本次生成的输出文件（含内容）")
|
||||
|
||||
|
||||
def _resolve_agent_id(agent_id: str | None, agent_name: str | None) -> str | None:
|
||||
"""把 agent_id / 中文名称 解析成运行 id。
|
||||
|
||||
优先级:显式 agent_id > agent_name(按 .deer-flow/agents/*/config.yaml 的 name 匹配)。
|
||||
都没传 = None(默认 lead agent)。名称匹配先精确、再忽略大小写/首尾空格。
|
||||
"""
|
||||
if agent_id and agent_id.strip():
|
||||
try:
|
||||
return validate_agent_id(agent_id)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=422, detail=f"非法的 agent_id:{exc}") from exc
|
||||
|
||||
if not agent_name or not agent_name.strip():
|
||||
return None
|
||||
|
||||
target = agent_name.strip()
|
||||
agents = list_custom_agents()
|
||||
for a in agents:
|
||||
if (a.name or "").strip() == target:
|
||||
return a.id
|
||||
lowered = target.lower()
|
||||
for a in agents:
|
||||
if (a.name or "").strip().lower() == lowered:
|
||||
return a.id
|
||||
raise HTTPException(status_code=404, detail=f"未找到名称为「{agent_name}」的智能体。")
|
||||
|
||||
|
||||
def _build_lc_messages(body: OpenChatRequest) -> list[BaseMessage]:
|
||||
"""把请求体转成 LangChain 消息列表。"""
|
||||
if body.messages:
|
||||
out: list[BaseMessage] = []
|
||||
for m in body.messages:
|
||||
text = (m.content or "").strip()
|
||||
if not text:
|
||||
continue
|
||||
if m.role == "assistant":
|
||||
out.append(AIMessage(content=m.content))
|
||||
elif m.role == "system":
|
||||
out.append(SystemMessage(content=m.content))
|
||||
else:
|
||||
out.append(HumanMessage(content=m.content))
|
||||
if out:
|
||||
return out
|
||||
if body.message and body.message.strip():
|
||||
return [HumanMessage(content=body.message)]
|
||||
raise HTTPException(status_code=422, detail="必须提供非空的 'message' 或 'messages'。")
|
||||
|
||||
|
||||
def _extract_text(content: Any) -> str:
|
||||
"""从 AIMessage.content(str 或 block 列表)抽取纯文本。"""
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
if isinstance(content, list):
|
||||
parts: list[str] = []
|
||||
for block in content:
|
||||
if isinstance(block, str):
|
||||
parts.append(block)
|
||||
elif isinstance(block, dict):
|
||||
text_value = block.get("text")
|
||||
if isinstance(text_value, str):
|
||||
parts.append(text_value)
|
||||
return "".join(parts) if parts else ""
|
||||
return ""
|
||||
|
||||
|
||||
def _extract_answer(result: Any, *, strip_think: bool) -> str:
|
||||
"""从 ainvoke 的最终状态里取最后一条 AI 文本。"""
|
||||
messages = (result or {}).get("messages") if isinstance(result, dict) else None
|
||||
answer = ""
|
||||
if isinstance(messages, list):
|
||||
for msg in reversed(messages):
|
||||
if isinstance(msg, AIMessage):
|
||||
text = _extract_text(msg.content).strip()
|
||||
if text:
|
||||
answer = text
|
||||
break
|
||||
if strip_think and answer:
|
||||
answer = _THINK_TAG_RE.sub("", answer).strip()
|
||||
return answer
|
||||
|
||||
|
||||
_OUTPUTS_VIRTUAL_PREFIX = "/mnt/user-data/outputs"
|
||||
_MAX_OUTPUT_FILES = 20
|
||||
_MAX_OUTPUT_FILE_BYTES = 5 * 1024 * 1024 # 单文件 5MB 上限,避免把大二进制塞进 JSON 响应
|
||||
|
||||
|
||||
def _collect_output_files(state: Any, *, include: bool) -> list[OpenChatFile]:
|
||||
"""读取本次运行在沙箱 outputs 目录生成的文件,随结果一并回传。
|
||||
|
||||
开放问答是无状态的(不持久化线程),客户端没法用 artifacts 路由按线程取件;但本端点
|
||||
就跑在服务端进程里,沙箱把生成的文件落在 ``state['thread_data']['outputs_path']``,
|
||||
所以这里直接扫描该目录,把文件内容带回(文本按 utf-8,二进制按 base64)。
|
||||
受 ``_MAX_OUTPUT_FILES`` / ``_MAX_OUTPUT_FILE_BYTES`` 限制。
|
||||
"""
|
||||
if not include or not isinstance(state, dict):
|
||||
return []
|
||||
thread_data = state.get("thread_data") or {}
|
||||
outputs_path = thread_data.get("outputs_path")
|
||||
if not outputs_path:
|
||||
return []
|
||||
base = Path(outputs_path)
|
||||
if not base.is_dir():
|
||||
return []
|
||||
try:
|
||||
entries = sorted((p for p in base.rglob("*") if p.is_file()), key=lambda p: p.as_posix())
|
||||
except OSError:
|
||||
return []
|
||||
|
||||
files: list[OpenChatFile] = []
|
||||
for entry in entries:
|
||||
if len(files) >= _MAX_OUTPUT_FILES:
|
||||
break
|
||||
try:
|
||||
if entry.stat().st_size > _MAX_OUTPUT_FILE_BYTES:
|
||||
continue
|
||||
data = entry.read_bytes()
|
||||
except OSError:
|
||||
continue
|
||||
rel = entry.relative_to(base).as_posix()
|
||||
try:
|
||||
content = data.decode("utf-8")
|
||||
encoding: Literal["text", "base64"] = "text"
|
||||
except UnicodeDecodeError:
|
||||
content = base64.b64encode(data).decode("ascii")
|
||||
encoding = "base64"
|
||||
files.append(OpenChatFile(name=rel, virtual_path=f"{_OUTPUTS_VIRTUAL_PREFIX}/{rel}", encoding=encoding, content=content))
|
||||
return files
|
||||
|
||||
|
||||
def _cleanup_ephemeral_thread(state: Any, thread_id: str | None) -> None:
|
||||
"""删除本次开放问答的临时线程目录(open-*),落实"聊天记录不存储"。
|
||||
|
||||
生成文件已在 _collect_output_files 读进内存(随响应回传),这里把磁盘上整个临时线程
|
||||
目录清掉,避免无状态调用在服务端堆积残留。仅对 ``open-*`` 线程生效,best-effort。
|
||||
"""
|
||||
if not thread_id or not str(thread_id).startswith("open-"):
|
||||
return
|
||||
thread_data = state.get("thread_data") if isinstance(state, dict) else None
|
||||
outputs_path = (thread_data or {}).get("outputs_path")
|
||||
if not outputs_path:
|
||||
return
|
||||
try:
|
||||
p = Path(outputs_path).resolve()
|
||||
# outputs_path 形如 .../threads/<thread_id>/user-data/outputs;只删到 <thread_id> 这层
|
||||
thread_root = next((parent for parent in p.parents if parent.name == thread_id), None)
|
||||
if thread_root is not None and thread_root.exists():
|
||||
shutil.rmtree(thread_root, ignore_errors=True)
|
||||
except Exception as exc: # noqa: BLE001 — 清理失败不影响返回结果
|
||||
logger.warning("清理临时线程目录失败 (%s): %s", thread_id, exc)
|
||||
|
||||
|
||||
# 流式响应每隔这么久没有真实事件就发一个 SSE 注释心跳,持续制造下行字节,
|
||||
# 让前置反向代理(nginx proxy_read_timeout 默认 60s 等)不会把长任务判定成空闲超时。
|
||||
_HEARTBEAT_SECONDS = 15.0
|
||||
|
||||
|
||||
def _build_run_context(body: OpenChatRequest) -> tuple[dict[str, Any], str | None, str]:
|
||||
"""把请求体翻译成 (config, agent_id, thread_id)。两个端点共用,确保关思考等行为一致。"""
|
||||
agent_id = _resolve_agent_id(body.agent_id, body.agent_name)
|
||||
thread_id = body.thread_id or f"open-{uuid.uuid4()}"
|
||||
configurable: dict[str, Any] = {
|
||||
"thread_id": thread_id,
|
||||
"model_name": body.model_name,
|
||||
"thinking_enabled": body.thinking_enabled,
|
||||
# 关闭思考时强制下发,确保内网模型也真的关掉(见模块 docstring)。
|
||||
"thinking_force_disabled": not body.thinking_enabled,
|
||||
# 一次性问答:跳过自动起标题(省一次 LLM 调用)、不挂定时任务工具。
|
||||
"is_scheduled_run": True,
|
||||
# 临时/无状态运行:跳过记忆与知识库的读写(不召回、不留存、不自动沉淀),
|
||||
# 做到"通过本接口的聊天记录不存储"(见 _build_middlewares 的 is_ephemeral)。
|
||||
"ephemeral": True,
|
||||
"memory_injection_enabled": False,
|
||||
}
|
||||
if agent_id:
|
||||
configurable["agent_id"] = agent_id
|
||||
config: dict[str, Any] = {"configurable": configurable, "recursion_limit": body.recursion_limit}
|
||||
return config, agent_id, thread_id
|
||||
|
||||
|
||||
def _sse(event: str | None, data: Any) -> str:
|
||||
"""序列化一条 SSE 消息。event=None 时只发 data 行。"""
|
||||
payload = json.dumps(data, ensure_ascii=False)
|
||||
head = f"event: {event}\n" if event else ""
|
||||
return f"{head}data: {payload}\n\n"
|
||||
|
||||
|
||||
@router.post("/chat", response_model=OpenChatResponse)
|
||||
async def open_chat(body: OpenChatRequest) -> OpenChatResponse:
|
||||
"""指定模型 + 智能体做一次问答(无状态、可关闭思考)。
|
||||
|
||||
同步阻塞版:整段调研跑完才返回。**长任务(联网调研)请改用 /chat/stream**——
|
||||
本端点期间不产生任何下行字节,容易被前置反代的 proxy_read_timeout 掐成 504。
|
||||
"""
|
||||
lc_messages = _build_lc_messages(body)
|
||||
config, agent_id, thread_id = _build_run_context(body)
|
||||
|
||||
try:
|
||||
agent = make_lead_agent(config)
|
||||
result = await agent.ainvoke({"messages": lc_messages}, config=config)
|
||||
except Exception as exc: # noqa: BLE001 — 开放接口需把内部错误归一成 502
|
||||
logger.exception("open_chat 调用智能体失败 (model=%s, agent=%s)", body.model_name, agent_id)
|
||||
raise HTTPException(status_code=502, detail=f"智能体调用失败:{exc}") from exc
|
||||
|
||||
answer = _extract_answer(result, strip_think=body.strip_think and not body.thinking_enabled)
|
||||
files = _collect_output_files(result, include=body.include_files)
|
||||
_cleanup_ephemeral_thread(result, thread_id) # 取完文件即清理临时目录,不在服务端留存
|
||||
if not answer and not files:
|
||||
raise HTTPException(status_code=502, detail="智能体未返回任何文本答案或文件。")
|
||||
|
||||
return OpenChatResponse(
|
||||
answer=answer,
|
||||
model_name=body.model_name,
|
||||
agent_name=body.agent_name,
|
||||
agent_id=agent_id,
|
||||
thinking_enabled=body.thinking_enabled,
|
||||
thread_id=thread_id,
|
||||
files=files,
|
||||
)
|
||||
|
||||
|
||||
async def _open_chat_event_stream(body: OpenChatRequest) -> AsyncIterator[str]:
|
||||
"""驱动智能体并产出 SSE 事件流。
|
||||
|
||||
- 边跑边转发 AI 文本增量(``event: delta``,仅供实时观看,客户端可忽略);
|
||||
- 没有真实事件时每 ``_HEARTBEAT_SECONDS`` 发一个心跳注释行保活;
|
||||
- 全部跑完后用 ``event: result`` 给出最终(已按需剥离 <think> 的)答案;
|
||||
- 出错则 ``event: error``。
|
||||
|
||||
用「后台生产任务 + 队列 + 超时取队列」的方式把「智能体进度」与「定时心跳」解耦,
|
||||
所以即便某次工具调用静默很久,心跳仍照常下行,连接不会被反代判空闲。
|
||||
"""
|
||||
lc_messages = _build_lc_messages(body)
|
||||
config, agent_id, thread_id = _build_run_context(body)
|
||||
strip_think = body.strip_think and not body.thinking_enabled
|
||||
|
||||
queue: asyncio.Queue[tuple[str, Any]] = asyncio.Queue()
|
||||
final_state: dict[str, Any] = {}
|
||||
|
||||
async def produce() -> None:
|
||||
nonlocal final_state
|
||||
try:
|
||||
agent = make_lead_agent(config)
|
||||
async for mode, chunk in agent.astream({"messages": lc_messages}, config=config, stream_mode=["values", "messages"]):
|
||||
if mode == "values" and isinstance(chunk, dict):
|
||||
final_state = chunk
|
||||
await queue.put(("progress", {"messages": len(chunk.get("messages") or [])}))
|
||||
elif mode == "messages":
|
||||
msg = chunk[0] if isinstance(chunk, tuple) and chunk else chunk
|
||||
# 仅转发 AI 正文增量;DeepSeek 等把思考放在 additional_kwargs.reasoning_content,
|
||||
# 不在 content 里,所以这里天然不会泄漏 <think>。
|
||||
if isinstance(msg, AIMessage):
|
||||
text = _extract_text(getattr(msg, "content", "")).strip()
|
||||
if text:
|
||||
await queue.put(("delta", {"text": text}))
|
||||
except Exception as exc: # noqa: BLE001 — 开放接口需把内部错误归一成 error 事件
|
||||
logger.exception("open_chat_stream 调用智能体失败 (model=%s, agent=%s)", body.model_name, agent_id)
|
||||
await queue.put(("error", {"detail": f"智能体调用失败:{exc}"}))
|
||||
finally:
|
||||
await queue.put(("__end__", None))
|
||||
|
||||
task = asyncio.create_task(produce())
|
||||
try:
|
||||
while True:
|
||||
try:
|
||||
event, data = await asyncio.wait_for(queue.get(), timeout=_HEARTBEAT_SECONDS)
|
||||
except asyncio.TimeoutError:
|
||||
yield ": keepalive\n\n" # SSE 注释行,客户端忽略,仅用于保活
|
||||
continue
|
||||
if event == "__end__":
|
||||
break
|
||||
if event == "error":
|
||||
yield _sse("error", data)
|
||||
return
|
||||
yield _sse(event, data)
|
||||
|
||||
answer = _extract_answer(final_state, strip_think=strip_think)
|
||||
files = _collect_output_files(final_state, include=body.include_files)
|
||||
_cleanup_ephemeral_thread(final_state, thread_id) # 取完文件即清理临时目录,不在服务端留存
|
||||
if not answer and not files:
|
||||
yield _sse("error", {"detail": "智能体未返回任何文本答案或文件。"})
|
||||
return
|
||||
yield _sse(
|
||||
"result",
|
||||
{
|
||||
"answer": answer,
|
||||
"model_name": body.model_name,
|
||||
"agent_name": body.agent_name,
|
||||
"agent_id": agent_id,
|
||||
"thinking_enabled": body.thinking_enabled,
|
||||
"thread_id": thread_id,
|
||||
"files": [f.model_dump() for f in files],
|
||||
},
|
||||
)
|
||||
finally:
|
||||
if not task.done():
|
||||
task.cancel()
|
||||
|
||||
|
||||
@router.post("/chat/stream")
|
||||
async def open_chat_stream(body: OpenChatRequest) -> StreamingResponse:
|
||||
"""流式版 /chat:边跑边发心跳保活,跑完用 ``event: result`` 给最终答案。
|
||||
|
||||
解决两件事:(1) 长任务(联网调研、报告)同步阻塞会被前置反代 proxy_read_timeout
|
||||
掐成 504;流式持续有字节下行,代理不再判定空闲超时。(2) 客户端只需读到
|
||||
``event: result`` 解析 ``answer`` 即可,中间的 ``delta``/``progress``/心跳都可忽略。
|
||||
"""
|
||||
# 入参校验放在流外:非法请求直接抛 4xx,而不是开了流再报错。
|
||||
_build_lc_messages(body)
|
||||
return StreamingResponse(
|
||||
_open_chat_event_stream(body),
|
||||
media_type="text/event-stream",
|
||||
headers={
|
||||
"Cache-Control": "no-cache",
|
||||
"Connection": "keep-alive",
|
||||
# 关掉 nginx 对该响应的缓冲,确保 SSE/心跳实时下行(即便前置 nginx 默认开缓冲)。
|
||||
"X-Accel-Buffering": "no",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.get("/agents")
|
||||
async def open_list_agents() -> dict[str, Any]:
|
||||
"""列出可用智能体的名称(方便知道 agent_name 该填什么)。"""
|
||||
agents = list_custom_agents()
|
||||
items = [{"name": a.name, "id": a.id} for a in agents if (a.name or "").strip()]
|
||||
return {"agents": items, "count": len(items)}
|
||||
@@ -67,24 +67,11 @@ async def list_recommended_questions(
|
||||
) -> RecommendedQuestionsListResponse:
|
||||
store = _get_store(request)
|
||||
rows = await store.list_all()
|
||||
if not all:
|
||||
# Filter by the caller's 岗位 (position) visibility. ``None`` =
|
||||
# unrestricted (no skill/question tag binding → show all default
|
||||
# questions); a set = show only the questions carrying the bound tags.
|
||||
sync = getattr(request.app.state, "position_sync", None)
|
||||
if sync is not None:
|
||||
try:
|
||||
visible = await sync.resolve_visible(_current_user_id(request), "recommended_question")
|
||||
except Exception:
|
||||
logger.debug("recommended-question visibility resolve failed", exc_info=True)
|
||||
visible = None
|
||||
if visible is not None:
|
||||
rows = [r for r in rows if r["id"] in visible]
|
||||
responses = [RecommendedQuestionResponse(**r) for r in rows]
|
||||
|
||||
# Bulk-attach the tags assigned to each question so the management list can
|
||||
# display them and filter by 标签. Best-effort: a missing/failing tag store
|
||||
# never breaks the question list.
|
||||
# display them and the 岗位 filter below can tell tagged from untagged.
|
||||
# Best-effort: a missing/failing tag store never breaks the question list.
|
||||
tag_store = getattr(request.app.state, "tag_store", None)
|
||||
if tag_store is not None and responses:
|
||||
try:
|
||||
@@ -95,6 +82,24 @@ async def list_recommended_questions(
|
||||
for resp in responses:
|
||||
resp.tags = assignments.get(resp.id, [])
|
||||
|
||||
if not all:
|
||||
# Filter by the caller's 岗位 (position) visibility:
|
||||
# * ``set`` — the position binds recommended_question 标签 → show only
|
||||
# the questions carrying those bound tags.
|
||||
# * ``None`` — 默认岗位 / 无标签绑定 → show only **untagged** questions.
|
||||
# 打标的题目只在绑定了对应标签的岗位展示，不再泄漏到默认岗位。
|
||||
sync = getattr(request.app.state, "position_sync", None)
|
||||
if sync is not None:
|
||||
try:
|
||||
visible = await sync.resolve_visible(_current_user_id(request), "recommended_question")
|
||||
except Exception:
|
||||
logger.debug("recommended-question visibility resolve failed", exc_info=True)
|
||||
visible = None
|
||||
if visible is not None:
|
||||
responses = [r for r in responses if r.id in visible]
|
||||
else:
|
||||
responses = [r for r in responses if not r.tags]
|
||||
|
||||
if tag_ids:
|
||||
wanted = set(tag_ids)
|
||||
responses = [r for r in responses if any(t.get("id") in wanted for t in r.tags)]
|
||||
|
||||
@@ -0,0 +1,713 @@
|
||||
"""技能地址(URL/IP)批量替换 API (技能地址管理).
|
||||
|
||||
Scans the **running** skills directory (``skills/{public,custom}``) — the files
|
||||
the agent actually reads — for ``http(s)://`` URLs and IPv4 (``ip[:port]``)
|
||||
addresses, aggregates each unique address with every place it appears (across
|
||||
``.md`` and ``.py``, across files), and lets an admin rewrite them in bulk.
|
||||
|
||||
Why this shape: the same address routinely lives in both a skill's ``SKILL.md``
|
||||
(documented endpoint) and its ``scripts/*.py`` (the actual call), so the natural
|
||||
unit is *"one address → replace all of its occurrences at once"*. The pure
|
||||
scan/replace logic lives in ``deerflow.skills.address_scan`` (harness layer,
|
||||
substring-safe span replacement); this router only walks the tree, enforces
|
||||
admin auth + path safety, backs up before writing, and refreshes the skills
|
||||
system-prompt cache so edits take effect immediately.
|
||||
|
||||
**Scale / availability (150+ skills):** every filesystem walk / read / write is
|
||||
offloaded to a worker thread via ``asyncio.to_thread`` so the event loop is
|
||||
never blocked while scanning or rewriting — the Gateway stays responsive. The
|
||||
streaming endpoints additionally emit NDJSON progress (``已处理 X / 共 N``) so the
|
||||
UI can render a progress bar for both the scan and the apply phases.
|
||||
|
||||
Routes (prefix ``/api/skill-addresses``, **admin only**):
|
||||
GET /scan?kinds=url,ip&skill=<name> read-only aggregated scan (one shot)
|
||||
GET /scan/stream?kinds=...&skill=... scan with per-skill NDJSON progress
|
||||
POST /preview dry-run diff of a replacement set
|
||||
POST /apply write changes (one shot, with backup)
|
||||
POST /apply/stream write changes with per-file progress
|
||||
GET /backups list restorable backups (newest first)
|
||||
POST /rollback/{backup_id} restore a backup
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import shutil
|
||||
from collections.abc import AsyncIterator
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query, Request
|
||||
from fastapi.responses import StreamingResponse
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
from app.gateway.deps import get_optional_user_from_request
|
||||
from deerflow.agents.lead_agent.prompt import refresh_skills_system_prompt_cache_async
|
||||
from deerflow.skills.address_scan import (
|
||||
VALID_KINDS,
|
||||
apply_replacements,
|
||||
infer_kind,
|
||||
is_valid_replacement,
|
||||
normalize_kinds,
|
||||
scan_occurrences,
|
||||
)
|
||||
from deerflow.skills.storage import get_or_new_skill_storage
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/skill-addresses", tags=["skill-addresses"])
|
||||
|
||||
_BACKUP_DIRNAME = ".address-edits"
|
||||
_SCAN_SUBDIRS = ("public", "custom")
|
||||
_SCAN_EXTS = (".md", ".py")
|
||||
|
||||
|
||||
# ── schemas ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class OccurrenceModel(BaseModel):
|
||||
skill: str
|
||||
category: str # "public" | "custom"
|
||||
file: str # POSIX path relative to the skills root
|
||||
line: int
|
||||
col: int
|
||||
snippet: str
|
||||
|
||||
|
||||
class AddressGroup(BaseModel):
|
||||
address: str
|
||||
kind: str # "url" | "ip"
|
||||
count: int
|
||||
occurrences: list[OccurrenceModel]
|
||||
|
||||
|
||||
class ScanResponse(BaseModel):
|
||||
groups: list[AddressGroup]
|
||||
scanned_files: int
|
||||
scanned_skills: int
|
||||
kinds: list[str]
|
||||
scanned_at: str
|
||||
|
||||
|
||||
class Replacement(BaseModel):
|
||||
model_config = ConfigDict(populate_by_name=True)
|
||||
|
||||
from_: str = Field(alias="from", min_length=1)
|
||||
to: str = Field(min_length=1)
|
||||
kind: str | None = None
|
||||
|
||||
|
||||
class PreviewRequest(BaseModel):
|
||||
replacements: list[Replacement] = Field(default_factory=list)
|
||||
|
||||
|
||||
class PreviewFileHit(BaseModel):
|
||||
file: str
|
||||
line: int
|
||||
before: str
|
||||
after: str
|
||||
|
||||
|
||||
class PreviewItem(BaseModel):
|
||||
model_config = ConfigDict(populate_by_name=True)
|
||||
|
||||
from_: str = Field(serialization_alias="from")
|
||||
to: str
|
||||
kind: str
|
||||
count: int
|
||||
files: list[PreviewFileHit]
|
||||
|
||||
|
||||
class PreviewResponse(BaseModel):
|
||||
items: list[PreviewItem]
|
||||
total: int
|
||||
warnings: list[str]
|
||||
expected_total: int
|
||||
|
||||
|
||||
class ApplyRequest(BaseModel):
|
||||
replacements: list[Replacement] = Field(default_factory=list)
|
||||
expected_total: int | None = None
|
||||
backup: bool = True
|
||||
|
||||
|
||||
class ChangedFile(BaseModel):
|
||||
file: str
|
||||
count: int
|
||||
|
||||
|
||||
class ApplyResponse(BaseModel):
|
||||
changed_files: list[ChangedFile]
|
||||
replaced_total: int
|
||||
backup_id: str | None
|
||||
warnings: list[str]
|
||||
|
||||
|
||||
class ReplacementSummary(BaseModel):
|
||||
model_config = ConfigDict(populate_by_name=True)
|
||||
|
||||
from_: str = Field(serialization_alias="from")
|
||||
to: str
|
||||
kind: str
|
||||
|
||||
|
||||
class BackupEntry(BaseModel):
|
||||
backup_id: str
|
||||
files: int
|
||||
created_at: str
|
||||
replaced_total: int = 0
|
||||
replacements: list[ReplacementSummary] = Field(default_factory=list)
|
||||
|
||||
|
||||
class BackupListResponse(BaseModel):
|
||||
backups: list[BackupEntry]
|
||||
|
||||
|
||||
class RollbackResponse(BaseModel):
|
||||
backup_id: str
|
||||
restored_files: list[str]
|
||||
|
||||
|
||||
# ── auth / path helpers ──────────────────────────────────────────────────────
|
||||
|
||||
|
||||
async def _require_admin(request: Request) -> None:
|
||||
"""Admin-only. Lets the call through when auth is disabled (no user)."""
|
||||
|
||||
user = await get_optional_user_from_request(request)
|
||||
if user is None:
|
||||
return
|
||||
if getattr(user, "system_role", None) != "admin":
|
||||
raise HTTPException(status_code=403, detail="技能地址管理仅限管理员")
|
||||
|
||||
|
||||
def _skills_root() -> Path:
|
||||
return Path(get_or_new_skill_storage().get_skills_root_path()).resolve()
|
||||
|
||||
|
||||
def _is_within(root: Path, p: Path) -> bool:
|
||||
try:
|
||||
p.resolve().relative_to(root)
|
||||
return True
|
||||
except (ValueError, OSError):
|
||||
return False
|
||||
|
||||
|
||||
def _skill_of(root: Path, path: Path) -> tuple[str, str, str]:
|
||||
"""Return ``(category, skill_name, posix_rel_path)`` for a scanned file."""
|
||||
|
||||
rel = path.relative_to(root)
|
||||
parts = rel.parts
|
||||
category = parts[0] if parts else ""
|
||||
skill_name = parts[1] if len(parts) > 1 else ""
|
||||
return category, skill_name, rel.as_posix()
|
||||
|
||||
|
||||
def _read_text(path: Path) -> str | None:
|
||||
"""Read UTF-8 text preserving original line endings; skip binary/bad files."""
|
||||
|
||||
try:
|
||||
with open(path, "r", encoding="utf-8", newline="") as fh:
|
||||
return fh.read()
|
||||
except (UnicodeDecodeError, OSError):
|
||||
return None
|
||||
|
||||
|
||||
def _write_text(path: Path, text: str) -> None:
|
||||
with open(path, "w", encoding="utf-8", newline="") as fh:
|
||||
fh.write(text)
|
||||
|
||||
|
||||
# ── synchronous workers (run via asyncio.to_thread; never block the loop) ─────
|
||||
|
||||
|
||||
def _iter_skill_files(root: Path, skill: str | None = None) -> list[Path]:
|
||||
"""All ``*.md`` / ``*.py`` under ``{root}/{public,custom}``.
|
||||
|
||||
Hidden directories (``.address-edits`` backups, ``.history``) are skipped.
|
||||
When *skill* is given, only that skill's directory is walked.
|
||||
"""
|
||||
|
||||
files: list[Path] = []
|
||||
for sub in _SCAN_SUBDIRS:
|
||||
base = root / sub
|
||||
if skill:
|
||||
base = base / skill
|
||||
if not base.is_dir():
|
||||
continue
|
||||
for p in base.rglob("*"):
|
||||
if not p.is_file() or p.suffix.lower() not in _SCAN_EXTS:
|
||||
continue
|
||||
try:
|
||||
rel_parts = p.relative_to(root).parts
|
||||
except ValueError:
|
||||
continue
|
||||
if any(part.startswith(".") for part in rel_parts):
|
||||
continue
|
||||
files.append(p)
|
||||
return files
|
||||
|
||||
|
||||
def _group_files_by_skill(root: Path, files: list[Path]) -> dict[tuple[str, str], list[Path]]:
|
||||
"""Bucket scanned files under their owning ``(category, skill_name)``."""
|
||||
|
||||
buckets: dict[tuple[str, str], list[Path]] = {}
|
||||
for p in files:
|
||||
category, skill_name, _ = _skill_of(root, p)
|
||||
buckets.setdefault((category, skill_name), []).append(p)
|
||||
return buckets
|
||||
|
||||
|
||||
def _occurrence_kind(value: str) -> str:
|
||||
return infer_kind(value) or "url"
|
||||
|
||||
|
||||
def _merge_occurrences(groups: dict[str, AddressGroup], pairs: list[tuple[str, OccurrenceModel]]) -> None:
|
||||
"""Merge ``(address, occurrence)`` pairs into the aggregate group map."""
|
||||
|
||||
for address, occ in pairs:
|
||||
g = groups.get(address)
|
||||
if g is None:
|
||||
g = AddressGroup(address=address, kind=_occurrence_kind(address), count=0, occurrences=[])
|
||||
groups[address] = g
|
||||
g.count += 1
|
||||
g.occurrences.append(occ)
|
||||
|
||||
|
||||
def _plan_apply(root: Path, mapping: dict[str, str]) -> tuple[list[tuple[Path, str, int, str]], int]:
|
||||
"""Compute the rewrite plan without touching disk → (plan, current_total)."""
|
||||
|
||||
plan: list[tuple[Path, str, int, str]] = []
|
||||
current_total = 0
|
||||
for path in _iter_skill_files(root):
|
||||
text = _read_text(path)
|
||||
if text is None:
|
||||
continue
|
||||
new_text, cnt = apply_replacements(text, mapping, VALID_KINDS)
|
||||
if cnt <= 0:
|
||||
continue
|
||||
if not _is_within(root, path):
|
||||
raise HTTPException(status_code=400, detail=f"拒绝写入技能目录之外的文件：{path}")
|
||||
_, _, rel = _skill_of(root, path)
|
||||
plan.append((path, new_text, cnt, rel))
|
||||
current_total += cnt
|
||||
return plan, current_total
|
||||
|
||||
|
||||
def _backup_file(root: Path, backup_id: str, path: Path, rel: str) -> None:
|
||||
dest = root / _BACKUP_DIRNAME / backup_id / rel
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(path, dest)
|
||||
|
||||
|
||||
_MANIFEST_SUFFIX = ".manifest.json"
|
||||
|
||||
|
||||
def _manifest_path(root: Path, backup_id: str) -> Path:
|
||||
"""Manifest lives as a *sibling* of the backup dir (``{id}.manifest.json``),
|
||||
not inside it — so it is neither restored back into the skills tree on
|
||||
rollback nor counted as a backed-up file."""
|
||||
|
||||
return root / _BACKUP_DIRNAME / f"{backup_id}{_MANIFEST_SUFFIX}"
|
||||
|
||||
|
||||
def _write_manifest(
|
||||
root: Path,
|
||||
backup_id: str,
|
||||
norm: list[tuple[str, str, str]],
|
||||
changed: list["ChangedFile"],
|
||||
replaced_total: int,
|
||||
created_at: str,
|
||||
) -> None:
|
||||
"""Record what this edit changed so the UI can show a meaningful, per-edit
|
||||
history (原地址 → 新地址, 文件数, 处数) and roll a single record back."""
|
||||
|
||||
data = {
|
||||
"created_at": created_at,
|
||||
"replaced_total": replaced_total,
|
||||
"replacements": [{"from": frm, "to": to, "kind": kind} for frm, to, kind in norm],
|
||||
"changed_files": [{"file": c.file, "count": c.count} for c in changed],
|
||||
}
|
||||
try: # best-effort: a metadata write failure must not fail the apply itself
|
||||
path = _manifest_path(root, backup_id)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(path, "w", encoding="utf-8") as fh:
|
||||
json.dump(data, fh, ensure_ascii=False)
|
||||
except OSError:
|
||||
logger.warning("Failed to write skill-address backup manifest for %s", backup_id, exc_info=True)
|
||||
|
||||
|
||||
def _read_manifest(root: Path, backup_id: str) -> dict | None:
|
||||
try:
|
||||
with open(_manifest_path(root, backup_id), "r", encoding="utf-8") as fh:
|
||||
return json.load(fh)
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return None
|
||||
|
||||
|
||||
# ── scan extraction shared by one-shot + stream ───────────────────────────────
|
||||
|
||||
|
||||
def _scan_files_with_addresses(root: Path, files: list[Path], wanted: tuple[str, ...]) -> tuple[list[tuple[str, OccurrenceModel]], int]:
|
||||
"""Scan files → ([(address, occurrence)], files_read)."""
|
||||
|
||||
pairs: list[tuple[str, OccurrenceModel]] = []
|
||||
read = 0
|
||||
for path in files:
|
||||
text = _read_text(path)
|
||||
if text is None:
|
||||
continue
|
||||
read += 1
|
||||
category, skill_name, rel = _skill_of(root, path)
|
||||
for o in scan_occurrences(text, wanted):
|
||||
pairs.append(
|
||||
(
|
||||
o.value,
|
||||
OccurrenceModel(skill=skill_name, category=category, file=rel, line=o.line, col=o.col, snippet=o.snippet),
|
||||
)
|
||||
)
|
||||
return pairs, read
|
||||
|
||||
|
||||
def _ordered_groups(groups: dict[str, AddressGroup]) -> list[AddressGroup]:
|
||||
return sorted(groups.values(), key=lambda g: (-g.count, g.address))
|
||||
|
||||
|
||||
def _build_mapping(replacements: list[Replacement]) -> tuple[dict[str, str], list[tuple[str, str, str]], list[str]]:
|
||||
"""Validate replacements → ``(mapping, [(from,to,kind)], warnings)``.
|
||||
|
||||
Drops (with a warning) any entry that is empty, duplicated, not a recognized
|
||||
URL/IP, a no-op (``from == to``), or whose ``to`` is malformed for its kind.
|
||||
"""
|
||||
|
||||
mapping: dict[str, str] = {}
|
||||
norm: list[tuple[str, str, str]] = []
|
||||
warnings: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for r in replacements:
|
||||
frm = r.from_.strip()
|
||||
to = r.to.strip()
|
||||
if not frm:
|
||||
continue
|
||||
if frm in seen:
|
||||
warnings.append(f"重复的原始地址已忽略：{frm}")
|
||||
continue
|
||||
kind = (r.kind or "").strip().lower() or infer_kind(frm)
|
||||
if kind not in VALID_KINDS:
|
||||
warnings.append(f"无法识别为 URL/IP，已忽略：{frm}")
|
||||
continue
|
||||
if to == frm:
|
||||
warnings.append(f"替换值与原值相同，已忽略：{frm}")
|
||||
continue
|
||||
if not is_valid_replacement(kind, to):
|
||||
warnings.append(f"替换值不是合法的 {kind}，已忽略：{to}")
|
||||
continue
|
||||
seen.add(frm)
|
||||
mapping[frm] = to
|
||||
norm.append((frm, to, kind))
|
||||
return mapping, norm, warnings
|
||||
|
||||
|
||||
def _ndjson(obj: dict) -> bytes:
|
||||
return (json.dumps(obj, ensure_ascii=False) + "\n").encode("utf-8")
|
||||
|
||||
|
||||
# ── routes: scan ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.get("/scan", response_model=ScanResponse)
|
||||
async def scan_addresses(
|
||||
request: Request,
|
||||
kinds: str | None = Query(default=None, description="逗号分隔：url,ip（默认全部）"),
|
||||
skill: str | None = Query(default=None, description="仅扫描指定技能目录"),
|
||||
) -> ScanResponse:
|
||||
await _require_admin(request)
|
||||
root = _skills_root()
|
||||
wanted = normalize_kinds(kinds)
|
||||
files = await asyncio.to_thread(_iter_skill_files, root, skill)
|
||||
buckets = _group_files_by_skill(root, files)
|
||||
groups: dict[str, AddressGroup] = {}
|
||||
scanned = 0
|
||||
for paths in buckets.values():
|
||||
pairs, read = await asyncio.to_thread(_scan_files_with_addresses, root, paths, wanted)
|
||||
_merge_occurrences(groups, pairs)
|
||||
scanned += read
|
||||
return ScanResponse(
|
||||
groups=_ordered_groups(groups),
|
||||
scanned_files=scanned,
|
||||
scanned_skills=len(buckets),
|
||||
kinds=list(wanted),
|
||||
scanned_at=datetime.now(UTC).isoformat(),
|
||||
)
|
||||
|
||||
|
||||
@router.get("/scan/stream")
|
||||
async def scan_addresses_stream(
|
||||
request: Request,
|
||||
kinds: str | None = Query(default=None),
|
||||
skill: str | None = Query(default=None),
|
||||
) -> StreamingResponse:
|
||||
"""Scan with per-skill NDJSON progress.
|
||||
|
||||
Emits ``{"type":"start","total":N}`` then one
|
||||
``{"type":"progress","done":i,"total":N,"skill":name}`` per skill, finally
|
||||
``{"type":"result", ...ScanResponse...}``.
|
||||
"""
|
||||
|
||||
await _require_admin(request)
|
||||
root = _skills_root()
|
||||
wanted = normalize_kinds(kinds)
|
||||
|
||||
async def gen() -> AsyncIterator[bytes]:
|
||||
files = await asyncio.to_thread(_iter_skill_files, root, skill)
|
||||
buckets = _group_files_by_skill(root, files)
|
||||
total = len(buckets)
|
||||
yield _ndjson({"type": "start", "total": total})
|
||||
groups: dict[str, AddressGroup] = {}
|
||||
scanned = 0
|
||||
done = 0
|
||||
for (category, skill_name), paths in buckets.items():
|
||||
pairs, read = await asyncio.to_thread(_scan_files_with_addresses, root, paths, wanted)
|
||||
_merge_occurrences(groups, pairs)
|
||||
scanned += read
|
||||
done += 1
|
||||
yield _ndjson({"type": "progress", "done": done, "total": total, "skill": skill_name or category})
|
||||
result = ScanResponse(
|
||||
groups=_ordered_groups(groups),
|
||||
scanned_files=scanned,
|
||||
scanned_skills=total,
|
||||
kinds=list(wanted),
|
||||
scanned_at=datetime.now(UTC).isoformat(),
|
||||
)
|
||||
yield _ndjson({"type": "result", **result.model_dump()})
|
||||
|
||||
return StreamingResponse(gen(), media_type="application/x-ndjson")
|
||||
|
||||
|
||||
# ── routes: preview ──────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _compute_preview(root: Path, mapping: dict[str, str], norm: list[tuple[str, str, str]]) -> tuple[list[PreviewItem], int]:
|
||||
counts: dict[str, int] = {frm: 0 for frm, _, _ in norm}
|
||||
files: dict[str, list[PreviewFileHit]] = {frm: [] for frm, _, _ in norm}
|
||||
total = 0
|
||||
if mapping:
|
||||
for path in _iter_skill_files(root):
|
||||
text = _read_text(path)
|
||||
if text is None:
|
||||
continue
|
||||
_, _, rel = _skill_of(root, path)
|
||||
for o in scan_occurrences(text, VALID_KINDS):
|
||||
if o.value not in mapping:
|
||||
continue
|
||||
to = mapping[o.value]
|
||||
counts[o.value] += 1
|
||||
total += 1
|
||||
files[o.value].append(PreviewFileHit(file=rel, line=o.line, before=o.snippet, after=o.snippet.replace(o.value, to)))
|
||||
items = [PreviewItem(from_=frm, to=to, kind=kind, count=counts[frm], files=files[frm]) for frm, to, kind in norm]
|
||||
return items, total
|
||||
|
||||
|
||||
@router.post("/preview", response_model=PreviewResponse)
|
||||
async def preview_replacements(request: Request, body: PreviewRequest) -> PreviewResponse:
|
||||
await _require_admin(request)
|
||||
root = _skills_root()
|
||||
mapping, norm, warnings = _build_mapping(body.replacements)
|
||||
items, total = await asyncio.to_thread(_compute_preview, root, mapping, norm)
|
||||
return PreviewResponse(items=items, total=total, warnings=warnings, expected_total=total)
|
||||
|
||||
|
||||
# ── routes: apply ────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _new_backup_id() -> str:
|
||||
return datetime.now(UTC).strftime("%Y%m%d-%H%M%S-%f")
|
||||
|
||||
|
||||
@router.post("/apply", response_model=ApplyResponse)
|
||||
async def apply_replacements_route(request: Request, body: ApplyRequest) -> ApplyResponse:
|
||||
await _require_admin(request)
|
||||
root = _skills_root()
|
||||
mapping, norm, warnings = _build_mapping(body.replacements)
|
||||
if not mapping:
|
||||
return ApplyResponse(changed_files=[], replaced_total=0, backup_id=None, warnings=warnings)
|
||||
|
||||
plan, current_total = await asyncio.to_thread(_plan_apply, root, mapping)
|
||||
if body.expected_total is not None and body.expected_total != current_total:
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail=f"文件自上次扫描后已变化（预期 {body.expected_total} 处，现为 {current_total} 处），请重新扫描后再应用。",
|
||||
)
|
||||
if not plan:
|
||||
return ApplyResponse(changed_files=[], replaced_total=0, backup_id=None, warnings=warnings)
|
||||
|
||||
backup_id: str | None = None
|
||||
if body.backup:
|
||||
backup_id = _new_backup_id()
|
||||
for path, _new, _cnt, rel in plan:
|
||||
await asyncio.to_thread(_backup_file, root, backup_id, path, rel)
|
||||
|
||||
changed: list[ChangedFile] = []
|
||||
replaced_total = 0
|
||||
for path, new_text, cnt, rel in plan:
|
||||
await asyncio.to_thread(_write_text, path, new_text)
|
||||
changed.append(ChangedFile(file=rel, count=cnt))
|
||||
replaced_total += cnt
|
||||
|
||||
if backup_id:
|
||||
await asyncio.to_thread(_write_manifest, root, backup_id, norm, changed, replaced_total, datetime.now(UTC).isoformat())
|
||||
|
||||
await _refresh_cache_safe()
|
||||
logger.info("Skill address apply: %d files, %d replacements, backup=%s", len(changed), replaced_total, backup_id)
|
||||
return ApplyResponse(changed_files=changed, replaced_total=replaced_total, backup_id=backup_id, warnings=warnings)
|
||||
|
||||
|
||||
@router.post("/apply/stream")
|
||||
async def apply_replacements_stream(request: Request, body: ApplyRequest) -> StreamingResponse:
|
||||
"""Apply with per-file NDJSON progress.
|
||||
|
||||
Validation + the optimistic-lock check happen *before* streaming (so a 409
|
||||
is a normal HTTP error). Then: ``{"type":"start","total_files":M,...}`` →
|
||||
one ``{"type":"progress","done":k,...}`` per written file →
|
||||
``{"type":"result", ...ApplyResponse...}``.
|
||||
"""
|
||||
|
||||
await _require_admin(request)
|
||||
root = _skills_root()
|
||||
mapping, norm, warnings = _build_mapping(body.replacements)
|
||||
|
||||
plan: list[tuple[Path, str, int, str]] = []
|
||||
current_total = 0
|
||||
if mapping:
|
||||
plan, current_total = await asyncio.to_thread(_plan_apply, root, mapping)
|
||||
if body.expected_total is not None and body.expected_total != current_total:
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail=f"文件自上次扫描后已变化（预期 {body.expected_total} 处，现为 {current_total} 处），请重新扫描后再应用。",
|
||||
)
|
||||
|
||||
async def gen() -> AsyncIterator[bytes]:
|
||||
if not plan:
|
||||
result = ApplyResponse(changed_files=[], replaced_total=0, backup_id=None, warnings=warnings)
|
||||
yield _ndjson({"type": "result", **result.model_dump()})
|
||||
return
|
||||
|
||||
backup_id: str | None = None
|
||||
if body.backup:
|
||||
backup_id = _new_backup_id()
|
||||
|
||||
yield _ndjson({"type": "start", "total_files": len(plan), "total_replacements": current_total})
|
||||
|
||||
changed: list[ChangedFile] = []
|
||||
replaced_total = 0
|
||||
done = 0
|
||||
for path, new_text, cnt, rel in plan:
|
||||
if backup_id:
|
||||
await asyncio.to_thread(_backup_file, root, backup_id, path, rel)
|
||||
await asyncio.to_thread(_write_text, path, new_text)
|
||||
changed.append(ChangedFile(file=rel, count=cnt))
|
||||
replaced_total += cnt
|
||||
done += 1
|
||||
yield _ndjson({"type": "progress", "done": done, "total_files": len(plan), "file": rel, "count": cnt})
|
||||
|
||||
if backup_id:
|
||||
await asyncio.to_thread(_write_manifest, root, backup_id, norm, changed, replaced_total, datetime.now(UTC).isoformat())
|
||||
|
||||
await _refresh_cache_safe()
|
||||
logger.info("Skill address apply(stream): %d files, %d replacements, backup=%s", len(changed), replaced_total, backup_id)
|
||||
result = ApplyResponse(changed_files=changed, replaced_total=replaced_total, backup_id=backup_id, warnings=warnings)
|
||||
yield _ndjson({"type": "result", **result.model_dump()})
|
||||
|
||||
return StreamingResponse(gen(), media_type="application/x-ndjson")
|
||||
|
||||
|
||||
# ── routes: backups / rollback ───────────────────────────────────────────────
|
||||
|
||||
|
||||
def _list_backups(root: Path) -> list[BackupEntry]:
|
||||
backup_root = root / _BACKUP_DIRNAME
|
||||
entries: list[BackupEntry] = []
|
||||
if backup_root.is_dir():
|
||||
for d in backup_root.iterdir():
|
||||
if not d.is_dir():
|
||||
continue
|
||||
file_count = sum(1 for p in d.rglob("*") if p.is_file())
|
||||
manifest = _read_manifest(root, d.name)
|
||||
created = ""
|
||||
replaced_total = 0
|
||||
replacements: list[ReplacementSummary] = []
|
||||
if manifest:
|
||||
created = str(manifest.get("created_at") or "")
|
||||
replaced_total = int(manifest.get("replaced_total") or 0)
|
||||
replacements = [
|
||||
ReplacementSummary(from_=str(r.get("from", "")), to=str(r.get("to", "")), kind=str(r.get("kind", "")))
|
||||
for r in manifest.get("replacements", [])
|
||||
if isinstance(r, dict)
|
||||
]
|
||||
if not created:
|
||||
try:
|
||||
created = datetime.fromtimestamp(d.stat().st_mtime, UTC).isoformat()
|
||||
except OSError:
|
||||
created = ""
|
||||
entries.append(
|
||||
BackupEntry(
|
||||
backup_id=d.name,
|
||||
files=file_count,
|
||||
created_at=created,
|
||||
replaced_total=replaced_total,
|
||||
replacements=replacements,
|
||||
)
|
||||
)
|
||||
entries.sort(key=lambda e: e.backup_id, reverse=True)
|
||||
return entries
|
||||
|
||||
|
||||
@router.get("/backups", response_model=BackupListResponse)
|
||||
async def list_backups(request: Request) -> BackupListResponse:
|
||||
await _require_admin(request)
|
||||
root = _skills_root()
|
||||
entries = await asyncio.to_thread(_list_backups, root)
|
||||
return BackupListResponse(backups=entries)
|
||||
|
||||
|
||||
def _restore_backup(root: Path, backup_root: Path) -> list[str]:
|
||||
restored: list[str] = []
|
||||
for src in backup_root.rglob("*"):
|
||||
if not src.is_file():
|
||||
continue
|
||||
rel = src.relative_to(backup_root)
|
||||
dest = root / rel
|
||||
if not _is_within(root, dest):
|
||||
continue
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(src, dest)
|
||||
restored.append(rel.as_posix())
|
||||
return restored
|
||||
|
||||
|
||||
@router.post("/rollback/{backup_id}", response_model=RollbackResponse)
|
||||
async def rollback(request: Request, backup_id: str) -> RollbackResponse:
|
||||
await _require_admin(request)
|
||||
root = _skills_root()
|
||||
if "/" in backup_id or "\\" in backup_id or backup_id in (".", ".."):
|
||||
raise HTTPException(status_code=400, detail="非法的备份编号")
|
||||
backup_root = (root / _BACKUP_DIRNAME / backup_id).resolve()
|
||||
if not _is_within(root, backup_root) or not backup_root.is_dir():
|
||||
raise HTTPException(status_code=404, detail="备份不存在")
|
||||
|
||||
restored = await asyncio.to_thread(_restore_backup, root, backup_root)
|
||||
await _refresh_cache_safe()
|
||||
logger.info("Skill address rollback: backup=%s, restored %d files", backup_id, len(restored))
|
||||
return RollbackResponse(backup_id=backup_id, restored_files=restored)
|
||||
|
||||
|
||||
# ── misc ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
async def _refresh_cache_safe() -> None:
|
||||
try:
|
||||
await refresh_skills_system_prompt_cache_async()
|
||||
except Exception: # noqa: BLE001 — cache refresh is best-effort
|
||||
logger.warning("Failed to refresh skills prompt cache", exc_info=True)
|
||||
@@ -222,6 +222,10 @@ _CONTEXT_CONFIGURABLE_KEYS: frozenset[str] = frozenset(
|
||||
"model_name",
|
||||
"mode",
|
||||
"thinking_enabled",
|
||||
# 强制关闭思考(内网 vLLM/Qwen 模型只声明 supports_thinking 时,单靠
|
||||
# thinking_enabled=False 不会注入 enable_thinking=false)。透传到 run config
|
||||
# 后由 lead agent 交给 create_chat_model(force_disable_thinking=...)。
|
||||
"thinking_force_disabled",
|
||||
"reasoning_effort",
|
||||
"rag_mode_enabled",
|
||||
"writing_mode",
|
||||
|
||||
@@ -123,6 +123,12 @@ models:
|
||||
temperature: 1.0
|
||||
supports_vision: true
|
||||
supports_thinking: true
|
||||
# SiliconFlow 混合推理模型关思考用「请求体顶层 enable_thinking: false」,
|
||||
# 不认 chat_template_kwargs / thinking.type 那两种嵌套形状。声明这个后,
|
||||
# 网页端「关闭思考」对本模型才生效(开放接口另由 force_disable_thinking 兜底)。
|
||||
when_thinking_disabled:
|
||||
extra_body:
|
||||
enable_thinking: false
|
||||
- name: Pro/MiniMaxAI/MiniMax-M2.5
|
||||
display_name: Pro/MiniMaxAI/MiniMax-M2.5
|
||||
use: langchain_openai:ChatOpenAI
|
||||
|
||||
+20
-7
@@ -306,17 +306,25 @@ def _build_middlewares(
|
||||
if not cfg.get("is_scheduled_run", False):
|
||||
middlewares.append(TitleMiddleware(app_config=resolved_app_config))
|
||||
|
||||
# Add MemoryMiddleware (after TitleMiddleware) — skip in bootstrap mode (agent creation flow)
|
||||
# Ephemeral runs (e.g. the open Q&A API /api/open/chat) must NOT touch any
|
||||
# server-side per-user state: no memory recall/retain, no knowledge-base
|
||||
# recall/auto-ingest. Those runs use the shared "default" user bucket, so
|
||||
# persisting their conversations there would both store records the caller
|
||||
# explicitly asked NOT to keep and leak content across unrelated callers.
|
||||
# Skip both write-capable middlewares entirely.
|
||||
is_ephemeral = cfg.get("ephemeral", False)
|
||||
|
||||
# Add MemoryMiddleware (after TitleMiddleware) — skip in bootstrap mode (agent creation flow) and ephemeral runs
|
||||
memory_injection_enabled = cfg.get("memory_injection_enabled", True)
|
||||
if not is_bootstrap and not is_writing_setup:
|
||||
if not is_bootstrap and not is_writing_setup and not is_ephemeral:
|
||||
middlewares.append(MemoryMiddleware(agent_name=agent_name, memory_config=resolved_app_config.memory, injection_enabled=memory_injection_enabled))
|
||||
|
||||
# KnowledgeRagMiddleware — inject knowledge-base recall (phase 2). Gated
|
||||
# per-conversation by the ``knowledge_rag_enabled`` runtime flag, defaulting
|
||||
# to the global ``knowledge.rag_enabled`` config. No-op when knowledge is
|
||||
# disabled. Skipped in bootstrap/writing-setup like MemoryMiddleware.
|
||||
# disabled. Skipped in bootstrap/writing-setup/ephemeral like MemoryMiddleware.
|
||||
knowledge_cfg = getattr(resolved_app_config, "knowledge", None)
|
||||
if not is_bootstrap and not is_writing_setup and knowledge_cfg is not None and knowledge_cfg.enabled:
|
||||
if not is_bootstrap and not is_writing_setup and not is_ephemeral and knowledge_cfg is not None and knowledge_cfg.enabled:
|
||||
from deerflow.agents.middlewares.knowledge_rag_middleware import KnowledgeRagMiddleware
|
||||
|
||||
middlewares.append(KnowledgeRagMiddleware(limit=knowledge_cfg.rag_limit, default_enabled=knowledge_cfg.rag_enabled))
|
||||
@@ -412,6 +420,11 @@ def _make_lead_agent(config: RunnableConfig, *, app_config: AppConfig):
|
||||
resolved_app_config = app_config
|
||||
|
||||
thinking_enabled = cfg.get("thinking_enabled", True)
|
||||
# Hard "thinking off" switch for internal vLLM/Qwen models whose config only
|
||||
# declares `supports_thinking: true` (no disable shape) — passes through to
|
||||
# create_chat_model(force_disable_thinking=...) so enable_thinking=false is
|
||||
# actually injected. Default off → no behavior change for normal chats.
|
||||
thinking_force_disabled = bool(cfg.get("thinking_force_disabled", False))
|
||||
reasoning_effort = cfg.get("reasoning_effort", None)
|
||||
requested_model_name: str | None = cfg.get("model_name") or cfg.get("model")
|
||||
is_plan_mode = cfg.get("is_plan_mode", False)
|
||||
@@ -501,7 +514,7 @@ def _make_lead_agent(config: RunnableConfig, *, app_config: AppConfig):
|
||||
from deerflow.agents.middlewares.writing_setup_round_cap_middleware import WritingSetupRoundCapMiddleware
|
||||
|
||||
return create_agent(
|
||||
model=create_chat_model(name=model_name, thinking_enabled=thinking_enabled, app_config=resolved_app_config),
|
||||
model=create_chat_model(name=model_name, thinking_enabled=thinking_enabled, force_disable_thinking=thinking_force_disabled, app_config=resolved_app_config),
|
||||
tools=[setup_writing, ask_clarification_tool],
|
||||
middleware=[
|
||||
*_build_middlewares(config, model_name=model_name, app_config=resolved_app_config),
|
||||
@@ -514,7 +527,7 @@ def _make_lead_agent(config: RunnableConfig, *, app_config: AppConfig):
|
||||
if is_bootstrap:
|
||||
# Special bootstrap agent with minimal prompt for initial custom agent creation flow
|
||||
return create_agent(
|
||||
model=create_chat_model(name=model_name, thinking_enabled=thinking_enabled, app_config=resolved_app_config),
|
||||
model=create_chat_model(name=model_name, thinking_enabled=thinking_enabled, force_disable_thinking=thinking_force_disabled, app_config=resolved_app_config),
|
||||
tools=get_available_tools(
|
||||
model_name=model_name,
|
||||
subagent_enabled=subagent_enabled,
|
||||
@@ -593,7 +606,7 @@ def _make_lead_agent(config: RunnableConfig, *, app_config: AppConfig):
|
||||
)
|
||||
|
||||
return create_agent(
|
||||
model=create_chat_model(name=model_name, thinking_enabled=thinking_enabled, reasoning_effort=reasoning_effort, app_config=resolved_app_config),
|
||||
model=create_chat_model(name=model_name, thinking_enabled=thinking_enabled, reasoning_effort=reasoning_effort, force_disable_thinking=thinking_force_disabled, app_config=resolved_app_config),
|
||||
tools=default_tools,
|
||||
middleware=_build_middlewares(config, model_name=model_name, agent_name=agent_id, app_config=resolved_app_config),
|
||||
system_prompt=system_prompt,
|
||||
|
||||
+4
-3
@@ -5,7 +5,8 @@
|
||||
随时改 ``config.yaml`` 即可（Gateway 有 mtime 热重载，无需重新构建前端）。
|
||||
|
||||
测试开关（线上接口未通时用假数据，互相独立）：
|
||||
- ``test`` → 目的树查询（``purpose_detail_url``）走 ``mock.purpose_detail`` 假数据。
|
||||
- ``test`` → 目的树查询（``purpose_detail_url``）走 ``mock.purpose_detail`` 假数据；
|
||||
同时 rwfx 进入「3q 是否完成」预检（``selectSqReport``，外部接口不可用）走 ``mock.sq_report`` 假数据。
|
||||
- ``delete_test`` → 删除接口（``delete_url``）走 ``mock.delete`` 假数据（该接口尚未开发完时打开）。
|
||||
- ``action_test`` → 行动详情接口（``action_detail_url``）走 ``mock.action_detail`` 假数据。
|
||||
xdfx 深链传入的是「行动id」，需先调该接口拿 ``data.taskId`` 解析出真正的「任务id」。
|
||||
@@ -25,7 +26,7 @@ class TaskDeeplinkConfig(BaseModel):
|
||||
|
||||
test: bool = Field(
|
||||
default=False,
|
||||
description="为真时前端目的树查询走 mock.purpose_detail 假数据（不打真实接口）",
|
||||
description="为真时前端目的树查询走 mock.purpose_detail 假数据；同时 rwfx 进入「3q 是否完成」预检（selectSqReport）走 mock.sq_report 假数据（都不打真实接口）",
|
||||
)
|
||||
delete_test: bool = Field(
|
||||
default=False,
|
||||
@@ -61,5 +62,5 @@ class TaskDeeplinkConfig(BaseModel):
|
||||
)
|
||||
mock: dict[str, Any] = Field(
|
||||
default_factory=dict,
|
||||
description="test/delete_test/action_test 为真时返回的假数据：{purpose_detail: {...}, delete: {message, success}, action_detail: {state, msg, data: {taskId, ...}}}",
|
||||
description="test/delete_test/action_test 为真时返回的假数据：{purpose_detail: {...}, delete: {message, success}, action_detail: {state, msg, data: {taskId, ...}}, sq_report: {state, msg, data: [...]}}（sq_report 供 rwfx 3q 预检，test 为真时用）",
|
||||
)
|
||||
@@ -47,11 +47,22 @@ def _enable_stream_usage_by_default(model_use_path: str, model_settings_from_con
|
||||
model_settings_from_config["stream_usage"] = True
|
||||
|
||||
|
||||
def create_chat_model(name: str | None = None, thinking_enabled: bool = False, *, app_config: AppConfig | None = None, **kwargs) -> BaseChatModel:
|
||||
def create_chat_model(name: str | None = None, thinking_enabled: bool = False, *, app_config: AppConfig | None = None, force_disable_thinking: bool = False, **kwargs) -> BaseChatModel:
|
||||
"""Create a chat model instance from the config.
|
||||
|
||||
Args:
|
||||
name: The name of the model to create. If None, the first model in the config will be used.
|
||||
thinking_enabled: Turn the model's extended thinking / reasoning on.
|
||||
force_disable_thinking: Hard override that disables thinking even when the
|
||||
per-model config declares no ``when_thinking_*`` shape. Many internal
|
||||
vLLM / Qwen models only set ``supports_thinking: true`` (no disable
|
||||
shape), so passing ``thinking_enabled=False`` alone injects nothing and
|
||||
the model keeps reasoning on. When this flag is set (and thinking is
|
||||
off) we additionally inject both common OpenAI-compatible disable shapes
|
||||
— vLLM/Qwen ``chat_template_kwargs.enable_thinking=False`` and Zhipu GLM
|
||||
``thinking.type="disabled"`` — mirroring
|
||||
``TitleMiddleware._force_disable_thinking``. No effect when thinking is
|
||||
enabled, and skipped for model classes that don't accept ``extra_body``.
|
||||
|
||||
Returns:
|
||||
A chat model instance.
|
||||
@@ -120,6 +131,21 @@ def create_chat_model(name: str | None = None, thinking_enabled: bool = False, *
|
||||
kwargs.pop("reasoning_effort", None)
|
||||
model_settings_from_config.pop("reasoning_effort", None)
|
||||
|
||||
# Hard override: force-disable thinking regardless of the per-model config shape.
|
||||
# Injected only for OpenAI-compatible classes (those declaring `extra_body`);
|
||||
# other providers would reject the kwarg, so they are skipped (they ignore it).
|
||||
# Three disable shapes are injected so the common OpenAI-compatible gateways are
|
||||
# all covered: SiliconFlow / DashScope-style **top-level** ``enable_thinking``
|
||||
# (DeepSeek-V3.1+/Qwen3/GLM hybrid models on api.siliconflow.cn — the knob the
|
||||
# nested ``chat_template_kwargs`` shape does NOT reach), vLLM/Qwen native
|
||||
# ``chat_template_kwargs.enable_thinking``, and Zhipu GLM ``thinking.type``.
|
||||
# Unknown fields are ignored by these gateways, so sending all three is safe.
|
||||
if force_disable_thinking and not thinking_enabled and "extra_body" in getattr(model_class, "model_fields", {}):
|
||||
model_settings_from_config["extra_body"] = _deep_merge_dicts(
|
||||
model_settings_from_config.get("extra_body"),
|
||||
{"enable_thinking": False, "chat_template_kwargs": {"enable_thinking": False}, "thinking": {"type": "disabled"}},
|
||||
)
|
||||
|
||||
_enable_stream_usage_by_default(model_config.use, model_settings_from_config)
|
||||
|
||||
# For Codex Responses API models: map thinking mode to reasoning_effort
|
||||
|
||||
+11
-6
@@ -16,8 +16,11 @@ Resolution rule (per resource type):
|
||||
- ``scheduled_task`` → empty (no built-in tasks)
|
||||
- ``skill`` → empty (the 广场 still shows every visible skill;
|
||||
a position only *adds* its tagged skills to the member's "我的技能")
|
||||
- ``recommended_question`` → *unrestricted* (handled at query time, never
|
||||
materialized — see :meth:`resolve_visible`)
|
||||
- ``recommended_question`` → *untagged-only* (handled at query time, never
|
||||
materialized — see :meth:`resolve_visible`). The default set is the
|
||||
questions carrying **no** 标签; tagged questions surface only for a position
|
||||
that binds their tag. (The recommended-questions router interprets a
|
||||
``None`` from :meth:`resolve_visible` as "show only untagged".)
|
||||
|
||||
Materialization (``sync_user``) only ever touches rows it created itself
|
||||
(``origin='position'``), so a user's own favorites / subscriptions are never
|
||||
@@ -67,12 +70,14 @@ class PositionSyncService:
|
||||
return await self._positions.get_default_position_id()
|
||||
|
||||
async def resolve_visible(self, user_id: str, resource_type: str) -> set[str] | None:
|
||||
"""Return the set of visible ``resource_type`` ids, or ``None`` = unrestricted.
|
||||
"""Return the set of visible ``resource_type`` ids, or ``None`` = no tag binding.
|
||||
|
||||
Used by the recommended-questions filter (the only remaining query-time
|
||||
visibility type; skills are now *materialized*, not filtered). ``None``
|
||||
means "no position restriction → show everything"; a set means "restrict
|
||||
to exactly these ids".
|
||||
visibility type; skills are now *materialized*, not filtered). A set
|
||||
means "restrict to exactly these ids". ``None`` means the effective
|
||||
position binds no tag for this type — the recommended-questions router
|
||||
treats that as "默认岗位 → show only untagged questions" (see
|
||||
``app/gateway/routers/recommended_questions.py``).
|
||||
"""
|
||||
pid = await self._effective_position_id(user_id)
|
||||
if not pid:
|
||||
|
||||
@@ -0,0 +1,255 @@
|
||||
"""Scan skill text for URL / IP addresses and apply safe batch replacements.
|
||||
|
||||
Pure-logic layer for the 技能地址(URL/IP)批量替换 feature. This module knows
|
||||
**only about text** — it never walks the filesystem. The HTTP router
|
||||
(``app/gateway/routers/skill_addresses.py``) walks ``skills/{public,custom}``,
|
||||
reads each ``*.md`` / ``*.py`` file, and feeds the text here. Keeping the
|
||||
extraction / replacement logic here (no ``app.*`` import) makes it unit-testable
|
||||
in isolation and respects the harness/app boundary.
|
||||
|
||||
Two address kinds are supported (the feature scope):
|
||||
|
||||
* ``"url"`` — full ``http(s)://…`` URLs.
|
||||
* ``"ip"`` — dotted-quad IPv4, optionally ``:port``. Each octet is range
|
||||
checked (0–255) so version-number noise like ``1.2.3.400`` is not matched.
|
||||
|
||||
The single most important invariant is **substring-safe replacement**: we never
|
||||
``str.replace`` raw file text (that would rewrite ``1.2.3.4`` *inside*
|
||||
``11.2.3.40``). Instead we re-detect the exact token spans and only rewrite a
|
||||
span whose text is *exactly* one of the requested ``from`` values.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
|
||||
__all__ = [
|
||||
"VALID_KINDS",
|
||||
"AddressSpan",
|
||||
"Occurrence",
|
||||
"normalize_kinds",
|
||||
"find_spans",
|
||||
"scan_occurrences",
|
||||
"count_matches",
|
||||
"apply_replacements",
|
||||
"is_valid_replacement",
|
||||
"infer_kind",
|
||||
]
|
||||
|
||||
# Supported address kinds (the configured feature scope: URL + IP).
|
||||
VALID_KINDS: tuple[str, ...] = ("url", "ip")
|
||||
|
||||
# A URL body stops at whitespace, quotes, brackets, angle brackets, backticks
|
||||
# and backslashes — everything that commonly *delimits* a URL in md / py. The
|
||||
# trailing punctuation that often clings to the end (a sentence period, a
|
||||
# closing paren already excluded, a Markdown comma) is stripped afterwards.
|
||||
_URL_RE = re.compile(r"https?://[^\s'\"`<>)\]}\\]+", re.IGNORECASE)
|
||||
|
||||
# Dotted-quad IPv4 (optionally :port). Guarded so it is not glued to a longer
|
||||
# number run (``(?<![\d.])`` / ``(?![\d.])``); octets are range-checked below.
|
||||
_IP_RE = re.compile(r"(?<![\d.])(\d{1,3}(?:\.\d{1,3}){3})(?::(\d{1,5}))?(?![\d.])")
|
||||
|
||||
# Punctuation (ASCII + CJK) that should not be considered part of a URL token
|
||||
# when it appears at the very end of the match.
|
||||
_URL_TRAILING = set(".,;:!?'\")]}>`" + "）。，；：！？】》、…")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AddressSpan:
|
||||
"""An address token located at an exact ``[start, end)`` offset in text."""
|
||||
|
||||
kind: str
|
||||
value: str
|
||||
start: int
|
||||
end: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Occurrence:
|
||||
"""A located address enriched with 1-based line/column + the source line."""
|
||||
|
||||
kind: str
|
||||
value: str
|
||||
line: int
|
||||
col: int
|
||||
snippet: str
|
||||
|
||||
|
||||
def normalize_kinds(kinds: object) -> tuple[str, ...]:
|
||||
"""Coerce arbitrary input into an ordered, de-duped subset of ``VALID_KINDS``.
|
||||
|
||||
``None`` / empty means "all kinds". Unknown kinds are dropped.
|
||||
"""
|
||||
|
||||
if kinds is None:
|
||||
return VALID_KINDS
|
||||
if isinstance(kinds, str):
|
||||
raw = [p.strip() for p in kinds.split(",")]
|
||||
else:
|
||||
try:
|
||||
raw = [str(p).strip() for p in kinds] # type: ignore[assignment]
|
||||
except TypeError:
|
||||
return VALID_KINDS
|
||||
seen: list[str] = []
|
||||
for k in raw:
|
||||
kl = k.lower()
|
||||
if kl in VALID_KINDS and kl not in seen:
|
||||
seen.append(kl)
|
||||
return tuple(seen) if seen else VALID_KINDS
|
||||
|
||||
|
||||
def _valid_ipv4(quad: str) -> bool:
|
||||
parts = quad.split(".")
|
||||
if len(parts) != 4:
|
||||
return False
|
||||
for p in parts:
|
||||
if not p.isdigit() or len(p) > 3:
|
||||
return False
|
||||
if int(p) > 255:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _valid_port(port: str | None) -> bool:
|
||||
if port is None:
|
||||
return True
|
||||
return port.isdigit() and 0 <= int(port) <= 65535
|
||||
|
||||
|
||||
def _url_spans(text: str) -> list[AddressSpan]:
|
||||
spans: list[AddressSpan] = []
|
||||
for m in _URL_RE.finditer(text):
|
||||
start, end = m.start(), m.end()
|
||||
value = m.group(0)
|
||||
# Strip clinging trailing punctuation.
|
||||
while value and value[-1] in _URL_TRAILING:
|
||||
value = value[:-1]
|
||||
end -= 1
|
||||
if value:
|
||||
spans.append(AddressSpan(kind="url", value=value, start=start, end=end))
|
||||
return spans
|
||||
|
||||
|
||||
def _ip_spans(text: str, exclude: list[AddressSpan]) -> list[AddressSpan]:
|
||||
spans: list[AddressSpan] = []
|
||||
for m in _IP_RE.finditer(text):
|
||||
quad = m.group(1)
|
||||
port = m.group(2)
|
||||
if not _valid_ipv4(quad) or not _valid_port(port):
|
||||
continue
|
||||
start, end = m.start(), m.end()
|
||||
# Skip an IP that lives *inside* a URL (e.g. http://1.2.3.4:8080/x) —
|
||||
# replacing just the IP there would corrupt the URL.
|
||||
if any(s.start <= start < s.end for s in exclude):
|
||||
continue
|
||||
spans.append(AddressSpan(kind="ip", value=m.group(0), start=start, end=end))
|
||||
return spans
|
||||
|
||||
|
||||
def find_spans(text: str, kinds: object = None) -> list[AddressSpan]:
|
||||
"""Return every address token span in *text*, ordered by position.
|
||||
|
||||
URL regions are always computed first so that IPs embedded inside a URL are
|
||||
not separately surfaced, even when only ``"ip"`` is requested.
|
||||
"""
|
||||
|
||||
wanted = normalize_kinds(kinds)
|
||||
url_spans = _url_spans(text)
|
||||
result: list[AddressSpan] = []
|
||||
if "url" in wanted:
|
||||
result.extend(url_spans)
|
||||
if "ip" in wanted:
|
||||
result.extend(_ip_spans(text, exclude=url_spans))
|
||||
result.sort(key=lambda s: s.start)
|
||||
return result
|
||||
|
||||
|
||||
def scan_occurrences(text: str, kinds: object = None, *, snippet_max: int = 240) -> list[Occurrence]:
|
||||
"""Locate addresses with 1-based line/column and a trimmed source line."""
|
||||
|
||||
spans = find_spans(text, kinds)
|
||||
if not spans:
|
||||
return []
|
||||
# Precompute line-start offsets for O(log n) line lookup.
|
||||
line_starts = [0]
|
||||
for i, ch in enumerate(text):
|
||||
if ch == "\n":
|
||||
line_starts.append(i + 1)
|
||||
|
||||
occ: list[Occurrence] = []
|
||||
import bisect
|
||||
|
||||
for s in spans:
|
||||
line_idx = bisect.bisect_right(line_starts, s.start) - 1
|
||||
line_no = line_idx + 1
|
||||
col = s.start - line_starts[line_idx] + 1
|
||||
line_end = line_starts[line_idx + 1] - 1 if line_idx + 1 < len(line_starts) else len(text)
|
||||
raw_line = text[line_starts[line_idx]:line_end].rstrip("\r")
|
||||
snippet = raw_line.strip()
|
||||
if len(snippet) > snippet_max:
|
||||
snippet = snippet[: snippet_max - 1] + "…"
|
||||
occ.append(Occurrence(kind=s.kind, value=s.value, line=line_no, col=col, snippet=snippet))
|
||||
return occ
|
||||
|
||||
|
||||
def count_matches(text: str, mapping: dict[str, str], kinds: object = None) -> int:
|
||||
"""How many address spans in *text* would be rewritten by *mapping*."""
|
||||
|
||||
if not mapping:
|
||||
return 0
|
||||
return sum(1 for s in find_spans(text, kinds) if s.value in mapping)
|
||||
|
||||
|
||||
def apply_replacements(text: str, mapping: dict[str, str], kinds: object = None) -> tuple[str, int]:
|
||||
"""Substring-safe batch replace.
|
||||
|
||||
Only spans whose text is *exactly* a key of *mapping* are rewritten, applied
|
||||
right-to-left so earlier offsets stay valid. Returns ``(new_text, count)``.
|
||||
A no-op mapping (or no hits) returns the original text and ``0``.
|
||||
"""
|
||||
|
||||
if not mapping:
|
||||
return text, 0
|
||||
spans = [s for s in find_spans(text, kinds) if s.value in mapping]
|
||||
if not spans:
|
||||
return text, 0
|
||||
out = text
|
||||
for s in sorted(spans, key=lambda s: s.start, reverse=True):
|
||||
out = out[: s.start] + mapping[s.value] + out[s.end:]
|
||||
return out, len(spans)
|
||||
|
||||
|
||||
def infer_kind(value: str) -> str | None:
|
||||
"""Return the address kind of *value* if it is a single, whole address.
|
||||
|
||||
Used to classify a user-supplied ``from`` value (so the matching ``to`` can
|
||||
be validated against the right rules). Returns ``None`` when *value* is not
|
||||
exactly one URL or IP token.
|
||||
"""
|
||||
|
||||
if not value:
|
||||
return None
|
||||
spans = find_spans(value, VALID_KINDS)
|
||||
if len(spans) == 1 and spans[0].value == value:
|
||||
return spans[0].kind
|
||||
return None
|
||||
|
||||
|
||||
def is_valid_replacement(kind: str, value: str) -> bool:
|
||||
"""Validate that a user-supplied replacement is well-formed for its kind.
|
||||
|
||||
Keeps an admin from turning a URL into garbage (e.g. dropping the scheme) or
|
||||
an IP into a non-IP. Empty replacements are rejected.
|
||||
"""
|
||||
|
||||
if not value or value.strip() != value or any(c.isspace() for c in value):
|
||||
return False
|
||||
if kind == "url":
|
||||
return bool(re.fullmatch(r"https?://[^\s'\"`<>]+", value, re.IGNORECASE))
|
||||
if kind == "ip":
|
||||
m = _IP_RE.fullmatch(value)
|
||||
if not m:
|
||||
return False
|
||||
return _valid_ipv4(m.group(1)) and _valid_port(m.group(2))
|
||||
return False
|
||||
@@ -77,3 +77,65 @@ async def test_list_without_tag_store_does_not_break() -> None:
|
||||
|
||||
assert [q.id for q in resp.questions] == ["q1", "q2"]
|
||||
assert all(q.tags == [] for q in resp.questions)
|
||||
|
||||
|
||||
class _FakeSync:
|
||||
"""Stub position-sync whose ``resolve_visible`` returns a fixed value.
|
||||
|
||||
``None`` mimics 默认岗位 / 无标签绑定; a set mimics a position that binds
|
||||
recommended_question 标签.
|
||||
"""
|
||||
|
||||
def __init__(self, visible: set[str] | None) -> None:
|
||||
self._visible = visible
|
||||
|
||||
async def resolve_visible(self, _user_id, _resource_type): # noqa: ANN001
|
||||
return self._visible
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_default_position_shows_only_untagged() -> None:
|
||||
"""默认岗位（resolve_visible → None）只看到未打标的题目。"""
|
||||
tag_store = MemoryTagStore()
|
||||
tag = await tag_store.create_tag(
|
||||
{"id": "t1", "name": "理赔", "tag_type": "recommended_question"}
|
||||
)
|
||||
await tag_store.set_target_tags("recommended_question", "q2", [tag["id"]])
|
||||
|
||||
req = _request(_FakeQuestionStore(_rows()), tag_store)
|
||||
req.app.state.position_sync = _FakeSync(None)
|
||||
resp = await rq_router.list_recommended_questions(req, all=False)
|
||||
|
||||
assert [q.id for q in resp.questions] == ["q1"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_bound_position_shows_only_its_tagged_questions() -> None:
|
||||
"""绑定了标签的岗位（resolve_visible → set）只看到对应打标的题目。"""
|
||||
tag_store = MemoryTagStore()
|
||||
tag = await tag_store.create_tag(
|
||||
{"id": "t1", "name": "理赔", "tag_type": "recommended_question"}
|
||||
)
|
||||
await tag_store.set_target_tags("recommended_question", "q2", [tag["id"]])
|
||||
|
||||
req = _request(_FakeQuestionStore(_rows()), tag_store)
|
||||
req.app.state.position_sync = _FakeSync({"q2"})
|
||||
resp = await rq_router.list_recommended_questions(req, all=False)
|
||||
|
||||
assert [q.id for q in resp.questions] == ["q2"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_admin_view_ignores_position_filter() -> None:
|
||||
"""``all=True`` 管理视图照旧返回全部（含打标），不受岗位过滤影响。"""
|
||||
tag_store = MemoryTagStore()
|
||||
tag = await tag_store.create_tag(
|
||||
{"id": "t1", "name": "理赔", "tag_type": "recommended_question"}
|
||||
)
|
||||
await tag_store.set_target_tags("recommended_question", "q2", [tag["id"]])
|
||||
|
||||
req = _request(_FakeQuestionStore(_rows()), tag_store)
|
||||
req.app.state.position_sync = _FakeSync(None)
|
||||
resp = await rq_router.list_recommended_questions(req, all=True)
|
||||
|
||||
assert [q.id for q in resp.questions] == ["q1", "q2"]
|
||||
@@ -77,6 +77,12 @@ models:
|
||||
temperature: 1.0
|
||||
supports_vision: true
|
||||
supports_thinking: true
|
||||
# SiliconFlow 混合推理模型关思考用「请求体顶层 enable_thinking: false」,
|
||||
# 不认 chat_template_kwargs / thinking.type 那两种嵌套形状。声明这个后,
|
||||
# 网页端「关闭思考」对本模型才生效(开放接口另由 force_disable_thinking 兜底)。
|
||||
when_thinking_disabled:
|
||||
extra_body:
|
||||
enable_thinking: false
|
||||
- name: Pro/MiniMaxAI/MiniMax-M2.5
|
||||
display_name: Pro/MiniMaxAI/MiniMax-M2.5
|
||||
use: langchain_openai:ChatOpenAI
|
||||
@@ -331,7 +337,7 @@ summarization:
|
||||
# ?taskId= / ?id=），随时改这里即可（Gateway mtime 热重载，前端无需重建）。
|
||||
task_deeplink:
|
||||
# 线上接口未通时用假数据；三个开关互相独立。
|
||||
test: true # 目的树查询走 mock.purpose_detail 假数据
|
||||
test: true # 目的树查询走 mock.purpose_detail 假数据；同时 rwfx 进入「3q 是否完成」预检走 mock.sq_report 假数据
|
||||
delete_test: true # 删除接口走 mock.delete 假数据（该接口尚未开发完）
|
||||
action_test: true # 行动→任务解析走 mock.action_detail 假数据（xdfx 行动id→任务id）
|
||||
purpose_detail_url: "https://ch1.b.uat.4.cn/consumer/taskAnalyseSearch/getTaskPurposeDetail"
|
||||
@@ -382,3 +388,15 @@ task_deeplink:
|
||||
actionName: ""
|
||||
actionStatus: 25
|
||||
narration: ""
|
||||
# test=true 时 rwfx 进入「3q 是否完成」预检（selectSqReport，外部不可用）走这份假数据。
|
||||
# 结构同真实 GET /consumer/cop/selectSqReport?taskId= 返回；前端判定 data 为空 / 每条
|
||||
# contentJson 都是 null/空 = 「3q 未完成」→ 弹框引导去 3qfx；否则视为已完成（正常，不弹框）。
|
||||
# 默认给一条有内容的数据 = 已完成（不打扰）。要测「3q 未完成」弹框：把下面 data 改成 []。
|
||||
sq_report:
|
||||
state: "200"
|
||||
msg: "操作成功！"
|
||||
data:
|
||||
- id: "sq-mock-1"
|
||||
taskId: 3338
|
||||
categoryType: "敌情关键事件 JSON"
|
||||
contentJson: "[{\"topic\":\"示例研判事件\",\"count\":1}]"
|
||||
Reference in new issue
Block a user