feat: 敏感词管理页面（脱敏词后台化，限 lqq 账号）

- 后端 /api/sensitive-words：全局共享词表，开放读、admin 或 lqq 整体替换写 - 前端 desensitize 种子化：SEED_DESENSITIZE_MAP + 运行时 replaceDesensitizeMap - 启动加载后台配置回灌脱敏映射；管理页支持增删改启停 + 一键导入种子词 - 入口置于左下角设置下拉（全量/简洁两种侧边栏均有），仅用户名 lqq 可见 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 18:05:07 +08:00
1 parent 39d7dab3e2
commit 25e4e3dbc9
@@ -253,6 +253,7 @@ FastAPI application on port 8001 with health check at `GET /health`. Set `GATEWA
 | **Public Scheduled Tasks** (`/api/public/scheduled-tasks`) | **Unauthenticated** mirror for embedding. Adds `GET /runs/{rid}/html` and `GET /{task_id}/latest-html` for the `html_page` preview; the run result keeps `result.html` (only file paths are redacted). |
 | **Light Apps** (`/api/light-apps`) | Light-app registry (????? / ????), **global/shared** (no user scoping; `created_by`/`updated_by` audit-only). Two kinds: `window_open` (launched in a new tab from the ???? card grid) and `iframe` (embedded under a parent sidebar menu via `mount_location` and dynamically injected into the sidebar). Both carry `route_params` ? a list of `{key, paramType: login\|theme, source, customValue, themeValue}` descriptors the **frontend** resolves at launch time (login params read from `localStorage.userInfo`; theme params use a literal value) and appends to `url` as query string. `GET /` - list (filters `app_type`/`mount_location`/`enabled_only`/`search`); `GET /{app_id}` - read one (both open to any authenticated user); `POST /`, `PUT /{app_id}`, `DELETE /{app_id}` - **admin only**; iframe apps must carry a `mount_location` (400 otherwise); the unique `name` collides as HTTP 409. Backed by `deerflow.persistence.light_apps` (table `light_apps`, auto-created by `create_all`); store wired in `deps.py` as `app.state.light_app_store`. Tests: `tests/test_light_apps.py`. |
 | **Menu Overrides** (`/api/menu-overrides`) | Sidebar-menu overrides (菜单管理), **global/shared** (no user scoping; `updated_by` audit-only). The static menu tree (icons/routes/`adminOnly`) lives in the **frontend** code (`core/page-layout/sidebar-menu.ts`); iframe light-apps are injected at render time. This router persists a per-node **overlay** the frontend merges onto that tree: `parent_id` (re-parent, moves the whole subtree; NULL = top level), `sort_order` (reorder among siblings), `label` (rename; NULL keeps code label), `disabled` (停用 — hides the node **and its subtree**). A node with no row renders exactly as code defines it, so new code menus / newly-registered iframe apps appear automatically and are immediately configurable. `GET /` - list all overrides (open to any authenticated user); `PUT /` - **admin only**, atomically **replaces the whole set** (`{overrides: [...]}`); rejects re-parenting that would exceed **3 levels** or form a cycle (`_validate_depth`, HTTP 400). Backed by `deerflow.persistence.menu_overrides` (table `menu_overrides`, auto-created by `create_all`); store wired in `deps.py` as `app.state.menu_override_store`. Frontend: merge logic `core/page-layout/menu-overrides.ts` (`applyMenuOverrides` in `use-sidebar-menu.ts`), client `strategy-components/api/menu-overrides.ts`, admin page `pages/MenuManagementPage.tsx` (@dnd-kit drag-reorder/re-parent + 「移动到」parent selector + ↑↓ arrows + 改名/停用), entry in the bottom-left settings dropdown (`components/page-sidebar.tsx`, admin-only). Tests: `tests/test_menu_overrides.py`, `frontend-web/src/core/page-layout/menu-overrides.test.ts`. |
+| **Sensitive Words** (`/api/sensitive-words`) | Sensitive-word / desensitize map (敏感词管理 / 脱敏词), **global/shared** (no user scoping; `updated_by` audit-only). Each row maps an original `word` → `replacement`, with `enabled` (停用 parks a word without deleting) + `sort_order`. The **frontend** ships a seed map in code (`src/lib/desensitize.ts` `SEED_DESENSITIZE_MAP`) used as the default on a fresh deployment; once an admin saves, this table is the source of truth and the frontend loads it at startup (`DesensitizeWordsLoader` → `replaceDesensitizeMap`, **enabled** rows only). `GET /` - list all words (open to any authenticated user — every page desensitizes); `PUT /` - **admin OR the `lqq` account** (`_require_manager`: `system_role == "admin"` or email `@`-prefix `== "lqq"`), atomically **replaces the whole set** (`{words: [...]}`, de-duped by `word`, last wins). Backed by `deerflow.persistence.sensitive_words` (table `sensitive_words`, PK `word` `String(191)`, auto-created by `create_all`); store wired in `deps.py` as `app.state.sensitive_word_store`. Frontend: client `strategy-components/api/sensitive-words.ts`, page `pages/SensitiveWordsPage.tsx` (route `/page/strategy/admin/sensitive-words`, entry in the bottom-left settings dropdown — **gated on username `lqq`, not admin role**, present in both sidebar modes; 「导入种子词」 merges missing seed entries). Tests: `tests/test_sensitive_words.py`. |
 | **Knowledge** (`/api/knowledge`) | Product knowledge base, **global/shared** (no user isolation; writes record `created_by`/`updated_by` for audit). `POST /ingest/thread/{thread_id}` - sediment one thread into a note (reads messages from the checkpointer, `threads:read` owner-checked); `POST /ingest/threads/batch` - batch-sediment the caller's recent threads (`skip_existing` via `.manifest.json` dedup). **Defaults to `background=true`**: the slow per-thread extraction loop is deferred to a FastAPI background task and the endpoint returns `{queued:true, processed:N}` immediately (each thread's transcript is truncated to `max_input_chars` before the LLM call). Running the loop inline for a large batch outlives the nginx gateway timeout (504), which the frontend's `parseResponse` surfaces as a generic "Request failed"; `background=false` keeps the blocking path (full per-thread `created/skipped/failed` tally) for small batches/scripts. Shared loop: `_run_batch_ingest`. **Optional model selection**: both ingest endpoints accept `model_name` (overrides `knowledge.extract_model_name` for that capture; threaded through `KnowledgeService.capture_thread` ? `extract_thread_llm_multi`). **Model-error surfacing**: when the chosen model produces nothing usable (timeout / API error / non-JSON) capture still **falls back to rule-based extraction and lands a draft** (`llm_fallback:true`); the single-thread (non-background) response and the batch response (`llm_fallback` count) carry this so the UI tells the user "???????". The "??????" chat button stays background/fire-and-forget (passes `model_name`, no per-save error toast); the batch-import dialog is synchronous and shows the fallback count. `POST /ingest/file` - **import an uploaded file** (multipart: `file` + optional `title`/`tags`/`status`/`model_name`) into wiki note(s): the file is converted to Markdown (md/txt read directly, pdf/word/ppt/excel via `convert_file_to_markdown`), distilled by the same wiki-capture LLM pipeline as thread capture (`source_type="file"`), and falls back to a verbatim **draft** note when the model returns nothing usable (`llm_fallback:true`); `POST /ingest/search` - sediment search/tool results as a reference note; `POST /notes` - manual note; `GET /notes` (keyword/tag/source_type/status + pagination); `GET/PUT/DELETE /notes/{id}` (DELETE is soft = archive, `?hard=true` removes the vault file); `POST /search` - `mode=keyword\|vector\|hybrid` (vector?keyword fallback when no embedding endpoint); `GET /notes/{id}/versions` - edit history; `GET /duplicates` + `POST /merge` - dedup; `POST /reindex` - rebuild vector index; `GET /graph` - product graph data (notes + entities, graph-blacklist filtered); `GET/POST /graph/blacklist` + `DELETE /graph/blacklist/{id}` - graph node blacklist (labels = entity names / note titles, case-insensitive match; blacklisted nodes and their edges are dropped from `/graph` and the wiki-export artifacts; table `knowledge_graph_blacklist`, auto-created); `POST /export/graph` + `GET /export/files/{graph.json\|graph.html\|graph.graphml\|cypher.txt}` - obsidian-wiki `wiki-export` artifacts (self-contained `graph.html` for iframe); `POST /resolve` `{targets:[?]}` + `GET /notes/{id}/wikilinks` - resolve `[[wikilink]]` targets to a note id / vault page / missing (powers clickable wikilinks); `GET /vault-page?path=` - read a raw vault meta/hub Markdown page (no DB note). `GET/POST /folders` + `PUT/DELETE /folders/{id}` - user-managed directories (notes carry a `folder` path; `GET /notes?folder=` filters a dir + sub-dirs); `GET/POST /extract-templates` + `GET/PUT/DELETE /extract-templates/{id}` - editable extraction-prompt templates (thread/document ingest accept `template_id`; capture/PUT-note accept `folder`). Backed by the `deerflow.knowledge` module (see below). |

 Proxied through nginx: `/api/langgraph/*` ? LangGraph, all other `/api/*` ? Gateway.
@@ -63,6 +63,7 @@ from app.gateway.routers import (
    roundtable_jobs,
    runs,
    scheduled_tasks,
+    sensitive_words,
    skills,
    suggestions,
    system_settings,
@@ -1343,6 +1344,9 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
    # Sidebar-menu overrides (菜单管理) at /api/menu-overrides
    app.include_router(menu_overrides.router)

+    # Sensitive-word map (敏感词管理 / 脱敏词) at /api/sensitive-words
+    app.include_router(sensitive_words.router)
+
    # Standalone parallel multi-agent panel: fan-out streaming + saved tasks
    app.include_router(parallel_agents.router)
    app.include_router(parallel_agent_tasks.router)
@@ -243,6 +243,10 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:

        app.state.menu_override_store = make_menu_override_store(sf)

+        from deerflow.persistence.sensitive_words import make_sensitive_word_store
+
+        app.state.sensitive_word_store = make_sensitive_word_store(sf)
+
        from deerflow.persistence.parallel_agent_tasks import make_parallel_agent_task_store

        app.state.parallel_agent_task_store = make_parallel_agent_task_store(sf)
@@ -0,0 +1,106 @@
+"""Sensitive-word API (脱敏词 / 敏感词管理).
+
+The frontend ships a seed desensitize map in code (``src/lib/desensitize.ts``);
+it is the default on a fresh deployment and the source admins can import from.
+This router persists a **global / shared** word map that the frontend loads at
+startup and uses to build its desensitize regex. Each entry maps an original
+``word`` to its ``replacement``; ``enabled`` parks a word without deleting it.
+
+Reads are open to any authenticated user (every page desensitizes); the whole
+map is replaced atomically and only by an admin.
+
+Routes (prefix ``/api/sensitive-words``):
+    GET ""   list all words
+    PUT ""   replace the entire word set (admin)
+"""
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from fastapi import APIRouter, HTTPException, Request
+from pydantic import BaseModel, Field
+
+from app.gateway.deps import get_current_user, get_optional_user_from_request
+from deerflow.persistence.sensitive_words import SensitiveWordStore
+
+router = APIRouter(prefix="/api/sensitive-words", tags=["sensitive-words"])
+
+# 敏感词管理对应的指定账号：用户名（邮箱 @ 前缀）为该值的用户可写，admin 也可写。
+# 与前端 `getAccountDisplayName() === "lqq"` 的可见性门保持一致。
+_MANAGER_USERNAME = "lqq"
+
+
+def _username_of(user) -> str:
+    email = getattr(user, "email", None) or ""
+    return email.split("@")[0].strip().lower()
+
+
+# ── schemas ──────────────────────────────────────────────────────────────
+
+
+class SensitiveWord(BaseModel):
+    word: str = Field(..., min_length=1, max_length=191)
+    replacement: str = Field(default="", max_length=191)
+    enabled: bool = True
+    sort_order: int = 0
+
+
+class SensitiveWordResponse(SensitiveWord):
+    updated_by: str | None = None
+    updated_at: datetime | str | None = None
+
+
+class SensitiveWordListResponse(BaseModel):
+    words: list[SensitiveWordResponse]
+
+
+class SensitiveWordReplaceRequest(BaseModel):
+    words: list[SensitiveWord] = Field(default_factory=list)
+
+
+# ── helpers ──────────────────────────────────────────────────────────────
+
+
+def _get_store(request: Request) -> SensitiveWordStore:
+    store = getattr(request.app.state, "sensitive_word_store", None)
+    if store is None:
+        raise HTTPException(status_code=503, detail="Sensitive word store not available")
+    return store
+
+
+async def _require_manager(request: Request) -> None:
+    """Allow the designated manager username (``lqq``) or an admin.
+
+    Lets the call through when auth is disabled (no user resolved).
+    """
+    user = await get_optional_user_from_request(request)
+    if user is None:
+        return
+    if getattr(user, "system_role", None) == "admin":
+        return
+    if _username_of(user) == _MANAGER_USERNAME:
+        return
+    raise HTTPException(status_code=403, detail="敏感词管理仅限指定账号")
+
+
+# ── routes ───────────────────────────────────────────────────────────────
+
+
+@router.get("", response_model=SensitiveWordListResponse)
+async def list_sensitive_words(request: Request) -> SensitiveWordListResponse:
+    store = _get_store(request)
+    rows = await store.list_words()
+    return SensitiveWordListResponse(words=[SensitiveWordResponse(**r) for r in rows])
+
+
+@router.put("", response_model=SensitiveWordListResponse)
+async def replace_sensitive_words(request: Request, body: SensitiveWordReplaceRequest) -> SensitiveWordListResponse:
+    await _require_manager(request)
+    store = _get_store(request)
+    user_id = await get_current_user(request)
+    rows = await store.replace_all(
+        [w.model_dump() for w in body.words],
+        updated_by=user_id,
+    )
+    return SensitiveWordListResponse(words=[SensitiveWordResponse(**r) for r in rows])
@@ -55,6 +55,7 @@ from deerflow.persistence.roundtable_drafts.model import (
 )
 from deerflow.persistence.roundtable_jobs.model import RoundtableJobRow
 from deerflow.persistence.run.model import RunRow
+from deerflow.persistence.sensitive_words.model import SensitiveWordRow
 from deerflow.persistence.scheduled_tasks.model import (
    ScheduledTaskDeliveryProfileRow,
    ScheduledTaskRow,
@@ -112,6 +113,7 @@ __all__ = [
    "ScheduledTaskRunRow",
    "ScheduledTaskSubscriptionRow",
    "SchedulerThreadRow",
+    "SensitiveWordRow",
    "SkillRow",
    "SkillDisplayAdapterRow",
    "SkillFavoriteRow",
@@ -0,0 +1,22 @@
+"""Sensitive-word (脱敏词 / 敏感词管理) persistence."""
+
+from __future__ import annotations
+
+from deerflow.persistence.sensitive_words.base import SensitiveWordStore
+from deerflow.persistence.sensitive_words.memory import MemorySensitiveWordStore
+from deerflow.persistence.sensitive_words.model import SensitiveWordRow
+from deerflow.persistence.sensitive_words.sql import SensitiveWordRepository
+
+__all__ = [
+    "MemorySensitiveWordStore",
+    "SensitiveWordRepository",
+    "SensitiveWordRow",
+    "SensitiveWordStore",
+    "make_sensitive_word_store",
+]
+
+
+def make_sensitive_word_store(session_factory):
+    if session_factory is None:
+        return MemorySensitiveWordStore()
+    return SensitiveWordRepository(session_factory)
@@ -0,0 +1,33 @@
+"""Abstract sensitive-word (脱敏词) persistence API.
+
+Global / shared store: no ``user_id`` scoping; every account desensitizes with
+the same word map. Reads are open to any caller (the frontend needs the map to
+render desensitized text); the router gates writes behind admin. ``updated_by``
+is audit-only.
+
+The whole word map is treated as one document: the management page computes the
+full desired set and replaces every row atomically via :meth:`replace_all`, so
+the stored set never drifts out of sync with what the admin sees. This mirrors
+the menu-override store.
+"""
+
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from typing import Any
+
+
+class SensitiveWordStore(ABC):
+    @abstractmethod
+    async def list_words(self) -> list[dict[str, Any]]:
+        """Return every stored word row, ordered by ``sort_order``."""
+        raise NotImplementedError
+
+    @abstractmethod
+    async def replace_all(self, rows: list[dict[str, Any]], *, updated_by: str | None = None) -> list[dict[str, Any]]:
+        """Atomically replace the entire word set with ``rows``.
+
+        Each row is ``{word, replacement, enabled, sort_order}``. Returns the
+        persisted rows.
+        """
+        raise NotImplementedError
@@ -0,0 +1,36 @@
+"""In-memory sensitive-word store (used when no DB is configured / tests)."""
+
+from __future__ import annotations
+
+from datetime import UTC, datetime
+from typing import Any
+
+from deerflow.persistence.sensitive_words.base import SensitiveWordStore
+
+
+def _normalize(data: dict[str, Any], updated_by: str | None) -> dict[str, Any]:
+    return {
+        "word": str(data["word"]),
+        "replacement": str(data.get("replacement") or ""),
+        "enabled": bool(data.get("enabled", True)),
+        "sort_order": int(data.get("sort_order", 0)),
+        "updated_by": updated_by,
+        "updated_at": datetime.now(UTC).isoformat(),
+    }
+
+
+class MemorySensitiveWordStore(SensitiveWordStore):
+    def __init__(self) -> None:
+        self._rows: dict[str, dict[str, Any]] = {}
+
+    async def list_words(self) -> list[dict[str, Any]]:
+        return [dict(r) for r in sorted(self._rows.values(), key=lambda r: (r.get("sort_order", 0), r.get("word", "")))]
+
+    async def replace_all(self, rows: list[dict[str, Any]], *, updated_by: str | None = None) -> list[dict[str, Any]]:
+        self._rows = {}
+        for r in rows:
+            word = r.get("word")
+            if word in (None, ""):
+                continue
+            self._rows[str(word)] = _normalize(r, updated_by)
+        return await self.list_words()
@@ -0,0 +1,38 @@
+"""ORM model for sensitive words (脱敏词 / 敏感词管理).
+
+Global / shared — there is no ``user_id`` scoping; every account desensitizes
+with the same map. The frontend ships a seed map in code (used as the default on
+a fresh deployment); once an admin saves here, this table is the source of truth.
+
+Each row maps an original term (``word``, the primary key) to its desensitized
+replacement (``replacement``). ``enabled`` lets an admin park a word without
+deleting it; ``sort_order`` controls display order in the management page. The
+desensitize regex is rebuilt from the **enabled** rows. ``updated_by`` is
+audit-only.
+
+``word`` is the primary key at ``String(191)`` so the utf8mb4 index stays within
+the 767-byte legacy limit (191 × 4 = 764) — sensitive terms are never that long.
+"""
+
+from __future__ import annotations
+
+from datetime import UTC, datetime
+
+from sqlalchemy import Boolean, Integer, String
+from sqlalchemy.orm import Mapped, mapped_column
+
+from deerflow.persistence.base import Base
+from deerflow.persistence.types import BeijingDateTime
+
+
+class SensitiveWordRow(Base):
+    __tablename__ = "sensitive_words"
+
+    word: Mapped[str] = mapped_column(String(191), primary_key=True)
+    replacement: Mapped[str] = mapped_column(String(191), default="")
+    enabled: Mapped[bool] = mapped_column(Boolean, default=True)
+    sort_order: Mapped[int] = mapped_column(Integer, default=0)
+    updated_by: Mapped[str | None] = mapped_column(String(64))
+    updated_at: Mapped[datetime] = mapped_column(
+        BeijingDateTime(), default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC)
+    )
@@ -0,0 +1,57 @@
+"""SQLAlchemy-backed sensitive-word store (global / shared)."""
+
+from __future__ import annotations
+
+from datetime import datetime
+from typing import Any
+
+from sqlalchemy import delete, select
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from deerflow.persistence.sensitive_words.base import SensitiveWordStore
+from deerflow.persistence.sensitive_words.model import SensitiveWordRow
+
+
+def _to_dict(row: SensitiveWordRow) -> dict[str, Any]:
+    data = row.to_dict()
+    value = data.get("updated_at")
+    if isinstance(value, datetime):
+        data["updated_at"] = value.isoformat()
+    return data
+
+
+def _row_from(data: dict[str, Any], updated_by: str | None) -> SensitiveWordRow:
+    return SensitiveWordRow(
+        word=str(data["word"]),
+        replacement=str(data.get("replacement") or ""),
+        enabled=bool(data.get("enabled", True)),
+        sort_order=int(data.get("sort_order", 0)),
+        updated_by=updated_by,
+    )
+
+
+class SensitiveWordRepository(SensitiveWordStore):
+    def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
+        self._sf = session_factory
+
+    async def list_words(self) -> list[dict[str, Any]]:
+        stmt = select(SensitiveWordRow).order_by(SensitiveWordRow.sort_order.asc(), SensitiveWordRow.word.asc())
+        async with self._sf() as session:
+            result = await session.execute(stmt)
+            return [_to_dict(row) for row in result.scalars()]
+
+    async def replace_all(self, rows: list[dict[str, Any]], *, updated_by: str | None = None) -> list[dict[str, Any]]:
+        # De-dupe by word (last wins) so a malformed payload can't violate the PK.
+        by_word: dict[str, dict[str, Any]] = {}
+        for r in rows:
+            word = r.get("word")
+            if word not in (None, ""):
+                by_word[str(word)] = r
+        async with self._sf() as session:
+            await session.execute(delete(SensitiveWordRow))
+            new_rows = [_row_from(r, updated_by) for r in by_word.values()]
+            session.add_all(new_rows)
+            await session.commit()
+            for row in new_rows:
+                await session.refresh(row)
+            return [_to_dict(row) for row in new_rows]