diff --git a/CLAUDE.md b/CLAUDE.md index b011329e..1ea3101b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -79,7 +79,7 @@ The frontend (`frontend-web/`) is a React 19 + Vite SPA: - **Routing**: React Router v7; top-level routes in `src/pages/PageRoutes.tsx`, workspace sub-routes in `src/pages/WorkspaceRoutes.tsx` - **Pages**: `ChatPage` (thread chat), `AgentChatPage` (custom agent chat), `AgentsPage`/`NewAgentPage` (agent management), `ScheduledTasksPage`/`ScheduledTaskRunDetailPage` (scheduled tasks), `LightAppManagementPage`/`LightAppListPage`/`LightAppIframeView` (轻应用中心 — register/list/embed mini-apps) -- **定时任务「JSON → HTML 模板填充」(模板数据转换器)**: a third scheduled-task execution mode. In the create/edit dialog (`ScheduledTasksPage.tsx`), selecting the built-in agent **「模板数据转换器」** (`execution_agent_name === "template-json-builder"`, `TEMPLATE_BUILDER_AGENT_ID`) reveals a 「HTML模板」 select (`useHtmlTemplates` → `GET /api/scheduled-tasks/html-templates`) + a **.json 上传** button (reads the file's text into the prompt box) and **hides** the 必须生成 HTML/Markdown switches (this mode always outputs HTML). The chosen template persists as `execution_context.template_config.template_id` (`TemplateFillConfig` in `core/scheduled-tasks/types.ts`). At run time the backend converts the source JSON (the task prompt) into that template's `DATA` structure, validates (retry-on-fail), fills the template, and stores an HTML page previewable via the existing run-detail HTML preview. The source JSON is **any** shape — the agent's `SOUL.md` maps it to the template's schema skeleton. See the backend CLAUDE.md「Template-fill mode」for the engine (`deerflow/runtime/scheduler/template_fill.py`), agent seed, and dispatch. +- **定时任务「JSON → HTML 模板填充」(模板数据转换器)**: a third scheduled-task execution mode. In the create/edit dialog (`ScheduledTasksPage.tsx`), selecting the built-in agent **「模板数据转换器」** (`execution_agent_name === "template-json-builder"`, `TEMPLATE_BUILDER_AGENT_ID`) reveals a 「HTML模板」 select (`useHtmlTemplates` → `GET /api/scheduled-tasks/html-templates`) + a **.json 上传** button (reads the file's text into the prompt box) and **hides** the 必须生成 HTML/Markdown switches (this mode always outputs HTML). The chosen template persists as `execution_context.template_config.template_id` (`TemplateFillConfig` in `core/scheduled-tasks/types.ts`). At run time the backend converts the source JSON (the task prompt) into that template's `DATA` structure, validates (retry-on-fail), fills the template, and stores an HTML page previewable via the existing run-detail HTML preview. The source can be **pure JSON, plain text, or text-with-embedded-JSON** — the backend pre-extracts any embedded JSON and the agent's `SOUL.md` maps it (or the plain text's semantics) to the template's schema skeleton. See the backend CLAUDE.md「Template-fill mode」for the engine (`deerflow/runtime/scheduler/template_fill.py`), agent seed, and dispatch. - **轻应用中心 (Light App Center)**: a sibling of 任务管理 in the sidebar with two pages — 应用管理 (admin registers `window_open` or `iframe` mini-apps with a route-param builder: login params resolved from `localStorage.userInfo`, theme params literal) and 应用列表 (card grid of `window_open` apps, `window.open` launch). `iframe` apps are mounted under a chosen parent menu (XK/PG/TY/综合管理) and **dynamically injected into the sidebar** by `core/page-layout/use-sidebar-menu.ts` (`useSidebarMenuItems`), opening in the `LightAppIframeView` host. Client + React Query hooks in `strategy-components/api/light-apps.ts`; URL builder/constants in `strategy-components/lib/light-app.ts`. Backend: `/api/light-apps` (global/shared, admin-gated writes). - **菜单管理 (Menu Management, admin-only)**: an **overlay** over the static sidebar menu — `MenuManagementPage` (`pages/MenuManagementPage.tsx`, route `/page/strategy/admin/menu-management`, opened from the admin-only 「菜单管理」 item in the **bottom-left settings dropdown** of `components/page-sidebar.tsx`). Admins rename / 停用·启用 / reorder (↑↓ or @dnd-kit drag) / re-parent (horizontal drag **or** the edit dialog's 「移动到」parent selector) any node across 一级/二级/三级, **including injected iframe light-app nodes** (`light-app-`). The static tree (`core/page-layout/sidebar-menu.ts`) stays the source of icons/routes; only per-node deltas (`label`/`disabled`/`parentId`/`sortOrder`) persist to the backend. Merge logic in `core/page-layout/menu-overrides.ts` (`applyMenuOverrides`, applied in `use-sidebar-menu.ts` after light-app injection, before role filter; disabling a node hides its whole subtree) — strict **3-level cap** (`wouldExceedDepth`). Client `strategy-components/api/menu-overrides.ts`. Backend: `/api/menu-overrides` (global/shared; open read, admin atomic replace-all). - **敏感词管理 (Sensitive Word Management)**: configures the **展示层脱敏映射** (原始词 → 替换词). **Gated on the username `lqq`** (not the admin role) — `getAccountDisplayName().toLowerCase() === "lqq"` (username = email `@` prefix, else `user_id`, same as the profile page); the dropdown entry, the route guard, and the page-level guard all use this check. `src/lib/desensitize.ts` ships a frozen seed map `SEED_DESENSITIZE_MAP` (the deployment seed) plus a live mutable `DESENSITIZE_MAP` (initialized from the seed so `desensitize()` works before any fetch). `DesensitizeWordsLoader` (mounted in `App.tsx` `AppProviders`) fetches `/api/sensitive-words` on startup and, if non-empty, calls `replaceDesensitizeMap(...)` to make the backend config the source of truth (empty/failed → keep seed). Page `SensitiveWordsPage.tsx` (route `/page/strategy/admin/sensitive-words`, opened from the 「敏感词管理」 item in the **bottom-left settings dropdown** of `components/page-sidebar.tsx`, present in **both** full + collapsed sidebar modes) lists/adds/edits/启停/deletes words and saves the **whole set atomically** (replace-all); a 「导入种子词」 button merges any missing `SEED_DESENSITIZE_MAP` entries into the editor. Client + hooks in `strategy-components/api/sensitive-words.ts` (`wordsToMap` folds enabled rows into the runtime map). Backend: `/api/sensitive-words` (global/shared; open read, write allowed for admin **or** the `lqq` account), mirroring 菜单管理's store shape. **侧边栏 + 导航栏 menu labels are desensitized** at the data source: `useSidebarMenuItems`/`useActiveThirdLevel` (`core/page-layout/use-sidebar-menu.ts`) map every node's `label` through a desensitizer built **directly from the `/api/sensitive-words` query** (`makeDesensitizer(wordsToMap(words))`, falling back to the global seed `desensitize` while empty) — this sidesteps the `replaceDesensitizeMap` effect-timing race. Only `label` is rewritten; `path`/`id`/`icon` are untouched so routing/highlight are unaffected. All nav surfaces (`page-sidebar.tsx`, `page-sidebar-dark.tsx`, `page-sidebar-v1.tsx`, `strategy-components/Header.tsx`, `PageRoutes.tsx`) consume this hook and are covered; `MenuManagementPage` uses the raw tree so admins still edit real labels. diff --git a/offline-backend-20260512/backend/CLAUDE.md b/offline-backend-20260512/backend/CLAUDE.md index 599ea614..f7685098 100644 --- a/offline-backend-20260512/backend/CLAUDE.md +++ b/offline-backend-20260512/backend/CLAUDE.md @@ -300,7 +300,7 @@ Two on-demand/automatic refresh paths layer on top: **Unified output switches (`require_html` + `require_markdown`).** The task form no longer splits "???? / HTML ??" into separate kinds. `execution_context` now carries two **independent** booleans that may both be on: `require_html` (???? HTML) and `require_markdown` (???? Markdown). `_task_requires_html()` returns true for `require_html` **or** legacy `task_kind == "html_page"` (so old tasks need no data migration), and `_run_task_attempt()` dispatches to `_run_html_page_attempt` when it is true, else the regular agent attempt. When **both** switches are on, `_gather_html_data()` returns `(text, files)` ? the gather agent run is additionally told to save a `.md` file ? and `_run_html_page_attempt` validates it (`MarkdownRequirementError` on failure) and surfaces it alongside the generated page, so the run delivers both artifacts. The single main `prompt` doubles as the HTML page requirement (`html_config.page_prompt`); there is no separate page-prompt field. Tests: `tests/test_scheduled_task_require_html.py` + the markdown-coexistence cases in `tests/test_html_page_tasks.py`. -**Template-fill mode (JSON → 固定 HTML 模板填充).** A third execution mode, distinct from the free-form `require_html` render pipeline: convert a user-supplied JSON into a **fixed** HTML template's data structure and fill it. Triggered when `execution_context.template_config.template_id` is set (chosen in the task form after selecting the built-in **「模板数据转换器」** agent `template-json-builder`). `_task_template_id()` reads it and `_run_task_attempt()` dispatches to `_run_template_fill_attempt` **before** the `require_html` check. Pipeline: take source JSON (the task `prompt`, pasted or uploaded .json) → load the selected agent's `SOUL.md`+model and run a **direct** `_invoke_conversion_model` call (system=SOUL, human=conversion prompt carrying the template's compact **schema skeleton** + source) → `extract_json_object` → `template_fill.validate_data` (top-level keys + container types) → **on failure feed the structure errors + `field_coverage_gaps` back and retry up to `_MAX_TEMPLATE_FILL_RETRIES`=2** → `template_fill.fill_template` replaces the template's `const DATA={…}` line (and refreshes `MANIFEST.系统.更新时间`) → archive `index.html` + persist a `scheduled_html_pages` row → return `format:"html"` (reusing the existing HTML preview at `/runs/{id}/html`). Any terminal failure raises `TemplateFillError` → run marked failed (no retry, message shown). **Pure engine** `deerflow/runtime/scheduler/template_fill.py` (harness, no `app.*`): the template registry (`TEMPLATES`: id=`no1` name=`no.1` 台湾滨海防卫大屏, and id=`no2` name=`no.2` 台湾飞行部队平台 with **`convertible=False`** — a 「只展示」 template that renders its existing HTML **as-is**, no model/validation/fill, until conversion is wired in later; `_run_template_fill_attempt` short-circuits to `read_template_text` for it, and `list_templates`/`is_convertible` expose the flag so the form hides the JSON input), `list_templates`/`reference_data`/`schema_skeleton_json`/`validate_data`/`field_coverage_gaps`/`fill_template` — **the template's own embedded `DATA` is the schema source of truth** (parsed at load, 1-sample-per-array skeleton fed to the LLM, top-level structure as the validation basis); template HTML files live under `runtime/scheduler/templates//`, add a `TemplateSpec` to register a no.2. The agent is seeded like the AI-writing ones: `app/gateway/routers/_template_builder_seed.py` + `_template_builder_seed_assets/template-json-builder/{config.yaml,SOUL.md}`, called from the lifespan before `_sync_legacy_agents` (which upserts it into the `agents` table as built-in/published so it shows in `/api/agents` → the task form's 执行智能体 dropdown). Template list API: `GET /api/scheduled-tasks/html-templates` → `{templates:[{id,name,description}]}` (open). Frontend: when the agent dropdown = `template-json-builder` the form hides the HTML/Markdown switches and shows a 「HTML模板」 select (`useHtmlTemplates`) + a .json upload (reads file text into the prompt box). Tests: `tests/test_template_fill.py`, `tests/test_template_fill_attempt.py`, `tests/test_template_builder_seed.py`. +**Template-fill mode (JSON → 固定 HTML 模板填充).** A third execution mode, distinct from the free-form `require_html` render pipeline: convert a user-supplied JSON into a **fixed** HTML template's data structure and fill it. Triggered when `execution_context.template_config.template_id` is set (chosen in the task form after selecting the built-in **「模板数据转换器」** agent `template-json-builder`). `_task_template_id()` reads it and `_run_task_attempt()` dispatches to `_run_template_fill_attempt` **before** the `require_html` check. Pipeline: take the source (the task `prompt`, pasted or uploaded .json) — which may be **pure JSON, plain text, or text-with-embedded-JSON** (`template_fill.extract_embedded_json` pre-extracts any embedded `{…}`/`[…]` with a string-aware balanced scan and surfaces it as a labeled block while keeping the full original text; plain text → no block, the model reads it semantically) → load the selected agent's `SOUL.md`+model and run a **direct** `_invoke_conversion_model` call (system=SOUL, human=conversion prompt carrying the template's compact **schema skeleton** + the recognized-JSON block + source) → `extract_json_object` → `template_fill.validate_data` (top-level keys + container types) → **on failure feed the structure errors + `field_coverage_gaps` back and retry up to `_MAX_TEMPLATE_FILL_RETRIES`=2** → `template_fill.fill_template` replaces the template's `const DATA={…}` line (and refreshes `MANIFEST.系统.更新时间`) → archive `index.html` + persist a `scheduled_html_pages` row → return `format:"html"` (reusing the existing HTML preview at `/runs/{id}/html`). Any terminal failure raises `TemplateFillError` → run marked failed (no retry, message shown). **Pure engine** `deerflow/runtime/scheduler/template_fill.py` (harness, no `app.*`): the template registry (`TEMPLATES`: id=`no1` name=`no.1` 台湾滨海防卫大屏, and id=`no2` name=`no.2` 台湾飞行部队平台 with **`convertible=False`** — a 「只展示」 template that renders its existing HTML **as-is**, no model/validation/fill, until conversion is wired in later; `_run_template_fill_attempt` short-circuits to `read_template_text` for it, and `list_templates`/`is_convertible` expose the flag so the form hides the JSON input), `list_templates`/`reference_data`/`schema_skeleton_json`/`validate_data`/`field_coverage_gaps`/`fill_template` — **the template's own embedded `DATA` is the schema source of truth** (parsed at load, 1-sample-per-array skeleton fed to the LLM, top-level structure as the validation basis); template HTML files live under `runtime/scheduler/templates//`, add a `TemplateSpec` to register a no.2. The agent is seeded like the AI-writing ones: `app/gateway/routers/_template_builder_seed.py` + `_template_builder_seed_assets/template-json-builder/{config.yaml,SOUL.md}`, called from the lifespan before `_sync_legacy_agents` (which upserts it into the `agents` table as built-in/published so it shows in `/api/agents` → the task form's 执行智能体 dropdown). Template list API: `GET /api/scheduled-tasks/html-templates` → `{templates:[{id,name,description}]}` (open). Frontend: when the agent dropdown = `template-json-builder` the form hides the HTML/Markdown switches and shows a 「HTML模板」 select (`useHtmlTemplates`) + a .json upload (reads file text into the prompt box). Tests: `tests/test_template_fill.py`, `tests/test_template_fill_attempt.py`, `tests/test_template_builder_seed.py`. ### Per-user Prompts & UI Preferences diff --git a/offline-backend-20260512/backend/app/gateway/routers/_template_builder_seed_assets/template-json-builder/SOUL.md b/offline-backend-20260512/backend/app/gateway/routers/_template_builder_seed_assets/template-json-builder/SOUL.md index f5624edf..f8af30de 100644 --- a/offline-backend-20260512/backend/app/gateway/routers/_template_builder_seed_assets/template-json-builder/SOUL.md +++ b/offline-backend-20260512/backend/app/gateway/routers/_template_builder_seed_assets/template-json-builder/SOUL.md @@ -7,7 +7,12 @@ 1. **目标结构骨架**:一份 JSON,展示了目标格式的**完整字段与类型**(每个数组只给 1 个 样例元素,但该元素已列出全部字段)。这是你必须严格对齐的契约。 -2. **源 JSON**:用户提供的原始数据,结构任意、字段名可能与目标不同、可能有多有少。 +2. **源数据**:用户提供的原始材料,形态可能是以下任意一种—— + - 一段 **JSON**(结构任意、字段名可能与目标不同、可能有多有少); + - 一段 **纯文本**(没有 JSON,需你理解其语义后据实提炼填充); + - **含 JSON 的文本**(正文里夹带 JSON——以其中的 JSON 为主、正文文字为补充说明)。 + 当系统已替你识别出内嵌的结构化数据时,会以「从源数据中识别到的结构化数据」单独给出, + **优先以它为准**,再结合原文里的其它说明。 3. (重试时) **上一轮的校验错误 / 缺失字段清单**:你必须据此修正后重新输出。 ## 转换规则 diff --git a/offline-backend-20260512/backend/packages/harness/deerflow/runtime/scheduler/service.py b/offline-backend-20260512/backend/packages/harness/deerflow/runtime/scheduler/service.py index 0880ed7a..eff446cd 100644 --- a/offline-backend-20260512/backend/packages/harness/deerflow/runtime/scheduler/service.py +++ b/offline-backend-20260512/backend/packages/harness/deerflow/runtime/scheduler/service.py @@ -3,6 +3,7 @@ from __future__ import annotations import asyncio +import json import logging import mimetypes import re @@ -1375,14 +1376,29 @@ class ScheduledTaskService: } @staticmethod - def _build_conversion_prompt(skeleton_json: str, source: str) -> str: - return ( - "请把下面的【源数据】转换为【目标结构】要求的 JSON。\n\n" - "【目标结构】(展示了完整字段与类型,其中数组只示意 1 个样例元素,实际请按源数据填充全部元素):\n" - "```json\n" + skeleton_json + "\n```\n\n" - "【源数据】:\n" + source + "\n\n" - "只输出一个 ```json 代码块,内含填充好的完整目标 JSON,不要任何解释文字。" - ) + def _build_conversion_prompt(skeleton_json: str, source: str, embedded_json: Any | None = None) -> str: + parts = [ + "请把下面的【源数据】转换为【目标结构】要求的 JSON。", + "源数据可能是一段 JSON、一段纯文本、或包含 JSON 的文本——" + "其中若有结构化 JSON 以 JSON 内容为准;若是纯文本请理解其语义后据实填充;" + "源里没有对应内容的字段按其类型留空(\"\" / [] / null),不要臆造。", + "", + "【目标结构】(展示了完整字段与类型,其中数组只示意 1 个样例元素,实际请按源数据填充全部元素):", + "```json\n" + skeleton_json + "\n```", + "", + ] + if embedded_json is not None: + # 已从源数据里识别出结构化 JSON,单独标注帮助(尤其弱)模型对齐。 + parts.append("【从源数据中识别到的结构化数据】(优先以此为准):") + parts.append("```json\n" + json.dumps(embedded_json, ensure_ascii=False) + "\n```") + parts.append("") + parts.append("【源数据原文】(含上面 JSON 之外的补充说明,一并参考):") + else: + parts.append("【源数据】:") + parts.append(source) + parts.append("") + parts.append("只输出一个 ```json 代码块,内含填充好的完整目标 JSON,不要任何解释文字。") + return "\n".join(parts) @staticmethod def _build_retry_prompt(base_prompt: str, errors: list[str], gaps: list[str]) -> str: @@ -1467,7 +1483,10 @@ class ScheduledTaskService: model_name = None skeleton = template_fill.schema_skeleton_json(template_id) - base_prompt = self._build_conversion_prompt(skeleton, source) + # 源数据可能是纯 JSON / 纯文本 / 含 JSON 的文本:先尽力抽出内嵌 JSON + # 单独标注,原文一并保留(纯文本则无内嵌 JSON,让模型按语义理解)。 + embedded_json = template_fill.extract_embedded_json(source) + base_prompt = self._build_conversion_prompt(skeleton, source, embedded_json) # 转换 → 校验 → 失败回喂重试。 last_errors: list[str] = [] diff --git a/offline-backend-20260512/backend/packages/harness/deerflow/runtime/scheduler/template_fill.py b/offline-backend-20260512/backend/packages/harness/deerflow/runtime/scheduler/template_fill.py index f1274dfb..025168fe 100644 --- a/offline-backend-20260512/backend/packages/harness/deerflow/runtime/scheduler/template_fill.py +++ b/offline-backend-20260512/backend/packages/harness/deerflow/runtime/scheduler/template_fill.py @@ -30,6 +30,73 @@ from functools import lru_cache from pathlib import Path from typing import Any +_FENCE_RE = re.compile(r"```(?:json)?\s*(.*?)```", re.DOTALL | re.IGNORECASE) + + +def _scan_balanced(text: str, open_ch: str, close_ch: str) -> Any | None: + """从 text 里扫第一个**平衡**的 ``open_ch…close_ch`` 块并 ``json.loads``。 + + 字符串感知(跳过引号内的括号/转义),失败返回 None。供从「含 JSON 的文本」里 + 把 JSON 段抠出来用。 + """ + start = text.find(open_ch) + if start == -1: + return None + depth = 0 + in_string = False + escape = False + for i in range(start, len(text)): + ch = text[i] + if in_string: + if escape: + escape = False + elif ch == "\\": + escape = True + elif ch == '"': + in_string = False + continue + if ch == '"': + in_string = True + elif ch == open_ch: + depth += 1 + elif ch == close_ch: + depth -= 1 + if depth == 0: + blob = text[start : i + 1] + try: + return json.loads(blob) + except (ValueError, TypeError): + return None + return None + + +def extract_embedded_json(text: str | None) -> Any | None: + """尽力从一段文本里抽出内嵌的 JSON(对象或数组)。 + + 覆盖三种用户输入:纯 JSON(直接解析)、含 JSON 的文本(抠出第一个平衡的 + ``{…}`` 或 ``[…]``)、纯文本(返回 ``None``)。也识别 ```json fence。 + """ + if not text or not isinstance(text, str): + return None + s = text.strip() + fence = _FENCE_RE.search(s) + if fence: + inner = fence.group(1).strip() + try: + return json.loads(inner) + except (ValueError, TypeError): + s = inner # fence 内不是完整 JSON,继续往下扫 + try: + return json.loads(s) + except (ValueError, TypeError): + pass + # 取最靠前出现的那种括号块(对象优先于数组,谁先出现谁优先)。 + brace = _scan_balanced(s, "{", "}") + bracket = _scan_balanced(s, "[", "]") + if brace is not None and bracket is not None: + return brace if s.find("{") <= s.find("[") else bracket + return brace if brace is not None else bracket + # 模板资源根目录(随 git 发布,与本模块同级 ``templates/`` 子目录)。 _TEMPLATES_ROOT = Path(__file__).parent / "templates"