feat(定时任务): 模板转换支持纯文本/含JSON文本/纯JSON三种源输入

- template_fill.extract_embedded_json：字符串感知平衡扫描，从含 JSON 的文本里抠出内嵌 {…}/[…]（纯文本返回 None） - 调度器转换 prompt 预识别内嵌 JSON，单独标注「识别到的结构化数据」并保留原文；纯文本则让模型按语义理解；conversion prompt 重写覆盖三种形态 - 内置 agent SOUL 说明三种源数据形态 - 文档同步 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 20:28:05 +08:00
1 parent 8b022bffc5
commit 00181ebde8
@@ -79,7 +79,7 @@ The frontend (`frontend-web/`) is a React 19 + Vite SPA:

 - **Routing**: React Router v7; top-level routes in `src/pages/PageRoutes.tsx`, workspace sub-routes in `src/pages/WorkspaceRoutes.tsx`
 - **Pages**: `ChatPage` (thread chat), `AgentChatPage` (custom agent chat), `AgentsPage`/`NewAgentPage` (agent management), `ScheduledTasksPage`/`ScheduledTaskRunDetailPage` (scheduled tasks), `LightAppManagementPage`/`LightAppListPage`/`LightAppIframeView` (轻应用中心 — register/list/embed mini-apps)
- **定时任务「JSON → HTML 模板填充」(模板数据转换器)**: a third scheduled-task execution mode. In the create/edit dialog (`ScheduledTasksPage.tsx`), selecting the built-in agent **「模板数据转换器」** (`execution_agent_name === "template-json-builder"`, `TEMPLATE_BUILDER_AGENT_ID`) reveals a 「HTML模板」 select (`useHtmlTemplates` → `GET /api/scheduled-tasks/html-templates`) + a **.json 上传** button (reads the file's text into the prompt box) and **hides** the 必须生成 HTML/Markdown switches (this mode always outputs HTML). The chosen template persists as `execution_context.template_config.template_id` (`TemplateFillConfig` in `core/scheduled-tasks/types.ts`). At run time the backend converts the source JSON (the task prompt) into that template's `DATA` structure, validates (retry-on-fail), fills the template, and stores an HTML page previewable via the existing run-detail HTML preview. The source JSON is **any** shape — the agent's `SOUL.md` maps it to the template's schema skeleton. See the backend CLAUDE.md「Template-fill mode」for the engine (`deerflow/runtime/scheduler/template_fill.py`), agent seed, and dispatch.
+- **定时任务「JSON → HTML 模板填充」(模板数据转换器)**: a third scheduled-task execution mode. In the create/edit dialog (`ScheduledTasksPage.tsx`), selecting the built-in agent **「模板数据转换器」** (`execution_agent_name === "template-json-builder"`, `TEMPLATE_BUILDER_AGENT_ID`) reveals a 「HTML模板」 select (`useHtmlTemplates` → `GET /api/scheduled-tasks/html-templates`) + a **.json 上传** button (reads the file's text into the prompt box) and **hides** the 必须生成 HTML/Markdown switches (this mode always outputs HTML). The chosen template persists as `execution_context.template_config.template_id` (`TemplateFillConfig` in `core/scheduled-tasks/types.ts`). At run time the backend converts the source JSON (the task prompt) into that template's `DATA` structure, validates (retry-on-fail), fills the template, and stores an HTML page previewable via the existing run-detail HTML preview. The source can be **pure JSON, plain text, or text-with-embedded-JSON** — the backend pre-extracts any embedded JSON and the agent's `SOUL.md` maps it (or the plain text's semantics) to the template's schema skeleton. See the backend CLAUDE.md「Template-fill mode」for the engine (`deerflow/runtime/scheduler/template_fill.py`), agent seed, and dispatch.
 - **轻应用中心 (Light App Center)**: a sibling of 任务管理 in the sidebar with two pages — 应用管理 (admin registers `window_open` or `iframe` mini-apps with a route-param builder: login params resolved from `localStorage.userInfo`, theme params literal) and 应用列表 (card grid of `window_open` apps, `window.open` launch). `iframe` apps are mounted under a chosen parent menu (XK/PG/TY/综合管理) and **dynamically injected into the sidebar** by `core/page-layout/use-sidebar-menu.ts` (`useSidebarMenuItems`), opening in the `LightAppIframeView` host. Client + React Query hooks in `strategy-components/api/light-apps.ts`; URL builder/constants in `strategy-components/lib/light-app.ts`. Backend: `/api/light-apps` (global/shared, admin-gated writes).
 - **菜单管理 (Menu Management, admin-only)**: an **overlay** over the static sidebar menu — `MenuManagementPage` (`pages/MenuManagementPage.tsx`, route `/page/strategy/admin/menu-management`, opened from the admin-only 「菜单管理」 item in the **bottom-left settings dropdown** of `components/page-sidebar.tsx`). Admins rename / 停用·启用 / reorder (↑↓ or @dnd-kit drag) / re-parent (horizontal drag **or** the edit dialog's 「移动到」parent selector) any node across 一级/二级/三级, **including injected iframe light-app nodes** (`light-app-<appId>`). The static tree (`core/page-layout/sidebar-menu.ts`) stays the source of icons/routes; only per-node deltas (`label`/`disabled`/`parentId`/`sortOrder`) persist to the backend. Merge logic in `core/page-layout/menu-overrides.ts` (`applyMenuOverrides`, applied in `use-sidebar-menu.ts` after light-app injection, before role filter; disabling a node hides its whole subtree) — strict **3-level cap** (`wouldExceedDepth`). Client `strategy-components/api/menu-overrides.ts`. Backend: `/api/menu-overrides` (global/shared; open read, admin atomic replace-all).
 - **敏感词管理 (Sensitive Word Management)**: configures the **展示层脱敏映射** (原始词 → 替换词). **Gated on the username `lqq`** (not the admin role) — `getAccountDisplayName().toLowerCase() === "lqq"` (username = email `@` prefix, else `user_id`, same as the profile page); the dropdown entry, the route guard, and the page-level guard all use this check. `src/lib/desensitize.ts` ships a frozen seed map `SEED_DESENSITIZE_MAP` (the deployment seed) plus a live mutable `DESENSITIZE_MAP` (initialized from the seed so `desensitize()` works before any fetch). `DesensitizeWordsLoader` (mounted in `App.tsx` `AppProviders`) fetches `/api/sensitive-words` on startup and, if non-empty, calls `replaceDesensitizeMap(...)` to make the backend config the source of truth (empty/failed → keep seed). Page `SensitiveWordsPage.tsx` (route `/page/strategy/admin/sensitive-words`, opened from the 「敏感词管理」 item in the **bottom-left settings dropdown** of `components/page-sidebar.tsx`, present in **both** full + collapsed sidebar modes) lists/adds/edits/启停/deletes words and saves the **whole set atomically** (replace-all); a 「导入种子词」 button merges any missing `SEED_DESENSITIZE_MAP` entries into the editor. Client + hooks in `strategy-components/api/sensitive-words.ts` (`wordsToMap` folds enabled rows into the runtime map). Backend: `/api/sensitive-words` (global/shared; open read, write allowed for admin **or** the `lqq` account), mirroring 菜单管理's store shape. **侧边栏 + 导航栏 menu labels are desensitized** at the data source: `useSidebarMenuItems`/`useActiveThirdLevel` (`core/page-layout/use-sidebar-menu.ts`) map every node's `label` through a desensitizer built **directly from the `/api/sensitive-words` query** (`makeDesensitizer(wordsToMap(words))`, falling back to the global seed `desensitize` while empty) — this sidesteps the `replaceDesensitizeMap` effect-timing race. Only `label` is rewritten; `path`/`id`/`icon` are untouched so routing/highlight are unaffected. All nav surfaces (`page-sidebar.tsx`, `page-sidebar-dark.tsx`, `page-sidebar-v1.tsx`, `strategy-components/Header.tsx`, `PageRoutes.tsx`) consume this hook and are covered; `MenuManagementPage` uses the raw tree so admins still edit real labels.
@@ -300,7 +300,7 @@ Two on-demand/automatic refresh paths layer on top:

 **Unified output switches (`require_html` + `require_markdown`).** The task form no longer splits "???? / HTML ??" into separate kinds. `execution_context` now carries two **independent** booleans that may both be on: `require_html` (???? HTML) and `require_markdown` (???? Markdown). `_task_requires_html()` returns true for `require_html` **or** legacy `task_kind == "html_page"` (so old tasks need no data migration), and `_run_task_attempt()` dispatches to `_run_html_page_attempt` when it is true, else the regular agent attempt. When **both** switches are on, `_gather_html_data()` returns `(text, files)` ? the gather agent run is additionally told to save a `.md` file ? and `_run_html_page_attempt` validates it (`MarkdownRequirementError` on failure) and surfaces it alongside the generated page, so the run delivers both artifacts. The single main `prompt` doubles as the HTML page requirement (`html_config.page_prompt`); there is no separate page-prompt field. Tests: `tests/test_scheduled_task_require_html.py` + the markdown-coexistence cases in `tests/test_html_page_tasks.py`.

-**Template-fill mode (JSON → 固定 HTML 模板填充).** A third execution mode, distinct from the free-form `require_html` render pipeline: convert a user-supplied JSON into a **fixed** HTML template's data structure and fill it. Triggered when `execution_context.template_config.template_id` is set (chosen in the task form after selecting the built-in **「模板数据转换器」** agent `template-json-builder`). `_task_template_id()` reads it and `_run_task_attempt()` dispatches to `_run_template_fill_attempt` **before** the `require_html` check. Pipeline: take source JSON (the task `prompt`, pasted or uploaded .json) → load the selected agent's `SOUL.md`+model and run a **direct** `_invoke_conversion_model` call (system=SOUL, human=conversion prompt carrying the template's compact **schema skeleton** + source) → `extract_json_object` → `template_fill.validate_data` (top-level keys + container types) → **on failure feed the structure errors + `field_coverage_gaps` back and retry up to `_MAX_TEMPLATE_FILL_RETRIES`=2** → `template_fill.fill_template` replaces the template's `const DATA={…}` line (and refreshes `MANIFEST.系统.更新时间`) → archive `index.html` + persist a `scheduled_html_pages` row → return `format:"html"` (reusing the existing HTML preview at `/runs/{id}/html`). Any terminal failure raises `TemplateFillError` → run marked failed (no retry, message shown). **Pure engine** `deerflow/runtime/scheduler/template_fill.py` (harness, no `app.*`): the template registry (`TEMPLATES`: id=`no1` name=`no.1` 台湾滨海防卫大屏, and id=`no2` name=`no.2` 台湾飞行部队平台 with **`convertible=False`** — a 「只展示」 template that renders its existing HTML **as-is**, no model/validation/fill, until conversion is wired in later; `_run_template_fill_attempt` short-circuits to `read_template_text` for it, and `list_templates`/`is_convertible` expose the flag so the form hides the JSON input), `list_templates`/`reference_data`/`schema_skeleton_json`/`validate_data`/`field_coverage_gaps`/`fill_template` — **the template's own embedded `DATA` is the schema source of truth** (parsed at load, 1-sample-per-array skeleton fed to the LLM, top-level structure as the validation basis); template HTML files live under `runtime/scheduler/templates/<id>/`, add a `TemplateSpec` to register a no.2. The agent is seeded like the AI-writing ones: `app/gateway/routers/_template_builder_seed.py` + `_template_builder_seed_assets/template-json-builder/{config.yaml,SOUL.md}`, called from the lifespan before `_sync_legacy_agents` (which upserts it into the `agents` table as built-in/published so it shows in `/api/agents` → the task form's 执行智能体 dropdown). Template list API: `GET /api/scheduled-tasks/html-templates` → `{templates:[{id,name,description}]}` (open). Frontend: when the agent dropdown = `template-json-builder` the form hides the HTML/Markdown switches and shows a 「HTML模板」 select (`useHtmlTemplates`) + a .json upload (reads file text into the prompt box). Tests: `tests/test_template_fill.py`, `tests/test_template_fill_attempt.py`, `tests/test_template_builder_seed.py`.
+**Template-fill mode (JSON → 固定 HTML 模板填充).** A third execution mode, distinct from the free-form `require_html` render pipeline: convert a user-supplied JSON into a **fixed** HTML template's data structure and fill it. Triggered when `execution_context.template_config.template_id` is set (chosen in the task form after selecting the built-in **「模板数据转换器」** agent `template-json-builder`). `_task_template_id()` reads it and `_run_task_attempt()` dispatches to `_run_template_fill_attempt` **before** the `require_html` check. Pipeline: take the source (the task `prompt`, pasted or uploaded .json) — which may be **pure JSON, plain text, or text-with-embedded-JSON** (`template_fill.extract_embedded_json` pre-extracts any embedded `{…}`/`[…]` with a string-aware balanced scan and surfaces it as a labeled block while keeping the full original text; plain text → no block, the model reads it semantically) → load the selected agent's `SOUL.md`+model and run a **direct** `_invoke_conversion_model` call (system=SOUL, human=conversion prompt carrying the template's compact **schema skeleton** + the recognized-JSON block + source) → `extract_json_object` → `template_fill.validate_data` (top-level keys + container types) → **on failure feed the structure errors + `field_coverage_gaps` back and retry up to `_MAX_TEMPLATE_FILL_RETRIES`=2** → `template_fill.fill_template` replaces the template's `const DATA={…}` line (and refreshes `MANIFEST.系统.更新时间`) → archive `index.html` + persist a `scheduled_html_pages` row → return `format:"html"` (reusing the existing HTML preview at `/runs/{id}/html`). Any terminal failure raises `TemplateFillError` → run marked failed (no retry, message shown). **Pure engine** `deerflow/runtime/scheduler/template_fill.py` (harness, no `app.*`): the template registry (`TEMPLATES`: id=`no1` name=`no.1` 台湾滨海防卫大屏, and id=`no2` name=`no.2` 台湾飞行部队平台 with **`convertible=False`** — a 「只展示」 template that renders its existing HTML **as-is**, no model/validation/fill, until conversion is wired in later; `_run_template_fill_attempt` short-circuits to `read_template_text` for it, and `list_templates`/`is_convertible` expose the flag so the form hides the JSON input), `list_templates`/`reference_data`/`schema_skeleton_json`/`validate_data`/`field_coverage_gaps`/`fill_template` — **the template's own embedded `DATA` is the schema source of truth** (parsed at load, 1-sample-per-array skeleton fed to the LLM, top-level structure as the validation basis); template HTML files live under `runtime/scheduler/templates/<id>/`, add a `TemplateSpec` to register a no.2. The agent is seeded like the AI-writing ones: `app/gateway/routers/_template_builder_seed.py` + `_template_builder_seed_assets/template-json-builder/{config.yaml,SOUL.md}`, called from the lifespan before `_sync_legacy_agents` (which upserts it into the `agents` table as built-in/published so it shows in `/api/agents` → the task form's 执行智能体 dropdown). Template list API: `GET /api/scheduled-tasks/html-templates` → `{templates:[{id,name,description}]}` (open). Frontend: when the agent dropdown = `template-json-builder` the form hides the HTML/Markdown switches and shows a 「HTML模板」 select (`useHtmlTemplates`) + a .json upload (reads file text into the prompt box). Tests: `tests/test_template_fill.py`, `tests/test_template_fill_attempt.py`, `tests/test_template_builder_seed.py`.

 ### Per-user Prompts & UI Preferences

@@ -7,7 +7,12 @@

 1. **目标结构骨架**：一份 JSON,展示了目标格式的**完整字段与类型**(每个数组只给 1 个
   样例元素,但该元素已列出全部字段)。这是你必须严格对齐的契约。
-2. **源 JSON**：用户提供的原始数据,结构任意、字段名可能与目标不同、可能有多有少。
+2. **源数据**：用户提供的原始材料,形态可能是以下任意一种——
+   - 一段 **JSON**(结构任意、字段名可能与目标不同、可能有多有少);
+   - 一段 **纯文本**(没有 JSON,需你理解其语义后据实提炼填充);
+   - **含 JSON 的文本**(正文里夹带 JSON——以其中的 JSON 为主、正文文字为补充说明)。
+   当系统已替你识别出内嵌的结构化数据时,会以「从源数据中识别到的结构化数据」单独给出,
+   **优先以它为准**,再结合原文里的其它说明。
 3. (重试时) **上一轮的校验错误 / 缺失字段清单**：你必须据此修正后重新输出。

 ## 转换规则
@@ -3,6 +3,7 @@
 from __future__ import annotations

 import asyncio
+import json
 import logging
 import mimetypes
 import re
@@ -1375,14 +1376,29 @@ class ScheduledTaskService:
        }

    @staticmethod
-    def _build_conversion_prompt(skeleton_json: str, source: str) -> str:
-        return (
-            "请把下面的【源数据】转换为【目标结构】要求的 JSON。\n\n"
-            "【目标结构】(展示了完整字段与类型,其中数组只示意 1 个样例元素,实际请按源数据填充全部元素):\n"
-            "```json\n" + skeleton_json + "\n```\n\n"
-            "【源数据】:\n" + source + "\n\n"
-            "只输出一个 ```json 代码块,内含填充好的完整目标 JSON,不要任何解释文字。"
-        )
+    def _build_conversion_prompt(skeleton_json: str, source: str, embedded_json: Any | None = None) -> str:
+        parts = [
+            "请把下面的【源数据】转换为【目标结构】要求的 JSON。",
+            "源数据可能是一段 JSON、一段纯文本、或包含 JSON 的文本——"
+            "其中若有结构化 JSON 以 JSON 内容为准；若是纯文本请理解其语义后据实填充；"
+            "源里没有对应内容的字段按其类型留空（\"\" / [] / null），不要臆造。",
+            "",
+            "【目标结构】(展示了完整字段与类型,其中数组只示意 1 个样例元素,实际请按源数据填充全部元素):",
+            "```json\n" + skeleton_json + "\n```",
+            "",
+        ]
+        if embedded_json is not None:
+            # 已从源数据里识别出结构化 JSON,单独标注帮助(尤其弱)模型对齐。
+            parts.append("【从源数据中识别到的结构化数据】(优先以此为准):")
+            parts.append("```json\n" + json.dumps(embedded_json, ensure_ascii=False) + "\n```")
+            parts.append("")
+            parts.append("【源数据原文】(含上面 JSON 之外的补充说明,一并参考):")
+        else:
+            parts.append("【源数据】:")
+        parts.append(source)
+        parts.append("")
+        parts.append("只输出一个 ```json 代码块,内含填充好的完整目标 JSON,不要任何解释文字。")
+        return "\n".join(parts)

    @staticmethod
    def _build_retry_prompt(base_prompt: str, errors: list[str], gaps: list[str]) -> str:
@@ -1467,7 +1483,10 @@ class ScheduledTaskService:
                    model_name = None

            skeleton = template_fill.schema_skeleton_json(template_id)
-            base_prompt = self._build_conversion_prompt(skeleton, source)
+            # 源数据可能是纯 JSON / 纯文本 / 含 JSON 的文本:先尽力抽出内嵌 JSON
+            # 单独标注,原文一并保留(纯文本则无内嵌 JSON,让模型按语义理解)。
+            embedded_json = template_fill.extract_embedded_json(source)
+            base_prompt = self._build_conversion_prompt(skeleton, source, embedded_json)

            # 转换 → 校验 → 失败回喂重试。
            last_errors: list[str] = []
@@ -30,6 +30,73 @@ from functools import lru_cache
 from pathlib import Path
 from typing import Any

+_FENCE_RE = re.compile(r"```(?:json)?\s*(.*?)```", re.DOTALL | re.IGNORECASE)
+
+
+def _scan_balanced(text: str, open_ch: str, close_ch: str) -> Any | None:
+    """从 text 里扫第一个**平衡**的 ``open_ch…close_ch`` 块并 ``json.loads``。
+
+    字符串感知(跳过引号内的括号/转义),失败返回 None。供从「含 JSON 的文本」里
+    把 JSON 段抠出来用。
+    """
+    start = text.find(open_ch)
+    if start == -1:
+        return None
+    depth = 0
+    in_string = False
+    escape = False
+    for i in range(start, len(text)):
+        ch = text[i]
+        if in_string:
+            if escape:
+                escape = False
+            elif ch == "\\":
+                escape = True
+            elif ch == '"':
+                in_string = False
+            continue
+        if ch == '"':
+            in_string = True
+        elif ch == open_ch:
+            depth += 1
+        elif ch == close_ch:
+            depth -= 1
+            if depth == 0:
+                blob = text[start : i + 1]
+                try:
+                    return json.loads(blob)
+                except (ValueError, TypeError):
+                    return None
+    return None
+
+
+def extract_embedded_json(text: str | None) -> Any | None:
+    """尽力从一段文本里抽出内嵌的 JSON(对象或数组)。
+
+    覆盖三种用户输入:纯 JSON(直接解析)、含 JSON 的文本(抠出第一个平衡的
+    ``{…}`` 或 ``[…]``)、纯文本(返回 ``None``)。也识别 ```json fence。
+    """
+    if not text or not isinstance(text, str):
+        return None
+    s = text.strip()
+    fence = _FENCE_RE.search(s)
+    if fence:
+        inner = fence.group(1).strip()
+        try:
+            return json.loads(inner)
+        except (ValueError, TypeError):
+            s = inner  # fence 内不是完整 JSON,继续往下扫
+    try:
+        return json.loads(s)
+    except (ValueError, TypeError):
+        pass
+    # 取最靠前出现的那种括号块(对象优先于数组,谁先出现谁优先)。
+    brace = _scan_balanced(s, "{", "}")
+    bracket = _scan_balanced(s, "[", "]")
+    if brace is not None and bracket is not None:
+        return brace if s.find("{") <= s.find("[") else bracket
+    return brace if brace is not None else bracket
+
 # 模板资源根目录(随 git 发布,与本模块同级 ``templates/`` 子目录)。
 _TEMPLATES_ROOT = Path(__file__).parent / "templates"