diff --git a/offline-backend-20260512/backend/CLAUDE.md b/offline-backend-20260512/backend/CLAUDE.md index 96629205..fd12f906 100644 --- a/offline-backend-20260512/backend/CLAUDE.md +++ b/offline-backend-20260512/backend/CLAUDE.md @@ -300,7 +300,7 @@ Two on-demand/automatic refresh paths layer on top: **Unified output switches (`require_html` + `require_markdown`).** The task form no longer splits "???? / HTML ??" into separate kinds. `execution_context` now carries two **independent** booleans that may both be on: `require_html` (???? HTML) and `require_markdown` (???? Markdown). `_task_requires_html()` returns true for `require_html` **or** legacy `task_kind == "html_page"` (so old tasks need no data migration), and `_run_task_attempt()` dispatches to `_run_html_page_attempt` when it is true, else the regular agent attempt. When **both** switches are on, `_gather_html_data()` returns `(text, files)` ? the gather agent run is additionally told to save a `.md` file ? and `_run_html_page_attempt` validates it (`MarkdownRequirementError` on failure) and surfaces it alongside the generated page, so the run delivers both artifacts. The single main `prompt` doubles as the HTML page requirement (`html_config.page_prompt`); there is no separate page-prompt field. Tests: `tests/test_scheduled_task_require_html.py` + the markdown-coexistence cases in `tests/test_html_page_tasks.py`. -**Template-fill mode (JSON → 固定 HTML 模板填充).** A third execution mode, distinct from the free-form `require_html` render pipeline: convert a user-supplied JSON into a **fixed** HTML template's data structure and fill it. Triggered when `execution_context.template_config.template_id` is set (chosen in the task form after selecting the built-in **「模板数据转换器」** agent `template-json-builder`). `_task_template_id()` reads it and `_run_task_attempt()` dispatches to `_run_template_fill_attempt` **before** the `require_html` check. Pipeline: take the source (the task `prompt`, pasted or uploaded .json) — which may be **pure JSON, plain text, or text-with-embedded-JSON** (`template_fill.extract_embedded_json` pre-extracts any embedded `{…}`/`[…]` with a string-aware balanced scan and surfaces it as a labeled block while keeping the full original text; plain text → no block, the model reads it semantically) → load the selected agent's `SOUL.md`+model and run a **direct** `_invoke_conversion_model` call (system=SOUL, human=conversion prompt carrying the template's compact **schema skeleton** + the recognized-JSON block + source) → `extract_json_object` → `template_fill.validate_data` (top-level keys + container types) → **on failure feed the structure errors + `field_coverage_gaps` back and retry up to `_MAX_TEMPLATE_FILL_RETRIES`=2** → `template_fill.fill_template` replaces the template's `const DATA={…}` line (and refreshes `MANIFEST.系统.更新时间`) → archive `index.html` + persist a `scheduled_html_pages` row → return `format:"html"` (reusing the existing HTML preview at `/runs/{id}/html`). Any terminal failure raises `TemplateFillError` → run marked failed (no retry, message shown). **Pure engine** `deerflow/runtime/scheduler/template_fill.py` (harness, no `app.*`): the template registry (`TEMPLATES`: id=`no1` name=`no.1` 台湾滨海防卫大屏, and id=`no2` name=`no.2` 台湾飞行部队平台 with **`convertible=False`** — a 「只展示」 template that renders its existing HTML **as-is**, no model/validation/fill, until conversion is wired in later; `_run_template_fill_attempt` short-circuits to `read_template_text` for it, and `list_templates`/`is_convertible` expose the flag so the form hides the JSON input), `list_templates`/`reference_data`/`schema_skeleton_json`/`validate_data`/`field_coverage_gaps`/`fill_template` — **the template's own embedded `DATA` is the schema source of truth** (parsed at load, 1-sample-per-array skeleton fed to the LLM, top-level structure as the validation basis); template HTML files live under `runtime/scheduler/templates//`, add a `TemplateSpec` to register a no.2. The agent is seeded like the AI-writing ones: `app/gateway/routers/_template_builder_seed.py` + `_template_builder_seed_assets/template-json-builder/{config.yaml,SOUL.md}`, called from the lifespan before `_sync_legacy_agents` (which upserts it into the `agents` table as built-in/published so it shows in `/api/agents` → the task form's 执行智能体 dropdown). Template list API: `GET /api/scheduled-tasks/html-templates` → `{templates:[{id,name,description,convertible}]}` (open). **Debug-render API** `POST /api/scheduled-tasks/template-fill/render` `{template_id, text?|data?}` → `{ok, display_only?, errors, gaps, html}` (open): no model call — `extract_embedded_json(text)` (or `data`) → `validate_data` → `fill_template`, returning the filled HTML **plus** the validation errors/field-gaps (fills even when invalid so you can see the effect). Powers the chat-page debug button below; non-convertible templates return their raw HTML. **Templates as `require_html` 参考收藏页面 (use the REAL template, all sub-pages)**: the 参考收藏页面 list in the task form also offers the templates (value `tpl:`, e.g. `tpl:no1`). These templates are **JS-driven multi-page SPAs** (sub-pages/tabs are rendered by script from `DATA`), so a free-form model render would only ever produce the main page and **drop every sub-page**. Therefore, when `_run_html_page_attempt` sees a `tpl:` reference (`template_fill.reference_template_id` parses it; distinguishes a template marker from a real favorite id), it short-circuits **before** the free-form pipeline to `_run_html_template_reference`, which renders the **actual template file** (all sub-pages intact): convertible templates gather real data (`_gather_html_data`) → convert to the template `DATA` (`_convert_to_template_data`, shared with template-fill mode) → **`merge_with_reference`** (per top-level key = per sub-page: any sub-page/field with no extracted content falls back to the template's **original** data, so sub-pages never render blank) → `fill_template`; non-convertible templates (no.2) output the raw template as-is; if the task prompt **and** description are both empty, or gather/conversion yields nothing, it uses the template's original data outright. **数据抽取开关** `html_config.template_extract` (default **off**, shown in the form only when a `tpl:` reference is picked): off → skip gather/convert entirely and just fill the template's **default data**; on → run the real gather+convert extraction. `merge_with_reference` is also applied in template-fill mode (`_run_template_fill_attempt`) so empty sub-pages there fall back too. Archive/persist/result are shared via `_finalize_template_html`. (The earlier `style_reference` CSS-extraction approach was removed — it lost the sub-pages.) Frontend: when the agent dropdown = `template-json-builder` the form hides the HTML/Markdown switches and shows a 「HTML模板」 select (`useHtmlTemplates`) + a .json upload (reads file text into the prompt box). Tests: `tests/test_template_fill.py`, `tests/test_template_fill_attempt.py`, `tests/test_template_builder_seed.py`. +**Template-fill mode (JSON → 固定 HTML 模板填充).** A third execution mode, distinct from the free-form `require_html` render pipeline: convert a user-supplied JSON into a **fixed** HTML template's data structure and fill it. Triggered when `execution_context.template_config.template_id` is set (chosen in the task form after selecting the built-in **「模板数据转换器」** agent `template-json-builder`). `_task_template_id()` reads it and `_run_task_attempt()` dispatches to `_run_template_fill_attempt` **before** the `require_html` check. Pipeline: take the source (the task `prompt`, pasted or uploaded .json) — which may be **pure JSON, plain text, or text-with-embedded-JSON** (`template_fill.extract_embedded_json` pre-extracts any embedded `{…}`/`[…]` with a string-aware balanced scan and surfaces it as a labeled block while keeping the full original text; plain text → no block, the model reads it semantically) → load the selected agent's `SOUL.md`+model and run a **direct** `_invoke_conversion_model` call (system=SOUL, human=conversion prompt carrying the template's compact **schema skeleton** + the recognized-JSON block + source) → `extract_json_object` → `template_fill.validate_data` (top-level keys + container types) → **on failure feed the structure errors + `field_coverage_gaps` back and retry up to `_MAX_TEMPLATE_FILL_RETRIES`=2** → `template_fill.fill_template` replaces the template's `const DATA={…}` line (and refreshes `MANIFEST.系统.更新时间`) → archive `index.html` + persist a `scheduled_html_pages` row → return `format:"html"` (reusing the existing HTML preview at `/runs/{id}/html`). Any terminal failure raises `TemplateFillError` → run marked failed (no retry, message shown). **Pure engine** `deerflow/runtime/scheduler/template_fill.py` (harness, no `app.*`): the template registry (`TEMPLATES`: id=`no1` name=`no.1` 台湾滨海防卫大屏, and id=`no2` name=`no.2` 台湾飞行部队平台 with **`convertible=False`** — a 「只展示」 template that renders its existing HTML **as-is**, no model/validation/fill, until conversion is wired in later; `_run_template_fill_attempt` short-circuits to `read_template_text` for it, and `list_templates`/`is_convertible` expose the flag so the form hides the JSON input), `list_templates`/`reference_data`/`schema_skeleton_json`/`validate_data`/`field_coverage_gaps`/`fill_template` — **the template's own embedded `DATA` is the schema source of truth** (parsed at load, 1-sample-per-array skeleton fed to the LLM, top-level structure as the validation basis); template HTML files live under `runtime/scheduler/templates//`, add a `TemplateSpec` to register a no.2. The agent is seeded like the AI-writing ones: `app/gateway/routers/_template_builder_seed.py` + `_template_builder_seed_assets/template-json-builder/{config.yaml,SOUL.md}`, called from the lifespan before `_sync_legacy_agents` (which upserts it into the `agents` table as built-in/published so it shows in `/api/agents` → the task form's 执行智能体 dropdown). Template list API: `GET /api/scheduled-tasks/html-templates` → `{templates:[{id,name,description,convertible}]}` (open). **Debug-render API** `POST /api/scheduled-tasks/template-fill/render` `{template_id, text?|data?}` → `{ok, display_only?, errors, gaps, html}` (open): no model call — `extract_embedded_json(text)` (or `data`) → `validate_data` → `fill_template`, returning the filled HTML **plus** the validation errors/field-gaps (fills even when invalid so you can see the effect). Powers the chat-page debug button below; non-convertible templates return their raw HTML. **Templates as `require_html` 参考收藏页面 (use the REAL template, all sub-pages)**: the 参考收藏页面 list in the task form also offers the templates (value `tpl:`, e.g. `tpl:no1`). These templates are **JS-driven multi-page SPAs** (sub-pages/tabs are rendered by script from `DATA`), so a free-form model render would only ever produce the main page and **drop every sub-page**. Therefore, when `_run_html_page_attempt` sees a `tpl:` reference (`template_fill.reference_template_id` parses it; distinguishes a template marker from a real favorite id), it short-circuits **before** the free-form pipeline to `_run_html_template_reference`, which renders the **actual template file** (all sub-pages intact): convertible templates gather real data (`_gather_html_data`) → convert to the template `DATA` (`_convert_to_template_data`, shared with template-fill mode) → **`merge_with_reference`** (per top-level key = per sub-page: any sub-page/field with no extracted content falls back to the template's **original** data, so sub-pages never render blank) → `fill_template`; non-convertible templates (no.2) output the raw template as-is; if the task prompt **and** description are both empty, or gather/conversion yields nothing, it uses the template's original data outright. **数据抽取开关** `html_config.template_extract` (default **off**, shown in the form only when a `tpl:` reference is picked): it **only** controls whether the HTML uses **extracted** data vs the template's **default** data — it does **not** affect anything else. Off → HTML fills the template's default data (no convert); on → gather+convert+merge. **Markdown is independent**: when `require_markdown` is also on, the gather agent run still runs and the `.md` is produced/validated/surfaced **regardless** of the extract switch (so turning extraction off never blocks the Markdown). Display-only templates (no.2) never extract; they still honor `require_markdown`. `_run_html_template_reference` runs gather once when `do_extract` (extract on + has prompt/desc + convertible) **or** `require_markdown`, and threads any `.md` through `_finalize_template_html(extra_files=…)`. `merge_with_reference` is also applied in template-fill mode (`_run_template_fill_attempt`) so empty sub-pages there fall back too. Archive/persist/result are shared via `_finalize_template_html`. (The earlier `style_reference` CSS-extraction approach was removed — it lost the sub-pages.) Frontend: when the agent dropdown = `template-json-builder` the form hides the HTML/Markdown switches and shows a 「HTML模板」 select (`useHtmlTemplates`) + a .json upload (reads file text into the prompt box). Tests: `tests/test_template_fill.py`, `tests/test_template_fill_attempt.py`, `tests/test_template_builder_seed.py`. ### Per-user Prompts & UI Preferences diff --git a/offline-backend-20260512/backend/packages/harness/deerflow/runtime/scheduler/service.py b/offline-backend-20260512/backend/packages/harness/deerflow/runtime/scheduler/service.py index 9c9db56c..39a30c32 100644 --- a/offline-backend-20260512/backend/packages/harness/deerflow/runtime/scheduler/service.py +++ b/offline-backend-20260512/backend/packages/harness/deerflow/runtime/scheduler/service.py @@ -1496,13 +1496,17 @@ class ScheduledTaskService: content: str, source_chars: int, used_default: bool, + extra_files: list[dict[str, Any]] | None = None, ) -> dict[str, Any]: - """归档 index.html + 持久化 scheduled_html_pages + 构造 format=html 结果。""" + """归档 index.html + 持久化 scheduled_html_pages + 构造 format=html 结果。 + + ``extra_files`` 会排在 index.html 之前一并带上(如必须生成 Markdown 时的 .md)。 + """ task_tz = ZoneInfo(task.get("timezone") or "UTC") local_time = scheduled_for.astimezone(task_tz) archive_path = self._archive_html_index(run_id, html) title = str(task.get("name") or spec.name).strip() - files: list[dict[str, Any]] = [] + files: list[dict[str, Any]] = list(extra_files or []) if archive_path is not None: files.append( { @@ -1569,8 +1573,40 @@ class ScheduledTaskService: raise HtmlGenerationError(f"未知的 HTML 模板：{tpl_id}") cfg = self._html_config(task) model_name = str(cfg.get("model_name") or "").strip() or None + # 「执行数据抽取」开关**只**控制 HTML 用「抽取数据」还是「模板默认数据」, + # 不影响 Markdown 生成等其它正常操作。 + extract = bool(cfg.get("template_extract")) + also_markdown = _task_requires_markdown(task) + prompt = str(task.get("prompt") or "").strip() + description = str(task.get("description") or "").strip() + # 真正抽取 HTML 数据：开关开 且 有描述/任务 prompt 且 模板可转换(只展示模板恒不抽取)。 + do_extract = extract and bool(prompt or description) and spec.convertible + task_tz = ZoneInfo(task.get("timezone") or "UTC") + local_time = scheduled_for.astimezone(task_tz) - # 只展示模板：原样输出(含全部子页面)。 + def _pick_md(gathered: list[dict[str, Any]]) -> list[dict[str, Any]]: + return [ + item + for item in (gathered or []) + if isinstance(item, dict) and str(item.get("name") or "").lower().endswith((".md", ".markdown")) + ] + + # gather：HTML 要抽取 或 需要 Markdown 时才跑(同一次 agent 运行可兼顾两者); + # 都不需要则完全不跑 agent。Markdown 校验独立于抽取开关。 + data_text = "" + md_files: list[dict[str, Any]] = [] + if do_extract or also_markdown: + data_text, gathered = await self._gather_html_data( + task, runtime_thread_id=runtime_thread_id, scheduled_for=scheduled_for, run_id=run_id + ) + if also_markdown: + if not _has_valid_markdown_file(gathered): + raise MarkdownRequirementError( + f"未满足输出要求：需生成 Markdown(.md) 文件，且文件正文不少于 {_MARKDOWN_MIN_CHARS} 字" + ) + md_files = _pick_md(gathered) + + # 只展示模板：HTML 原样输出(含全部子页面);Markdown 已按需独立生成。 if not spec.convertible: html = template_fill.read_template_text(tpl_id) return await self._finalize_template_html( @@ -1583,54 +1619,13 @@ class ScheduledTaskService: content=f"已套用模板「{spec.name}」（含全部子页面）", source_chars=0, used_default=True, + extra_files=md_files, ) - # 数据抽取开关(html_config.template_extract):关闭 → 直接用模板默认数据,不检索/ - # 不转换;开启 → 才执行下面的检索+转换抽取。 - if not bool(cfg.get("template_extract")): - data_obj = template_fill.reference_data(tpl_id) - return await self._finalize_template_html( - task, - spec=spec, - template_id=tpl_id, - html=template_fill.fill_template( - tpl_id, data_obj, update_date=scheduled_for.astimezone(ZoneInfo(task.get("timezone") or "UTC")).date() - ), - run_id=run_id, - scheduled_for=scheduled_for, - content=f"未开启数据抽取，已用模板「{spec.name}」默认数据（含全部子页面）", - source_chars=0, - used_default=True, - ) - - # 用户的描述与任务 prompt 都为空 → 不检索/不转换,直接用模板原有数据。 - prompt = str(task.get("prompt") or "").strip() - description = str(task.get("description") or "").strip() - if not prompt and not description: - data_obj = template_fill.reference_data(tpl_id) - return await self._finalize_template_html( - task, - spec=spec, - template_id=tpl_id, - html=template_fill.fill_template(tpl_id, data_obj, update_date=scheduled_for.astimezone(ZoneInfo(task.get("timezone") or "UTC")).date()), - run_id=run_id, - scheduled_for=scheduled_for, - content=f"未提供描述，已用模板「{spec.name}」原有数据（含全部子页面）", - source_chars=0, - used_default=True, - ) - - # 可转换模板：检索真实数据(复用 gather) → 转换成模板 DATA → 填进真模板。 - data_text, _files = await self._gather_html_data( - task, runtime_thread_id=runtime_thread_id, scheduled_for=scheduled_for, run_id=run_id - ) + # HTML 数据来源：抽取(开关开 + 有数据)→ 转换填充;否则用模板默认数据(子页面齐全)。 source = (data_text or "").strip() - used_default = False - if not source: - data_obj = template_fill.reference_data(tpl_id) - content = f"未获取到数据，已用模板「{spec.name}」默认数据生成（含全部子页面）" - used_default = True - else: + used_default = True + if do_extract and source: try: converted = await self._convert_to_template_data( tpl_id, source, model_name=model_name, soul=None, run_id=run_id @@ -1638,14 +1633,19 @@ class ScheduledTaskService: # 子页面没抽到有效内容的,回退模板原有数据,避免子页面空白。 data_obj = template_fill.merge_with_reference(tpl_id, converted) content = f"已检索数据并套用模板「{spec.name}」（空子页面回退原数据，含全部子页面）" + used_default = False except TemplateFillError: logger.warning("模板参考转换失败,回退默认数据 run=%s tpl=%s", run_id, tpl_id, exc_info=True) data_obj = template_fill.reference_data(tpl_id) content = f"数据转换未通过，已回退模板「{spec.name}」默认数据（含全部子页面）" - used_default = True + else: + data_obj = template_fill.reference_data(tpl_id) + content = ( + f"未开启数据抽取，已用模板「{spec.name}」默认数据（含全部子页面）" + if not extract + else f"未提供描述/未获取到数据，已用模板「{spec.name}」默认数据（含全部子页面）" + ) - task_tz = ZoneInfo(task.get("timezone") or "UTC") - local_time = scheduled_for.astimezone(task_tz) try: html = template_fill.fill_template(tpl_id, data_obj, update_date=local_time.date()) except template_fill.TemplateError as exc: @@ -1660,6 +1660,7 @@ class ScheduledTaskService: content=content, source_chars=len(source), used_default=used_default, + extra_files=md_files, ) async def _run_template_fill_attempt(