Merge branch 'dhk-1' of http://192.168.0.200/knowledge-base-bj/deerflow-server into dhk-1
This commit is contained in:
commit
8f6e08eb2f
102 files changed
+6816
-6016
No files matched your search
@@ -245,7 +245,9 @@ FastAPI application on port 8001 with health check at `GET /health`. Set `GATEWA
|
||||
| **Thread Shares** (`/api/thread-shares`) | Per-conversation sharing with two modes (`thread_shares.mode`). **`import`**: `POST /` `{thread_id, mode}` - create (or reuse) a revocable share code; `GET /{code}/preview` + `POST /{code}/import` - copy that conversation into the caller's account (fresh `thread_id` via `app/gateway/thread_copy.py` + new `threads_meta` row + `user-data` copy; dedup on `metadata.shared_origin`). **`view`**: a public, read-only static page — see Public Shares below. Both modes: `GET /` - list my codes; `DELETE /{code}` - revoke. |
|
||||
| **Public Shares** (`/api/public/shares`) | **Unauthenticated** (the `/api/public/` prefix is whitelisted in `auth_middleware._PUBLIC_PATH_PREFIXES`). `GET /{code}` - read-only conversation snapshot (title + serialized messages + artifacts) for a `view`-mode share; messages are read straight from the latest checkpoint. `GET /{code}/files/{path}` - serves a file referenced by that conversation (images, Markdown, text, PDF, …) so the shared page's links are openable; path resolution is confined to the owner's thread directory. Active content (`text/html`, `application/xhtml+xml`, `image/svg+xml`) is forced to download (`_public_share_disposition`); everything else is served inline. Missing/revoked/non-`view` codes all return 404. |
|
||||
| **Roundtable Drafts** (`/api/roundtable-drafts`) | Per-user persistence for the roundtable-planning page (replaces the old browser `localStorage`). `GET /` - list drafts (lightweight meta); `POST /` - create; `GET/PUT/DELETE /{id}` - full read / partial update / delete. A draft = one full session (Step 1 intent + Step 2 roundtable), step snapshots stored as JSON in `PortableLongText`. Recommendation history bound to a draft: `GET /{id}/recommendations` - list newest-first; `POST /{id}/recommendations` - append one run (objective/status/rationale/picks/candidates). Backed by `deerflow.persistence.roundtable_drafts` (tables `roundtable_drafts`, `roundtable_recommend_history`); store wired in `deps.py` as `app.state.roundtable_draft_store`. |
|
||||
| **Scheduled Tasks** (`/api/scheduled-tasks`) | Per-user cron-like agent tasks. `GET/POST /` - list/create owned tasks; `PATCH/DELETE /{id}` - update/delete (owner); `POST /{id}/{pause,resume,trigger}`; `GET /{id}/runs`, `GET /runs/{rid}`, run-file preview/edit/download; `GET /runs/{rid}/messages` - agent messages for the run's underlying agent run (used to inspect a stuck/running run; authorized via `_load_run`); `DELETE /runs/{rid}` - owner-scoped, with an admin fallback so admins can delete any user's run; publish/subscribe under `/{id}/{publish,unpublish,subscribe}` and `GET /published`. **Admin (cross-user):** `GET /admin/all` - every task; `PATCH/DELETE /admin/{id}` - edit/delete any task; `POST /admin/{id}/trigger`; `GET /admin/{id}/runs`. Admin routes require `system_role == "admin"`. |
|
||||
| **Scheduled Tasks** (`/api/scheduled-tasks`) | Per-user cron-like agent tasks. `GET/POST /` - list/create owned tasks; `PATCH/DELETE /{id}` - update/delete (owner); `POST /{id}/{pause,resume,trigger}`; `GET /{id}/runs`, `GET /runs/{rid}`, run-file preview/edit/download; `GET /runs/{rid}/messages` - agent messages for the run's underlying agent run (used to inspect a stuck/running run; authorized via `_load_run`); `GET /runs/{rid}/html` - generated HTML document for an `html_page` run; `DELETE /runs/{rid}` - owner-scoped, with an admin fallback so admins can delete any user's run; publish/subscribe under `/{id}/{publish,unpublish,subscribe}` and `GET /published`. **Admin (cross-user):** `GET /admin/all` - every task; `PATCH/DELETE /admin/{id}` - edit/delete any task; `POST /admin/{id}/trigger`; `GET /admin/{id}/runs`. Admin routes require `system_role == "admin"`. |
|
||||
| **HTML Page Favorites** (`/api/html-page-favorites`) | Bookmarked generated HTML pages, owner-scoped. `GET /` - list; `POST /` `{page_id, title?, description?, tags?}` - bookmark a `scheduled_html_pages` row (extracts a reusable style/layout summary via GLM5 at favorite time); a page can only be favorited once per user (`DuplicateFavoriteError` → HTTP 409). `GET/DELETE /{id}` - read/delete (the create-task reference picker deletes favorites here). A favorite can be selected as a style reference when creating a new `html_page` task. |
|
||||
| **Public Scheduled Tasks** (`/api/public/scheduled-tasks`) | **Unauthenticated** mirror for embedding. Adds `GET /runs/{rid}/html` and `GET /{task_id}/latest-html` for the `html_page` preview; the run result keeps `result.html` (only file paths are redacted). |
|
||||
|
||||
Proxied through nginx: `/api/langgraph/*` → LangGraph, all other `/api/*` → Gateway.
|
||||
|
||||
@@ -257,6 +259,10 @@ Proxied through nginx: `/api/langgraph/*` → LangGraph, all other `/api/*` →
|
||||
|
||||
**Unique task names**: task names are unique per user (`uq_scheduled_tasks_user_name`). `create_task`/`update_task`/`admin_update_task` proactively check via `_assert_name_available()` and raise `DuplicateTaskNameError` on a collision; the router translates that into HTTP `409` instead of letting the raw DB `IntegrityError` surface as a `500`.
|
||||
|
||||
**HTML page tasks (`task_kind == "html_page"`)**: a second task kind that renders a full HTML document each run instead of a Markdown report. The kind + its config (`html_config`: `page_prompt`, `negative_prompt`, `reference_favorite_id`, `model_name`) live inside `execution_context` — no schema change to `scheduled_tasks`. Data gathering is delegated to the agent's own search skills (driven by `page_prompt`), so there is no explicit search-query/data-source field; the render model is user-selectable (`model_name`, default `zai-org/GLM-5-FP8`). `_run_task_attempt()` dispatches by kind; `_run_html_page_attempt()` runs the pipeline: gather real data via the lead agent (its search tools) → optionally load a reference favorite's reusable style/layout → render with the chosen model (thinking off) → fault-tolerant JSON parse → sanitize (BeautifulSoup, strips `<script>`/`<iframe>`/`on*`/`javascript:`) → structural validate → one GLM5 repair pass if invalid → **completeness gate**: if the HTML is still not a full valid document (missing doctype/html/head/body, empty body, or residual scripts) after the repair pass, raise `HtmlGenerationError` so the run is marked **failed** (a page only counts as successful when it is a complete, previewable HTML document) → best-effort Playwright screenshot (no-op when Playwright absent) → archive `index.html` → persist a `scheduled_html_pages` row → return `result` with `format: "html"` and the full HTML inline. Pure helpers live in `deerflow/runtime/scheduler/html_page.py`. The frontend previews the HTML in a sandboxed `iframe` (no `allow-scripts`) on the run-detail and public pages; a user can bookmark a page (style summary extracted via GLM5) and later pick it as a style reference. New tables `scheduled_html_pages` + `html_page_favorites` (`deerflow/persistence/html_pages/`, wired as `app.state.html_page_store`) are created by `Base.metadata.create_all` with no migration. Tests: `tests/test_html_page_tasks.py`.
|
||||
|
||||
**HTML skills (tools the render step uses).** Two offline skills under `skills/public/` shape the generated HTML and are toggled in the skills settings page (enabled by default): `html-page-builder` (a self-contained inline-CSS design system — colors/cards/tables/KPI/responsive) and `html-charts` (inline-SVG charts that render in the no-`allow-scripts` preview iframe; optional local ECharts documented in its `assets/README.md`). A skill opts into HTML injection via `metadata.kind: html_page` in its SKILL.md frontmatter; `_collect_html_skill_instructions()` reads the bodies of **enabled** such skills and `build_generation_prompt(skill_instructions=…)` injects them, so the (raw-model) render step honors the user's skill selection. The data-gathering step and the render/repair/style passes all use the task's selected `model_name` (no silent fallback to the default model — and there is **no hardcoded GLM-5**; an unset selection falls back to config `models[0]`). To stop auxiliary LLM calls from leaking onto the default model, **`TitleMiddleware` is skipped when `is_scheduled_run` is set** (`_build_middlewares` in `lead_agent/agent.py`) — scheduled/HTML runs use throwaway threads whose titles are never shown, and `config.title.model_name` is usually unset (→ would call `models[0]`, not the chosen model). The completeness gate (`validate_html`) also requires a non-empty `<title>` and forbids external stylesheets, on top of the doctype/html/head/body + non-empty-body + no-script checks.
|
||||
|
||||
### Custom Agent Persistence
|
||||
|
||||
Custom agent metadata lives in the `agents` ORM table (`deerflow.persistence.agents`). `agents.id` is the stable runtime id and the filesystem directory name under `.deer-flow/agents/{id}/`; `agents.name` is display-only, can contain Chinese characters, and is not unique. `user_id IS NULL` marks built-in agents, user-owned agents are visible only to their owner unless `published = true`, and update/delete operations require ownership. The runtime accepts `agent_id` in context/configurable, and non-default `assistant_id` values are treated as agent ids after visibility checks.
|
||||
@@ -497,6 +503,11 @@ Per-user runtime overrides (whitelisted fields) live at `{base_dir}/users/{user_
|
||||
- `summarization` - Context summarization (enabled, trigger conditions, keep policy)
|
||||
- `subagents.enabled` - Master switch for subagent delegation
|
||||
- `memory` - Memory system (enabled, storage_path, debounce_seconds, model_name, max_facts, fact_confidence_threshold, injection_enabled, max_injection_tokens)
|
||||
- `auth_login` - Custom login modes layered on the base auth (`deerflow/config/login_config.py`):
|
||||
- `username_login.{require_password, password}` - Password gate for `POST /api/v1/auth/login/username`. When `require_password=true`, the (otherwise passwordless) username login additionally requires `?password=<shared-secret>`; empty `password` falls back to `123ewq`. Gate only — it does not change which account logs in.
|
||||
- `token_login.{enabled, token_info_url, service_authorization, timeout_seconds, verify_ssl, ca_cert_path}` - Token-exchange login (`POST /api/v1/auth/login/token`). The gateway POSTs to `token_info_url` with the caller-supplied token placed directly in the `Authorization: Bearer <token>` header (no cookie; the static `service_authorization` field is deprecated/unused, kept only for backward compat), reads `data.username` from a `{"state":"200",...}` body, then signs the matching local user in (auto-registering if absent). `token_info_url` is HTTPS — `verify_ssl: false` skips cert verification for self-signed/internal UAT hosts, or set `ca_cert_path` to a PEM bundle (takes precedence, keeps verification on). See [docs/AUTH_LOGIN.md](docs/AUTH_LOGIN.md) for the full request/response contract.
|
||||
|
||||
Both endpoints are whitelisted in `auth_middleware._PUBLIC_EXACT_PATHS`. The shared `_get_or_create_user_by_username()` helper (in `routers/auth.py`) backs both `/login/username` and `/login/token`.
|
||||
|
||||
**`extensions_config.json`**:
|
||||
- `mcpServers` - Map of server name → config (enabled, type, command, args, env, url, headers, oauth, description)
|
||||
@@ -637,12 +648,21 @@ Automatic conversation summarization when approaching token limits:
|
||||
|
||||
See [docs/summarization.md](docs/summarization.md) for details.
|
||||
|
||||
### Vision Support
|
||||
### Vision Support / Image Recognition (识图)
|
||||
|
||||
For models with `supports_vision: true`:
|
||||
- `ViewImageMiddleware` processes images in conversation
|
||||
- `view_image_tool` added to agent's toolset
|
||||
- Images automatically converted to base64 and injected into state
|
||||
Two paths, selected by `config.yaml` → `vision`:
|
||||
|
||||
**Dedicated vision model (delegated recognition)** — set `vision.model_name` to a model from `models[]`. The `view_image` tool sends the image to that model and returns its **text description** as the tool result, so a text-only main chat model can still "see" images. `view_image` accepts an optional `question` argument for targeted queries; the default prompt is `vision.prompt` (overridable), and `vision.max_tokens` caps the recognition output. Resolution/gating lives in `deerflow/models/vision.py` (`resolve_vision_model_name`, `vision_enabled`, `is_delegated_vision`, `recognize_image`).
|
||||
|
||||
**Native vision (fallback)** — when `vision.model_name` is unset, behavior falls back to the main chat model, which only works if it has `supports_vision: true`:
|
||||
- `ViewImageMiddleware` injects base64 image data into the conversation before the LLM call
|
||||
- `view_image` stores the image in `state.viewed_images` for that middleware
|
||||
|
||||
Both the `view_image` tool and `ViewImageMiddleware` are registered whenever `vision_enabled(model_name)` is true (dedicated model configured **or** main model supports vision). Config schema: `deerflow/config/vision_config.py` (`VisionConfig`); tests: `tests/test_vision.py`.
|
||||
|
||||
### Upload symlink-safety (cross-platform)
|
||||
|
||||
`deerflow/uploads/manager.py:open_upload_file_no_symlink` writes uploads without following a pre-existing destination symlink. `O_NOFOLLOW` is applied where available (POSIX) but is **optional** — Windows lacks `os.O_NOFOLLOW`/`O_NONBLOCK`, so requiring it previously made every Windows upload fail with `UnsafeUploadPathError`. Pre-existing symlinks are still rejected via `lstat`, and the opened fd is re-checked with `fstat`. `O_BINARY` is added on Windows. Regression: `tests/test_uploads_windows_safe.py`.
|
||||
|
||||
## Code Style
|
||||
|
||||
|
||||
@@ -32,6 +32,7 @@ from app.gateway.routers import (
|
||||
auth,
|
||||
channels,
|
||||
feedback,
|
||||
html_page_favorites,
|
||||
intent,
|
||||
llm_metrics,
|
||||
mcp,
|
||||
@@ -881,6 +882,7 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
|
||||
|
||||
# Public embed Q&A status polling (/api/public/embed/*)
|
||||
app.include_router(public_embed.router)
|
||||
app.include_router(html_page_favorites.router)
|
||||
|
||||
# System settings API is mounted at /api/system-settings
|
||||
app.include_router(system_settings.router)
|
||||
|
||||
@@ -41,6 +41,7 @@ _PUBLIC_EXACT_PATHS: frozenset[str] = frozenset(
|
||||
{
|
||||
"/api/v1/auth/login/local",
|
||||
"/api/v1/auth/login/username",
|
||||
"/api/v1/auth/login/token",
|
||||
"/api/v1/auth/register",
|
||||
"/api/v1/auth/logout",
|
||||
"/api/v1/auth/setup-status",
|
||||
|
||||
@@ -93,6 +93,10 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
|
||||
app.state.scheduled_task_store = make_scheduled_task_store(sf)
|
||||
|
||||
from deerflow.persistence.html_pages import make_html_page_store
|
||||
|
||||
app.state.html_page_store = make_html_page_store(sf)
|
||||
|
||||
from deerflow.persistence.llm_metrics import make_llm_metrics_store
|
||||
|
||||
app.state.llm_metrics_store = make_llm_metrics_store(sf)
|
||||
|
||||
@@ -6,10 +6,12 @@ import time
|
||||
from ipaddress import ip_address, ip_network
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request, Response, status
|
||||
import httpx
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request, Response, status
|
||||
from fastapi.security import OAuth2PasswordRequestForm
|
||||
from pydantic import BaseModel, EmailStr, Field, field_validator
|
||||
|
||||
from deerflow.config import get_app_config
|
||||
from deerflow.persistence.agents.base import AgentStore
|
||||
from deerflow.persistence.skills.base import SkillStore
|
||||
|
||||
@@ -174,12 +176,12 @@ def _normalize_username_local_part(username: str) -> str | None:
|
||||
"""
|
||||
raw = username.strip()
|
||||
if not raw:
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="username cannot be empty")
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="用户名不能为空")
|
||||
if "@" in raw:
|
||||
return None
|
||||
normalized = "".join(ch if (ch.isalnum() or ch in "._-") else "-" for ch in raw.lower()).strip("._-")
|
||||
if not normalized:
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="username is invalid")
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="用户名不合法")
|
||||
return normalized
|
||||
|
||||
|
||||
@@ -200,6 +202,117 @@ def _username_to_email(username: str) -> str:
|
||||
return f"{local}@{domain}"
|
||||
|
||||
|
||||
async def _get_or_create_user_by_username(username: str):
|
||||
"""Resolve a username to a local user, auto-registering when absent.
|
||||
|
||||
Shared by ``/login/username`` and ``/login/token`` — both turn an
|
||||
externally-supplied username into a local account and sign it in. Returns
|
||||
``(user, created)`` where ``created`` is True only when this call inserted
|
||||
the row. Mirrors the legacy domain-agnostic fallback and the canonical
|
||||
``admin`` promotion so both entry points behave identically.
|
||||
"""
|
||||
provider = get_local_provider()
|
||||
email = _username_to_email(username)
|
||||
local_part = _normalize_username_local_part(username)
|
||||
|
||||
created = False
|
||||
user = await provider.get_user_by_email(email)
|
||||
# Domain-agnostic fallback: if no row exists at the canonical
|
||||
# ``<username>@<current-default-domain>``, look for an existing user with
|
||||
# the same local part under any domain. This is the safety net that keeps
|
||||
# usernames globally unique when the default email domain changes (e.g.
|
||||
# ``local.deerflow`` → ``cm.com``) — without it, switching the default
|
||||
# would let the same username log into two different accounts.
|
||||
if user is None and local_part is not None:
|
||||
legacy = await provider.find_user_by_email_local_part(local_part)
|
||||
if legacy is not None:
|
||||
user = legacy
|
||||
logger.info("Matched username %r to legacy account %s", username, legacy.email)
|
||||
if user is None:
|
||||
try:
|
||||
role = "admin" if username == "admin" else "user"
|
||||
user = await provider.create_user(email=email, password=None, system_role=role, needs_setup=False)
|
||||
created = True
|
||||
logger.info("Auto-registered passwordless user: %s", email)
|
||||
except ValueError:
|
||||
# Concurrent first login may create the same user in parallel.
|
||||
user = await provider.get_user_by_email(email)
|
||||
if user is None and local_part is not None:
|
||||
user = await provider.find_user_by_email_local_part(local_part)
|
||||
if user is None:
|
||||
raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail="创建或加载用户失败")
|
||||
elif username == "admin" and user.system_role != "admin":
|
||||
# Historical accounts created before the username-based role assignment
|
||||
# may still have system_role="user" — promote the canonical admin user
|
||||
# so /login/admin behaves consistently.
|
||||
user.system_role = "admin"
|
||||
user = await provider.update_user(user)
|
||||
logger.info("Promoted historical 'admin' account to system_role=admin: %s", email)
|
||||
|
||||
return user, created
|
||||
|
||||
|
||||
async def _fetch_username_from_token(token: str) -> str:
|
||||
"""Resolve an upstream access token to a username via the configured endpoint.
|
||||
|
||||
Calls ``auth_login.token_login.token_info_url`` with the caller-supplied
|
||||
token placed directly in the ``Authorization: Bearer <token>`` header — the
|
||||
upstream reads the bearer token from there, not from a cookie. Expects
|
||||
``{"state": "200", "data": {"username": ...}}`` and returns ``data.username``.
|
||||
Any upstream failure, non-200 ``state``, or missing username raises HTTP 401.
|
||||
"""
|
||||
config = get_app_config().auth_login.token_login
|
||||
if not config.token_info_url:
|
||||
raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail="token 登录未配置(token_info_url 为空)")
|
||||
|
||||
# TLS verification for the (HTTPS) upstream. A custom CA bundle wins; else
|
||||
# the boolean toggle. UAT/internal hosts often need verify_ssl=false because
|
||||
# they serve self-signed or private-CA certs — without this the call dies
|
||||
# with an SSLError and every token login silently 401s.
|
||||
verify: bool | str = config.ca_cert_path if config.ca_cert_path else config.verify_ssl
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=config.timeout_seconds, verify=verify) as client:
|
||||
upstream = await client.post(
|
||||
config.token_info_url,
|
||||
headers={
|
||||
"Authorization": f"Bearer {token}",
|
||||
"Content-Type": "application/x-www-form-urlencoded",
|
||||
},
|
||||
)
|
||||
except httpx.HTTPError as exc:
|
||||
# Surface the concrete failure (SSL handshake, DNS, timeout, ...) in logs
|
||||
# so HTTPS/cert misconfiguration is diagnosable; client still sees 401.
|
||||
logger.warning(
|
||||
"Token login: upstream call to %s failed (verify=%r): %s: %s",
|
||||
config.token_info_url,
|
||||
verify,
|
||||
type(exc).__name__,
|
||||
exc,
|
||||
)
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail=AuthErrorResponse(code=AuthErrorCode.INVALID_CREDENTIALS, message="token 校验失败").model_dump())
|
||||
|
||||
if upstream.status_code != 200:
|
||||
logger.warning("Token login: upstream returned HTTP %s", upstream.status_code)
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail=AuthErrorResponse(code=AuthErrorCode.INVALID_CREDENTIALS, message="token 无效").model_dump())
|
||||
|
||||
try:
|
||||
payload = upstream.json()
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail=AuthErrorResponse(code=AuthErrorCode.INVALID_CREDENTIALS, message="token 接口返回格式异常").model_dump())
|
||||
|
||||
# Upstream signals success with the string state "200" (see API docs).
|
||||
if str(payload.get("state")) != "200":
|
||||
logger.warning("Token login: upstream state=%r msg=%r", payload.get("state"), payload.get("msg"))
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail=AuthErrorResponse(code=AuthErrorCode.INVALID_CREDENTIALS, message="token 无效或已过期").model_dump())
|
||||
|
||||
username = (payload.get("data") or {}).get("username")
|
||||
if not username or not str(username).strip():
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail=AuthErrorResponse(code=AuthErrorCode.INVALID_CREDENTIALS, message="未能从 token 解析出用户名").model_dump())
|
||||
|
||||
return str(username).strip()
|
||||
|
||||
|
||||
def _set_session_cookie(response: Response, token: str, request: Request) -> None:
|
||||
"""Set the access_token HttpOnly cookie on the response."""
|
||||
config = get_auth_config()
|
||||
@@ -301,7 +414,7 @@ def _check_rate_limit(ip: str) -> None:
|
||||
if time.time() < lock_until:
|
||||
raise HTTPException(
|
||||
status_code=429,
|
||||
detail="Too many login attempts. Try again later.",
|
||||
detail="登录尝试次数过多,请稍后再试",
|
||||
)
|
||||
del _login_attempts[ip]
|
||||
|
||||
@@ -357,7 +470,7 @@ async def login_local(
|
||||
_record_login_failure(client_ip)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail=AuthErrorResponse(code=AuthErrorCode.INVALID_CREDENTIALS, message="Incorrect email or password").model_dump(),
|
||||
detail=AuthErrorResponse(code=AuthErrorCode.INVALID_CREDENTIALS, message="邮箱或密码错误").model_dump(),
|
||||
)
|
||||
|
||||
_record_login_success(client_ip)
|
||||
@@ -371,53 +484,39 @@ async def login_local(
|
||||
|
||||
|
||||
@router.post("/login/username", response_model=UsernameLoginResponse)
|
||||
async def login_by_username(request: Request, response: Response, body: UsernameLoginRequest):
|
||||
async def login_by_username(
|
||||
request: Request,
|
||||
response: Response,
|
||||
body: UsernameLoginRequest,
|
||||
password: str | None = Query(default=None, description="Shared gate secret; required only when auth_login.username_login.require_password is on."),
|
||||
):
|
||||
"""Passwordless username login with auto-registration.
|
||||
|
||||
Rules:
|
||||
- If mapped account exists, sign in directly (no password required).
|
||||
- If account does not exist, create a regular user and sign in.
|
||||
|
||||
Password gate: when ``auth_login.username_login.require_password`` is
|
||||
enabled, callers must additionally pass ``?password=`` matching the
|
||||
configured shared secret (default ``123ewq`` when unset). The gate is a
|
||||
single shared password — it does not change which account is logged in.
|
||||
|
||||
Returns the JWT access token in the response body and also sets it as
|
||||
a session cookie for browser callers.
|
||||
"""
|
||||
provider = get_local_provider()
|
||||
email = _username_to_email(body.username)
|
||||
local_part = _normalize_username_local_part(body.username)
|
||||
username_login_config = get_app_config().auth_login.username_login
|
||||
if username_login_config.require_password:
|
||||
client_ip = _get_client_ip(request)
|
||||
_check_rate_limit(client_ip)
|
||||
if password != username_login_config.effective_password:
|
||||
_record_login_failure(client_ip)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail=AuthErrorResponse(code=AuthErrorCode.INVALID_CREDENTIALS, message="口令错误").model_dump(),
|
||||
)
|
||||
_record_login_success(client_ip)
|
||||
|
||||
created = False
|
||||
user = await provider.get_user_by_email(email)
|
||||
# Domain-agnostic fallback: if no row exists at the canonical
|
||||
# ``<username>@<current-default-domain>``, look for an existing user with
|
||||
# the same local part under any domain. This is the safety net that keeps
|
||||
# usernames globally unique when the default email domain changes (e.g.
|
||||
# ``local.deerflow`` → ``cm.com``) — without it, switching the default
|
||||
# would let the same username log into two different accounts.
|
||||
if user is None and local_part is not None:
|
||||
legacy = await provider.find_user_by_email_local_part(local_part)
|
||||
if legacy is not None:
|
||||
user = legacy
|
||||
logger.info("Matched username %r to legacy account %s", body.username, legacy.email)
|
||||
if user is None:
|
||||
try:
|
||||
role = "admin" if body.username == "admin" else "user"
|
||||
user = await provider.create_user(email=email, password=None, system_role=role, needs_setup=False)
|
||||
created = True
|
||||
logger.info("Auto-registered passwordless user: %s", email)
|
||||
except ValueError:
|
||||
# Concurrent first login may create the same user in parallel.
|
||||
user = await provider.get_user_by_email(email)
|
||||
if user is None and local_part is not None:
|
||||
user = await provider.find_user_by_email_local_part(local_part)
|
||||
if user is None:
|
||||
raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail="Failed to create or load user")
|
||||
elif body.username == "admin" and user.system_role != "admin":
|
||||
# Historical accounts created before the username-based role assignment
|
||||
# may still have system_role="user" — promote the canonical admin user
|
||||
# so /login/admin behaves consistently.
|
||||
user.system_role = "admin"
|
||||
user = await provider.update_user(user)
|
||||
logger.info("Promoted historical 'admin' account to system_role=admin: %s", email)
|
||||
user, created = await _get_or_create_user_by_username(body.username)
|
||||
|
||||
token = create_access_token(str(user.id), token_version=user.token_version)
|
||||
_set_session_cookie(response, token, request)
|
||||
@@ -434,6 +533,56 @@ async def login_by_username(request: Request, response: Response, body: Username
|
||||
)
|
||||
|
||||
|
||||
@router.post("/login/token", response_model=UsernameLoginResponse)
|
||||
async def login_by_token(
|
||||
request: Request,
|
||||
response: Response,
|
||||
token: str = Query(..., min_length=1, description="Upstream access token to exchange for a username."),
|
||||
):
|
||||
"""Token-exchange login.
|
||||
|
||||
Verifies an upstream access token by calling the configured
|
||||
``token_info_url`` (see ``auth_login.token_login``), reads the resolved
|
||||
username, then signs the matching local account in — auto-registering it
|
||||
when the username is not yet in the database. Gated by
|
||||
``auth_login.token_login.enabled``.
|
||||
|
||||
Returns the JWT access token in the response body and also sets it as a
|
||||
session cookie for browser callers.
|
||||
"""
|
||||
config = get_app_config().auth_login.token_login
|
||||
if not config.enabled:
|
||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="token 登录未开启")
|
||||
|
||||
client_ip = _get_client_ip(request)
|
||||
_check_rate_limit(client_ip)
|
||||
|
||||
try:
|
||||
username = await _fetch_username_from_token(token)
|
||||
except HTTPException as exc:
|
||||
if exc.status_code == status.HTTP_401_UNAUTHORIZED:
|
||||
_record_login_failure(client_ip)
|
||||
raise
|
||||
|
||||
_record_login_success(client_ip)
|
||||
|
||||
user, created = await _get_or_create_user_by_username(username)
|
||||
|
||||
access_token = create_access_token(str(user.id), token_version=user.token_version)
|
||||
_set_session_cookie(response, access_token, request)
|
||||
|
||||
return UsernameLoginResponse(
|
||||
access_token=access_token,
|
||||
token_type="bearer",
|
||||
expires_in=get_auth_config().token_expiry_days * 24 * 3600,
|
||||
user_id=str(user.id),
|
||||
email=user.email,
|
||||
system_role=user.system_role,
|
||||
needs_setup=user.needs_setup,
|
||||
created=created,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/register", response_model=UserResponse, status_code=status.HTTP_201_CREATED)
|
||||
async def register(request: Request, response: Response, body: RegisterRequest):
|
||||
"""Register a new user account (always 'user' role).
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
"""HTML page favorites API.
|
||||
|
||||
A favorite bookmarks a generated ``scheduled_html_pages`` row and stores a
|
||||
reusable style/layout summary (extracted via GLM5 at favorite time) so a later
|
||||
``html_page`` task can be generated "in the style of" the favorite.
|
||||
|
||||
All routes are owner-scoped; the underlying logic lives on
|
||||
``ScheduledTaskService`` so GLM5 access and the page store stay in one place.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.gateway.deps import get_current_user
|
||||
|
||||
# Importing the ORM models here (at app-module load, before the lifespan runs
|
||||
# ``Base.metadata.create_all``) registers ``scheduled_html_pages`` +
|
||||
# ``html_page_favorites`` with the metadata so the tables are auto-created on
|
||||
# startup. Without this the store's models are only imported later (inside
|
||||
# ``langgraph_runtime``), after create_all has already run — yielding
|
||||
# "no such table: html_page_favorites".
|
||||
from deerflow.persistence.html_pages.model import ( # noqa: F401
|
||||
HtmlPageFavoriteRow,
|
||||
ScheduledHtmlPageRow,
|
||||
)
|
||||
from deerflow.runtime.scheduler import get_scheduled_task_service
|
||||
from deerflow.runtime.scheduler.service import DuplicateFavoriteError
|
||||
|
||||
router = APIRouter(prefix="/api/html-page-favorites", tags=["html-page-favorites"])
|
||||
|
||||
|
||||
class HtmlPageFavoriteCreateRequest(BaseModel):
|
||||
page_id: str = Field(..., description="scheduled_html_pages.id to bookmark")
|
||||
title: str | None = None
|
||||
description: str | None = None
|
||||
tags: list[str] | None = None
|
||||
|
||||
|
||||
async def _user_id(request: Request) -> str:
|
||||
user_id = await get_current_user(request)
|
||||
if not user_id:
|
||||
raise HTTPException(status_code=401, detail="Authentication required")
|
||||
return user_id
|
||||
|
||||
|
||||
@router.get("")
|
||||
async def list_html_page_favorites(request: Request) -> list[dict[str, Any]]:
|
||||
service = get_scheduled_task_service()
|
||||
user_id = await _user_id(request)
|
||||
return await service.list_html_page_favorites(user_id)
|
||||
|
||||
|
||||
@router.post("")
|
||||
async def create_html_page_favorite(body: HtmlPageFavoriteCreateRequest, request: Request) -> dict[str, Any]:
|
||||
service = get_scheduled_task_service()
|
||||
user_id = await _user_id(request)
|
||||
try:
|
||||
favorite = await service.create_html_page_favorite(
|
||||
user_id,
|
||||
body.page_id,
|
||||
title=body.title,
|
||||
description=body.description,
|
||||
tags=body.tags,
|
||||
)
|
||||
except DuplicateFavoriteError:
|
||||
raise HTTPException(status_code=409, detail="该页面已收藏,请勿重复收藏")
|
||||
if favorite is None:
|
||||
raise HTTPException(status_code=404, detail="页面不存在或无权收藏")
|
||||
return favorite
|
||||
|
||||
|
||||
@router.get("/{favorite_id}")
|
||||
async def get_html_page_favorite(favorite_id: str, request: Request) -> dict[str, Any]:
|
||||
service = get_scheduled_task_service()
|
||||
user_id = await _user_id(request)
|
||||
favorite = await service.get_html_page_favorite(user_id, favorite_id)
|
||||
if favorite is None:
|
||||
raise HTTPException(status_code=404, detail=f"Favorite {favorite_id} not found")
|
||||
return favorite
|
||||
|
||||
|
||||
@router.delete("/{favorite_id}")
|
||||
async def delete_html_page_favorite(favorite_id: str, request: Request) -> dict[str, Any]:
|
||||
service = get_scheduled_task_service()
|
||||
user_id = await _user_id(request)
|
||||
deleted = await service.delete_html_page_favorite(user_id, favorite_id)
|
||||
if not deleted:
|
||||
raise HTTPException(status_code=404, detail=f"Favorite {favorite_id} not found")
|
||||
return {"success": True}
|
||||
@@ -243,6 +243,39 @@ async def public_get_run(run_id: str) -> dict[str, Any]:
|
||||
return _redact_run(run)
|
||||
|
||||
|
||||
def _html_payload(run_id: str, page: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
"run_id": run_id,
|
||||
"task_id": page.get("task_id"),
|
||||
"title": page.get("title"),
|
||||
"html": page.get("html_content"),
|
||||
"style_summary": page.get("style_summary"),
|
||||
"layout_summary": page.get("layout_summary"),
|
||||
"page_id": page.get("id"),
|
||||
"created_at": page.get("created_at"),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/runs/{run_id}/html")
|
||||
async def public_get_run_html(run_id: str) -> dict[str, Any]:
|
||||
"""Return the generated HTML document for an ``html_page`` run (anonymous)."""
|
||||
service = get_scheduled_task_service()
|
||||
page = await service.get_html_page_for_run(run_id)
|
||||
if page is None:
|
||||
raise HTTPException(status_code=404, detail="该执行记录没有 HTML 页面")
|
||||
return _html_payload(run_id, page)
|
||||
|
||||
|
||||
@router.get("/{task_id}/latest-html")
|
||||
async def public_get_latest_html(task_id: str) -> dict[str, Any]:
|
||||
"""Return the most recent generated HTML page for a task (anonymous)."""
|
||||
service = get_scheduled_task_service()
|
||||
page = await service.get_latest_html_page_for_task(task_id)
|
||||
if page is None:
|
||||
raise HTTPException(status_code=404, detail="该任务暂无 HTML 页面")
|
||||
return _html_payload(str(page.get("run_id") or ""), page)
|
||||
|
||||
|
||||
@router.get("/runs/{run_id}/files/{file_index}")
|
||||
async def public_get_run_file(run_id: str, file_index: int) -> dict[str, Any]:
|
||||
"""Return a single result-file entry with its inline text content where
|
||||
|
||||
@@ -251,6 +251,29 @@ async def get_scheduled_task_run(run_id: str, request: Request) -> dict:
|
||||
return run
|
||||
|
||||
|
||||
@router.get("/runs/{run_id}/html")
|
||||
async def get_scheduled_task_run_html(run_id: str, request: Request) -> dict:
|
||||
"""Return the generated HTML document for an ``html_page`` run."""
|
||||
service = get_scheduled_task_service()
|
||||
user_id = await _user_id(request)
|
||||
run = await _load_run(service, request, run_id, user_id)
|
||||
if run is None:
|
||||
raise HTTPException(status_code=404, detail=f"Run {run_id} not found")
|
||||
page = await service.get_html_page_for_run(run_id)
|
||||
if page is None:
|
||||
raise HTTPException(status_code=404, detail="该执行记录没有 HTML 页面")
|
||||
return {
|
||||
"run_id": run_id,
|
||||
"task_id": page.get("task_id") or run.get("task_id"),
|
||||
"title": page.get("title"),
|
||||
"html": page.get("html_content"),
|
||||
"style_summary": page.get("style_summary"),
|
||||
"layout_summary": page.get("layout_summary"),
|
||||
"page_id": page.get("id"),
|
||||
"created_at": page.get("created_at"),
|
||||
}
|
||||
|
||||
|
||||
def _event_type_for_checkpoint_message(message: dict) -> str:
|
||||
"""Map a serialized LangChain message to the same ``event_type`` strings
|
||||
the run-event store uses, so the frontend's existing role labels keep
|
||||
|
||||
@@ -30,7 +30,7 @@ from deerflow.skills import Skill
|
||||
from deerflow.skills.cascade import cascade_skill_unpublished_or_deleted
|
||||
from deerflow.skills.installer import SkillAlreadyExistsError
|
||||
from deerflow.skills.security_scanner import scan_skill_content
|
||||
from deerflow.skills.storage import get_or_new_skill_storage
|
||||
from deerflow.skills.storage import SkillStorage, get_or_new_skill_storage
|
||||
from deerflow.skills.types import SKILL_MD_FILE, SkillCategory
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -210,6 +210,12 @@ class CustomSkillUpdateRequest(BaseModel):
|
||||
content: str = Field(..., description="Replacement SKILL.md content")
|
||||
|
||||
|
||||
class SkillDuplicateRequest(BaseModel):
|
||||
"""Request model for duplicating an existing skill into a new custom skill."""
|
||||
|
||||
new_name: str = Field(..., description="Name for the new custom skill copy (lowercase letters/digits, '-'/'_' separators)")
|
||||
|
||||
|
||||
class CustomSkillHistoryResponse(BaseModel):
|
||||
history: list[dict]
|
||||
|
||||
@@ -950,6 +956,121 @@ async def install_skill(
|
||||
raise HTTPException(status_code=500, detail=f"Failed to install skill: {str(e)}")
|
||||
|
||||
|
||||
def _rewrite_skill_frontmatter_name(content: str, new_name: str) -> str:
|
||||
"""Return *content* with the YAML-frontmatter ``name:`` field set to *new_name*.
|
||||
|
||||
A duplicated skill must declare the new name in its SKILL.md frontmatter or
|
||||
``validate_skill_markdown_content`` will reject it (frontmatter name must
|
||||
match the directory name). We rewrite only the first ``name:`` line inside
|
||||
the leading ``---`` block, leaving every other field byte-for-byte intact.
|
||||
"""
|
||||
match = re.match(r"^---\n(.*?)\n---", content, re.DOTALL)
|
||||
if not match:
|
||||
raise ValueError("SKILL.md is missing the leading YAML frontmatter block")
|
||||
fm = match.group(1)
|
||||
new_fm, replaced = re.subn(r"(?m)^(name:\s*).*$", rf"\g<1>{new_name}", fm, count=1)
|
||||
if replaced == 0:
|
||||
new_fm = f"name: {new_name}\n{fm}"
|
||||
return content[: match.start(1)] + new_fm + content[match.end(1) :]
|
||||
|
||||
|
||||
@router.post(
|
||||
"/skills/custom/{source_name}/duplicate",
|
||||
response_model=SkillInstallResponse,
|
||||
summary="Duplicate Skill",
|
||||
description=(
|
||||
"Copy an existing skill (built-in/public OR custom) into a brand-new custom "
|
||||
"skill named ``new_name``, owned by the caller and unpublished. All support "
|
||||
"files are copied verbatim; only the SKILL.md frontmatter name is rewritten. "
|
||||
"The source already passed the install-time security scan, so no re-scan is run."
|
||||
),
|
||||
)
|
||||
async def duplicate_skill(
|
||||
source_name: str,
|
||||
body: SkillDuplicateRequest,
|
||||
request: Request,
|
||||
config: AppConfig = Depends(get_config),
|
||||
skill_store: SkillStore = Depends(get_skill_store),
|
||||
) -> SkillInstallResponse:
|
||||
import shutil
|
||||
|
||||
try:
|
||||
user_id = _current_user_id(request)
|
||||
is_admin = _is_admin_user(request)
|
||||
storage = get_or_new_skill_storage(app_config=config)
|
||||
|
||||
source_name = SkillStorage.validate_skill_name(source_name.replace("\r\n", "").replace("\n", ""))
|
||||
new_name = SkillStorage.validate_skill_name(body.new_name.replace("\r\n", "").replace("\n", ""))
|
||||
|
||||
# Resolve the source directory (custom takes precedence over public, so a
|
||||
# user-shadowed copy is the one duplicated when both exist).
|
||||
if storage.custom_skill_exists(source_name):
|
||||
source_dir = storage.get_custom_skill_dir(source_name)
|
||||
elif storage.public_skill_exists(source_name):
|
||||
source_dir = storage.get_skills_root_path() / SkillCategory.PUBLIC.value / source_name
|
||||
else:
|
||||
raise HTTPException(status_code=404, detail=f"Skill '{source_name}' not found")
|
||||
|
||||
if new_name == source_name or storage.custom_skill_exists(new_name) or storage.public_skill_exists(new_name):
|
||||
raise HTTPException(status_code=409, detail=f"A skill named '{new_name}' already exists")
|
||||
if not is_admin:
|
||||
existing = await skill_store.get_any(new_name)
|
||||
if existing and existing.get("owner_user_id") not in (None, user_id):
|
||||
raise HTTPException(status_code=409, detail=f"Skill '{new_name}' already exists and is owned by another user")
|
||||
|
||||
dest_dir = storage.get_custom_skill_dir(new_name)
|
||||
# copytree refuses to overwrite, which is the behaviour we want — the
|
||||
# existence checks above already guarantee a clean target.
|
||||
shutil.copytree(source_dir, dest_dir)
|
||||
try:
|
||||
source_content = (dest_dir / SKILL_MD_FILE).read_text(encoding="utf-8")
|
||||
new_content = _rewrite_skill_frontmatter_name(source_content, new_name)
|
||||
SkillStorage.validate_skill_markdown_content(new_name, new_content)
|
||||
storage.write_custom_skill(new_name, SKILL_MD_FILE, new_content)
|
||||
except Exception:
|
||||
# Roll back the half-created directory so a failed copy never leaves
|
||||
# an unparseable skill on disk.
|
||||
shutil.rmtree(dest_dir, ignore_errors=True)
|
||||
raise
|
||||
|
||||
await skill_store.ensure_legacy({"name": new_name, "owner_user_id": user_id, "published": False})
|
||||
try:
|
||||
storage.append_history(
|
||||
new_name,
|
||||
{
|
||||
"action": "duplicate",
|
||||
"author": "human",
|
||||
"thread_id": None,
|
||||
"file_path": SKILL_MD_FILE,
|
||||
"prev_content": None,
|
||||
"new_content": new_content,
|
||||
"source_skill": source_name,
|
||||
},
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
from deerflow.skills.usage import mark_uploaded
|
||||
mark_uploaded(new_name)
|
||||
except Exception:
|
||||
pass
|
||||
await refresh_skills_system_prompt_cache_async()
|
||||
return SkillInstallResponse(
|
||||
success=True,
|
||||
skill_name=new_name,
|
||||
message=f"Skill '{source_name}' duplicated as '{new_name}'",
|
||||
overwritten=False,
|
||||
category=SkillCategory.CUSTOM.value,
|
||||
)
|
||||
except HTTPException:
|
||||
raise
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
except Exception as e:
|
||||
logger.error("Failed to duplicate skill %s: %s", source_name, e, exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to duplicate skill: {str(e)}")
|
||||
|
||||
|
||||
@router.post(
|
||||
"/skills/reconcile-custom",
|
||||
response_model=SkillReconcileResponse,
|
||||
|
||||
@@ -228,6 +228,8 @@ class ReferenceSourceResponse(BaseModel):
|
||||
id: str | None = None
|
||||
skillName: str = "backend-reference"
|
||||
mode: str = "citation"
|
||||
resultSetKey: str | None = None
|
||||
resultSetLabel: str | None = None
|
||||
title: str
|
||||
type: str = "result"
|
||||
snippet: str = ""
|
||||
|
||||
@@ -84,13 +84,23 @@ def normalize_input(raw_input: dict[str, Any] | None) -> dict[str, Any]:
|
||||
converted = []
|
||||
for msg in messages:
|
||||
if isinstance(msg, dict):
|
||||
role = msg.get("role", msg.get("type", "user"))
|
||||
content = msg.get("content", "")
|
||||
if role in ("user", "human"):
|
||||
converted.append(HumanMessage(content=content))
|
||||
else:
|
||||
# TODO: handle other message types (system, ai, tool)
|
||||
converted.append(HumanMessage(content=content))
|
||||
# Preserve additional_kwargs (e.g. uploaded `files`, `client_ts`,
|
||||
# `prompt_prefix`) and optional id/name so they round-trip through
|
||||
# the checkpoint to the UI and to UploadsMiddleware. Dropping them
|
||||
# here previously made uploaded files invisible in the chat and
|
||||
# broke the per-message ask-time timestamp.
|
||||
msg_kwargs: dict[str, Any] = {"content": content}
|
||||
extra = msg.get("additional_kwargs")
|
||||
if isinstance(extra, dict) and extra:
|
||||
msg_kwargs["additional_kwargs"] = extra
|
||||
if msg.get("id"):
|
||||
msg_kwargs["id"] = msg["id"]
|
||||
if msg.get("name"):
|
||||
msg_kwargs["name"] = msg["name"]
|
||||
# TODO: handle other message types (system, ai, tool); for now all
|
||||
# inbound dict messages are treated as human turns.
|
||||
converted.append(HumanMessage(**msg_kwargs))
|
||||
else:
|
||||
converted.append(msg)
|
||||
return {**raw_input, "messages": converted}
|
||||
|
||||
@@ -33,7 +33,7 @@ models:
|
||||
max_retries: 2
|
||||
max_tokens: 4096
|
||||
temperature: 1.0
|
||||
supports_vision: true
|
||||
supports_vision: false
|
||||
supports_thinking: true
|
||||
|
||||
- name: deepseek-v4-flash
|
||||
@@ -46,7 +46,7 @@ models:
|
||||
max_retries: 2
|
||||
max_tokens: 4096
|
||||
temperature: 1.0
|
||||
supports_vision: true
|
||||
supports_vision: false
|
||||
supports_thinking: true
|
||||
|
||||
- name: deepseek-v4-pro
|
||||
@@ -59,7 +59,7 @@ models:
|
||||
max_retries: 2
|
||||
max_tokens: 4096
|
||||
temperature: 1.0
|
||||
supports_vision: true
|
||||
supports_vision: false
|
||||
supports_thinking: true
|
||||
- name: MiniMax-M2.7
|
||||
display_name: MiniMax-M2.7
|
||||
@@ -81,6 +81,37 @@ models:
|
||||
temperature: 1.0
|
||||
supports_vision: true
|
||||
supports_thinking: true
|
||||
- name: deepseek-ai/DeepSeek-OCR
|
||||
display_name: deepseek-ai/DeepSeek-OCR
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: deepseek-ai/DeepSeek-OCR
|
||||
api_key: sk-beshfxfxfrllsczpqfrgjbugfbpzxgqgedbehceytpcvvtyb
|
||||
base_url: https://api.siliconflow.cn/v1/
|
||||
max_tokens: 4096
|
||||
temperature: 1.0
|
||||
supports_vision: true
|
||||
supports_thinking: true
|
||||
# 识图(图片识别)配置。
|
||||
# - model_name:指定一个 models[] 中的模型名,作为“默认识图大模型”。view_image 工具会把
|
||||
# 图片交给该模型识别,并把识别出的文字结果返回给主对话模型 —— 这样即使主对话模型不支持
|
||||
# 视觉,也能“看图”。请改成你实际用于识图的模型名。
|
||||
# - 若留空 / 注释掉 model_name,则回退到主对话模型自身的视觉能力(需该模型 supports_vision: true)。
|
||||
# - prompt:不带具体问题时使用的默认识图指令(可选)。
|
||||
# - max_tokens:识图模型输出上限(可选,默认用该模型自身的配置)。
|
||||
vision:
|
||||
model_name: deepseek-ai/DeepSeek-OCR
|
||||
# prompt: 请详细描述这张图片的内容。
|
||||
# max_tokens: 2048
|
||||
|
||||
# 上传文件识别配置。
|
||||
# - auto_convert_documents: true 时,上传的 PDF / Word / Excel / PPT 会在服务端自动转换为
|
||||
# Markdown(.md),agent 通过 read_file / grep 读取其内容 —— 即“识别”文档内容。
|
||||
# 纯文本类(txt / csv / md / json / 代码等)无需转换,agent 用 read_file 直接读取。
|
||||
# 图片(png/jpg/webp)由 view_image + 上面的 vision 识图模型处理。
|
||||
# - pdf_converter: auto(优先 pymupdf4llm,缺失时回退 markitdown)| pymupdf4llm | markitdown
|
||||
uploads:
|
||||
auto_convert_documents: true
|
||||
pdf_converter: auto
|
||||
|
||||
title:
|
||||
enabled: true
|
||||
@@ -156,6 +187,27 @@ skills:
|
||||
agents_api:
|
||||
enabled: true
|
||||
|
||||
# 登录鉴权扩展:在基础登录之上叠加两种自定义模式。
|
||||
auth_login:
|
||||
# 用户名直登口令门:开关打开后,/login/<用户名> 直登必须再带 ?password=口令,
|
||||
# 口令不对直接 401。password 留空则默认口令为 123ewq。仅作用于用户名直登。
|
||||
username_login:
|
||||
require_password: false
|
||||
password: ""
|
||||
# token 换登录:前端用 ?authToken=上游token 进入,后端调用 token_info_url
|
||||
# (把该 token 直接拼成 Authorization: Bearer <token> 头)换出用户名,再登录;
|
||||
# 用户名不在库里时自动注册。
|
||||
token_login:
|
||||
enabled: false
|
||||
token_info_url: "https://ch1.b.uat.4.cn/consumer/login/getTokenInfo"
|
||||
# 已弃用:鉴权头现由前端传入的 token 拼成 "Bearer <token>",此项不再生效,保留仅兼容。
|
||||
service_authorization: "Bearer admin"
|
||||
timeout_seconds: 10
|
||||
# 上游是 HTTPS。UAT/内网常用自签或私有 CA 证书,默认 verify=true 会握手失败导致
|
||||
# 永远 401。此处 uat 域名按自签处理,先关校验;生产请置 true 或配 ca_cert_path。
|
||||
verify_ssl: false
|
||||
ca_cert_path: "" # 可选:PEM 格式 CA 包路径,设置后优先生效并启用校验
|
||||
|
||||
database:
|
||||
backend: sqlite
|
||||
sqlite_dir: .deer-flow/data
|
||||
@@ -164,7 +216,7 @@ database:
|
||||
|
||||
checkpointer:
|
||||
type: sqlite
|
||||
connection_string: D:\05271502\deerflow-server\offline-backend-20260512\backend\.deer-flow\data\deerflow.db
|
||||
connection_string: .deer-flow/data/deerflow.db
|
||||
|
||||
# 记忆子系统:跨会话持久化用户画像与 agent 私有记忆。
|
||||
# 三层覆盖优先级(高->低):
|
||||
|
||||
@@ -0,0 +1,141 @@
|
||||
# 登录鉴权扩展接口文档(auth_login)
|
||||
|
||||
在原有登录之上新增两种自定义登录模式,均由 `config.yaml` 的 `auth_login` 段控制。
|
||||
|
||||
## 配置
|
||||
|
||||
```yaml
|
||||
auth_login:
|
||||
# 功能一:用户名直登口令门
|
||||
username_login:
|
||||
require_password: false # true 时 /login/username 必须带 ?password=
|
||||
password: "" # 共享口令;留空默认 123ewq
|
||||
# 功能二:token 换登录
|
||||
token_login:
|
||||
enabled: false # 总开关
|
||||
token_info_url: "https://ch1.b.uat.4.cn/consumer/login/getTokenInfo"
|
||||
service_authorization: "Bearer admin" # 调上游接口的鉴权头
|
||||
timeout_seconds: 10
|
||||
verify_ssl: false # 上游 HTTPS 证书校验;UAT/内网自签证书需置 false,生产置 true
|
||||
ca_cert_path: "" # 可选:PEM 格式 CA 包路径,设置后优先生效并强制校验
|
||||
```
|
||||
|
||||
> **HTTPS 注意**:`token_info_url` 是 HTTPS。若上游用自签名 / 私有 CA 证书(UAT、内网常见),`verify_ssl: true` 时 httpx 的 TLS 握手会失败,导致 token 登录**恒定 401**。此时应:
|
||||
> - 临时/内网:`verify_ssl: false`(跳过证书校验);
|
||||
> - 正规做法:把私有 CA 证书导出为 PEM,配 `ca_cert_path: /path/to/ca.pem`(优先于 `verify_ssl`,仍做校验);
|
||||
> - 生产对外域名(有效证书):保持 `verify_ssl: true`。
|
||||
>
|
||||
> 握手失败时后端日志会打印具体异常类型(如 `SSLError`)与当前 `verify` 取值,便于排查。
|
||||
|
||||
配置模型:`packages/harness/deerflow/config/login_config.py`。修改 `config.yaml` 后无需重启(`get_app_config()` 按 mtime 自动重载)。
|
||||
|
||||
---
|
||||
|
||||
## 功能一:用户名直登 + 口令门
|
||||
|
||||
在原 `POST /api/v1/auth/login/username` 上叠加一个共享口令校验。`require_password=false`(默认)时行为完全不变(免密直登 + 自动注册)。
|
||||
|
||||
### 请求
|
||||
|
||||
```
|
||||
POST /api/v1/auth/login/username[?password=<共享口令>]
|
||||
Content-Type: application/json
|
||||
|
||||
{ "username": "alice" }
|
||||
```
|
||||
|
||||
- `?password=`:仅当 `require_password=true` 时需要。值需等于 `username_login.password`;该项留空时默认口令为 `123ewq`。
|
||||
- 口令门是**单一共享口令**,只决定"放不放行",不影响登录成的是哪个账号。
|
||||
- 校验失败按 IP 计入登录限速(5 次/5 分钟锁定),与本地密码登录共用同一限速器。
|
||||
|
||||
### 响应
|
||||
|
||||
成功 `200`:
|
||||
|
||||
```json
|
||||
{
|
||||
"access_token": "<JWT>",
|
||||
"token_type": "bearer",
|
||||
"expires_in": 604800,
|
||||
"user_id": "<uuid>",
|
||||
"email": "alice@cm.com",
|
||||
"system_role": "user",
|
||||
"needs_setup": false,
|
||||
"created": false
|
||||
}
|
||||
```
|
||||
|
||||
同时种下 `access_token` HttpOnly Cookie。
|
||||
|
||||
失败:
|
||||
|
||||
| 状态码 | 场景 |
|
||||
|--------|------|
|
||||
| `401` | `require_password=true` 但口令缺失或不匹配 |
|
||||
| `429` | 触发登录限速 |
|
||||
|
||||
### 前端入口
|
||||
|
||||
访问 `/login/<用户名>?password=<口令>`。[LoginPage.tsx](../../../frontend-web/src/pages/LoginPage.tsx) 从 `?password=` 读取并透传给 [`loginByUsername`](../../../frontend-web/src/core/auth/api.ts)。
|
||||
|
||||
---
|
||||
|
||||
## 功能二:token 换登录
|
||||
|
||||
前端用 URL 把上游 token 传进来,后端调上游接口换出用户名再登录。
|
||||
|
||||
### 请求
|
||||
|
||||
```
|
||||
POST /api/v1/auth/login/token?token=<上游access_token>
|
||||
```
|
||||
|
||||
后端据此向 `token_info_url` 发起(与下方 curl 对齐):**前端传入的 token 直接拼成 `Authorization: Bearer <token>` 头**,上游从该头读取令牌(不再用 cookie,也不再用静态 `service_authorization`)。
|
||||
|
||||
```bash
|
||||
curl 'https://ch1.b.uat.4.cn/consumer/login/getTokenInfo' \
|
||||
-X POST \
|
||||
-H 'authorization: Bearer <上游access_token>' \
|
||||
-H 'content-type: application/x-www-form-urlencoded'
|
||||
```
|
||||
|
||||
上游返回示例:
|
||||
|
||||
```json
|
||||
{ "state": "200", "msg": "操作成功!", "data": { "username": "b1admin", "...": "..." } }
|
||||
```
|
||||
|
||||
后端取 `data.username` → 经 `_username_to_email` 映射 → 库内查找(含跨域名 local-part 兜底)→ **查不到则自动注册** → 发本系统 JWT 并种 Cookie。
|
||||
|
||||
### 响应
|
||||
|
||||
成功 `200`:结构同功能一的 `UsernameLoginResponse`(首次自动注册时 `created=true`)。
|
||||
|
||||
失败:
|
||||
|
||||
| 状态码 | 场景 |
|
||||
|--------|------|
|
||||
| `403` | `token_login.enabled=false` |
|
||||
| `401` | 上游调用失败 / HTTP 非 200 / `state != "200"` / 未取到 username |
|
||||
| `429` | 触发登录限速 |
|
||||
| `500` | `token_info_url` 未配置 |
|
||||
|
||||
### 前端入口
|
||||
|
||||
访问 `/login?authToken=<上游token>`(用 `authToken` 而非 `token`,以避开既有的 `tools_token` 参数 `?token=`)。LoginPage 命中 `authToken` 时走 [`loginByToken`](../../../frontend-web/src/core/auth/api.ts),不再调用户名直登。
|
||||
|
||||
---
|
||||
|
||||
## 安全说明
|
||||
|
||||
- 两个端点都在 `auth_middleware._PUBLIC_EXACT_PATHS` 白名单内(免会话鉴权,自身完成鉴权)。
|
||||
- 口令门是**最低限度**的共享密钥防护,适合内网/受控环境;非每用户独立密码。
|
||||
- token 换登录信任上游 `getTokenInfo` 的鉴权结论,务必保证 `token_info_url` 配置正确且走内网/HTTPS。前端传入的 token 会直接作为 `Authorization: Bearer <token>` 发给上游,因此该 token 必须来自可信入口。(`service_authorization` 配置项已不再参与请求,保留仅为兼容,后续可移除。)
|
||||
|
||||
## 相关代码
|
||||
|
||||
- 后端端点 / 助手:`app/gateway/routers/auth.py`(`login_by_username`、`login_by_token`、`_get_or_create_user_by_username`、`_fetch_username_from_token`)
|
||||
- 白名单:`app/gateway/auth_middleware.py`
|
||||
- 配置模型:`packages/harness/deerflow/config/login_config.py`(挂载于 `AppConfig.auth_login`)
|
||||
- 前端:`frontend-web/src/core/auth/api.ts`、`frontend-web/src/pages/LoginPage.tsx`
|
||||
- 测试:`tests/test_login_config.py`、`tests/test_auth_login_modes.py`
|
||||
@@ -6,6 +6,8 @@
|
||||
"code-retrieval": { "enabled": true },
|
||||
"news-briefing": { "enabled": true },
|
||||
"workspace-search": { "enabled": true },
|
||||
"policy-knowledge": { "enabled": true }
|
||||
"policy-knowledge": { "enabled": true },
|
||||
"html-page-builder": { "enabled": true },
|
||||
"html-charts": { "enabled": true }
|
||||
}
|
||||
}
|
||||
+91
-2
@@ -1,8 +1,9 @@
|
||||
"""素材收集专家节点:关键词提取 → 并发搜索 → 相关度排序。"""
|
||||
"""素材收集专家节点:关键词提取 → 并发搜索(含笔记) → 相关度排序。"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
import uuid
|
||||
@@ -21,6 +22,10 @@ from ._utils import get_ai_writing_config, llm_json, safe_get_stream_writer
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_NOTEBOOK_SEARCH_PATH = "/api/v1/source/mentioned-documents/search"
|
||||
_NOTEBOOK_MAX_KEYWORDS = 3 # 每次写作最多对笔记发起的检索轮数
|
||||
_NOTEBOOK_TOP_K = 5 # 每次检索取回的 chunk 数
|
||||
|
||||
# 素材总量上限边界:config.yaml 的 ai_writing.max_materials,非法值回退默认 50
|
||||
_DEFAULT_MAX_MATERIALS = 50
|
||||
_MIN_MAX_MATERIALS = 1
|
||||
@@ -95,6 +100,67 @@ def _notice(title: str, detail: str, raw: str = "") -> dict:
|
||||
return event
|
||||
|
||||
|
||||
async def _search_notebook(
|
||||
keywords: list[str],
|
||||
search_space_id: int,
|
||||
source_ids: list[int],
|
||||
token: str,
|
||||
) -> list[Material]:
|
||||
"""从笔记 API 按关键词检索内容,返回 source_type='notebook' 的 Material 列表。"""
|
||||
import httpx # 懒导入:仅在笔记模式下才需要
|
||||
|
||||
base = (os.getenv("DEER_FLOW_NOTEBOOK_API_BASE") or "").rstrip("/")
|
||||
if not base:
|
||||
logger.warning("DEER_FLOW_NOTEBOOK_API_BASE not set, skipping notebook search")
|
||||
return []
|
||||
|
||||
results: list[Material] = []
|
||||
mat_idx = 0
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=20) as client:
|
||||
for kw in keywords[:_NOTEBOOK_MAX_KEYWORDS]:
|
||||
try:
|
||||
resp = await client.post(
|
||||
f"{base}{_NOTEBOOK_SEARCH_PATH}",
|
||||
headers={"Authorization": f"Bearer {token}"},
|
||||
json={
|
||||
"query": kw,
|
||||
"source_ids": source_ids,
|
||||
"search_space_id": search_space_id,
|
||||
"top_k": _NOTEBOOK_TOP_K,
|
||||
},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
# 解析 documents 列表,每个 chunk 作为独立素材
|
||||
documents: list[dict] = data.get("documents") if isinstance(data, dict) else []
|
||||
for doc in documents or []:
|
||||
doc_title: str = doc.get("title") or "笔记文档"
|
||||
doc_url: str = doc.get("url") or ""
|
||||
for chunk in doc.get("chunks") or []:
|
||||
chunk_content: str = (chunk.get("content") or "").strip()
|
||||
if not chunk_content:
|
||||
continue
|
||||
mat_idx += 1
|
||||
raw_score = chunk.get("score")
|
||||
chunk_score = float(raw_score) if raw_score is not None else max(0.5, 0.95 - (mat_idx - 1) * 0.03)
|
||||
results.append(Material(
|
||||
id=f"nb{mat_idx:03d}",
|
||||
title=doc_title,
|
||||
content=chunk_content[:500],
|
||||
source=f"笔记·{doc_title}",
|
||||
source_type="notebook",
|
||||
relevance_score=chunk_score,
|
||||
url=doc_url,
|
||||
))
|
||||
except Exception as e:
|
||||
logger.warning("notebook search kw=%r failed: %s", kw, e)
|
||||
except Exception as e:
|
||||
logger.warning("notebook search client error: %s", e)
|
||||
|
||||
return results
|
||||
|
||||
|
||||
async def researcher_node(state: AIWritingState, config: RunnableConfig) -> dict:
|
||||
intent = state["user_intent"]
|
||||
article_type = state.get("article_type", "analysis")
|
||||
@@ -181,6 +247,22 @@ async def researcher_node(state: AIWritingState, config: RunnableConfig) -> dict
|
||||
if extra_keywords:
|
||||
keywords = list(dict.fromkeys(keywords + extra_keywords))
|
||||
|
||||
# Step 1.5: 笔记检索(如果传入了 notebook_search_space_id)
|
||||
notebook_search_space_id: int | None = state.get("notebook_search_space_id")
|
||||
notebook_materials: list[Material] = []
|
||||
if notebook_search_space_id:
|
||||
notebook_source_ids: list[int] = state.get("notebook_source_ids") or []
|
||||
notebook_token: str = cfg.get("notebook_token") or ""
|
||||
_emit("正在检索笔记内容…")
|
||||
notebook_materials = await _search_notebook(
|
||||
keywords[:_NOTEBOOK_MAX_KEYWORDS],
|
||||
notebook_search_space_id,
|
||||
notebook_source_ids,
|
||||
notebook_token,
|
||||
)
|
||||
if notebook_materials:
|
||||
_emit(f"从笔记中获取到 {len(notebook_materials)} 条素材")
|
||||
|
||||
# Step 2: 搜索
|
||||
# 技能检索:技能不是素材,是"检索策略 / 视角"。流程:
|
||||
# a. SkillPicker 选出按相关度排序的 skills(推 list_skills + select_skills 事件)
|
||||
@@ -330,10 +412,17 @@ async def researcher_node(state: AIWritingState, config: RunnableConfig) -> dict
|
||||
existing_pkg = state.get("material_package")
|
||||
if is_skill_search:
|
||||
# 技能编排循环已经完成素材累积,直接用 raw_materials_pre 当 raw_materials
|
||||
# 笔记素材追加在技能素材之后
|
||||
raw_materials: list[Material] = raw_materials_pre # type: ignore[name-defined]
|
||||
raw_materials.extend(notebook_materials)
|
||||
else:
|
||||
seen_urls: set[str] = set()
|
||||
raw_materials = []
|
||||
# 笔记素材优先排在最前面,赋予高相关度
|
||||
raw_materials = list(notebook_materials)
|
||||
for m in notebook_materials:
|
||||
if m.get("url"):
|
||||
seen_urls.add(m["url"])
|
||||
|
||||
preserve_existing = existing_pkg and (is_supplement or is_append_only)
|
||||
if preserve_existing:
|
||||
for m in existing_pkg.get("materials", []):
|
||||
|
||||
@@ -143,6 +143,9 @@ class AIWritingState(TypedDict):
|
||||
# 经 checkpoint 持久化 —— interrupt/resume 是新 invocation、其 config 不带原
|
||||
# model_name,存进 state 才能让 resume 后才运行的节点拿到用户模型。
|
||||
model_name: str
|
||||
# 笔记来源(可选):传入后 researcher 阶段会额外从笔记检索素材
|
||||
notebook_search_space_id: Optional[int]
|
||||
notebook_source_ids: Optional[List[int]]
|
||||
|
||||
# Agent A 输出
|
||||
material_package: Optional[MaterialPackage]
|
||||
|
||||
+16
-6
@@ -285,18 +285,28 @@ def _build_middlewares(
|
||||
if resolved_app_config.token_usage.enabled:
|
||||
middlewares.append(TokenUsageMiddleware())
|
||||
|
||||
# Add TitleMiddleware
|
||||
middlewares.append(TitleMiddleware(app_config=resolved_app_config))
|
||||
# Add TitleMiddleware — but skip it for scheduled runs. Each scheduled run
|
||||
# executes on a throwaway thread whose title is never shown to a user, and
|
||||
# generating one would fire an extra LLM call on the *default* model
|
||||
# (config.title.model_name is usually unset → models[0]), which is not the
|
||||
# model the task selected. Skipping it keeps scheduled/HTML runs on exactly
|
||||
# the chosen model.
|
||||
if not cfg.get("is_scheduled_run", False):
|
||||
middlewares.append(TitleMiddleware(app_config=resolved_app_config))
|
||||
|
||||
# Add MemoryMiddleware (after TitleMiddleware) — skip in bootstrap mode (agent creation flow)
|
||||
memory_injection_enabled = cfg.get("memory_injection_enabled", True)
|
||||
if not is_bootstrap and not is_writing_setup:
|
||||
middlewares.append(MemoryMiddleware(agent_name=agent_name, memory_config=resolved_app_config.memory, injection_enabled=memory_injection_enabled))
|
||||
|
||||
# Add ViewImageMiddleware only if the current model supports vision.
|
||||
# Use the resolved runtime model_name from make_lead_agent to avoid stale config values.
|
||||
model_config = resolved_app_config.get_model_config(model_name) if model_name else None
|
||||
if model_config is not None and model_config.supports_vision:
|
||||
# Add ViewImageMiddleware only for the NATIVE-vision path (main model
|
||||
# ingests raw image data). In delegated mode the view_image tool already
|
||||
# returns a text description and stores nothing in `viewed_images`, so the
|
||||
# middleware would otherwise inject a spurious "No images have been viewed."
|
||||
# message. Use the resolved runtime model_name to avoid stale config values.
|
||||
from deerflow.models.vision import is_delegated_vision, vision_enabled
|
||||
|
||||
if vision_enabled(model_name, app_config=resolved_app_config) and not is_delegated_vision(app_config=resolved_app_config):
|
||||
middlewares.append(ViewImageMiddleware())
|
||||
|
||||
# Add DeferredToolFilterMiddleware to hide deferred tool schemas from model binding
|
||||
|
||||
+16
@@ -18,6 +18,12 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
_OUTLINE_PREVIEW_LINES = 5
|
||||
|
||||
# Image files are understood via the `view_image` tool (which routes to the
|
||||
# configured vision model), never by reading their bytes. Listing them with the
|
||||
# generic read_file/grep guidance made the agent try to parse image binaries
|
||||
# with bash/python, which fails and wastes turns.
|
||||
_IMAGE_EXTENSIONS = {".jpg", ".jpeg", ".png", ".webp", ".gif", ".bmp"}
|
||||
|
||||
|
||||
def _extract_outline_for_file(file_path: Path) -> tuple[list[dict], list[str]]:
|
||||
"""Return the document outline and fallback preview for *file_path*.
|
||||
@@ -88,6 +94,16 @@ class UploadsMiddleware(AgentMiddleware[UploadsMiddlewareState]):
|
||||
size_str = f"{size_kb:.1f} KB" if size_kb < 1024 else f"{size_kb / 1024:.1f} MB"
|
||||
lines.append(f"- {file['filename']} ({size_str})")
|
||||
lines.append(f" Path: {file['path']}")
|
||||
|
||||
# Images must be understood through the vision model, not by reading bytes.
|
||||
extension = str(file.get("extension") or Path(file["filename"]).suffix).lower()
|
||||
if extension in _IMAGE_EXTENSIONS:
|
||||
lines.append(f" This is an image. To understand it, call the `view_image` tool with image_path='{file['path']}' (optionally pass a `question`).")
|
||||
lines.append(" The configured vision model will analyze it and return a description — trust that result.")
|
||||
lines.append(" Do NOT read, open, or parse the image with read_file, grep, bash, python, PIL, or any byte-level inspection; those cannot interpret image content.")
|
||||
lines.append("")
|
||||
return
|
||||
|
||||
outline = file.get("outline") or []
|
||||
if outline:
|
||||
truncated = outline[-1].get("truncated", False)
|
||||
|
||||
+6
@@ -139,6 +139,12 @@ class ViewImageMiddleware(AgentMiddleware[ViewImageMiddlewareState]):
|
||||
if not messages:
|
||||
return False
|
||||
|
||||
# Nothing to inject when no image data was captured (e.g. delegated
|
||||
# recognition handled the image in the tool and stored no base64).
|
||||
# Avoids injecting a useless "No images have been viewed." message.
|
||||
if not state.get("viewed_images"):
|
||||
return False
|
||||
|
||||
# Get the last assistant message
|
||||
last_assistant_msg = self._get_last_assistant_message(messages)
|
||||
if not last_assistant_msg:
|
||||
|
||||
@@ -15,6 +15,7 @@ from deerflow.config.database_config import DatabaseConfig
|
||||
from deerflow.config.extensions_config import ExtensionsConfig
|
||||
from deerflow.config.gateway_config import GatewayConfig
|
||||
from deerflow.config.guardrails_config import GuardrailsConfig, load_guardrails_config_from_dict
|
||||
from deerflow.config.login_config import AuthLoginConfig
|
||||
from deerflow.config.memory_config import MemoryConfig, load_memory_config_from_dict
|
||||
from deerflow.config.model_config import ModelConfig
|
||||
from deerflow.config.run_events_config import RunEventsConfig
|
||||
@@ -32,6 +33,7 @@ from deerflow.config.title_config import TitleConfig, load_title_config_from_dic
|
||||
from deerflow.config.token_usage_config import TokenUsageConfig
|
||||
from deerflow.config.tool_config import ToolConfig, ToolGroupConfig
|
||||
from deerflow.config.tool_search_config import ToolSearchConfig, load_tool_search_config_from_dict
|
||||
from deerflow.config.vision_config import VisionConfig
|
||||
|
||||
load_dotenv()
|
||||
|
||||
@@ -104,6 +106,8 @@ class AppConfig(BaseModel):
|
||||
log_level: str = Field(default="info", description="Logging level for deerflow and app modules (debug/info/warning/error); third-party libraries are not affected")
|
||||
token_usage: TokenUsageConfig = Field(default_factory=TokenUsageConfig, description="Token usage tracking configuration")
|
||||
models: list[ModelConfig] = Field(default_factory=list, description="Available models")
|
||||
vision: VisionConfig = Field(default_factory=VisionConfig, description="Image recognition (识图) configuration")
|
||||
auth_login: AuthLoginConfig = Field(default_factory=AuthLoginConfig, description="Custom login modes (username password gate, token exchange)")
|
||||
sandbox: SandboxConfig = Field(description="Sandbox configuration")
|
||||
tools: list[ToolConfig] = Field(default_factory=list, description="Available tools")
|
||||
tool_groups: list[ToolGroupConfig] = Field(default_factory=list, description="Available tool groups")
|
||||
|
||||
+51
@@ -62,6 +62,52 @@ class SkillDisplayTableColumn(BaseModel):
|
||||
model_config = ConfigDict(extra="allow", populate_by_name=True)
|
||||
|
||||
|
||||
class SkillDisplayResultSetConfig(BaseModel):
|
||||
"""Display adapter for one result array inside a skill response."""
|
||||
|
||||
key: str = Field(description="Stable key for this result set")
|
||||
label: str | None = Field(default=None, description="Human-readable result set label")
|
||||
enabled: bool = Field(default=True, description="Whether this result set is enabled")
|
||||
mode: Literal["citation", "list", "table", "card", "custom", "none"] = Field(
|
||||
default="citation",
|
||||
description="How the frontend should render this result set",
|
||||
)
|
||||
result_path: str = Field(default="results", description="Dot path to the array of result items", alias="resultPath")
|
||||
title_field: str | None = Field(default=None, description="Field used as item title", alias="titleField")
|
||||
summary_field: str | None = Field(default=None, description="Field used as short summary/snippet", alias="summaryField")
|
||||
content_field: str | None = Field(default=None, description="Field used as full item content", alias="contentField")
|
||||
source_field: str | None = Field(default=None, description="Field used as source label", alias="sourceField")
|
||||
author_field: str | None = Field(default=None, description="Field used as author/byline label", alias="authorField")
|
||||
type_field: str | None = Field(default=None, description="Field used as source/type badge", alias="typeField")
|
||||
time_field: str | None = Field(default=None, description="Field used as publication/update time", alias="timeField")
|
||||
score_field: str | None = Field(default=None, description="Field used as relevance score", alias="scoreField")
|
||||
url_field: str | None = Field(default=None, description="Field containing a direct URL", alias="urlField")
|
||||
id_field: str | None = Field(default=None, description="Field containing an ID for template URLs", alias="idField")
|
||||
link_strategy: Literal["auto", "url", "id"] = Field(
|
||||
default="auto",
|
||||
description="Whether links use a URL field, an ID template, or automatic fallback",
|
||||
alias="linkStrategy",
|
||||
)
|
||||
link_template: str | None = Field(default=None, description="Template used when linkStrategy is id", alias="linkTemplate")
|
||||
citation_token: Literal["doubleBracket", "singleBracket"] = Field(
|
||||
default="singleBracket",
|
||||
description="Citation token style expected in model output",
|
||||
alias="citationToken",
|
||||
)
|
||||
max_items: int = Field(default=100, ge=1, le=100, description="Maximum displayed items", alias="maxItems")
|
||||
custom_instruction: str = Field(
|
||||
default="",
|
||||
description="User-authored instruction for custom rendering schema generation",
|
||||
alias="customInstruction",
|
||||
)
|
||||
table_columns: list[SkillDisplayTableColumn] = Field(
|
||||
default_factory=list,
|
||||
description="Columns used when mode is table",
|
||||
alias="tableColumns",
|
||||
)
|
||||
model_config = ConfigDict(extra="allow", populate_by_name=True)
|
||||
|
||||
|
||||
class SkillDisplayConfig(BaseModel):
|
||||
"""Frontend display adapter for structured skill results."""
|
||||
|
||||
@@ -103,6 +149,11 @@ class SkillDisplayConfig(BaseModel):
|
||||
description="Columns used when mode is table",
|
||||
alias="tableColumns",
|
||||
)
|
||||
result_sets: list[SkillDisplayResultSetConfig] = Field(
|
||||
default_factory=list,
|
||||
description="Optional per-result-array display adapters",
|
||||
alias="resultSets",
|
||||
)
|
||||
model_config = ConfigDict(extra="allow", populate_by_name=True)
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
# Fallback shared password for the username-login gate. Used when
|
||||
# ``UsernameLoginConfig.password`` is left empty but ``require_password`` is on.
|
||||
DEFAULT_USERNAME_LOGIN_PASSWORD = "123ewq"
|
||||
|
||||
|
||||
class UsernameLoginConfig(BaseModel):
|
||||
"""Gate for the passwordless ``POST /api/v1/auth/login/username`` flow.
|
||||
|
||||
By default this flow logs a user in (auto-registering if needed) from a
|
||||
username alone — convenient, but it lets anyone sign in as any username.
|
||||
Turning ``require_password`` on forces callers to additionally pass a
|
||||
shared secret as ``?password=...``; mismatches are rejected with 401.
|
||||
"""
|
||||
|
||||
require_password: bool = Field(
|
||||
default=False,
|
||||
description="When true, login/username requires a matching ?password= query value.",
|
||||
)
|
||||
password: str = Field(
|
||||
default="",
|
||||
description=f"Shared secret for the username-login gate. Empty falls back to '{DEFAULT_USERNAME_LOGIN_PASSWORD}'.",
|
||||
)
|
||||
|
||||
@property
|
||||
def effective_password(self) -> str:
|
||||
"""The password actually checked at login time (config value or default)."""
|
||||
return self.password or DEFAULT_USERNAME_LOGIN_PASSWORD
|
||||
|
||||
|
||||
class TokenLoginConfig(BaseModel):
|
||||
"""Token-exchange login.
|
||||
|
||||
The frontend passes an upstream access token via the URL. The gateway calls
|
||||
``token_info_url`` with that token placed directly in the
|
||||
``Authorization: Bearer <token>`` header, reads ``data.username`` from the
|
||||
response, then signs the matching local user in (auto-registering if the
|
||||
username is not yet in the database).
|
||||
"""
|
||||
|
||||
enabled: bool = Field(
|
||||
default=False,
|
||||
description="Master switch for the POST /api/v1/auth/login/token endpoint.",
|
||||
)
|
||||
token_info_url: str = Field(
|
||||
default="",
|
||||
description="Upstream endpoint that resolves an access token to a username.",
|
||||
)
|
||||
service_authorization: str = Field(
|
||||
default="Bearer admin",
|
||||
description=(
|
||||
"Deprecated/unused: the Authorization header is now built from the "
|
||||
"caller's token as 'Bearer <token>'. Retained for backward compat."
|
||||
),
|
||||
)
|
||||
timeout_seconds: float = Field(
|
||||
default=10.0,
|
||||
gt=0,
|
||||
description="HTTP timeout (seconds) for the token_info_url call.",
|
||||
)
|
||||
verify_ssl: bool = Field(
|
||||
default=True,
|
||||
description=(
|
||||
"Whether to verify the upstream TLS certificate. Set false for "
|
||||
"internal/UAT hosts with self-signed or private-CA certs. Leave "
|
||||
"true in production."
|
||||
),
|
||||
)
|
||||
ca_cert_path: str = Field(
|
||||
default="",
|
||||
description=(
|
||||
"Optional path to a CA bundle (PEM) used to verify the upstream "
|
||||
"certificate. Takes precedence over verify_ssl when set."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class AuthLoginConfig(BaseModel):
|
||||
"""Container for the custom login modes layered on top of the base auth."""
|
||||
|
||||
username_login: UsernameLoginConfig = Field(default_factory=UsernameLoginConfig)
|
||||
token_login: TokenLoginConfig = Field(default_factory=TokenLoginConfig)
|
||||
@@ -0,0 +1,29 @@
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
# Default instruction sent to the vision model when the agent does not provide a
|
||||
# more specific question. Kept in Chinese to match the product's primary locale.
|
||||
DEFAULT_VISION_PROMPT = "请仔细观察这张图片,尽可能详细、准确地描述其中的全部内容,包括文字、图表、物体、场景与布局等关键信息。"
|
||||
|
||||
|
||||
class VisionConfig(BaseModel):
|
||||
"""Configuration for image recognition (识图).
|
||||
|
||||
When ``model_name`` names a model from ``models[]``, the lead agent's
|
||||
``view_image`` tool delegates image understanding to that dedicated model
|
||||
and returns a text description — so a text-only chat model can still "see"
|
||||
uploaded images. When ``model_name`` is unset, vision falls back to the main
|
||||
chat model, which only works when that model has ``supports_vision: true``.
|
||||
"""
|
||||
|
||||
model_name: str | None = Field(
|
||||
default=None,
|
||||
description="Name of the model (from models[]) used for image recognition. When unset, falls back to the main chat model if it supports vision.",
|
||||
)
|
||||
prompt: str = Field(
|
||||
default=DEFAULT_VISION_PROMPT,
|
||||
description="Default instruction for the vision model when no per-call question is given.",
|
||||
)
|
||||
max_tokens: int | None = Field(
|
||||
default=None,
|
||||
description="Optional override for the vision model's max output tokens; falls back to the model's own config when unset.",
|
||||
)
|
||||
@@ -1,3 +1,10 @@
|
||||
from .factory import create_chat_model
|
||||
from .vision import is_delegated_vision, recognize_image, resolve_vision_model_name, vision_enabled
|
||||
|
||||
__all__ = ["create_chat_model"]
|
||||
__all__ = [
|
||||
"create_chat_model",
|
||||
"is_delegated_vision",
|
||||
"recognize_image",
|
||||
"resolve_vision_model_name",
|
||||
"vision_enabled",
|
||||
]
|
||||
@@ -0,0 +1,121 @@
|
||||
"""Image recognition (识图) via a configurable vision model.
|
||||
|
||||
When ``config.yaml`` defines ``vision.model_name``, the lead agent's
|
||||
``view_image`` tool delegates image understanding to that dedicated model and
|
||||
returns a text description — so a text-only chat model can still "see" uploaded
|
||||
images. When no dedicated model is configured, vision falls back to the main
|
||||
chat model, which only works when that model has ``supports_vision: true``.
|
||||
"""
|
||||
|
||||
import logging
|
||||
|
||||
from deerflow.config import get_app_config
|
||||
from deerflow.config.app_config import AppConfig
|
||||
from deerflow.config.vision_config import DEFAULT_VISION_PROMPT
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _configured_vision_model(config: AppConfig) -> str | None:
|
||||
"""Return a valid dedicated vision model name from config, or None."""
|
||||
vision_cfg = getattr(config, "vision", None)
|
||||
name = getattr(vision_cfg, "model_name", None) if vision_cfg else None
|
||||
if not name:
|
||||
return None
|
||||
if config.get_model_config(name) is None:
|
||||
logger.warning("vision.model_name %r is not defined in models[]; ignoring", name)
|
||||
return None
|
||||
return name
|
||||
|
||||
|
||||
def resolve_vision_model_name(model_name: str | None, *, app_config: AppConfig | None = None) -> str | None:
|
||||
"""Resolve which model should handle image recognition for this run.
|
||||
|
||||
Priority:
|
||||
1. A dedicated ``vision.model_name`` configured in ``config.yaml``.
|
||||
2. The main chat model, only if it declares ``supports_vision: true``.
|
||||
3. ``None`` — vision is unavailable.
|
||||
"""
|
||||
config = app_config or get_app_config()
|
||||
configured = _configured_vision_model(config)
|
||||
if configured:
|
||||
return configured
|
||||
if model_name:
|
||||
mc = config.get_model_config(model_name)
|
||||
if mc is not None and mc.supports_vision:
|
||||
return model_name
|
||||
return None
|
||||
|
||||
|
||||
def vision_enabled(model_name: str | None, *, app_config: AppConfig | None = None) -> bool:
|
||||
"""Whether the ``view_image`` tool/middleware should be available this run."""
|
||||
return resolve_vision_model_name(model_name, app_config=app_config) is not None
|
||||
|
||||
|
||||
def is_delegated_vision(*, app_config: AppConfig | None = None) -> bool:
|
||||
"""Whether image recognition is delegated to a dedicated vision model.
|
||||
|
||||
True when ``vision.model_name`` is configured and valid — in that case
|
||||
``view_image`` calls the dedicated model and returns a text description
|
||||
instead of injecting raw image data into the main chat model.
|
||||
"""
|
||||
config = app_config or get_app_config()
|
||||
return _configured_vision_model(config) is not None
|
||||
|
||||
|
||||
def _extract_text(message) -> str:
|
||||
"""Flatten an LLM response message into plain text."""
|
||||
content = getattr(message, "content", message)
|
||||
if isinstance(content, str):
|
||||
return content.strip()
|
||||
if isinstance(content, list):
|
||||
parts: list[str] = []
|
||||
for block in content:
|
||||
if isinstance(block, str):
|
||||
parts.append(block)
|
||||
elif isinstance(block, dict) and block.get("type") == "text":
|
||||
parts.append(str(block.get("text", "")))
|
||||
return "\n".join(p for p in parts if p).strip()
|
||||
return str(content).strip()
|
||||
|
||||
|
||||
def recognize_image(
|
||||
image_base64: str,
|
||||
mime_type: str,
|
||||
*,
|
||||
question: str | None = None,
|
||||
app_config: AppConfig | None = None,
|
||||
) -> str:
|
||||
"""Describe/analyze an image using the configured dedicated vision model.
|
||||
|
||||
Returns the model's text answer. Raises ``ValueError`` if no dedicated
|
||||
vision model is configured, and propagates model/network errors to the
|
||||
caller for handling.
|
||||
"""
|
||||
from langchain_core.messages import HumanMessage
|
||||
|
||||
from deerflow.models.factory import create_chat_model
|
||||
|
||||
config = app_config or get_app_config()
|
||||
model_name = _configured_vision_model(config)
|
||||
if model_name is None:
|
||||
raise ValueError("No dedicated vision model is configured (vision.model_name).")
|
||||
|
||||
vision_cfg = getattr(config, "vision", None)
|
||||
default_prompt = getattr(vision_cfg, "prompt", None) or DEFAULT_VISION_PROMPT
|
||||
prompt = (question or "").strip() or default_prompt
|
||||
|
||||
extra: dict = {}
|
||||
max_tokens = getattr(vision_cfg, "max_tokens", None) if vision_cfg else None
|
||||
if max_tokens:
|
||||
extra["max_tokens"] = max_tokens
|
||||
|
||||
model = create_chat_model(name=model_name, thinking_enabled=False, app_config=config, **extra)
|
||||
message = HumanMessage(
|
||||
content=[
|
||||
{"type": "text", "text": prompt},
|
||||
{"type": "image_url", "image_url": {"url": f"data:{mime_type};base64,{image_base64}"}},
|
||||
]
|
||||
)
|
||||
result = model.invoke([message])
|
||||
return _extract_text(result)
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
"""Generated HTML page + favorite persistence."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from deerflow.persistence.html_pages.base import HtmlPageStore
|
||||
from deerflow.persistence.html_pages.memory import MemoryHtmlPageStore
|
||||
from deerflow.persistence.html_pages.model import HtmlPageFavoriteRow, ScheduledHtmlPageRow
|
||||
from deerflow.persistence.html_pages.sql import HtmlPageRepository
|
||||
|
||||
__all__ = [
|
||||
"HtmlPageStore",
|
||||
"HtmlPageRepository",
|
||||
"MemoryHtmlPageStore",
|
||||
"HtmlPageFavoriteRow",
|
||||
"ScheduledHtmlPageRow",
|
||||
"make_html_page_store",
|
||||
]
|
||||
|
||||
|
||||
def make_html_page_store(session_factory):
|
||||
if session_factory is None:
|
||||
return MemoryHtmlPageStore()
|
||||
return HtmlPageRepository(session_factory)
|
||||
+53
@@ -0,0 +1,53 @@
|
||||
"""Abstract persistence API for generated HTML pages and favorites."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Any
|
||||
|
||||
|
||||
class HtmlPageStore(ABC):
|
||||
# --- Generated pages ---
|
||||
|
||||
@abstractmethod
|
||||
async def create_page(self, data: dict[str, Any]) -> dict[str, Any]:
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def get_page(self, page_id: str) -> dict[str, Any] | None:
|
||||
"""Return a page by id without restricting by owner."""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def get_page_by_run(self, run_id: str) -> dict[str, Any] | None:
|
||||
"""Return the page produced by a given run, if any."""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def latest_page_for_task(self, task_id: str) -> dict[str, Any] | None:
|
||||
"""Return the most recently created page for a task, if any."""
|
||||
raise NotImplementedError
|
||||
|
||||
# --- Favorites ---
|
||||
|
||||
@abstractmethod
|
||||
async def create_favorite(self, data: dict[str, Any]) -> dict[str, Any]:
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def list_favorites(self, user_id: str, *, limit: int = 100) -> list[dict[str, Any]]:
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def get_favorite(self, favorite_id: str) -> dict[str, Any] | None:
|
||||
"""Return a favorite by id without restricting by owner."""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def get_favorite_by_page(self, user_id: str, page_id: str) -> dict[str, Any] | None:
|
||||
"""Return the user's existing favorite for a page, if any (dedup check)."""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def delete_favorite(self, favorite_id: str, user_id: str) -> bool:
|
||||
raise NotImplementedError
|
||||
+102
@@ -0,0 +1,102 @@
|
||||
"""In-memory HtmlPageStore for no-database / test deployments."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from deerflow.persistence.html_pages.base import HtmlPageStore
|
||||
|
||||
|
||||
def _now_iso() -> str:
|
||||
return datetime.now(UTC).isoformat()
|
||||
|
||||
|
||||
class MemoryHtmlPageStore(HtmlPageStore):
|
||||
def __init__(self) -> None:
|
||||
self._pages: dict[str, dict[str, Any]] = {}
|
||||
self._favorites: dict[str, dict[str, Any]] = {}
|
||||
|
||||
# --- Generated pages ---
|
||||
|
||||
async def create_page(self, data: dict[str, Any]) -> dict[str, Any]:
|
||||
row = {
|
||||
"id": data["id"],
|
||||
"task_id": data["task_id"],
|
||||
"run_id": data["run_id"],
|
||||
"user_id": data["user_id"],
|
||||
"title": data.get("title") or "",
|
||||
"html_content": data.get("html_content") or "",
|
||||
"html_file_path": data.get("html_file_path"),
|
||||
"data_snapshot": data.get("data_snapshot") or {},
|
||||
"prompt_snapshot": data.get("prompt_snapshot") or {},
|
||||
"reference_favorite_id": data.get("reference_favorite_id"),
|
||||
"style_summary": data.get("style_summary"),
|
||||
"layout_summary": data.get("layout_summary"),
|
||||
"created_at": _now_iso(),
|
||||
"updated_at": _now_iso(),
|
||||
}
|
||||
self._pages[row["id"]] = row
|
||||
return copy.deepcopy(row)
|
||||
|
||||
async def get_page(self, page_id: str) -> dict[str, Any] | None:
|
||||
row = self._pages.get(page_id)
|
||||
return copy.deepcopy(row) if row is not None else None
|
||||
|
||||
async def get_page_by_run(self, run_id: str) -> dict[str, Any] | None:
|
||||
matches = [r for r in self._pages.values() if r["run_id"] == run_id]
|
||||
if not matches:
|
||||
return None
|
||||
matches.sort(key=lambda r: r["created_at"], reverse=True)
|
||||
return copy.deepcopy(matches[0])
|
||||
|
||||
async def latest_page_for_task(self, task_id: str) -> dict[str, Any] | None:
|
||||
matches = [r for r in self._pages.values() if r["task_id"] == task_id]
|
||||
if not matches:
|
||||
return None
|
||||
matches.sort(key=lambda r: r["created_at"], reverse=True)
|
||||
return copy.deepcopy(matches[0])
|
||||
|
||||
# --- Favorites ---
|
||||
|
||||
async def create_favorite(self, data: dict[str, Any]) -> dict[str, Any]:
|
||||
row = {
|
||||
"id": data["id"],
|
||||
"user_id": data["user_id"],
|
||||
"page_id": data["page_id"],
|
||||
"title": data.get("title") or "",
|
||||
"description": data.get("description"),
|
||||
"tags": data.get("tags") or [],
|
||||
"html_snapshot": data.get("html_snapshot") or "",
|
||||
"style_summary": data.get("style_summary"),
|
||||
"layout_summary": data.get("layout_summary"),
|
||||
"content_structure_summary": data.get("content_structure_summary"),
|
||||
"reuse_prompt": data.get("reuse_prompt"),
|
||||
"created_at": _now_iso(),
|
||||
"updated_at": _now_iso(),
|
||||
}
|
||||
self._favorites[row["id"]] = row
|
||||
return copy.deepcopy(row)
|
||||
|
||||
async def list_favorites(self, user_id: str, *, limit: int = 100) -> list[dict[str, Any]]:
|
||||
matches = [r for r in self._favorites.values() if r["user_id"] == user_id]
|
||||
matches.sort(key=lambda r: r["created_at"], reverse=True)
|
||||
return [copy.deepcopy(r) for r in matches[: max(1, limit)]]
|
||||
|
||||
async def get_favorite(self, favorite_id: str) -> dict[str, Any] | None:
|
||||
row = self._favorites.get(favorite_id)
|
||||
return copy.deepcopy(row) if row is not None else None
|
||||
|
||||
async def get_favorite_by_page(self, user_id: str, page_id: str) -> dict[str, Any] | None:
|
||||
for row in self._favorites.values():
|
||||
if row["user_id"] == user_id and row["page_id"] == page_id:
|
||||
return copy.deepcopy(row)
|
||||
return None
|
||||
|
||||
async def delete_favorite(self, favorite_id: str, user_id: str) -> bool:
|
||||
row = self._favorites.get(favorite_id)
|
||||
if row is None or row["user_id"] != user_id:
|
||||
return False
|
||||
del self._favorites[favorite_id]
|
||||
return True
|
||||
+65
@@ -0,0 +1,65 @@
|
||||
"""ORM models for generated HTML pages and user favorites.
|
||||
|
||||
These back the "HTML 页面定时任务" feature: a scheduled task of kind
|
||||
``html_page`` renders a full HTML document each run. Every rendered page is
|
||||
recorded in :class:`ScheduledHtmlPageRow`; a user may bookmark one into
|
||||
:class:`HtmlPageFavoriteRow`, which also stores a reusable style/layout
|
||||
summary so a later task can be generated "in the style of" the favorite.
|
||||
|
||||
Both tables are brand new, so ``Base.metadata.create_all`` creates them on the
|
||||
next startup without touching any existing schema.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from sqlalchemy import String, Text
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from deerflow.persistence.base import Base
|
||||
from deerflow.persistence.types import BeijingDateTime, PortableJSON, PortableLongText
|
||||
|
||||
|
||||
class ScheduledHtmlPageRow(Base):
|
||||
"""One HTML page produced by a single run of an ``html_page`` task."""
|
||||
|
||||
__tablename__ = "scheduled_html_pages"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
task_id: Mapped[str] = mapped_column(String(64), index=True)
|
||||
run_id: Mapped[str] = mapped_column(String(64), index=True)
|
||||
user_id: Mapped[str] = mapped_column(String(64), index=True)
|
||||
title: Mapped[str] = mapped_column(String(255))
|
||||
# Full HTML document — can be large, so LONGTEXT on MySQL.
|
||||
html_content: Mapped[str] = mapped_column(PortableLongText())
|
||||
html_file_path: Mapped[str | None] = mapped_column(Text)
|
||||
data_snapshot_json: Mapped[dict | None] = mapped_column(PortableJSON(), default=dict)
|
||||
prompt_snapshot_json: Mapped[dict | None] = mapped_column(PortableJSON(), default=dict)
|
||||
reference_favorite_id: Mapped[str | None] = mapped_column(String(64), index=True)
|
||||
style_summary: Mapped[str | None] = mapped_column(Text)
|
||||
layout_summary: Mapped[str | None] = mapped_column(Text)
|
||||
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), default=lambda: datetime.now(UTC))
|
||||
updated_at: Mapped[datetime] = mapped_column(BeijingDateTime(), default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC))
|
||||
|
||||
|
||||
class HtmlPageFavoriteRow(Base):
|
||||
"""A user-bookmarked HTML page, with a reusable style/layout summary."""
|
||||
|
||||
__tablename__ = "html_page_favorites"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
user_id: Mapped[str] = mapped_column(String(64), index=True)
|
||||
page_id: Mapped[str] = mapped_column(String(64), index=True)
|
||||
title: Mapped[str] = mapped_column(String(255))
|
||||
description: Mapped[str | None] = mapped_column(Text)
|
||||
tags_json: Mapped[list | None] = mapped_column(PortableJSON(), default=list)
|
||||
# Snapshot of the page HTML at favorite time, decoupled from the source run
|
||||
# so deleting the run never breaks the favorite reference.
|
||||
html_snapshot: Mapped[str] = mapped_column(PortableLongText())
|
||||
style_summary: Mapped[str | None] = mapped_column(Text)
|
||||
layout_summary: Mapped[str | None] = mapped_column(Text)
|
||||
content_structure_summary: Mapped[str | None] = mapped_column(Text)
|
||||
reuse_prompt: Mapped[str | None] = mapped_column(Text)
|
||||
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), default=lambda: datetime.now(UTC))
|
||||
updated_at: Mapped[datetime] = mapped_column(BeijingDateTime(), default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC))
|
||||
+141
@@ -0,0 +1,141 @@
|
||||
"""SQLAlchemy repository for generated HTML pages and favorites."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import delete, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
from deerflow.persistence.html_pages.base import HtmlPageStore
|
||||
from deerflow.persistence.html_pages.model import HtmlPageFavoriteRow, ScheduledHtmlPageRow
|
||||
|
||||
|
||||
def _to_dict(row) -> dict[str, Any]:
|
||||
data = row.to_dict()
|
||||
for key in ("created_at", "updated_at"):
|
||||
value = data.get(key)
|
||||
if isinstance(value, datetime):
|
||||
data[key] = value.isoformat()
|
||||
# Normalize the JSON column names to API-friendly keys.
|
||||
if "data_snapshot_json" in data:
|
||||
data["data_snapshot"] = data.pop("data_snapshot_json") or {}
|
||||
if "prompt_snapshot_json" in data:
|
||||
data["prompt_snapshot"] = data.pop("prompt_snapshot_json") or {}
|
||||
if "tags_json" in data:
|
||||
data["tags"] = data.pop("tags_json") or []
|
||||
return data
|
||||
|
||||
|
||||
class HtmlPageRepository(HtmlPageStore):
|
||||
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
|
||||
self._sf = session_factory
|
||||
|
||||
# --- Generated pages ---
|
||||
|
||||
async def create_page(self, data: dict[str, Any]) -> dict[str, Any]:
|
||||
row = ScheduledHtmlPageRow(
|
||||
id=data["id"],
|
||||
task_id=data["task_id"],
|
||||
run_id=data["run_id"],
|
||||
user_id=data["user_id"],
|
||||
title=data.get("title") or "",
|
||||
html_content=data.get("html_content") or "",
|
||||
html_file_path=data.get("html_file_path"),
|
||||
data_snapshot_json=data.get("data_snapshot") or {},
|
||||
prompt_snapshot_json=data.get("prompt_snapshot") or {},
|
||||
reference_favorite_id=data.get("reference_favorite_id"),
|
||||
style_summary=data.get("style_summary"),
|
||||
layout_summary=data.get("layout_summary"),
|
||||
)
|
||||
async with self._sf() as session:
|
||||
session.add(row)
|
||||
await session.commit()
|
||||
await session.refresh(row)
|
||||
return _to_dict(row)
|
||||
|
||||
async def get_page(self, page_id: str) -> dict[str, Any] | None:
|
||||
async with self._sf() as session:
|
||||
row = await session.get(ScheduledHtmlPageRow, page_id)
|
||||
return _to_dict(row) if row is not None else None
|
||||
|
||||
async def get_page_by_run(self, run_id: str) -> dict[str, Any] | None:
|
||||
async with self._sf() as session:
|
||||
result = await session.execute(
|
||||
select(ScheduledHtmlPageRow)
|
||||
.where(ScheduledHtmlPageRow.run_id == run_id)
|
||||
.order_by(ScheduledHtmlPageRow.created_at.desc())
|
||||
.limit(1)
|
||||
)
|
||||
row = result.scalar_one_or_none()
|
||||
return _to_dict(row) if row is not None else None
|
||||
|
||||
async def latest_page_for_task(self, task_id: str) -> dict[str, Any] | None:
|
||||
async with self._sf() as session:
|
||||
result = await session.execute(
|
||||
select(ScheduledHtmlPageRow)
|
||||
.where(ScheduledHtmlPageRow.task_id == task_id)
|
||||
.order_by(ScheduledHtmlPageRow.created_at.desc())
|
||||
.limit(1)
|
||||
)
|
||||
row = result.scalar_one_or_none()
|
||||
return _to_dict(row) if row is not None else None
|
||||
|
||||
# --- Favorites ---
|
||||
|
||||
async def create_favorite(self, data: dict[str, Any]) -> dict[str, Any]:
|
||||
row = HtmlPageFavoriteRow(
|
||||
id=data["id"],
|
||||
user_id=data["user_id"],
|
||||
page_id=data["page_id"],
|
||||
title=data.get("title") or "",
|
||||
description=data.get("description"),
|
||||
tags_json=data.get("tags") or [],
|
||||
html_snapshot=data.get("html_snapshot") or "",
|
||||
style_summary=data.get("style_summary"),
|
||||
layout_summary=data.get("layout_summary"),
|
||||
content_structure_summary=data.get("content_structure_summary"),
|
||||
reuse_prompt=data.get("reuse_prompt"),
|
||||
)
|
||||
async with self._sf() as session:
|
||||
session.add(row)
|
||||
await session.commit()
|
||||
await session.refresh(row)
|
||||
return _to_dict(row)
|
||||
|
||||
async def list_favorites(self, user_id: str, *, limit: int = 100) -> list[dict[str, Any]]:
|
||||
async with self._sf() as session:
|
||||
result = await session.execute(
|
||||
select(HtmlPageFavoriteRow)
|
||||
.where(HtmlPageFavoriteRow.user_id == user_id)
|
||||
.order_by(HtmlPageFavoriteRow.created_at.desc())
|
||||
.limit(max(1, limit))
|
||||
)
|
||||
return [_to_dict(row) for row in result.scalars()]
|
||||
|
||||
async def get_favorite(self, favorite_id: str) -> dict[str, Any] | None:
|
||||
async with self._sf() as session:
|
||||
row = await session.get(HtmlPageFavoriteRow, favorite_id)
|
||||
return _to_dict(row) if row is not None else None
|
||||
|
||||
async def get_favorite_by_page(self, user_id: str, page_id: str) -> dict[str, Any] | None:
|
||||
async with self._sf() as session:
|
||||
result = await session.execute(
|
||||
select(HtmlPageFavoriteRow)
|
||||
.where(HtmlPageFavoriteRow.user_id == user_id, HtmlPageFavoriteRow.page_id == page_id)
|
||||
.limit(1)
|
||||
)
|
||||
row = result.scalar_one_or_none()
|
||||
return _to_dict(row) if row is not None else None
|
||||
|
||||
async def delete_favorite(self, favorite_id: str, user_id: str) -> bool:
|
||||
async with self._sf() as session:
|
||||
result = await session.execute(
|
||||
delete(HtmlPageFavoriteRow).where(
|
||||
HtmlPageFavoriteRow.id == favorite_id,
|
||||
HtmlPageFavoriteRow.user_id == user_id,
|
||||
)
|
||||
)
|
||||
await session.commit()
|
||||
return (result.rowcount or 0) > 0
|
||||
@@ -7,7 +7,11 @@ import re
|
||||
from typing import Any
|
||||
from urllib.parse import quote
|
||||
|
||||
from deerflow.config.extensions_config import SkillDisplayConfig, get_extensions_config
|
||||
from deerflow.config.extensions_config import (
|
||||
SkillDisplayConfig,
|
||||
SkillDisplayResultSetConfig,
|
||||
get_extensions_config,
|
||||
)
|
||||
|
||||
|
||||
_SKILL_PATH_RE = re.compile(r"[\\/]skills[\\/](?:public|custom)[\\/]([^\\/]+)(?:[\\/]|$)")
|
||||
@@ -222,7 +226,11 @@ def adapt_reference_batches_with_displays(
|
||||
)
|
||||
except Exception:
|
||||
continue
|
||||
if not config.enabled or config.mode == "none":
|
||||
has_enabled_result_set = any(
|
||||
result_set.enabled and result_set.mode != "none"
|
||||
for result_set in config.result_sets
|
||||
)
|
||||
if not config.enabled or (config.mode == "none" and not has_enabled_result_set):
|
||||
continue
|
||||
normalized_displays[skill_name] = config
|
||||
|
||||
@@ -257,13 +265,30 @@ def adapt_reference_batches_with_displays(
|
||||
raw = source.get("raw")
|
||||
if display is not None and isinstance(raw, dict):
|
||||
index_value = int(source.get("index") or len(sources) + 1)
|
||||
display_for_source: SkillDisplayConfig | SkillDisplayResultSetConfig = display
|
||||
result_set_key = source.get("resultSetKey")
|
||||
if result_set_key:
|
||||
display_for_source = next(
|
||||
(
|
||||
result_set
|
||||
for result_set in display.result_sets
|
||||
if result_set.key == result_set_key
|
||||
),
|
||||
display,
|
||||
)
|
||||
elif display.result_sets and display.mode == "none":
|
||||
sources.append(source)
|
||||
continue
|
||||
configured = _normalize_with_display_config(
|
||||
raw,
|
||||
index_value,
|
||||
skill_name,
|
||||
display,
|
||||
display_for_source,
|
||||
)
|
||||
if configured is not None:
|
||||
if isinstance(display_for_source, SkillDisplayResultSetConfig):
|
||||
configured["resultSetKey"] = display_for_source.key
|
||||
configured["resultSetLabel"] = display_for_source.label
|
||||
sources.append(
|
||||
{
|
||||
**source,
|
||||
@@ -318,6 +343,8 @@ def _source_debug_item(source: dict[str, Any]) -> dict[str, Any]:
|
||||
"id": source.get("id"),
|
||||
"skillName": source.get("skillName"),
|
||||
"mode": source.get("mode"),
|
||||
"resultSetKey": source.get("resultSetKey"),
|
||||
"resultSetLabel": source.get("resultSetLabel"),
|
||||
"title": source.get("title"),
|
||||
"source": source.get("source"),
|
||||
"author": source.get("author"),
|
||||
@@ -359,6 +386,16 @@ def _collect_sources(messages: list[Any]) -> list[dict[str, Any]]:
|
||||
tool_contexts_by_id,
|
||||
)
|
||||
payload = _extract_tool_payload(message)
|
||||
configured_sources = _collect_configured_sources(
|
||||
payload,
|
||||
len(sources) + 1,
|
||||
tool_context,
|
||||
)
|
||||
if configured_sources is not None:
|
||||
for normalized in configured_sources:
|
||||
normalized["index"] = len(sources) + 1
|
||||
sources.append(normalized)
|
||||
continue
|
||||
for raw_item in _first_list(payload):
|
||||
normalized = _normalize_reference_item(
|
||||
raw_item,
|
||||
@@ -373,6 +410,44 @@ def _collect_sources(messages: list[Any]) -> list[dict[str, Any]]:
|
||||
return sources
|
||||
|
||||
|
||||
def _collect_configured_sources(
|
||||
payload: Any,
|
||||
fallback_index: int,
|
||||
tool_context: dict[str, Any] | None,
|
||||
) -> list[dict[str, Any]] | None:
|
||||
display_match = _resolve_display_match({}, payload, tool_context)
|
||||
if display_match is None:
|
||||
return None
|
||||
skill_name, display = display_match
|
||||
configs: list[SkillDisplayConfig | SkillDisplayResultSetConfig] = [
|
||||
result_set
|
||||
for result_set in display.result_sets
|
||||
if result_set.enabled and result_set.mode != "none"
|
||||
]
|
||||
if not configs:
|
||||
configs = [display]
|
||||
|
||||
sources: list[dict[str, Any]] = []
|
||||
for config in configs:
|
||||
raw_items = _list_at_path(payload, config.result_path)
|
||||
if raw_items is None:
|
||||
continue
|
||||
for raw_item in raw_items:
|
||||
normalized = _normalize_with_display_config(
|
||||
raw_item,
|
||||
fallback_index + len(sources),
|
||||
skill_name,
|
||||
config,
|
||||
)
|
||||
if normalized is None:
|
||||
continue
|
||||
if isinstance(config, SkillDisplayResultSetConfig):
|
||||
normalized["resultSetKey"] = config.key
|
||||
normalized["resultSetLabel"] = config.label
|
||||
sources.append(normalized)
|
||||
return sources
|
||||
|
||||
|
||||
def _find_reference_batch_anchor(messages: list[Any], start: int, end: int) -> Any | None:
|
||||
answer = _find_last_assistant_answer(messages, start, end)
|
||||
if answer is not None:
|
||||
@@ -622,6 +697,19 @@ def _first_list(payload: Any) -> list[Any]:
|
||||
return []
|
||||
|
||||
|
||||
def _list_at_path(payload: Any, path: str | None) -> list[dict[str, Any]] | None:
|
||||
cursor = payload
|
||||
if path:
|
||||
for part in path.split("."):
|
||||
if isinstance(cursor, dict):
|
||||
cursor = cursor.get(part)
|
||||
else:
|
||||
return None
|
||||
if not isinstance(cursor, list):
|
||||
return None
|
||||
return [item for item in cursor if isinstance(item, dict)]
|
||||
|
||||
|
||||
def _nested_records(raw: Any) -> list[dict[str, Any]]:
|
||||
if not isinstance(raw, dict):
|
||||
return []
|
||||
@@ -654,7 +742,13 @@ def _enabled_skill_displays() -> dict[str, SkillDisplayConfig]:
|
||||
if state.display is not None
|
||||
and state.enabled
|
||||
and state.display.enabled
|
||||
and state.display.mode != "none"
|
||||
and (
|
||||
state.display.mode != "none"
|
||||
or any(
|
||||
result_set.enabled and result_set.mode != "none"
|
||||
for result_set in state.display.result_sets
|
||||
)
|
||||
)
|
||||
}
|
||||
except Exception:
|
||||
return {}
|
||||
@@ -705,7 +799,7 @@ def _normalize_with_display_config(
|
||||
raw: dict[str, Any],
|
||||
fallback_index: int,
|
||||
skill_name: str,
|
||||
display: SkillDisplayConfig,
|
||||
display: SkillDisplayConfig | SkillDisplayResultSetConfig,
|
||||
) -> dict[str, Any] | None:
|
||||
title = _stringify(_get_by_path(raw, display.title_field), f"结果 {fallback_index}")
|
||||
snippet = _stringify(_get_by_path(raw, display.summary_field), "")
|
||||
@@ -737,7 +831,10 @@ def _normalize_with_display_config(
|
||||
}
|
||||
|
||||
|
||||
def _resolve_display_link(raw: dict[str, Any], display: SkillDisplayConfig) -> str | None:
|
||||
def _resolve_display_link(
|
||||
raw: dict[str, Any],
|
||||
display: SkillDisplayConfig | SkillDisplayResultSetConfig,
|
||||
) -> str | None:
|
||||
url = _stringify(_get_by_path(raw, display.url_field), "")
|
||||
item_id = _stringify(_get_by_path(raw, display.id_field), "")
|
||||
if display.link_strategy == "url":
|
||||
|
||||
+9
-1
@@ -1,8 +1,16 @@
|
||||
from .registry import get_scheduled_task_service, set_scheduled_task_service
|
||||
from .service import DuplicateTaskNameError, MarkdownRequirementError, ScheduledTaskService
|
||||
from .service import (
|
||||
DuplicateFavoriteError,
|
||||
DuplicateTaskNameError,
|
||||
HtmlGenerationError,
|
||||
MarkdownRequirementError,
|
||||
ScheduledTaskService,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"DuplicateFavoriteError",
|
||||
"DuplicateTaskNameError",
|
||||
"HtmlGenerationError",
|
||||
"MarkdownRequirementError",
|
||||
"ScheduledTaskService",
|
||||
"get_scheduled_task_service",
|
||||
|
||||
+1
-1
@@ -18,7 +18,7 @@ class ElementDeliveryClient:
|
||||
self.access_token = access_token
|
||||
|
||||
@classmethod
|
||||
def from_env(cls) -> "ElementDeliveryClient | None":
|
||||
def from_env(cls) -> ElementDeliveryClient | None:
|
||||
enabled = os.environ.get("ELEMENT_ENABLED", "true").strip().lower()
|
||||
if enabled in {"0", "false", "no", "off"}:
|
||||
return None
|
||||
|
||||
+365
@@ -0,0 +1,365 @@
|
||||
"""HTML page generation helpers for ``html_page`` scheduled tasks.
|
||||
|
||||
Pure, dependency-light utilities the scheduler uses to turn a GLM5 response
|
||||
into a safe, previewable HTML document:
|
||||
|
||||
- :func:`extract_json_object` — fault-tolerant JSON extraction from a model
|
||||
reply that may be wrapped in prose or ```json fences.
|
||||
- :func:`sanitize_html` — strip script-execution vectors (``<script>``,
|
||||
``<iframe>``, ``on*`` handlers, ``javascript:`` URLs, …) using BeautifulSoup.
|
||||
- :func:`validate_html` — cheap structural checks (doctype/html/head/body,
|
||||
non-empty body, no residual scripts) that gate the optional repair pass.
|
||||
- ``build_*_prompt`` — the GLM5 prompt templates (generation / repair / style
|
||||
extraction).
|
||||
- :func:`screenshot_html` — best-effort Playwright screenshot, a no-op when
|
||||
Playwright is not installed (the offline image ships without a browser).
|
||||
|
||||
Kept free of any ``app.*`` import so it stays inside the harness boundary.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Tags removed wholesale — they either execute code or load remote active
|
||||
# content that the iframe sandbox alone shouldn't be relied on to contain.
|
||||
_DANGEROUS_TAGS = frozenset(
|
||||
{"script", "iframe", "object", "embed", "applet", "frame", "frameset", "base"}
|
||||
)
|
||||
|
||||
# URL schemes that can execute script when navigated/loaded.
|
||||
_DANGEROUS_URL_RE = re.compile(r"^\s*(javascript|vbscript|data:text/html)", re.IGNORECASE)
|
||||
|
||||
# Attributes carrying a URL we must scheme-check.
|
||||
_URL_ATTRS = frozenset({"href", "src", "action", "formaction", "xlink:href", "background", "poster"})
|
||||
|
||||
|
||||
def parse_frontmatter(text: str) -> tuple[dict[str, Any], str]:
|
||||
"""Split a ``---`` YAML frontmatter block from the body of a SKILL.md.
|
||||
|
||||
Returns ``(frontmatter_dict, body)``. When there is no frontmatter the dict
|
||||
is empty and the body is the whole text.
|
||||
"""
|
||||
if not text or not text.lstrip().startswith("---"):
|
||||
return {}, text or ""
|
||||
lines = text.splitlines()
|
||||
# Skip leading blank lines before the opening fence.
|
||||
start = 0
|
||||
while start < len(lines) and not lines[start].strip():
|
||||
start += 1
|
||||
end = None
|
||||
for i in range(start + 1, len(lines)):
|
||||
if lines[i].strip() == "---":
|
||||
end = i
|
||||
break
|
||||
if end is None:
|
||||
return {}, text
|
||||
fm_block = "\n".join(lines[start + 1 : end])
|
||||
body = "\n".join(lines[end + 1 :])
|
||||
try:
|
||||
import yaml
|
||||
|
||||
data = yaml.safe_load(fm_block) or {}
|
||||
if not isinstance(data, dict):
|
||||
data = {}
|
||||
except Exception:
|
||||
data = {}
|
||||
return data, body
|
||||
|
||||
|
||||
def extract_json_object(text: str | None) -> dict[str, Any] | None:
|
||||
"""Best-effort parse of a JSON object from a model reply.
|
||||
|
||||
Handles three common shapes: a bare JSON object, a ```json fenced block,
|
||||
and a JSON object embedded in surrounding prose. Returns ``None`` when no
|
||||
balanced object can be recovered.
|
||||
"""
|
||||
if not text or not isinstance(text, str):
|
||||
return None
|
||||
candidate = text.strip()
|
||||
|
||||
# Strip a single ```json ... ``` (or ``` ... ```) fence if present.
|
||||
fence = re.search(r"```(?:json)?\s*(.*?)```", candidate, flags=re.DOTALL | re.IGNORECASE)
|
||||
if fence:
|
||||
candidate = fence.group(1).strip()
|
||||
|
||||
# Fast path: the whole thing is already valid JSON.
|
||||
try:
|
||||
parsed = json.loads(candidate)
|
||||
return parsed if isinstance(parsed, dict) else None
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
# Fallback: scan for the first balanced ``{ ... }`` and parse that.
|
||||
start = candidate.find("{")
|
||||
if start == -1:
|
||||
return None
|
||||
depth = 0
|
||||
in_string = False
|
||||
escape = False
|
||||
for index in range(start, len(candidate)):
|
||||
char = candidate[index]
|
||||
if in_string:
|
||||
if escape:
|
||||
escape = False
|
||||
elif char == "\\":
|
||||
escape = True
|
||||
elif char == '"':
|
||||
in_string = False
|
||||
continue
|
||||
if char == '"':
|
||||
in_string = True
|
||||
elif char == "{":
|
||||
depth += 1
|
||||
elif char == "}":
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
blob = candidate[start : index + 1]
|
||||
try:
|
||||
parsed = json.loads(blob)
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
return parsed if isinstance(parsed, dict) else None
|
||||
return None
|
||||
|
||||
|
||||
def sanitize_html(html: str) -> tuple[str, list[str]]:
|
||||
"""Remove script-execution vectors from ``html``.
|
||||
|
||||
Returns ``(cleaned_html, removed)`` where ``removed`` is a human-readable
|
||||
report of what was stripped (used for logging / repair feedback). The
|
||||
cleaner is defensive but not a full HTML validator — pair it with an
|
||||
``iframe sandbox`` (no ``allow-scripts``) on the preview side.
|
||||
"""
|
||||
if not html or not isinstance(html, str):
|
||||
return "", []
|
||||
try:
|
||||
from bs4 import BeautifulSoup
|
||||
except Exception: # pragma: no cover - bs4 is a declared dependency
|
||||
logger.warning("BeautifulSoup unavailable; returning HTML unsanitized")
|
||||
return html, []
|
||||
|
||||
removed: list[str] = []
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
|
||||
# 1) Drop dangerous tags entirely.
|
||||
for tag in soup.find_all(True):
|
||||
name = (tag.name or "").lower()
|
||||
if name in _DANGEROUS_TAGS:
|
||||
removed.append(f"<{name}>")
|
||||
tag.decompose()
|
||||
continue
|
||||
# External stylesheet / preload links pull remote resources.
|
||||
if name == "link":
|
||||
href = str(tag.get("href") or "")
|
||||
rel = " ".join(tag.get("rel") or []) if isinstance(tag.get("rel"), list) else str(tag.get("rel") or "")
|
||||
if re.match(r"^\s*(https?:)?//", href) and "stylesheet" in rel.lower():
|
||||
removed.append("<link stylesheet>")
|
||||
tag.decompose()
|
||||
continue
|
||||
# meta refresh can redirect to arbitrary URLs.
|
||||
if name == "meta" and str(tag.get("http-equiv") or "").lower() == "refresh":
|
||||
removed.append("<meta refresh>")
|
||||
tag.decompose()
|
||||
continue
|
||||
|
||||
# 2) Strip event handlers and dangerous URL schemes from surviving tags.
|
||||
for tag in soup.find_all(True):
|
||||
for attr in list(tag.attrs.keys()):
|
||||
lowered = attr.lower()
|
||||
if lowered.startswith("on"):
|
||||
removed.append(f"{lowered}=")
|
||||
del tag.attrs[attr]
|
||||
continue
|
||||
if lowered in _URL_ATTRS:
|
||||
value = tag.attrs[attr]
|
||||
value_str = value if isinstance(value, str) else " ".join(value)
|
||||
if _DANGEROUS_URL_RE.match(value_str or ""):
|
||||
removed.append(f"{lowered}={value_str[:24]}")
|
||||
del tag.attrs[attr]
|
||||
|
||||
return str(soup), removed
|
||||
|
||||
|
||||
def validate_html(html: str) -> list[str]:
|
||||
"""Return a list of structural problems with ``html`` (empty == valid).
|
||||
|
||||
A page only counts as a complete document when it has the full skeleton
|
||||
(doctype/html/head/body), a non-empty ``<title>``, visible body content
|
||||
(no white-screen), no residual scripts, and no external stylesheet links.
|
||||
"""
|
||||
errors: list[str] = []
|
||||
if not html or not isinstance(html, str):
|
||||
return ["empty html"]
|
||||
lowered = html.lower()
|
||||
if "<!doctype html" not in lowered:
|
||||
errors.append("缺少 <!doctype html> 声明")
|
||||
if "<html" not in lowered:
|
||||
errors.append("缺少 <html> 根元素")
|
||||
if "<head" not in lowered:
|
||||
errors.append("缺少 <head> 元素")
|
||||
if "<body" not in lowered:
|
||||
errors.append("缺少 <body> 元素")
|
||||
if "<script" in lowered:
|
||||
errors.append("仍包含 <script> 标签(不允许脚本)")
|
||||
|
||||
try:
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
# Title must exist and be non-empty.
|
||||
title_el = soup.find("title")
|
||||
if title_el is None or not title_el.get_text(strip=True):
|
||||
errors.append("缺少非空 <title> 标题")
|
||||
# External stylesheet links pull remote resources — not allowed offline.
|
||||
for link in soup.find_all("link"):
|
||||
href = str(link.get("href") or "")
|
||||
rel = " ".join(link.get("rel") or []) if isinstance(link.get("rel"), list) else str(link.get("rel") or "")
|
||||
if re.match(r"^\s*(https?:)?//", href) and "stylesheet" in rel.lower():
|
||||
errors.append("包含外链样式表(不允许外部 CDN/资源)")
|
||||
break
|
||||
# Body must carry some visible content — guards against white-screen.
|
||||
body = soup.body or soup
|
||||
text = body.get_text(strip=True) if body else ""
|
||||
has_media = bool(body.find(["img", "svg", "canvas", "table", "ul", "ol"])) if body else False
|
||||
if len(text) < 5 and not has_media:
|
||||
errors.append("页面正文为空,可能白屏")
|
||||
except Exception: # pragma: no cover
|
||||
pass
|
||||
return errors
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Prompt templates
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_GENERATION_TEMPLATE = """你是一个内网页面生成助手。请根据输入数据生成一个完整的 HTML 页面。
|
||||
|
||||
用户页面要求:
|
||||
{page_prompt}
|
||||
|
||||
限制条件:
|
||||
{negative_prompt}
|
||||
|
||||
本次数据:
|
||||
{data}
|
||||
|
||||
参考页面风格:
|
||||
{reference_style}
|
||||
|
||||
参考页面布局:
|
||||
{reference_layout}
|
||||
{skill_instructions}
|
||||
要求:
|
||||
1. 只输出 JSON,不要输出 Markdown,不要输出多余解释。
|
||||
2. JSON 字段必须包含 title、html、style_summary、layout_summary。
|
||||
3. html 必须是完整 HTML 文档,包含 <!doctype html>、html、head、body。
|
||||
4. 所有 CSS 写在 <style> 标签内,不要引用外部 CDN 或外链样式表。
|
||||
5. 不要包含 <script>、外链脚本、iframe、表单提交、on* 事件属性。
|
||||
6. 参考页面只用于风格和布局,页面内容必须来自“本次数据”,不要复用参考页旧数据。
|
||||
7. 页面使用中文,排版适合在浏览器中直接预览。
|
||||
"""
|
||||
|
||||
_REPAIR_TEMPLATE = """你是一个 HTML 修复助手。下面的 HTML 存在问题,请修复后重新输出完整页面。
|
||||
|
||||
原始页面要求:
|
||||
{requirements}
|
||||
|
||||
发现的问题:
|
||||
{errors}
|
||||
|
||||
待修复 HTML:
|
||||
{html}
|
||||
|
||||
要求:
|
||||
1. 只输出 JSON,字段包含 fixed_html、changes(修复说明数组)。
|
||||
2. fixed_html 必须是完整 HTML 文档,包含 <!doctype html>、html、head、body。
|
||||
3. 移除所有 <script>、iframe、外链脚本、on* 事件属性、表单提交。
|
||||
4. 保留原有内容与数据,只修复结构与安全问题。
|
||||
"""
|
||||
|
||||
_STYLE_EXTRACT_TEMPLATE = """你是一个页面风格分析助手。请分析下面的 HTML 页面,提取可复用的风格与布局特征。
|
||||
|
||||
HTML 页面:
|
||||
{html}
|
||||
|
||||
要求:
|
||||
1. 只输出 JSON,字段包含 style_summary、layout_summary、content_structure_summary、reuse_prompt。
|
||||
2. style_summary:颜色、字体、间距、圆角、阴影、视觉密度等视觉风格。
|
||||
3. layout_summary:页面分区、卡片、表格、图表、导航等布局结构。
|
||||
4. content_structure_summary:信息组织顺序与层级。
|
||||
5. reuse_prompt:一段可直接拼接到下次生成 prompt 的中文风格说明,不要包含本页的具体数据。
|
||||
"""
|
||||
|
||||
|
||||
def _clip(value: Any, limit: int) -> str:
|
||||
text = value if isinstance(value, str) else json.dumps(value, ensure_ascii=False, default=str)
|
||||
text = text or ""
|
||||
return text if len(text) <= limit else text[:limit] + "\n…(内容过长已截断)"
|
||||
|
||||
|
||||
def build_generation_prompt(
|
||||
*,
|
||||
page_prompt: str,
|
||||
negative_prompt: str | None,
|
||||
data: Any,
|
||||
reference_style: str | None,
|
||||
reference_layout: str | None,
|
||||
skill_instructions: str | None = None,
|
||||
data_limit: int = 16000,
|
||||
skill_limit: int = 16000,
|
||||
) -> str:
|
||||
skill_block = ""
|
||||
if skill_instructions and skill_instructions.strip():
|
||||
skill_block = "\n可用页面生成技能规范(请严格遵循):\n" + _clip(skill_instructions.strip(), skill_limit) + "\n"
|
||||
return _GENERATION_TEMPLATE.format(
|
||||
page_prompt=(page_prompt or "(未提供,使用通用看板风格)").strip(),
|
||||
negative_prompt=(negative_prompt or "无").strip(),
|
||||
data=_clip(data, data_limit),
|
||||
reference_style=(reference_style or "无").strip(),
|
||||
reference_layout=(reference_layout or "无").strip(),
|
||||
skill_instructions=skill_block,
|
||||
)
|
||||
|
||||
|
||||
def build_repair_prompt(*, html: str, errors: list[str], requirements: str | None) -> str:
|
||||
return _REPAIR_TEMPLATE.format(
|
||||
requirements=(requirements or "无").strip(),
|
||||
errors="\n".join(f"- {e}" for e in errors) or "- 未知问题",
|
||||
html=_clip(html, 32000),
|
||||
)
|
||||
|
||||
|
||||
def build_style_extract_prompt(*, html: str) -> str:
|
||||
return _STYLE_EXTRACT_TEMPLATE.format(html=_clip(html, 32000))
|
||||
|
||||
|
||||
async def screenshot_html(html: str, output_path) -> str | None:
|
||||
"""Best-effort PNG screenshot of ``html`` for white-screen / console QA.
|
||||
|
||||
Returns the screenshot path on success, or ``None`` when Playwright (or a
|
||||
usable browser) is unavailable — the offline image ships without one, so
|
||||
this must degrade silently rather than fail the run.
|
||||
"""
|
||||
try:
|
||||
from playwright.async_api import async_playwright
|
||||
except Exception:
|
||||
return None
|
||||
try:
|
||||
async with async_playwright() as pw:
|
||||
browser = await pw.chromium.launch()
|
||||
try:
|
||||
page = await browser.new_page()
|
||||
await page.set_content(html, wait_until="networkidle")
|
||||
await page.screenshot(path=str(output_path), full_page=True)
|
||||
finally:
|
||||
await browser.close()
|
||||
return str(output_path)
|
||||
except Exception:
|
||||
logger.debug("Playwright screenshot failed; skipping QA screenshot", exc_info=True)
|
||||
return None
|
||||
+479
-1
@@ -22,6 +22,7 @@ from deerflow.config.agents_config import AGENT_ID_PATTERN
|
||||
from deerflow.config.paths import get_paths
|
||||
from deerflow.persistence.scheduled_tasks import ScheduledTaskStore
|
||||
from deerflow.runtime import serialize_channel_values
|
||||
from deerflow.runtime.scheduler import html_page as html_page_utils
|
||||
from deerflow.runtime.scheduler.element import ElementDeliveryClient
|
||||
from deerflow.runtime.user_context import reset_current_user, set_current_user
|
||||
from deerflow.utils.time import now_iso
|
||||
@@ -36,6 +37,11 @@ _RUN_ATTEMPT_TIMEOUT_SECONDS = 30 * 60 # 30 minutes
|
||||
# Retries allowed after the first attempt before the run is marked "failed".
|
||||
_MAX_RUN_RETRIES = 3
|
||||
|
||||
# Max HTML length (chars) inlined into the run's ``result_json``. Beyond this
|
||||
# the inline copy is dropped to stay under MySQL's 64 KB ``TEXT`` cap; the full
|
||||
# document is always retrievable from the page store (LONGTEXT) + index.html.
|
||||
_MAX_INLINE_RESULT_HTML = 60000
|
||||
|
||||
# Minimum length (in characters) the generated Markdown file's content must
|
||||
# reach when a task has ``require_markdown`` enabled. The model is NOT told about
|
||||
# this threshold — it is enforced silently by the service.
|
||||
@@ -58,6 +64,22 @@ class MarkdownRequirementError(Exception):
|
||||
"""
|
||||
|
||||
|
||||
class HtmlGenerationError(Exception):
|
||||
"""Raised when an ``html_page`` run did not produce a complete, valid HTML
|
||||
document (missing doctype/html/head/body, empty body, or residual scripts
|
||||
after the sanitize + repair passes). The run is marked failed — a page only
|
||||
counts as successful when it is a full, previewable HTML document."""
|
||||
|
||||
|
||||
class DuplicateFavoriteError(Exception):
|
||||
"""A user already bookmarked this page. Surfaced as HTTP 409 by the router
|
||||
so the same page cannot be favorited twice."""
|
||||
|
||||
def __init__(self, page_id: str) -> None:
|
||||
self.page_id = page_id
|
||||
super().__init__(f"Page {page_id!r} is already favorited")
|
||||
|
||||
|
||||
class DuplicateTaskNameError(Exception):
|
||||
"""A user already owns a scheduled task with the requested name.
|
||||
|
||||
@@ -835,7 +857,7 @@ class ScheduledTaskService:
|
||||
runtime_thread_id = str(uuid.uuid4())
|
||||
try:
|
||||
result = await asyncio.wait_for(
|
||||
self._run_agent_attempt(task, runtime_thread_id=runtime_thread_id, scheduled_for=scheduled_for, run_id=run_id),
|
||||
self._run_task_attempt(task, runtime_thread_id=runtime_thread_id, scheduled_for=scheduled_for, run_id=run_id),
|
||||
timeout=_RUN_ATTEMPT_TIMEOUT_SECONDS,
|
||||
)
|
||||
except TimeoutError:
|
||||
@@ -875,6 +897,333 @@ class ScheduledTaskService:
|
||||
await self._deliver_element_result(task, result["content"])
|
||||
return
|
||||
|
||||
async def _run_task_attempt(
|
||||
self,
|
||||
task: dict[str, Any],
|
||||
*,
|
||||
runtime_thread_id: str,
|
||||
scheduled_for: datetime,
|
||||
run_id: str,
|
||||
) -> dict[str, Any]:
|
||||
"""Dispatch a single execution attempt by task kind.
|
||||
|
||||
``html_page`` tasks render an HTML document via GLM5; everything else
|
||||
runs the regular lead-agent attempt. The kind lives in
|
||||
``execution_context.task_kind`` so no schema migration is needed.
|
||||
"""
|
||||
ctx = task.get("execution_context")
|
||||
kind = ctx.get("task_kind") if isinstance(ctx, dict) else None
|
||||
if kind == "html_page":
|
||||
return await self._run_html_page_attempt(
|
||||
task, runtime_thread_id=runtime_thread_id, scheduled_for=scheduled_for, run_id=run_id
|
||||
)
|
||||
return await self._run_agent_attempt(
|
||||
task, runtime_thread_id=runtime_thread_id, scheduled_for=scheduled_for, run_id=run_id
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _html_config(task: dict[str, Any]) -> dict[str, Any]:
|
||||
ctx = task.get("execution_context")
|
||||
cfg = ctx.get("html_config") if isinstance(ctx, dict) else None
|
||||
return cfg if isinstance(cfg, dict) else {}
|
||||
|
||||
async def _invoke_chat_model(self, model_name: str | None, prompt: str) -> str:
|
||||
"""Invoke a chat model once and return its text content.
|
||||
|
||||
Used for the GLM5 HTML generation / repair / style-extraction passes.
|
||||
Thinking is disabled to keep structured-JSON output stable.
|
||||
"""
|
||||
from langchain_core.messages import HumanMessage
|
||||
|
||||
from deerflow.models import create_chat_model
|
||||
|
||||
# No hardcoded model: use exactly the selected model; only when none was
|
||||
# selected do we fall back to the system default (config models[0]).
|
||||
model = create_chat_model(name=model_name or None, thinking_enabled=False)
|
||||
response = await model.ainvoke([HumanMessage(content=prompt)])
|
||||
content = getattr(response, "content", response)
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
if isinstance(content, list):
|
||||
parts = [part.get("text", "") if isinstance(part, dict) else str(part) for part in content]
|
||||
return "\n".join(p for p in parts if p)
|
||||
return str(content)
|
||||
|
||||
async def _gather_html_data(
|
||||
self,
|
||||
task: dict[str, Any],
|
||||
*,
|
||||
runtime_thread_id: str,
|
||||
scheduled_for: datetime,
|
||||
run_id: str,
|
||||
) -> str:
|
||||
"""Run the lead agent to collect source data for the HTML page.
|
||||
|
||||
Reuses the regular agent-run primitives (with the task's search query
|
||||
folded into the prompt) and returns the agent's final text, which feeds
|
||||
the GLM5 render step as ``本次数据``. Returns an empty string if the
|
||||
agent produced nothing.
|
||||
"""
|
||||
cfg = self._html_config(task)
|
||||
page_prompt = str(cfg.get("page_prompt") or "").strip()
|
||||
gather_instruction = (
|
||||
"请运用你自身的检索/搜索等技能,围绕下述页面需求获取最新、真实的资料,"
|
||||
"并整理成结构化的中文要点(可用列表/小标题),供后续生成 HTML 页面使用,本步骤不要生成 HTML:\n"
|
||||
)
|
||||
parts = [gather_instruction]
|
||||
if page_prompt:
|
||||
parts.append(f"页面需求:{page_prompt}")
|
||||
base_prompt = str(task.get("prompt") or "").strip()
|
||||
if base_prompt:
|
||||
parts.append(f"补充说明:{base_prompt}")
|
||||
gather_task = {**task, "prompt": "\n".join(parts)}
|
||||
# Never apply the markdown gate to the gather pass.
|
||||
gather_ctx = dict(task.get("execution_context") or {})
|
||||
gather_ctx.pop("require_markdown", None)
|
||||
gather_ctx.pop("task_kind", None)
|
||||
gather_ctx.pop("html_config", None)
|
||||
# Use the task's selected model for the data-gathering agent too, so the
|
||||
# whole HTML pipeline honors ``html_config.model_name`` instead of
|
||||
# silently falling back to the default model for this step.
|
||||
gather_model = str(cfg.get("model_name") or "").strip()
|
||||
if gather_model:
|
||||
gather_ctx["model_name"] = gather_model
|
||||
gather_task["execution_context"] = gather_ctx
|
||||
|
||||
await self._ensure_thread_meta(task["user_id"], runtime_thread_id)
|
||||
async with self._agent_run_lock:
|
||||
record = await self._start_agent_run(gather_task, scheduled_for=scheduled_for, runtime_thread_id=runtime_thread_id)
|
||||
await self.store.update_run(run_id, {"agent_run_id": record.run_id})
|
||||
if record.task is not None:
|
||||
await record.task
|
||||
text = await self._extract_latest_ai_text(runtime_thread_id)
|
||||
return text or ""
|
||||
|
||||
def _collect_html_skill_instructions(self) -> str:
|
||||
"""Concatenate the bodies of enabled skills marked for HTML pages.
|
||||
|
||||
A skill opts in via ``metadata.kind: html_page`` in its SKILL.md
|
||||
frontmatter. Only enabled skills (per the skills settings the user
|
||||
toggles) are injected, so the render step honors the user's skill
|
||||
selection. Best-effort — any failure yields an empty string.
|
||||
"""
|
||||
try:
|
||||
from deerflow.skills import get_or_new_skill_storage
|
||||
|
||||
storage = get_or_new_skill_storage()
|
||||
skills = storage.load_skills(enabled_only=True)
|
||||
except Exception:
|
||||
logger.debug("Failed to load skills for HTML render injection", exc_info=True)
|
||||
return ""
|
||||
blocks: list[str] = []
|
||||
for skill in skills:
|
||||
try:
|
||||
text = Path(skill.skill_file).read_text(encoding="utf-8")
|
||||
except Exception:
|
||||
continue
|
||||
frontmatter, body = html_page_utils.parse_frontmatter(text)
|
||||
meta = frontmatter.get("metadata") if isinstance(frontmatter, dict) else None
|
||||
if not (isinstance(meta, dict) and str(meta.get("kind")) == "html_page"):
|
||||
continue
|
||||
body = body.strip()
|
||||
if body:
|
||||
blocks.append(f"### 技能:{skill.name}\n{body}")
|
||||
return "\n\n".join(blocks)
|
||||
|
||||
def _archive_html_index(self, run_id: str, html: str) -> Path | None:
|
||||
"""Write the generated HTML to the run's archive dir as ``index.html``."""
|
||||
try:
|
||||
archive_dir = get_paths().base_dir / "scheduled-task-runs" / run_id / "files"
|
||||
archive_dir.mkdir(parents=True, exist_ok=True)
|
||||
target = archive_dir / "index.html"
|
||||
target.write_text(html, encoding="utf-8")
|
||||
return target
|
||||
except Exception:
|
||||
logger.warning("Failed to archive generated HTML for run %s", run_id, exc_info=True)
|
||||
return None
|
||||
|
||||
async def _run_html_page_attempt(
|
||||
self,
|
||||
task: dict[str, Any],
|
||||
*,
|
||||
runtime_thread_id: str,
|
||||
scheduled_for: datetime,
|
||||
run_id: str,
|
||||
) -> dict[str, Any]:
|
||||
"""Generate an HTML page for an ``html_page`` task.
|
||||
|
||||
Pipeline: gather data via the lead agent → optionally load a reference
|
||||
favorite's style/layout → GLM5 render → sanitize → validate → (one
|
||||
GLM5 repair pass if invalid) → best-effort screenshot → archive
|
||||
``index.html`` → persist a ``scheduled_html_pages`` row → build the run
|
||||
result payload (``format: "html"`` with the full HTML inline).
|
||||
"""
|
||||
cfg = self._html_config(task)
|
||||
# The user's selected model only — no hardcoded GLM-5. Empty → None →
|
||||
# system default (config models[0]).
|
||||
model_name = str(cfg.get("model_name") or "").strip() or None
|
||||
page_prompt = str(cfg.get("page_prompt") or "").strip()
|
||||
negative_prompt = str(cfg.get("negative_prompt") or "").strip()
|
||||
reference_favorite_id = cfg.get("reference_favorite_id") or None
|
||||
|
||||
# 1) Gather source data (real, via the agent's search tools).
|
||||
data_text = await self._gather_html_data(
|
||||
task, runtime_thread_id=runtime_thread_id, scheduled_for=scheduled_for, run_id=run_id
|
||||
)
|
||||
|
||||
# 2) Load reference style/layout from a favorite if one was selected.
|
||||
reference_style = None
|
||||
reference_layout = None
|
||||
if reference_favorite_id:
|
||||
store = self._html_store
|
||||
if store is not None:
|
||||
try:
|
||||
favorite = await store.get_favorite(str(reference_favorite_id))
|
||||
except Exception:
|
||||
favorite = None
|
||||
if favorite is not None:
|
||||
reference_style = favorite.get("reuse_prompt") or favorite.get("style_summary")
|
||||
reference_layout = favorite.get("layout_summary")
|
||||
|
||||
# 3) Render with the chosen model, injecting any enabled HTML skills.
|
||||
skill_instructions = self._collect_html_skill_instructions()
|
||||
gen_prompt = html_page_utils.build_generation_prompt(
|
||||
page_prompt=page_prompt,
|
||||
negative_prompt=negative_prompt,
|
||||
data=data_text,
|
||||
reference_style=reference_style,
|
||||
reference_layout=reference_layout,
|
||||
skill_instructions=skill_instructions,
|
||||
)
|
||||
raw = await self._invoke_chat_model(model_name, gen_prompt)
|
||||
parsed = html_page_utils.extract_json_object(raw) or {}
|
||||
title = str(parsed.get("title") or task.get("name") or "生成页面").strip()
|
||||
html = parsed.get("html")
|
||||
if not isinstance(html, str) or "<" not in html:
|
||||
# GLM5 may have replied with bare HTML instead of JSON — accept it.
|
||||
if isinstance(raw, str) and ("<html" in raw.lower() or "<!doctype" in raw.lower()):
|
||||
html = raw
|
||||
else:
|
||||
raise ValueError("GLM5 未返回有效的 HTML(JSON 解析失败)")
|
||||
style_summary = str(parsed.get("style_summary") or "").strip() or None
|
||||
layout_summary = str(parsed.get("layout_summary") or "").strip() or None
|
||||
|
||||
# 4) Sanitize + validate, with one repair pass on failure.
|
||||
html, _removed = html_page_utils.sanitize_html(html)
|
||||
errors = html_page_utils.validate_html(html)
|
||||
if errors:
|
||||
logger.info("Generated HTML failed validation (%s); attempting repair", errors)
|
||||
repair_prompt = html_page_utils.build_repair_prompt(
|
||||
html=html, errors=errors, requirements=page_prompt or task.get("prompt")
|
||||
)
|
||||
try:
|
||||
repaired_raw = await self._invoke_chat_model(model_name, repair_prompt)
|
||||
repaired = html_page_utils.extract_json_object(repaired_raw) or {}
|
||||
fixed = repaired.get("fixed_html")
|
||||
if isinstance(fixed, str) and "<" in fixed:
|
||||
fixed_clean, _ = html_page_utils.sanitize_html(fixed)
|
||||
if not html_page_utils.validate_html(fixed_clean):
|
||||
html = fixed_clean
|
||||
except Exception:
|
||||
logger.warning("HTML repair pass failed; keeping sanitized original", exc_info=True)
|
||||
|
||||
# 4b) Completeness gate — a page only counts as successful when it is a
|
||||
# full, valid HTML document. If it still fails validation after the
|
||||
# repair pass, fail the run (no half-baked page is archived/persisted).
|
||||
final_errors = html_page_utils.validate_html(html)
|
||||
if final_errors:
|
||||
raise HtmlGenerationError(
|
||||
"HTML 生成未通过完整性校验:" + ";".join(final_errors) + "。请调整页面要求或重试。"
|
||||
)
|
||||
|
||||
# 5) Best-effort screenshot QA (no-op without Playwright).
|
||||
screenshot_path = None
|
||||
try:
|
||||
shot_target = get_paths().base_dir / "scheduled-task-runs" / run_id / "files" / "preview.png"
|
||||
shot_target.parent.mkdir(parents=True, exist_ok=True)
|
||||
screenshot_path = await html_page_utils.screenshot_html(html, shot_target)
|
||||
except Exception:
|
||||
screenshot_path = None
|
||||
|
||||
# 6) Archive index.html.
|
||||
archive_path = self._archive_html_index(run_id, html)
|
||||
|
||||
files: list[dict[str, Any]] = []
|
||||
if archive_path is not None:
|
||||
files.append(
|
||||
{
|
||||
"name": "index.html",
|
||||
"path": str(archive_path),
|
||||
"virtual_path": "/mnt/user-data/outputs/index.html",
|
||||
"mime_type": "text/html",
|
||||
"size": archive_path.stat().st_size if archive_path.exists() else None,
|
||||
"kind": "html",
|
||||
}
|
||||
)
|
||||
if screenshot_path:
|
||||
shot = Path(screenshot_path)
|
||||
files.append(
|
||||
{
|
||||
"name": "preview.png",
|
||||
"path": str(shot),
|
||||
"mime_type": "image/png",
|
||||
"size": shot.stat().st_size if shot.exists() else None,
|
||||
"kind": "artifact",
|
||||
}
|
||||
)
|
||||
|
||||
data_snapshot = {"data_text": data_text[:8000] if isinstance(data_text, str) else ""}
|
||||
prompt_snapshot = {
|
||||
"page_prompt": page_prompt,
|
||||
"negative_prompt": negative_prompt,
|
||||
"reference_favorite_id": reference_favorite_id,
|
||||
"model_name": model_name,
|
||||
}
|
||||
|
||||
# 7) Persist the page asset (best-effort — failure must not fail the run).
|
||||
store = self._html_store
|
||||
if store is not None:
|
||||
try:
|
||||
await store.create_page(
|
||||
{
|
||||
"id": str(uuid.uuid4()),
|
||||
"task_id": task["task_id"],
|
||||
"run_id": run_id,
|
||||
"user_id": task["user_id"],
|
||||
"title": title,
|
||||
"html_content": html,
|
||||
"html_file_path": str(archive_path) if archive_path else None,
|
||||
"data_snapshot": data_snapshot,
|
||||
"prompt_snapshot": prompt_snapshot,
|
||||
"reference_favorite_id": reference_favorite_id,
|
||||
"style_summary": style_summary,
|
||||
"layout_summary": layout_summary,
|
||||
}
|
||||
)
|
||||
except Exception:
|
||||
logger.warning("Failed to persist scheduled_html_pages row for run %s", run_id, exc_info=True)
|
||||
|
||||
task_tz = ZoneInfo(task.get("timezone") or "UTC")
|
||||
local_time = scheduled_for.astimezone(task_tz)
|
||||
# ``scheduled_task_runs.result_json`` is TEXT (64 KB) on MySQL, so only
|
||||
# inline the HTML when it comfortably fits. The full document always
|
||||
# lives in ``scheduled_html_pages.html_content`` (LONGTEXT) + the
|
||||
# archived ``index.html``; the ``/runs/{id}/html`` endpoints read from
|
||||
# there, so previews work even when the inline copy is omitted here.
|
||||
inline_html = html if len(html) <= _MAX_INLINE_RESULT_HTML else None
|
||||
return {
|
||||
"title": f"{title} - {local_time.strftime('%Y-%m-%d %H:%M')}",
|
||||
"content": page_prompt or f"HTML 页面已生成:{title}",
|
||||
"format": "html",
|
||||
"html": inline_html,
|
||||
"style_summary": style_summary,
|
||||
"layout_summary": layout_summary,
|
||||
"data_snapshot": data_snapshot,
|
||||
"prompt_snapshot": prompt_snapshot,
|
||||
"files": files,
|
||||
"created_at": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
|
||||
async def _run_agent_attempt(
|
||||
self,
|
||||
task: dict[str, Any],
|
||||
@@ -1275,3 +1624,132 @@ class ScheduledTaskService:
|
||||
|
||||
async def unsubscribe_task(self, user_id: str, task_id: str) -> bool:
|
||||
return await self.store.delete_subscription(task_id, user_id)
|
||||
|
||||
# --- HTML pages / favorites ---
|
||||
|
||||
@property
|
||||
def _html_store(self):
|
||||
state = getattr(self.app, "state", None)
|
||||
return getattr(state, "html_page_store", None)
|
||||
|
||||
async def get_html_page_for_run(self, run_id: str) -> dict[str, Any] | None:
|
||||
"""Return the generated HTML page for a run (page-store row or run result)."""
|
||||
store = self._html_store
|
||||
if store is not None:
|
||||
page = await store.get_page_by_run(run_id)
|
||||
if page is not None:
|
||||
return page
|
||||
# Fallback to the inline HTML on the run result.
|
||||
run = await self.store.get_run_any(run_id)
|
||||
if run is None:
|
||||
return None
|
||||
result = run.get("result") if isinstance(run.get("result"), dict) else None
|
||||
if not result or not isinstance(result.get("html"), str):
|
||||
return None
|
||||
return {
|
||||
"run_id": run_id,
|
||||
"task_id": run.get("task_id"),
|
||||
"title": result.get("title"),
|
||||
"html_content": result.get("html"),
|
||||
"style_summary": result.get("style_summary"),
|
||||
"layout_summary": result.get("layout_summary"),
|
||||
"created_at": result.get("created_at"),
|
||||
}
|
||||
|
||||
async def get_latest_html_page_for_task(self, task_id: str) -> dict[str, Any] | None:
|
||||
store = self._html_store
|
||||
if store is not None:
|
||||
page = await store.latest_page_for_task(task_id)
|
||||
if page is not None:
|
||||
return page
|
||||
# Fallback: newest run that carries inline HTML.
|
||||
runs = await self.store.list_runs_any(task_id, limit=20)
|
||||
for run in runs:
|
||||
result = run.get("result") if isinstance(run.get("result"), dict) else None
|
||||
if result and isinstance(result.get("html"), str):
|
||||
return await self.get_html_page_for_run(run["id"])
|
||||
return None
|
||||
|
||||
async def list_html_page_favorites(self, user_id: str) -> list[dict[str, Any]]:
|
||||
store = self._html_store
|
||||
if store is None:
|
||||
return []
|
||||
return await store.list_favorites(user_id)
|
||||
|
||||
async def get_html_page_favorite(self, user_id: str, favorite_id: str) -> dict[str, Any] | None:
|
||||
store = self._html_store
|
||||
if store is None:
|
||||
return None
|
||||
favorite = await store.get_favorite(favorite_id)
|
||||
if favorite is None or favorite.get("user_id") != user_id:
|
||||
return None
|
||||
return favorite
|
||||
|
||||
async def delete_html_page_favorite(self, user_id: str, favorite_id: str) -> bool:
|
||||
store = self._html_store
|
||||
if store is None:
|
||||
return False
|
||||
return await store.delete_favorite(favorite_id, user_id)
|
||||
|
||||
async def create_html_page_favorite(
|
||||
self,
|
||||
user_id: str,
|
||||
page_id: str,
|
||||
*,
|
||||
title: str | None = None,
|
||||
description: str | None = None,
|
||||
tags: list[str] | None = None,
|
||||
) -> dict[str, Any] | None:
|
||||
"""Bookmark a generated page, extracting a reusable style summary via GLM5.
|
||||
|
||||
Returns ``None`` when the page does not exist or is not owned by the
|
||||
caller. Style extraction is best-effort — a failure still produces a
|
||||
favorite (without the reuse hints) so the user's bookmark is never lost.
|
||||
"""
|
||||
store = self._html_store
|
||||
if store is None:
|
||||
return None
|
||||
page = await store.get_page(page_id)
|
||||
if page is None or page.get("user_id") != user_id:
|
||||
return None
|
||||
# A page can only be bookmarked once per user.
|
||||
existing = await store.get_favorite_by_page(user_id, page_id)
|
||||
if existing is not None:
|
||||
raise DuplicateFavoriteError(page_id)
|
||||
html = page.get("html_content") or ""
|
||||
|
||||
style_summary = page.get("style_summary")
|
||||
layout_summary = page.get("layout_summary")
|
||||
content_structure_summary = None
|
||||
reuse_prompt = None
|
||||
if html:
|
||||
try:
|
||||
# Reuse the model the page was generated with (recorded in the
|
||||
# page's prompt snapshot) so favoriting doesn't fall back to the
|
||||
# default model.
|
||||
extract_model = (page.get("prompt_snapshot") or {}).get("model_name")
|
||||
extract_prompt = html_page_utils.build_style_extract_prompt(html=html)
|
||||
raw = await self._invoke_chat_model(extract_model, extract_prompt)
|
||||
extracted = html_page_utils.extract_json_object(raw) or {}
|
||||
style_summary = extracted.get("style_summary") or style_summary
|
||||
layout_summary = extracted.get("layout_summary") or layout_summary
|
||||
content_structure_summary = extracted.get("content_structure_summary")
|
||||
reuse_prompt = extracted.get("reuse_prompt")
|
||||
except Exception:
|
||||
logger.warning("HTML style extraction failed for page %s; saving favorite without hints", page_id, exc_info=True)
|
||||
|
||||
return await store.create_favorite(
|
||||
{
|
||||
"id": str(uuid.uuid4()),
|
||||
"user_id": user_id,
|
||||
"page_id": page_id,
|
||||
"title": (title or page.get("title") or "收藏页面").strip(),
|
||||
"description": (description or None),
|
||||
"tags": tags or [],
|
||||
"html_snapshot": html,
|
||||
"style_summary": style_summary,
|
||||
"layout_summary": layout_summary,
|
||||
"content_structure_summary": content_structure_summary,
|
||||
"reuse_prompt": reuse_prompt,
|
||||
}
|
||||
)
|
||||
+1
-1
@@ -56,7 +56,7 @@ def notebook_search_tool(query: str, top_k: int = 10) -> str:
|
||||
headers={"Authorization": f"Bearer {token}"},
|
||||
json={
|
||||
"query": query,
|
||||
"source_ids": [source_ids],
|
||||
"source_ids": source_ids,
|
||||
"search_space_id": search_space_id,
|
||||
"top_k": top_k,
|
||||
},
|
||||
|
||||
+37
-5
@@ -51,20 +51,33 @@ def view_image_tool(
|
||||
runtime: ToolRuntime[ContextT, ThreadState],
|
||||
image_path: str,
|
||||
tool_call_id: Annotated[str, InjectedToolCallId],
|
||||
question: str | None = None,
|
||||
) -> Command:
|
||||
"""Read an image file.
|
||||
"""Read and understand an image file.
|
||||
|
||||
Use this tool to read an image file and make it available for display.
|
||||
This is the ONLY correct way to understand an image. The image is analyzed by
|
||||
the configured vision model and its description/answer is returned to you as
|
||||
the tool result. Treat that result as authoritative.
|
||||
|
||||
When to use the view_image tool:
|
||||
- When you need to view an image file.
|
||||
- Whenever you need to view, describe, or answer a question about an image file.
|
||||
|
||||
When NOT to use the view_image tool:
|
||||
- For non-image files (use present_files instead)
|
||||
- For multiple files at once (use present_files instead)
|
||||
|
||||
IMPORTANT — do NOT attempt to understand images any other way:
|
||||
- Never read, open, decode, or parse an image with read_file, grep, bash,
|
||||
python, PIL/Pillow, OpenCV, EXIF parsing, or byte/hex inspection. Those
|
||||
cannot interpret visual content and will only waste turns.
|
||||
- If `view_image` returns a weak or incomplete description, do not fall back
|
||||
to code. Either call `view_image` again with a more specific `question`,
|
||||
or tell the user the configured vision model could not extract more detail
|
||||
(the vision model may need to be changed in config).
|
||||
|
||||
Args:
|
||||
image_path: Absolute /mnt/user-data virtual path to the image file. Common formats supported: jpg, jpeg, png, webp.
|
||||
question: Optional specific question to ask about the image (e.g. "What is the total on this invoice?"). When omitted, a general detailed description is produced.
|
||||
"""
|
||||
from deerflow.sandbox.exceptions import SandboxRuntimeError
|
||||
from deerflow.sandbox.tools import (
|
||||
@@ -153,8 +166,27 @@ def view_image_tool(
|
||||
mime_type = detected_mime_type
|
||||
image_base64 = base64.b64encode(image_data).decode("utf-8")
|
||||
|
||||
# Update viewed_images in state
|
||||
# The merge_viewed_images reducer will handle merging with existing images
|
||||
# Delegated recognition: when a dedicated vision model is configured, send
|
||||
# the image to that model and return its description as the tool result, so
|
||||
# a text-only main chat model can still "see" the image.
|
||||
from deerflow.models.vision import is_delegated_vision, recognize_image
|
||||
|
||||
if is_delegated_vision():
|
||||
try:
|
||||
description = recognize_image(image_base64, mime_type, question=question)
|
||||
except Exception as e: # noqa: BLE001 - surface as a recoverable tool error
|
||||
return Command(
|
||||
update={"messages": [ToolMessage(f"Error recognizing image: {_sanitize_image_error(e, thread_data)}", tool_call_id=tool_call_id)]},
|
||||
)
|
||||
if not description:
|
||||
description = "The vision model returned an empty description."
|
||||
return Command(
|
||||
update={"messages": [ToolMessage(f"Image recognition result for {image_path}:\n\n{description}", tool_call_id=tool_call_id)]},
|
||||
)
|
||||
|
||||
# Native vision: store the image so ViewImageMiddleware injects it into the
|
||||
# (vision-capable) main model before the next LLM call.
|
||||
# The merge_viewed_images reducer will handle merging with existing images.
|
||||
new_viewed_images = {image_path: {"base64": image_base64, "mime_type": mime_type}}
|
||||
|
||||
return Command(
|
||||
|
||||
@@ -138,11 +138,14 @@ def get_available_tools(
|
||||
if model_name is None and config.models:
|
||||
model_name = config.models[0].name
|
||||
|
||||
# Add view_image_tool only if the model supports vision
|
||||
model_config = config.get_model_config(model_name) if model_name else None
|
||||
if model_config is not None and model_config.supports_vision:
|
||||
# Add view_image_tool when vision is available — either the main model
|
||||
# supports vision natively, or a dedicated vision model is configured
|
||||
# (vision.model_name) so recognition can be delegated to it.
|
||||
from deerflow.models.vision import vision_enabled
|
||||
|
||||
if vision_enabled(model_name, app_config=config):
|
||||
builtin_tools.append(view_image_tool)
|
||||
logger.info(f"Including view_image_tool for model '{model_name}' (supports_vision=True)")
|
||||
logger.info(f"Including view_image_tool for model '{model_name}' (vision available)")
|
||||
|
||||
# Get cached MCP tools if enabled
|
||||
# NOTE: We use ExtensionsConfig.from_file() instead of config.extensions
|
||||
|
||||
@@ -138,12 +138,21 @@ def open_upload_file_no_symlink(base_dir: Path, filename: str) -> tuple[Path, ob
|
||||
|
||||
validate_path_traversal(dest, base_dir)
|
||||
|
||||
if not hasattr(os, "O_NOFOLLOW"):
|
||||
raise UnsafeUploadPathError("Upload writes require O_NOFOLLOW support")
|
||||
|
||||
flags = os.O_WRONLY | os.O_CREAT | os.O_NOFOLLOW
|
||||
# ``O_NOFOLLOW`` closes the TOCTOU window between the ``lstat`` above and the
|
||||
# ``open`` below, but it is a POSIX-only flag — it does not exist on Windows
|
||||
# (``os.O_NOFOLLOW`` is undefined there). Windows also lacks ``O_NONBLOCK``
|
||||
# and needs ``O_BINARY`` to avoid CRLF translation on the raw fd. Where
|
||||
# ``O_NOFOLLOW`` is unavailable we still reject pre-existing symlinks via the
|
||||
# ``lstat`` check and re-verify the opened file with ``fstat`` below; the
|
||||
# residual race only matters under a hostile local user, which is out of
|
||||
# scope for the single-user local deployment this path targets.
|
||||
flags = os.O_WRONLY | os.O_CREAT
|
||||
if hasattr(os, "O_NOFOLLOW"):
|
||||
flags |= os.O_NOFOLLOW
|
||||
if hasattr(os, "O_NONBLOCK"):
|
||||
flags |= os.O_NONBLOCK
|
||||
if hasattr(os, "O_BINARY"):
|
||||
flags |= os.O_BINARY
|
||||
|
||||
try:
|
||||
fd = os.open(dest, flags, 0o600)
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
"""Regression tests for the upstream call shape of token-exchange login.
|
||||
|
||||
The upstream ``getTokenInfo`` endpoint reads the bearer token from the
|
||||
``Authorization`` header — NOT from an ``access_token`` cookie. These tests pin
|
||||
that contract for ``_fetch_username_from_token``:
|
||||
|
||||
- ``Authorization`` is exactly ``Bearer <caller-token>`` (the token passed in),
|
||||
not the static ``service_authorization`` config value.
|
||||
- No ``access_token`` cookie is sent (the old, wrong placement).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import types
|
||||
|
||||
import pytest
|
||||
|
||||
from app.gateway.routers import auth as auth_module
|
||||
from app.gateway.routers.auth import _fetch_username_from_token
|
||||
|
||||
|
||||
class _FakeResponse:
|
||||
status_code = 200
|
||||
|
||||
def json(self) -> dict:
|
||||
return {"state": "200", "msg": "ok", "data": {"username": "b1admin"}}
|
||||
|
||||
|
||||
class _CapturingClient:
|
||||
"""Stand-in for ``httpx.AsyncClient`` that records the post() kwargs."""
|
||||
|
||||
captured: dict = {}
|
||||
|
||||
def __init__(self, *args, **kwargs) -> None:
|
||||
type(self).init_kwargs = kwargs
|
||||
|
||||
async def __aenter__(self) -> _CapturingClient:
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *exc) -> None:
|
||||
return None
|
||||
|
||||
async def post(self, url, **kwargs):
|
||||
type(self).captured = {"url": url, **kwargs}
|
||||
return _FakeResponse()
|
||||
|
||||
|
||||
def _patch_config(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
token_login = types.SimpleNamespace(
|
||||
token_info_url="https://upstream.example/getTokenInfo",
|
||||
service_authorization="Bearer admin", # must NOT be used anymore
|
||||
timeout_seconds=10.0,
|
||||
verify_ssl=False,
|
||||
ca_cert_path="",
|
||||
)
|
||||
fake_cfg = types.SimpleNamespace(auth_login=types.SimpleNamespace(token_login=token_login))
|
||||
monkeypatch.setattr(auth_module, "get_app_config", lambda: fake_cfg)
|
||||
monkeypatch.setattr(auth_module.httpx, "AsyncClient", _CapturingClient)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sends_caller_token_as_bearer_header(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
_patch_config(monkeypatch)
|
||||
|
||||
username = await _fetch_username_from_token("UPSTREAM_TOK_123")
|
||||
|
||||
assert username == "b1admin"
|
||||
headers = _CapturingClient.captured["headers"]
|
||||
assert headers["Authorization"] == "Bearer UPSTREAM_TOK_123"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_does_not_use_static_service_authorization(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
_patch_config(monkeypatch)
|
||||
|
||||
await _fetch_username_from_token("UPSTREAM_TOK_123")
|
||||
|
||||
headers = _CapturingClient.captured["headers"]
|
||||
assert headers["Authorization"] != "Bearer admin"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_no_access_token_cookie_is_sent(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
_patch_config(monkeypatch)
|
||||
|
||||
await _fetch_username_from_token("UPSTREAM_TOK_123")
|
||||
|
||||
# The token must live in the header now; no cookie param should be passed.
|
||||
assert "cookies" not in _CapturingClient.captured
|
||||
Generated
-4917
File diff suppressed because it is too large.
Load diff
@@ -11,7 +11,7 @@ models:
|
||||
max_retries: 2
|
||||
max_tokens: 4096
|
||||
temperature: 1.0
|
||||
supports_vision: true
|
||||
supports_vision: false
|
||||
supports_thinking: true
|
||||
- name: zai-org/GLM-5-FP8
|
||||
display_name: zai-org/GLM-5-FP8
|
||||
@@ -35,7 +35,7 @@ models:
|
||||
max_retries: 2
|
||||
max_tokens: 4096
|
||||
temperature: 1.0
|
||||
supports_vision: true
|
||||
supports_vision: false
|
||||
supports_thinking: true
|
||||
- name: minimax-m2.7
|
||||
display_name: MiniMax M2.7
|
||||
@@ -57,6 +57,36 @@ models:
|
||||
temperature: 1.0
|
||||
supports_vision: true
|
||||
supports_thinking: true
|
||||
- name: deepseek-ai/DeepSeek-OCR
|
||||
display_name: deepseek-ai/DeepSeek-OCR
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: deepseek-ai/DeepSeek-OCR
|
||||
api_key: sk-beshfxfxfrllsczpqfrgjbugfbpzxgqgedbehceytpcvvtyb
|
||||
base_url: https://api.siliconflow.cn/v1/
|
||||
max_tokens: 4096
|
||||
temperature: 1.0
|
||||
supports_vision: true
|
||||
supports_thinking: true
|
||||
# 识图(图片识别)配置。
|
||||
# - model_name:指定一个 models[] 中的模型名,作为“默认识图大模型”。view_image 工具会把
|
||||
# 图片交给该模型识别,并把识别出的文字结果返回给主对话模型 —— 这样即使主对话模型不支持
|
||||
# 视觉,也能“看图”。请改成你实际用于识图的模型名。
|
||||
# - 若留空 / 注释掉 model_name,则回退到主对话模型自身的视觉能力(需该模型 supports_vision: true)。
|
||||
# - prompt:不带具体问题时使用的默认识图指令(可选)。
|
||||
# - max_tokens:识图模型输出上限(可选,默认用该模型自身的配置)。
|
||||
vision:
|
||||
model_name: deepseek-ai/DeepSeek-OCR
|
||||
# prompt: 请详细描述这张图片的内容。
|
||||
# max_tokens: 2048
|
||||
# 上传文件识别配置。
|
||||
# - auto_convert_documents: true 时,上传的 PDF / Word / Excel / PPT 会在服务端自动转换为
|
||||
# Markdown(.md),agent 通过 read_file / grep 读取其内容 —— 即“识别”文档内容。
|
||||
# 纯文本类(txt / csv / md / json / 代码等)无需转换,agent 用 read_file 直接读取。
|
||||
# 图片(png/jpg/webp)由 view_image + 上面的 vision 识图模型处理。
|
||||
# - pdf_converter: auto(优先 pymupdf4llm,缺失时回退 markitdown)| pymupdf4llm | markitdown
|
||||
uploads:
|
||||
auto_convert_documents: true
|
||||
pdf_converter: auto
|
||||
tool_groups:
|
||||
- name: web
|
||||
- name: file:read
|
||||
@@ -113,6 +143,26 @@ skills:
|
||||
enabled: false
|
||||
agents_api:
|
||||
enabled: true
|
||||
# 登录鉴权扩展:在基础登录之上叠加两种自定义模式。
|
||||
auth_login:
|
||||
# 用户名直登口令门:开关打开后,/login/<用户名> 直登必须再带 ?password=口令,
|
||||
# 口令不对直接 401。password 留空则默认口令为 123ewq。仅作用于用户名直登。
|
||||
username_login:
|
||||
require_password: false
|
||||
password: ""
|
||||
# token 换登录:前端用 ?authToken=上游token 进入,后端调用 token_info_url
|
||||
# (把该 token 直接拼成 Authorization: Bearer <token> 头)换出用户名,再登录;
|
||||
# 用户名不在库里时自动注册。
|
||||
token_login:
|
||||
enabled: true
|
||||
token_info_url: "https://ch1.b.uat.4.cn/consumer/login/getTokenInfo"
|
||||
# 已弃用:鉴权头现由前端传入的 token 拼成 "Bearer <token>",此项不再生效,保留仅兼容。
|
||||
service_authorization: "Bearer admin"
|
||||
timeout_seconds: 10
|
||||
# 上游是 HTTPS。UAT/内网常用自签或私有 CA 证书,默认 verify=true 会握手失败导致
|
||||
# 永远 401。此处 uat 域名按自签处理,先关校验;生产请置 true 或配 ca_cert_path。
|
||||
verify_ssl: false
|
||||
ca_cert_path: "" # 可选:PEM 格式 CA 包路径,设置后优先生效并启用校验
|
||||
database:
|
||||
backend: mysql
|
||||
mysql_url: $MYSQL_DATABASE_URL
|
||||
|
||||
Reference in new issue
Block a user