识图与文件上传/预览:可配置视觉模型 + 修复上传与回显

后端:
- 新增可配置识图模型(config: vision.model_name/prompt/max_tokens)。配置后 view_image 委托该模型识别并返回文字结果,文本主模型也能“看图”;未配置时回退到主模型自带视觉能力。新增 deerflow/models/vision.py 与 VisionConfig。
- 修复 Windows 上传全部报 unsafe:open_upload_file_no_symlink 不再强制要求 POSIX 专有的 O_NOFOLLOW,缺失时降级(仍用 lstat/fstat 防符号链接),Windows 追加 O_BINARY。
- 修复 normalize_input 丢弃 human 消息 additional_kwargs:保留 files/client_ts 等,使上传文件能回传到 UI 并被 UploadsMiddleware 识别。
- 约束 agent 行为:上传图片时引导只用 view_image,禁止用 read_file/grep/bash/python 解析图片字节;view_image 工具说明同步加固。
- 启用文档转换(uploads.auto_convert_documents),PDF/Word/Excel/PPT 自动转 Markdown 供 read_file/grep 读取。
- 订正若干模型错误的 supports_vision 标记。

前端:
- 修复上传文件不回显:parseUploadedFiles 不再因 "(empty)" 提前返回。
- 修复 artifact 401 破图:改用带 Authorization 头的 apiFetch 取 blob 显示(新增 useAuthedObjectUrl)。
- 新增上传文件预览弹窗:点击卡片可查看图片/文本/Markdown、Office(读转换后的 .md)、PDF;文件卡片改为可点击。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
吉鑫升andClaude Opus 4.8 committed 2026-06-02 16:43:53 +08:00
1 parent 43cbcd86be
commit 5aa1fdca04
17 files changed
+706 -74

No files matched your search

@@ -288,10 +288,14 @@ def _build_middlewares(
if not is_bootstrap:
middlewares.append(MemoryMiddleware(agent_name=agent_name, memory_config=resolved_app_config.memory, injection_enabled=memory_injection_enabled))
# Add ViewImageMiddleware only if the current model supports vision.
# Use the resolved runtime model_name from make_lead_agent to avoid stale config values.
model_config = resolved_app_config.get_model_config(model_name) if model_name else None
if model_config is not None and model_config.supports_vision:
# Add ViewImageMiddleware only for the NATIVE-vision path (main model
# ingests raw image data). In delegated mode the view_image tool already
# returns a text description and stores nothing in `viewed_images`, so the
# middleware would otherwise inject a spurious "No images have been viewed."
# message. Use the resolved runtime model_name to avoid stale config values.
from deerflow.models.vision import is_delegated_vision, vision_enabled
if vision_enabled(model_name, app_config=resolved_app_config) and not is_delegated_vision(app_config=resolved_app_config):
middlewares.append(ViewImageMiddleware())
# Add DeferredToolFilterMiddleware to hide deferred tool schemas from model binding
@@ -18,6 +18,12 @@ logger = logging.getLogger(__name__)
_OUTLINE_PREVIEW_LINES = 5
# Image files are understood via the `view_image` tool (which routes to the
# configured vision model), never by reading their bytes. Listing them with the
# generic read_file/grep guidance made the agent try to parse image binaries
# with bash/python, which fails and wastes turns.
_IMAGE_EXTENSIONS = {".jpg", ".jpeg", ".png", ".webp", ".gif", ".bmp"}
def _extract_outline_for_file(file_path: Path) -> tuple[list[dict], list[str]]:
"""Return the document outline and fallback preview for *file_path*.
@@ -88,6 +94,16 @@ class UploadsMiddleware(AgentMiddleware[UploadsMiddlewareState]):
size_str = f"{size_kb:.1f} KB" if size_kb < 1024 else f"{size_kb / 1024:.1f} MB"
lines.append(f"- {file['filename']} ({size_str})")
lines.append(f" Path: {file['path']}")
# Images must be understood through the vision model, not by reading bytes.
extension = str(file.get("extension") or Path(file["filename"]).suffix).lower()
if extension in _IMAGE_EXTENSIONS:
lines.append(f" This is an image. To understand it, call the `view_image` tool with image_path='{file['path']}' (optionally pass a `question`).")
lines.append(" The configured vision model will analyze it and return a description — trust that result.")
lines.append(" Do NOT read, open, or parse the image with read_file, grep, bash, python, PIL, or any byte-level inspection; those cannot interpret image content.")
lines.append("")
return
outline = file.get("outline") or []
if outline:
truncated = outline[-1].get("truncated", False)
@@ -139,6 +139,12 @@ class ViewImageMiddleware(AgentMiddleware[ViewImageMiddlewareState]):
if not messages:
return False
# Nothing to inject when no image data was captured (e.g. delegated
# recognition handled the image in the tool and stored no base64).
# Avoids injecting a useless "No images have been viewed." message.
if not state.get("viewed_images"):
return False
# Get the last assistant message
last_assistant_msg = self._get_last_assistant_message(messages)
if not last_assistant_msg: