识图与文件上传/预览:可配置视觉模型 + 修复上传与回显

后端:
- 新增可配置识图模型(config: vision.model_name/prompt/max_tokens)。配置后 view_image 委托该模型识别并返回文字结果,文本主模型也能“看图”;未配置时回退到主模型自带视觉能力。新增 deerflow/models/vision.py 与 VisionConfig。
- 修复 Windows 上传全部报 unsafe:open_upload_file_no_symlink 不再强制要求 POSIX 专有的 O_NOFOLLOW,缺失时降级(仍用 lstat/fstat 防符号链接),Windows 追加 O_BINARY。
- 修复 normalize_input 丢弃 human 消息 additional_kwargs:保留 files/client_ts 等,使上传文件能回传到 UI 并被 UploadsMiddleware 识别。
- 约束 agent 行为:上传图片时引导只用 view_image,禁止用 read_file/grep/bash/python 解析图片字节;view_image 工具说明同步加固。
- 启用文档转换(uploads.auto_convert_documents),PDF/Word/Excel/PPT 自动转 Markdown 供 read_file/grep 读取。
- 订正若干模型错误的 supports_vision 标记。

前端:
- 修复上传文件不回显:parseUploadedFiles 不再因 "(empty)" 提前返回。
- 修复 artifact 401 破图:改用带 Authorization 头的 apiFetch 取 blob 显示(新增 useAuthedObjectUrl)。
- 新增上传文件预览弹窗:点击卡片可查看图片/文本/Markdown、Office(读转换后的 .md)、PDF;文件卡片改为可点击。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
吉鑫升andClaude Opus 4.8 committed 2026-06-02 16:43:53 +08:00
1 parent 43cbcd86be
commit 5aa1fdca04
17 files changed
+706 -74

No files matched your search

+14 -5
View File
@@ -637,12 +637,21 @@ Automatic conversation summarization when approaching token limits:
See [docs/summarization.md](docs/summarization.md) for details.
### Vision Support
### Vision Support / Image Recognition (识图)
For models with `supports_vision: true`:
- `ViewImageMiddleware` processes images in conversation
- `view_image_tool` added to agent's toolset
- Images automatically converted to base64 and injected into state
Two paths, selected by `config.yaml` → `vision`:
**Dedicated vision model (delegated recognition)** — set `vision.model_name` to a model from `models[]`. The `view_image` tool sends the image to that model and returns its **text description** as the tool result, so a text-only main chat model can still "see" images. `view_image` accepts an optional `question` argument for targeted queries; the default prompt is `vision.prompt` (overridable), and `vision.max_tokens` caps the recognition output. Resolution/gating lives in `deerflow/models/vision.py` (`resolve_vision_model_name`, `vision_enabled`, `is_delegated_vision`, `recognize_image`).
**Native vision (fallback)** — when `vision.model_name` is unset, behavior falls back to the main chat model, which only works if it has `supports_vision: true`:
- `ViewImageMiddleware` injects base64 image data into the conversation before the LLM call
- `view_image` stores the image in `state.viewed_images` for that middleware
Both the `view_image` tool and `ViewImageMiddleware` are registered whenever `vision_enabled(model_name)` is true (dedicated model configured **or** main model supports vision). Config schema: `deerflow/config/vision_config.py` (`VisionConfig`); tests: `tests/test_vision.py`.
### Upload symlink-safety (cross-platform)
`deerflow/uploads/manager.py:open_upload_file_no_symlink` writes uploads without following a pre-existing destination symlink. `O_NOFOLLOW` is applied where available (POSIX) but is **optional** — Windows lacks `os.O_NOFOLLOW`/`O_NONBLOCK`, so requiring it previously made every Windows upload fail with `UnsafeUploadPathError`. Pre-existing symlinks are still rejected via `lstat`, and the opened fd is re-checked with `fstat`. `O_BINARY` is added on Windows. Regression: `tests/test_uploads_windows_safe.py`.
## Code Style