feat: 新增 multi-llm 路由智能体及 API 端点
添加 multi-llm 路由智能体（LangGraph），支持多 LLM 分发能力；在 Gateway 中注册 /api/multi-llm 路由，并豁免其 CSRF 和认证校验；同时新增 asyncmy 依赖和聊天流格式文档。 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
xuhdandClaude Sonnet 4.6 committed
1 parent
a8a545690f
commit
adeace0f7b
15 files changed
+3789
-2767
No files matched your search
@@ -0,0 +1,474 @@
|
||||
# New Chat 接口流式返回格式说明
|
||||
|
||||
## 接口概览
|
||||
|
||||
- **路径**：`POST /new_chat`
|
||||
- **传输协议**：Server-Sent Events (SSE)
|
||||
- **Content-Type**：`text/event-stream`
|
||||
- **协议基础**：Vercel AI SDK Data Stream Protocol
|
||||
|
||||
---
|
||||
|
||||
## 请求格式
|
||||
|
||||
```json
|
||||
{
|
||||
"chat_id": 123,
|
||||
"search_space_id": 456,
|
||||
"user_query": "用户问题",
|
||||
"pic_query": [{"image_url": "/tmp/xxx.png"}],
|
||||
"model_name": "gpt-4o",
|
||||
"enable_citations": true,
|
||||
}
|
||||
```
|
||||
|
||||
| 字段 | 类型 | 必填 | 说明 |
|
||||
|------|------|------|------|
|
||||
| `chat_id` | int | ✅ | 会话 ID，同时作为 LangGraph 的 thread_id |
|
||||
| `search_space_id` | int | ✅ | 搜索空间 ID |
|
||||
| `user_query` | string | ✅ | 用户问题 |
|
||||
| `pic_query` | list\|null | ❌ | 图片路径列表，格式 `[{"image_url": "..."}]` |
|
||||
| `model_name` | string\|null | ❌ | 指定模型名称（插件场景使用） |
|
||||
| `enable_citations` | bool | ❌ | 是否启用引用（默认 true） |
|
||||
| `messages` | list\|null | ❌ | 历史消息列表 |
|
||||
| `attachments` | list\|null | ❌ | 上传附件（已解析为 Markdown） |
|
||||
| `mentioned_document_ids` | list\|null | ❌ | 前端 @ 引用的文档 ID 列表 |
|
||||
|
||||
---
|
||||
|
||||
## 响应 Headers
|
||||
|
||||
```
|
||||
Content-Type: text/event-stream
|
||||
Cache-Control: no-cache
|
||||
Connection: keep-alive
|
||||
X-Accel-Buffering: no
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## SSE 数据帧格式
|
||||
|
||||
每条消息均为标准 SSE 格式：
|
||||
|
||||
```
|
||||
data: {JSON 对象}\n\n
|
||||
```
|
||||
|
||||
流结束时发送：
|
||||
|
||||
```
|
||||
data: [DONE]\n\n
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 事件类型详解
|
||||
|
||||
### 1. 消息生命周期
|
||||
|
||||
#### `start` — 消息开始
|
||||
|
||||
```json
|
||||
{"type": "start", "messageId": "msg_abc123"}
|
||||
```
|
||||
|
||||
| 字段 | 说明 |
|
||||
|------|------|
|
||||
| `messageId` | 本次消息的唯一 ID |
|
||||
|
||||
---
|
||||
|
||||
#### `finish` — 消息结束
|
||||
|
||||
```json
|
||||
{"type": "finish"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `[DONE]` — 流终止标记
|
||||
|
||||
```
|
||||
data: [DONE]
|
||||
```
|
||||
|
||||
> 这是流中最后一帧，为纯字符串，不是 JSON。
|
||||
|
||||
---
|
||||
|
||||
### 2. 步骤控制（多轮工具调用）
|
||||
|
||||
#### `start-step` — 步骤开始
|
||||
|
||||
```json
|
||||
{"type": "start-step"}
|
||||
```
|
||||
|
||||
#### `finish-step` — 步骤结束
|
||||
|
||||
```json
|
||||
{"type": "finish-step"}
|
||||
```
|
||||
|
||||
> 每次 LLM 调用算一个 step。工具调用期间会有多个 step 交替出现。
|
||||
|
||||
---
|
||||
|
||||
### 3. 文本块（start / delta / end 模式）
|
||||
|
||||
#### `text-start` — 文本块开始
|
||||
|
||||
```json
|
||||
{"type": "text-start", "id": "text_abc123"}
|
||||
```
|
||||
|
||||
#### `text-delta` — 文本增量内容
|
||||
|
||||
```json
|
||||
{"type": "text-delta", "id": "text_abc123", "delta": "Hello, "}
|
||||
```
|
||||
|
||||
#### `text-end` — 文本块结束
|
||||
|
||||
```json
|
||||
{"type": "text-end", "id": "text_abc123"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4. 推理块（Reasoning，适用于 DeepSeek R1 等思维链模型）
|
||||
|
||||
#### `reasoning-start` — 推理块开始
|
||||
|
||||
```json
|
||||
{"type": "reasoning-start", "id": "reasoning_abc123"}
|
||||
```
|
||||
|
||||
#### `reasoning-delta` — 推理增量内容
|
||||
|
||||
```json
|
||||
{"type": "reasoning-delta", "id": "reasoning_abc123", "delta": "让我先思考..."}
|
||||
```
|
||||
|
||||
#### `reasoning-end` — 推理块结束
|
||||
|
||||
```json
|
||||
{"type": "reasoning-end", "id": "reasoning_abc123"}
|
||||
```
|
||||
|
||||
> 触发时机：
|
||||
> - 模型返回 `additional_kwargs.reasoning_content`（DeepSeek R1 原生）
|
||||
> - 模型输出中包含 `<think>...</think>` 标签（自动剥离）
|
||||
|
||||
---
|
||||
|
||||
### 5. 工具调用
|
||||
|
||||
#### `tool-input-start` — 工具调用开始
|
||||
|
||||
```json
|
||||
{"type": "tool-input-start", "toolCallId": "call_abc123", "toolName": "search_knowledge_base"}
|
||||
```
|
||||
|
||||
#### `tool-input-delta` — 工具输入流式内容（增量）
|
||||
|
||||
```json
|
||||
{"type": "tool-input-delta", "toolCallId": "call_abc123", "inputTextDelta": "San Fra"}
|
||||
```
|
||||
|
||||
#### `tool-input-available` — 工具输入完整参数
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "tool-input-available",
|
||||
"toolCallId": "call_abc123",
|
||||
"toolName": "search_knowledge_base",
|
||||
"input": {"query": "机器学习入门"}
|
||||
}
|
||||
```
|
||||
|
||||
#### `tool-output-available` — 工具执行结果
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "tool-output-available",
|
||||
"toolCallId": "call_abc123",
|
||||
"output": {"status": "completed", "result_length": 1234}
|
||||
}
|
||||
```
|
||||
|
||||
> 注意：`search_knowledge_base` 只返回摘要（不含全文），`write_todos`、`link_preview`、`display_image`、`scrape_webpage`、`generate_podcast` 会返回完整 output 供前端渲染组件。
|
||||
|
||||
---
|
||||
|
||||
### 6. 来源引用
|
||||
|
||||
#### `source-url` — URL 来源
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "source-url",
|
||||
"sourceId": "https://example.com",
|
||||
"url": "https://example.com",
|
||||
"title": "示例页面"
|
||||
}
|
||||
```
|
||||
|
||||
#### `source-document` — 文档来源
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "source-document",
|
||||
"sourceId": "doc_123",
|
||||
"mediaType": "file",
|
||||
"title": "报告.pdf",
|
||||
"description": "第三季度报告"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 7. 文件
|
||||
|
||||
```json
|
||||
{"type": "file", "url": "https://example.com/image.png", "mediaType": "image/png"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 8. 自定义数据（`data-*` 类型）
|
||||
|
||||
#### `data-terminal-info` — 终端信息提示
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "data-terminal-info",
|
||||
"data": {
|
||||
"text": "知识库搜索已完成",
|
||||
"type": "success"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`type` 枚举值：`info` | `error` | `success` | `warning`
|
||||
|
||||
---
|
||||
|
||||
#### `data-further-questions` — 推荐追问问题
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "data-further-questions",
|
||||
"data": {
|
||||
"questions": ["问题1", "问题2", "问题3"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `data-thinking-step` — 思考步骤（思维链 UI 展示）
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "data-thinking-step",
|
||||
"data": {
|
||||
"id": "thinking-1",
|
||||
"title": "正在搜索知识库",
|
||||
"status": "in_progress",
|
||||
"items": ["Query: 机器学习入门"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| 字段 | 说明 |
|
||||
|------|------|
|
||||
| `id` | 步骤唯一 ID（格式 `thinking-N`） |
|
||||
| `title` | 步骤标题 |
|
||||
| `status` | `pending` \| `in_progress` \| `completed` |
|
||||
| `items` | 步骤详情列表（可选） |
|
||||
|
||||
> 同一个 `id` 的步骤事件会被重复发送以更新状态（先 `in_progress` 后 `completed`）。
|
||||
|
||||
**常见步骤标题对照：**
|
||||
|
||||
| 工具 / 场景 | 标题 |
|
||||
|------------|------|
|
||||
| 初始分析 | 正在理解您的请求 / 正在分析您的内容 |
|
||||
| `search_knowledge_base` | 查询个人知识库中 |
|
||||
| `search_web` | 网查搜索中 |
|
||||
| `link_preview` | 正在获取链接预览 |
|
||||
| `display_image` | 正在分析图片 |
|
||||
| `scrape_webpage` | 正在抓取网页 |
|
||||
| `write_todos`（首次） | 正在创建计划 |
|
||||
| `write_todos`（更新） | 正在更新进度 |
|
||||
| `generate_podcast` | 正在生成播客 |
|
||||
|
||||
---
|
||||
|
||||
### 9. 错误
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "error",
|
||||
"errorText": "API rate limit exceeded",
|
||||
"errorType": "llm",
|
||||
"errorCode": "rate_limit"
|
||||
}
|
||||
```
|
||||
|
||||
| 字段 | 说明 |
|
||||
|------|------|
|
||||
| `errorText` | 错误描述文本 |
|
||||
| `errorType` | 错误类型（见下表） |
|
||||
| `errorCode` | 具体错误代码（可选） |
|
||||
|
||||
**errorType 枚举：**
|
||||
|
||||
| 值 | 含义 |
|
||||
|----|------|
|
||||
| `llm` | LLM / API 错误（模型不存在、限流、密钥错误等） |
|
||||
| `network` | 网络连通性错误 |
|
||||
| `timeout` | 请求超时 |
|
||||
| `authentication` | 鉴权 / 权限错误 |
|
||||
| `rate_limit` | 速率限制 |
|
||||
| `validation` | 输入验证错误 |
|
||||
| `internal` | 内部服务器错误 |
|
||||
| `unknown` | 未知错误 |
|
||||
|
||||
---
|
||||
|
||||
### 10. 中断（Human-in-the-loop）
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "interrupt",
|
||||
"messageId": "msg_abc123",
|
||||
"interrupt": {
|
||||
"action_requests": [...],
|
||||
"review_configs": [...]
|
||||
},
|
||||
"payload": {"expertAgentId": 42}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 完整流示例（普通问答）
|
||||
|
||||
```
|
||||
data: {"type":"start","messageId":"msg_abc123"}
|
||||
|
||||
data: {"type":"start-step"}
|
||||
|
||||
data: {"type":"data-thinking-step","data":{"id":"thinking-1","title":"正在理解您的请求","status":"in_progress","items":["Processing: 什么是向量数据库？"]}}
|
||||
|
||||
data: {"type":"data-thinking-step","data":{"id":"thinking-1","title":"正在理解您的请求","status":"completed","items":["Processing: 什么是向量数据库？"]}}
|
||||
|
||||
data: {"type":"text-start","id":"text_xyz789"}
|
||||
|
||||
data: {"type":"text-delta","id":"text_xyz789","delta":"向量数据库是"}
|
||||
|
||||
data: {"type":"text-delta","id":"text_xyz789","delta":"一种专为存储和检索"}
|
||||
|
||||
data: {"type":"text-delta","id":"text_xyz789","delta":"高维向量而设计的数据库..."}
|
||||
|
||||
data: {"type":"text-end","id":"text_xyz789"}
|
||||
|
||||
data: {"type":"finish-step"}
|
||||
|
||||
data: {"type":"finish"}
|
||||
|
||||
data: [DONE]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 完整流示例（含工具调用）
|
||||
|
||||
```
|
||||
data: {"type":"start","messageId":"msg_abc123"}
|
||||
|
||||
data: {"type":"start-step"}
|
||||
|
||||
data: {"type":"data-thinking-step","data":{"id":"thinking-1","title":"正在理解您的请求","status":"in_progress","items":["Processing: 搜索最新 AI 论文"]}}
|
||||
|
||||
data: {"type":"data-thinking-step","data":{"id":"thinking-1","title":"正在理解您的请求","status":"completed","items":["Processing: 搜索最新 AI 论文"]}}
|
||||
|
||||
data: {"type":"data-thinking-step","data":{"id":"thinking-2","title":"查询个人知识库中","status":"in_progress","items":["Query: 最新 AI 论文"]}}
|
||||
|
||||
data: {"type":"tool-input-start","toolCallId":"call_xxx","toolName":"search_knowledge_base"}
|
||||
|
||||
data: {"type":"tool-input-available","toolCallId":"call_xxx","toolName":"search_knowledge_base","input":{"query":"最新 AI 论文"}}
|
||||
|
||||
data: {"type":"data-thinking-step","data":{"id":"thinking-2","title":"Searching knowledge base","status":"completed","items":["Query: 最新 AI 论文","Found relevant information (2048 chars)"]}}
|
||||
|
||||
data: {"type":"tool-output-available","toolCallId":"call_xxx","output":{"status":"completed","result_length":2048}}
|
||||
|
||||
data: {"type":"data-terminal-info","data":{"text":"知识库搜索已完成","type":"success"}}
|
||||
|
||||
data: {"type":"text-start","id":"text_xyz789"}
|
||||
|
||||
data: {"type":"text-delta","id":"text_xyz789","delta":"根据知识库中的内容..."}
|
||||
|
||||
data: {"type":"text-end","id":"text_xyz789"}
|
||||
|
||||
data: {"type":"finish-step"}
|
||||
|
||||
data: {"type":"finish"}
|
||||
|
||||
data: [DONE]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 完整流示例（含推理过程）
|
||||
|
||||
```
|
||||
data: {"type":"start","messageId":"msg_abc123"}
|
||||
|
||||
data: {"type":"start-step"}
|
||||
|
||||
data: {"type":"reasoning-start","id":"reasoning_r01"}
|
||||
|
||||
data: {"type":"reasoning-delta","id":"reasoning_r01","delta":"让我先分析这个问题..."}
|
||||
|
||||
data: {"type":"reasoning-delta","id":"reasoning_r01","delta":"需要考虑以下几个方面..."}
|
||||
|
||||
data: {"type":"reasoning-end","id":"reasoning_r01"}
|
||||
|
||||
data: {"type":"text-start","id":"text_xyz789"}
|
||||
|
||||
data: {"type":"text-delta","id":"text_xyz789","delta":"综合分析后，结论是..."}
|
||||
|
||||
data: {"type":"text-end","id":"text_xyz789"}
|
||||
|
||||
data: {"type":"finish-step"}
|
||||
|
||||
data: {"type":"finish"}
|
||||
|
||||
data: [DONE]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 标准消息流顺序
|
||||
|
||||
```
|
||||
start
|
||||
└─ start-step
|
||||
├─ [data-thinking-step × N] 思考步骤（可多次）
|
||||
├─ [reasoning-start/delta/end] 推理过程（可选）
|
||||
├─ [tool-input-start] 工具调用开始（可多次）
|
||||
├─ [tool-input-available] 工具参数确认
|
||||
├─ [tool-output-available] 工具结果
|
||||
├─ [data-terminal-info] 终端提示（可多次）
|
||||
├─ [source-url / source-document] 来源引用（可选）
|
||||
├─ text-start
|
||||
│ └─ text-delta × N 流式文本
|
||||
└─ text-end
|
||||
finish-step
|
||||
finish
|
||||
[DONE]
|
||||
```
|
||||
@@ -34,6 +34,7 @@ from app.gateway.routers import (
|
||||
mcp,
|
||||
memory,
|
||||
models,
|
||||
multi_llm,
|
||||
notifications,
|
||||
public_scheduled_tasks,
|
||||
recommended_questions,
|
||||
@@ -796,6 +797,9 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
|
||||
# AI Writing multi-agent API is mounted at /api/ai-writing
|
||||
app.include_router(ai_writing.router)
|
||||
|
||||
# Multi-LLM routing agent API is mounted at /api/multi-llm
|
||||
app.include_router(multi_llm.router)
|
||||
|
||||
# Scheduled task API is mounted at /api/scheduled-tasks
|
||||
app.include_router(scheduled_tasks.router)
|
||||
|
||||
|
||||
@@ -33,6 +33,7 @@ _PUBLIC_PATH_PREFIXES: tuple[str, ...] = (
|
||||
"/redoc",
|
||||
"/openapi.json",
|
||||
"/api/public/",
|
||||
"/api/multi-llm/",
|
||||
)
|
||||
|
||||
# Exact auth paths that are public (login/register/status check).
|
||||
|
||||
@@ -41,6 +41,9 @@ def should_check_csrf(request: Request) -> bool:
|
||||
# Exempt /api/v1/auth/me endpoint
|
||||
if path == "/api/v1/auth/me":
|
||||
return False
|
||||
# Exempt multi-llm streaming endpoint (API-to-API calls, no browser session)
|
||||
if path.startswith("/api/multi-llm"):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
from . import ai_writing, artifacts, assistants_compat, mcp, models, recommended_questions, skills, suggestions, thread_runs, threads, uploads, writing
|
||||
from . import ai_writing, artifacts, assistants_compat, mcp, models, multi_llm, recommended_questions, skills, suggestions, thread_runs, threads, uploads, writing
|
||||
|
||||
__all__ = ["ai_writing", "artifacts", "assistants_compat", "mcp", "models", "recommended_questions", "skills", "suggestions", "threads", "thread_runs", "uploads", "writing"]
|
||||
__all__ = ["ai_writing", "artifacts", "assistants_compat", "mcp", "models", "multi_llm", "recommended_questions", "skills", "suggestions", "threads", "thread_runs", "uploads", "writing"]
|
||||
@@ -0,0 +1,89 @@
|
||||
"""Multi-LLM routing agent — SSE streaming endpoint.
|
||||
|
||||
POST /api/multi-llm/{thread_id}/stream
|
||||
Body: {"message": "..."}
|
||||
Returns text/event-stream with frames (all as "data: <json>\\n\\n"):
|
||||
{"type": "routing", "system": "B"} ← router decision
|
||||
{"type": "thinking", "content": "...", "system": "B"} ← reasoning (if available)
|
||||
{"type": "text", "content": "...", "system": "B"} ← final text chunks
|
||||
{"type": "done"} ← stream end
|
||||
{"type": "error", "message": "..."} ← on failure
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
from fastapi.responses import StreamingResponse
|
||||
from langchain_core.messages import HumanMessage
|
||||
from langgraph.types import Checkpointer
|
||||
from pydantic import BaseModel
|
||||
|
||||
from app.gateway.deps import get_checkpointer
|
||||
from deerflow.agents.multi_llm.graph import build_multi_llm_graph
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/multi-llm", tags=["multi-llm"])
|
||||
|
||||
# Lazy singleton: built once on first request, reusing the app-scoped checkpointer.
|
||||
_graph = None
|
||||
_graph_lock = asyncio.Lock()
|
||||
|
||||
|
||||
async def _get_graph(checkpointer: Checkpointer):
|
||||
global _graph
|
||||
if _graph is None:
|
||||
async with _graph_lock:
|
||||
if _graph is None:
|
||||
_graph = build_multi_llm_graph(checkpointer=checkpointer)
|
||||
return _graph
|
||||
|
||||
|
||||
class ChatRequest(BaseModel):
|
||||
message: str
|
||||
|
||||
|
||||
@router.post("/{thread_id}/stream", summary="Multi-LLM routing agent (SSE stream)")
|
||||
async def multi_llm_stream(
|
||||
thread_id: str,
|
||||
body: ChatRequest,
|
||||
request: Request,
|
||||
checkpointer: Checkpointer = Depends(get_checkpointer),
|
||||
) -> StreamingResponse:
|
||||
graph = await _get_graph(checkpointer)
|
||||
lg_config = {"configurable": {"thread_id": thread_id}}
|
||||
input_state = {"messages": [HumanMessage(content=body.message)]}
|
||||
|
||||
async def generate():
|
||||
try:
|
||||
async for event in graph.astream_events(input_state, config=lg_config, version="v2"):
|
||||
if await request.is_disconnected():
|
||||
break
|
||||
|
||||
ev = event["event"]
|
||||
|
||||
if ev == "on_custom_event" and event.get("name") == "llm_chunk":
|
||||
d = event["data"]
|
||||
yield f"data: {json.dumps({'type': d['type'], 'content': d['content'], 'system': d['system']}, ensure_ascii=False)}\n\n"
|
||||
|
||||
elif ev == "on_chain_end" and event.get("name") == "router":
|
||||
output = event.get("data", {}).get("output", {})
|
||||
system = output.get("selected_system", "") if isinstance(output, dict) else ""
|
||||
if system:
|
||||
yield f"data: {json.dumps({'type': 'routing', 'system': system})}\n\n"
|
||||
|
||||
yield f"data: {json.dumps({'type': 'done'})}\n\n"
|
||||
|
||||
except Exception as e:
|
||||
logger.exception("multi_llm stream failed: thread=%s", thread_id)
|
||||
yield f"data: {json.dumps({'type': 'error', 'message': str(e)})}\n\n"
|
||||
|
||||
return StreamingResponse(
|
||||
generate(),
|
||||
media_type="text/event-stream",
|
||||
headers={"X-Accel-Buffering": "no", "Cache-Control": "no-cache"},
|
||||
)
|
||||
Whitespace-only changes.
@@ -0,0 +1,51 @@
|
||||
"""Multi-LLM routing graph.
|
||||
|
||||
Systems: fast, bigmodel, hxchat, moreModes, rag, taskChat
|
||||
|
||||
Graph layout:
|
||||
START → router → (conditional) → fast / bigmodel / hxchat / more_modes / rag / task_chat → END
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from langgraph.graph import END, START, StateGraph
|
||||
|
||||
from deerflow.agents.multi_llm.nodes.router import route_by_system, router_node
|
||||
from deerflow.agents.multi_llm.nodes.systems import (
|
||||
bigmodel_node,
|
||||
fast_node,
|
||||
hxchat_node,
|
||||
more_modes_node,
|
||||
rag_node,
|
||||
task_chat_node,
|
||||
)
|
||||
from deerflow.agents.multi_llm.state import MultiLLMState
|
||||
|
||||
# LangGraph node name → node function
|
||||
_SYSTEM_NODES = {
|
||||
"fast": fast_node,
|
||||
"bigmodel": bigmodel_node,
|
||||
"hxchat": hxchat_node,
|
||||
"more_modes": more_modes_node,
|
||||
"rag": rag_node,
|
||||
"task_chat": task_chat_node,
|
||||
}
|
||||
|
||||
|
||||
def build_multi_llm_graph(checkpointer=None):
|
||||
g = StateGraph(MultiLLMState)
|
||||
|
||||
g.add_node("router", router_node)
|
||||
for name, fn in _SYSTEM_NODES.items():
|
||||
g.add_node(name, fn)
|
||||
|
||||
g.add_edge(START, "router")
|
||||
g.add_conditional_edges(
|
||||
"router",
|
||||
route_by_system,
|
||||
{name: name for name in _SYSTEM_NODES},
|
||||
)
|
||||
for name in _SYSTEM_NODES:
|
||||
g.add_edge(name, END)
|
||||
|
||||
return g.compile(checkpointer=checkpointer)
|
||||
Whitespace-only changes.
+77
@@ -0,0 +1,77 @@
|
||||
"""Router node: uses an LLM to decide which system to invoke."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Literal
|
||||
|
||||
from langchain_core.messages import HumanMessage, SystemMessage
|
||||
from langchain_core.runnables import RunnableConfig
|
||||
|
||||
from deerflow.agents.multi_llm.state import MultiLLMState
|
||||
from deerflow.config.app_config import get_app_config
|
||||
from deerflow.models import create_chat_model
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
SystemName = Literal["fast", "bigmodel", "hxchat", "moreModes", "rag", "taskChat"]
|
||||
|
||||
_VALID_SYSTEMS: tuple[str, ...] = ("fast", "bigmodel", "hxchat", "moreModes", "rag", "taskChat")
|
||||
|
||||
_ROUTER_SYSTEM_PROMPT = f"""你是一个路由助手，负责将用户请求分配给最合适的AI系统。
|
||||
|
||||
可用系统（共 {len(_VALID_SYSTEMS)} 个）：
|
||||
- fast : 快速问答，适合通用问题、知识库检索
|
||||
- bigmodel : 大模型，适合复杂推理、长文本分析
|
||||
- hxchat : HX 对话，适合...（TODO: 补充说明）
|
||||
- moreModes : 多模态，适合...（TODO: 补充说明）
|
||||
- rag : RAG 检索增强，适合需要查找文档的问题
|
||||
- taskChat : 任务对话，适合需要执行多步骤任务的场景
|
||||
|
||||
请根据用户请求，仅输出一个系统名称（区分大小写），不要输出任何其他内容。
|
||||
有效值：fast / bigmodel / hxchat / moreModes / rag / taskChat"""
|
||||
|
||||
|
||||
async def router_node(state: MultiLLMState, config: RunnableConfig) -> dict:
|
||||
messages = state.get("messages", [])
|
||||
if not messages:
|
||||
return {"selected_system": "fast"}
|
||||
|
||||
last_human = next(
|
||||
(
|
||||
m
|
||||
for m in reversed(messages)
|
||||
if getattr(m, "type", None) == "human" or (isinstance(m, dict) and m.get("role") == "user")
|
||||
),
|
||||
None,
|
||||
)
|
||||
user_content = ""
|
||||
if last_human is not None:
|
||||
user_content = last_human.content if hasattr(last_human, "content") else last_human.get("content", "")
|
||||
|
||||
try:
|
||||
app_config = get_app_config()
|
||||
llm = create_chat_model(app_config=app_config)
|
||||
prompt = [
|
||||
SystemMessage(content=_ROUTER_SYSTEM_PROMPT),
|
||||
HumanMessage(content=user_content or "请帮我处理这个请求"),
|
||||
]
|
||||
response = await llm.ainvoke(prompt, config={"run_name": "multi_llm_router"})
|
||||
raw = (response.content or "").strip()
|
||||
selected = raw if raw in _VALID_SYSTEMS else "fast"
|
||||
except Exception:
|
||||
logger.exception("multi_llm router failed, defaulting to fast")
|
||||
selected = "fast"
|
||||
|
||||
return {"selected_system": selected}
|
||||
|
||||
|
||||
def route_by_system(state: MultiLLMState) -> str:
|
||||
system = state.get("selected_system", "fast")
|
||||
if system not in _VALID_SYSTEMS:
|
||||
system = "fast"
|
||||
# LangGraph node names use underscores
|
||||
return system.lower() if system in ("fast", "bigmodel", "hxchat", "rag") else {
|
||||
"moreModes": "more_modes",
|
||||
"taskChat": "task_chat",
|
||||
}[system]
|
||||
+223
@@ -0,0 +1,223 @@
|
||||
"""System nodes: call each external LLM system and stream chunks via custom events.
|
||||
|
||||
Systems: fast, bigmodel, hxchat, moreModes, rag, taskChat
|
||||
|
||||
System URLs are read from config.yaml under multi_llm_systems.<name>_url,
|
||||
falling back to environment variables FAST_URL / BIGMODEL_URL / etc.
|
||||
|
||||
fast uses a custom request format (Vercel AI SDK / POST /new_chat):
|
||||
{"chat_id": <thread_id>, "search_space_id": <id>, "user_query": "..."}
|
||||
search_space_id from config multi_llm_systems.fast_search_space_id
|
||||
or env FAST_SEARCH_SPACE_ID (default 0).
|
||||
|
||||
All other systems use generic {"messages": [...]} format (placeholder until
|
||||
their real formats are confirmed).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
|
||||
import httpx
|
||||
from langchain_core.callbacks.manager import adispatch_custom_event
|
||||
from langchain_core.messages import AIMessage, HumanMessage
|
||||
from langchain_core.runnables import RunnableConfig
|
||||
|
||||
from deerflow.agents.multi_llm.parsers import PARSERS
|
||||
from deerflow.agents.multi_llm.state import MultiLLMState
|
||||
from deerflow.config.app_config import get_app_config
|
||||
from deerflow.models import create_chat_model
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
SYSTEM_NAMES = ["fast", "bigmodel", "hxchat", "moreModes", "rag", "taskChat"]
|
||||
|
||||
# Maps system name → env var suffix for URL lookup
|
||||
_ENV_VAR_MAP = {
|
||||
"fast": "FAST_URL",
|
||||
"bigmodel": "BIGMODEL_URL",
|
||||
"hxchat": "HXCHAT_URL",
|
||||
"moreModes": "MOREMODES_URL",
|
||||
"rag": "RAG_URL",
|
||||
"taskChat": "TASKCHAT_URL",
|
||||
}
|
||||
|
||||
|
||||
def _get_multi_llm_config() -> dict:
|
||||
try:
|
||||
cfg = get_app_config()
|
||||
return getattr(cfg, "multi_llm_systems", None) or {}
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def _get_url(system: str) -> str:
|
||||
c = _get_multi_llm_config()
|
||||
config_key = f"{system}_url"
|
||||
env_key = _ENV_VAR_MAP.get(system, f"{system.upper()}_URL")
|
||||
return c.get(config_key) or os.environ.get(env_key, "")
|
||||
|
||||
|
||||
def _extract_last_user_content(messages: list) -> str:
|
||||
for m in reversed(messages):
|
||||
if hasattr(m, "type") and m.type == "human":
|
||||
return m.content if isinstance(m.content, str) else str(m.content)
|
||||
if isinstance(m, dict) and m.get("role") == "user":
|
||||
return m.get("content", "")
|
||||
return ""
|
||||
|
||||
|
||||
def _build_generic_payload(messages: list) -> dict:
|
||||
result = []
|
||||
for m in messages:
|
||||
if hasattr(m, "type"):
|
||||
role = "user" if m.type == "human" else "assistant"
|
||||
content = m.content if isinstance(m.content, str) else str(m.content)
|
||||
elif isinstance(m, dict):
|
||||
role = m.get("role", "user")
|
||||
content = m.get("content", "")
|
||||
else:
|
||||
continue
|
||||
result.append({"role": role, "content": content})
|
||||
return {"messages": result}
|
||||
|
||||
|
||||
_FAST_TEST_URL = "http://47.88.25.99:8000/api/v1/new_chat"
|
||||
_FAST_TEST_TOKEN = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiI5MmFhNmRkMC03NDA5LTQ0NDEtOGQzNC1iNTY4YmZjMzg5ODIiLCJhdWQiOlsiZmFzdGFwaS11c2VyczphdXRoIl0sImV4cCI6MTc3OTQzNjY0NH0.Da_KDlQaEc-cH95AyUzcx82pwZmU3t-9fGmJC9tRF7E"
|
||||
_FAST_TEST_MODEL = "Qwen3.5-plus"
|
||||
_FAST_TEST_SEARCH_SPACE_ID = 197
|
||||
_FAST_TEST_CHAT_ID = 600
|
||||
|
||||
|
||||
def _build_fast_payload(messages: list, thread_id: str = "") -> dict: # noqa: ARG001
|
||||
"""Build the /new_chat request payload for the fast system."""
|
||||
return {
|
||||
"model_name": _FAST_TEST_MODEL,
|
||||
"chat_id": _FAST_TEST_CHAT_ID,
|
||||
"search_space_id": _FAST_TEST_SEARCH_SPACE_ID,
|
||||
"user_query": _extract_last_user_content(messages),
|
||||
}
|
||||
|
||||
|
||||
async def _call_system(system: str, messages: list, thread_id: str = "") -> str:
|
||||
if system == "fast":
|
||||
url = _FAST_TEST_URL
|
||||
headers = {"Authorization": f"Bearer {_FAST_TEST_TOKEN}"}
|
||||
else:
|
||||
url = _get_url(system)
|
||||
headers = {}
|
||||
if not url:
|
||||
raise ValueError(
|
||||
f"No URL configured for system '{system}'. "
|
||||
f"Set multi_llm_systems.{system}_url in config.yaml or {_ENV_VAR_MAP.get(system, system.upper() + '_URL')} env var."
|
||||
)
|
||||
|
||||
parser = PARSERS[system]
|
||||
payload = _build_fast_payload(messages, thread_id) if system == "fast" else _build_generic_payload(messages)
|
||||
text_buf = ""
|
||||
|
||||
async with httpx.AsyncClient(timeout=120) as client:
|
||||
async with client.stream("POST", url, json=payload, headers=headers) as resp:
|
||||
resp.raise_for_status()
|
||||
async for chunk in parser.parse(resp):
|
||||
if chunk.content:
|
||||
if chunk.type == "thinking":
|
||||
await adispatch_custom_event("llm_chunk", {"type": "thinking", "content": chunk.content, "system": system})
|
||||
else:
|
||||
text_buf += chunk.content
|
||||
await adispatch_custom_event("llm_chunk", {"type": "text", "content": chunk.content, "system": system})
|
||||
|
||||
return text_buf
|
||||
|
||||
|
||||
def _extract_chunk_parts(chunk: object) -> tuple[str, str]:
|
||||
"""Return (thinking_delta, text_delta) from a LangChain streaming chunk."""
|
||||
thinking = ""
|
||||
text = ""
|
||||
content = getattr(chunk, "content", "")
|
||||
if isinstance(content, str):
|
||||
text = content
|
||||
elif isinstance(content, list):
|
||||
for block in content:
|
||||
if isinstance(block, str):
|
||||
text += block
|
||||
elif isinstance(block, dict):
|
||||
btype = block.get("type", "")
|
||||
if btype in ("thinking", "thinking_delta"):
|
||||
thinking += block.get("thinking", "")
|
||||
elif btype in ("text", "text_delta"):
|
||||
text += block.get("text", "")
|
||||
elif btype == "reasoning":
|
||||
thinking += block.get("content", "")
|
||||
if not thinking:
|
||||
kwargs = getattr(chunk, "additional_kwargs", {}) or {}
|
||||
r = kwargs.get("reasoning_content") or ""
|
||||
if isinstance(r, str):
|
||||
thinking = r
|
||||
return thinking, text
|
||||
|
||||
|
||||
async def _fallback_to_local_llm(messages: list, config: RunnableConfig) -> str:
|
||||
"""Fall back to the local configured LLM when an external system fails."""
|
||||
app_config = get_app_config()
|
||||
llm = create_chat_model(app_config=app_config)
|
||||
|
||||
lc_messages = []
|
||||
for m in messages:
|
||||
if hasattr(m, "type"):
|
||||
content = m.content if isinstance(m.content, str) else str(m.content)
|
||||
lc_messages.append(HumanMessage(content=content) if m.type == "human" else AIMessage(content=content))
|
||||
elif isinstance(m, dict):
|
||||
content = m.get("content", "")
|
||||
lc_messages.append(HumanMessage(content=content) if m.get("role") == "user" else AIMessage(content=content))
|
||||
|
||||
text_buf = ""
|
||||
async for chunk in llm.astream(lc_messages):
|
||||
thinking, text = _extract_chunk_parts(chunk)
|
||||
if thinking:
|
||||
await adispatch_custom_event("llm_chunk", {"type": "thinking", "content": thinking, "system": "local"})
|
||||
if text:
|
||||
text_buf += text
|
||||
await adispatch_custom_event("llm_chunk", {"type": "text", "content": text, "system": "local"})
|
||||
return text_buf
|
||||
|
||||
|
||||
async def _system_node(system: str, state: MultiLLMState, config: RunnableConfig) -> dict:
|
||||
messages = state.get("messages", [])
|
||||
thread_id = (config or {}).get("configurable", {}).get("thread_id", "")
|
||||
try:
|
||||
result = await _call_system(system, messages, thread_id=thread_id)
|
||||
except Exception as e:
|
||||
logger.warning("multi_llm system '%s' failed (%s), falling back to local LLM", system, e)
|
||||
await adispatch_custom_event("llm_chunk", {"type": "fallback", "system": "local", "reason": str(e)})
|
||||
try:
|
||||
result = await _fallback_to_local_llm(messages, config)
|
||||
except Exception as e2:
|
||||
logger.exception("local LLM fallback also failed")
|
||||
result = f"[系统异常，请稍后重试: {e2}]"
|
||||
return {"messages": [AIMessage(content=result)]}
|
||||
|
||||
|
||||
async def fast_node(state: MultiLLMState, config: RunnableConfig) -> dict:
|
||||
return await _system_node("fast", state, config)
|
||||
|
||||
|
||||
async def bigmodel_node(state: MultiLLMState, config: RunnableConfig) -> dict:
|
||||
return await _system_node("bigmodel", state, config)
|
||||
|
||||
|
||||
async def hxchat_node(state: MultiLLMState, config: RunnableConfig) -> dict:
|
||||
return await _system_node("hxchat", state, config)
|
||||
|
||||
|
||||
async def more_modes_node(state: MultiLLMState, config: RunnableConfig) -> dict:
|
||||
return await _system_node("moreModes", state, config)
|
||||
|
||||
|
||||
async def rag_node(state: MultiLLMState, config: RunnableConfig) -> dict:
|
||||
return await _system_node("rag", state, config)
|
||||
|
||||
|
||||
async def task_chat_node(state: MultiLLMState, config: RunnableConfig) -> dict:
|
||||
return await _system_node("taskChat", state, config)
|
||||
+88
@@ -0,0 +1,88 @@
|
||||
"""Stream parsers for 6 external LLM systems.
|
||||
|
||||
Each system returns a different streaming format. All parsers normalize the
|
||||
raw HTTP byte stream into a sequence of StreamChunk(type, content) where
|
||||
type is either "thinking" (reasoning process) or "text" (final output).
|
||||
|
||||
Systems: fast, bigmodel, hxchat, moreModes, rag, taskChat
|
||||
Currently implemented: fast (Vercel AI SDK Data Stream Protocol / POST /new_chat)
|
||||
Others: TODO — using plain-text placeholder until formats are confirmed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import abc
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from typing import AsyncIterator, Literal
|
||||
|
||||
|
||||
@dataclass
|
||||
class StreamChunk:
|
||||
type: Literal["thinking", "text"]
|
||||
content: str
|
||||
|
||||
|
||||
class BaseSystemParser(abc.ABC):
|
||||
"""Parse a raw httpx response stream into normalized StreamChunks."""
|
||||
|
||||
@abc.abstractmethod
|
||||
async def parse(self, response) -> AsyncIterator[StreamChunk]:
|
||||
...
|
||||
|
||||
|
||||
class FastParser(BaseSystemParser):
|
||||
"""fast: Vercel AI SDK Data Stream Protocol (POST /new_chat).
|
||||
|
||||
Relevant event types:
|
||||
text-delta → StreamChunk(type="text", content=payload["delta"])
|
||||
reasoning-delta → StreamChunk(type="thinking", content=payload["delta"])
|
||||
|
||||
All other event types (start, finish, start-step, finish-step,
|
||||
tool-input-*, tool-output-*, data-*, source-*, error, etc.) are silently
|
||||
skipped — they are UI-control signals, not content.
|
||||
"""
|
||||
|
||||
async def parse(self, response) -> AsyncIterator[StreamChunk]:
|
||||
async for line in response.aiter_lines():
|
||||
if not line.startswith("data:"):
|
||||
continue
|
||||
raw = line[5:].strip()
|
||||
if not raw or raw == "[DONE]":
|
||||
continue
|
||||
try:
|
||||
payload = json.loads(raw)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
ev_type = payload.get("type", "")
|
||||
if ev_type == "text-delta":
|
||||
delta = payload.get("delta", "")
|
||||
if delta:
|
||||
yield StreamChunk(type="text", content=delta)
|
||||
elif ev_type == "reasoning-delta":
|
||||
delta = payload.get("delta", "")
|
||||
if delta:
|
||||
yield StreamChunk(type="thinking", content=delta)
|
||||
|
||||
|
||||
class PlainTextParser(BaseSystemParser):
|
||||
"""Fallback: plain text stream, no thinking content.
|
||||
|
||||
Used as placeholder for bigmodel / hxchat / moreModes / rag / taskChat
|
||||
until their real streaming formats are confirmed.
|
||||
"""
|
||||
|
||||
async def parse(self, response) -> AsyncIterator[StreamChunk]:
|
||||
async for chunk in response.aiter_text():
|
||||
if chunk:
|
||||
yield StreamChunk(type="text", content=chunk)
|
||||
|
||||
|
||||
PARSERS: dict[str, BaseSystemParser] = {
|
||||
"fast": FastParser(),
|
||||
"bigmodel": PlainTextParser(), # TODO: replace with real parser
|
||||
"hxchat": PlainTextParser(), # TODO: replace with real parser
|
||||
"moreModes": PlainTextParser(), # TODO: replace with real parser
|
||||
"rag": PlainTextParser(), # TODO: replace with real parser
|
||||
"taskChat": PlainTextParser(), # TODO: replace with real parser
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
from typing import Annotated
|
||||
|
||||
from langgraph.graph.message import add_messages
|
||||
from typing_extensions import NotRequired, TypedDict
|
||||
|
||||
|
||||
class MultiLLMState(TypedDict):
|
||||
messages: Annotated[list, add_messages]
|
||||
selected_system: NotRequired[str]
|
||||
@@ -1,51 +1,52 @@
|
||||
[project]
|
||||
name = "deer-flow"
|
||||
version = "0.1.0"
|
||||
description = "LangGraph-based AI agent system with sandbox execution capabilities"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
dependencies = [
|
||||
"deerflow-harness",
|
||||
"fastapi>=0.115.0",
|
||||
"httpx>=0.28.0",
|
||||
"python-multipart>=0.0.26",
|
||||
"sse-starlette>=2.1.0",
|
||||
"uvicorn[standard]>=0.34.0",
|
||||
"lark-oapi>=1.4.0",
|
||||
"slack-sdk>=3.33.0",
|
||||
"python-telegram-bot>=21.0",
|
||||
"langgraph-sdk>=0.1.51",
|
||||
"markdown-to-mrkdwn>=0.3.1",
|
||||
"wecom-aibot-python-sdk>=0.1.6",
|
||||
"dingtalk-stream>=0.24.3",
|
||||
"bcrypt>=4.0.0",
|
||||
"pyjwt>=2.9.0",
|
||||
"email-validator>=2.0.0",
|
||||
"croniter>=6.2.2",
|
||||
]
|
||||
|
||||
[project]
|
||||
name = "deer-flow"
|
||||
version = "0.1.0"
|
||||
description = "LangGraph-based AI agent system with sandbox execution capabilities"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
dependencies = [
|
||||
"deerflow-harness",
|
||||
"fastapi>=0.115.0",
|
||||
"httpx>=0.28.0",
|
||||
"python-multipart>=0.0.26",
|
||||
"sse-starlette>=2.1.0",
|
||||
"uvicorn[standard]>=0.34.0",
|
||||
"lark-oapi>=1.4.0",
|
||||
"slack-sdk>=3.33.0",
|
||||
"python-telegram-bot>=21.0",
|
||||
"langgraph-sdk>=0.1.51",
|
||||
"markdown-to-mrkdwn>=0.3.1",
|
||||
"wecom-aibot-python-sdk>=0.1.6",
|
||||
"dingtalk-stream>=0.24.3",
|
||||
"bcrypt>=4.0.0",
|
||||
"pyjwt>=2.9.0",
|
||||
"email-validator>=2.0.0",
|
||||
"croniter>=6.2.2",
|
||||
"asyncmy>=0.2.11",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
postgres = ["deerflow-harness[postgres]"]
|
||||
mysql = ["deerflow-harness[mysql]"]
|
||||
|
||||
[dependency-groups]
|
||||
dev = [
|
||||
"prompt-toolkit>=3.0.0",
|
||||
"pytest>=9.0.3",
|
||||
"pytest-asyncio>=1.3.0",
|
||||
"ruff>=0.14.11",
|
||||
]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
markers = [
|
||||
"no_auto_user: disable the conftest autouse contextvar fixture for this test",
|
||||
]
|
||||
|
||||
[tool.uv]
|
||||
index-url = "https://pypi.org/simple"
|
||||
|
||||
[tool.uv.workspace]
|
||||
members = ["packages/harness"]
|
||||
|
||||
[tool.uv.sources]
|
||||
deerflow-harness = { workspace = true }
|
||||
|
||||
[dependency-groups]
|
||||
dev = [
|
||||
"prompt-toolkit>=3.0.0",
|
||||
"pytest>=9.0.3",
|
||||
"pytest-asyncio>=1.3.0",
|
||||
"ruff>=0.14.11",
|
||||
]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
markers = [
|
||||
"no_auto_user: disable the conftest autouse contextvar fixture for this test",
|
||||
]
|
||||
|
||||
[tool.uv]
|
||||
index-url = "https://pypi.org/simple"
|
||||
|
||||
[tool.uv.workspace]
|
||||
members = ["packages/harness"]
|
||||
|
||||
[tool.uv.sources]
|
||||
deerflow-harness = { workspace = true }
|
||||
Generated
+2719
-2717
File diff suppressed because it is too large.
Load diff
Reference in new issue
Block a user