feat(roundtable): 第二/三步模型自动容错(出问题自动换下一个模型)
模型出问题时按 config.yaml models[] 顺序自动切到下一个模型继续,并告知用户 「『X』模型异常 → 已切换『Y』继续」。前台实时 + 后台挂起两条路径都覆盖。 - 中间件兜底/熔断 AIMessage 打 LLM_ERROR_MARKER 标记,is_llm_error_message 可识别 - roundtable_model_fallback: fallback_model_chain 候选链 + reason_text + 总开关 env ROUNDTABLE_MODEL_FALLBACK - 触发: 异常(start_run失败/idle卡死) / 兜底错误文案 / 交付为空(仅seat/leader) - 后台 InProcessRoundtableGateway._run_turn_with_fallback 包住每轮 run - 前台 _special_run/_leader_run 收尾检测失败→model_switch 帧+换模型重流 - 结构化抽取 flow_extract round1 astream 同款换模型循环 - Step2 单席位异常隔离: 单席位失败不拖垮整批/整链/整个作业 - 时间线插「系统提示」气泡 + toast + 诊断 model_switch 事件 - Tests: test_roundtable_model_fallback.py Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
ae2706dad3
commit
d383b0e9dd
18 files changed
+1033
-174
No files matched your search
@@ -18,12 +18,18 @@ import asyncio
|
||||
import logging
|
||||
import os
|
||||
import uuid
|
||||
from collections.abc import Callable
|
||||
from datetime import UTC, datetime
|
||||
from types import SimpleNamespace
|
||||
from typing import Any
|
||||
|
||||
from langgraph.checkpoint.base import empty_checkpoint
|
||||
|
||||
from app.gateway.roundtable_model_fallback import (
|
||||
fallback_model_chain,
|
||||
is_llm_error_message,
|
||||
reason_text,
|
||||
)
|
||||
from app.gateway.roundtable_run_policy import roundtable_run_policy
|
||||
from app.gateway.roundtable_seat_skills import append_seat_skill_directive
|
||||
from app.gateway.routers._roundtable_seed import (
|
||||
@@ -36,6 +42,7 @@ from app.gateway.routers._roundtable_seed import (
|
||||
from deerflow.agents.roundtable_orchestrator.report_json import extract_report_json
|
||||
from deerflow.agents.roundtable_orchestrator.types import (
|
||||
LeaderTurn,
|
||||
ModelSwitch,
|
||||
ReportResult,
|
||||
SeatRef,
|
||||
SeatTurn,
|
||||
@@ -521,6 +528,31 @@ _SEAT_MODE_PARAMS: dict[str, tuple[bool | None, str | None, bool]] = {
|
||||
}
|
||||
|
||||
|
||||
def _exception_reason(exc: BaseException) -> str:
|
||||
"""把单轮 run 抛出的异常归类成模型容错原因码(用于换模型提示)。
|
||||
|
||||
idle 守护判定的卡死(``_await_run_with_idle_guard`` 抛 ``... stalled ...``)→ ``timeout``;
|
||||
其余(start_run 失败 / DB 抖动 / 其它)→ ``exception``。
|
||||
"""
|
||||
text = str(exc).lower()
|
||||
if "stall" in text or "idle" in text or "timeout" in text or "timed out" in text:
|
||||
return "timeout"
|
||||
return "exception"
|
||||
|
||||
|
||||
def _seat_empty_reason(messages: list[dict[str, Any]]) -> str | None:
|
||||
"""席位/取数轮校验:最后一条有正文的 AI 消息为空 → ``empty``(触发换模型)。"""
|
||||
return "empty" if not _latest_ai_text(messages).strip() else None
|
||||
|
||||
|
||||
def _leader_failed_reason(messages: list[dict[str, Any]]) -> str | None:
|
||||
"""总控轮校验:既无正文、也无任何工具调用(派活/澄清)→ ``empty``(视为模型没干活)。"""
|
||||
ai = _latest_ai_message(messages)
|
||||
has_text = bool(_latest_ai_text(messages).strip())
|
||||
has_tools = bool(_normalize_tool_calls((ai or {}).get("tool_calls")))
|
||||
return None if (has_text or has_tools) else "empty"
|
||||
|
||||
|
||||
# ── 网关实现 ─────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@@ -585,25 +617,116 @@ class InProcessRoundtableGateway:
|
||||
self._seat_intro = _build_seat_intro(self._agents)
|
||||
return thread_ids, COORDINATOR_AGENT_ID
|
||||
|
||||
async def _run_turn_with_fallback(
|
||||
self,
|
||||
*,
|
||||
thread_id: str,
|
||||
agent_name: str,
|
||||
message: str,
|
||||
requested_model: str | None,
|
||||
excluded_tools: list[str],
|
||||
skill_stop_names: list[str],
|
||||
thinking_enabled: bool,
|
||||
reasoning_effort: str,
|
||||
subagent_enabled: bool = False,
|
||||
stage: str,
|
||||
cycle: int | None = None,
|
||||
validate: Callable[[list[dict[str, Any]]], str | None] | None = None,
|
||||
) -> tuple[list[dict[str, Any]], list[ModelSwitch], str | None]:
|
||||
"""跑一轮 agent,并在「模型出问题」时自动切到下一个模型重试。
|
||||
|
||||
触发换模型的三种情形(与用户选择一致):
|
||||
① ``_run_agent_turn`` 抛异常(start_run 失败 / idle 卡死超时);
|
||||
② 返回了中间件兜底错误文案(``is_llm_error_message`` 命中);
|
||||
③ 调用方 ``validate`` 判定无效(如席位交付为空 / 总控既无正文又无派活)。
|
||||
|
||||
候选模型链按 ``config.yaml models[]`` 顺序(``fallback_model_chain``),首个为
|
||||
``requested_model``(空则作业级 ``self._model``,再空则配置默认)。返回
|
||||
``(最终 messages, 发生的换模型记录, 实际所用模型)``。全部模型都失败时返回最后一次的
|
||||
messages(让既有的「空交付 / 产物缺失」逻辑兜底,绝不抛异常拖垮整个作业)。
|
||||
"""
|
||||
base = requested_model if (requested_model not in (None, "")) else self._model
|
||||
chain = fallback_model_chain(base) or [base]
|
||||
switches: list[ModelSwitch] = []
|
||||
messages: list[dict[str, Any]] = []
|
||||
prev_model: str | None = None
|
||||
prev_reason: str | None = None
|
||||
used_model: str | None = chain[0]
|
||||
for idx, candidate in enumerate(chain):
|
||||
if idx > 0:
|
||||
label = reason_text(prev_reason)
|
||||
switches.append(ModelSwitch(failed_model=prev_model or "", next_model=candidate or "", reason_label=label))
|
||||
logger.warning(
|
||||
"roundtable %s '%s' switching model %s -> %s (%s)",
|
||||
stage, agent_name, prev_model, candidate, prev_reason,
|
||||
)
|
||||
await self._diag.record(
|
||||
stage=stage, level="warning", event="model_switch",
|
||||
message=f"「{agent_name}」模型「{prev_model}」{label},已自动切换到「{candidate}」重试",
|
||||
detail={"failed_model": prev_model, "next_model": candidate, "reason": prev_reason, "attempt": idx + 1},
|
||||
agent_name=agent_name, cycle=cycle,
|
||||
)
|
||||
used_model = candidate
|
||||
try:
|
||||
messages = await _run_agent_turn(
|
||||
self._app,
|
||||
user_id=self._user_id,
|
||||
thread_id=thread_id,
|
||||
agent_name=agent_name,
|
||||
message=message,
|
||||
model=candidate,
|
||||
excluded_tools=excluded_tools,
|
||||
skill_stop_names=skill_stop_names,
|
||||
thinking_enabled=thinking_enabled,
|
||||
reasoning_effort=reasoning_effort,
|
||||
subagent_enabled=subagent_enabled,
|
||||
diag=self._diag,
|
||||
stage=stage,
|
||||
cycle=cycle,
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001 — 模型故障:归类后尝试下一个模型
|
||||
prev_model, prev_reason, messages = candidate, _exception_reason(exc), []
|
||||
if idx + 1 < len(chain):
|
||||
continue
|
||||
await self._diag.record(
|
||||
stage=stage, level="error", event="model_fallback_exhausted",
|
||||
message=f"「{agent_name}」所有备选模型均失败(最后 {candidate}:{prev_reason})",
|
||||
detail={"reason": prev_reason, "tried": chain}, agent_name=agent_name, cycle=cycle,
|
||||
)
|
||||
return messages, switches, used_model
|
||||
# 跑通了:再看是否「兜底错误文案」或 validate 判定无效。
|
||||
reason = is_llm_error_message(_latest_ai_message(messages))
|
||||
if reason is None and validate is not None:
|
||||
reason = validate(messages)
|
||||
if reason is None:
|
||||
return messages, switches, used_model
|
||||
prev_model, prev_reason = candidate, reason
|
||||
if idx + 1 >= len(chain):
|
||||
await self._diag.record(
|
||||
stage=stage, level="error", event="model_fallback_exhausted",
|
||||
message=f"「{agent_name}」所有备选模型均未产出有效结果(最后 {candidate}:{reason})",
|
||||
detail={"reason": reason, "tried": chain}, agent_name=agent_name, cycle=cycle,
|
||||
)
|
||||
return messages, switches, used_model
|
||||
return messages, switches, used_model
|
||||
|
||||
async def run_leader(self, *, thread_ids, coordinator_name, message, model) -> LeaderTurn:
|
||||
msg = message
|
||||
if self._first_leader and self._seat_intro:
|
||||
msg = self._seat_intro + "\n\n" + message
|
||||
self._first_leader = False
|
||||
policy = roundtable_run_policy("leader")
|
||||
messages = await _run_agent_turn(
|
||||
self._app,
|
||||
user_id=self._user_id,
|
||||
messages, switches, _used = await self._run_turn_with_fallback(
|
||||
thread_id=thread_ids[coordinator_name],
|
||||
agent_name=coordinator_name,
|
||||
message=msg,
|
||||
model=model or self._model,
|
||||
requested_model=model,
|
||||
excluded_tools=policy.excluded_tools,
|
||||
skill_stop_names=policy.skill_stop_names,
|
||||
thinking_enabled=policy.thinking_enabled,
|
||||
reasoning_effort=policy.reasoning_effort,
|
||||
diag=self._diag,
|
||||
stage="step2_leader",
|
||||
validate=_leader_failed_reason,
|
||||
)
|
||||
ai = _latest_ai_message(messages)
|
||||
# tool_calls 取最后一条 AI 消息(派活就在这条);正文取最后一条有文字的 AI 消息。
|
||||
@@ -617,7 +740,7 @@ class InProcessRoundtableGateway:
|
||||
if clar is not None:
|
||||
cargs = clar.get("args") or {}
|
||||
question = str(cargs.get("question") or "").strip() or "总控需要补充信息,请补充后继续。"
|
||||
return LeaderTurn(final_text=final_text, clarification=question, tool_calls=turn_tool_calls)
|
||||
return LeaderTurn(final_text=final_text, clarification=question, tool_calls=turn_tool_calls, switches=switches)
|
||||
|
||||
# 派活解析(复用 multi_agent 纯函数)
|
||||
from app.gateway.routers.multi_agent import _extract_dispatch
|
||||
@@ -630,7 +753,7 @@ class InProcessRoundtableGateway:
|
||||
dispatched.append((name, task))
|
||||
except Exception: # noqa: BLE001
|
||||
logger.warning("dispatch parse failed for tool_call", exc_info=True)
|
||||
return LeaderTurn(final_text=final_text, dispatched=dispatched, tool_calls=turn_tool_calls)
|
||||
return LeaderTurn(final_text=final_text, dispatched=dispatched, tool_calls=turn_tool_calls, switches=switches)
|
||||
|
||||
async def run_seat(self, *, thread_ids, agent_id, task, model) -> SeatTurn:
|
||||
policy = roundtable_run_policy("seat")
|
||||
@@ -654,13 +777,11 @@ class InProcessRoundtableGateway:
|
||||
override = _SEAT_MODE_PARAMS.get(self._seat_modes.get(agent_id, ""))
|
||||
if override is not None:
|
||||
seat_thinking, seat_reasoning, seat_subagent = override
|
||||
messages = await _run_agent_turn(
|
||||
self._app,
|
||||
user_id=self._user_id,
|
||||
messages, switches, _used = await self._run_turn_with_fallback(
|
||||
thread_id=thread_ids[agent_id],
|
||||
agent_name=agent_id,
|
||||
message=seat_task,
|
||||
model=model or self._model,
|
||||
requested_model=model,
|
||||
excluded_tools=policy.excluded_tools,
|
||||
skill_stop_names=policy.skill_stop_names,
|
||||
thinking_enabled=(
|
||||
@@ -670,10 +791,14 @@ class InProcessRoundtableGateway:
|
||||
policy.reasoning_effort if seat_reasoning is None else seat_reasoning
|
||||
),
|
||||
subagent_enabled=seat_subagent,
|
||||
diag=self._diag,
|
||||
stage="step2_seat",
|
||||
validate=_seat_empty_reason,
|
||||
)
|
||||
return SeatTurn(
|
||||
final_text=_latest_ai_text(messages),
|
||||
tool_calls=_tool_calls_since_last_human(messages),
|
||||
switches=switches,
|
||||
)
|
||||
return SeatTurn(final_text=_latest_ai_text(messages), tool_calls=_tool_calls_since_last_human(messages))
|
||||
|
||||
async def create_report_thread(self) -> str:
|
||||
ensure_roundtable_functional_agents()
|
||||
@@ -686,21 +811,22 @@ class InProcessRoundtableGateway:
|
||||
summary = ""
|
||||
html: str | None = None
|
||||
message = prompt
|
||||
switches: list[ModelSwitch] = []
|
||||
current_model = model # 跨产物自愈轮记住实际所用模型:一旦换到可用模型,后续续写轮就从它起。
|
||||
for attempt in range(1 + _ARTIFACT_RETRY_LIMIT):
|
||||
messages = await _run_agent_turn(
|
||||
self._app,
|
||||
user_id=self._user_id,
|
||||
messages, sw, used = await self._run_turn_with_fallback(
|
||||
thread_id=report_thread_id,
|
||||
agent_name=REPORT_AGENT_ID,
|
||||
message=message,
|
||||
model=model or self._model,
|
||||
requested_model=current_model,
|
||||
excluded_tools=policy.excluded_tools,
|
||||
skill_stop_names=policy.skill_stop_names,
|
||||
thinking_enabled=policy.thinking_enabled,
|
||||
reasoning_effort=policy.reasoning_effort,
|
||||
diag=self._diag,
|
||||
stage="step3_report",
|
||||
)
|
||||
switches.extend(sw)
|
||||
current_model = used or current_model
|
||||
summary = _latest_ai_text(messages) or summary
|
||||
html = _read_report_html(self._app, report_thread_id, self._user_id)
|
||||
if _html_complete(html):
|
||||
@@ -727,7 +853,7 @@ class InProcessRoundtableGateway:
|
||||
)
|
||||
# 截断的 HTML 也比纯文字强(浏览器容忍未闭合标签),仅在完全没有文件时才回退正文。
|
||||
final_html = html if (html and html.strip()) else summary
|
||||
return ReportResult(html=final_html, summary=summary, model=model or self._model)
|
||||
return ReportResult(html=final_html, summary=summary, model=current_model or self._model, switches=switches)
|
||||
|
||||
async def create_summary_thread(self) -> str:
|
||||
ensure_roundtable_functional_agents()
|
||||
@@ -743,21 +869,22 @@ class InProcessRoundtableGateway:
|
||||
closing = ""
|
||||
md: str | None = None
|
||||
message = prompt
|
||||
switches: list[ModelSwitch] = []
|
||||
current_model = model
|
||||
for attempt in range(1 + _ARTIFACT_RETRY_LIMIT):
|
||||
messages = await _run_agent_turn(
|
||||
self._app,
|
||||
user_id=self._user_id,
|
||||
messages, sw, used = await self._run_turn_with_fallback(
|
||||
thread_id=summary_thread_id,
|
||||
agent_name=SUMMARY_AGENT_ID,
|
||||
message=message,
|
||||
model=model or self._model,
|
||||
requested_model=current_model,
|
||||
excluded_tools=policy.excluded_tools,
|
||||
skill_stop_names=policy.skill_stop_names,
|
||||
thinking_enabled=policy.thinking_enabled,
|
||||
reasoning_effort=policy.reasoning_effort,
|
||||
diag=self._diag,
|
||||
stage="step3_summary",
|
||||
)
|
||||
switches.extend(sw)
|
||||
current_model = used or current_model
|
||||
closing = _latest_ai_text(messages) or closing
|
||||
md = _read_summary_md(self._app, summary_thread_id, self._user_id)
|
||||
if md and md.strip():
|
||||
@@ -783,7 +910,7 @@ class InProcessRoundtableGateway:
|
||||
agent_name=SUMMARY_AGENT_ID,
|
||||
)
|
||||
final_md = md if (md and md.strip()) else closing
|
||||
return SummaryResult(md=final_md, closing=closing, model=model or self._model, thread_id=summary_thread_id)
|
||||
return SummaryResult(md=final_md, closing=closing, model=current_model or self._model, thread_id=summary_thread_id, switches=switches)
|
||||
|
||||
async def run_dashboard_data(self, *, prompt, model) -> str | None:
|
||||
"""大屏多页的结构化数据:进程内驱动**专职**「大屏多页数据智能体」
|
||||
@@ -799,21 +926,20 @@ class InProcessRoundtableGateway:
|
||||
thread_id = await _create_thread(self._app)
|
||||
policy = roundtable_run_policy("dashboard")
|
||||
message = prompt
|
||||
current_model = model
|
||||
for attempt in range(1 + _ARTIFACT_RETRY_LIMIT):
|
||||
messages = await _run_agent_turn(
|
||||
self._app,
|
||||
user_id=self._user_id,
|
||||
messages, _sw, used = await self._run_turn_with_fallback(
|
||||
thread_id=thread_id,
|
||||
agent_name=DASHBOARD_AGENT_ID,
|
||||
message=message,
|
||||
model=model or self._model,
|
||||
requested_model=current_model,
|
||||
excluded_tools=policy.excluded_tools,
|
||||
skill_stop_names=policy.skill_stop_names,
|
||||
thinking_enabled=policy.thinking_enabled,
|
||||
reasoning_effort=policy.reasoning_effort,
|
||||
diag=self._diag,
|
||||
stage="step3_dashboard",
|
||||
)
|
||||
current_model = used or current_model
|
||||
body = _extract_report_json(_latest_ai_text(messages))
|
||||
if body:
|
||||
return body
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
"""圆桌会商「模型自动容错(换下一个模型)」共享助手。
|
||||
|
||||
会商的第二步(leader / seat)与第三步(summary / report / dashboard / structure)单轮执行时,
|
||||
若所用模型出现问题(抛异常 / idle 卡死 / 中间件兜底错误文案 / 交付为空),应自动切到
|
||||
**下一个可用模型**继续,并把「『X』模型异常 → 已切换『Y』模型」告知用户。
|
||||
|
||||
本模块只放**与执行路径无关**的纯逻辑,供前台(``multi_agent.py``)与后台
|
||||
(``roundtable_inprocess_gateway.py``)两条路径共用:
|
||||
|
||||
- ``model_fallback_enabled()``:总开关(env ``ROUNDTABLE_MODEL_FALLBACK``,默认开);
|
||||
- ``fallback_model_chain(current)``:按 ``config.yaml`` ``models[]`` 顺序给出候选模型链
|
||||
``[当前模型, …其余]``(去重保序);
|
||||
- ``reason_text(reason)``:把失败原因码翻译成简短中文(用于用户提示);
|
||||
- 复用 harness 的 ``is_llm_error_message`` 识别中间件兜底文案。
|
||||
|
||||
放在 app 层(依赖 ``deerflow.config`` + harness 的中间件常量,皆为 app→harness 合法方向)。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
|
||||
from deerflow.agents.middlewares.llm_error_handling_middleware import (
|
||||
is_llm_error_message as is_llm_error_message, # re-export for callers
|
||||
)
|
||||
from deerflow.config import get_app_config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# 失败原因码 → 用户可见简短中文(用于「『X』模型<原因>,已切换…」提示)。
|
||||
_REASON_TEXT: dict[str, str] = {
|
||||
"quota": "额度不足/账单异常",
|
||||
"auth": "鉴权失败/无访问权限",
|
||||
"busy": "服务繁忙",
|
||||
"transient": "网络/连接异常",
|
||||
"upstream": "上游响应异常",
|
||||
"circuit": "连续失败被熔断",
|
||||
"generic": "调用失败",
|
||||
"exception": "调用异常",
|
||||
"timeout": "长时间无响应(卡死)",
|
||||
"empty": "未产出有效内容",
|
||||
}
|
||||
|
||||
|
||||
def reason_text(reason: str | None) -> str:
|
||||
"""把失败原因码翻译成简短中文(未知码 → 「调用失败」)。"""
|
||||
if not reason:
|
||||
return "调用失败"
|
||||
return _REASON_TEXT.get(reason, "调用失败")
|
||||
|
||||
|
||||
def model_fallback_enabled() -> bool:
|
||||
"""模型自动容错总开关(默认开;设 ``ROUNDTABLE_MODEL_FALLBACK=0/false/off`` 关闭)。
|
||||
|
||||
关闭时 ``fallback_model_chain`` 只返回单个候选(原模型),即恢复旧行为:模型失败就失败,
|
||||
不再尝试其它模型。
|
||||
"""
|
||||
raw = (os.getenv("ROUNDTABLE_MODEL_FALLBACK", "1") or "1").strip().lower()
|
||||
return raw not in {"0", "false", "no", "off"}
|
||||
|
||||
|
||||
def _max_attempts() -> int:
|
||||
"""最多尝试的模型个数(含首个)。``ROUNDTABLE_MODEL_FALLBACK_MAX``,0/未设 = 不限(试完全部)。"""
|
||||
try:
|
||||
return max(0, int(os.getenv("ROUNDTABLE_MODEL_FALLBACK_MAX", "0") or "0"))
|
||||
except (TypeError, ValueError):
|
||||
return 0
|
||||
|
||||
|
||||
def _config_model_names() -> list[str]:
|
||||
try:
|
||||
cfg = get_app_config()
|
||||
return [m.name for m in (cfg.models or []) if getattr(m, "name", None)]
|
||||
except Exception: # noqa: BLE001 — 拿不到配置(内存后端/测试桩)时退化为空
|
||||
logger.debug("fallback_model_chain: get_app_config().models unavailable", exc_info=True)
|
||||
return []
|
||||
|
||||
|
||||
def fallback_model_chain(current: str | None) -> list[str]:
|
||||
"""按 ``config.yaml`` ``models[]`` 顺序构造候选模型链:``[当前模型, …其余]``,去重保序。
|
||||
|
||||
- ``current`` 为空/None → 从 ``models[0]`` 起(即后端默认模型),后接其余;
|
||||
- ``current`` 在配置内 → 它排第一,其余按配置顺序补齐;
|
||||
- ``current`` 不在配置内(自定义/未知名)→ 仍排第一,再补齐全部配置模型;
|
||||
- 总开关关闭 → 只返回单个候选(``current`` 或默认模型),等同旧行为;
|
||||
- 拿不到任何配置模型 → 返回 ``[current]``(current 为空则空列表,调用方自行退化为不换模型)。
|
||||
"""
|
||||
all_names = _config_model_names()
|
||||
cur = (current or "").strip()
|
||||
ordered: list[str] = []
|
||||
if cur:
|
||||
ordered.append(cur)
|
||||
for n in all_names:
|
||||
if n and n not in ordered:
|
||||
ordered.append(n)
|
||||
if not ordered:
|
||||
return [cur] if cur else []
|
||||
if not model_fallback_enabled():
|
||||
return ordered[:1]
|
||||
cap = _max_attempts()
|
||||
if cap > 0:
|
||||
ordered = ordered[:cap]
|
||||
return ordered
|
||||
@@ -31,12 +31,12 @@ from fastapi.responses import StreamingResponse
|
||||
from langchain_core.messages import HumanMessage, SystemMessage
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from deerflow.agents.roundtable_orchestrator.extraction_spec import build_structure_directive
|
||||
|
||||
from app.gateway.roundtable_model_fallback import fallback_model_chain, reason_text
|
||||
from app.gateway.routers._roundtable_seed import (
|
||||
STRUCTURE_AGENT_ID,
|
||||
ensure_roundtable_functional_agents,
|
||||
)
|
||||
from deerflow.agents.roundtable_orchestrator.extraction_spec import build_structure_directive
|
||||
from deerflow.config import get_app_config
|
||||
from deerflow.config.agents_config import load_agent_config, load_agent_soul
|
||||
from deerflow.models.factory import create_chat_model
|
||||
@@ -596,28 +596,60 @@ async def extract_flow_stream(request: Request, payload: FlowExtractRequest = Bo
|
||||
return
|
||||
|
||||
# ① 抽取 / 编辑(正文 + 思考(reasoning) 拼成 `<think>…</think>正文` 增量逐帧吐,前端实时显示)
|
||||
# 模型自动容错:结构化抽取非常重要——某模型 astream 抛错 / 产出空时,按 config.yaml
|
||||
# models[] 顺序切到下一个模型重抽,并发 model_switch 帧让前端把抽取面板已累计的半截内容重置。
|
||||
round1_messages = _edit_messages(payload) if is_edit else _extract_messages(payload)
|
||||
model = create_chat_model(name=resolved_model, thinking_enabled=True)
|
||||
chain = fallback_model_chain(resolved_model) or [resolved_model]
|
||||
extracted = ""
|
||||
think_open = False
|
||||
async for chunk in model.astream(round1_messages):
|
||||
c, r = _chunk_parts(chunk)
|
||||
out = ""
|
||||
if r:
|
||||
if not think_open:
|
||||
out += "<think>"
|
||||
think_open = True
|
||||
out += r
|
||||
if c:
|
||||
used_model = chain[0]
|
||||
prev_model: str | None = None
|
||||
prev_reason: str | None = None
|
||||
for idx, candidate in enumerate(chain):
|
||||
if idx > 0:
|
||||
yield _frame(
|
||||
{
|
||||
"event": "model_switch",
|
||||
"agent_name": "结构化抽取",
|
||||
"failed_model": prev_model or "",
|
||||
"next_model": candidate or "",
|
||||
"reason": reason_text(prev_reason),
|
||||
}
|
||||
)
|
||||
used_model = candidate
|
||||
extracted = ""
|
||||
think_open = False
|
||||
try:
|
||||
model = create_chat_model(name=candidate, thinking_enabled=True)
|
||||
async for chunk in model.astream(round1_messages):
|
||||
c, r = _chunk_parts(chunk)
|
||||
out = ""
|
||||
if r:
|
||||
if not think_open:
|
||||
out += "<think>"
|
||||
think_open = True
|
||||
out += r
|
||||
if c:
|
||||
if think_open:
|
||||
out += "</think>"
|
||||
think_open = False
|
||||
out += c
|
||||
extracted += c
|
||||
if out:
|
||||
yield _frame({"event": "extract_delta", "content": out})
|
||||
if think_open:
|
||||
out += "</think>"
|
||||
think_open = False
|
||||
out += c
|
||||
extracted += c
|
||||
if out:
|
||||
yield _frame({"event": "extract_delta", "content": out})
|
||||
if think_open:
|
||||
yield _frame({"event": "extract_delta", "content": "</think>"})
|
||||
yield _frame({"event": "extract_delta", "content": "</think>"})
|
||||
except Exception as exc: # noqa: BLE001 — 该模型抽取失败:换下一个模型重抽
|
||||
prev_model, prev_reason = candidate, "exception"
|
||||
logger.warning("flow-extract round1 model %s failed: %s", candidate, exc)
|
||||
if idx + 1 < len(chain):
|
||||
continue
|
||||
raise
|
||||
if extracted.strip():
|
||||
break
|
||||
# 产出为空:换下一个模型(已是最后一个则继续走后续解析兜底)。
|
||||
prev_model, prev_reason = candidate, "empty"
|
||||
if idx + 1 >= len(chain):
|
||||
break
|
||||
yield _frame({"event": "extract_done"})
|
||||
|
||||
# ② 校验 / 修复(同样流式 + 思考实时显示)
|
||||
@@ -625,7 +657,7 @@ async def extract_flow_stream(request: Request, payload: FlowExtractRequest = Bo
|
||||
validated = ""
|
||||
graph: dict[str, Any] | None = None
|
||||
try:
|
||||
vmodel = create_chat_model(name=resolved_model, thinking_enabled=True)
|
||||
vmodel = create_chat_model(name=used_model, thinking_enabled=True)
|
||||
think_open = False
|
||||
async for chunk in vmodel.astream(_validate_messages(payload, extracted)):
|
||||
c, r = _chunk_parts(chunk)
|
||||
|
||||
@@ -63,6 +63,7 @@ from fastapi.responses import StreamingResponse
|
||||
from langgraph.checkpoint.base import empty_checkpoint
|
||||
|
||||
from app.gateway.roundtable_diag import record_foreground_diag
|
||||
from app.gateway.roundtable_model_fallback import fallback_model_chain, is_llm_error_message, reason_text
|
||||
from app.gateway.roundtable_run_policy import merge_skill_stop_names, roundtable_run_policy
|
||||
from deerflow.agents.roundtable_orchestrator.extraction_spec import build_summary_directive
|
||||
|
||||
@@ -1547,21 +1548,31 @@ async def _leader_run(
|
||||
if files:
|
||||
_read_tools = {"read_file", "ls", "grep", "view_image"}
|
||||
leader_excluded = [t for t in leader_excluded if t not in _read_tools]
|
||||
body = _run_payload(
|
||||
agent_name=agent_name,
|
||||
model_name=model_name,
|
||||
thread_id=thread_id,
|
||||
# 注入了席位清单的输入(leader_input);seat_roster 为空时等同 new_message。
|
||||
new_message=leader_input,
|
||||
excluded_tools=leader_excluded,
|
||||
skill_stop_names=effective_stop_names,
|
||||
thinking_enabled=policy.thinking_enabled,
|
||||
reasoning_effort=policy.reasoning_effort,
|
||||
# 人工干预/恢复会重发 leader run 到同一总控 thread;抢占掉没结束的旧 run,避免 409。
|
||||
multitask_strategy="rollback",
|
||||
# 用户随干预上传的参考文件 → 总控可用 read_file 等工具读取后再派活。
|
||||
files=files,
|
||||
)
|
||||
|
||||
# 模型自动容错:按 config.yaml models[] 顺序的候选链。当前模型出问题(兜底错误文案 /
|
||||
# 既无正文又无派活)时切到下一个模型重跑(见循环内检测)。``_mk_body`` 用候选模型重建 body。
|
||||
model_chain = fallback_model_chain(model_name) or [model_name]
|
||||
model_idx = 0
|
||||
current_model = model_chain[0]
|
||||
|
||||
def _mk_body(m: str) -> Any:
|
||||
return _run_payload(
|
||||
agent_name=agent_name,
|
||||
model_name=m,
|
||||
thread_id=thread_id,
|
||||
# 注入了席位清单的输入(leader_input);seat_roster 为空时等同 new_message。
|
||||
new_message=leader_input,
|
||||
excluded_tools=leader_excluded,
|
||||
skill_stop_names=effective_stop_names,
|
||||
thinking_enabled=policy.thinking_enabled,
|
||||
reasoning_effort=policy.reasoning_effort,
|
||||
# 人工干预/恢复会重发 leader run 到同一总控 thread;抢占掉没结束的旧 run,避免 409。
|
||||
multitask_strategy="rollback",
|
||||
# 用户随干预上传的参考文件 → 总控可用 read_file 等工具读取后再派活。
|
||||
files=files,
|
||||
)
|
||||
|
||||
body = _mk_body(current_model)
|
||||
t_payload_built = time.perf_counter()
|
||||
logger.info(
|
||||
"[leader-timing] %s payload_built=%.3fs (broadcast running in background)",
|
||||
@@ -1645,6 +1656,28 @@ async def _leader_run(
|
||||
leader_text = _flatten_content(dispatch_msg.get("content", ""))
|
||||
tool_calls = dispatch_msg.get("tool_calls") or []
|
||||
|
||||
# 模型自动容错:本轮总控产出是「中间件兜底错误文案」或「既无正文又无派活」(模型没干活)
|
||||
# → 若还有备选模型,切到下一个重跑(不消耗派活纠正预算)。先发 model_switch 帧让前端把
|
||||
# 这条失败气泡重置,再用新模型重新流式。最后一个模型仍失败则照常往下走(错误文案当共识展示)。
|
||||
fail_reason = is_llm_error_message(dispatch_msg)
|
||||
if fail_reason is None and not (leader_text or "").strip() and not tool_calls:
|
||||
fail_reason = "empty"
|
||||
if fail_reason and model_idx + 1 < len(model_chain):
|
||||
model_idx += 1
|
||||
next_model = model_chain[model_idx]
|
||||
logger.warning("[leader-fallback] %s 模型 %s 失败(%s),切换到 %s", agent_name, current_model, fail_reason, next_model)
|
||||
yield {
|
||||
"status": "model_switch",
|
||||
"role": "leader",
|
||||
"agent_name": agent_name,
|
||||
"failed_model": current_model,
|
||||
"next_model": next_model,
|
||||
"reason": reason_text(fail_reason),
|
||||
}
|
||||
current_model = next_model
|
||||
body = _mk_body(current_model)
|
||||
continue
|
||||
|
||||
# 澄清分支:与 intent.py 相同字段;status 为字符串 "clarification"(非数组),
|
||||
# 前端据此暂停 runOrchestration 循环,展示 Step 2 澄清条。
|
||||
clarification_call = next(
|
||||
@@ -1665,12 +1698,13 @@ async def _leader_run(
|
||||
"allow_custom": bool(cargs.get("allow_custom", True)),
|
||||
"allow_multiple": resolve_allow_multiple(cargs),
|
||||
"agent_name": agent_name,
|
||||
"model_used": current_model,
|
||||
}
|
||||
return
|
||||
|
||||
if not tool_calls:
|
||||
# 空 status 数组 = 总控不再派活,前端视为共识(hasConsensus)。
|
||||
yield {"status": [], "content": leader_text, "agent_name": agent_name}
|
||||
yield {"status": [], "content": leader_text, "agent_name": agent_name, "model_used": current_model}
|
||||
return
|
||||
|
||||
# 派活前按真实 roster 校验:合法保留、非法丢弃(只丢非法,不殃及同轮合法席位)。
|
||||
@@ -1706,7 +1740,7 @@ async def _leader_run(
|
||||
# 关键路径优先:**先 yield 派活帧**,让前端立刻启动席位 special run;派活广播(告知
|
||||
# 其它席位"总控给 X 派了什么活")改成 detached 后台任务,不再 await 阻塞 —— 否则广播
|
||||
# 读写各席位巨大 thread 状态会卡几十秒,白白拖慢席位启动(见 _spawn_detached_broadcast)。
|
||||
yield {"status": dispatched, "content": leader_text, "agent_name": agent_name}
|
||||
yield {"status": dispatched, "content": leader_text, "agent_name": agent_name, "model_used": current_model}
|
||||
# DAG 派活轮(suppress_pre_broadcast)同样**不**把「总控给 X 派活」广播给兄弟席位 ——
|
||||
# 否则弱模型从历史里看到派活语句又会模仿派活(席位任务由前端 pickDagSeatTask 干净下发,
|
||||
# 跨席位承上靠各 special run 的"完成交付"后广播,不需要这条派活广播)。recommend/chain 照旧。
|
||||
@@ -1805,95 +1839,128 @@ async def _special_run(
|
||||
directive = build_summary_directive(business_code)
|
||||
if directive:
|
||||
seat_message = f"{new_message}\n\n---\n\n{directive}"
|
||||
body = _run_payload(
|
||||
agent_name=agent_name,
|
||||
model_name=model_name,
|
||||
thread_id=thread_id,
|
||||
new_message=seat_message,
|
||||
excluded_tools=seat_excluded,
|
||||
skill_stop_names=merge_skill_stop_names(policy.skill_stop_names, skill_stop_names),
|
||||
# 席位执行模式覆盖 policy 的 thinking/reasoning(None → 回退 policy 默认);
|
||||
# subagent 仅 ultra 档置真。工具裁剪(excluded_tools)仍完全由 policy 决定。
|
||||
thinking_enabled=(policy.thinking_enabled if thinking_enabled is None else thinking_enabled),
|
||||
reasoning_effort=(policy.reasoning_effort if reasoning_effort is None else reasoning_effort),
|
||||
subagent_enabled=subagent_enabled,
|
||||
# ultra 席位才会派子智能体 → 才需要 custom 流回传子任务实时进度(task_running 等)。
|
||||
stream_custom=subagent_enabled,
|
||||
# 干预/重新派活会把同一席位 thread 再跑一次;抢占掉没结束的旧 run,避免 409。
|
||||
multitask_strategy="rollback",
|
||||
# 前序席位完整交付 → run context,供 read_peer_delivery 工具按需取全文(§12.9)。
|
||||
extra_context=(
|
||||
{"roundtable_peer_deliveries": peer_deliveries} if peer_deliveries else None
|
||||
),
|
||||
# 用户随干预上传的参考文件 → 席位可用 read_file 等工具读取。
|
||||
files=files,
|
||||
)
|
||||
# 模型自动容错:按 config.yaml models[] 顺序的候选链。某模型出问题(兜底错误文案;
|
||||
# 真正的研讨席位还含「交付为空」)→ 发 model_switch 帧让前端重置该席位气泡,再用下一个
|
||||
# 模型重跑。Step3 单例(report/summary/dashboard/ingest)的产物是落盘文件,正文可能很短,
|
||||
# 故**不**把「正文为空」当失败(与后台 run_report/summary 的 validate=None 一致),只在真正
|
||||
# 的模型错误时换模型。
|
||||
model_chain = fallback_model_chain(model_name) or [model_name]
|
||||
empty_is_failure = role in ("seat", "seat_parallel")
|
||||
|
||||
def _mk_body(m: str) -> Any:
|
||||
return _run_payload(
|
||||
agent_name=agent_name,
|
||||
model_name=m,
|
||||
thread_id=thread_id,
|
||||
new_message=seat_message,
|
||||
excluded_tools=seat_excluded,
|
||||
skill_stop_names=merge_skill_stop_names(policy.skill_stop_names, skill_stop_names),
|
||||
# 席位执行模式覆盖 policy 的 thinking/reasoning(None → 回退 policy 默认);
|
||||
# subagent 仅 ultra 档置真。工具裁剪(excluded_tools)仍完全由 policy 决定。
|
||||
thinking_enabled=(policy.thinking_enabled if thinking_enabled is None else thinking_enabled),
|
||||
reasoning_effort=(policy.reasoning_effort if reasoning_effort is None else reasoning_effort),
|
||||
subagent_enabled=subagent_enabled,
|
||||
# ultra 席位才会派子智能体 → 才需要 custom 流回传子任务实时进度(task_running 等)。
|
||||
stream_custom=subagent_enabled,
|
||||
# 干预/重新派活会把同一席位 thread 再跑一次;抢占掉没结束的旧 run,避免 409。
|
||||
multitask_strategy="rollback",
|
||||
# 前序席位完整交付 → run context,供 read_peer_delivery 工具按需取全文(§12.9)。
|
||||
extra_context=(
|
||||
{"roundtable_peer_deliveries": peer_deliveries} if peer_deliveries else None
|
||||
),
|
||||
# 用户随干预上传的参考文件 → 席位可用 read_file 等工具读取。
|
||||
files=files,
|
||||
)
|
||||
|
||||
t_payload_built = time.perf_counter()
|
||||
logger.info(
|
||||
"[special-timing] %s payload_built=%.3fs",
|
||||
agent_name,
|
||||
t_payload_built - t0,
|
||||
)
|
||||
|
||||
captured: list[str] = []
|
||||
first_line_ts: float | None = None
|
||||
line_count = 0
|
||||
async for line, done in _stream_upstream(client, base, headers, thread_id, body):
|
||||
if line is not None:
|
||||
if first_line_ts is None:
|
||||
first_line_ts = time.perf_counter()
|
||||
logger.info(
|
||||
"[special-timing] %s first_upstream_line=%.3fs since_entry=%.3fs",
|
||||
agent_name,
|
||||
first_line_ts - t_payload_built,
|
||||
first_line_ts - t0,
|
||||
)
|
||||
line_count += 1
|
||||
yield line
|
||||
elif done is not None:
|
||||
captured = done
|
||||
t_stream_done = time.perf_counter()
|
||||
logger.info(
|
||||
"[special-timing] %s upstream_stream_complete=%.3fs lines=%d since_entry=%.3fs",
|
||||
agent_name,
|
||||
t_stream_done - (first_line_ts or t_payload_built),
|
||||
line_count,
|
||||
t_stream_done - t0,
|
||||
)
|
||||
|
||||
messages = _parse_last_messages(captured)
|
||||
if messages is None:
|
||||
# 与 _leader_run 同理:席位 run 被抢占/断开导致流截断,不抛 502,静默收尾。
|
||||
# WARNING 级:正常是 rollback 抢占;否则查上游 run 的服务端报错(如 DB 连接断)。
|
||||
logger.warning(
|
||||
"[special-timing] %s upstream truncated (no values frame) — ending quietly "
|
||||
"(expected on rollback preemption; otherwise check the upstream run's server-side error)",
|
||||
agent_name,
|
||||
)
|
||||
return
|
||||
# Sub-agent's final deliverable is the latest AIMessage's text content.
|
||||
last_msg: dict[str, Any] = {}
|
||||
for msg in reversed(messages):
|
||||
if isinstance(msg, dict) and (msg.get("type") or "").lower() in ("ai", "aimessage", "aimessagechunk"):
|
||||
last_msg = msg
|
||||
break
|
||||
content = _flatten_content(last_msg.get("content", ""))
|
||||
# 优先用 display_name(中文友好,例如"情报收集")提示 leader,id 放括号备查;
|
||||
# display_name 缺失时退化为仅 agent_name —— 与旧行为一致。
|
||||
speaker_label = f"{display_name}({agent_name})" if display_name else agent_name
|
||||
_prefix = f"子智能体{speaker_label}完成{new_message}工作,交付内容为:"
|
||||
# 广播给**所有**席位(含总控)的都只是**截断摘要** —— 总控 thread 不再囤积各席位全文,
|
||||
# 每轮 leader 的上下文因此保持小、响应快。总控做最终综合需要全文时,由 _leader_run 在
|
||||
# 「综合轮」按需现读各席位 thread 注入(见 _build_full_deliverables_block),不靠历史囤积。
|
||||
brief_msg = _truncate_broadcast_content(_prefix, content)
|
||||
# 先 yield 交付结果让前端立即收口本席位;后广播改 detached,不再 await 阻塞 ——
|
||||
# 否则它读写各席位大状态会让 SSE 多挂十几秒,把下一轮 leader 的启动也一起拖慢。
|
||||
yield {
|
||||
"status": f"子智能体{speaker_label}完成{new_message}工作",
|
||||
"content": content,
|
||||
"agent_name": agent_name,
|
||||
}
|
||||
_spawn_detached_broadcast(base, headers, brief_msg, agent_threads, [agent_name])
|
||||
|
||||
for idx, candidate in enumerate(model_chain):
|
||||
body = _mk_body(candidate)
|
||||
captured: list[str] = []
|
||||
first_line_ts: float | None = None
|
||||
line_count = 0
|
||||
async for line, done in _stream_upstream(client, base, headers, thread_id, body):
|
||||
if line is not None:
|
||||
if first_line_ts is None:
|
||||
first_line_ts = time.perf_counter()
|
||||
logger.info(
|
||||
"[special-timing] %s first_upstream_line=%.3fs since_entry=%.3fs",
|
||||
agent_name,
|
||||
first_line_ts - t_payload_built,
|
||||
first_line_ts - t0,
|
||||
)
|
||||
line_count += 1
|
||||
yield line
|
||||
elif done is not None:
|
||||
captured = done
|
||||
t_stream_done = time.perf_counter()
|
||||
logger.info(
|
||||
"[special-timing] %s upstream_stream_complete=%.3fs lines=%d since_entry=%.3fs",
|
||||
agent_name,
|
||||
t_stream_done - (first_line_ts or t_payload_built),
|
||||
line_count,
|
||||
t_stream_done - t0,
|
||||
)
|
||||
|
||||
messages = _parse_last_messages(captured)
|
||||
if messages is None:
|
||||
# 与 _leader_run 同理:席位 run 被抢占/断开导致流截断,不抛 502,静默收尾。
|
||||
# 这不是模型错误(多半 rollback 抢占),不触发换模型。
|
||||
logger.warning(
|
||||
"[special-timing] %s upstream truncated (no values frame) — ending quietly "
|
||||
"(expected on rollback preemption; otherwise check the upstream run's server-side error)",
|
||||
agent_name,
|
||||
)
|
||||
return
|
||||
# Sub-agent's final deliverable is the latest AIMessage's text content.
|
||||
last_msg: dict[str, Any] = {}
|
||||
for msg in reversed(messages):
|
||||
if isinstance(msg, dict) and (msg.get("type") or "").lower() in ("ai", "aimessage", "aimessagechunk"):
|
||||
last_msg = msg
|
||||
break
|
||||
content = _flatten_content(last_msg.get("content", ""))
|
||||
|
||||
# 模型失败检测:兜底错误文案;研讨席位再加「交付为空」。还有备选模型则换。
|
||||
fail_reason = is_llm_error_message(last_msg)
|
||||
if fail_reason is None and empty_is_failure and not content.strip():
|
||||
fail_reason = "empty"
|
||||
if fail_reason and idx + 1 < len(model_chain):
|
||||
next_model = model_chain[idx + 1]
|
||||
logger.warning("[special-fallback] %s 模型 %s 失败(%s),切换到 %s", agent_name, candidate, fail_reason, next_model)
|
||||
yield {
|
||||
"status": "model_switch",
|
||||
"role": role,
|
||||
"agent_name": agent_name,
|
||||
"failed_model": candidate,
|
||||
"next_model": next_model,
|
||||
"reason": reason_text(fail_reason),
|
||||
}
|
||||
continue
|
||||
|
||||
# 成功(或已用尽备选):发交付终态帧 + detached 后广播。
|
||||
_prefix = f"子智能体{speaker_label}完成{new_message}工作,交付内容为:"
|
||||
# 广播给**所有**席位(含总控)的都只是**截断摘要** —— 总控 thread 不再囤积各席位全文,
|
||||
# 每轮 leader 的上下文因此保持小、响应快。总控做最终综合需要全文时,由 _leader_run 在
|
||||
# 「综合轮」按需现读各席位 thread 注入(见 _build_full_deliverables_block),不靠历史囤积。
|
||||
brief_msg = _truncate_broadcast_content(_prefix, content)
|
||||
# 先 yield 交付结果让前端立即收口本席位;后广播改 detached,不再 await 阻塞 ——
|
||||
# 否则它读写各席位大状态会让 SSE 多挂十几秒,把下一轮 leader 的启动也一起拖慢。
|
||||
yield {
|
||||
"status": f"子智能体{speaker_label}完成{new_message}工作",
|
||||
"content": content,
|
||||
"agent_name": agent_name,
|
||||
"model_used": candidate,
|
||||
}
|
||||
_spawn_detached_broadcast(base, headers, brief_msg, agent_threads, [agent_name])
|
||||
return
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
Reference in new issue
Block a user