1143 lines
52 KiB
Markdown
1143 lines
52 KiB
Markdown
---
|
||
id: plan.week12
|
||
title: "第 12 周：项目三 M3——检查点续跑、人机协作、后台作业、SSE、长期记忆"
|
||
type: week
|
||
domain: plan
|
||
stack: [asyncio, fastapi, langgraph, agent-ops]
|
||
week: [12]
|
||
est_hours: 12
|
||
order: 12
|
||
---
|
||
|
||
# 第 12 周：项目三 M3——检查点续跑、人机协作、后台作业、SSE、长期记忆
|
||
|
||
> 项目三的第 3 周，**整个计划最核心的一周**：第 9 周案例卡里“若重做的设计”在这里落地。上周 `week11-agent-m2-langgraph.md`，下周 `week13-agent-m4-eval-lowcode.md`。
|
||
|
||
**本周目标**：做出能演示的“可靠性”——
|
||
|
||
1. 中途杀进程 → 重启 → 从检查点继续（重点 C）
|
||
2. 信息不足时停下来问人，回答后继续；模型失败后换模型从失败点重试（重点 D）
|
||
3. 编排跑在服务端后台；同一任务只有一个执行者（租约）；业务状态以 `runs` 表为权威（重点 A、B）
|
||
4. SSE 推送进度，关页不影响任务，重连按事件序号补发（重点 C）
|
||
5. 长期记忆：记住用户偏好，下次用上（Memory）
|
||
6. 报告可审计（重点 E）
|
||
|
||
**接口以官方文档为准**：持久化 https://docs.langchain.com/oss/python/langgraph/persistence ，中断 https://docs.langchain.com/oss/python/langgraph/interrupts ，记忆 https://docs.langchain.com/oss/python/langgraph/add-memory 。本周代码在 `langgraph 1.2`、`langgraph-checkpoint-sqlite 3.1` 上跑通。
|
||
|
||
```bash
|
||
uv add langgraph-checkpoint-sqlite aiosqlite fastapi uvicorn
|
||
```
|
||
|
||
## 核心设计：两层状态
|
||
|
||
| 层 | 存什么 | 谁用 | 存在哪 |
|
||
| --- | --- | --- | --- |
|
||
| **检查点**（LangGraph） | 图的内部状态快照（每个节点执行完一份），按 `thread_id` 区分 | 框架用来“从哪一步继续” | SQLite（演示）/ PostgreSQL（生产） |
|
||
| **业务表** `runs` / `events` | 对外状态（排队、运行、等人、完成、失败、取消）、租约、事件流、结论 | 列表页、SSE、权限、审计 | SQLite / PostgreSQL |
|
||
|
||
为什么要两层：检查点是框架内部格式，不适合直接给业务查询和权限控制；**业务表是对外的唯一权威来源**。这正是你在工作里见过的“草稿 / 作业 / 任务状态谁说了算”的问题，这里给出你自己的答案。
|
||
|
||
```mermaid
|
||
stateDiagram-v2
|
||
[*] --> queued: POST /runs
|
||
queued --> running: 领取租约
|
||
running --> completed
|
||
running --> failed: 模型错误等（带错误码）
|
||
running --> awaiting_input: interrupt
|
||
running --> cancelled: 用户取消
|
||
awaiting_input --> queued: POST /input
|
||
failed --> queued: POST /retry（可换模型）
|
||
running --> queued: 进程退出 / 租约过期，看门狗重新领取
|
||
```
|
||
|
||
**本周新增文件**
|
||
|
||
```text
|
||
agent-mini/src/agent_mini/
|
||
├── graph/memory.py # load_memory / save_memory 节点
|
||
├── graph/build.py # 覆盖：加上记忆节点
|
||
├── demo_model.py # 无状态的演示假模型（可加延迟、可模拟失败）
|
||
├── store/__init__.py
|
||
├── store/run_store.py # runs / events 表、租约
|
||
├── runner.py # RunManager：后台运行、续租、看门狗、取消、重试
|
||
├── report.py # 可审计报告
|
||
└── api/
|
||
├── __init__.py
|
||
└── app.py # FastAPI + SSE
|
||
web/index.html
|
||
scripts/demo_crash_resume.sh
|
||
tests/
|
||
├── test_resume.py
|
||
├── test_memory.py
|
||
├── test_run_store.py
|
||
└── test_api.py
|
||
```
|
||
|
||
---
|
||
|
||
## 第 1 天：检查点和续跑
|
||
|
||
### 学习内容
|
||
|
||
- 说明：
|
||
- **checkpointer**：每个节点（准确说是每个“超步”）执行完，自动把 State 存一份快照。用同一个 `thread_id` 再次调用 `ainvoke(None, config)`，就从最近的快照继续。
|
||
- **已完成的节点不会重做；崩溃时正在执行的节点会重做**（至少执行一次）。所以有副作用的操作（建待办、发通知）要幂等——第 10 周 `create_todo` 的 `request_id` 就是为这个准备的。
|
||
- SQLite 检查点只适合演示和单机，官方推荐生产用 PostgreSQL 检查点（`langgraph-checkpoint-postgres`）；这一点写进 README 的“取舍”。
|
||
- `AsyncSqliteSaver.from_conn_string(...)` 是异步上下文管理器，用完自动关闭连接。
|
||
- 案例：用“每次调用模型都慢一点”的假模型，跑到一半取消任务（模拟崩溃），再续跑。
|
||
|
||
```python
|
||
# 文件：agent-mini/src/agent_mini/demo_model.py
|
||
import json
|
||
import re
|
||
import time
|
||
|
||
from langchain_core.messages import AIMessage, BaseMessage
|
||
|
||
from agent_mini.fakes import role_of
|
||
from agent_mini.llm import FakeChatModel
|
||
|
||
|
||
def demo_model(delay: float = 0.0, fail_role: str | None = None) -> FakeChatModel:
|
||
"""无状态的演示模型：只根据收到的上下文决定回答，所以重启、续跑后行为一致。
|
||
- 问题里带“[需要澄清]”且用户还没补充时，协调者会请求问人
|
||
- 第 2 版方案才会被评审通过
|
||
- fail_role 指定的角色会抛错（模拟额度不足），用来演示“换模型重试”"""
|
||
|
||
def respond(messages: list[BaseMessage]) -> AIMessage:
|
||
if delay:
|
||
time.sleep(delay)
|
||
role = role_of(messages)
|
||
if role == fail_role:
|
||
raise RuntimeError("QUOTA_EXCEEDED: 模拟模型额度不足")
|
||
text = str(messages[-1].content)
|
||
roles = re.findall(r"^\[(proposer|critic|human)\]", text, re.M)
|
||
if role == "coordinator":
|
||
if "[需要澄清]" in text and "用户补充：\n（无）" in text:
|
||
return AIMessage(content=json.dumps({"action": "ask_human", "instruction": "预算大概多少？",
|
||
"reason": "缺少预算"}, ensure_ascii=False))
|
||
target = "critic" if roles and roles[-1] == "proposer" else "proposer"
|
||
return AIMessage(content=json.dumps({"action": "dispatch", "target": target,
|
||
"instruction": "继续", "reason": "演示"}))
|
||
if role == "proposer":
|
||
return AIMessage(content=f"方案 v{roles.count('proposer') + 1}：前端 Vue，后端 Go，部署用 Docker")
|
||
if role == "critic":
|
||
ok = "方案 v2" in text
|
||
return AIMessage(content=("考虑周全\nVERDICT: APPROVE" if ok else "缺少成本估算\nVERDICT: REVISE"))
|
||
if role == "summarizer":
|
||
return AIMessage(content="结论：采用方案 v2（Vue + Go + Docker）")
|
||
has_pref = "预算" in text and "→" in text
|
||
return AIMessage(content=json.dumps({"memories": ["用户的预算偏好见补充回答"] if has_pref else []},
|
||
ensure_ascii=False))
|
||
|
||
return FakeChatModel(responder=respond)
|
||
```
|
||
|
||
```python
|
||
# 文件：agent-mini/tests/test_resume.py
|
||
import asyncio
|
||
|
||
import pytest
|
||
from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver
|
||
from langgraph.types import Command
|
||
|
||
from agent_mini.demo_model import demo_model
|
||
from agent_mini.graph.build import build_graph
|
||
|
||
|
||
async def test_crash_then_resume(tmp_path):
|
||
db = str(tmp_path / "cp.db")
|
||
cfg = {"configurable": {"thread_id": "run-1"}}
|
||
async with AsyncSqliteSaver.from_conn_string(db) as cp:
|
||
model = demo_model(delay=0.05)
|
||
graph = build_graph(model, checkpointer=cp)
|
||
seen = []
|
||
|
||
async def consume():
|
||
async for update in graph.astream({"question": "选技术栈", "max_rounds": 4, "round": 0}, cfg,
|
||
stream_mode="updates"):
|
||
seen.extend(update)
|
||
|
||
task = asyncio.create_task(consume())
|
||
while len(seen) < 3: # 跑完 3 个节点后“崩溃”
|
||
await asyncio.sleep(0.01)
|
||
task.cancel()
|
||
with pytest.raises(asyncio.CancelledError):
|
||
await task
|
||
|
||
# 模拟进程重启：新的连接、新的图对象、新的模型对象，只有数据库文件还在
|
||
async with AsyncSqliteSaver.from_conn_string(db) as cp:
|
||
model2 = demo_model()
|
||
graph2 = build_graph(model2, checkpointer=cp)
|
||
snap = await graph2.aget_state(cfg)
|
||
assert snap.next # 还有没执行完的节点
|
||
out = await graph2.ainvoke(None, cfg)
|
||
assert out["end_reason"] == "consensus"
|
||
# 续跑后的模型调用次数明显少于从头跑（已完成的节点没有重做）
|
||
fresh = demo_model()
|
||
await build_graph(fresh).ainvoke({"question": "选技术栈", "max_rounds": 4, "round": 0})
|
||
assert model2.calls < fresh.calls
|
||
|
||
|
||
async def test_interrupt_and_resume(tmp_path):
|
||
async with AsyncSqliteSaver.from_conn_string(str(tmp_path / "cp.db")) as cp:
|
||
graph = build_graph(demo_model(), checkpointer=cp)
|
||
cfg = {"configurable": {"thread_id": "run-2"}}
|
||
out = await graph.ainvoke({"question": "[需要澄清] 做个网站", "max_rounds": 3, "round": 0}, cfg)
|
||
assert out["__interrupt__"][0].value == {"question": "预算大概多少？"}
|
||
snap = await graph.aget_state(cfg)
|
||
assert snap.next == ("ask_human",)
|
||
out = await graph.ainvoke(Command(resume="5 万以内"), cfg) # 人的回答就是 interrupt() 的返回值
|
||
assert out["end_reason"] == "consensus"
|
||
assert out["human_answers"] == ["预算大概多少？ → 5 万以内"]
|
||
|
||
|
||
async def test_fail_then_retry_with_another_model(tmp_path):
|
||
async with AsyncSqliteSaver.from_conn_string(str(tmp_path / "cp.db")) as cp:
|
||
cfg = {"configurable": {"thread_id": "run-3"}}
|
||
broken = build_graph(demo_model(fail_role="critic"), checkpointer=cp)
|
||
with pytest.raises(RuntimeError, match="QUOTA_EXCEEDED"):
|
||
await broken.ainvoke({"question": "q", "max_rounds": 3, "round": 0}, cfg)
|
||
good_model = demo_model()
|
||
out = await build_graph(good_model, checkpointer=cp).ainvoke(None, cfg) # 换模型，从失败的节点继续
|
||
assert out["end_reason"] == "consensus"
|
||
assert out["turns"][0]["role"] == "proposer" # 失败前已完成的提议没有丢
|
||
```
|
||
|
||
**易错点 / 面试常问（补充）**
|
||
|
||
- 易错点：检查点按 `thread_id` 区分，同一个任务续跑必须用同一个 `thread_id`；续跑时传入 `None` 作为输入，表示“从检查点接着跑”。
|
||
- 易错点：崩溃时正在执行的节点会在恢复后**重新执行一遍**（至少一次），所以节点里有副作用的操作（写数据库、发消息、调用付费接口）要做成幂等的。
|
||
- 面试常问：检查点里存了什么（每一步之后的完整状态和下一步要执行的节点）；检查点和业务状态表的区别（前者是编排的草稿，后者是对外的权威状态）。
|
||
- 文档：https://docs.langchain.com/oss/python/langgraph/persistence ，以官方文档为准。
|
||
|
||
### 要建的文件
|
||
|
||
- `demo_model.py`、`tests/test_resume.py`（明天写完记忆节点、覆盖 `build.py` 后一起跑）
|
||
|
||
### 运行方式
|
||
|
||
```bash
|
||
uv run pytest tests/test_resume.py -v
|
||
```
|
||
|
||
### 完成标准
|
||
|
||
- 能讲清：检查点保存了什么、恢复时从哪里继续、崩溃时正在执行的节点会怎样、`interrupt` 恢复时节点为什么会重新执行。
|
||
|
||
---
|
||
|
||
## 第 2 天：长期记忆（Store）
|
||
|
||
### 学习内容
|
||
|
||
- 说明：
|
||
- **短期记忆**：当前任务（线程）内的状态，检查点已经管了。
|
||
- **长期记忆**：跨任务记住的东西——用户偏好（“我们团队只用 Go 和 Vue”）、以前讨论得出的结论。LangGraph 提供 **Store**：按命名空间（如 `(user_id, "memories")`）存取键值，可以配向量索引做语义搜索。
|
||
- 节点通过 `runtime: Runtime` 参数拿到 Store（`runtime.store`），编译图时传入 `store=...`。
|
||
- **写什么进记忆要有规则**：只写稳定的事实和偏好（0~3 条），不写整段对话（成本、隐私、干扰后续回答）；记忆要能查看、能删除。
|
||
- 持久化：`InMemoryStore` 只适合测试；本项目用 `AsyncSqliteStore`（`langgraph.store.sqlite.aio`），生产可用 PostgreSQL 版本。
|
||
- 案例：
|
||
|
||
```python
|
||
# 文件：agent-mini/src/agent_mini/graph/memory.py
|
||
import json
|
||
import re
|
||
import uuid
|
||
|
||
from langchain_core.language_models import BaseChatModel
|
||
from langchain_core.messages import HumanMessage, SystemMessage
|
||
from langgraph.runtime import Runtime
|
||
|
||
from agent_mini.graph.state import AgentState
|
||
|
||
EXTRACT = """你是记忆整理助手。从下面的讨论中提炼 0~3 条值得长期记住的、关于用户的稳定事实或偏好
|
||
（例如团队规模、技术偏好、预算范围）。不要记录一次性的内容。只输出 JSON：{"memories": ["……"]}"""
|
||
|
||
|
||
def namespace(state: AgentState) -> tuple[str, str]:
|
||
return (state.get("user_id", "demo"), "memories")
|
||
|
||
|
||
class MemoryNodes:
|
||
def __init__(self, model: BaseChatModel, limit: int = 5):
|
||
self.model, self.limit = model, limit
|
||
|
||
async def load_memory(self, state: AgentState, runtime: Runtime) -> dict:
|
||
if runtime.store is None:
|
||
return {"memories": []}
|
||
items = await runtime.store.asearch(namespace(state), query=state["question"], limit=self.limit)
|
||
return {"memories": [i.value["text"] for i in items]}
|
||
|
||
async def save_memory(self, state: AgentState, runtime: Runtime) -> dict:
|
||
if runtime.store is None:
|
||
return {} # 没有配置 Store 时跳过，也不调用模型
|
||
answers = "\n".join(state.get("human_answers", [])) or "（无）"
|
||
ai = await self.model.ainvoke([
|
||
SystemMessage(EXTRACT),
|
||
HumanMessage(f"问题：{state['question']}\n用户补充：{answers}\n结论：{state.get('final_answer', '')}"),
|
||
])
|
||
m = re.search(r"\{.*\}", str(ai.content), re.S)
|
||
try:
|
||
memories = json.loads(m.group(0))["memories"][:3] if m else []
|
||
except (json.JSONDecodeError, KeyError, TypeError):
|
||
memories = [] # 记忆提取失败不影响主流程
|
||
for text in memories:
|
||
await runtime.store.aput(namespace(state), uuid.uuid4().hex[:12], {"text": str(text)[:200]})
|
||
return {}
|
||
```
|
||
|
||
```python
|
||
# 文件：agent-mini/src/agent_mini/graph/build.py
|
||
from langchain_core.language_models import BaseChatModel
|
||
from langchain_core.tools import BaseTool
|
||
from langgraph.graph import END, START, StateGraph
|
||
|
||
from agent_mini.graph.memory import MemoryNodes
|
||
from agent_mini.graph.nodes import Nodes
|
||
from agent_mini.graph.routing import route_after_coordinator
|
||
from agent_mini.graph.state import AgentState
|
||
|
||
|
||
def build_graph(model: BaseChatModel, tools: list[BaseTool] | None = None, checkpointer=None, store=None):
|
||
n, mem = Nodes(model, tools), MemoryNodes(model)
|
||
g = StateGraph(AgentState)
|
||
g.add_node("load_memory", mem.load_memory)
|
||
g.add_node("coordinator", n.coordinator)
|
||
g.add_node("proposer", n.proposer)
|
||
g.add_node("critic", n.critic)
|
||
g.add_node("ask_human", n.ask_human)
|
||
g.add_node("summarize", n.summarize)
|
||
g.add_node("save_memory", mem.save_memory)
|
||
g.add_edge(START, "load_memory")
|
||
g.add_edge("load_memory", "coordinator")
|
||
g.add_conditional_edges("coordinator", route_after_coordinator,
|
||
["proposer", "critic", "ask_human", "summarize"])
|
||
for node in ("proposer", "critic", "ask_human"):
|
||
g.add_edge(node, "coordinator")
|
||
g.add_edge("summarize", "save_memory")
|
||
g.add_edge("save_memory", END)
|
||
return g.compile(checkpointer=checkpointer, store=store)
|
||
```
|
||
|
||
```python
|
||
# 文件：agent-mini/tests/test_memory.py
|
||
from langgraph.store.memory import InMemoryStore
|
||
|
||
from agent_mini.demo_model import demo_model
|
||
from agent_mini.graph.build import build_graph
|
||
from agent_mini.llm import FakeChatModel
|
||
|
||
|
||
async def test_memory_saved_and_used_next_time():
|
||
store = InMemoryStore()
|
||
model = demo_model()
|
||
graph = build_graph(model, store=store)
|
||
await graph.ainvoke({"question": "选技术栈", "user_id": "u1", "max_rounds": 3, "round": 0,
|
||
"human_answers": ["预算大概多少？ → 5 万以内"]})
|
||
saved = await store.asearch(("u1", "memories"))
|
||
assert len(saved) == 1
|
||
|
||
seen_prompts: list[str] = []
|
||
inner = demo_model().responder
|
||
|
||
def spy(messages):
|
||
seen_prompts.append(str(messages[-1].content))
|
||
return inner(messages)
|
||
|
||
await build_graph(FakeChatModel(responder=spy), store=store).ainvoke(
|
||
{"question": "再选一次技术栈", "user_id": "u1", "max_rounds": 3, "round": 0})
|
||
assert any("预算偏好" in p for p in seen_prompts) # 第二次任务的提示词里用上了记忆
|
||
assert await store.asearch(("u2", "memories")) == [] # 不同用户互相隔离
|
||
```
|
||
|
||
**易错点 / 面试常问（补充）**
|
||
|
||
- 易错点：长期记忆要按用户分命名空间（本项目是 `(user_id, "memories")`），否则会把 A 用户的偏好用到 B 用户身上。
|
||
- 易错点：不是什么都存成记忆：只存稳定的偏好和事实，并允许用户查看和删除。
|
||
- 面试常问：短期记忆（检查点里的对话状态）和长期记忆（Store，跨会话）怎么分工；记忆内容错了怎么办（可更新、可删除、带来源和时间）。
|
||
|
||
### 要建的文件
|
||
|
||
- `graph/memory.py`、`graph/build.py`（覆盖）、`tests/test_memory.py`
|
||
|
||
### 运行方式
|
||
|
||
```bash
|
||
uv run pytest tests/test_resume.py tests/test_memory.py tests/test_graph.py -v
|
||
```
|
||
|
||
### 完成标准
|
||
|
||
- 测试通过；在 `docs/design-decisions.md` 写下“长期记忆写什么、不写什么、怎么删除”。
|
||
|
||
---
|
||
|
||
## 第 3 天：业务表和租约
|
||
|
||
### 学习内容
|
||
|
||
- 说明：
|
||
- `runs` 表是**对外权威**：状态、问题、模型、结论、错误码、租约；`events` 表是给前端的事件流（自增 `seq` 就是 SSE 的事件 ID）。
|
||
- **租约（lease）防双跑**：执行前用一条**带条件的 UPDATE** 领取——“只有没人持有、或租约已过期时才能领取”，影响行数为 1 才算成功。数据库保证这条语句是原子的，两个进程同时抢只有一个成功。执行中定期续租；进程崩溃后不再续租，租约过期，看门狗让别的进程接手。
|
||
- 和第 3 周 Redis 分布式锁对比：思路一样（带过期时间的独占），这里放在业务库里，状态和锁在同一处，少一个依赖。
|
||
- 演示用同步 `sqlite3`（调用很快）；生产换 PostgreSQL，并把阻塞调用放进线程池或改用异步驱动。
|
||
- 案例：
|
||
|
||
```python
|
||
# 文件：agent-mini/src/agent_mini/store/__init__.py
|
||
```
|
||
|
||
```python
|
||
# 文件：agent-mini/src/agent_mini/store/run_store.py
|
||
import json
|
||
import sqlite3
|
||
import time
|
||
import uuid
|
||
|
||
SCHEMA = """
|
||
CREATE TABLE IF NOT EXISTS runs (
|
||
id TEXT PRIMARY KEY, -- 同时作为 LangGraph 的 thread_id
|
||
user_id TEXT NOT NULL,
|
||
question TEXT NOT NULL,
|
||
status TEXT NOT NULL, -- queued/running/awaiting_input/completed/failed/cancelled
|
||
model TEXT NOT NULL,
|
||
max_rounds INTEGER NOT NULL,
|
||
human_question TEXT, final_answer TEXT, end_reason TEXT, error_code TEXT, error_message TEXT,
|
||
lease_owner TEXT, lease_until REAL,
|
||
created_at REAL NOT NULL, updated_at REAL NOT NULL
|
||
);
|
||
CREATE TABLE IF NOT EXISTS events (
|
||
seq INTEGER PRIMARY KEY AUTOINCREMENT,
|
||
run_id TEXT NOT NULL REFERENCES runs(id),
|
||
type TEXT NOT NULL,
|
||
data TEXT NOT NULL,
|
||
created_at REAL NOT NULL
|
||
);
|
||
CREATE INDEX IF NOT EXISTS idx_events_run_seq ON events(run_id, seq);
|
||
CREATE INDEX IF NOT EXISTS idx_runs_status_lease ON runs(status, lease_until);
|
||
"""
|
||
TERMINAL = {"completed", "failed", "cancelled"}
|
||
|
||
|
||
class RunStore:
|
||
def __init__(self, path: str):
|
||
self.db = sqlite3.connect(path, check_same_thread=False, isolation_level=None) # 自动提交
|
||
self.db.row_factory = sqlite3.Row
|
||
self.db.execute("PRAGMA journal_mode=WAL") # 读写并发更好
|
||
self.db.executescript(SCHEMA)
|
||
|
||
def create_run(self, question: str, user_id: str = "demo", model: str = "fake", max_rounds: int = 3) -> dict:
|
||
now, rid = time.time(), uuid.uuid4().hex[:12]
|
||
self.db.execute("INSERT INTO runs (id, user_id, question, status, model, max_rounds, created_at, updated_at)"
|
||
" VALUES (?, ?, ?, 'queued', ?, ?, ?, ?)", (rid, user_id, question, model, max_rounds, now, now))
|
||
self.append_event(rid, "run_created", {"question": question})
|
||
return self.get_run(rid) # type: ignore[return-value]
|
||
|
||
def get_run(self, run_id: str) -> dict | None:
|
||
row = self.db.execute("SELECT * FROM runs WHERE id = ?", (run_id,)).fetchone()
|
||
return dict(row) if row else None
|
||
|
||
def list_runs(self, user_id: str, limit: int = 50) -> list[dict]:
|
||
rows = self.db.execute("SELECT id, question, status, end_reason, created_at FROM runs WHERE user_id = ?"
|
||
" ORDER BY created_at DESC LIMIT ?", (user_id, limit))
|
||
return [dict(r) for r in rows]
|
||
|
||
def update(self, run_id: str, **fields) -> None:
|
||
fields["updated_at"] = time.time()
|
||
cols = ", ".join(f"{k} = ?" for k in fields)
|
||
self.db.execute(f"UPDATE runs SET {cols} WHERE id = ?", (*fields.values(), run_id))
|
||
|
||
def append_event(self, run_id: str, type_: str, data: dict) -> int:
|
||
cur = self.db.execute("INSERT INTO events (run_id, type, data, created_at) VALUES (?, ?, ?, ?)",
|
||
(run_id, type_, json.dumps(data, ensure_ascii=False), time.time()))
|
||
return cur.lastrowid or 0
|
||
|
||
def list_events(self, run_id: str, after_seq: int = 0) -> list[dict]:
|
||
rows = self.db.execute("SELECT seq, type, data FROM events WHERE run_id = ? AND seq > ? ORDER BY seq",
|
||
(run_id, after_seq))
|
||
return [{"seq": r["seq"], "type": r["type"], "data": json.loads(r["data"])} for r in rows]
|
||
|
||
# ---------- 租约 ----------
|
||
def try_acquire(self, run_id: str, worker: str, lease_s: float) -> bool:
|
||
now = time.time()
|
||
cur = self.db.execute(
|
||
"UPDATE runs SET lease_owner = ?, lease_until = ?, updated_at = ? WHERE id = ?"
|
||
" AND status IN ('queued', 'running') AND (lease_owner IS NULL OR lease_until < ? OR lease_owner = ?)",
|
||
(worker, now + lease_s, now, run_id, now, worker))
|
||
return cur.rowcount == 1
|
||
|
||
def renew(self, run_id: str, worker: str, lease_s: float) -> bool:
|
||
cur = self.db.execute("UPDATE runs SET lease_until = ? WHERE id = ? AND lease_owner = ?",
|
||
(time.time() + lease_s, run_id, worker))
|
||
return cur.rowcount == 1 # 返回 False 说明租约已被别人拿走，应该停止执行
|
||
|
||
def release(self, run_id: str, worker: str) -> None:
|
||
self.db.execute("UPDATE runs SET lease_owner = NULL, lease_until = NULL WHERE id = ? AND lease_owner = ?",
|
||
(run_id, worker))
|
||
|
||
def find_resumable(self) -> list[str]:
|
||
rows = self.db.execute("SELECT id FROM runs WHERE status IN ('queued', 'running')"
|
||
" AND (lease_owner IS NULL OR lease_until < ?)", (time.time(),))
|
||
return [r["id"] for r in rows]
|
||
```
|
||
|
||
```python
|
||
# 文件：agent-mini/tests/test_run_store.py
|
||
import time
|
||
|
||
from agent_mini.store.run_store import RunStore
|
||
|
||
|
||
def test_only_one_worker_gets_the_lease(tmp_path):
|
||
store = RunStore(str(tmp_path / "a.db"))
|
||
run = store.create_run("q")
|
||
assert store.try_acquire(run["id"], "w1", lease_s=30)
|
||
assert not store.try_acquire(run["id"], "w2", lease_s=30) # 别人持有，抢不到
|
||
assert store.try_acquire(run["id"], "w1", lease_s=30) # 自己可以重入
|
||
|
||
|
||
def test_expired_lease_can_be_taken_over(tmp_path):
|
||
store = RunStore(str(tmp_path / "a.db"))
|
||
run = store.create_run("q")
|
||
assert store.try_acquire(run["id"], "w1", lease_s=0.05)
|
||
time.sleep(0.1) # w1“崩溃”，没有续租
|
||
assert store.find_resumable() == [run["id"]]
|
||
assert store.try_acquire(run["id"], "w2", lease_s=30)
|
||
assert not store.renew(run["id"], "w1", lease_s=30) # w1 回来也续不上，应停止
|
||
|
||
|
||
def test_terminal_runs_are_not_acquired_and_events_replay(tmp_path):
|
||
store = RunStore(str(tmp_path / "a.db"))
|
||
run = store.create_run("q")
|
||
store.append_event(run["id"], "node_finished", {"node": "coordinator"})
|
||
store.update(run["id"], status="completed")
|
||
assert not store.try_acquire(run["id"], "w1", lease_s=30)
|
||
events = store.list_events(run["id"])
|
||
assert [e["type"] for e in events] == ["run_created", "node_finished"]
|
||
assert store.list_events(run["id"], after_seq=events[0]["seq"])[0]["type"] == "node_finished"
|
||
```
|
||
|
||
**易错点 / 面试常问（补充）**
|
||
|
||
- 易错点：领取任务要用带条件的 `UPDATE ... WHERE 租约已过期或没人持有`，再看影响行数判断是否抢到；“先查再改”在并发下会被两个进程同时抢到。
|
||
- 易错点：租约时间要大于续租间隔的几倍，否则网络抖一下就被别人抢走；同时要接受“租约过期后旧进程可能还在跑”的情况，每一步之后续租，续租失败（`renew` 返回 `False`）就立即停止，本项目的 `RunManager` 就是这样做的。
|
||
- 面试常问：分布式锁和租约的区别（租约带过期时间，持有者挂了别人能接手）；为什么不用进程内的锁（多个进程 / 多台机器之间不共享内存）。
|
||
|
||
### 要建的文件
|
||
|
||
- `store/__init__.py`、`store/run_store.py`、`tests/test_run_store.py`
|
||
|
||
### 运行方式
|
||
|
||
```bash
|
||
uv run pytest tests/test_run_store.py -v
|
||
```
|
||
|
||
### 完成标准
|
||
|
||
- 测试通过；能讲清“带条件的 UPDATE + 影响行数”为什么能防双跑，租约为什么必须有过期时间。
|
||
|
||
---
|
||
|
||
## 第 4 天：后台运行器（RunManager）和报告
|
||
|
||
### 学习内容
|
||
|
||
- 说明：
|
||
- `RunManager` 把“执行”从“HTTP 连接”里拿出来：`submit()` 只是创建后台任务，接口立刻返回；后台任务领取租约 → 用 `thread_id = run_id` 流式运行图 → 每个节点完成写一条事件 → 续租 → 根据结束时的快照更新 `runs`。
|
||
- 结束时怎么判断：`aget_state()` 的快照里有 `interrupts` → 等待输入；否则 → 完成。抛异常 → 失败（记错误码）。
|
||
- **看门狗**：定期找“该有人跑但没人跑”（`queued/running` 且租约过期）的任务重新提交，这是崩溃恢复的入口。
|
||
- 取消：取消后台任务，状态改为 `cancelled`；服务正常关闭时把任务放回 `queued`，重启后继续。
|
||
- 重试：失败的任务可以换模型（不同模型对应不同的图对象，共享同一个检查点库），从失败的节点继续。
|
||
- **报告**（重点 E）：问题、每一步谁做了什么、调用了哪些工具、结束原因、结论、用的模型——结果可审计。
|
||
- 案例：
|
||
|
||
```python
|
||
# 文件：agent-mini/src/agent_mini/runner.py
|
||
import asyncio
|
||
import os
|
||
import uuid
|
||
from collections.abc import Callable
|
||
from typing import Any
|
||
|
||
from langgraph.types import Command
|
||
|
||
from agent_mini.store.run_store import TERMINAL, RunStore
|
||
|
||
|
||
def error_code(e: Exception) -> str:
|
||
text = str(e)
|
||
for code in ("QUOTA_EXCEEDED", "RATE_LIMITED", "AUTH", "TIMEOUT"):
|
||
if code in text.upper():
|
||
return code
|
||
return type(e).__name__
|
||
|
||
|
||
class RunManager:
|
||
def __init__(self, store: RunStore, graph_for: Callable[[str], Any], lease_s: float = 30):
|
||
self.store, self.graph_for, self.lease_s = store, graph_for, lease_s
|
||
self.worker = f"{os.getpid()}-{uuid.uuid4().hex[:6]}"
|
||
self.tasks: dict[str, asyncio.Task] = {}
|
||
self.cancelling: set[str] = set()
|
||
self.notify: dict[str, asyncio.Event] = {}
|
||
|
||
# ---------- 事件 ----------
|
||
def emit(self, run_id: str, type_: str, data: dict) -> None:
|
||
self.store.append_event(run_id, type_, data)
|
||
self.notify.setdefault(run_id, asyncio.Event()).set() # 唤醒正在等待的 SSE 连接
|
||
|
||
async def wait_event(self, run_id: str, timeout: float) -> None:
|
||
ev = self.notify.setdefault(run_id, asyncio.Event())
|
||
try:
|
||
await asyncio.wait_for(ev.wait(), timeout)
|
||
except TimeoutError:
|
||
pass # 超时也返回，SSE 会去数据库再查一次（兜底：多实例时通知不到）
|
||
ev.clear()
|
||
|
||
# ---------- 提交 / 取消 ----------
|
||
def submit(self, run_id: str, resume: Any = None) -> bool:
|
||
if run_id in self.tasks and not self.tasks[run_id].done():
|
||
return False
|
||
self.tasks[run_id] = asyncio.create_task(self._drive(run_id, resume))
|
||
return True
|
||
|
||
async def cancel(self, run_id: str) -> None:
|
||
self.cancelling.add(run_id)
|
||
task = self.tasks.get(run_id)
|
||
if task and not task.done():
|
||
task.cancel()
|
||
await asyncio.gather(task, return_exceptions=True)
|
||
else:
|
||
self.store.update(run_id, status="cancelled")
|
||
self.emit(run_id, "run_cancelled", {})
|
||
|
||
async def shutdown(self) -> None:
|
||
for task in self.tasks.values():
|
||
task.cancel()
|
||
await asyncio.gather(*self.tasks.values(), return_exceptions=True)
|
||
|
||
async def watchdog(self, interval: float = 5) -> None:
|
||
while True:
|
||
for run_id in self.store.find_resumable():
|
||
self.submit(run_id)
|
||
await asyncio.sleep(interval)
|
||
|
||
# ---------- 执行 ----------
|
||
async def _drive(self, run_id: str, resume: Any) -> None:
|
||
if not self.store.try_acquire(run_id, self.worker, self.lease_s):
|
||
return # 别的进程在跑
|
||
run = self.store.get_run(run_id)
|
||
assert run is not None
|
||
graph = self.graph_for(run["model"])
|
||
cfg = {"configurable": {"thread_id": run_id}}
|
||
self.store.update(run_id, status="running", error_code=None, error_message=None)
|
||
self.emit(run_id, "run_started", {"model": run["model"], "worker": self.worker})
|
||
try:
|
||
snap = await graph.aget_state(cfg)
|
||
if resume is not None:
|
||
payload: Any = Command(resume=resume)
|
||
elif snap.values:
|
||
payload = None # 有检查点：从断点继续
|
||
else:
|
||
payload = {"question": run["question"], "user_id": run["user_id"],
|
||
"max_rounds": run["max_rounds"], "round": 0}
|
||
async for update in graph.astream(payload, cfg, stream_mode="updates"):
|
||
for node, data in update.items():
|
||
if node != "__interrupt__":
|
||
self.emit(run_id, "node_finished", {"node": node, **summarize_update(data)})
|
||
if not self.store.renew(run_id, self.worker, self.lease_s):
|
||
raise RuntimeError("LEASE_LOST") # 租约被别人拿走：立刻停止，避免双跑
|
||
snap = await graph.aget_state(cfg)
|
||
if snap.interrupts:
|
||
question = snap.interrupts[0].value.get("question", "")
|
||
self.store.update(run_id, status="awaiting_input", human_question=question)
|
||
self.emit(run_id, "awaiting_input", {"question": question})
|
||
else:
|
||
v = snap.values
|
||
self.store.update(run_id, status="completed", final_answer=v.get("final_answer"),
|
||
end_reason=v.get("end_reason"), human_question=None)
|
||
self.emit(run_id, "run_completed", {"end_reason": v.get("end_reason"),
|
||
"final_answer": v.get("final_answer")})
|
||
except asyncio.CancelledError:
|
||
if run_id in self.cancelling:
|
||
self.store.update(run_id, status="cancelled")
|
||
self.emit(run_id, "run_cancelled", {})
|
||
else:
|
||
self.store.update(run_id, status="queued") # 服务关闭：放回队列，重启后继续
|
||
raise
|
||
except Exception as e:
|
||
self.store.update(run_id, status="failed", error_code=error_code(e), error_message=str(e)[:300])
|
||
self.emit(run_id, "run_failed", {"error_code": error_code(e), "message": str(e)[:300]})
|
||
finally:
|
||
self.cancelling.discard(run_id)
|
||
self.store.release(run_id, self.worker)
|
||
|
||
|
||
def summarize_update(data: dict | None) -> dict:
|
||
"""事件里只放摘要，完整状态在检查点里。"""
|
||
data = data or {}
|
||
out: dict = {}
|
||
if data.get("turns"):
|
||
t = data["turns"][-1]
|
||
out.update(role=t["role"], content=t["content"][:300], tools=t["tools"])
|
||
if data.get("decision"):
|
||
out["decision"] = {k: data["decision"].get(k) for k in ("action", "target", "reason")}
|
||
if data.get("final_answer"):
|
||
out["final_answer"] = data["final_answer"][:300]
|
||
return out
|
||
|
||
|
||
def is_terminal(status: str) -> bool:
|
||
return status in TERMINAL
|
||
```
|
||
|
||
```python
|
||
# 文件：agent-mini/src/agent_mini/report.py
|
||
def build_report(run: dict, values: dict, events: list[dict]) -> str:
|
||
lines = [f"# 运行报告 {run['id']}", "",
|
||
f"- 问题：{run['question']}", f"- 状态：{run['status']}", f"- 模型：{run['model']}",
|
||
f"- 结束原因：{values.get('end_reason') or run.get('end_reason') or '-'}", ""]
|
||
if values.get("memories"):
|
||
lines += ["## 用到的长期记忆", *[f"- {m}" for m in values["memories"]], ""]
|
||
lines.append("## 过程")
|
||
for i, t in enumerate(values.get("turns", []), start=1):
|
||
tools = f"（调用工具：{', '.join(t['tools'])}）" if t.get("tools") else ""
|
||
lines.append(f"{i}. **{t['role']}**{tools}：{t['content'][:300]}")
|
||
for a in values.get("human_answers", []):
|
||
lines.append(f"- 用户补充：{a}")
|
||
lines += ["", "## 结论", values.get("final_answer") or "（未完成）", "",
|
||
f"## 事件（共 {len(events)} 条）"]
|
||
lines += [f"- #{e['seq']} {e['type']}" for e in events]
|
||
return "\n".join(lines)
|
||
```
|
||
|
||
**易错点 / 面试常问（补充）**
|
||
|
||
- 易错点：后台任务里的异常一定要捕获并写进业务状态（`failed` + 原因），否则任务会一直显示“运行中”。
|
||
- 易错点：服务关闭时要停止领取新任务、释放自己持有的租约，方便其他进程尽快接手。
|
||
- 面试常问：长任务为什么不能跑在 HTTP 请求里（请求会超时、断开；进程重启就丢）；看门狗做什么（定期找出租约过期的任务重新执行）。
|
||
|
||
### 要建的文件
|
||
|
||
- `runner.py`、`report.py`（明天和接口一起测）
|
||
|
||
### 完成标准
|
||
|
||
- 能画出“提交 → 领取 → 运行 → 续租 → 结束 / 中断 / 失败 → 释放”的流程；能讲清看门狗什么时候接手。
|
||
|
||
---
|
||
|
||
## 第 5 天：FastAPI 接口和 SSE
|
||
|
||
### 学习内容
|
||
|
||
- 说明：
|
||
- `lifespan` 里依次打开：业务库、检查点库、记忆库，构建图，启动看门狗；关闭时反向清理（`AsyncExitStack` 管理多个异步上下文）。
|
||
- SSE：先从 `events` 表补发 `seq > Last-Event-ID` 的事件，再等新事件（进程内通知 + 超时回查数据库兜底）；任务到终态后发 `end` 并结束；15 秒无事件发心跳注释，防止代理断开空闲连接。**客户端断开只结束这条连接，不影响任务**。
|
||
- 接口：
|
||
|
||
| 方法 | 路径 | 说明 |
|
||
| --- | --- | --- |
|
||
| POST | `/runs` | 创建任务，立即返回 `{id, status: "queued"}` |
|
||
| GET | `/runs`、`/runs/{id}` | 列表 / 详情 |
|
||
| GET | `/runs/{id}/events` | SSE，支持 `Last-Event-ID` |
|
||
| POST | `/runs/{id}/input` | 回答问题（仅 `awaiting_input`） |
|
||
| POST | `/runs/{id}/retry` | 失败后重试，可换模型 |
|
||
| POST | `/runs/{id}/cancel` | 取消 |
|
||
| GET | `/runs/{id}/report` | Markdown 报告 |
|
||
| GET / DELETE | `/memories`、`/memories/{key}` | 查看 / 删除长期记忆 |
|
||
|
||
- 案例：
|
||
|
||
```python
|
||
# 文件：agent-mini/src/agent_mini/api/__init__.py
|
||
```
|
||
|
||
```python
|
||
# 文件：agent-mini/src/agent_mini/api/app.py
|
||
import asyncio
|
||
import json
|
||
import os
|
||
from contextlib import AsyncExitStack, asynccontextmanager
|
||
from pathlib import Path
|
||
|
||
from fastapi import FastAPI, HTTPException, Request
|
||
from fastapi.responses import FileResponse, PlainTextResponse, StreamingResponse
|
||
from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver
|
||
from langgraph.store.sqlite.aio import AsyncSqliteStore
|
||
from pydantic import BaseModel, Field
|
||
|
||
from agent_mini.demo_model import demo_model
|
||
from agent_mini.graph.build import build_graph
|
||
from agent_mini.report import build_report
|
||
from agent_mini.runner import RunManager, is_terminal
|
||
from agent_mini.store.run_store import RunStore
|
||
|
||
|
||
class RunIn(BaseModel):
|
||
question: str = Field(min_length=1, max_length=1000)
|
||
max_rounds: int = Field(default=3, ge=1, le=8)
|
||
model: str | None = None
|
||
|
||
|
||
class InputIn(BaseModel):
|
||
answer: str = Field(min_length=1, max_length=1000)
|
||
|
||
|
||
class RetryIn(BaseModel):
|
||
model: str | None = None
|
||
|
||
|
||
def make_model(name: str):
|
||
"""fake / fake-broken / fake-slow 是演示模型；其他名字走真实模型。"""
|
||
delay = float(os.getenv("FAKE_DELAY", "0"))
|
||
if name == "fake":
|
||
return demo_model(delay=delay)
|
||
if name == "fake-broken":
|
||
return demo_model(delay=delay, fail_role="critic")
|
||
from agent_mini.llm import build_chat_model
|
||
|
||
return build_chat_model(name)
|
||
|
||
|
||
def create_app(data_dir: str | None = None, default_model: str | None = None, watchdog_interval: float = 5) -> FastAPI:
|
||
data = Path(data_dir or os.getenv("AGENT_DATA", "data"))
|
||
model_name = default_model or os.getenv("AGENT_MODEL", "fake")
|
||
|
||
@asynccontextmanager
|
||
async def lifespan(app: FastAPI):
|
||
data.mkdir(parents=True, exist_ok=True)
|
||
async with AsyncExitStack() as stack:
|
||
checkpointer = await stack.enter_async_context(AsyncSqliteSaver.from_conn_string(str(data / "checkpoints.db")))
|
||
mem_store = await stack.enter_async_context(AsyncSqliteStore.from_conn_string(str(data / "memory.db")))
|
||
await mem_store.setup()
|
||
graphs: dict[str, object] = {}
|
||
|
||
def graph_for(name: str):
|
||
if name not in graphs: # 每个模型一张图，共享同一个检查点库和记忆库
|
||
graphs[name] = build_graph(make_model(name), checkpointer=checkpointer, store=mem_store)
|
||
return graphs[name]
|
||
|
||
app.state.runs = RunStore(str(data / "runs.db"))
|
||
app.state.mgr = RunManager(app.state.runs, graph_for)
|
||
app.state.graph_for, app.state.mem = graph_for, mem_store
|
||
dog = asyncio.create_task(app.state.mgr.watchdog(watchdog_interval)) # 启动时也会接手未完成的任务
|
||
yield
|
||
dog.cancel()
|
||
await app.state.mgr.shutdown()
|
||
|
||
app = FastAPI(title="agent-mini", lifespan=lifespan)
|
||
|
||
def get_run(request: Request, run_id: str) -> dict:
|
||
run = request.app.state.runs.get_run(run_id)
|
||
if run is None:
|
||
raise HTTPException(404, "任务不存在")
|
||
return run
|
||
|
||
@app.post("/runs", status_code=201)
|
||
async def create_run(body: RunIn, request: Request):
|
||
run = request.app.state.runs.create_run(body.question, model=body.model or model_name, max_rounds=body.max_rounds)
|
||
request.app.state.mgr.submit(run["id"])
|
||
return {"id": run["id"], "status": run["status"]}
|
||
|
||
@app.get("/runs")
|
||
def list_runs(request: Request):
|
||
return request.app.state.runs.list_runs("demo")
|
||
|
||
@app.get("/runs/{run_id}")
|
||
def run_detail(run_id: str, request: Request):
|
||
return get_run(request, run_id)
|
||
|
||
@app.post("/runs/{run_id}/input")
|
||
async def give_input(run_id: str, body: InputIn, request: Request):
|
||
if get_run(request, run_id)["status"] != "awaiting_input":
|
||
raise HTTPException(409, "当前不在等待输入状态")
|
||
request.app.state.runs.update(run_id, status="queued")
|
||
request.app.state.mgr.emit(run_id, "input_received", {"answer": body.answer})
|
||
request.app.state.mgr.submit(run_id, resume=body.answer)
|
||
return {"status": "queued"}
|
||
|
||
@app.post("/runs/{run_id}/retry")
|
||
async def retry(run_id: str, body: RetryIn, request: Request):
|
||
if get_run(request, run_id)["status"] != "failed":
|
||
raise HTTPException(409, "只有失败的任务可以重试")
|
||
fields = {"status": "queued"} | ({"model": body.model} if body.model else {})
|
||
request.app.state.runs.update(run_id, **fields)
|
||
request.app.state.mgr.submit(run_id)
|
||
return {"status": "queued"}
|
||
|
||
@app.post("/runs/{run_id}/cancel")
|
||
async def cancel(run_id: str, request: Request):
|
||
if is_terminal(get_run(request, run_id)["status"]):
|
||
raise HTTPException(409, "任务已结束")
|
||
await request.app.state.mgr.cancel(run_id)
|
||
return {"status": "cancelled"}
|
||
|
||
@app.get("/runs/{run_id}/events")
|
||
async def events(run_id: str, request: Request):
|
||
get_run(request, run_id)
|
||
last = int(request.headers.get("last-event-id") or request.query_params.get("after") or 0)
|
||
runs, mgr = request.app.state.runs, request.app.state.mgr
|
||
|
||
async def stream():
|
||
nonlocal last
|
||
idle = 0.0
|
||
while True:
|
||
for e in runs.list_events(run_id, last):
|
||
last = e["seq"]
|
||
idle = 0.0
|
||
yield f"id: {e['seq']}\nevent: {e['type']}\ndata: {json.dumps(e['data'], ensure_ascii=False)}\n\n"
|
||
if is_terminal(runs.get_run(run_id)["status"]) and not runs.list_events(run_id, last):
|
||
yield "event: end\ndata: {}\n\n"
|
||
return
|
||
if await request.is_disconnected():
|
||
return # 只结束这条连接，任务继续在后台跑
|
||
await mgr.wait_event(run_id, timeout=1.0)
|
||
idle += 1.0
|
||
if idle >= 15:
|
||
idle = 0.0
|
||
yield ": ping\n\n" # 心跳
|
||
|
||
return StreamingResponse(stream(), media_type="text/event-stream",
|
||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"})
|
||
|
||
@app.get("/runs/{run_id}/report", response_class=PlainTextResponse)
|
||
async def report(run_id: str, request: Request):
|
||
run = get_run(request, run_id)
|
||
snap = await request.app.state.graph_for(run["model"]).aget_state({"configurable": {"thread_id": run_id}})
|
||
return build_report(run, snap.values or {}, request.app.state.runs.list_events(run_id))
|
||
|
||
@app.get("/memories")
|
||
async def memories(request: Request, user_id: str = "demo"):
|
||
items = await request.app.state.mem.asearch((user_id, "memories"), limit=50)
|
||
return [{"key": i.key, "text": i.value["text"]} for i in items]
|
||
|
||
@app.delete("/memories/{key}", status_code=204)
|
||
async def delete_memory(key: str, request: Request, user_id: str = "demo"):
|
||
await request.app.state.mem.adelete((user_id, "memories"), key)
|
||
|
||
@app.get("/")
|
||
def index():
|
||
return FileResponse(Path(__file__).resolve().parents[3] / "web" / "index.html")
|
||
|
||
return app
|
||
```
|
||
|
||
```python
|
||
# 文件：agent-mini/tests/test_api.py
|
||
from fastapi.testclient import TestClient
|
||
|
||
from agent_mini.api.app import create_app
|
||
|
||
|
||
def read_until_end(client: TestClient, run_id: str, last_id: int | None = None) -> list[tuple[str, str]]:
|
||
headers = {"Last-Event-ID": str(last_id)} if last_id else {}
|
||
events, ev_id = [], ""
|
||
with client.stream("GET", f"/runs/{run_id}/events", headers=headers) as s:
|
||
for line in s.iter_lines():
|
||
if line.startswith("id: "):
|
||
ev_id = line[4:]
|
||
elif line.startswith("event: "):
|
||
events.append((line[7:], ev_id))
|
||
if line == "event: end" or line == "event: awaiting_input":
|
||
break
|
||
return events
|
||
|
||
|
||
def test_run_to_completion_and_report(tmp_path):
|
||
with TestClient(create_app(str(tmp_path), "fake")) as client:
|
||
run_id = client.post("/runs", json={"question": "5 人团队选技术栈"}).json()["id"]
|
||
events = read_until_end(client, run_id)
|
||
types = [t for t, _ in events]
|
||
assert types[0] == "run_created" and "run_completed" in types and types[-1] == "end"
|
||
assert client.get(f"/runs/{run_id}").json()["end_reason"] == "consensus"
|
||
|
||
# 断线重连：只补发 Last-Event-ID 之后的事件
|
||
third_id = int(events[2][1])
|
||
replay = read_until_end(client, run_id, last_id=third_id)
|
||
assert all(int(i) > third_id for t, i in replay if t != "end")
|
||
|
||
report = client.get(f"/runs/{run_id}/report").text
|
||
assert "## 过程" in report and "方案 v2" in report
|
||
|
||
|
||
def test_ask_human_flow_and_memory(tmp_path):
|
||
with TestClient(create_app(str(tmp_path), "fake")) as client:
|
||
run_id = client.post("/runs", json={"question": "[需要澄清] 做个网站"}).json()["id"]
|
||
first = read_until_end(client, run_id)
|
||
assert first[-1][0] == "awaiting_input"
|
||
assert client.get(f"/runs/{run_id}").json()["human_question"] == "预算大概多少？"
|
||
assert client.post(f"/runs/{run_id}/input", json={"answer": "5 万以内"}).status_code == 200
|
||
assert read_until_end(client, run_id, last_id=int(first[-1][1]))[-1][0] == "end" # 从上次看到的位置接着读
|
||
assert client.get(f"/runs/{run_id}").json()["status"] == "completed"
|
||
assert len(client.get("/memories").json()) == 1 # 偏好被记住
|
||
|
||
|
||
def test_fail_then_retry_with_another_model(tmp_path):
|
||
with TestClient(create_app(str(tmp_path), "fake-broken")) as client:
|
||
run_id = client.post("/runs", json={"question": "q"}).json()["id"]
|
||
read_until_end(client, run_id)
|
||
run = client.get(f"/runs/{run_id}").json()
|
||
assert (run["status"], run["error_code"]) == ("failed", "QUOTA_EXCEEDED")
|
||
assert client.post(f"/runs/{run_id}/retry", json={"model": "fake"}).status_code == 200
|
||
read_until_end(client, run_id)
|
||
assert client.get(f"/runs/{run_id}").json()["status"] == "completed"
|
||
|
||
|
||
def test_not_found(tmp_path):
|
||
with TestClient(create_app(str(tmp_path))) as client:
|
||
assert client.get("/runs/nope/events").status_code == 404
|
||
```
|
||
|
||
**易错点 / 面试常问（补充）**
|
||
|
||
- 易错点：SSE 事件要先写进数据库（事件表）再推送，事件带递增的序号；前端断线重连时带上 `Last-Event-ID`，服务端从这个序号之后补发。
|
||
- 易错点：读事件流时要从“上次看到的序号”继续，不然会从头重放，误把旧的 `awaiting_input` 事件当成当前状态（本周测试里就踩过这个坑）。
|
||
- 面试常问：为什么事件要落库（断线续传、多实例部署时任意实例都能推送、事后可审计）；SSE 连接很多时服务端怎么扛（异步 I/O、心跳、连接数限制）。
|
||
|
||
### 要建的文件
|
||
|
||
- `api/__init__.py`、`api/app.py`、`tests/test_api.py`
|
||
|
||
### 运行方式
|
||
|
||
```bash
|
||
uv run pytest -v
|
||
AGENT_MODEL=fake FAKE_DELAY=0.5 uv run uvicorn agent_mini.api.app:create_app --factory --port 8100
|
||
curl -s -X POST localhost:8100/runs -H 'Content-Type: application/json' -d '{"question":"选技术栈"}'
|
||
curl -N localhost:8100/runs/<id>/events
|
||
```
|
||
|
||
### 完成标准
|
||
|
||
- 全部测试通过（不需要密钥）；`curl -N` 能看到事件逐条出现，`Ctrl+C` 后任务照跑，再连能补发。
|
||
|
||
---
|
||
|
||
## 第 6 天：页面 + 崩溃恢复演示
|
||
|
||
### 学习内容
|
||
|
||
- 说明：一个原生 JS 页面（第 7 周的 Vue / React 版本可选替换），以及一个**可重复的崩溃恢复演示脚本**——面试时最有说服力的就是“我现在杀掉进程，你看它接着跑”。
|
||
- 案例：
|
||
|
||
```html
|
||
<!-- web/index.html：极简页面，原生 JS -->
|
||
<!doctype html>
|
||
<html lang="zh-CN">
|
||
<head><meta charset="UTF-8" /><title>agent-mini</title>
|
||
<style>body{font-family:sans-serif;max-width:860px;margin:24px auto} li{margin:6px 0} .muted{color:#888}</style>
|
||
</head>
|
||
<body>
|
||
<h2>多 Agent 讨论</h2>
|
||
<form id="f"><input id="q" size="60" placeholder="问题；带 [需要澄清] 可以演示问人" /> <button>开始</button></form>
|
||
<p id="status" class="muted"></p>
|
||
<div id="ask" hidden><span id="askq"></span> <input id="ans" /> <button id="send">回答</button></div>
|
||
<ol id="timeline"></ol>
|
||
<script>
|
||
const $ = (id) => document.getElementById(id);
|
||
let runId = null, es = null;
|
||
function add(text) { const li = document.createElement("li"); li.textContent = text; $("timeline").append(li); }
|
||
function listen() {
|
||
es?.close();
|
||
es = new EventSource(`/runs/${runId}/events`); // 断线后浏览器自动重连并带上 Last-Event-ID
|
||
for (const type of ["run_created", "run_started", "node_finished", "awaiting_input", "input_received",
|
||
"run_completed", "run_failed", "run_cancelled"]) {
|
||
es.addEventListener(type, (e) => {
|
||
const d = JSON.parse(e.data);
|
||
$("status").textContent = `#${e.lastEventId} ${type}`;
|
||
if (type === "node_finished") add(`${d.node}${d.role ? "（" + d.role + "）" : ""}：${d.content ?? d.decision?.reason ?? ""}`);
|
||
if (type === "awaiting_input") { $("askq").textContent = d.question; $("ask").hidden = false; }
|
||
if (type === "run_completed") add(`结论：${d.final_answer}`);
|
||
if (type === "run_failed") add(`失败：${d.error_code}`);
|
||
});
|
||
}
|
||
es.addEventListener("end", () => es.close());
|
||
}
|
||
$("f").onsubmit = async (e) => {
|
||
e.preventDefault(); $("timeline").innerHTML = "";
|
||
const r = await fetch("/runs", { method: "POST", headers: { "Content-Type": "application/json" },
|
||
body: JSON.stringify({ question: $("q").value }) });
|
||
runId = (await r.json()).id; history.replaceState(null, "", `?run=${runId}`); listen();
|
||
};
|
||
$("send").onclick = async () => {
|
||
await fetch(`/runs/${runId}/input`, { method: "POST", headers: { "Content-Type": "application/json" },
|
||
body: JSON.stringify({ answer: $("ans").value }) });
|
||
$("ask").hidden = true;
|
||
};
|
||
runId = new URLSearchParams(location.search).get("run"); if (runId) listen(); // 刷新页面后继续看
|
||
</script>
|
||
</body>
|
||
</html>
|
||
```
|
||
|
||
```bash
|
||
#!/usr/bin/env bash
|
||
# scripts/demo_crash_resume.sh：启动 → 创建任务 → kill -9 → 重启 → 等完成 → 打印报告
|
||
set -euo pipefail
|
||
export AGENT_MODEL=fake FAKE_DELAY=0.6 AGENT_DATA=data/demo
|
||
rm -rf "$AGENT_DATA"
|
||
start() { uv run uvicorn agent_mini.api.app:create_app --factory --port 8100 >/tmp/agent-mini.log 2>&1 & echo $!; }
|
||
wait_up() { until curl -sf localhost:8100/runs >/dev/null; do sleep 0.3; done; }
|
||
|
||
PID=$(start); wait_up
|
||
RUN=$(curl -s -X POST localhost:8100/runs -H 'Content-Type: application/json' \
|
||
-d '{"question":"5 人团队做内部工具选什么技术栈","max_rounds":4}' | python3 -c 'import sys,json;print(json.load(sys.stdin)["id"])')
|
||
echo "任务 $RUN 已创建，3 秒后杀进程"; sleep 3
|
||
curl -s "localhost:8100/runs/$RUN" | python3 -c 'import sys,json;print("杀之前状态：", json.load(sys.stdin)["status"])'
|
||
kill -9 "$PID"; echo "已 kill -9"
|
||
|
||
PID=$(start); wait_up; echo "已重启，等待看门狗接手（租约过期后）"
|
||
until [ "$(curl -s "localhost:8100/runs/$RUN" | python3 -c 'import sys,json;print(json.load(sys.stdin)["status"])')" = completed ]; do sleep 1; done
|
||
curl -s "localhost:8100/runs/$RUN/report"
|
||
kill "$PID"
|
||
```
|
||
|
||
### 要建的文件
|
||
|
||
- `web/index.html`、`scripts/demo_crash_resume.sh`（`chmod +x`）
|
||
|
||
### 运行方式
|
||
|
||
```bash
|
||
./scripts/demo_crash_resume.sh
|
||
```
|
||
|
||
### 完成标准
|
||
|
||
- 脚本跑通：报告里能看到事件序列中有两次 `run_started`（两个不同的 worker），并且最终完成。
|
||
- 注意：租约是 30 秒，重启后要等租约过期看门狗才会接手；演示时可以把租约调短（`RunManager(lease_s=...)`），在 README 里说明这是“恢复速度”和“误判崩溃”之间的取舍。
|
||
|
||
---
|
||
|
||
## 第 7 天：走查 + M3 验收
|
||
|
||
### 学习内容
|
||
|
||
- 说明：手动走查 6 个场景，每个记录“预期 / 实际”：
|
||
1. 正常跑完；
|
||
2. 中途关页再打开（`?run=` 链接），事件补齐；
|
||
3. 中途停服务再启动，任务继续；
|
||
4. 触发问人并回答；
|
||
5. 用 `fake-broken` 制造失败，换 `fake` 重试成功；
|
||
6. 第二次提问用上了记忆，在 `/memories` 里删除后不再出现。
|
||
|
||
### 要建的文件
|
||
|
||
- `docs/walkthrough.md`（6 个场景的记录）；`docs/design-decisions.md` 补充：两层状态、租约、至少一次执行与幂等、SQLite 只用于演示、记忆规则、没做的事（鉴权、多实例实时通知、PostgreSQL 版本）
|
||
|
||
### 运行方式
|
||
|
||
```bash
|
||
uv run ruff check . && uv run pyright src && uv run pytest -v
|
||
```
|
||
|
||
### 完成标准（M3 验收）
|
||
|
||
- [ ] 测试全部通过：续跑、中断恢复、换模型重试、租约、记忆、SSE 补发
|
||
- [ ] 崩溃恢复演示脚本成功
|
||
- [ ] 6 个走查场景符合预期
|
||
- [ ] 能讲清：两层状态、为什么编排不能放在浏览器里、什么时候停下来问人、怎么防双跑、长期记忆写什么不写什么
|
||
- [ ] 在总纲进度表勾掉第 12 周
|