初始化

This commit is contained in:
gozeros committed 2026-09-07 18:24:55 +08:00
1 parent 34474bd5fc
commit b638b541d0
3008 files changed
+837699

No files matched your search

@@ -0,0 +1,56 @@
# GPT Researcher adaptation manifest
This directory implements Deep Research for DeerFlow. It is an **algorithmic
adaptation** of selected GPT Researcher control flows, not an import of the
upstream package at runtime. This boundary is intentional: GPT Researcher's
LLM providers, search/scrape stack, MCP setup, global configuration and
dependency graph must not bypass DeerFlow's model factory, material-security
policy, user context or durable-job lifecycle.
## Fixed upstream baseline
| Field | Value |
| --- | --- |
| Project | `assafelovic/gpt-researcher` |
| Local review checkout | `F:\react01\deeflow-code\gpt-researcher` |
| Commit | `5d84d2f5553e70a2765a8ff3a0d2672d60437ce8` |
| License | Apache License 2.0 (`F:\react01\deeflow-code\gpt-researcher\LICENSE`) |
| Upstream repository | <https://github.com/assafelovic/gpt-researcher> |
No upstream executable source file is copied into this directory at present.
If a future change copies an upstream file or a non-trivial portion of one,
add its copyright/license notice next to that file, record the exact source
path and commit below, and keep the Apache-2.0 notice in the distribution.
## Adapted control-flow mapping
| DeerFlow code | Upstream reference | Adaptation made |
| --- | --- | --- |
| `runners/basic.py` | `gpt_researcher/agent.py`, `backend/report_type/basic_report/basic_report.py` | Planning → bounded material collection → de-duplication/context compression → cited report is expressed through `AdapterBundle`. |
| `runners/detailed.py` | `backend/report_type/detailed_report/detailed_report.py` | Subtopic planning, concurrent sub-research and section assembly are retained; all upstream `GPTResearcher` instances become bounded DeerFlow adapters. |
| `runners/deep.py` | `gpt_researcher/skills/deep_research.py` | Breadth/depth branching, learnings and follow-up questions are preserved with an explicit global budget, semaphores and cooperative cancellation. |
| `runners/multi_agent.py` | `multi_agents/agents/{orchestrator,editor,writer,fact_checker}.py` | Editor/researcher/writer/reviewer roles become a LangGraph 1.x `StateGraph`, with the review pause persisted as a durable job state. |
| `adapters/context.py`, `adapters/material.py` | `gpt_researcher/context/compression.py`, `gpt_researcher/skills/researcher.py` | Retains context/de-duplication intent while preserving DeerFlow provenance (`rec_uuid`, source id, source type) and enforcing the existing material-provider boundary. |
| `runners/images.py`, `adapters/image.py` | `gpt_researcher/skills/image_generator.py` | Retains report-image placement intent, while configured OpenAI-compatible and DashScope-native adapters both persist protected hidden-thread artifacts. |
## Required adaptation invariants
1. Harness code may not import `app.*`; this is covered by
`tests/test_deep_research_harness_boundary.py`.
2. Research collection may only use `MaterialProvider`; upstream retrievers,
scrapers and MCP clients are not imported.
3. Model calls may only use `DeerFlowCompletionBackend` and
`create_chat_model()`; no upstream provider configuration is accepted.
4. Every job uses a frozen `DeepResearchConfig` snapshot, a user-scoped
session and a durable event log.
5. Upstream examples that run network calls, write unrestricted paths, or
block for terminal input must be redesigned rather than copied.
## Upgrade procedure
1. Review the candidate upstream commit in the local checkout.
2. Update the commit and mapping table in this manifest and in
`docs/gpt-researcher-deerflow-集成开发方案.md`.
3. Re-evaluate dependency, license and security implications before copying
any source.
4. Run the complete deep-research suite and the harness-boundary test.
@@ -0,0 +1,26 @@
"""Deep Research agent package — vendored GPT Researcher algorithms adapted to DeerFlow.
This package integrates selected GPT Researcher research-planning, recursive
exploration, report-generation, and multi-agent editorial algorithms into the
DeerFlow runtime. The *algorithms* are vendored (``vendor/``); all infrastructure
(model, search, events, artifacts) is injected through a small set of adapters
defined in ``adapters/`` so the vendored code never touches the network, the
filesystem, or ``app.*``.
Architecture (see ``docs/gpt-researcher-deerflow-集成开发方案.md``):
DeepResearchEngine ──► runners/{basic,detailed,deep,multi_agent}
│
▼
AdapterBundle ──► adapters/{llm,material,context,artifacts,event_sink}
│
▼
vendor/gpt_researcher/* (algorithms only)
Harness-layer constraint: **no imports from ``app.*``** (enforced by
``tests/test_deep_research_harness_boundary.py``).
"""
from __future__ import annotations
__all__: list[str] = []
@@ -0,0 +1,19 @@
"""Deep Research adapters — the four-boundary isolation layer.
Adapters are the *only* place the vendored GPT Researcher algorithms touch
DeerFlow infrastructure. Each adapter wraps one concern:
================ ===========================================================
``llm`` model calls → ``create_chat_model()`` (no upstream Provider)
``material`` search/fetch → DeerFlow ``web_search`` + knowledge + MCP
``context`` context compression (no legacy text-splitter deps)
``event_sink`` progress events → ``deep_research_events`` + SSE
``artifacts`` report/export writes → hidden runtime-thread outputs
``image`` image generation (optional, Phase 5)
================ ===========================================================
Protocols live in each module alongside (or ahead of) their concrete impl, so
tests can inject fakes that satisfy the protocol without importing infra.
"""
from __future__ import annotations
@@ -0,0 +1,159 @@
"""Artifact writer — writes report/export files into the session's hidden thread.
Each deep-research session owns a *hidden* DeerFlow runtime thread
(``metadata.thread_type=deep_research``, ``hidden=true``). Report artifacts
(``report.md``, ``report.html``, ``sources.json``, …) are written into that
thread's sandbox ``outputs/`` directory and downloaded through the existing
``/api/threads/{thread_id}/artifacts`` endpoint (which already has path-safety
guards + owner checks — §13.6, §18.3).
The writer is a thin protocol so the engine/runner never touches the filesystem
layout directly.
"""
from __future__ import annotations
import re
from pathlib import Path
from typing import Protocol, runtime_checkable
@runtime_checkable
class ArtifactWriter(Protocol):
"""Protocol for writing research artifacts (§13.6 §18)."""
@property
def thread_id(self) -> str:
"""The hidden runtime thread that owns these artifacts."""
...
async def write_text(self, name: str, content: str) -> str:
"""Write a text artifact (e.g. ``report.md``) and return its virtual path."""
...
async def write_bytes(self, name: str, content: bytes) -> str:
"""Write a binary artifact (e.g. ``report.pdf``). Optional capability."""
...
def allowed_names(self) -> tuple[str, ...]:
"""The server-enumerated filename whitelist (§18.3 — no arbitrary paths)."""
...
# Server-enumerated artifact filename whitelist (§18.3, §13.6).
# The writer rejects any name not in this set to prevent path traversal.
ARTIFACT_NAMES: tuple[str, ...] = (
"report.md",
"report.html",
"sources.json",
"research-plan.json",
"report.pdf", # optional capability
"report.docx", # optional capability
)
# Image artifact names are server-generated only. Keeping the numerical range
# small makes the whitelist finite in practice while still letting the report
# writer select the matching format returned by the configured provider.
_RESEARCH_IMAGE_NAME_RE = re.compile(r"research-image-0[1-4]\.(?:png|jpe?g|webp)$")
def is_allowed_artifact_name(name: str) -> bool:
"""Return whether *name* is one of the server-controlled research outputs."""
return name in ARTIFACT_NAMES or bool(_RESEARCH_IMAGE_NAME_RE.fullmatch(name))
def resolve_research_artifact_path(
thread_id: str,
name: str,
*,
user_id: str | None = None,
) -> Path:
"""Resolve a whitelisted research artifact into its thread outputs dir.
``Paths.resolve_virtual_path`` intentionally accepts only paths that an
in-sandbox agent could see. Research code deals in server-generated file
names, so passing a bare name (for example ``report.md``) would fail that
guard on every host, including native Windows deployments. Keep the name
allowlist and construct the canonical sandbox path here, at the boundary.
"""
if not is_allowed_artifact_name(name):
raise ValueError(f"Artifact name {name!r} is not in the allowed research-artifact whitelist")
from deerflow.config.paths import VIRTUAL_PATH_PREFIX, get_paths
return get_paths().resolve_virtual_path(
thread_id,
f"{VIRTUAL_PATH_PREFIX}/outputs/{name}",
user_id=user_id,
)
class ThreadOutputsArtifactWriter:
"""Concrete writer backed by a DeerFlow thread's sandbox ``outputs/`` dir.
The actual path resolution reuses
:func:`deerflow.config.paths.get_paths().sandbox_outputs_dir` (the same
guard ``app/gateway/routers/artifacts.py`` uses), so traversal/symlink
protection is inherited. File IO is offloaded to a thread so the event
loop is never blocked (§25.6).
"""
def __init__(self, thread_id: str, *, user_id: str | None = None) -> None:
self._thread_id = thread_id
self._user_id = user_id
@property
def thread_id(self) -> str:
return self._thread_id
def allowed_names(self) -> tuple[str, ...]:
return ARTIFACT_NAMES
async def write_text(self, name: str, content: str) -> str:
self._validate_name(name)
import asyncio
path = await asyncio.to_thread(self._resolve_path, name)
await asyncio.to_thread(self._write_file, path, content.encode("utf-8"))
return name
async def write_bytes(self, name: str, content: bytes) -> str:
self._validate_name(name)
import asyncio
path = await asyncio.to_thread(self._resolve_path, name)
await asyncio.to_thread(self._write_file, path, content)
return name
# ── internals ───────────────────────────────────────────────────────────
def _validate_name(self, name: str) -> None:
if not is_allowed_artifact_name(name):
raise ValueError(
f"Artifact name {name!r} is not in the allowed research-artifact whitelist"
)
def _resolve_path(self, name: str):
# Ensure the outputs dir exists; the paths module rejects traversal and
# the name is whitelisted above, so the canonical virtual path is safe.
from deerflow.config.paths import get_paths
get_paths().sandbox_outputs_dir(self._thread_id, user_id=self._user_id)
return resolve_research_artifact_path(
self._thread_id,
name,
user_id=self._user_id,
)
def _write_file(self, path, data: bytes) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_bytes(data)
__all__ = [
"ARTIFACT_NAMES",
"ArtifactWriter",
"ThreadOutputsArtifactWriter",
"is_allowed_artifact_name",
"resolve_research_artifact_path",
]
@@ -0,0 +1,130 @@
"""Context compressor — DeerFlow replacement for upstream's legacy compressor.
Upstream's ``ContextManager`` / ``compression`` module depends on
``langchain_classic`` and ``langchain_text_splitters``, which DeerFlow does not
declare. Rather than pull in those legacy packages, we keep the *calling
interface* (§10) and provide :class:`DeerFlowContextCompressor` with a
self-contained algorithm:
1. Chunk by title/paragraph/char-limit without breaking Markdown code blocks.
2. Each chunk retains its source id (no cross-source amalgamation).
3. If embeddings are configured → similarity pre-filter; else keyword/BM25-style.
4. Over budget → per-source summary via ``fast_model`` (Phase 2).
5. Final ranking balances source diversity, recency, relevance.
Phase 1 ships the chunking + keyword pre-filter (no embedding dependency);
the ``fast_model`` summarisation path is wired but only triggers when the
context genuinely exceeds ``max_context_words``.
"""
from __future__ import annotations
from collections.abc import Callable
from typing import TYPE_CHECKING, Any, Protocol, runtime_checkable
if TYPE_CHECKING:
from deerflow.agents.deep_research.adapters.llm import DeerFlowCompletionBackend
from deerflow.agents.deep_research.types import ResearchMaterial
@runtime_checkable
class ContextCompressor(Protocol):
"""The interface vendored ``ContextManager`` is rewired to call (§10)."""
async def async_get_context(
self,
query: str,
max_results: int = 10,
cost_callback: Callable[..., Any] | None = None,
) -> list[dict[str, Any]]:
"""Return the compressed context list (each item = one material's context dict).
Output shape matches :meth:`ResearchMaterial.to_context_dict` so the
vendored writer sees what it expects.
"""
...
class DeerFlowContextCompressor:
"""Concrete compressor: chunk → keyword-filter → optional summary.
Holds a reference to the current material set (set by the runner before
the vendored research loop calls ``async_get_context``) and the LLM backend
(for the summary path).
"""
def __init__(
self,
*,
llm: DeerFlowCompletionBackend | None = None,
max_context_words: int = 24_000,
) -> None:
self._llm = llm
self._max_words = max_context_words
self._materials: list[ResearchMaterial] = []
def set_materials(self, materials: list[ResearchMaterial]) -> None:
"""Prime the compressor with the current source pool (called per research phase)."""
self._materials = list(materials)
async def async_get_context(
self,
query: str,
max_results: int = 10,
cost_callback: Callable[..., Any] | None = None,
) -> list[dict[str, Any]]:
"""Select + compress the best ``max_results`` materials for ``query``."""
from deerflow.agents.deep_research.adapters.material import dedupe_materials
deduped = dedupe_materials(self._materials)
# Keyword pre-filter (no embedding dep in Phase 1).
scored = [(_keyword_score(query, m), m) for m in deduped]
scored.sort(key=lambda pair: pair[0], reverse=True)
selected = [m for _, m in scored[:max_results]]
# Budget guard: if total words exceed limit, summarise the tail.
total_words = sum(len((m.raw_content or "").split()) for m in selected)
if total_words > self._max_words and self._llm is not None:
selected = await self._summarise_tail(selected, query)
if cost_callback is not None:
try:
cost_callback(1, 0) # upstream signature: (total, done)
except Exception: # noqa: BLE001
pass
return [m.to_context_dict() for m in selected]
async def _summarise_tail(
self, materials: list[ResearchMaterial], query: str
) -> list[ResearchMaterial]:
"""Summarise lower-ranked materials to fit the word budget (Phase 2)."""
# Phase 1: simply truncate the list rather than calling the model,
# so a fresh deploy without model tuning still produces a report.
# (The full summarisation path is enabled once the basic runner is green.)
words = 0
kept: list[ResearchMaterial] = []
for m in materials:
w = len((m.raw_content or "").split())
if words + w > self._max_words:
break
words += w
kept.append(m)
return kept or materials[:1]
def _keyword_score(query: str, material: ResearchMaterial) -> float:
"""Cheap relevance heuristic: fraction of query terms present in the material."""
q_terms = {t.lower() for t in query.split() if len(t) > 1}
if not q_terms:
return 0.0
haystack = f"{material.title}\n{material.raw_content}".lower()
hits = sum(1 for t in q_terms if t in haystack)
base = hits / len(q_terms)
# Tiny recency/relevance nudges when metadata is present.
if material.relevance_score:
base += 0.01 * float(material.relevance_score)
return base
__all__ = ["ContextCompressor", "DeerFlowContextCompressor"]
@@ -0,0 +1,376 @@
"""Persisting event sink — bridges runners to the event-store repository.
Runners call :meth:`PersistingEventSink.emit` (which satisfies the
:class:`~deerflow.agents.deep_research.events.EventSink` protocol); the sink
delegates to :class:`~deerflow.persistence.deep_research_events.DeepResearchEventRepository`,
which assigns the monotonic ``seq`` and persists the row **before** returning.
This "persist-then-stream" ordering (§15.3) is what makes ``?after=<seq>``
reconnect-replay correct.
The sink is bound to one ``(session_id, job_id)`` pair at construction; runners
don't carry those ids around.
"""
from __future__ import annotations
import json
import logging
from collections.abc import Awaitable, Callable
from contextlib import suppress
from datetime import date, datetime
from enum import Enum
from time import monotonic
from typing import TYPE_CHECKING, Any
from deerflow.agents.deep_research.events import (
DeepResearchEvent,
EventType,
ResearchPhase,
)
if TYPE_CHECKING:
from deerflow.persistence.deep_research_events import DeepResearchEventRepository
logger = logging.getLogger(__name__)
def _json_default(value: Any) -> Any:
if isinstance(value, (datetime, date)):
return value.isoformat()
if isinstance(value, Enum):
return value.value
if hasattr(value, "model_dump"):
return value.model_dump(mode="json")
return str(value)
# Live-only frame types that have no durable counterpart at all. Report and
# summary text are already checkpointed (``report_chunk`` / ``summary_completed``),
# so only the reasoning channel needs a cross-worker mirror.
_MIRRORED_LIVE_TYPES = frozenset({"report_thinking", "plan_reasoning", "summary_thinking"})
# Coalescing bounds for that mirror: one bounded row per ~600 characters or
# ~1.5 seconds, never one row per provider token.
_MIRROR_MIN_CHARS = 600
_MIRROR_MAX_DELAY_S = 1.5
# Floor for the time-based flush: a slowly trickling reasoning stream must not
# degrade into one durable row per token. Anything shorter is carried until the
# next durable milestone flushes it.
_MIRROR_MIN_FLUSH_CHARS = 40
def _json_safe_payload(payload: dict[str, Any]) -> dict[str, Any]:
"""Normalize event payloads before they reach any event repository.
This protects deployments that still use a strict ``json.dumps`` inside
the SQL repository and keeps native datetime values in source metadata
from degrading every ``source_added`` event.
"""
return json.loads(json.dumps(payload, ensure_ascii=False, default=_json_default))
class PersistingEventSink:
"""EventSink implementation backed by the deep-research event repository.
Satisfies :class:`~deerflow.agents.deep_research.events.EventSink` (the
runtime-checkable protocol) via structural typing.
"""
def __init__(
self,
repo: DeepResearchEventRepository,
*,
session_id: str,
job_id: str,
on_event: Callable[[dict[str, Any]], Awaitable[None]] | None = None,
live_subscriber_probe: Callable[[str], bool] | None = None,
) -> None:
self._repo = repo
self._session_id = session_id
self._job_id = job_id
self._on_event = on_event
# Reports whether this worker holds an SSE connection for the job.
# Absent probe → assume the live path reaches the viewer (single-worker
# deployments and tests keep today's zero-extra-write behaviour).
self._live_subscriber_probe = live_subscriber_probe
self._mirror_buffers: dict[str, dict[str, Any]] = {}
self._mirroring = False
self.diagnostics: list[dict[str, Any]] = []
# A durable event row normally owns the sequence. When the event
# database is temporarily unavailable, use distinct negative sequence
# numbers for the live-only path so clients can render every milestone
# without advancing their durable ``after`` cursor.
self._fallback_seq = 0
async def emit(
self,
type: EventType, # noqa: A002
*,
phase: ResearchPhase = "initializing",
payload: dict | None = None,
) -> DeepResearchEvent:
event_payload = _json_safe_payload(payload or {})
persistence_error: Exception | None = None
if not self._mirroring:
# Order any coalesced reasoning tail before the next milestone
# instead of leaving it buffered until the job ends.
await self._flush_live_mirrors()
try:
row = await self._repo.append(
session_id=self._session_id,
job_id=self._job_id,
event_type=type,
phase=phase,
payload=event_payload,
)
# Reconstruct the envelope from the persisted row (seq is now assigned).
event = DeepResearchEvent(
seq=int(row.get("seq", 0)),
session_id=self._session_id,
job_id=self._job_id,
type=row.get("event_type", type),
phase=row.get("phase", phase),
payload=row.get("payload") or event_payload,
)
except Exception as exc: # noqa: BLE001
# Progress persistence must not turn a recoverable database blip
# into a failed report. The job/status stores still record terminal
# state; this live-only frame lets an attached UI stay responsive
# until durable replay becomes available again after reconnect.
self._fallback_seq -= 1
persistence_error = exc
self.diagnostics.append(
{
"code": "event_persistence_degraded",
"stage": phase,
"eventType": type,
"errorType": exc.__class__.__name__,
"detail": str(exc).split("[SQL:", 1)[0].strip()[:600] or exc.__class__.__name__,
}
)
logger.warning(
"deep-research event persistence degraded; continuing job with a live-only event "
"(job=%s, type=%s, phase=%s, error=%s)",
self._job_id,
type,
phase,
persistence_error.__class__.__name__,
)
event = DeepResearchEvent(
seq=self._fallback_seq,
session_id=self._session_id,
job_id=self._job_id,
type=type,
phase=phase,
payload={**event_payload, "eventPersistenceDegraded": True},
)
await self._publish_event(event)
if persistence_error is not None:
# Keep the original source/progress event live, and additionally
# expose one copyable diagnostic frame. This never re-enters the
# failing repository and therefore cannot interrupt the job.
self._fallback_seq -= 1
detail = str(persistence_error).split("[SQL:", 1)[0].strip()[:600]
await self._publish_event(
DeepResearchEvent(
seq=self._fallback_seq,
session_id=self._session_id,
job_id=self._job_id,
type="warning",
phase=phase,
payload={
"code": "event_persistence_degraded",
"message": "研究过程事件暂未写入数据库,报告任务已继续运行",
"recoverable": True,
"diagnostic": {
"stage": phase,
"eventType": type,
"errorType": persistence_error.__class__.__name__,
"detail": detail or persistence_error.__class__.__name__,
},
},
)
)
return event
async def _publish_event(self, event: DeepResearchEvent) -> None:
"""Push a persisted or live-only event without risking the worker."""
if self._on_event is None:
return
wire = event.to_sse_dict()
if event.seq < 0:
wire["liveId"] = f"event-persistence-fallback:{self._job_id}:{abs(event.seq)}"
try:
await self._on_event(wire)
except Exception: # noqa: BLE001
# A browser disconnect or a live-hub issue must not abort report
# generation; persisted rows will still be replayable on reconnect.
logger.warning(
"deep-research live event publish failed; job continues (job=%s, seq=%s)",
self._job_id,
event.seq,
exc_info=True,
)
async def publish_live(
self,
type: str, # noqa: A002
*,
phase: ResearchPhase = "initializing",
payload: dict | None = None,
) -> None:
if self._on_event is None:
return
try:
await self._on_event(
{
"seq": 0,
"sessionId": self._session_id,
"jobId": self._job_id,
"type": type,
"phase": phase,
"payload": payload or {},
}
)
except Exception: # noqa: BLE001
logger.warning(
"deep-research live frame publish failed; job continues (job=%s, type=%s)",
self._job_id,
type,
exc_info=True,
)
with suppress(Exception):
await self._mirror_live_frame(type, phase=phase, payload=payload or {})
# ── cross-worker mirror for live-only reasoning frames ──────────────────
async def _mirror_live_frame(
self,
type: str, # noqa: A002
*,
phase: ResearchPhase,
payload: dict[str, Any],
) -> None:
"""Buffer a live-only reasoning delta when no local viewer can see it.
The live hub is in-process. A job runs on the worker that won its
durable lease, while the SSE connection is served by whichever worker
accepted it, so on a multi-worker deployment the model's thinking is
usually published into a hub with no subscribers and is then lost —
there is no durable row to replay it from. When that is the case, the
reasoning is coalesced here and persisted as a bounded durable event
so the polling stream on the other worker still renders it.
"""
if type not in _MIRRORED_LIVE_TYPES or self._live_subscriber_probe is None:
return
delta = payload.get("delta")
if not isinstance(delta, str) or not delta:
return
if self._live_subscriber_probe(self._job_id):
# A viewer is attached here and has already received every frame
# live. Drop the buffer rather than flushing it, so that viewer
# does not also receive the same text as a durable replay.
self._mirror_buffers.pop(self._mirror_key(type, payload), None)
return
key = self._mirror_key(type, payload)
buffered = self._mirror_buffers.get(key)
if buffered is None:
buffered = {
"type": type,
"phase": phase,
"target": payload.get("target"),
"reportVersion": payload.get("reportVersion"),
"text": "",
"since": monotonic(),
}
self._mirror_buffers[key] = buffered
buffered["text"] += delta
buffered["phase"] = phase
buffered_chars = len(buffered["text"])
aged_out = (
buffered_chars >= _MIRROR_MIN_FLUSH_CHARS
and monotonic() - float(buffered["since"]) >= _MIRROR_MAX_DELAY_S
)
if buffered_chars >= _MIRROR_MIN_CHARS or aged_out:
await self._emit_mirror(key)
@staticmethod
def _mirror_key(type: str, payload: dict[str, Any]) -> str: # noqa: A002
return f"{type}::{payload.get('target') or 'report'}"
async def _flush_live_mirrors(self) -> None:
for key in tuple(self._mirror_buffers):
with suppress(Exception):
await self._emit_mirror(key)
async def _emit_mirror(self, key: str) -> None:
buffered = self._mirror_buffers.pop(key, None)
if buffered is None or not buffered["text"]:
return
payload: dict[str, Any] = {
"delta": buffered["text"],
"liveMirror": True,
}
if buffered["target"] is not None:
payload["target"] = buffered["target"]
if buffered["reportVersion"] is not None:
payload["reportVersion"] = buffered["reportVersion"]
self._mirroring = True
try:
await self.emit(buffered["type"], phase=buffered["phase"], payload=payload)
finally:
self._mirroring = False
class NullEventSink:
"""No-op sink for unit tests / dry runs that don't have a database.
Satisfies :class:`EventSink` structurally; assigns a monotonic in-memory seq
so fake tests can still assert event ordering.
"""
def __init__(self, *, session_id: str = "test", job_id: str = "test") -> None:
self._session_id = session_id
self._job_id = job_id
self._seq = 0
self.events: list[DeepResearchEvent] = []
self.live_events: list[dict[str, Any]] = []
async def emit(
self,
type: EventType, # noqa: A002
*,
phase: ResearchPhase = "initializing",
payload: dict | None = None,
) -> DeepResearchEvent:
self._seq += 1
ev = DeepResearchEvent(
seq=self._seq,
session_id=self._session_id,
job_id=self._job_id,
type=type,
phase=phase,
payload=payload or {},
)
self.events.append(ev)
return ev
async def publish_live(
self,
type: str, # noqa: A002
*,
phase: ResearchPhase = "initializing",
payload: dict | None = None,
) -> None:
self.live_events.append(
{
"seq": 0,
"sessionId": self._session_id,
"jobId": self._job_id,
"type": type,
"phase": phase,
"payload": payload or {},
}
)
__all__ = ["NullEventSink", "PersistingEventSink"]
@@ -0,0 +1,432 @@
"""Optional, DeerFlow-configured report image generation.
The upstream GPT Researcher image skill owns its provider configuration, so it
cannot be safely reused in DeerFlow. This adapter supports both a deployment
configured OpenAI-compatible ``/images/generations`` endpoint and the native
DashScope multimodal-generation endpoint used by Qwen Image 3.0 and Wan 2.7.
It is never configured from a browser request: model credentials stay in
``config.yaml`` and the per-session snapshot contains only the boolean
``generate_images``.
OpenAI-compatible images are requested as base64; DashScope image URLs are
downloaded immediately. The runner therefore always puts exact bytes into the
session's protected artifact directory, and reports never depend on an expiring
public image URL.
"""
from __future__ import annotations
import base64
import binascii
from typing import Any, Literal, Protocol, runtime_checkable
from urllib.parse import urlparse
import httpx
from pydantic import BaseModel, Field, field_validator
_MAX_IMAGE_BYTES = 20 * 1024 * 1024
_DASHSCOPE_GENERATION_PATH = "/api/v1/services/aigc/multimodal-generation/generation"
class ImageGenerationError(RuntimeError):
"""A recoverable image-provider failure (the report still completes)."""
class ImageGenerationSettings(BaseModel):
"""Server-only configuration for a report image endpoint.
Example ``config.yaml`` section::
deep_research:
image_generation:
enabled: true
provider: dashscope_native
base_url: https://token-plan.cn-beijing.maas.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation
api_key: $DEEP_RESEARCH_IMAGE_API_KEY
model: qwen-image-3.0-pro
``api_key`` is resolved by :class:`AppConfig` while loading YAML, but this
object is deliberately not placed in ``DeepResearchConfig`` and is never
persisted to a session/job row.
"""
enabled: bool = False
provider: Literal["openai_compatible", "dashscope_native"] = "openai_compatible"
base_url: str = ""
api_key: str = ""
model: str = ""
size: str = "1024x1024"
quality: str | None = None
watermark: bool = False
timeout_seconds: float = Field(default=120, ge=1, le=300)
model_config = {"extra": "ignore"}
@field_validator("base_url")
@classmethod
def _normalise_base_url(cls, value: str) -> str:
return value.strip().rstrip("/")
@field_validator("size")
@classmethod
def _validate_size(cls, value: str) -> str:
text = value.strip()
if not text or len(text) > 32:
raise ValueError("image_generation.size must be a short non-empty value")
return text
@property
def is_configured(self) -> bool:
return (
self.enabled
and self.base_url.startswith(("https://", "http://"))
and bool(self.api_key.strip())
and bool(self.model.strip())
)
@classmethod
def from_app_config(cls) -> ImageGenerationSettings:
"""Read the optional ``deep_research.image_generation`` section.
The core ``AppConfig`` allows product-specific extra sections, which
keeps this optional capability backward-compatible with old installs.
Any malformed optional configuration disables image generation rather
than preventing the gateway from serving normal research requests.
"""
try:
from deerflow.config import get_app_config
root = (get_app_config().model_extra or {}).get("deep_research")
raw = root.get("image_generation") if isinstance(root, dict) else None
return cls.model_validate(raw if isinstance(raw, dict) else {})
except Exception: # noqa: BLE001
return cls()
@runtime_checkable
class ImageProvider(Protocol):
"""Generate an image for a research section.
Successful providers return a mapping with binary ``data``, ``mime_type``,
``alt`` and ``prompt`` fields. The runner, not the provider, writes the
result into a protected research artifact.
"""
async def generate(
self,
prompt: str,
*,
section_id: str = "",
user_id: str = "",
) -> dict[str, Any] | None:
"""Return image bytes metadata, or ``None`` when generation is disabled."""
...
class NullImageProvider:
"""No-op image provider — ``generate_images=false`` default."""
async def generate(
self,
prompt: str,
*,
section_id: str = "",
user_id: str = "",
) -> None:
return None
class OpenAICompatibleImageProvider:
"""Generate one image through a configured OpenAI-compatible endpoint.
``transport`` is injection-only for adapter tests. Production instances
use a normal ``httpx.AsyncClient`` and make no assumptions about a specific
vendor beyond the standard ``/images/generations`` response shape.
"""
def __init__(
self,
settings: ImageGenerationSettings,
*,
transport: httpx.AsyncBaseTransport | None = None,
) -> None:
if not settings.is_configured:
raise ValueError("Image generation provider is not configured")
self._settings = settings
self._transport = transport
async def generate(
self,
prompt: str,
*,
section_id: str = "",
user_id: str = "",
) -> dict[str, Any]:
del user_id # Credentials are deployment-scoped; users never supply them.
body: dict[str, Any] = {
"model": self._settings.model,
"prompt": prompt,
"n": 1,
"size": self._settings.size,
# Base64 is mandatory: it yields a stable, protected local artifact.
"response_format": "b64_json",
}
if self._settings.quality:
body["quality"] = self._settings.quality
headers = {
"Authorization": f"Bearer {self._settings.api_key}",
"Content-Type": "application/json",
}
endpoint = f"{self._settings.base_url}/images/generations"
try:
async with httpx.AsyncClient(
timeout=self._settings.timeout_seconds,
transport=self._transport,
) as client:
response = await client.post(endpoint, json=body, headers=headers)
response.raise_for_status()
except httpx.TimeoutException as exc:
raise ImageGenerationError("图片生成服务响应超时") from exc
except httpx.HTTPStatusError as exc:
raise ImageGenerationError(f"图片生成服务返回 HTTP {exc.response.status_code}") from exc
except httpx.HTTPError as exc:
raise ImageGenerationError("图片生成服务暂时不可用") from exc
try:
payload = response.json()
except ValueError as exc:
raise ImageGenerationError("图片生成服务返回了无效响应") from exc
items = payload.get("data") if isinstance(payload, dict) else None
item = items[0] if isinstance(items, list) and items and isinstance(items[0], dict) else None
encoded = str((item or {}).get("b64_json") or "")
if not encoded:
raise ImageGenerationError("图片生成服务未返回可保存的图片数据")
# A few compatible endpoints prepend a data-URL header. Strip it
# without accepting arbitrary content as a URL or another fetch target.
if encoded.startswith("data:") and "," in encoded:
encoded = encoded.split(",", 1)[1]
try:
data = base64.b64decode(encoded, validate=True)
except (ValueError, binascii.Error) as exc:
raise ImageGenerationError("图片生成服务返回的图片编码无效") from exc
if not data or len(data) > _MAX_IMAGE_BYTES:
raise ImageGenerationError("生成图片为空或超过大小限制")
mime_type = str((item or {}).get("mime_type") or "image/png").lower()
if mime_type not in {"image/png", "image/jpeg", "image/webp"}:
mime_type = "image/png"
alt = f"{section_id or '研究报告'}配图"
return {"data": data, "mime_type": mime_type, "alt": alt, "prompt": prompt}
class DashScopeNativeImageProvider:
"""Generate report illustrations through DashScope's native image API.
Qwen Image 3.0 and Wan 2.7 do not implement the OpenAI
``/images/generations`` request/response contract. They receive a
multimodal ``input.messages`` payload and normally return a short-lived
image URL. The URL is downloaded immediately so the research report keeps
a protected local artifact, just like the OpenAI-compatible provider.
"""
def __init__(
self,
settings: ImageGenerationSettings,
*,
transport: httpx.AsyncBaseTransport | None = None,
) -> None:
if not settings.is_configured:
raise ValueError("Image generation provider is not configured")
self._settings = settings
self._transport = transport
async def generate(
self,
prompt: str,
*,
section_id: str = "",
user_id: str = "",
) -> dict[str, Any]:
del user_id # Credentials are deployment-scoped; users never supply them.
body = _dashscope_request_body(self._settings, prompt)
headers = {
"Authorization": f"Bearer {self._settings.api_key}",
"Content-Type": "application/json",
}
try:
async with httpx.AsyncClient(
timeout=self._settings.timeout_seconds,
transport=self._transport,
follow_redirects=True,
) as client:
response = await client.post(
_dashscope_generation_endpoint(self._settings.base_url),
json=body,
headers=headers,
)
response.raise_for_status()
payload = response.json()
encoded, image_url, mime_type = _dashscope_image_result(payload)
if encoded:
data = _decode_image_base64(encoded)
elif image_url:
data, mime_type = await _download_dashscope_image(client, image_url, mime_type)
else:
raise ImageGenerationError("千问图片生成服务未返回图片地址或图片数据")
except ImageGenerationError:
raise
except httpx.TimeoutException as exc:
raise ImageGenerationError("图片生成服务响应超时") from exc
except httpx.HTTPStatusError as exc:
raise ImageGenerationError(_http_error_message(exc.response)) from exc
except httpx.HTTPError as exc:
raise ImageGenerationError("图片生成服务暂时不可用") from exc
except ValueError as exc:
raise ImageGenerationError("图片生成服务返回了无效响应") from exc
if not data or len(data) > _MAX_IMAGE_BYTES:
raise ImageGenerationError("生成图片为空或超过大小限制")
alt = f"{section_id or '研究报告'}配图"
return {"data": data, "mime_type": _normalise_mime_type(mime_type), "alt": alt, "prompt": prompt}
def _dashscope_generation_endpoint(base_url: str) -> str:
"""Resolve a native DashScope endpoint without relying on a workspace id.
Token Plan deployments commonly expose the full native endpoint, while
regular DashScope configuration often uses just an API origin. Supporting
both avoids an accidental ``.../generation/images/generations`` path.
"""
base = base_url.rstrip("/")
if base.endswith(_DASHSCOPE_GENERATION_PATH):
return base
parsed = urlparse(base)
if not parsed.scheme or not parsed.netloc:
return base
return f"{parsed.scheme}://{parsed.netloc}{_DASHSCOPE_GENERATION_PATH}"
def _dashscope_request_body(settings: ImageGenerationSettings, prompt: str) -> dict[str, Any]:
parameters: dict[str, Any] = {
"n": 1,
"size": settings.size.replace("x", "*").replace("X", "*"),
"watermark": settings.watermark,
}
# ``prompt_extend`` is a Qwen Image-only parameter. Wan 2.7 rejects
# unrelated generation options, so do not send it there.
if settings.model.strip().lower().startswith("qwen-image-"):
parameters["prompt_extend"] = True
return {
"model": settings.model,
"input": {
"messages": [
{
"role": "user",
"content": [{"text": prompt}],
}
]
},
"parameters": parameters,
}
def _dashscope_image_result(payload: object) -> tuple[str, str, str]:
"""Return optional base64, image URL and MIME type from native output."""
if not isinstance(payload, dict):
return "", "", ""
output = payload.get("output")
choices = output.get("choices") if isinstance(output, dict) else None
if not isinstance(choices, list):
return "", "", ""
for choice in choices:
message = choice.get("message") if isinstance(choice, dict) else None
content = message.get("content") if isinstance(message, dict) else None
if not isinstance(content, list):
continue
for part in content:
if not isinstance(part, dict):
continue
encoded = str(part.get("b64_json") or "")
raw_image_url = part.get("image") or part.get("image_url")
if isinstance(raw_image_url, dict):
raw_image_url = raw_image_url.get("url")
image_url = str(raw_image_url or "")
if encoded or image_url:
return encoded, image_url, str(part.get("mime_type") or "")
return "", "", ""
async def _download_dashscope_image(
client: httpx.AsyncClient,
image_url: str,
mime_type: str,
) -> tuple[bytes, str]:
"""Download only an HTTPS model result, never forwarding the API key."""
parsed = urlparse(image_url)
if parsed.scheme != "https" or not parsed.netloc:
raise ImageGenerationError("千问图片生成服务返回了不安全的图片地址")
response = await client.get(image_url, headers={"Accept": "image/png,image/jpeg,image/webp"})
response.raise_for_status()
final_url = urlparse(str(response.url))
if final_url.scheme != "https" or not final_url.netloc:
raise ImageGenerationError("千问图片下载地址不安全")
response_mime_type = response.headers.get("Content-Type", "").split(";", 1)[0].strip().lower()
return response.content, response_mime_type or mime_type
def _decode_image_base64(encoded: str) -> bytes:
if encoded.startswith("data:") and "," in encoded:
encoded = encoded.split(",", 1)[1]
try:
return base64.b64decode(encoded, validate=True)
except (ValueError, binascii.Error) as exc:
raise ImageGenerationError("图片生成服务返回的图片编码无效") from exc
def _normalise_mime_type(value: str) -> str:
mime_type = value.lower()
return mime_type if mime_type in {"image/png", "image/jpeg", "image/webp"} else "image/png"
def _http_error_message(response: httpx.Response) -> str:
"""Expose a short provider message without leaking credentials or payloads."""
detail = ""
try:
payload = response.json()
if isinstance(payload, dict):
error = payload.get("error")
source = error if isinstance(error, dict) else payload
message = source.get("message") if isinstance(source, dict) else None
if isinstance(message, str) and message.strip():
detail = f":{message.strip()[:300]}"
except ValueError:
pass
return f"图片生成服务返回 HTTP {response.status_code}{detail}"
def build_image_provider() -> ImageProvider:
"""Build the optional provider from server config, or a safe no-op."""
settings = ImageGenerationSettings.from_app_config()
if not settings.is_configured:
return NullImageProvider()
if settings.provider == "dashscope_native":
return DashScopeNativeImageProvider(settings)
return OpenAICompatibleImageProvider(settings)
__all__ = [
"ImageGenerationError",
"ImageGenerationSettings",
"ImageProvider",
"NullImageProvider",
"OpenAICompatibleImageProvider",
"DashScopeNativeImageProvider",
"build_image_provider",
]
@@ -0,0 +1,639 @@
"""LLM adapter — the single chokepoint between vendored code and DeerFlow models.
The vendored GPT Researcher code expects ``create_chat_completion()`` /
``get_llm()`` style helpers (``gpt_researcher/utils/llm.py``). Rather than
letting it dynamically import providers, we keep those function *signatures* in
the vendored copy but route them through :class:`DeerFlowCompletionBackend`,
which delegates to DeerFlow's :func:`create_chat_model`.
Responsibilities (§9):
- Resolve the model name from the per-task ``DeepResearchConfig`` by role
(``fast`` / ``smart`` / ``strategic``); ``None`` → system default.
- Convert ``[{role, content}, ...]`` dicts to LangChain messages.
- Call ``model.ainvoke()`` and normalise text + ``usage_metadata``.
- Support structured output via a Pydantic ``response_schema`` (one JSON-repair
retry, no ``json-repair`` hard dep — §9.5).
- Emit a ``model_usage`` event per call.
- Honour the :class:`CancellationToken`.
Retry policy (§9.5): exactly **one** retry layer here — a structured-output
parse failure triggers one "fix the JSON" re-prompt. Cancel / permission /
config errors are never retried.
"""
from __future__ import annotations
import asyncio
import logging
from collections.abc import Awaitable, Callable
from typing import TYPE_CHECKING, Any
from deerflow.agents.deep_research.cancellation import CancellationToken
from deerflow.agents.deep_research.config import DeepResearchConfig, ModelRole
from deerflow.agents.deep_research.events import EventSink
from deerflow.agents.deep_research.types import CompletionResult
from deerflow.utils.stream_text import InlineThinkTagFilter, strip_inline_think_tags
if TYPE_CHECKING:
from langchain.chat_models import BaseChatModel
from langchain_core.messages import BaseMessage
from pydantic import BaseModel
logger = logging.getLogger(__name__)
def _parent_isolated_run_config(run_name: str) -> dict[str, Any]:
"""Keep LangGraph runtime, drop only inherited LLM streaming callbacks.
``StreamWriter`` reads ``get_config()[configurable]`` at emit time. Replacing
the child config with a blank dict makes ``report_delta`` / ``report_thinking``
raise and get swallowed, so the sandbox only appears after the file lands.
"""
parent: dict[str, Any] = {}
try:
from langchain_core.runnables.config import var_child_runnable_config
current = var_child_runnable_config.get()
if isinstance(current, dict):
parent = dict(current)
except Exception: # noqa: BLE001 — isolation must never fail the completion
parent = {}
tags = [tag for tag in (parent.get("tags") or []) if tag != "nostream"]
tags.append("nostream")
parent["callbacks"] = []
parent["tags"] = tags
parent["run_name"] = run_name
return parent
def _clear_inherited_runnable_config(isolated: dict[str, Any]) -> Callable[[], None]:
"""Install the isolated config for the duration of ``astream``."""
try:
from langchain_core.runnables.config import var_child_runnable_config
except ImportError:
return lambda: None
token = var_child_runnable_config.set(isolated)
return lambda: var_child_runnable_config.reset(token)
# ── message conversion ──────────────────────────────────────────────────────
def to_langchain_messages(messages: list[dict[str, str]]) -> list[BaseMessage]:
"""Convert ``[{role, content}, ...]`` dicts to LangChain message objects.
``role`` accepts the usual OpenAI-ish names: ``system`` / ``user`` /
``assistant`` / ``human`` / ``ai``. Unknown roles fall back to ``HumanMessage``
so a bad prompt never crashes the whole research run.
"""
from langchain_core.messages import (
AIMessage,
HumanMessage,
SystemMessage,
)
out: list[BaseMessage] = []
for m in messages:
role = (m.get("role") or "user").lower()
content = m.get("content") or ""
if role in ("system", "developer"):
out.append(SystemMessage(content=content))
elif role in ("assistant", "ai"):
out.append(AIMessage(content=content))
else: # user / human / unknown → human (safe default)
out.append(HumanMessage(content=content))
return out
def _content_to_text(resp: Any) -> str:
"""Best-effort extraction of text from a LangChain AIMessage-like response."""
content = getattr(resp, "content", None)
if isinstance(content, str):
return content
if isinstance(content, list):
# Some providers return content blocks; concatenate text parts.
parts: list[str] = []
for block in content:
if isinstance(block, str):
parts.append(block)
elif isinstance(block, dict):
# Never fall through to ``content`` for structured reasoning
# blocks. Providers differ here, but those blocks are not
# report prose and must not reach ReportChunkEmitter.
if block.get("type") in {
"thinking",
"thinking_delta",
"reasoning",
"reasoning_delta",
}:
continue
parts.append(str(block.get("text") or block.get("content") or ""))
return "".join(parts)
return str(content or "")
def _reasoning_delta(chunk: Any) -> str:
"""Extract the incremental reasoning/thinking text from a streamed chunk.
Reasoning models (deepseek-reasoner, GLM thinking, QwQ, ...) stream their
thinking in ``additional_kwargs.reasoning_content`` while ``content`` stays
empty until the final answer starts. A stream loop that only reads
``content`` skips these chunks entirely, so callers get a dedicated
callback for them.
"""
ak = getattr(chunk, "additional_kwargs", None) or {}
val = ak.get("reasoning_content") or ak.get("reasoning") or ""
if isinstance(val, str) and val:
return val
# Anthropic/OpenAI-compatible adapters may put the same increment in an
# explicitly typed content block instead of ``additional_kwargs``.
content = getattr(chunk, "content", None)
if isinstance(content, list):
parts = [
str(block.get("text") or block.get("content") or "")
for block in content
if isinstance(block, dict)
and block.get("type") in {"thinking", "thinking_delta", "reasoning", "reasoning_delta"}
]
return "".join(parts)
return ""
def _extract_usage(resp: Any) -> dict[str, Any]:
"""Read ``AIMessage.usage_metadata`` (§9.4) — model-agnostic token accounting."""
um = getattr(resp, "usage_metadata", None)
if isinstance(um, dict):
return {
"prompt_tokens": um.get("input_tokens", 0),
"completion_tokens": um.get("output_tokens", 0),
"total_tokens": um.get("total_tokens", 0),
}
# Fallback: some providers populate response_metadata.
rm = getattr(resp, "response_metadata", None) or {}
token_usage = rm.get("token_usage") or rm.get("usage") or {}
if isinstance(token_usage, dict) and token_usage:
return {
"prompt_tokens": token_usage.get("prompt_tokens", 0),
"completion_tokens": token_usage.get("completion_tokens", 0),
"total_tokens": token_usage.get("total_tokens", 0),
}
return {}
# ── errors ──────────────────────────────────────────────────────────────────
class CompletionError(Exception):
"""Non-retryable completion failure (config / permission / budget)."""
def __init__(self, message: str, *, code: str = "completion_error", retryable: bool = False) -> None:
super().__init__(message)
self.code = code
self.retryable = retryable
# ── the backend ─────────────────────────────────────────────────────────────
class DeerFlowCompletionBackend:
"""The LLM adapter every vendored call funnels through.
Constructed once per job (with the frozen config) and handed to runners via
:class:`AdapterBundle`. Model instances are cached by role so repeated
calls don't rebuild the provider.
"""
def __init__(
self,
config: DeepResearchConfig,
event_sink: EventSink | None = None,
cancellation: CancellationToken | None = None,
force_disable_thinking: bool = False,
) -> None:
self._config = config
self._events = event_sink
self._cancel = cancellation
# 与 lead agent 同语义的「思考硬关」开关(内网 vLLM/Qwen 只声明
# supports_thinking 时,thinking_enabled=False 不会注入 enable_thinking=false)。
self._force_disable_thinking = force_disable_thinking
self._models: dict[str, BaseChatModel] = {}
# Running token totals (keyed by model name) for the final usage report.
self._usage_totals: dict[str, dict[str, int]] = {}
# ── model resolution ────────────────────────────────────────────────────
def _resolve_model_name(self, role: ModelRole) -> str | None:
return self._config.resolve_model_name(role)
def _get_model(self, role: ModelRole) -> BaseChatModel:
name = self._resolve_model_name(role)
cache_key = name or "__default__"
cached = self._models.get(cache_key)
if cached is not None:
return cached
from deerflow.models import create_chat_model
model = create_chat_model(
name=name,
thinking_enabled=self._config.thinking_enabled,
force_disable_thinking=self._force_disable_thinking,
)
self._models[cache_key] = model
return model
# ── the main entry point ────────────────────────────────────────────────
async def complete(
self,
*,
model_role: ModelRole = "smart",
messages: list[dict[str, str]],
temperature: float | None = None,
response_schema: type[BaseModel] | None = None,
operation: str = "unknown",
) -> CompletionResult:
"""Run one completion, normalise output, bill usage.
Args:
model_role: which config role to use (fast/cheap research queries →
``fast``; report writing → ``smart``; strategic planning → ``strategic``).
messages: OpenAI-style ``[{role, content}]`` dicts.
temperature: optional override (most vendored calls don't set one).
response_schema: if given, parse + validate the output against this
Pydantic model (structured output). One JSON-repair retry on
parse failure (§9.5).
operation: a short label for the ``model_usage`` event
(e.g. ``"generate_queries"``, ``"write_report"``).
"""
if self._cancel is not None:
self._cancel.raise_if_cancelled()
model = self._get_model(model_role)
model_name = self._resolve_model_name(model_role) or "default"
lc_messages = to_langchain_messages(messages)
kwargs: dict[str, Any] = {}
if temperature is not None:
kwargs["temperature"] = temperature
# ── plain completion ──
if response_schema is None:
resp = await self._invoke_with_timeout(model, lc_messages, kwargs, operation)
text = _content_to_text(resp)
usage = _extract_usage(resp)
await self._record_usage(model_name, operation, usage)
return CompletionResult(text=text, model=model_name, usage=usage)
# ── structured completion (with one repair retry) ──
schema_instructions = _schema_instructions(response_schema)
structured_messages = list(lc_messages)
# Append the schema instruction to the last system/user message block.
structured_messages = _inject_schema_hint(structured_messages, schema_instructions)
for attempt in range(2):
if self._cancel is not None:
self._cancel.raise_if_cancelled()
resp = await self._invoke_with_timeout(model, structured_messages, kwargs, operation)
text = _content_to_text(resp)
usage = _extract_usage(resp)
parsed = _try_parse_structured(text, response_schema)
if parsed is not None:
await self._record_usage(model_name, operation, usage)
return CompletionResult(text=text, model=model_name, usage=usage, structured=parsed)
if attempt == 0:
# One repair re-prompt (§9.5).
logger.warning("deep_research llm: JSON parse failed for %s, retrying", operation)
structured_messages = _inject_repair_hint(structured_messages, text, response_schema)
await self._record_usage(model_name, operation, usage)
raise CompletionError(
f"Structured output for '{operation}' could not be parsed after retry",
code="structured_parse_failed",
retryable=True,
)
async def _invoke_with_timeout(
self,
model: BaseChatModel,
messages: list[BaseMessage],
kwargs: dict[str, Any],
operation: str,
) -> Any:
"""Bound a non-streaming control-plane call without capping report prose.
Model configurations may have a much longer HTTP ``request_timeout``.
A research job cannot leave its visible planning/review stage frozen for
that whole interval, so these short structured calls get their own
deadline. ``wait_for`` cancels the pending provider coroutine before a
durable runner either uses its safe fallback or records a terminal error.
"""
timeout = self._config.control_plane_timeout_seconds
try:
return await asyncio.wait_for(model.ainvoke(messages, **kwargs), timeout=timeout)
except TimeoutError as exc:
if self._cancel is not None:
self._cancel.raise_if_cancelled()
raise CompletionError(
f"Model did not complete '{operation}' within {timeout:g} seconds",
code="completion_timeout",
retryable=True,
) from exc
async def stream_complete(
self,
*,
model_role: ModelRole = "smart",
messages: list[dict[str, str]],
temperature: float | None = None,
max_tokens: int | None = None,
operation: str = "unknown",
on_delta: Callable[[str], Awaitable[None]] | None = None,
on_reasoning: Callable[[str], Awaitable[None]] | None = None,
) -> CompletionResult:
"""Run a prose completion through LangChain's incremental stream API.
Structured planning/review calls intentionally stay on :meth:`complete`.
This method is reserved for report prose so the final returned text is
exactly the concatenation of the deltas the user has already seen.
``on_reasoning`` receives the thinking/reasoning increments reasoning
models emit before the answer; they are not part of the returned text.
"""
if self._cancel is not None:
self._cancel.raise_if_cancelled()
model = self._get_model(model_role)
model_name = self._resolve_model_name(model_role) or "default"
kwargs: dict[str, Any] = {}
if temperature is not None:
kwargs["temperature"] = temperature
if max_tokens is not None:
# Report-only callers may need a larger completion than a model's
# conservative deployment default. Codex Responses endpoints
# reject both token-limit parameters, so preserve the factory's
# provider-specific exclusion for that adapter.
from deerflow.models.openai_codex_provider import CodexChatModel
if not isinstance(model, CodexChatModel):
kwargs["max_tokens"] = max_tokens
text_parts: list[str] = []
usage: dict[str, Any] = {}
inline_think_filter = InlineThinkTagFilter()
# Isolate from the parent LangGraph run: inherited callbacks would
# replay every report token into the chat ``messages`` stream, so the
# left-hand bubble and the sandbox would write the same prose at once.
# Custom ``report_delta`` events remain the only live preview channel.
isolated_config = _parent_isolated_run_config(f"deep_research:{operation}")
reset_parent_config = _clear_inherited_runnable_config(isolated_config)
attempts = 2 if operation.startswith("write_report") else 1
last_error: Exception | None = None
try:
for attempt in range(attempts):
text_parts.clear()
inline_think_filter = InlineThinkTagFilter()
try:
async for chunk in model.astream(
to_langchain_messages(messages),
config=isolated_config,
**kwargs,
):
if self._cancel is not None:
self._cancel.raise_if_cancelled()
reasoning = _reasoning_delta(chunk)
if reasoning and on_reasoning is not None:
await on_reasoning(reasoning)
# A few OpenAI-compatible models put their reasoning in the
# ordinary content channel instead of ``reasoning_content``.
# Split it from Markdown even when tags span tokens: callers may
# display a live thought trace, but it never joins ``text``.
delta, inline_reasoning = inline_think_filter.push_parts(_content_to_text(chunk))
if inline_reasoning and on_reasoning is not None:
await on_reasoning(inline_reasoning)
if delta:
text_parts.append(delta)
if on_delta is not None:
await on_delta(delta)
chunk_usage = _extract_usage(chunk)
if any(chunk_usage.values()):
usage = chunk_usage
last_error = None
break
except Exception as exc: # noqa: BLE001 - report writing waits like ordinary chat
from deerflow.agents.deep_research.cancellation import ResearchCancelled
if isinstance(exc, (ResearchCancelled, asyncio.CancelledError)):
raise
last_error = exc
# Only replay a clean connection; a half-written stream is
# reset by the runner, not by appending a second attempt.
if text_parts or attempt + 1 >= attempts:
raise
logger.warning(
"deep_research llm: %s stream interrupted before the first token (%s); retrying once",
operation,
type(exc).__name__,
)
await asyncio.sleep(1.5)
if last_error is not None:
raise last_error
finally:
reset_parent_config()
tail, inline_reasoning_tail = inline_think_filter.finish_parts()
if inline_reasoning_tail and on_reasoning is not None:
await on_reasoning(inline_reasoning_tail)
if tail:
text_parts.append(tail)
if on_delta is not None:
await on_delta(tail)
await self._record_usage(model_name, operation, usage)
return CompletionResult(text="".join(text_parts), model=model_name, usage=usage)
async def complete_prose(
self,
*,
model_role: ModelRole = "smart",
messages: list[dict[str, str]],
temperature: float | None = None,
max_tokens: int | None = None,
operation: str = "unknown",
) -> CompletionResult:
"""Generate long prose without the provider's inter-chunk timeout.
This is the report retry path. Some reasoning models pause for longer
than the streaming client's 120-second chunk deadline after already
producing several chunks. A normal complete response can still finish
successfully, so the retry deliberately waits for that full response.
"""
if self._cancel is not None:
self._cancel.raise_if_cancelled()
model = self._get_model(model_role)
model_name = self._resolve_model_name(model_role) or "default"
kwargs: dict[str, Any] = {}
if temperature is not None:
kwargs["temperature"] = temperature
if max_tokens is not None:
from deerflow.models.openai_codex_provider import CodexChatModel
if not isinstance(model, CodexChatModel):
kwargs["max_tokens"] = max_tokens
isolated_config = _parent_isolated_run_config(f"deep_research:{operation}")
reset_parent_config = _clear_inherited_runnable_config(isolated_config)
try:
response = await model.ainvoke(
to_langchain_messages(messages),
config=isolated_config,
**kwargs,
)
finally:
reset_parent_config()
if self._cancel is not None:
self._cancel.raise_if_cancelled()
# ``ainvoke`` returns raw content, including the inline ``<think>``
# block some OpenAI-compatible deployments write there. The streamed
# path filters it out, so this retry path must not reintroduce it.
text = strip_inline_think_tags(_content_to_text(response))
usage = _extract_usage(response)
await self._record_usage(model_name, operation, usage)
return CompletionResult(text=text, model=model_name, usage=usage)
# ── usage accounting ────────────────────────────────────────────────────
async def _record_usage(self, model_name: str, operation: str, usage: dict[str, Any]) -> None:
"""Accumulate per-model totals + emit a ``model_usage`` event."""
bucket = self._usage_totals.setdefault(model_name, {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0})
for k in bucket:
bucket[k] += int(usage.get(k, 0) or 0)
if self._events is not None:
await self._events.emit(
"model_usage",
payload={
"model": model_name,
"operation": operation,
"usage": usage,
},
)
@property
def usage_totals(self) -> dict[str, dict[str, int]]:
"""Snapshot of cumulative token usage by model (for the final result)."""
return {k: dict(v) for k, v in self._usage_totals.items()}
# ── convenience: the upstream-compatible function family ────────────────
#
# These mirror ``gpt_researcher/utils/llm.py`` signatures so the vendored
# action/skill code needs minimal changes — it calls a context-local bound
# method instead of a module global.
async def create_chat_completion(
self,
messages: list[dict[str, str]],
*,
model_role: ModelRole = "smart",
temperature: float | None = None,
response_format: type[BaseModel] | None = None,
stream: bool = False,
) -> str:
"""Upstream-compatible ``create_chat_completion`` → returns raw text."""
if stream and response_format is None:
result = await self.stream_complete(
model_role=model_role,
messages=messages,
temperature=temperature,
operation="chat_completion",
)
else:
result = await self.complete(
model_role=model_role,
messages=messages,
temperature=temperature,
response_schema=response_format,
operation="chat_completion",
)
return result.text
# ── structured-output helpers (no json-repair dependency) ───────────────────
def _schema_instructions(schema: type[BaseModel]) -> str:
"""Build a compact "respond as JSON matching this schema" instruction."""
props = schema.model_json_schema().get("properties", {})
fields = ", ".join(f'"{k}"' for k in props)
return f"Respond with ONLY a valid JSON object (no markdown, no prose) containing these keys: {fields}."
def _inject_schema_hint(messages: list[BaseMessage], hint: str) -> list[BaseMessage]:
"""Append the schema instruction as a final system message."""
from langchain_core.messages import SystemMessage
return [*messages, SystemMessage(content=hint)]
def _inject_repair_hint(messages: list[BaseMessage], bad_text: str, schema: type[BaseModel]) -> list[BaseMessage]:
"""Re-prompt: 'your last reply was invalid JSON, fix it'."""
from langchain_core.messages import HumanMessage
return [
*messages,
HumanMessage(content=(f"Your previous reply was not valid JSON and could not be parsed:\n{bad_text[:500]}\n\nPlease respond again with ONLY a valid JSON object.")),
]
def parse_structured_text(text: str, schema: type[BaseModel]) -> BaseModel | None:
"""Parse a completed (possibly streamed) model output against a schema.
Public wrapper over the same fence-stripping, blob-extracting parse used
by :meth:`DeerFlowCompletionBackend.complete`, so runners that collect the
text through :meth:`stream_complete` can validate it afterwards.
"""
return _try_parse_structured(text, schema)
def _try_parse_structured(text: str, schema: type[BaseModel]) -> BaseModel | None:
"""Strip markdown fences + attempt ``schema.model_validate_json``."""
cleaned = _strip_code_fences(text).strip()
if not cleaned:
return None
try:
return schema.model_validate_json(cleaned)
except Exception:
# Try to locate the first {...} blob as a last resort.
start = cleaned.find("{")
end = cleaned.rfind("}")
if start != -1 and end > start:
try:
return schema.model_validate_json(cleaned[start : end + 1])
except Exception:
return None
return None
def _strip_code_fences(text: str) -> str:
"""Remove ```json ... ``` / ``` ... ``` wrapping if present."""
stripped = text.strip()
if stripped.startswith("```"):
# Drop first line (the fence + optional language tag).
lines = stripped.splitlines()
if len(lines) >= 2:
lines = lines[1:]
if lines and lines[-1].strip().startswith("```"):
lines = lines[:-1]
return "\n".join(lines)
return stripped
__all__ = [
"CompletionError",
"DeerFlowCompletionBackend",
"parse_structured_text",
"to_langchain_messages",
]
@@ -0,0 +1,75 @@
"""Material provider — DeerFlow's unified information-collection boundary.
The vendored GPT Researcher code normally fires off its own retrievers + scrapers
per source URL. We replace that entirely: the :class:`MaterialProvider` returns
**fully-fetched** :class:`ResearchMaterial` objects (``raw_content`` already
populated), so the vendored scraper/BrowserManager is never invoked (§6.4).
First version (Phase 1) wraps :func:`deerflow.community.configurable_search.execute_search`:
1. Run the configured ``web_search`` tool for the query.
2. Map normalised results (``recUuid / content / title / url / source / time``)
→ :class:`ResearchMaterial`.
3. If ``content`` is already full text → use as ``raw_content`` (no fetch needed).
4. If only a snippet → a DeerFlow fetch tool backfills (Phase 2).
5. Dedupe by ``rec_uuid`` → canonical URL → ``content_hash``.
The provider never imports ``app.*`` and never does raw ``requests.get``.
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Protocol, runtime_checkable
from deerflow.agents.deep_research.types import ResearchMaterial
if TYPE_CHECKING:
from deerflow.agents.deep_research.cancellation import CancellationToken
@runtime_checkable
class MaterialProvider(Protocol):
"""Protocol every material channel implements (§8.3)."""
async def search(
self,
query: str,
*,
domains: list[str] | None = None,
limit: int = 8,
user_id: str = "",
session_id: str = "",
cancellation: CancellationToken | None = None,
) -> list[ResearchMaterial]:
"""Return fully-fetched materials for ``query``.
Implementations guarantee ``raw_content`` is non-empty for every item
that has fetchable text (the vendored writer relies on this).
"""
...
def dedupe_materials(materials: list[ResearchMaterial]) -> list[ResearchMaterial]:
"""Stable dedup by rec_uuid → canonical url → content_hash (§8.3 §6).
Keeps the first occurrence; later duplicates are dropped (but their queries
are noted on the survivor's metadata so multi-query hits aren't lost).
"""
seen: dict[str, int] = {} # key → index in result
out: list[ResearchMaterial] = []
for m in materials:
key = (m.rec_uuid and f"rec:{m.rec_uuid}") or (m.url and f"url:{m.url}") or m.compute_hash()
if key in seen:
idx = seen[key]
# Record that another query also surfaced this material.
surviving = out[idx]
extra_queries = surviving.metadata.setdefault("_also_matched_queries", [])
if m.query and m.query not in extra_queries:
extra_queries.append(m.query)
continue
seen[key] = len(out)
out.append(m)
return out
__all__ = ["MaterialProvider", "dedupe_materials"]
@@ -0,0 +1,601 @@
"""Deep Research material provider backed by DeerFlow's Q&A retrieval stack.
This is the only place adapted GPT Researcher runners retrieve materials. It
uses the same hierarchy as DeerFlow Q&A instead of treating the legacy
configurable ``web_search`` endpoint as the only possible source:
1. Run enabled Q&A research skills through the existing ``SubagentExecutor``
runtime when their required search tool is registered.
2. Use a configured ``web_search`` provider when the deployment has one.
3. Fall back to DeerFlow's existing keyless DuckDuckGo search provider.
The fallback is important for this offline deployment: its configurable search
entry is deliberately disabled while it still contains a placeholder intranet
endpoint. A missing endpoint must never turn every Deep Research report into
"no materials collected".
"""
from __future__ import annotations
import asyncio
import hashlib
import importlib.util
import json
import logging
import re
from datetime import datetime
from typing import Any
from urllib.parse import urlparse
from deerflow.agents.deep_research.adapters.material import dedupe_materials
from deerflow.agents.deep_research.types import ResearchMaterial
logger = logging.getLogger(__name__)
# Treat shorter content as a snippet and try a full-text fetch when available.
SNIPPET_THRESHOLD = 300
ONLINE_SEARCH_TIMEOUT_SECONDS = 45
SECONDARY_ONLINE_SEARCH_TIMEOUT_SECONDS = 20
# Same skills that guide the Q&A assistant's public-web / knowledge retrieval.
# The order deliberately prefers the multi-round skill for a Deep Research job.
QUESTION_ANSWER_RESEARCH_SKILLS: tuple[str, ...] = (
"deep-search",
"web-research",
"weknora-knowledge-search",
)
# A skill is run only when the Q&A runtime exposes the search tool it needs.
# This prevents a disabled placeholder web_search config from launching an
# agent which is incapable of searching and can only wait for its timeout.
_SKILL_REQUIRED_TOOLS: dict[str, set[str]] = {
"deep-search": {"web_search"},
"web-research": {"web_search"},
"weknora-knowledge-search": {"weknora_search"},
}
class DeerFlowMaterialProvider:
"""MaterialProvider backed by the Q&A skills and online retrieval stack.
A configurable ``web_search`` endpoint remains supported, but it is no
longer a single point of failure. All returned materials preserve the
original source, URL and retrieval channel for citations and auditing.
"""
def __init__(
self,
*,
user_id: str = "",
session_id: str = "",
runtime_thread_id: str = "",
model_name: str | None = None,
fetch_enabled: bool = True,
retrieval_skills: list[str] | None = None,
) -> None:
self._user_id = user_id
self._session_id = session_id
self._runtime_thread_id = runtime_thread_id
self._model_name = model_name
self._fetch_enabled = fetch_enabled
self._retrieval_skills = _normalize_named_skills(retrieval_skills)
self._settings = None
self._settings_error: str | None = None
self._question_answer_tool_names = _load_question_answer_tool_names(model_name)
self._research_skills = _load_question_answer_research_skills()
self._online_search_available = importlib.util.find_spec("ddgs") is not None
self._online_search_semaphore: asyncio.Semaphore | None = None
try:
from deerflow.community.configurable_search.tools import ConfigurableSearchSettings
from deerflow.config import get_app_config
cfg = get_app_config().get_tool_config("web_search")
self._settings = ConfigurableSearchSettings.from_tool_config(cfg)
if not self._settings.enabled:
self._settings_error = "web_search tool is disabled in config"
self._settings = None
except Exception as exc: # noqa: BLE001
self._settings_error = f"web_search not configured: {exc}"
logger.warning("Deep Research configurable web_search unavailable: %s", self._settings_error)
async def search(
self,
query: str,
*,
domains: list[str] | None = None,
limit: int = 8,
user_id: str = "",
session_id: str = "",
cancellation=None,
) -> list[ResearchMaterial]:
"""Collect actual Q&A-compatible materials for one research query."""
if cancellation is not None:
cancellation.raise_if_cancelled()
materials: list[ResearchMaterial] = []
seen_keys: set[str] = set()
directed_skills = bool(self._retrieval_skills)
# 1. The genuine Q&A skill runtime. Skills may call web, MCP or another
# deployment-specific source according to their SKILL.md instructions.
# A non-empty 检索来源 list replaces the default research-skill trio.
skill_results = await self._search_via_question_answer_skills(query, limit=limit)
materials.extend(
self._normalise_search_results(
skill_results,
query=query,
channel="skill",
domains=domains,
seen_keys=seen_keys,
)
)
if cancellation is not None:
cancellation.raise_if_cancelled()
# Specified skills are the retrieval source: do not also dump generic
# web_search / DDG on top. Empty 检索来源 keeps the original fallbacks.
if directed_skills:
hydrated: list[ResearchMaterial] = []
for material in materials[:limit]:
if cancellation is not None:
cancellation.raise_if_cancelled()
if material.url and len(material.raw_content) < SNIPPET_THRESHOLD and self._fetch_enabled:
fetched = await asyncio.to_thread(_safe_fetch, material.url)
if fetched:
material.raw_content = fetched
if material.raw_content:
hydrated.append(material)
return dedupe_materials(hydrated)[:limit]
# 2. Retain compatibility with deployed configurable search providers.
if len(materials) < limit:
configured_results = await self._search_configured_web(query)
materials.extend(
self._normalise_configured_results(
configured_results,
query=query,
domains=domains,
seen_keys=seen_keys,
)
)
if cancellation is not None:
cancellation.raise_if_cancelled()
# 3. Reuse DeerFlow's maintained online provider if the Q&A search tool
# is absent or the configured provider returned too few materials.
if len(materials) < limit:
online_results = await self._search_via_online_fallback(query, limit=limit - len(materials))
materials.extend(
self._normalise_search_results(
online_results,
query=query,
channel="web_search",
domains=domains,
seen_keys=seen_keys,
)
)
hydrated: list[ResearchMaterial] = []
for material in materials[:limit]:
if cancellation is not None:
cancellation.raise_if_cancelled()
if material.url and len(material.raw_content) < SNIPPET_THRESHOLD and self._fetch_enabled:
fetched = await asyncio.to_thread(_safe_fetch, material.url)
if fetched:
material.raw_content = fetched
if material.raw_content:
hydrated.append(material)
return dedupe_materials(hydrated)[:limit]
async def _search_via_question_answer_skills(self, query: str, *, limit: int) -> list[Any]:
"""Run eligible research skills through the same runtime as Q&A."""
runnable = self._runnable_research_skills()
if not runnable:
logger.info(
"Deep Research Q&A skills are enabled but their required search tools are unavailable; "
"using the online fallback"
)
return []
try:
from deerflow.agents.ai_writing.nodes._skill_agent import run_skill_via_real_agent
except Exception as exc: # noqa: BLE001
logger.warning("Deep Research could not load the Q&A skill runtime: %s", exc)
return []
run_config = {
"configurable": {
"thread_id": self._runtime_thread_id,
"model_name": self._model_name or "",
}
}
state = {"model_name": self._model_name or ""}
collected: list[Any] = []
directed = bool(self._retrieval_skills)
for skill in runnable:
try:
results, error = await run_skill_via_real_agent(
skill=skill,
intent=f"Collect verifiable research materials for: {query}",
keywords=[query],
user_query=query,
config=run_config,
state=state,
model_name_fallback=self._model_name,
max_results=limit,
timeout_s=ONLINE_SEARCH_TIMEOUT_SECONDS,
)
except Exception as exc: # noqa: BLE001
logger.warning("Deep Research Q&A skill %s failed: %s", skill.name, exc)
continue
if results:
collected.extend(list(results)[:limit])
if not directed or len(collected) >= limit:
return collected[:limit]
if error:
logger.info("Deep Research Q&A skill %s returned no materials: %s", skill.name, error)
return collected[:limit]
def _runnable_research_skills(self) -> list[Any]:
"""Default Q&A trio, or the report-structure 检索来源 when named."""
skills = (
_load_named_research_skills(self._retrieval_skills)
if self._retrieval_skills
else self._research_skills
)
if self._retrieval_skills:
# User-named skills may use any tool; skip the trio's required-tool gate.
return list(skills)
return [
skill
for skill in skills
if _SKILL_REQUIRED_TOOLS.get(skill.name, set()) <= self._question_answer_tool_names
]
async def _search_configured_web(self, query: str) -> list[dict[str, Any]]:
"""Run the existing configurable ``web_search`` adapter when enabled."""
if self._settings is None:
return []
try:
from deerflow.community.configurable_search.tools import execute_search
raw = await execute_search(query, self._settings)
envelope = json.loads(raw)
except Exception as exc: # noqa: BLE001
logger.warning("Deep Research configured web_search failed: %s", exc)
return []
if not isinstance(envelope, dict) or envelope.get("error"):
logger.info("Deep Research configured web_search returned no materials: %s", envelope)
return []
return [item for item in envelope.get("results") or [] if isinstance(item, dict)]
async def _search_via_online_fallback(self, query: str, *, limit: int) -> list[Any]:
"""Use maintained DDG retrieval, then a relevant-result web fallback."""
if not self._online_search_available or limit < 1:
return []
if self._online_search_semaphore is None:
self._online_search_semaphore = asyncio.Semaphore(2)
try:
from deerflow.agents.ai_writing.search.duckduckgo_search import DuckDuckGoSearchProvider
async with self._online_search_semaphore:
provider = DuckDuckGoSearchProvider(max_timeout=ONLINE_SEARCH_TIMEOUT_SECONDS)
ddg_results = await asyncio.wait_for(
provider.search(query, max_results=limit),
timeout=ONLINE_SEARCH_TIMEOUT_SECONDS + 5,
)
relevant_results = _filter_relevant_search_results(ddg_results, query)
if relevant_results:
return relevant_results[:limit]
# DDGS can occasionally return generic login/tracking pages for
# Chinese queries. Do not turn those into report citations;
# collect a second set from the search HTML instead.
return await asyncio.to_thread(_search_360_web, query, limit)
except Exception as exc: # noqa: BLE001
logger.warning("Deep Research online fallback search failed for %r: %s", query, exc)
return []
def _normalise_configured_results(
self,
results: list[dict[str, Any]],
*,
query: str,
domains: list[str] | None,
seen_keys: set[str],
) -> list[ResearchMaterial]:
materials: list[ResearchMaterial] = []
for item in results:
material = result_to_material(item, query=query)
if not _material_matches_domains(material, domains) or not _accept_material(material, seen_keys):
continue
material.metadata["retrieval_channel"] = "configured_web_search"
materials.append(material)
return materials
def _normalise_search_results(
self,
results: list[Any],
*,
query: str,
channel: str,
domains: list[str] | None,
seen_keys: set[str],
) -> list[ResearchMaterial]:
materials: list[ResearchMaterial] = []
for result in results:
material = _search_result_to_material(result, query=query, channel=channel)
if material is None or not _material_matches_domains(material, domains) or not _accept_material(material, seen_keys):
continue
materials.append(material)
return materials
@property
def is_configured(self) -> bool:
"""Whether at least one material channel can run."""
return bool(self.material_channels)
@property
def material_channels(self) -> list[str]:
"""Advertise active Q&A-compatible retrieval channels to the frontend."""
channels: list[str] = []
if any(
_SKILL_REQUIRED_TOOLS.get(skill.name, set()) <= self._question_answer_tool_names
for skill in self._research_skills
):
channels.append("skill")
if self._settings is not None or self._online_search_available:
channels.append("web_search")
return channels
@property
def settings_error(self) -> str | None:
return self._settings_error
def result_to_material(item: dict[str, Any], *, query: str) -> ResearchMaterial:
"""Map a configurable-search result item into a research material.
Public so the chat-collection thread harvester can reuse the exact same
field mapping for ``web_search`` tool envelopes harvested off a thread.
"""
rec_uuid = str(item.get("recUuid") or "").strip() or None
url = str(item.get("url") or "").strip() or None
title = str(item.get("title") or "").strip()
content = str(item.get("content") or "").strip()
source = str(item.get("source") or "").strip() or "web"
raw_hash = hashlib.sha256((url or rec_uuid or content).encode("utf-8")).hexdigest()[:32]
published_at = _try_parse_datetime(str(item.get("publish_time") or item.get("time") or "").strip())
return ResearchMaterial(
id=f"src_{raw_hash[:16]}",
query=query,
title=title,
url=url,
raw_content=content,
snippet=content[:200] if content else None,
source=source,
source_type="web",
published_at=published_at,
rec_uuid=rec_uuid,
content_hash=raw_hash,
)
def _search_result_to_material(result: Any, *, query: str, channel: str) -> ResearchMaterial | None:
"""Map a Q&A/AI-writing SearchResult-like object into ResearchMaterial."""
title = str(getattr(result, "title", "") or "").strip()
content = str(getattr(result, "content", "") or "").strip()
if not title and not content:
return None
url = str(getattr(result, "url", "") or "").strip() or None
rec_uuid = str(getattr(result, "recUuid", "") or "").strip() or None
source = str(getattr(result, "source", "") or "").strip() or channel
source_type = str(getattr(result, "source_type", "") or "").strip()
if source_type not in {"web", "knowledge", "skill", "mcp", "file", "other"}:
source_type = "skill" if channel == "skill" else "web"
raw_hash = hashlib.sha256((url or rec_uuid or title or content).encode("utf-8")).hexdigest()[:32]
return ResearchMaterial(
id=f"src_{raw_hash[:16]}",
query=query,
title=title,
url=url,
raw_content=content,
snippet=content[:200] if content else None,
source=source,
source_type=source_type,
published_at=_try_parse_datetime(str(getattr(result, "published_at", "") or "").strip()),
relevance_score=_safe_float(getattr(result, "relevance_score", None)),
rec_uuid=rec_uuid,
content_hash=raw_hash,
metadata={"retrieval_channel": channel},
)
def _load_question_answer_tool_names(model_name: str | None) -> set[str]:
"""Return exactly the tool names the Q&A agent can currently call."""
try:
from deerflow.tools import get_available_tools
return {str(tool.name) for tool in get_available_tools(model_name=model_name, include_mcp=True)}
except Exception as exc: # noqa: BLE001
logger.warning("Deep Research failed to inspect Q&A tools: %s", exc)
return set()
def _load_question_answer_research_skills() -> list[Any]:
"""Load only enabled Q&A research skills, preserving normal skill state."""
try:
from deerflow.skills import get_or_new_skill_storage
by_name = {skill.name: skill for skill in get_or_new_skill_storage().load_skills(enabled_only=True)}
return [by_name[name] for name in QUESTION_ANSWER_RESEARCH_SKILLS if name in by_name]
except Exception as exc: # noqa: BLE001
logger.warning("Deep Research failed to load Q&A research skills: %s", exc)
return []
def _load_named_research_skills(names: list[str]) -> list[Any]:
"""Load enabled skills by name, preserving the requested order."""
if not names:
return []
try:
from deerflow.skills import get_or_new_skill_storage
by_name = {skill.name: skill for skill in get_or_new_skill_storage().load_skills(enabled_only=True)}
return [by_name[name] for name in names if name in by_name]
except Exception as exc: # noqa: BLE001
logger.warning("Deep Research failed to load named retrieval skills: %s", exc)
return []
def _normalize_named_skills(value: list[str] | None) -> list[str]:
from deerflow.persistence.report_structures.skills import normalize_retrieval_skills
return normalize_retrieval_skills(value)
def _material_matches_domains(material: ResearchMaterial, domains: list[str] | None) -> bool:
"""Apply the requested domain allow-list without dropping URL-less KB hits."""
if not domains or not material.url:
return True
host = (urlparse(material.url).hostname or "").lower()
allowed = {str(domain).lower().strip().lstrip(".") for domain in domains if str(domain).strip()}
return not allowed or any(host == domain or host.endswith(f".{domain}") for domain in allowed)
def _accept_material(material: ResearchMaterial, seen_keys: set[str]) -> bool:
"""Deduplicate cross-channel hits before optional fetch backfill."""
key = material.url or material.rec_uuid or material.content_hash
if not key or key in seen_keys:
return False
seen_keys.add(key)
return True
def _safe_float(value: Any) -> float | None:
try:
return float(value) if value is not None else None
except (TypeError, ValueError):
return None
def _filter_relevant_search_results(results: list[Any], query: str) -> list[Any]:
"""Discard obvious search-engine noise before it reaches report citations."""
tokens = _query_relevance_tokens(query)
if not tokens:
return list(results)
kept: list[Any] = []
for result in results:
haystack = " ".join(
[
str(getattr(result, "title", "") or ""),
str(getattr(result, "content", "") or ""),
str(getattr(result, "url", "") or ""),
]
).lower()
if any(token in haystack for token in tokens):
kept.append(result)
return kept
def _query_relevance_tokens(query: str) -> set[str]:
"""Build compact English/CJK tokens suitable for rejecting blank noise."""
tokens = {token.lower() for token in re.findall(r"[A-Za-z][A-Za-z0-9_-]{2,}", query)}
for chunk in re.findall(r"[\u4e00-\u9fff]+", query):
for width in (4, 3, 2):
tokens.update(chunk[index : index + width] for index in range(max(0, len(chunk) - width + 1)))
return tokens
def _search_360_web(query: str, limit: int) -> list[Any]:
"""Return parsed, real 360 Search snippets when DDG returns no relevant hit."""
try:
import httpx
from bs4 import BeautifulSoup
from deerflow.agents.ai_writing.search.base import SearchResult
response = httpx.get(
"https://www.so.com/s",
params={"q": query},
headers={"User-Agent": "Mozilla/5.0 (compatible; DeerFlow/1.0)"},
timeout=SECONDARY_ONLINE_SEARCH_TIMEOUT_SECONDS,
follow_redirects=True,
)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
rows: list[Any] = []
seen_urls: set[str] = set()
for card in soup.select("li.res-list"):
anchor = card.select_one("h3 a") or card.select_one("a.res-link")
if anchor is None:
continue
title = anchor.get_text(" ", strip=True)
url = str(anchor.get("href") or "").strip()
content_node = card.select_one(".res-desc") or card.select_one(".res-rich") or card
content = content_node.get_text(" ", strip=True)
if not title or not content or not url or url in seen_urls:
continue
seen_urls.add(url)
rows.append(
SearchResult(
title=title,
content=content[:500],
source=urlparse(url).hostname or "360 Search",
published_at="",
url=url,
relevance_score=0.7,
source_type="internet",
)
)
if len(rows) >= limit:
break
return _filter_relevant_search_results(rows, query)[:limit]
except Exception as exc: # noqa: BLE001
logger.warning("Deep Research 360 fallback search failed for %r: %s", query, exc)
return []
def _try_parse_datetime(value: str) -> datetime | None:
for fmt in ("%Y-%m-%d %H:%M:%S", "%Y-%m-%d", "%Y/%m/%d", "%Y-%m-%dT%H:%M:%S"):
try:
return datetime.strptime(value[:19], fmt)
except ValueError:
continue
return None
def _safe_fetch(url: str) -> str:
"""Best-effort full-text fetch via the configured DeerFlow web_fetch tool."""
try:
from deerflow.config import get_app_config
fetch_cfg = get_app_config().get_tool_config("web_fetch")
if fetch_cfg is None or not fetch_cfg.enabled:
return ""
use = fetch_cfg.use or ""
html = ""
if "jina_ai" in use:
from deerflow.community.jina_ai.jina_client import JinaClient
html = JinaClient().crawl(url, return_format="html", timeout=10) or ""
elif "infoquest" in use:
from deerflow.community.infoquest.infoquest_client import InfoQuestClient
html = InfoQuestClient().fetch(url, return_format="html") or ""
if not html or html.startswith("Error"):
return ""
from deerflow.utils.readability import ReadabilityExtractor
return ReadabilityExtractor().extract_article(html).to_markdown(including_title=True) or ""
except Exception as exc: # noqa: BLE001
logger.debug("Deep Research material fetch failed for %s: %s", url, exc)
return ""
__all__ = [
"DeerFlowMaterialProvider",
"ONLINE_SEARCH_TIMEOUT_SECONDS",
"QUESTION_ANSWER_RESEARCH_SKILLS",
"SNIPPET_THRESHOLD",
"result_to_material",
]
@@ -0,0 +1,135 @@
"""Cooperative cancellation token for deep-research jobs.
The vendored GPT Researcher loops (recursive deep research, sub-topic fan-out,
multi-agent review cycles) can run for minutes. Every adapter and runner polls
a single :class:`CancellationToken` so a user cancel (or a lease takeover)
promptly unwinds the whole tree instead of burning model budget to completion.
Design:
- ``CancellationToken`` is an asyncio-friendly flag — ``cancel()`` flips it,
``raise_if_cancelled()`` is the checked-call sites sprinkle at loop boundaries.
- A token is created **per job execution** by the executor and threaded into the
``AdapterBundle``; it is never a process global.
- The executor's cancellation watcher calls ``cancel()`` when it observes
``cancel_requested`` in the DB; the running ``asyncio.Task`` sees
``CancelledError`` propagate up.
"""
from __future__ import annotations
import asyncio
class CancellationToken:
"""A lightweight cooperative cancel flag.
Thread-safe in the asyncio sense (single event loop): ``cancel()`` and
``is_cancelled`` only touch the flag / an asyncio event.
"""
__slots__ = ("_cancelled", "_event", "_reason")
def __init__(self) -> None:
self._cancelled: bool = False
self._reason: str = ""
# Used by ``wait()``; lazily created on first await so we don't need a
# running loop at construction time (tokens are often built in sync code).
self._event: asyncio.Event | None = None
# ── state queries ───────────────────────────────────────────────────────
@property
def is_cancelled(self) -> bool:
return self._cancelled
@property
def reason(self) -> str:
return self._reason
# ── transitions ─────────────────────────────────────────────────────────
def cancel(self, reason: str = "cancelled") -> None:
"""Flip the flag and wake any coroutine blocked on ``wait()``."""
self._cancelled = True
self._reason = reason
if self._event is not None:
self._event.set()
# ── checked-call helpers ────────────────────────────────────────────────
def raise_if_cancelled(self) -> None:
"""Raise :class:`ResearchCancelled` if the token has been cancelled.
Sprinkle at the top of every loop iteration in vendored / runner code:
search fan-out, context compression, report chunking, graph nodes.
"""
if self._cancelled:
raise ResearchCancelled(self._reason)
async def wait(self, timeout: float | None = None) -> None:
"""Block until cancelled (or ``timeout`` elapses).
Useful when a node needs to *park* until either cancellation or an
external resume signal — though for human-review interrupts the
LangGraph ``interrupt()`` mechanism is preferred.
"""
if self._cancelled:
return
if self._event is None:
self._event = asyncio.Event()
try:
await asyncio.wait_for(self._event.wait(), timeout=timeout)
except TimeoutError:
pass
# ── context-manager sugar (check on enter/exit of a scoped unit) ────────
def __enter__(self) -> CancellationToken:
self.raise_if_cancelled()
return self
def __exit__(self, *exc: object) -> None:
self.raise_if_cancelled()
class ResearchCancelled(Exception):
"""Raised when a :class:`CancellationToken` is flipped mid-run.
Deliberately **not** a subclass of ``asyncio.CancelledError``: the latter is
used by the executor to kill the task object; ``ResearchCancelled`` is the
*checked* variant that lets adapters/runners unwind cleanly and record a
``cancelled`` terminal state rather than an error.
"""
def __init__(self, reason: str = "cancelled") -> None:
super().__init__(reason)
self.reason = reason
class ResearchAwaitingInput(Exception):
"""Signal a durable human-review pause to the job executor.
This is intentionally separate from cancellation: the runner has completed
a safe checkpoint and the executor must preserve its plan, release the
worker lease, and let a later ``resume`` job continue from that checkpoint.
"""
def __init__(self, interrupt_id: str, plan: dict) -> None:
super().__init__(f"Research is awaiting input at {interrupt_id}")
self.interrupt_id = interrupt_id
self.plan = plan
def raise_if_cancelled_or_return(value: CancellationToken | None) -> None:
"""Convenience for adapters that receive an optional token."""
if value is not None:
value.raise_if_cancelled()
__all__ = [
"CancellationToken",
"ResearchAwaitingInput",
"ResearchCancelled",
"raise_if_cancelled_or_return",
]
@@ -0,0 +1 @@
"""Chat collection package — harvest materials from a collector-agent thread."""
@@ -0,0 +1,311 @@
"""Harvest research materials from a collector-agent chat thread.
The chat collection flow replaces the runner-driven search tiers with a real
collector-agent conversation on the session's runtime LangGraph thread (same
pipeline as normal chat — skills, sandbox tools, repair middlewares all work
with weak models). When the user confirms report writing, the writing job
reads that thread's checkpoint and converts the tool results the agent already
produced into :class:`ResearchMaterial` objects:
- ``web_search`` tool envelopes reuse the exact configured-search field
mapping (``result_to_material``) so citations stay identical to the legacy
channel.
- other tool outputs (knowledge/skill/MCP/bash stdout, including
middle-truncated JSON) are normalized through ``deerflow.runtime.references``
— the same robust harvester the chat reference batches use.
- the agent's final collection summary becomes a fallback material so a weak
model that finished without usable tool output still yields a report.
The harvested pool is served to the runners through
:class:`StaticHarvestMaterialProvider` — runners and their dedupe/compress/
write stages stay untouched.
"""
from __future__ import annotations
import hashlib
import logging
from datetime import datetime
from typing import Any
from deerflow.agents.deep_research.adapters.material import dedupe_materials
from deerflow.agents.deep_research.adapters.material_provider import result_to_material
from deerflow.agents.deep_research.types import ResearchMaterial
from deerflow.runtime.references import (
_extract_tool_payload,
_first_list,
_message_content,
_message_name,
_message_type,
_normalize_reference_item,
)
logger = logging.getLogger(__name__)
# Upper bound on harvested materials so a chatty collector thread cannot blow
# up the report context; the curating/compressing stages trim further anyway.
MAX_HARVESTED_MATERIALS = 80
# A collector model may time out after tool calls were checkpointed but before
# its final assistant response advanced the thread head. Keep the scan bounded
# while retaining the richest checkpoint from that short history.
MAX_CHECKPOINT_HISTORY = 32
# Tool-name → ResearchMaterial.source_type. Everything unknown maps to
# ``other``; ``bash`` is treated as ``skill`` because skill scripts emit there.
_TOOL_SOURCE_TYPES: dict[str, str] = {
"web_search": "web",
"weknora_search": "knowledge",
"bash": "skill",
"read_file": "file",
"write_file": "file",
}
# Tool results that never carry report materials.
_IGNORED_TOOLS = {"ask_clarification", "write_todos", "deep_research_report"}
async def harvest_thread_materials(
*,
checkpointer: Any,
runtime_thread_id: str,
query: str,
) -> tuple[list[ResearchMaterial], str]:
"""Read the collector thread's checkpointed messages into materials.
Returns ``(materials, agent_summary)`` — the summary is the agent's last
visible text (its collection小结) and doubles as a fallback material when
the thread produced no harvestable tool results.
"""
config = {"configurable": {"thread_id": runtime_thread_id, "checkpoint_ns": ""}}
checkpoint_tuple = await checkpointer.aget_tuple(config)
if checkpoint_tuple is None:
logger.warning("deep-research harvest: no checkpoint for thread %s", runtime_thread_id)
return [], ""
best_materials, best_summary = _harvest_checkpoint_tuple(checkpoint_tuple, query=query)
# The latest checkpoint is the normal, fast path. Historical traversal is
# only a recovery for a stale/empty head after an interrupted collector
# stream. Scanning it after usable material was already found adds dozens
# of checkpoint reads and can make report writing appear to stop halfway
# on deployments whose checkpoint store is slow or isolated per worker.
if best_materials:
return best_materials, best_summary
# ``aget_tuple`` normally returns the latest state. When a streamed lead
# agent call fails, however, some checkpointers leave that head at the
# final input checkpoint even though an earlier checkpoint already holds
# the successful web-search tool results. With an empty head, prefer the
# richest available material pool rather than treating it as an empty run.
checkpoint_history = getattr(checkpointer, "alist", None)
if callable(checkpoint_history):
try:
history = checkpoint_history(config, limit=MAX_CHECKPOINT_HISTORY)
if hasattr(history, "__aiter__"):
async for candidate in history:
materials, summary = _harvest_checkpoint_tuple(candidate, query=query)
if len(materials) > len(best_materials):
best_materials, best_summary = materials, summary
else:
for candidate in history:
materials, summary = _harvest_checkpoint_tuple(candidate, query=query)
if len(materials) > len(best_materials):
best_materials, best_summary = materials, summary
except Exception: # noqa: BLE001 - newest checkpoint remains a safe fallback
logger.debug("deep-research harvest: checkpoint history unavailable for %s", runtime_thread_id, exc_info=True)
return best_materials, best_summary
def _harvest_checkpoint_tuple(checkpoint_tuple: Any, *, query: str) -> tuple[list[ResearchMaterial], str]:
checkpoint = getattr(checkpoint_tuple, "checkpoint", None)
if not isinstance(checkpoint, dict):
return [], ""
values = checkpoint.get("channel_values", {}) or {}
messages = values.get("messages") or []
return harvest_messages_materials(messages, query=query)
def harvest_messages_materials(
messages: list[Any],
*,
query: str,
) -> tuple[list[ResearchMaterial], str]:
"""Convert an in-memory message list into research materials.
Same conversion as :func:`harvest_thread_materials` minus the checkpoint
read — for callers that already hold the messages (e.g. a tool reading
``runtime.state`` inside the same run).
Returns ``(materials, agent_summary)`` — the summary is the agent's last
visible text (its collection小结) and doubles as a fallback material when
the thread produced no harvestable tool results.
"""
materials = _harvest_tool_messages(messages, query=query)
agent_summary = _last_assistant_text(messages)
if not materials and agent_summary.strip():
# Weak-model fallback: the agent narrated its findings but never called
# a harvestable tool — its summary is still real collected knowledge.
materials.append(
ResearchMaterial(
id=f"src_{hashlib.sha256(agent_summary.encode('utf-8')).hexdigest()[:16]}",
query=query,
title="收集员小结",
url=None,
raw_content=agent_summary,
snippet=agent_summary[:200],
source="collector",
source_type="other",
content_hash=hashlib.sha256(agent_summary.encode("utf-8")).hexdigest()[:32],
metadata={"retrieval_channel": "chat_collector_summary"},
)
)
return dedupe_materials(materials)[:MAX_HARVESTED_MATERIALS], agent_summary
def _harvest_tool_messages(messages: list[Any], *, query: str) -> list[ResearchMaterial]:
"""Convert every harvestable tool result in the thread into materials.
每条材料在 ``metadata["message_index"]`` 记录其来源工具消息在对话中的
位置——写作管线据此区分「课题请求之后检索的(属于当前课题)」与
「课题切换前的旧检索」(后者才做词面相关度过滤)。
"""
materials: list[ResearchMaterial] = []
for message_index, message in enumerate(messages):
if _message_type(message) != "tool":
continue
tool_name = str(_message_name(message) or "")
if tool_name in _IGNORED_TOOLS:
continue
payload = _extract_tool_payload(message)
if payload is None:
continue
if tool_name == "web_search" and isinstance(payload, dict):
# Highest-fidelity path: identical field mapping to the legacy
# configured-search channel (recUuid/url/content/publish_time).
envelope_query = str(payload.get("query") or "").strip() or query
for item in payload.get("results") or []:
if isinstance(item, dict):
material = result_to_material(item, query=envelope_query)
material.metadata["retrieval_channel"] = "chat_web_search"
material.metadata["message_index"] = message_index
materials.append(material)
continue
materials.extend(_harvest_generic_payload(payload, tool_name=tool_name, query=query, message_index=message_index))
return materials
def _harvest_generic_payload(
payload: Any,
*,
tool_name: str,
query: str,
message_index: int | None = None,
) -> list[ResearchMaterial]:
"""Normalize a non-web_search tool payload (skill/knowledge/bash/MCP)."""
source_type = _TOOL_SOURCE_TYPES.get(tool_name, "other")
materials: list[ResearchMaterial] = []
for index, raw_item in enumerate(_first_list(payload), 1):
normalized = _normalize_reference_item(raw_item, index, payload=payload, tool_context=None)
if not normalized:
continue
title = str(normalized.get("title") or "").strip()
content = str(normalized.get("content") or normalized.get("snippet") or "").strip()
url = str(normalized.get("url") or "").strip() or None
if not title and not content:
continue
raw_hash = hashlib.sha256((url or title or content).encode("utf-8")).hexdigest()[:32]
materials.append(
ResearchMaterial(
id=f"src_{raw_hash[:16]}",
query=query,
title=title or f"结果 {index}",
url=url,
raw_content=content,
snippet=content[:200] or None,
source=str(normalized.get("source") or tool_name).strip() or tool_name,
source_type=source_type, # type: ignore[arg-type]
published_at=_try_parse_datetime(str(normalized.get("time") or "")),
content_hash=raw_hash,
metadata={
"retrieval_channel": "chat_tool",
"tool_name": tool_name,
**({"message_index": message_index} if message_index is not None else {}),
},
)
)
return materials
def _last_assistant_text(messages: list[Any]) -> str:
"""The collector's last visible answer (its collection summary)."""
for message in reversed(messages):
if _message_type(message) != "ai":
continue
content = _message_content(message)
text = _content_to_text(content)
if text.strip():
return text.strip()
return ""
def _content_to_text(content: Any) -> str:
if isinstance(content, str):
return content
if isinstance(content, list):
parts: list[str] = []
for part in content:
if isinstance(part, str):
parts.append(part)
elif isinstance(part, dict) and isinstance(part.get("text"), str):
parts.append(part["text"])
return "".join(parts)
return ""
def _try_parse_datetime(value: str) -> datetime | None:
for fmt in ("%Y-%m-%d %H:%M:%S", "%Y-%m-%d", "%Y/%m/%d", "%Y-%m-%dT%H:%M:%S"):
try:
return datetime.strptime(value[:19], fmt)
except ValueError:
continue
return None
class StaticHarvestMaterialProvider:
"""MaterialProvider serving the pre-harvested collector-thread pool.
Runners call ``search`` per planned sub-query; for a chat-collected pool
every call returns the whole harvest (the runners' dedupe + compress
stages trim it). ``limit`` therefore does not truncate — it bounds
network fan-out, which this provider never performs.
"""
def __init__(self, materials: list[ResearchMaterial]) -> None:
self._materials = list(materials)
@property
def material_count(self) -> int:
return len(self._materials)
async def search(
self,
query: str,
*,
domains: list[str] | None = None,
limit: int = 8,
user_id: str = "",
session_id: str = "",
cancellation=None,
) -> list[ResearchMaterial]:
if cancellation is not None:
cancellation.raise_if_cancelled()
return list(self._materials)
__all__ = [
"MAX_CHECKPOINT_HISTORY",
"MAX_HARVESTED_MATERIALS",
"StaticHarvestMaterialProvider",
"harvest_messages_materials",
"harvest_thread_materials",
]
@@ -0,0 +1,230 @@
"""Deep Research configuration — frozen per-task snapshot.
Every deep-research job captures the *full* ``DeepResearchConfig`` as an immutable
``config_snapshot`` at creation time; the running job reads **only** that snapshot
and never consults mutable global config mid-run (prevents mid-flight setting
changes from corrupting a long research session).
Server-side clamps (``clamp()``) bound every numeric field regardless of what the
client sent — client-side limits are convenience only.
"""
from __future__ import annotations
from typing import Any, Literal
from pydantic import BaseModel, Field, field_validator
# ── enums (kept as plain Literal unions so the model is JSON-serialisable) ──
ResearchMode = Literal["quick", "basic", "detailed", "deep", "multi_agent"]
ResearchTone = Literal["objective", "formal", "analytical", "persuasive"]
ReportFormat = Literal["markdown", "apa", "mla", "chicago"]
ModelRole = Literal["fast", "smart", "strategic"]
MaterialChannel = Literal["web_search", "knowledge", "skill", "mcp", "file", "other"]
SourceType = Literal["web", "knowledge", "skill", "mcp", "file", "other"]
# How the collecting stage runs. ``chat`` (the default) means the frontend
# drives a real collector-agent conversation on the session's runtime thread
# and the writing job harvests materials from that thread; ``legacy`` keeps the
# runner-driven skill/web tiers searching on their own.
CollectionMode = Literal["chat", "legacy"]
# How the writer treats ``custom_outline``.
# ``fixed`` (固定大纲) reproduces the stored headings and hierarchy.
# ``adaptive`` (弹性大纲) keeps the current material-driven layout.
StructureMode = Literal["fixed", "adaptive"]
# ── server-side bounds (§4.3 / §19.4) ────────────────────────────────────────
# Centralised so the router, the executor, and tests all share one truth source.
_MAX_SEARCH_RESULTS_PER_QUERY = 20
_MAX_ITERATIONS = 10
_MAX_SUBTOPICS = 20
_MAX_DEEP_BREADTH = 8
_MAX_DEEP_DEPTH = 4
_MAX_CONCURRENCY = 8
_MAX_CONTEXT_WORDS = 120_000
_MAX_REVISION_LOOPS = 5
MAX_GENERATED_IMAGES = 4
_MAX_CUSTOM_OUTLINE_CHARS = 4_000
_MAX_REPORT_INSTRUCTION_CHARS = 4_000
_MAX_CONTROL_PLANE_TIMEOUT_SECONDS = 120.0
# Per-user active job ceiling (§19.4 / §26).
DEFAULT_MAX_ACTIVE_JOBS_PER_USER = 3
class DeepResearchConfig(BaseModel):
"""Per-task research configuration.
Field names mirror the upstream GPT Researcher knobs but use DeerFlow
conventions (``snake_case``, explicit clamps, no ``os.environ`` reads).
"""
mode: ResearchMode = "basic"
collection_mode: CollectionMode = "chat"
language: str = "zh-CN"
tone: ResearchTone = "objective"
report_format: ReportFormat = "markdown"
# Model role overrides; ``None`` → fall back to the system default model.
fast_model: str | None = None
smart_model: str | None = None
strategic_model: str | None = None
thinking_enabled: bool = False
# Research budget knobs (client may request, server clamps).
max_search_results_per_query: int = Field(default=8, ge=1)
max_iterations: int = Field(default=3, ge=1)
max_subtopics: int = Field(default=8, ge=1)
deep_breadth: int = Field(default=4, ge=1)
deep_depth: int = Field(default=2, ge=1)
max_concurrency: int = Field(default=3, ge=1)
max_context_words: int = Field(default=24_000, ge=500)
# A provider may retain a hung HTTP request for several minutes. Planning,
# query generation, and review calls must fail in a bounded time so the
# durable job can either fall back to the topic or finish with a real error.
# Report prose deliberately uses ``stream_complete`` and is not subject to
# this total-duration cap.
control_plane_timeout_seconds: float = Field(default=45.0, ge=0.1)
# Editorial / quality knobs.
curate_sources: bool = True
generate_images: bool = False
max_generated_images: int = Field(default=2, ge=1)
include_human_feedback: bool = False
max_revision_loops: int = Field(default=2, ge=0)
# Material channels — only admin-configured tool groups are accepted
# (router validates against capabilities; ``other`` reserved for future).
allowed_material_channels: list[MaterialChannel] = Field(default_factory=lambda: ["web_search"])
allowed_domains: list[str] = Field(default_factory=list)
# Optional user-supplied outline (sanitised; never overrides system safety
# prompts in the vendored writer).
custom_outline: str | None = Field(default=None, max_length=_MAX_CUSTOM_OUTLINE_CHARS)
# Optional write-time requirements confirmed by the user. This is kept
# separate from ``custom_outline`` so prose/style requirements can never be
# misread as extra chapter headings by detailed runners.
report_instruction: str | None = Field(default=None, max_length=_MAX_REPORT_INSTRUCTION_CHARS)
# Writing constraint copied from the selected report structure (结构约束).
# Default ``adaptive`` preserves today's material-driven layout.
structure_mode: StructureMode = "adaptive"
# Optional search facets copied from the selected report structure (检索方向).
# Empty → unconstrained query-planning. Non-empty → split retrievable terms
# along those facets; never treat the labels themselves as search queries.
retrieval_directions: list[str] = Field(default_factory=list)
# Optional skill names copied from the selected report structure (检索来源).
# Empty → collector / material provider keep today's default sources.
retrieval_skills: list[str] = Field(default_factory=list)
model_config = {"extra": "forbid"}
# ── server-side clamping ────────────────────────────────────────────────
def clamp(self) -> DeepResearchConfig:
"""Return a copy with every numeric field bounded to its server ceiling.
Call this at job-creation time so the persisted snapshot is already safe;
the executor can then trust ``config_snapshot`` unconditionally.
"""
return self.model_copy(
update={
"max_search_results_per_query": min(
self.max_search_results_per_query, _MAX_SEARCH_RESULTS_PER_QUERY
),
"max_iterations": min(self.max_iterations, _MAX_ITERATIONS),
"max_subtopics": min(self.max_subtopics, _MAX_SUBTOPICS),
"deep_breadth": min(self.deep_breadth, _MAX_DEEP_BREADTH),
"deep_depth": min(self.deep_depth, _MAX_DEEP_DEPTH),
"max_concurrency": min(self.max_concurrency, _MAX_CONCURRENCY),
"max_context_words": min(self.max_context_words, _MAX_CONTEXT_WORDS),
"control_plane_timeout_seconds": min(
self.control_plane_timeout_seconds, _MAX_CONTROL_PLANE_TIMEOUT_SECONDS
),
"max_revision_loops": min(self.max_revision_loops, _MAX_REVISION_LOOPS),
"max_generated_images": min(self.max_generated_images, MAX_GENERATED_IMAGES),
}
)
@field_validator("allowed_material_channels")
@classmethod
def _dedupe_channels(cls, v: list[MaterialChannel]) -> list[MaterialChannel]:
# Preserve order, drop duplicates.
seen: set[str] = set()
out: list[MaterialChannel] = []
for ch in v:
if ch not in seen:
seen.add(ch)
out.append(ch)
return out
@field_validator("custom_outline", mode="before")
@classmethod
def _sanitize_custom_outline(cls, value: Any) -> str | None:
"""Keep a bounded user outline as content, never as a system prompt.
Newlines are retained because Markdown/numbered headings are useful to
report planning; NUL/control separators are removed so the snapshot is
portable across the database, events and artifact JSON.
"""
if value is None:
return None
cleaned = str(value).replace("\x00", "").replace("\r", "").strip()
return cleaned or None
@field_validator("report_instruction", mode="before")
@classmethod
def _sanitize_report_instruction(cls, value: Any) -> str | None:
if value is None:
return None
cleaned = str(value).replace("\x00", "").replace("\r", "").strip()
return cleaned or None
@field_validator("structure_mode", mode="before")
@classmethod
def _normalize_structure_mode(cls, value: Any) -> StructureMode:
raw = str(value or "").strip().lower()
if raw in {"fixed", "canonical", "prescriptive", "rigid"}:
return "fixed"
return "adaptive"
@field_validator("retrieval_directions", mode="before")
@classmethod
def _normalize_retrieval_directions(cls, value: Any) -> list[str]:
from deerflow.persistence.report_structures.directions import normalize_retrieval_directions
return normalize_retrieval_directions(value)
@field_validator("retrieval_skills", mode="before")
@classmethod
def _normalize_retrieval_skills(cls, value: Any) -> list[str]:
from deerflow.persistence.report_structures.skills import normalize_retrieval_skills
return normalize_retrieval_skills(value)
def resolve_model_name(self, role: ModelRole) -> str | None:
"""Return the configured model name for a role, or ``None`` for system default."""
return {
"fast": self.fast_model,
"smart": self.smart_model,
"strategic": self.strategic_model,
}[role]
# Convenience: the full set of supported modes, for capability discovery.
ALL_MODES: tuple[ResearchMode, ...] = (
"quick",
"basic",
"detailed",
"deep",
"multi_agent",
)
def config_from_dict(data: dict[str, Any]) -> DeepResearchConfig:
"""Build + clamp a config from a raw dict (e.g. JSON column / request body)."""
return DeepResearchConfig.model_validate(data).clamp()
@@ -0,0 +1,90 @@
"""DeepResearchEngine — mode-dispatching entry point (§12.1).
The engine selects the appropriate runner for the request's ``mode`` and hands
it the assembled :class:`AdapterBundle`. It is a thin dispatcher: all research
logic lives in the runners; all infrastructure wiring lives in the adapter
construction (done by the executor before calling ``run``).
``basic``/``quick`` use the compact path. ``detailed`` and ``deep`` use their
dedicated runners; ``multi_agent`` runs the LangGraph editor/researcher/writer/
reviewer workflow with durable plan-review pause and resume.
"""
from __future__ import annotations
import logging
from typing import TYPE_CHECKING
from deerflow.agents.deep_research.runners.basic import BasicRunner
from deerflow.agents.deep_research.runners.deep import DeepRunner
from deerflow.agents.deep_research.runners.detailed import DetailedRunner
from deerflow.agents.deep_research.runners.multi_agent import MultiAgentRunner
from deerflow.agents.deep_research.types import DeepResearchRequest, DeepResearchResult
if TYPE_CHECKING:
from deerflow.agents.deep_research.cancellation import CancellationToken
from deerflow.agents.deep_research.types import AdapterBundle
logger = logging.getLogger(__name__)
class DeepResearchEngine:
"""Dispatches a research request to the mode-appropriate runner."""
def __init__(self) -> None:
self._basic = BasicRunner()
self._detailed = DetailedRunner()
self._deep = DeepRunner()
self._multi_agent = MultiAgentRunner()
async def run(
self,
request: DeepResearchRequest,
*,
adapters: AdapterBundle,
cancellation: CancellationToken | None = None,
) -> DeepResearchResult:
mode = request.config.mode
# Chat/regeneration entries use a static material provider. First chat
# writes select exactly one channel in order (browser snapshot,
# durable rows, checkpoint); regeneration reads its cloned durable
# source pool directly. Once the selected path is empty, that is
# authoritative:
# bypass every planning/search/curation branch and enter the supported
# no-material report path immediately, regardless of requested mode.
material_count = getattr(adapters.materials, "material_count", None)
if material_count == 0:
return await self._basic._no_materials_result(
request,
adapters,
{
"queries": [request.query],
"collection": "no_material_fallback",
"requestedMode": mode,
},
cancellation,
collection_outcomes=[
{
"query": request.query,
"status": "completed",
"resultCount": 0,
"route": "no_material_fallback",
}
],
)
if mode in ("basic", "quick"):
return await self._basic.run(request, adapters, cancellation)
if mode == "detailed":
return await self._detailed.run(request, adapters, cancellation)
if mode == "deep":
return await self._deep.run(request, adapters, cancellation)
if mode == "multi_agent":
return await self._multi_agent.run(request, adapters, cancellation)
# Phase 1 scope guard — these modes are planned for later phases.
raise NotImplementedError(
f"Research mode '{mode}' is not implemented. "
"Available modes: quick, basic, detailed, deep, multi_agent."
)
__all__ = ["DeepResearchEngine"]
@@ -0,0 +1,176 @@
"""Deep Research event protocol.
Every observable thing that happens during a research job — a search query fired,
a source added, a report chunk written, a model invocation billed — becomes an
event. Events are **persisted first** (to ``deep_research_events``) then pushed
to the SSE stream; this ordering lets a client reconnect with ``?after=<seq>``
and replay anything missed (§15.3).
This module defines:
- :data:`EventType` — the closed set of event ``type`` strings (§15.2).
- :data:`ResearchPhase` — the phase enum (§14.3).
- :class:`DeepResearchEvent` — the envelope persisted + serialised over SSE.
- :class:`EventSink` — the adapter protocol runners call.
The concrete :class:`PersistingEventSink` (adapter) lives in
``adapters/event_sink.py`` and bridges to the event-store repository.
"""
from __future__ import annotations
from datetime import UTC, datetime
from typing import Any, Literal, Protocol, runtime_checkable
from pydantic import BaseModel, Field
# ── phase enum (§14.3) ───────────────────────────────────────────────────────
ResearchPhase = Literal[
"initializing",
"planning",
"collecting",
"curating",
"compressing",
"deepening",
"writing",
"summarizing",
"reviewing",
"illustrating",
"exporting",
"done",
]
ALL_PHASES: tuple[ResearchPhase, ...] = (
"initializing",
"planning",
"collecting",
"curating",
"compressing",
"deepening",
"writing",
"summarizing",
"reviewing",
"illustrating",
"exporting",
"done",
)
# ── event type enum (§15.2) ──────────────────────────────────────────────────
#
# Kept as a Literal union (not an enum.IntEnum) so the persisted string and the
# SSE JSON ``type`` field are identical and stable across versions.
EventType = Literal[
"job_status",
# Terminal events are kept separate from job_status for the historical
# event timeline. The executor persists these after the DB state has
# been committed, so a reconnecting browser receives an explicit end.
"job_completed",
"job_failed",
"phase_changed",
"plan_created",
"queries_planned",
"query_started",
"query_completed",
"source_added",
"sources_curated",
"context_compressed",
"deep_branch_started",
"deep_branch_completed",
"report_reset",
"report_chunk",
# Model reasoning normally travels as a live-only frame. It becomes a
# durable event when no SSE connection for the job is attached to the
# worker running it, because the in-process live hub cannot reach a
# viewer served by a different worker (see PersistingEventSink).
"report_thinking",
"plan_reasoning",
"summary_thinking",
"report_completed",
"summary_completed",
"collection_completed",
"artifact_created",
"image_generation_started",
"image_generated",
"awaiting_input",
"model_usage",
# Durable whole-document rewrite milestones reuse the deep-research job
# event log, but have their own frontend SSE payload contract.
"document_rewrite",
"warning",
"error",
"heartbeat",
]
class DeepResearchEvent(BaseModel):
"""A single observable event in a research job.
``seq`` is assigned by the event store (monotonic per job); callers leave it
``0`` when emitting and read it back from the persisted row.
"""
seq: int = 0
session_id: str = ""
job_id: str = ""
type: EventType
phase: ResearchPhase = "initializing"
timestamp: datetime = Field(default_factory=lambda: datetime.now(UTC))
payload: dict[str, Any] = Field(default_factory=dict)
model_config = {"extra": "allow"}
def to_sse_dict(self) -> dict[str, Any]:
"""Serialise for the ``data:`` line of the SSE stream (§15.1)."""
return {
"seq": self.seq,
"sessionId": self.session_id,
"jobId": self.job_id,
"type": self.type,
"phase": self.phase,
"timestamp": self.timestamp.isoformat() if self.timestamp else None,
"payload": self.payload,
}
@runtime_checkable
class EventSink(Protocol):
"""Adapter protocol for emitting research events.
Implementations persist the event (assigning ``seq``) before returning, so a
crash after ``emit()`` never loses an already-observed event.
"""
async def emit(
self,
type: EventType, # noqa: A002 — intentional arg name for readability
*,
phase: ResearchPhase = "initializing",
payload: dict[str, Any] | None = None,
) -> DeepResearchEvent:
"""Persist + return an event. ``session_id``/``job_id`` are bound by the sink."""
...
async def publish_live(
self,
type: str, # noqa: A002 - intentionally outside the durable EventType union
*,
phase: ResearchPhase = "initializing",
payload: dict[str, Any] | None = None,
) -> None:
"""Publish a best-effort, non-persisted frame.
Provider text deltas use this path. Bounded ``report_chunk`` events
remain the durable recovery checkpoints.
"""
...
__all__ = [
"ALL_PHASES",
"DeepResearchEvent",
"EventSink",
"EventType",
"ResearchPhase",
]
@@ -0,0 +1,138 @@
"""Optional retrieval directions and skills attached to a report structure.
检索方向 (``retrieval_directions``): when listed, material collection **splits
search terms along those facets** instead of treating the labels as queries.
检索来源 (``retrieval_skills``): when listed, collection calls those skills
instead of the collector's default web_search / Q&A research-skill stack.
Empty lists are no-ops: runners and the chat collector keep today's behaviour.
"""
from __future__ import annotations
from typing import Any
from deerflow.persistence.report_structures.directions import (
MAX_DIRECTION_CHARS,
MAX_RETRIEVAL_DIRECTIONS,
normalize_retrieval_directions,
)
from deerflow.persistence.report_structures.skills import (
MAX_RETRIEVAL_SKILLS,
MAX_SKILL_NAME_CHARS,
normalize_retrieval_skills,
)
# Must match the built-in collector agent id (seed + frontend). Extra skills
# from a report structure are only merged onto this agent so other custom
# agents cannot widen their whitelist via run context.
COLLECTOR_AGENT_ID = "deep-research-collector"
# Keep in sync with frontend ``COLLECTOR_DIRECTION_INTRO``.
_COLLECTOR_DIRECTION_INTRO = (
"请按以下检索方向拆分检索词(这是拆词约束,不是检索词本身):"
"每个方向拆成若干具体、可检索的检索词(含课题中的专名、时间、地点、同义/中英对照等),"
"不要直接把方向名称拿去搜索——方向标签往往搜不到结果。"
"覆盖列出的每一个方向;未列出的方面不必主动扩展。"
"某个检索词没有结果时,换更具体或更短的词再试,仍不要改用未列出的方向。"
)
_COLLECTOR_SKILL_INTRO = (
"请严格按照以下技能收集素材:先用 read_file 读取每个技能的 SKILL.md,"
"再按其说明调用工具检索。不要改用未列出的技能,也不要自行改成纯 web_search"
"(除非技能说明要求联网)。每个技能至少使用一轮。"
)
__all__ = [
"COLLECTOR_AGENT_ID",
"MAX_DIRECTION_CHARS",
"MAX_RETRIEVAL_DIRECTIONS",
"MAX_RETRIEVAL_SKILLS",
"MAX_SKILL_NAME_CHARS",
"format_collector_direction_block",
"format_collector_skill_block",
"format_direction_lines",
"merge_collector_skill_allowlist",
"normalize_retrieval_directions",
"normalize_retrieval_skills",
"queries_from_retrieval_directions",
"runtime_extra_skills",
]
def format_direction_lines(directions: Any) -> str:
"""Numbered direction list, or ``""`` when empty."""
dirs = normalize_retrieval_directions(directions)
return "\n".join(f"{i}. {item}" for i, item in enumerate(dirs, 1))
def queries_from_retrieval_directions(topic: str, directions: Any) -> list[str]:
"""Last-resort search strings when directed query-planning fails.
Prefer LLM/collector splitting: direction labels concatenated with the
topic are often not retrievable. Empty directions → ``[]``.
"""
topic_text = " ".join((topic or "").split()).strip()
dirs = normalize_retrieval_directions(directions)
if not dirs:
return []
seen: set[str] = set()
queries: list[str] = []
for direction in dirs:
if topic_text and topic_text not in direction:
query = f"{topic_text} {direction}".strip()
else:
query = direction
key = query.casefold()
if not query or key in seen:
continue
seen.add(key)
queries.append(query)
return queries
def format_collector_direction_block(directions: Any) -> str:
"""XML block appended to the collector's first user message, or ``""``."""
lines = format_direction_lines(directions)
if not lines:
return ""
return f"<retrieval-directions>\n{_COLLECTOR_DIRECTION_INTRO}\n{lines}\n</retrieval-directions>"
def format_collector_skill_block(skills: Any) -> str:
"""XML block appended to the collector's first user message, or ``""``."""
names = normalize_retrieval_skills(skills)
if not names:
return ""
lines = "\n".join(f"{i}. {item}" for i, item in enumerate(names, 1))
return f"<retrieval-skills>\n{_COLLECTOR_SKILL_INTRO}\n{lines}\n</retrieval-skills>"
def runtime_extra_skills(cfg: Any) -> list[str]:
"""Read extra skill names from a LangGraph run config / context dict."""
if not isinstance(cfg, dict):
return []
return normalize_retrieval_skills(cfg.get("extra_skills") or cfg.get("retrieval_skills"))
def merge_collector_skill_allowlist(
agent_id: str | None,
base: list[str] | None,
extra: Any,
) -> list[str] | None:
"""Widen the collector agent's skill allowlist for one run.
Other agents are unchanged. Empty extras are a no-op. ``None`` base means
"all skills" (default lead agent) and is left alone.
"""
extras = normalize_retrieval_skills(extra)
if not extras or agent_id != COLLECTOR_AGENT_ID:
return list(base) if base is not None else None
seen: set[str] = set()
out: list[str] = []
for name in [*(base or []), *extras]:
if not name or name in seen:
continue
seen.add(name)
out.append(name)
return out
@@ -0,0 +1,19 @@
"""Deep Research runners — one per research mode.
Each runner maps a :class:`~deerflow.agents.deep_research.types.DeepResearchRequest`
to a concrete research pipeline using the injected :class:`AdapterBundle`.
Runners never touch the network, filesystem, or ``app.*`` directly — they call
only the adapters.
The detailed and deep runners use the same adapter boundary as the basic
runner, so they do not add a model/search runtime dependency.
"""
from __future__ import annotations
from deerflow.agents.deep_research.runners.basic import BasicRunner
from deerflow.agents.deep_research.runners.deep import DeepRunner
from deerflow.agents.deep_research.runners.detailed import DetailedRunner
from deerflow.agents.deep_research.runners.multi_agent import MultiAgentRunner
__all__ = ["BasicRunner", "DeepRunner", "DetailedRunner", "MultiAgentRunner"]
@@ -0,0 +1,462 @@
"""Recursive deep-research runner adapted from GPT Researcher's DeepResearchSkill.
The original skill expands search goals by breadth, extracts learnings from each
branch, and recursively investigates follow-up questions with a smaller width.
This module keeps those mechanics but routes all search/model work through the
DeerFlow adapter bundle. A deterministic query budget is reserved before any
branch runs, so recursive fan-out cannot exceed the persisted job configuration.
"""
from __future__ import annotations
import asyncio
import json
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any
from pydantic import BaseModel, Field
from deerflow.agents.deep_research.adapters.material import dedupe_materials
from deerflow.agents.deep_research.cancellation import ResearchCancelled
from deerflow.agents.deep_research.retrieval import (
format_direction_lines,
normalize_retrieval_directions,
queries_from_retrieval_directions,
)
from deerflow.agents.deep_research.runners.basic import (
_SYSTEM,
BasicRunner,
_format_context,
ensure_reference_section,
)
from deerflow.agents.deep_research.runners.images import generate_report_images
from deerflow.agents.deep_research.types import DeepResearchRequest, DeepResearchResult, ResearchMaterial
if TYPE_CHECKING:
from deerflow.agents.deep_research.types import AdapterBundle
class ResearchBranch(BaseModel):
query: str
research_goal: str = ""
class ResearchBranches(BaseModel):
branches: list[ResearchBranch] = Field(default_factory=list)
class ResearchLearnings(BaseModel):
learnings: list[str] = Field(default_factory=list)
follow_up_questions: list[str] = Field(default_factory=list)
_BRANCH_PROMPT = """Research topic: {query}
Generate at most {breadth} independent search branches. Each branch must include
a concrete search query and a one-sentence research goal. Return JSON only:
{{"branches": [{{"query": "...", "research_goal": "..."}}]}}.
"""
_DIRECTED_BRANCH_PROMPT = """Research topic: {query}
The user named these retrieval directions. Use them only as facets for splitting
search terms — do not use the direction labels themselves as queries (they often
return no results):
{directions}
Generate at most {breadth} independent search branches covering those directions.
Each branch must include a concrete, retrievable search query (entities, dates,
places, synonyms) and a one-sentence research goal. Do not invent branches
outside the listed directions. Return JSON only:
{{"branches": [{{"query": "...", "research_goal": "..."}}]}}.
"""
_LEARNING_PROMPT = """Research objective: {goal}
Extract concise, evidence-grounded learnings from the materials. Then propose
follow-up questions that would reduce the most important uncertainty. Return
JSON only: {{"learnings": ["..."], "follow_up_questions": ["..."]}}.
Materials:
{context}
"""
def calculate_query_budget(breadth: int, depth: int) -> int:
"""Return the maximum number of searches reachable by recursive expansion.
A depth level creates ``width`` branches per existing branch. Each next
level halves the width (with a floor of one), mirroring GPT Researcher's
``new_breadth=breadth // 2`` expansion while remaining bounded for
DeerFlow's small default budgets.
"""
width = max(1, breadth)
nodes = 1
total = 0
for _ in range(max(1, depth)):
total += nodes * width
nodes *= width
width = max(1, width // 2)
return total
@dataclass
class _Budget:
remaining: int
lock: asyncio.Lock
async def take(self) -> bool:
async with self.lock:
if self.remaining <= 0:
return False
self.remaining -= 1
return True
class DeepRunner(BasicRunner):
"""Run bounded recursive research and synthesize a final cited report."""
async def run(
self,
request: DeepResearchRequest,
adapters: AdapterBundle,
cancellation=None,
) -> DeepResearchResult:
config = request.config
events = adapters.events
all_materials: list[ResearchMaterial] = []
learnings: list[str] = []
materials_lock = asyncio.Lock()
learnings_lock = asyncio.Lock()
budget = _Budget(calculate_query_budget(config.deep_breadth, config.deep_depth), asyncio.Lock())
model_semaphore = asyncio.Semaphore(max(1, config.max_concurrency))
try:
await events.emit(
"phase_changed",
phase="planning",
payload={"from": "initializing", "to": "planning"},
)
self._raise_if_cancelled(cancellation)
roots = await self._generate_branches(
adapters.llm,
request.query,
config.deep_breadth,
retrieval_directions=config.retrieval_directions,
)
if not roots:
roots = [ResearchBranch(query=request.query, research_goal=request.query)]
plan = {
"branches": [branch.model_dump() for branch in roots],
"deepBreadth": config.deep_breadth,
"deepDepth": config.deep_depth,
"queryBudget": budget.remaining,
}
await events.emit("plan_created", phase="planning", payload={"plan": plan, "subtopics": []})
await events.emit(
"phase_changed",
phase="deepening",
payload={"from": "planning", "to": "deepening"},
)
await asyncio.gather(
*[
self._research_branch(
branch=branch,
depth_remaining=config.deep_depth,
branch_width=config.deep_breadth,
path=str(index + 1),
request=request,
adapters=adapters,
cancellation=cancellation,
model_semaphore=model_semaphore,
budget=budget,
all_materials=all_materials,
materials_lock=materials_lock,
learnings=learnings,
learnings_lock=learnings_lock,
)
for index, branch in enumerate(roots)
]
)
self._raise_if_cancelled(cancellation)
await events.emit(
"phase_changed",
phase="curating",
payload={"from": "deepening", "to": "curating"},
)
deduped = dedupe_materials(all_materials)
if not deduped:
return await self._no_materials_result(request, adapters, plan)
selected_ids = [material.id for material in deduped]
await events.emit(
"sources_curated",
phase="curating",
payload={"selectedIds": selected_ids, "removedIds": [], "reason": "deduped_recursive"},
)
await events.emit(
"phase_changed",
phase="compressing",
payload={"from": "curating", "to": "compressing"},
)
adapters.context.set_materials(deduped)
context_list = await adapters.context.async_get_context(
request.query,
max_results=min(len(deduped), config.max_search_results_per_query * 2),
)
if not context_list:
context_list = [
material.to_context_dict()
for material in deduped[: min(len(deduped), config.max_search_results_per_query * 2)]
]
await events.emit(
"warning",
phase="compressing",
payload={
"code": "context_fallback",
"message": "材料压缩未返回上下文,已直接使用已收集素材撰写报告",
"recoverable": True,
"sourceCount": len(deduped),
"diagnostic": {
"stage": "compressing",
"outcome": "empty_context",
"sourceCount": len(deduped),
},
},
)
material_context = _format_context(context_list, deduped)
learning_context = "\n".join(f"- {learning}" for learning in dict.fromkeys(learnings))
context_text = f"## Recursive research learnings\n{learning_context}\n\n## Source materials\n{material_context}" if learning_context else material_context
await events.emit(
"context_compressed",
phase="compressing",
payload={"sourceCount": len(context_list), "learningCount": len(learnings)},
)
await events.emit(
"phase_changed",
phase="writing",
payload={"from": "compressing", "to": "writing"},
)
self._raise_if_cancelled(cancellation)
report_md = await self._generate_report(
adapters.llm,
request,
context_text,
events,
cancellation,
source_count=len(deduped),
)
report_md, image_paths = await generate_report_images(request, adapters, report_md, cancellation)
# Ensure before _write_artifacts so the artifact AND the returned
# report_markdown (persisted by the executor) both carry it.
report_md = ensure_reference_section(report_md, context_list, deduped)
await self._write_artifacts(adapters, report_md, deduped, plan)
await events.emit(
"report_completed",
phase="writing",
payload={"reportVersion": 1, "sourceCount": len(deduped)},
)
await events.emit(
"artifact_created",
phase="exporting",
payload={"format": "markdown", "name": "report.md"},
)
return DeepResearchResult(
report_markdown=report_md,
source_ids=selected_ids,
artifact_paths={
"report.md": "report.md",
"sources.json": "sources.json",
**image_paths,
},
usage=adapters.llm.usage_totals,
plan=plan,
)
except ResearchCancelled:
await events.emit(
"warning",
phase="deepening",
payload={"code": "cancelled", "message": "Research job cancelled", "recoverable": False},
)
raise
async def _research_branch(
self,
*,
branch: ResearchBranch,
depth_remaining: int,
branch_width: int,
path: str,
request: DeepResearchRequest,
adapters: AdapterBundle,
cancellation,
model_semaphore: asyncio.Semaphore,
budget: _Budget,
all_materials: list[ResearchMaterial],
materials_lock: asyncio.Lock,
learnings: list[str],
learnings_lock: asyncio.Lock,
) -> None:
self._raise_if_cancelled(cancellation)
if not await budget.take():
await adapters.events.emit(
"warning",
phase="deepening",
payload={"code": "deep_budget_exhausted", "message": "Recursive query budget exhausted", "recoverable": True},
)
return
await adapters.events.emit(
"deep_branch_started",
phase="deepening",
payload={"path": path, "query": branch.query, "goal": branch.research_goal, "depthRemaining": depth_remaining},
)
try:
materials = await self._search_one(
adapters.materials,
branch.query,
request.config,
request,
cancellation,
adapters.events,
)
except ResearchCancelled:
raise
except Exception as exc: # noqa: BLE001 - one branch must not abort all research
await adapters.events.emit(
"warning",
phase="deepening",
payload={"code": "deep_branch_search_failed", "message": str(exc), "recoverable": True},
)
materials = []
async with materials_lock:
all_materials.extend(materials)
branch_learnings: list[str] = []
follow_ups: list[str] = []
if materials:
deduped_materials = dedupe_materials(materials)
context = _format_context(
[material.to_context_dict() for material in deduped_materials],
deduped_materials,
)
async with model_semaphore:
extraction = await self._extract_learnings(adapters.llm, branch.research_goal or branch.query, context)
branch_learnings = extraction.learnings
follow_ups = extraction.follow_up_questions
async with learnings_lock:
learnings.extend(branch_learnings)
await adapters.events.emit(
"deep_branch_completed",
phase="deepening",
payload={
"path": path,
"query": branch.query,
"resultCount": len(materials),
"learningCount": len(branch_learnings),
"nextQuestionCount": len(follow_ups),
},
)
self._raise_if_cancelled(cancellation)
if depth_remaining <= 1 or not follow_ups:
return
next_width = max(1, branch_width // 2)
await asyncio.gather(
*[
self._research_branch(
branch=ResearchBranch(query=question, research_goal=question),
depth_remaining=depth_remaining - 1,
branch_width=next_width,
path=f"{path}.{index + 1}",
request=request,
adapters=adapters,
cancellation=cancellation,
model_semaphore=model_semaphore,
budget=budget,
all_materials=all_materials,
materials_lock=materials_lock,
learnings=learnings,
learnings_lock=learnings_lock,
)
for index, question in enumerate(follow_ups[:next_width])
if question.strip()
]
)
async def _generate_branches(
self,
llm,
query: str,
breadth: int,
retrieval_directions: list[str] | None = None,
) -> list[ResearchBranch]:
dirs = normalize_retrieval_directions(retrieval_directions)
prompt = (
_DIRECTED_BRANCH_PROMPT.format(
query=query, breadth=breadth, directions=format_direction_lines(dirs)
)
if dirs
else _BRANCH_PROMPT.format(query=query, breadth=breadth)
)
result = await llm.complete(
model_role="strategic",
messages=[
{"role": "system", "content": _SYSTEM},
{"role": "user", "content": prompt},
],
response_schema=ResearchBranches,
operation="generate_deep_queries",
)
branches = result.structured.branches if result.structured else []
seen: set[str] = set()
output: list[ResearchBranch] = []
for branch in branches:
query_text = branch.query.strip()
if not query_text or query_text in seen:
continue
seen.add(query_text)
output.append(ResearchBranch(query=query_text, research_goal=branch.research_goal.strip()))
if len(output) >= breadth:
break
if output:
return output
fallback = queries_from_retrieval_directions(query, dirs)
return [ResearchBranch(query=item, research_goal=item) for item in fallback[:breadth]]
async def _extract_learnings(self, llm, goal: str, context: str) -> ResearchLearnings:
result = await llm.complete(
model_role="fast",
messages=[
{"role": "system", "content": _SYSTEM},
{"role": "user", "content": _LEARNING_PROMPT.format(goal=goal, context=context)},
],
response_schema=ResearchLearnings,
operation="extract_research_learnings",
)
return result.structured if result.structured else ResearchLearnings()
async def _write_artifacts(
self,
adapters: AdapterBundle,
report_md: str,
materials: list[ResearchMaterial],
plan: dict[str, Any],
) -> None:
await adapters.artifacts.write_text("report.md", report_md)
await adapters.artifacts.write_text("sources.json", json.dumps([material.to_context_dict() for material in materials], ensure_ascii=False, indent=2))
await adapters.artifacts.write_text("research-plan.json", json.dumps(plan, ensure_ascii=False, indent=2))
async def _no_materials_result(self, request: DeepResearchRequest, adapters: AdapterBundle, plan: dict[str, Any]) -> DeepResearchResult:
return await super()._no_materials_result(request, adapters, plan)
@staticmethod
def _raise_if_cancelled(cancellation) -> None:
if cancellation:
cancellation.raise_if_cancelled()
__all__ = ["DeepRunner", "ResearchBranch", "ResearchBranches", "ResearchLearnings", "calculate_query_budget"]
@@ -0,0 +1,502 @@
"""Detailed-report runner adapted from GPT Researcher's subtopic workflow.
The upstream ``DetailedReport`` flow plans a set of subtopics, researches each
one, writes independently usable sections, then assembles an introduction and
conclusion. This adaptation deliberately retains that control flow while using
only DeerFlow's injected adapters for models, materials, events, and artifacts.
"""
from __future__ import annotations
import asyncio
import json
import logging
from typing import TYPE_CHECKING, Any
from pydantic import BaseModel, Field
from deerflow.agents.deep_research.adapters.material import dedupe_materials
from deerflow.agents.deep_research.cancellation import ResearchCancelled
from deerflow.agents.deep_research.runners.basic import (
_SYSTEM,
BasicRunner,
_format_context,
ensure_outline_preamble,
ensure_reference_section,
extract_outline_sections,
)
from deerflow.agents.deep_research.runners.images import generate_report_images
from deerflow.agents.deep_research.types import DeepResearchRequest, DeepResearchResult, ResearchMaterial
if TYPE_CHECKING:
from deerflow.agents.deep_research.types import AdapterBundle
logger = logging.getLogger(__name__)
class DetailedSubtopics(BaseModel):
"""The bounded section plan for a detailed report."""
subtopics: list[str] = Field(default_factory=list)
_SUBTOPIC_PROMPT = """Research topic: {query}
Create at most {count} mutually exclusive report sections. Cover the most
decision-useful dimensions of the topic and make each section title concise.
Return JSON only: {{"subtopics": ["section one", "section two"]}}.
"""
_SECTION_PROMPT = """Write the Markdown section titled "{topic}" for a report on
"{query}". Use only the supplied evidence, distinguish facts from inference,
and cite evidence with [N]. Output the section body only; do not repeat a #
title.
Evidence:
{context}
"""
_INTRODUCTION_PROMPT = """Write a concise Markdown introduction for a detailed
research report about "{query}". State scope and method without inventing facts.
The report sections are: {sections}. Output prose only.
"""
_CONCLUSION_PROMPT = """Write a concise Markdown conclusion for a detailed
research report about "{query}". Synthesize only the supplied section findings;
note important uncertainty where appropriate. Output prose only.
Section findings:
{sections}
"""
def _confirmed_requirements(value: str | None) -> str:
if not value:
return ""
return f"\n\nConfirmed user generation requirements:\n<user-report-requirements>\n{value}\n</user-report-requirements>"
class DetailedRunner(BasicRunner):
"""Plan, research, and write bounded subtopic sections in parallel."""
async def run(
self,
request: DeepResearchRequest,
adapters: AdapterBundle,
cancellation=None,
) -> DeepResearchResult:
config = request.config
events = adapters.events
try:
await events.emit(
"phase_changed",
phase="planning",
payload={"from": "initializing", "to": "planning"},
)
self._raise_if_cancelled(cancellation)
subtopics = extract_outline_sections(
config.custom_outline,
config.max_subtopics,
structure_mode=config.structure_mode,
)
if not subtopics:
subtopics = await self._generate_subtopics(adapters.llm, request.query, config.max_subtopics, events)
if not subtopics:
subtopics = [request.query]
plan = {"subtopics": subtopics, "queries": []}
await events.emit(
"plan_created",
phase="planning",
payload={"plan": plan, "subtopics": subtopics},
)
await events.emit(
"phase_changed",
phase="collecting",
payload={"from": "planning", "to": "collecting"},
)
semaphore = asyncio.Semaphore(max(1, config.max_concurrency))
topic_results = await asyncio.gather(
*[
self._research_subtopic(
topic,
request,
adapters,
semaphore,
cancellation,
)
for topic in subtopics
],
return_exceptions=True,
)
self._raise_if_cancelled(cancellation)
materials_by_topic: dict[str, list[ResearchMaterial]] = {topic: [] for topic in subtopics}
all_materials: list[ResearchMaterial] = []
for result in topic_results:
if isinstance(result, ResearchCancelled):
raise result
if isinstance(result, Exception):
await events.emit(
"warning",
phase="collecting",
payload={"code": "subtopic_failed", "message": str(result), "recoverable": True},
)
continue
topic, materials = result
materials_by_topic[topic] = materials
all_materials.extend(materials)
await events.emit(
"phase_changed",
phase="curating",
payload={"from": "collecting", "to": "curating"},
)
self._raise_if_cancelled(cancellation)
deduped = dedupe_materials(all_materials)
if not deduped:
return await self._no_materials_result(request, adapters, plan)
selected_ids = [material.id for material in deduped]
await events.emit(
"sources_curated",
phase="curating",
payload={"selectedIds": selected_ids, "removedIds": [], "reason": "deduped"},
)
await events.emit(
"phase_changed",
phase="compressing",
payload={"from": "curating", "to": "compressing"},
)
adapters.context.set_materials(deduped)
global_context = await adapters.context.async_get_context(
request.query,
max_results=min(len(deduped), config.max_search_results_per_query * 2),
)
if not global_context:
global_context = [
material.to_context_dict()
for material in deduped[: min(len(deduped), config.max_search_results_per_query * 2)]
]
await events.emit(
"warning",
phase="compressing",
payload={
"code": "context_fallback",
"message": "材料压缩未返回上下文,已直接使用已收集素材撰写报告",
"recoverable": True,
"sourceCount": len(deduped),
"diagnostic": {
"stage": "compressing",
"outcome": "empty_context",
"sourceCount": len(deduped),
},
},
)
await events.emit(
"context_compressed",
phase="compressing",
payload={"sourceCount": len(global_context)},
)
await events.emit(
"phase_changed",
phase="writing",
payload={"from": "compressing", "to": "writing"},
)
self._raise_if_cancelled(cancellation)
if config.structure_mode == "fixed":
introduction = ""
writing_semaphore = asyncio.Semaphore(max(1, config.max_concurrency))
section_results = await asyncio.gather(
*[
self._write_section(
topic,
materials_by_topic.get(topic) or deduped,
request,
adapters,
writing_semaphore,
cancellation,
index,
)
for index, topic in enumerate(subtopics)
]
)
self._raise_if_cancelled(cancellation)
conclusion = ""
else:
introduction = await self._write_introduction(adapters.llm, request, adapters.events, subtopics)
writing_semaphore = asyncio.Semaphore(max(1, config.max_concurrency))
section_results = await asyncio.gather(
*[
self._write_section(
topic,
materials_by_topic.get(topic) or deduped,
request,
adapters,
writing_semaphore,
cancellation,
index,
)
for index, topic in enumerate(subtopics)
]
)
self._raise_if_cancelled(cancellation)
conclusion = await self._write_conclusion(adapters.llm, request, adapters.events, section_results)
report_md = self._assemble_report(
request.query,
introduction,
subtopics,
section_results,
conclusion,
outline=request.config.custom_outline,
structure_mode=request.config.structure_mode,
)
report_md, image_paths = await generate_report_images(request, adapters, report_md, cancellation)
# Sections cite with per-topic local numbering that cannot be
# mapped to the global list, so list every source instead of
# filtering by [N]. Ensured here so the artifact AND the returned
# report_markdown (persisted by the executor) both carry it.
report_md = ensure_reference_section(report_md, global_context, deduped, filter_by_citations=False)
await self._write_artifacts(adapters, report_md, deduped, plan)
await events.emit(
"report_completed",
phase="writing",
payload={"reportVersion": 1, "sourceCount": len(deduped)},
)
await events.emit(
"artifact_created",
phase="exporting",
payload={"format": "markdown", "name": "report.md"},
)
return DeepResearchResult(
report_markdown=report_md,
source_ids=selected_ids,
artifact_paths={
"report.md": "report.md",
"sources.json": "sources.json",
**image_paths,
},
usage=adapters.llm.usage_totals,
plan=plan,
)
except ResearchCancelled:
await events.emit(
"warning",
phase="writing",
payload={"code": "cancelled", "message": "Research job cancelled", "recoverable": False},
)
raise
async def _generate_subtopics(self, llm, query: str, max_subtopics: int, events) -> list[str]:
try:
result = await llm.complete(
model_role="strategic",
messages=[
{"role": "system", "content": _SYSTEM},
{"role": "user", "content": _SUBTOPIC_PROMPT.format(query=query, count=max_subtopics)},
],
response_schema=DetailedSubtopics,
operation="generate_detailed_subtopics",
)
except ResearchCancelled:
raise
except Exception:
logger.warning("detailed runner: subtopic planning failed; using original topic", exc_info=True)
await events.emit(
"warning",
phase="planning",
payload={
"code": "subtopic_planning_fallback",
"message": "研究范围规划超时或失败,已直接使用研究课题继续检索。",
"recoverable": True,
},
)
return []
raw = result.structured.subtopics if result.structured else []
seen: set[str] = set()
subtopics: list[str] = []
for item in raw:
title = item.strip()
if not title or title in seen:
continue
seen.add(title)
subtopics.append(title)
if len(subtopics) >= max_subtopics:
break
return subtopics
async def _research_subtopic(
self,
topic: str,
request: DeepResearchRequest,
adapters: AdapterBundle,
semaphore: asyncio.Semaphore,
cancellation,
) -> tuple[str, list[ResearchMaterial]]:
self._raise_if_cancelled(cancellation)
async with semaphore:
queries = await self._generate_queries(
adapters.llm,
topic,
request.config.max_iterations,
cancellation,
events=adapters.events,
phase="collecting",
retrieval_directions=request.config.retrieval_directions,
)
ordered_queries = list(dict.fromkeys([*queries, topic]))[: request.config.max_iterations + 1]
# The generated terms are useful user-facing progress, not internal
# reasoning. Persist them before dispatching the parallel searches so
# an SSE reconnect can faithfully replay the fan-out.
await adapters.events.emit(
"queries_planned",
phase="collecting",
payload={"parentTopic": topic, "queries": ordered_queries},
)
materials: list[ResearchMaterial] = []
for query in ordered_queries:
try:
materials.extend(
await self._search_one(
adapters.materials,
query,
request.config,
request,
cancellation,
adapters.events,
)
)
except ResearchCancelled:
raise
except Exception as exc: # noqa: BLE001 - one failed term must not abort the topic
logger.warning("detailed runner: subtopic search failed: %s", exc)
await adapters.events.emit(
"warning",
phase="collecting",
payload={"code": "search_failed", "message": str(exc), "recoverable": True},
)
return topic, materials
async def _write_introduction(self, llm, request: DeepResearchRequest, events, subtopics: list[str]) -> str:
from deerflow.agents.deep_research.runners.report_stream import ReportChunkEmitter
emitter = ReportChunkEmitter(events, target="introduction")
result = await llm.stream_complete(
model_role="smart",
messages=[
{"role": "system", "content": _SYSTEM},
{
"role": "user",
"content": _INTRODUCTION_PROMPT.format(query=request.query, sections=", ".join(subtopics))
+ _confirmed_requirements(request.config.report_instruction),
},
],
operation="write_detailed_introduction",
on_delta=emitter.add,
on_reasoning=emitter.add_thinking,
)
await emitter.flush()
return result.text
async def _write_section(
self,
topic: str,
materials: list[ResearchMaterial],
request: DeepResearchRequest,
adapters: AdapterBundle,
semaphore: asyncio.Semaphore,
cancellation,
index: int,
) -> str:
from deerflow.agents.deep_research.runners.report_stream import ReportChunkEmitter
async with semaphore:
self._raise_if_cancelled(cancellation)
deduped_materials = dedupe_materials(materials)
context = _format_context(
[material.to_context_dict() for material in deduped_materials],
deduped_materials,
)
emitter = ReportChunkEmitter(adapters.events, target=f"section:{index}")
result = await adapters.llm.stream_complete(
model_role="smart",
messages=[
{"role": "system", "content": _SYSTEM},
{
"role": "user",
"content": _SECTION_PROMPT.format(query=request.query, topic=topic, context=context)
+ _confirmed_requirements(request.config.report_instruction),
},
],
operation="write_detailed_section",
on_delta=emitter.add,
on_reasoning=emitter.add_thinking,
)
await emitter.flush()
return result.text
async def _write_conclusion(self, llm, request: DeepResearchRequest, events, sections: list[str]) -> str:
from deerflow.agents.deep_research.runners.report_stream import ReportChunkEmitter
emitter = ReportChunkEmitter(events, target="conclusion")
result = await llm.stream_complete(
model_role="smart",
messages=[
{"role": "system", "content": _SYSTEM},
{
"role": "user",
"content": _CONCLUSION_PROMPT.format(query=request.query, sections="\n\n".join(sections))
+ _confirmed_requirements(request.config.report_instruction),
},
],
operation="write_detailed_conclusion",
on_delta=emitter.add,
on_reasoning=emitter.add_thinking,
)
await emitter.flush()
return result.text
async def _write_artifacts(
self,
adapters: AdapterBundle,
report_md: str,
materials: list[ResearchMaterial],
plan: dict[str, Any],
) -> None:
await adapters.artifacts.write_text("report.md", report_md)
await adapters.artifacts.write_text("sources.json", json.dumps([material.to_context_dict() for material in materials], ensure_ascii=False, indent=2))
await adapters.artifacts.write_text("research-plan.json", json.dumps(plan, ensure_ascii=False, indent=2))
async def _no_materials_result(self, request: DeepResearchRequest, adapters: AdapterBundle, plan: dict[str, Any]) -> DeepResearchResult:
return await super()._no_materials_result(request, adapters, plan)
@staticmethod
def _assemble_report(
query: str,
introduction: str,
subtopics: list[str],
sections: list[str],
conclusion: str,
outline: str | None = None,
structure_mode: str = "adaptive",
) -> str:
section_md = "\n\n".join(f"## {topic}\n\n{body}" for topic, body in zip(subtopics, sections, strict=True))
if structure_mode == "fixed":
report = section_md + "\n"
else:
toc = "\n".join(f"- {topic}" for topic in subtopics)
report = f"# {query}\n\n## 摘要\n\n{introduction}\n\n## 目录\n\n{toc}\n\n{section_md}\n\n## 结论\n\n{conclusion}\n"
return ensure_outline_preamble(report, outline, query)
@staticmethod
def _raise_if_cancelled(cancellation) -> None:
if cancellation:
cancellation.raise_if_cancelled()
__all__ = ["DetailedRunner", "DetailedSubtopics"]
@@ -0,0 +1,180 @@
"""Shared optional report-illustration step for every research runner.
This module is intentionally independent of any particular runner. A basic,
detailed, recursive or multi-agent report receives the same bounded, best-effort
image behavior after its Markdown draft is complete and before ``report.md`` is
written. Image failures emit warnings and never discard an otherwise valid
research report.
"""
from __future__ import annotations
import re
from typing import TYPE_CHECKING
from deerflow.agents.deep_research.adapters.image import ImageGenerationError
if TYPE_CHECKING:
from deerflow.agents.deep_research.types import AdapterBundle, DeepResearchRequest
_HEADING_RE = re.compile(r"^##\s+(.+?)\s*$", re.MULTILINE)
_EXTENSIONS = {
"image/png": "png",
"image/jpeg": "jpg",
"image/webp": "webp",
}
async def generate_report_images(
request: DeepResearchRequest,
adapters: AdapterBundle,
report_markdown: str,
cancellation=None,
) -> tuple[str, dict[str, str]]:
"""Generate bounded section illustrations and inject protected references.
The generated Markdown uses a private ``deep-research://`` URI. The React
report renderer maps it to an authenticated session-artifact endpoint, so
image paths never expose the hidden runtime-thread id or an external model
provider URL.
"""
if not request.config.generate_images:
return report_markdown, {}
if adapters.images is None:
await adapters.events.emit(
"warning",
phase="illustrating",
payload={
"code": "image_provider_unavailable",
"message": "当前部署未配置图片生成服务,已跳过报告配图",
"recoverable": True,
},
)
return report_markdown, {}
sections = _report_sections(report_markdown, request.query, request.config.max_generated_images)
if not sections:
return report_markdown, {}
await adapters.events.emit(
"phase_changed",
phase="illustrating",
payload={"from": "writing", "to": "illustrating"},
)
image_paths: dict[str, str] = {}
amended_report = report_markdown
total = len(sections)
for position, section in enumerate(sections, 1):
if cancellation is not None:
cancellation.raise_if_cancelled()
prompt = _image_prompt(request.query, section)
await adapters.events.emit(
"image_generation_started",
phase="illustrating",
payload={"section": section, "position": position, "total": total},
)
try:
generated = await adapters.images.generate(
prompt,
section_id=section,
user_id=request.user_id,
)
if not generated:
raise ImageGenerationError("图片生成服务未返回图片")
data = generated.get("data")
if not isinstance(data, bytes) or not data:
raise ImageGenerationError("图片生成服务未返回可保存的图片数据")
mime_type = str(generated.get("mime_type") or "image/png").lower()
extension = _EXTENSIONS.get(mime_type, "png")
name = f"research-image-{position:02d}.{extension}"
await adapters.artifacts.write_bytes(name, data)
alt = _clean_alt(str(generated.get("alt") or f"{section}配图"))
amended_report = _insert_image(amended_report, section, alt, name)
image_paths[name] = name
await adapters.events.emit(
"image_generated",
phase="illustrating",
payload={
"section": section,
"position": position,
"total": total,
"name": name,
"alt": alt,
},
)
except ImageGenerationError as exc:
await adapters.events.emit(
"warning",
phase="illustrating",
payload={
"code": "image_generation_failed",
"message": str(exc),
"recoverable": True,
"section": section,
"position": position,
"total": total,
},
)
except Exception: # noqa: BLE001
# Third-party image APIs and artifact storage must never turn a
# completed text report into a failed job. Keep internal details
# out of the browser event stream.
await adapters.events.emit(
"warning",
phase="illustrating",
payload={
"code": "image_generation_failed",
"message": "报告配图生成失败,已继续保留文字报告",
"recoverable": True,
"section": section,
"position": position,
"total": total,
},
)
await adapters.events.emit(
"phase_changed",
phase="exporting",
payload={"from": "illustrating", "to": "exporting"},
)
return amended_report, image_paths
def _report_sections(report_markdown: str, query: str, maximum: int) -> list[str]:
seen: set[str] = set()
sections: list[str] = []
for match in _HEADING_RE.finditer(report_markdown):
section = _clean_heading(match.group(1))
if section and section not in seen:
seen.add(section)
sections.append(section)
if len(sections) >= maximum:
return sections
return sections or [_clean_heading(query) or "研究主题"]
def _image_prompt(query: str, section: str) -> str:
return (
f"为一份中文专业研究报告生成一张信息性配图。研究主题:{_clean_heading(query)}。对应章节:{section}。画面应当客观、简洁、适合商务报告;使用抽象示意、数据视觉化或场景插画表达主题。不要在图片中放置可读文字、品牌标志、水印或人物肖像。"
)
def _insert_image(report: str, section: str, alt: str, name: str) -> str:
marker = f"## {section}"
image = f"\n\n![{alt}](deep-research://{name})\n"
if marker in report:
return report.replace(marker, f"{marker}{image}", 1)
return f"{report.rstrip()}{image}"
def _clean_heading(value: str) -> str:
return " ".join(value.replace("\r", " ").replace("\n", " ").split())[:160]
def _clean_alt(value: str) -> str:
return _clean_heading(value).replace("[", "").replace("]", "") or "研究报告配图"
__all__ = ["generate_report_images"]
@@ -0,0 +1,449 @@
"""LangGraph 1.x multi-agent research runner with durable plan review.
The workflow adapts GPT Researcher's editor/researcher/writer/reviewer shape:
an editor plans sections, researchers collect evidence, a writer drafts the
report, and a reviewer can request a bounded revision loop. Human plan review
is durable at the runner/executor boundary: a first execution raises
``ResearchAwaitingInput`` after persisting its plan; a resumed job rebuilds the
same graph from the persisted plan and action payload.
"""
from __future__ import annotations
import asyncio
import json
import uuid
from typing import TYPE_CHECKING, Any, TypedDict
from langgraph.graph import END, START, StateGraph
from pydantic import BaseModel, Field
from deerflow.agents.deep_research.adapters.material import dedupe_materials
from deerflow.agents.deep_research.cancellation import ResearchAwaitingInput, ResearchCancelled
from deerflow.agents.deep_research.runners.basic import (
_SYSTEM,
BasicRunner,
_format_context,
ensure_reference_section,
extract_outline_sections,
)
from deerflow.agents.deep_research.runners.images import generate_report_images
from deerflow.agents.deep_research.types import DeepResearchRequest, DeepResearchResult, ResearchMaterial
if TYPE_CHECKING:
from deerflow.agents.deep_research.types import AdapterBundle
class MultiAgentPlan(BaseModel):
sections: list[str] = Field(default_factory=list)
class ReviewVerdict(BaseModel):
approved: bool = True
revision_notes: str = ""
class _MultiAgentState(TypedDict, total=False):
plan: dict[str, Any]
materials: list[ResearchMaterial]
report: str
review: dict[str, Any]
revision_count: int
context_list: list[dict[str, Any]]
_PLAN_PROMPT = """Research topic: {query}
Act as the research editor. Produce at most {count} focused sections for a
multi-agent report. Each section must describe a different decision-relevant
dimension. Return JSON only: {{"sections": ["...", "..."]}}.
"""
_REVIEW_PROMPT = """Review this research report for unsupported claims, missing
citations, and material contradictions. Approve it when it is publishable;
otherwise provide concise revision notes. Return JSON only:
{{"approved": true, "revision_notes": ""}}.
Report:
{report}
"""
_REVISE_PROMPT = """Revise the research report using the reviewer notes. Retain
valid citations, do not invent facts or sources, and return the full Markdown
report only.
Reviewer notes:
{notes}
Report:
{report}
"""
class MultiAgentRunner(BasicRunner):
"""Run a LangGraph editor/researcher/writer/reviewer workflow."""
async def run(
self,
request: DeepResearchRequest,
adapters: AdapterBundle,
cancellation=None,
) -> DeepResearchResult:
await adapters.events.emit(
"phase_changed",
phase="planning",
payload={"from": "initializing", "to": "planning"},
)
self._raise_if_cancelled(cancellation)
plan = await self._resolve_plan(request, adapters, cancellation)
if not request.interrupted_plan:
interrupt_id = f"plan-review-{uuid.uuid4().hex[:12]}"
await adapters.events.emit(
"plan_created",
phase="planning",
payload={"plan": plan, "subtopics": plan["sections"]},
)
await adapters.events.emit(
"awaiting_input",
phase="planning",
payload={
"interruptId": interrupt_id,
"kind": "plan_review",
"plan": plan,
"allowedActions": ["approve", "revise", "cancel"],
},
)
raise ResearchAwaitingInput(interrupt_id, plan)
graph = self._build_graph(request, adapters, cancellation)
initial_state: _MultiAgentState = {"plan": plan, "revision_count": 0}
state = await graph.ainvoke(initial_state)
materials = dedupe_materials(state.get("materials") or [])
report_md = str(state.get("report") or "")
if not materials or not report_md:
return await self._no_materials_result(request, adapters, plan)
report_md, image_paths = await generate_report_images(request, adapters, report_md, cancellation)
report_md = ensure_reference_section(report_md, state.get("context_list") or [], materials)
await adapters.artifacts.write_text("report.md", report_md)
await adapters.artifacts.write_text(
"sources.json",
json.dumps([material.to_context_dict() for material in materials], ensure_ascii=False, indent=2),
)
await adapters.artifacts.write_text("research-plan.json", json.dumps(plan, ensure_ascii=False, indent=2))
await adapters.events.emit(
"report_completed",
phase="writing",
payload={"reportVersion": 1, "sourceCount": len(materials)},
)
await adapters.events.emit(
"artifact_created",
phase="exporting",
payload={"format": "markdown", "name": "report.md"},
)
return DeepResearchResult(
report_markdown=report_md,
source_ids=[material.id for material in materials],
artifact_paths={
"report.md": "report.md",
"sources.json": "sources.json",
**image_paths,
},
usage=adapters.llm.usage_totals,
plan=plan,
)
async def _resolve_plan(self, request: DeepResearchRequest, adapters: AdapterBundle, cancellation) -> dict[str, Any]:
if request.interrupted_plan:
plan = dict(request.interrupted_plan)
action = str(request.resume_payload.get("action") or "approve")
if action == "cancel":
raise ResearchCancelled("plan review cancelled")
if action == "revise":
feedback = str((request.resume_payload.get("payload") or {}).get("feedback") or "")
plan = await self._revise_plan(adapters.llm, request.query, plan, feedback)
await adapters.events.emit(
"plan_created",
phase="planning",
payload={"plan": plan, "subtopics": plan.get("sections", []), "resumed": True},
)
return plan
outline_sections = extract_outline_sections(
request.config.custom_outline,
request.config.max_subtopics,
structure_mode=request.config.structure_mode,
)
if outline_sections:
return {"sections": outline_sections}
try:
result = await adapters.llm.complete(
model_role="strategic",
messages=[
{"role": "system", "content": _SYSTEM},
{
"role": "user",
"content": _PLAN_PROMPT.format(query=request.query, count=request.config.max_subtopics),
},
],
response_schema=MultiAgentPlan,
operation="generate_multi_agent_plan",
)
except ResearchCancelled:
raise
except Exception:
await adapters.events.emit(
"warning",
phase="planning",
payload={
"code": "multi_agent_plan_fallback",
"message": "研究计划生成超时或失败,已使用原始课题作为研究范围。",
"recoverable": True,
},
)
return {"sections": [request.query]}
sections = result.structured.sections if result.structured else []
normalised = self._normalise_sections(sections, request.config.max_subtopics)
return {"sections": normalised or [request.query]}
async def _revise_plan(self, llm, query: str, plan: dict[str, Any], feedback: str) -> dict[str, Any]:
prompt = f'Research topic: {query}\nExisting plan: {json.dumps(plan, ensure_ascii=False)}\nHuman feedback: {feedback}\nReturn JSON only: {{"sections": ["..."]}}.'
result = await llm.complete(
model_role="strategic",
messages=[{"role": "system", "content": _SYSTEM}, {"role": "user", "content": prompt}],
response_schema=MultiAgentPlan,
operation="revise_multi_agent_plan",
)
sections = result.structured.sections if result.structured else []
normalised = self._normalise_sections(sections, len(plan.get("sections") or []) or 1)
return {"sections": normalised or list(plan.get("sections") or [query])}
def _build_graph(self, request: DeepResearchRequest, adapters: AdapterBundle, cancellation):
model_semaphore = asyncio.Semaphore(max(1, request.config.max_concurrency))
async def research_node(state: _MultiAgentState) -> dict[str, Any]:
await adapters.events.emit(
"phase_changed",
phase="collecting",
payload={"from": "planning", "to": "collecting"},
)
sections = list(state["plan"].get("sections") or [])
results = await asyncio.gather(
*[self._research_section(section, request, adapters, cancellation, model_semaphore) for section in sections],
return_exceptions=True,
)
materials: list[ResearchMaterial] = []
for result in results:
if isinstance(result, ResearchCancelled):
raise result
if isinstance(result, Exception):
await adapters.events.emit(
"warning",
phase="collecting",
payload={"code": "multi_agent_researcher_failed", "message": str(result), "recoverable": True},
)
else:
materials.extend(result)
return {"materials": materials}
async def writer_node(state: _MultiAgentState) -> dict[str, Any]:
materials = dedupe_materials(state.get("materials") or [])
if not materials:
return {"report": ""}
await adapters.events.emit(
"phase_changed",
phase="curating",
payload={"from": "collecting", "to": "curating"},
)
await adapters.events.emit(
"sources_curated",
phase="curating",
payload={"selectedIds": [material.id for material in materials], "removedIds": [], "reason": "multi_agent_deduped"},
)
await adapters.events.emit(
"phase_changed",
phase="compressing",
payload={"from": "curating", "to": "compressing"},
)
adapters.context.set_materials(materials)
context_list = await adapters.context.async_get_context(
request.query,
max_results=min(len(materials), request.config.max_search_results_per_query * 2),
)
if not context_list:
context_list = [
material.to_context_dict()
for material in materials[: min(len(materials), request.config.max_search_results_per_query * 2)]
]
await adapters.events.emit(
"warning",
phase="compressing",
payload={
"code": "context_fallback",
"message": "材料压缩未返回上下文,已直接使用已收集素材撰写报告",
"recoverable": True,
"sourceCount": len(materials),
"diagnostic": {
"stage": "compressing",
"outcome": "empty_context",
"sourceCount": len(materials),
},
},
)
await adapters.events.emit(
"phase_changed",
phase="writing",
payload={"from": "compressing", "to": "writing"},
)
report = await self._generate_report(
adapters.llm,
request,
_format_context(context_list, materials),
adapters.events,
cancellation,
source_count=len(materials),
)
# Keep the numbered context around so the runner can append the
# deterministic reference section with the same [N] numbers the
# writer saw (revise_node rewrites the report but keeps citations).
return {"report": report, "context_list": context_list}
async def reviewer_node(state: _MultiAgentState) -> dict[str, Any]:
await adapters.events.emit(
"phase_changed",
phase="reviewing",
payload={"from": "writing", "to": "reviewing"},
)
async with model_semaphore:
result = await adapters.llm.complete(
model_role="smart",
messages=[
{"role": "system", "content": _SYSTEM},
{"role": "user", "content": _REVIEW_PROMPT.format(report=state.get("report") or "")},
],
response_schema=ReviewVerdict,
operation="review_multi_agent_report",
)
verdict = result.structured if result.structured else ReviewVerdict()
return {"review": verdict.model_dump()}
async def revise_node(state: _MultiAgentState) -> dict[str, Any]:
review = state.get("review") or {}
report_version = int(state.get("revision_count") or 0) + 2
await adapters.events.emit(
"report_reset",
phase="writing",
payload={"reportVersion": report_version, "reason": "review_revision"},
)
from deerflow.agents.deep_research.runners.report_stream import ReportChunkEmitter
emitter = ReportChunkEmitter(adapters.events, report_version=report_version)
async with model_semaphore:
result = await adapters.llm.stream_complete(
model_role="smart",
messages=[
{"role": "system", "content": _SYSTEM},
{
"role": "user",
"content": _REVISE_PROMPT.format(notes=review.get("revision_notes") or "", report=state.get("report") or ""),
},
],
operation="revise_multi_agent_report",
on_delta=emitter.add,
on_reasoning=emitter.add_thinking,
)
await emitter.flush()
return {"report": result.text, "revision_count": int(state.get("revision_count") or 0) + 1}
def route_review(state: _MultiAgentState) -> str:
review = state.get("review") or {}
if review.get("approved") or int(state.get("revision_count") or 0) >= request.config.max_revision_loops:
return "done"
return "revise"
workflow = StateGraph(_MultiAgentState)
workflow.add_node("researcher", research_node)
workflow.add_node("writer", writer_node)
workflow.add_node("reviewer", reviewer_node)
workflow.add_node("reviser", revise_node)
workflow.add_edge(START, "researcher")
workflow.add_edge("researcher", "writer")
workflow.add_edge("writer", "reviewer")
workflow.add_conditional_edges("reviewer", route_review, {"done": END, "revise": "reviser"})
workflow.add_edge("reviser", "reviewer")
return workflow.compile()
async def _research_section(
self,
section: str,
request: DeepResearchRequest,
adapters: AdapterBundle,
cancellation,
model_semaphore: asyncio.Semaphore,
) -> list[ResearchMaterial]:
self._raise_if_cancelled(cancellation)
async with model_semaphore:
queries = await self._generate_queries(
adapters.llm,
section,
request.config.max_iterations,
cancellation,
events=adapters.events,
phase="collecting",
retrieval_directions=request.config.retrieval_directions,
)
ordered_queries = list(dict.fromkeys([*queries, section]))[: request.config.max_iterations + 1]
await adapters.events.emit(
"queries_planned",
phase="collecting",
payload={"parentTopic": section, "queries": ordered_queries},
)
materials: list[ResearchMaterial] = []
for query in ordered_queries:
try:
materials.extend(
await self._search_one(
adapters.materials,
query,
request.config,
request,
cancellation,
adapters.events,
)
)
except ResearchCancelled:
raise
except Exception as exc: # noqa: BLE001 - one failed term must not abort the section
await adapters.events.emit(
"warning",
phase="collecting",
payload={"code": "search_failed", "message": str(exc), "recoverable": True},
)
return materials
async def _no_materials_result(self, request: DeepResearchRequest, adapters: AdapterBundle, plan: dict[str, Any]) -> DeepResearchResult:
return await super()._no_materials_result(request, adapters, plan)
@staticmethod
def _normalise_sections(sections: list[str], max_sections: int) -> list[str]:
seen: set[str] = set()
result: list[str] = []
for section in sections:
title = section.strip()
if not title or title in seen:
continue
seen.add(title)
result.append(title)
if len(result) >= max_sections:
break
return result
@staticmethod
def _raise_if_cancelled(cancellation) -> None:
if cancellation:
cancellation.raise_if_cancelled()
__all__ = ["MultiAgentPlan", "MultiAgentRunner", "ReviewVerdict"]
@@ -0,0 +1,101 @@
"""Small, durable report-delta emitter shared by research runners.
The event store intentionally persists every emitted event. Sending one event
for each provider token would therefore create excessive database writes and
make reconnect replay expensive. This helper keeps the first visible delta
immediate, then coalesces later deltas into bounded chunks.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from deerflow.agents.deep_research.events import EventSink, ResearchPhase
class ReportChunkEmitter:
"""Coalesce model deltas into replayable ``report_chunk`` events."""
def __init__(
self,
events: EventSink,
*,
target: str = "report",
report_version: int = 1,
phase: ResearchPhase = "writing",
chunk_size: int = 480,
) -> None:
self._events = events
self._target = target
self._report_version = report_version
self._phase = phase
self._chunk_size = chunk_size
self._buffer = ""
self._sent_first = False
self._next_offset = 0
self._persisted_offset = 0
async def add(self, delta: str) -> None:
"""Add a provider delta, making the first one visible immediately."""
if not delta:
return
# The sandbox UI receives the native model delta immediately. This is
# intentionally non-durable; ``report_chunk`` below remains the
# bounded checkpoint used for reconnect replay.
await self._events.publish_live(
"report_delta",
phase=self._phase,
payload={
"delta": delta,
"target": self._target,
"reportVersion": self._report_version,
"offset": self._next_offset,
},
)
self._next_offset += len(delta)
self._buffer += delta
if not self._sent_first:
first = self._buffer[: self._chunk_size]
await self._emit(first)
self._buffer = self._buffer[self._chunk_size :]
self._sent_first = True
while len(self._buffer) >= self._chunk_size:
await self._emit(self._buffer[: self._chunk_size])
self._buffer = self._buffer[self._chunk_size :]
async def add_thinking(self, delta: str) -> None:
"""Forward reasoning to the live UI without persisting it as report text."""
if not delta:
return
await self._events.publish_live(
"report_thinking",
phase=self._phase,
payload={
"delta": delta,
"target": self._target,
"reportVersion": self._report_version,
},
)
async def flush(self) -> None:
"""Persist the final tail once the model stream has finished."""
if self._buffer:
await self._emit(self._buffer)
self._buffer = ""
async def _emit(self, delta: str) -> None:
await self._events.emit(
"report_chunk",
phase=self._phase,
payload={
"delta": delta,
"target": self._target,
"reportVersion": self._report_version,
"offset": self._persisted_offset,
},
)
self._persisted_offset += len(delta)
__all__ = ["ReportChunkEmitter"]
@@ -0,0 +1,181 @@
"""Deep Research domain types.
These types are the **contract** between the DeerFlow adapters and the vendored
GPT Researcher algorithms:
- ``ResearchMaterial`` — the richer replacement for upstream's ``{url, raw_content}``
dict. Carries full DeerFlow provenance (title, source, ``rec_uuid``, publish time,
relevance) so report citations can always trace back to the originating record.
- ``DeepResearchRequest`` / ``DeepResearchResult`` — engine I/O.
- ``AdapterBundle`` — the named tuple of adapters handed to every runner.
"""
from __future__ import annotations
import hashlib
from datetime import datetime
from typing import TYPE_CHECKING, Any, NamedTuple
from pydantic import BaseModel, Field
from deerflow.agents.deep_research.config import DeepResearchConfig, ModelRole, SourceType
if TYPE_CHECKING:
from deerflow.agents.deep_research.adapters.artifacts import ArtifactWriter
from deerflow.agents.deep_research.adapters.context import DeerFlowContextCompressor
from deerflow.agents.deep_research.adapters.event_sink import EventSink
from deerflow.agents.deep_research.adapters.image import ImageProvider
from deerflow.agents.deep_research.adapters.llm import DeerFlowCompletionBackend
from deerflow.agents.deep_research.adapters.material import MaterialProvider
# ── Research material ────────────────────────────────────────────────────────
class ResearchMaterial(BaseModel):
"""A single piece of research material, fully provenance-tagged.
Supersedes upstream's bare ``{url, raw_content}``: the vendored code receives
``raw_content`` already fetched by the DeerFlow material provider, so it
**never** calls a scraper. Every field here survives into the source store
and into report citations.
"""
id: str
query: str = ""
title: str = ""
url: str | None = None
raw_content: str = ""
snippet: str | None = None
source: str = ""
source_type: SourceType = "web"
published_at: datetime | None = None
relevance_score: float | None = None
rec_uuid: str | None = None
content_hash: str = ""
metadata: dict[str, Any] = Field(default_factory=dict)
model_config = {"extra": "allow"}
def to_context_dict(self) -> dict[str, Any]:
"""The dict shape written into the vendored context list.
Mirrors upstream ``{url, raw_content}`` but preserves *all* DeerFlow
fields so downstream citation binding can recover the source id.
"""
return {
"id": self.id,
"query": self.query,
"title": self.title,
"url": self.url,
"raw_content": self.raw_content,
"snippet": self.snippet,
"source": self.source,
"source_type": self.source_type,
"published_at": self.published_at.isoformat() if self.published_at else None,
"relevance_score": self.relevance_score,
"recUuid": self.rec_uuid,
"metadata": self.metadata,
}
def compute_hash(self) -> str:
"""Stable content fingerprint for dedup (canonical URL preferred)."""
basis = self.url or self.rec_uuid or self.content_hash or self.raw_content
return hashlib.sha256(basis.encode("utf-8")).hexdigest()[:32]
def to_persistence_dict(self, query: str = "") -> dict[str, Any]:
"""Fields for ``DeepResearchSourceRepository.upsert()``.
Excludes ``session_id`` / ``job_id`` (the executor supplies those).
Backfills ``content_hash`` from :meth:`compute_hash` if empty so the
dedup-by-hash fallback works out of the box.
"""
return {
"id": self.id,
"first_seen_query": query or self.query,
"title": self.title,
"url": self.url,
"raw_content": self.raw_content,
"snippet": self.snippet,
"source": self.source,
"source_type": self.source_type,
"published_at": self.published_at,
"relevance_score": self.relevance_score,
"rec_uuid": self.rec_uuid,
"content_hash": self.content_hash or self.compute_hash(),
}
# ── Engine I/O ───────────────────────────────────────────────────────────────
class DeepResearchRequest(BaseModel):
"""Input to ``DeepResearchEngine.run()``."""
session_id: str
user_id: str
query: str
config: DeepResearchConfig
request_id: str = ""
# Filled only when a durable multi-agent plan-review job is resumed.
resume_payload: dict[str, Any] = Field(default_factory=dict)
interrupted_plan: dict[str, Any] | None = None
model_config = {"extra": "forbid"}
class DeepResearchResult(BaseModel):
"""Output of ``DeepResearchEngine.run()``."""
report_markdown: str = ""
report_html: str | None = None
source_ids: list[str] = Field(default_factory=list)
artifact_paths: dict[str, str] = Field(default_factory=dict)
usage: dict[str, Any] = Field(default_factory=dict)
plan: dict[str, Any] | None = None
diagnostics: list[dict[str, Any]] = Field(default_factory=list)
# ── Adapter bundle ───────────────────────────────────────────────────────────
class AdapterBundle(NamedTuple):
"""The five adapters every runner needs, bundled for clean passing.
The vendored GPT Researcher code is adapted to call *only* these (or objects
they produce) — never ``requests``, never ``create_chat_model`` directly,
never the filesystem outside ``ArtifactWriter``.
"""
llm: DeerFlowCompletionBackend
materials: MaterialProvider
context: DeerFlowContextCompressor
artifacts: ArtifactWriter
events: EventSink
images: ImageProvider | None
# ── Completion result (returned by the LLM adapter) ─────────────────────────
class CompletionResult(BaseModel):
"""Normalised LLM completion — text + usage, model-agnostic."""
text: str
model: str = ""
usage: dict[str, Any] = Field(default_factory=dict)
# When ``response_schema`` was requested, the parsed object (already validated).
structured: Any | None = None
model_config = {"extra": "allow"}
__all__ = [
"AdapterBundle",
"CompletionResult",
"DeepResearchRequest",
"DeepResearchResult",
"ModelRole",
"ResearchMaterial",
"SourceType",
]