wsk/deerflow-code
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

文档修改

Browse Source
This commit is contained in:
fengxiangdebaozi committed 2026-05-13 20:22:14 +08:00
1 parent af7e90c87a
commit 02111ab5b8
3 files changed
+1813 -988
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

offline-backend-20260512/backend/CLAUDE.md
+568 -568
View File
File diff suppressed because it is too large. Load diff
offline-backend-20260512/backend/README.md
+420 -420
View File
@@ -1,425 +1,425 @@
# DeerFlow Backend
DeerFlow is a LangGraph-based AI super agent with sandbox execution, persistent memory, and extensible tool integration. The backend enables AI agents to execute code, browse the web, manage files, delegate tasks to subagents, and retain context across conversations - all in isolated, per-thread environments.
---
## Architecture
```
┌──────────────────────────────────────┐
│ Nginx (Port 2026) │
│ Unified reverse proxy │
└───────┬──────────────────┬───────────┘
│ │
/api/langgraph/* │ │ /api/* (other)
▼ ▼
┌────────────────────┐ ┌────────────────────────┐
│ LangGraph Server │ │ Gateway API (8001) │
│ (Port 2024) │ │ FastAPI REST │
│ │ │ │
│ ┌────────────────┐ │ │ Models, MCP, Skills, │
│ │ Lead Agent │ │ │ Memory, Uploads, │
│ │ ┌──────────┐ │ │ │ Artifacts │
│ │ │Middleware│ │ │ └────────────────────────┘
│ │ │ Chain │ │ │
│ │ └──────────┘ │ │
│ │ ┌──────────┐ │ │
│ │ │ Tools │ │ │
│ │ └──────────┘ │ │
│ │ ┌──────────┐ │ │
│ │ │Subagents │ │ │
│ │ └──────────┘ │ │
│ └────────────────┘ │
└────────────────────┘
```
**Request Routing** (via Nginx):
- `/api/langgraph/*` → LangGraph Server - agent interactions, threads, streaming
- `/api/*` (other) → Gateway API - models, MCP, skills, memory, artifacts, uploads, thread-local cleanup
- `/` (non-API) → Frontend - Next.js web interface
---
## Core Components
### Lead Agent
The single LangGraph agent (`lead_agent`) is the runtime entry point, created via `make_lead_agent(config)`. It combines:
- **Dynamic model selection** with thinking and vision support
- **Middleware chain** for cross-cutting concerns (9 middlewares)
- **Tool system** with sandbox, MCP, community, and built-in tools
- **Subagent delegation** for parallel task execution
- **System prompt** with skills injection, memory context, and working directory guidance
### Middleware Chain
Middlewares execute in strict order, each handling a specific concern:
| # | Middleware | Purpose |
|---|-----------|---------|
| 1 | **ThreadDataMiddleware** | Creates per-thread isolated directories (workspace, uploads, outputs) |
| 2 | **UploadsMiddleware** | Injects newly uploaded files into conversation context |
| 3 | **SandboxMiddleware** | Acquires sandbox environment for code execution |
| 4 | **SummarizationMiddleware** | Reduces context when approaching token limits (optional) |
| 5 | **TodoListMiddleware** | Tracks multi-step tasks in plan mode (optional) |
| 6 | **TitleMiddleware** | Auto-generates conversation titles after first exchange |
| 7 | **MemoryMiddleware** | Queues conversations for async memory extraction |
| 8 | **ViewImageMiddleware** | Injects image data for vision-capable models (conditional) |
| 9 | **ClarificationMiddleware** | Intercepts clarification requests and interrupts execution (must be last) |
### Sandbox System
Per-thread isolated execution with virtual path translation:
- **Abstract interface**: `execute_command`, `read_file`, `write_file`, `list_dir`
- **Providers**: `LocalSandboxProvider` (filesystem) and `AioSandboxProvider` (Docker, in community/)
- **Virtual paths**: `/mnt/user-data/{workspace,uploads,outputs}` → thread-specific physical directories
- **Skills path**: `/mnt/skills` → `deer-flow/skills/` directory
- **Skills loading**: Recursively discovers nested `SKILL.md` files under `skills/{public,custom}` and preserves nested container paths
- **File-write safety**: `str_replace` serializes read-modify-write per `(sandbox.id, path)` so isolated sandboxes keep concurrency even when virtual paths match
- **Tools**: `bash`, `ls`, `read_file`, `write_file`, `str_replace` (`bash` is disabled by default when using `LocalSandboxProvider`; use `AioSandboxProvider` for isolated shell access)
### Subagent System
Async task delegation with concurrent execution:
- **Built-in agents**: `general-purpose` (full toolset) and `bash` (command specialist, exposed only when shell access is available)
- **Concurrency**: Max 3 subagents per turn, 15-minute timeout
- **Execution**: Background thread pools with status tracking and SSE events
- **Flow**: Agent calls `task()` tool → executor runs subagent in background → polls for completion → returns result
### Memory System
LLM-powered persistent context retention across conversations:
- **Automatic extraction**: Analyzes conversations for user context, facts, and preferences
- **Structured storage**: User context (work, personal, top-of-mind), history, and confidence-scored facts
- **Debounced updates**: Batches updates to minimize LLM calls (configurable wait time)
- **System prompt injection**: Top facts + context injected into agent prompts
- **Storage**: JSON file with mtime-based cache invalidation
### Tool Ecosystem
| Category | Tools |
|----------|-------|
| **Sandbox** | `bash`, `ls`, `read_file`, `write_file`, `str_replace` |
| **Built-in** | `present_files`, `ask_clarification`, `view_image`, `task` (subagent) |
| **Community** | Tavily (web search), Jina AI (web fetch), Firecrawl (scraping), DuckDuckGo (image search) |
| **MCP** | Any Model Context Protocol server (stdio, SSE, HTTP transports) |
| **Skills** | Domain-specific workflows injected via system prompt |
### Gateway API
FastAPI application providing REST endpoints for frontend integration:
| Route | Purpose |
|-------|---------|
| `GET /api/models` | List available LLM models |
| `GET/PUT /api/mcp/config` | Manage MCP server configurations |
| `GET/PUT /api/skills` | List and manage skills |
| `POST /api/skills/install` | Install skill from `.skill` archive |
| `GET/POST /api/agents` | List visible agents or create a user-owned agent |
| `GET/PUT/DELETE /api/agents/{id}` | Read, update, or delete an id-addressed agent |
| `GET /api/memory` | Retrieve memory data |
| `POST /api/memory/reload` | Force memory reload |
| `GET /api/memory/config` | Memory configuration |
| `GET /api/memory/status` | Combined config + data |
| `POST /api/threads/{id}/uploads` | Upload files (auto-converts PDF/PPT/Excel/Word to Markdown, rejects directory paths) |
| `GET /api/threads/{id}/uploads/list` | List uploaded files |
| `DELETE /api/threads/{id}` | Delete DeerFlow-managed local thread data after LangGraph thread deletion; unexpected failures are logged server-side and return a generic 500 detail |
| `GET /api/threads/{id}/artifacts/{path}` | Serve generated artifacts |
### Custom Agents
Custom agents are indexed in the `agents` database table. The `id` column is the stable runtime identifier and filesystem directory name (`.deer-flow/agents/{id}/SOUL.md`); `name` is display-only, may be Chinese, and may be duplicated. Rows with `user_id = NULL` are built-in agents, user rows are private unless `published = true`, and users can only update or delete agents they created.
### IM Channels
The IM bridge supports Feishu, Slack, and Telegram. Slack and Telegram still use the final `runs.wait()` response path, while Feishu now streams through `runs.stream(["messages-tuple", "values"])` and updates a single in-thread card in place.
For Feishu card updates, DeerFlow stores the running card's `message_id` per inbound message and patches that same card until the run finishes, preserving the existing `OK` / `DONE` reaction flow.
---
## Quick Start
### Prerequisites
- Python 3.12+
- [uv](https://docs.astral.sh/uv/) package manager
- API keys for your chosen LLM provider
### Installation
```bash
cd deer-flow
# Copy configuration files
cp config.example.yaml config.yaml
# Install backend dependencies
cd backend
make install
```
### Configuration
Edit `config.yaml` in the project root:
```yaml
models:
- name: gpt-4o
display_name: GPT-4o
use: langchain_openai:ChatOpenAI
model: gpt-4o
api_key: $OPENAI_API_KEY
supports_thinking: false
supports_vision: true
- name: gpt-5-responses
display_name: GPT-5 (Responses API)
use: langchain_openai:ChatOpenAI
model: gpt-5
api_key: $OPENAI_API_KEY
use_responses_api: true
output_version: responses/v1
supports_vision: true
```
Set your API keys:
```bash
export OPENAI_API_KEY="your-api-key-here"
```
### Running
**Full Application** (from project root):
```bash
make dev # Starts LangGraph + Gateway + Frontend + Nginx
```
Access at: http://localhost:2026
**Backend Only** (from backend directory):
```bash
# Terminal 1: LangGraph server
make dev
# Terminal 2: Gateway API
make gateway
```
Direct access: LangGraph at http://localhost:2024, Gateway at http://localhost:8001
---
## Project Structure
```
backend/
├── src/
│ ├── agents/ # Agent system
│ │ ├── lead_agent/ # Main agent (factory, prompts)
│ │ ├── middlewares/ # 9 middleware components
│ │ ├── memory/ # Memory extraction & storage
│ │ └── thread_state.py # ThreadState schema
│ ├── gateway/ # FastAPI Gateway API
│ │ ├── app.py # Application setup
│ │ └── routers/ # 6 route modules
│ ├── sandbox/ # Sandbox execution
│ │ ├── local/ # Local filesystem provider
│ │ ├── sandbox.py # Abstract interface
│ │ ├── tools.py # bash, ls, read/write/str_replace
│ │ └── middleware.py # Sandbox lifecycle
│ ├── subagents/ # Subagent delegation
│ │ ├── builtins/ # general-purpose, bash agents
│ │ ├── executor.py # Background execution engine
│ │ └── registry.py # Agent registry
│ ├── tools/builtins/ # Built-in tools
│ ├── mcp/ # MCP protocol integration
│ ├── models/ # Model factory
│ ├── skills/ # Skill discovery & loading
│ ├── config/ # Configuration system
│ ├── community/ # Community tools & providers
│ ├── reflection/ # Dynamic module loading
│ └── utils/ # Utilities
├── docs/ # Documentation
├── tests/ # Test suite
├── langgraph.json # LangGraph server configuration
├── pyproject.toml # Python dependencies
├── Makefile # Development commands
└── Dockerfile # Container build
```
---
## Configuration
### Main Configuration (`config.yaml`)
Place in project root. Config values starting with `$` resolve as environment variables.
Key sections:
- `models` - LLM configurations with class paths, API keys, thinking/vision flags
- `tools` - Tool definitions with module paths and groups
- `tool_groups` - Logical tool groupings
- `sandbox` - Execution environment provider
- `skills` - Skills directory paths
- `title` - Auto-title generation settings
- `summarization` - Context summarization settings
- `subagents` - Subagent system (enabled/disabled)
- `memory` - Memory system settings (enabled, storage, debounce, facts limits)
Provider note:
- `models[*].use` references provider classes by module path (for example `langchain_openai:ChatOpenAI`).
- If a provider module is missing, DeerFlow now returns an actionable error with install guidance (for example `uv add langchain-google-genai`).
### Extensions Configuration (`extensions_config.json`)
MCP servers and skill states in a single file:
```json
{
"mcpServers": {
"github": {
"enabled": true,
"type": "stdio",
"command": "npx",
"args": ["-y", "@modelcontextprotocol/server-github"],
"env": {"GITHUB_TOKEN": "$GITHUB_TOKEN"}
},
"secure-http": {
"enabled": true,
"type": "http",
"url": "https://api.example.com/mcp",
"oauth": {
"enabled": true,
"token_url": "https://auth.example.com/oauth/token",
"grant_type": "client_credentials",
"client_id": "$MCP_OAUTH_CLIENT_ID",
"client_secret": "$MCP_OAUTH_CLIENT_SECRET"
}
}
},
"skills": {
"pdf-processing": {"enabled": true}
}
}
```
### Environment Variables
- `DEER_FLOW_CONFIG_PATH` - Override config.yaml location
- `DEER_FLOW_EXTENSIONS_CONFIG_PATH` - Override extensions_config.json location
- Model API keys: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `DEEPSEEK_API_KEY`, etc.
- Tool API keys: `TAVILY_API_KEY`, `GITHUB_TOKEN`, etc.
### LangSmith Tracing
DeerFlow has built-in [LangSmith](https://smith.langchain.com) integration for observability. When enabled, all LLM calls, agent runs, tool executions, and middleware processing are traced and visible in the LangSmith dashboard.
**Setup:**
1. Sign up at [smith.langchain.com](https://smith.langchain.com) and create a project.
2. Add the following to your `.env` file in the project root:
```bash
LANGSMITH_TRACING=true
LANGSMITH_ENDPOINT=https://api.smith.langchain.com
LANGSMITH_API_KEY=lsv2_pt_xxxxxxxxxxxxxxxx
LANGSMITH_PROJECT=xxx
```
**Legacy variables:** The `LANGCHAIN_TRACING_V2`, `LANGCHAIN_API_KEY`, `LANGCHAIN_PROJECT`, and `LANGCHAIN_ENDPOINT` variables are also supported for backward compatibility. `LANGSMITH_*` variables take precedence when both are set.
### Langfuse Tracing
DeerFlow also supports [Langfuse](https://langfuse.com) observability for LangChain-compatible runs.
Add the following to your `.env` file:
```bash
LANGFUSE_TRACING=true
LANGFUSE_PUBLIC_KEY=pk-lf-xxxxxxxxxxxxxxxx
LANGFUSE_SECRET_KEY=sk-lf-xxxxxxxxxxxxxxxx
LANGFUSE_BASE_URL=https://cloud.langfuse.com
```
If you are using a self-hosted Langfuse deployment, set `LANGFUSE_BASE_URL` to your Langfuse host.
### Dual Provider Behavior
If both LangSmith and Langfuse are enabled, DeerFlow initializes and attaches both callbacks so the same run data is reported to both systems.
If a provider is explicitly enabled but required credentials are missing, or the provider callback cannot be initialized, DeerFlow raises an error when tracing is initialized during model creation instead of silently disabling tracing.
**Docker:** In `docker-compose.yaml`, tracing is disabled by default (`LANGSMITH_TRACING=false`). Set `LANGSMITH_TRACING=true` and/or `LANGFUSE_TRACING=true` in your `.env`, together with the required credentials, to enable tracing in containerized deployments.
---
## Development
### Commands
```bash
make install # Install dependencies
make dev # Run LangGraph server (port 2024)
make gateway # Run Gateway API (port 8001)
make lint # Run linter (ruff)
make format # Format code (ruff)
```
### Code Style
- **Linter/Formatter**: `ruff`
- **Line length**: 240 characters
- **Python**: 3.12+ with type hints
- **Quotes**: Double quotes
- **Indentation**: 4 spaces
### Testing
```bash
uv run pytest
```
---
## Technology Stack
- **LangGraph** (1.0.6+) - Agent framework and multi-agent orchestration
- **LangChain** (1.2.3+) - LLM abstractions and tool system
- **FastAPI** (0.115.0+) - Gateway REST API
- **langchain-mcp-adapters** - Model Context Protocol support
- **agent-sandbox** - Sandboxed code execution
- **markitdown** - Multi-format document conversion
- **tavily-python** / **firecrawl-py** - Web search and scraping
---
## Documentation
- [Configuration Guide](docs/CONFIGURATION.md)
- [Architecture Details](docs/ARCHITECTURE.md)
- [API Reference](docs/API.md)
- [File Upload](docs/FILE_UPLOAD.md)
# DeerFlow Backend
DeerFlow is a LangGraph-based AI super agent with sandbox execution, persistent memory, and extensible tool integration. The backend enables AI agents to execute code, browse the web, manage files, delegate tasks to subagents, and retain context across conversations - all in isolated, per-thread environments.
---
## Architecture
```
┌──────────────────────────────────────┐
│ Nginx (Port 2026) │
│ Unified reverse proxy │
└───────┬──────────────────┬───────────┘
│ │
/api/langgraph/* │ │ /api/* (other)
▼ ▼
┌────────────────────┐ ┌────────────────────────┐
│ LangGraph Server │ │ Gateway API (8001) │
│ (Port 2024) │ │ FastAPI REST │
│ │ │ │
│ ┌────────────────┐ │ │ Models, MCP, Skills, │
│ │ Lead Agent │ │ │ Memory, Uploads, │
│ │ ┌──────────┐ │ │ │ Artifacts │
│ │ │Middleware│ │ │ └────────────────────────┘
│ │ │ Chain │ │ │
│ │ └──────────┘ │ │
│ │ ┌──────────┐ │ │
│ │ │ Tools │ │ │
│ │ └──────────┘ │ │
│ │ ┌──────────┐ │ │
│ │ │Subagents │ │ │
│ │ └──────────┘ │ │
│ └────────────────┘ │
└────────────────────┘
```
**Request Routing** (via Nginx):
- `/api/langgraph/*` → LangGraph Server - agent interactions, threads, streaming
- `/api/*` (other) → Gateway API - models, MCP, skills, memory, artifacts, uploads, thread-local cleanup
- `/` (non-API) → Frontend - Next.js web interface
---
## Core Components
### Lead Agent
The single LangGraph agent (`lead_agent`) is the runtime entry point, created via `make_lead_agent(config)`. It combines:
- **Dynamic model selection** with thinking and vision support
- **Middleware chain** for cross-cutting concerns (9 middlewares)
- **Tool system** with sandbox, MCP, community, and built-in tools
- **Subagent delegation** for parallel task execution
- **System prompt** with skills injection, memory context, and working directory guidance
### Middleware Chain
Middlewares execute in strict order, each handling a specific concern:
| # | Middleware | Purpose |
|---|-----------|---------|
| 1 | **ThreadDataMiddleware** | Creates per-thread isolated directories (workspace, uploads, outputs) |
| 2 | **UploadsMiddleware** | Injects newly uploaded files into conversation context |
| 3 | **SandboxMiddleware** | Acquires sandbox environment for code execution |
| 4 | **SummarizationMiddleware** | Reduces context when approaching token limits (optional) |
| 5 | **TodoListMiddleware** | Tracks multi-step tasks in plan mode (optional) |
| 6 | **TitleMiddleware** | Auto-generates conversation titles after first exchange |
| 7 | **MemoryMiddleware** | Queues conversations for async memory extraction |
| 8 | **ViewImageMiddleware** | Injects image data for vision-capable models (conditional) |
| 9 | **ClarificationMiddleware** | Intercepts clarification requests and interrupts execution (must be last) |
### Sandbox System
Per-thread isolated execution with virtual path translation:
- **Abstract interface**: `execute_command`, `read_file`, `write_file`, `list_dir`
- **Providers**: `LocalSandboxProvider` (filesystem) and `AioSandboxProvider` (Docker, in community/)
- **Virtual paths**: `/mnt/user-data/{workspace,uploads,outputs}` → thread-specific physical directories
- **Skills path**: `/mnt/skills` → `deer-flow/skills/` directory
- **Skills loading**: Recursively discovers nested `SKILL.md` files under `skills/{public,custom}` and preserves nested container paths
- **File-write safety**: `str_replace` serializes read-modify-write per `(sandbox.id, path)` so isolated sandboxes keep concurrency even when virtual paths match
- **Tools**: `bash`, `ls`, `read_file`, `write_file`, `str_replace` (`bash` is disabled by default when using `LocalSandboxProvider`; use `AioSandboxProvider` for isolated shell access)
### Subagent System
Async task delegation with concurrent execution:
- **Built-in agents**: `general-purpose` (full toolset) and `bash` (command specialist, exposed only when shell access is available)
- **Concurrency**: Max 3 subagents per turn, 15-minute timeout
- **Execution**: Background thread pools with status tracking and SSE events
- **Flow**: Agent calls `task()` tool → executor runs subagent in background → polls for completion → returns result
### Memory System
LLM-powered persistent context retention across conversations:
- **Automatic extraction**: Analyzes conversations for user context, facts, and preferences
- **Structured storage**: User context (work, personal, top-of-mind), history, and confidence-scored facts
- **Debounced updates**: Batches updates to minimize LLM calls (configurable wait time)
- **System prompt injection**: Top facts + context injected into agent prompts
- **Storage**: JSON file with mtime-based cache invalidation
### Tool Ecosystem
| Category | Tools |
|----------|-------|
| **Sandbox** | `bash`, `ls`, `read_file`, `write_file`, `str_replace` |
| **Built-in** | `present_files`, `ask_clarification`, `view_image`, `task` (subagent) |
| **Community** | Tavily (web search), Jina AI (web fetch), Firecrawl (scraping), DuckDuckGo (image search) |
| **MCP** | Any Model Context Protocol server (stdio, SSE, HTTP transports) |
| **Skills** | Domain-specific workflows injected via system prompt |
### Gateway API
FastAPI application providing REST endpoints for frontend integration:
| Route | Purpose |
|-------|---------|
| `GET /api/models` | List available LLM models |
| `GET/PUT /api/mcp/config` | Manage MCP server configurations |
| `GET/PUT /api/skills` | List and manage skills |
| `POST /api/skills/install` | Install skill from `.skill` archive |
| `GET/POST /api/agents` | List visible agents or create a user-owned agent |
| `GET/PUT/DELETE /api/agents/{id}` | Read, update, or delete an id-addressed agent |
| `GET /api/memory` | Retrieve memory data |
| `POST /api/memory/reload` | Force memory reload |
| `GET /api/memory/config` | Memory configuration |
| `GET /api/memory/status` | Combined config + data |
| `POST /api/threads/{id}/uploads` | Upload files (auto-converts PDF/PPT/Excel/Word to Markdown, rejects directory paths) |
| `GET /api/threads/{id}/uploads/list` | List uploaded files |
| `DELETE /api/threads/{id}` | Delete DeerFlow-managed local thread data after LangGraph thread deletion; unexpected failures are logged server-side and return a generic 500 detail |
| `GET /api/threads/{id}/artifacts/{path}` | Serve generated artifacts |
### Custom Agents
Custom agents are indexed in the `agents` database table. The `id` column is the stable runtime identifier and filesystem directory name (`.deer-flow/agents/{id}/SOUL.md`); `name` is display-only, may be Chinese, and may be duplicated. Rows with `user_id = NULL` are built-in agents, user rows are private unless `published = true`, and users can only update or delete agents they created.
### IM Channels
The IM bridge supports Feishu, Slack, and Telegram. Slack and Telegram still use the final `runs.wait()` response path, while Feishu now streams through `runs.stream(["messages-tuple", "values"])` and updates a single in-thread card in place.
For Feishu card updates, DeerFlow stores the running card's `message_id` per inbound message and patches that same card until the run finishes, preserving the existing `OK` / `DONE` reaction flow.
---
## Quick Start
### Prerequisites
- Python 3.12+
- [uv](https://docs.astral.sh/uv/) package manager
- API keys for your chosen LLM provider
### Installation
```bash
cd deer-flow
# Copy configuration files
cp config.example.yaml config.yaml
# Install backend dependencies
cd backend
make install
```
### Configuration
Edit `config.yaml` in the project root:
```yaml
models:
- name: gpt-4o
display_name: GPT-4o
use: langchain_openai:ChatOpenAI
model: gpt-4o
api_key: $OPENAI_API_KEY
supports_thinking: false
supports_vision: true
- name: gpt-5-responses
display_name: GPT-5 (Responses API)
use: langchain_openai:ChatOpenAI
model: gpt-5
api_key: $OPENAI_API_KEY
use_responses_api: true
output_version: responses/v1
supports_vision: true
```
Set your API keys:
```bash
export OPENAI_API_KEY="your-api-key-here"
```
### Running
**Full Application** (from project root):
```bash
make dev # Starts LangGraph + Gateway + Frontend + Nginx
```
Access at: http://localhost:2026
**Backend Only** (from backend directory):
```bash
# Terminal 1: LangGraph server
make dev
# Terminal 2: Gateway API
make gateway
```
Direct access: LangGraph at http://localhost:2024, Gateway at http://localhost:8001
---
## Project Structure
```
backend/
├── src/
│ ├── agents/ # Agent system
│ │ ├── lead_agent/ # Main agent (factory, prompts)
│ │ ├── middlewares/ # 9 middleware components
│ │ ├── memory/ # Memory extraction & storage
│ │ └── thread_state.py # ThreadState schema
│ ├── gateway/ # FastAPI Gateway API
│ │ ├── app.py # Application setup
│ │ └── routers/ # 6 route modules
│ ├── sandbox/ # Sandbox execution
│ │ ├── local/ # Local filesystem provider
│ │ ├── sandbox.py # Abstract interface
│ │ ├── tools.py # bash, ls, read/write/str_replace
│ │ └── middleware.py # Sandbox lifecycle
│ ├── subagents/ # Subagent delegation
│ │ ├── builtins/ # general-purpose, bash agents
│ │ ├── executor.py # Background execution engine
│ │ └── registry.py # Agent registry
│ ├── tools/builtins/ # Built-in tools
│ ├── mcp/ # MCP protocol integration
│ ├── models/ # Model factory
│ ├── skills/ # Skill discovery & loading
│ ├── config/ # Configuration system
│ ├── community/ # Community tools & providers
│ ├── reflection/ # Dynamic module loading
│ └── utils/ # Utilities
├── docs/ # Documentation
├── tests/ # Test suite
├── langgraph.json # LangGraph server configuration
├── pyproject.toml # Python dependencies
├── Makefile # Development commands
└── Dockerfile # Container build
```
---
## Configuration
### Main Configuration (`config.yaml`)
Place in project root. Config values starting with `$` resolve as environment variables.
Key sections:
- `models` - LLM configurations with class paths, API keys, thinking/vision flags
- `tools` - Tool definitions with module paths and groups
- `tool_groups` - Logical tool groupings
- `sandbox` - Execution environment provider
- `skills` - Skills directory paths
- `title` - Auto-title generation settings
- `summarization` - Context summarization settings
- `subagents` - Subagent system (enabled/disabled)
- `memory` - Memory system settings (enabled, storage, debounce, facts limits)
Provider note:
- `models[*].use` references provider classes by module path (for example `langchain_openai:ChatOpenAI`).
- If a provider module is missing, DeerFlow now returns an actionable error with install guidance (for example `uv add langchain-google-genai`).
### Extensions Configuration (`extensions_config.json`)
MCP servers and skill states in a single file:
```json
{
"mcpServers": {
"github": {
"enabled": true,
"type": "stdio",
"command": "npx",
"args": ["-y", "@modelcontextprotocol/server-github"],
"env": {"GITHUB_TOKEN": "$GITHUB_TOKEN"}
},
"secure-http": {
"enabled": true,
"type": "http",
"url": "https://api.example.com/mcp",
"oauth": {
"enabled": true,
"token_url": "https://auth.example.com/oauth/token",
"grant_type": "client_credentials",
"client_id": "$MCP_OAUTH_CLIENT_ID",
"client_secret": "$MCP_OAUTH_CLIENT_SECRET"
}
}
},
"skills": {
"pdf-processing": {"enabled": true}
}
}
```
### Environment Variables
- `DEER_FLOW_CONFIG_PATH` - Override config.yaml location
- `DEER_FLOW_EXTENSIONS_CONFIG_PATH` - Override extensions_config.json location
- Model API keys: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `DEEPSEEK_API_KEY`, etc.
- Tool API keys: `TAVILY_API_KEY`, `GITHUB_TOKEN`, etc.
### LangSmith Tracing
DeerFlow has built-in [LangSmith](https://smith.langchain.com) integration for observability. When enabled, all LLM calls, agent runs, tool executions, and middleware processing are traced and visible in the LangSmith dashboard.
**Setup:**
1. Sign up at [smith.langchain.com](https://smith.langchain.com) and create a project.
2. Add the following to your `.env` file in the project root:
```bash
LANGSMITH_TRACING=true
LANGSMITH_ENDPOINT=https://api.smith.langchain.com
LANGSMITH_API_KEY=lsv2_pt_xxxxxxxxxxxxxxxx
LANGSMITH_PROJECT=xxx
```
**Legacy variables:** The `LANGCHAIN_TRACING_V2`, `LANGCHAIN_API_KEY`, `LANGCHAIN_PROJECT`, and `LANGCHAIN_ENDPOINT` variables are also supported for backward compatibility. `LANGSMITH_*` variables take precedence when both are set.
### Langfuse Tracing
DeerFlow also supports [Langfuse](https://langfuse.com) observability for LangChain-compatible runs.
Add the following to your `.env` file:
```bash
LANGFUSE_TRACING=true
LANGFUSE_PUBLIC_KEY=pk-lf-xxxxxxxxxxxxxxxx
LANGFUSE_SECRET_KEY=sk-lf-xxxxxxxxxxxxxxxx
LANGFUSE_BASE_URL=https://cloud.langfuse.com
```
If you are using a self-hosted Langfuse deployment, set `LANGFUSE_BASE_URL` to your Langfuse host.
### Dual Provider Behavior
If both LangSmith and Langfuse are enabled, DeerFlow initializes and attaches both callbacks so the same run data is reported to both systems.
If a provider is explicitly enabled but required credentials are missing, or the provider callback cannot be initialized, DeerFlow raises an error when tracing is initialized during model creation instead of silently disabling tracing.
**Docker:** In `docker-compose.yaml`, tracing is disabled by default (`LANGSMITH_TRACING=false`). Set `LANGSMITH_TRACING=true` and/or `LANGFUSE_TRACING=true` in your `.env`, together with the required credentials, to enable tracing in containerized deployments.
---
## Development
### Commands
```bash
make install # Install dependencies
make dev # Run LangGraph server (port 2024)
make gateway # Run Gateway API (port 8001)
make lint # Run linter (ruff)
make format # Format code (ruff)
```
### Code Style
- **Linter/Formatter**: `ruff`
- **Line length**: 240 characters
- **Python**: 3.12+ with type hints
- **Quotes**: Double quotes
- **Indentation**: 4 spaces
### Testing
```bash
uv run pytest
```
---
## Technology Stack
- **LangGraph** (1.0.6+) - Agent framework and multi-agent orchestration
- **LangChain** (1.2.3+) - LLM abstractions and tool system
- **FastAPI** (0.115.0+) - Gateway REST API
- **langchain-mcp-adapters** - Model Context Protocol support
- **agent-sandbox** - Sandboxed code execution
- **markitdown** - Multi-format document conversion
- **tavily-python** / **firecrawl-py** - Web search and scraping
---
## Documentation
- [Configuration Guide](docs/CONFIGURATION.md)
- [Architecture Details](docs/ARCHITECTURE.md)
- [API Reference](docs/API.md)
- [File Upload](docs/FILE_UPLOAD.md)
- [Path Examples](docs/PATH_EXAMPLES.md)
- [Context Summarization](docs/summarization.md)
- [Plan Mode](docs/plan_mode_usage.md)
- [Backend Run/Stop Guide (ZH)](docs/BACKEND_RUN_STOP_ZH.md)
- [Setup Guide](docs/SETUP.md)
---
## License
See the [LICENSE](../LICENSE) file in the project root.
## Contributing
See [CONTRIBUTING.md](CONTRIBUTING.md) for contribution guidelines.
---
## License
See the [LICENSE](../LICENSE) file in the project root.
## Contributing
See [CONTRIBUTING.md](CONTRIBUTING.md) for contribution guidelines.
offline-backend-20260512/backend/docs/houduankai.md
+825
View File
@@ -0,0 +1,825 @@
# DeerFlow 后端开发文档
> 本文档面向开发人员，梳理后端项目结构、核心模块、API 接口、配置系统和常用开发命令，方便快速上手和维护。
---
## 目录
1. [项目概述](#1-项目概述)
2. [技术栈](#2-技术栈)
3. [目录结构](#3-目录结构)
4. [架构分层](#4-架构分层)
5. [快速启动](#5-快速启动)
6. [配置系统](#6-配置系统)
7. [核心模块详解](#7-核心模块详解)
8. [API 接口汇总](#8-api-接口汇总)
9. [Agent 系统](#9-agent-系统)
10. [中间件链](#10-中间件链)
11. [沙箱执行系统](#11-沙箱执行系统)
12. [内存系统](#12-内存系统)
13. [工具系统](#13-工具系统)
14. [IM 频道集成](#14-im-频道集成)
15. [数据持久化](#15-数据持久化)
16. [测试](#16-测试)
17. [常见问题与排错](#17-常见问题与排错)
---
## 1. 项目概述
DeerFlow 是一个基于 **LangGraph** 的 AI 超级 Agent 系统。后端提供：
- **Lead Agent**：具备沙箱执行、持久内存、子 Agent 委托和可扩展工具集成的主 Agent
- **Gateway API**：FastAPI REST API（端口 8001），同时内嵌 LangGraph 兼容的 Agent 运行时
- **IM 频道集成**：飞书、Slack、Telegram、钉钉、Discord、微信、企业微信
- **每线程隔离**：沙箱和工作目录按 `用户 / 线程` 两级隔离
整体入口（通过 Nginx，端口 2026）：
- `/api/langgraph/*` → 内嵌 LangGraph 运行时（转发到 Gateway `/api/*`）
- `/api/*` → Gateway REST API（8001）
- `/` → 前端（3000）
---
## 2. 技术栈
| 层次 | 技术 |
|------|------|
| Web 框架 | FastAPI + Uvicorn |
| Agent 框架 | LangGraph（状态图 + 检查点） |
| LLM 调用 | LangChain（多模型适配） |
| 数据库 | SQLite（默认）/ PostgreSQL |
| ORM | SQLAlchemy + Alembic（迁移） |
| 包管理 | uv（workspace 模式） |
| 测试 | pytest |
| 代码规范 | ruff（检查 + 格式化） |
| Python 版本 | 3.12+ |
---
## 3. 目录结构
```
backend/
├── Makefile # 开发命令入口
├── pyproject.toml # 项目配置和依赖
├── config.yaml # 主配置文件（模型、沙箱、工具等）
├── langgraph.json # LangGraph Studio 图配置
├── ruff.toml # 代码风格配置（行长 240）
│
├── app/ # 应用层（FastAPI + IM 集成）
│ ├── gateway/ # FastAPI 网关 API
│ │ ├── app.py # 应用主入口，创建 FastAPI 实例，挂载所有路由
│ │ ├── auth/ # JWT 身份认证模块
│ │ ├── routers/ # 16 个 API 路由模块
│ │ ├── auth_middleware.py # 认证中间件
│ │ ├── csrf_middleware.py # CSRF 保护中间件
│ │ └── deps.py # FastAPI 依赖注入
│ └── channels/ # IM 平台集成
│ ├── manager.py # 频道管理器（核心调度）
│ ├── message_bus.py # 消息总线（pub/sub）
│ ├── store.py # 线程持久化（chat → thread_id 映射）
│ ├── feishu.py # 飞书集成
│ ├── slack.py # Slack 集成
│ ├── telegram.py # Telegram 集成
│ ├── dingtalk.py # 钉钉集成
│ ├── discord.py # Discord 集成
│ ├── wechat.py # 微信集成
│ └── wecom.py # 企业微信集成
│
├── packages/harness/ # deerflow-harness 核心包（可独立发布）
│ └── deerflow/ # 导入前缀：deerflow.*
│ ├── agents/ # Agent 系统
│ │ ├── lead_agent/ # 主 Agent（工厂 + 提示词）
│ │ ├── middlewares/ # 18 个中间件
│ │ ├── memory/ # 内存提取、队列、存储
│ │ └── thread_state.py # ThreadState 数据模型
│ ├── config/ # 配置系统（25 个模块）
│ ├── sandbox/ # 沙箱执行（本地 + Docker）
│ ├── tools/ # 工具工厂 + 内置工具
│ ├── subagents/ # 子 Agent 委托系统
│ ├── mcp/ # Model Context Protocol 集成
│ ├── models/ # LLM 模型工厂
│ ├── skills/ # 技能发现和加载
│ ├── runtime/ # 运行时管理（检查点、流、调度）
│ ├── persistence/ # 数据库 ORM 和迁移
│ ├── community/ # 社区工具（Tavily、Jina、Firecrawl 等）
│ ├── guardrails/ # 护栏系统（工具调用授权）
│ ├── reflection/ # 动态模块加载
│ ├── uploads/ # 文件上传和文档转换
│ ├── utils/ # 网络、可读性等工具函数
│ └── client.py # 嵌入式 Python 客户端（无需 HTTP）
│
├── tests/ # 测试套件
├── scripts/ # 辅助脚本
├── docs/ # 文档目录
└── skills/ # 技能目录
├── public/ # 公开技能（提交到 git）
└── custom/ # 自定义技能（git 忽略）
```
---
## 4. 架构分层
### Harness / App 分层原则
后端严格分为两层，依赖方向单向：
```
app.* →（允许导入）→ deerflow.*
deerflow.* ×（禁止导入）× app.*
```
- **Harness**（`packages/harness/deerflow/`）：可发布的 Agent 框架包，包含 Agent 编排、工具、沙箱、模型、MCP、技能、配置——一切构建和运行 Agent 所需的内容。
- **App**（`app/`）：不对外发布的应用代码，包含 FastAPI Gateway API 和 IM 频道集成。
此边界通过 `tests/test_harness_boundary.py` 在 CI 中强制检查。
### 运行时数据流
```
前端 / IM 平台
↓
Nginx（2026）
↓
Gateway API（8001）— FastAPI
↓
RunManager + run_agent() + StreamBridge
↓
Lead Agent（LangGraph 状态图）
↓
[中间件链] → [LLM] → [工具调用] → [沙箱执行]
```
---
## 5. 快速启动
### 环境要求
- Python 3.12+
- [uv](https://github.com/astral-sh/uv)（包管理器）
- 可选：Docker（用于容器化沙箱）
### 安装依赖
```bash
# 在 backend/ 目录下
make install
# 等价于：uv sync
```
### 启动开发服务器
```bash
# 方式一：仅启动后端 Gateway（带热重载，端口 8001）
make dev
# 方式二：启动完整应用（含前端和 Nginx）
# 在项目根目录执行
make dev
```
启动后访问：
- API 文档：`http://localhost:8001/docs`
- 健康检查：`http://localhost:8001/health`
### 停止服务
```bash
# 项目根目录
make stop
```
### 开发常用命令
```bash
make lint # ruff 代码检查
make format # ruff 自动格式化
make test # 运行全部测试
make gateway # 启动 Gateway（不带热重载）
# 运行特定测试文件
PYTHONPATH=. uv run pytest tests/test_memory_updater.py -v
```
---
## 6. 配置系统
### 主配置文件 `config.yaml`
**查找顺序**（优先级从高到低）：
1. 显式 `config_path` 参数
2. 环境变量 `DEER_FLOW_CONFIG_PATH`
3. `backend/config.yaml`
4. 项目根目录 `config.yaml`（**推荐放置位置**）
**关键配置段**：
```yaml
config_version: 1 # 配置版本，升级时自动提示
log_level: info
database:
backend: sqlite # 数据库后端（sqlite / postgres）
sqlite_dir: .deer-flow/data
sandbox:
use: deerflow.sandbox.local:LocalSandboxProvider # 沙箱提供者类路径
allow_host_bash: false
models: # LLM 模型配置列表
- name: my-model # 模型标识名
display_name: 我的模型
use: langchain_openai:ChatOpenAI # 提供者类（通过反射加载）
model: gpt-4o
api_key: $OPENAI_API_KEY # 以 $ 开头代表读取环境变量
base_url: https://api.openai.com/v1
supports_vision: true
supports_thinking: false
tools: # 工具配置
- use: deerflow.community.tavily:tavily_search
group: search
memory:
enabled: true
injection_enabled: true
debounce_seconds: 30
max_facts: 100
fact_confidence_threshold: 0.7
max_injection_tokens: 2000
subagents:
enabled: true # 是否启用子 Agent 委托
title:
enabled: true # 是否自动生成对话标题
```
### 扩展配置 `extensions_config.json`
MCP 服务器和技能的启用状态存放在此文件：
```json
{
"mcpServers": {
"my-mcp-server": {
"enabled": true,
"type": "stdio",
"command": "npx",
"args": ["-y", "@my/mcp-server"]
}
},
"skills": {
"my-skill": { "enabled": true }
}
}
```
**查找顺序**同 `config.yaml`，环境变量为 `DEER_FLOW_EXTENSIONS_CONFIG_PATH`。
### 环境变量
| 变量名 | 说明 |
|--------|------|
| `DEER_FLOW_CONFIG_PATH` | 指定 config.yaml 路径 |
| `DEER_FLOW_EXTENSIONS_CONFIG_PATH` | 指定 extensions_config.json 路径 |
| `GATEWAY_CORS_ORIGINS` | 允许的跨域来源（逗号分隔） |
| `GATEWAY_CORS_ALLOW_ALL` | 允许所有跨域（1/true） |
| `GATEWAY_ENABLE_DOCS` | 是否启用 `/docs` 页面（默认开启） |
| `AUTH_JWT_SECRET` | JWT 签名密钥 |
| `DEER_FLOW_AUTH_DISABLED` | 禁用认证（开发用） |
---
## 7. 核心模块详解
### 7.1 应用主入口 `app/gateway/app.py`
FastAPI 应用工厂，负责：
- 创建 FastAPI 实例，配置 CORS、认证、CSRF 中间件
- 挂载全部 16 个路由模块
- 通过 `lifespan` 管理启动/关闭钩子：初始化数据库、运行 Alembic 迁移、启动 IM 频道服务
- 首次启动时创建管理员用户
### 7.2 模型工厂 `deerflow/models/factory.py`
```python
from deerflow.models import create_chat_model
model = create_chat_model(name="my-model", thinking_enabled=False)
```
- 通过反射（`resolve_variable`）按 `config.yaml` 中的 `use` 字段实例化 LangChain 模型
- 支持 `thinking_enabled` 标志和按模型的 `when_thinking_enabled` 覆盖
- 支持 vLLM Qwen 推理模型的 `enable_thinking` 参数
- 配置值以 `$` 开头时自动解析为环境变量
- 缺少提供者模块时给出可操作的安装提示
### 7.3 嵌入式客户端 `deerflow/client.py`
无需启动 HTTP 服务即可直接调用所有 DeerFlow 能力：
```python
from deerflow.client import DeerFlowClient
client = DeerFlowClient()
# 同步对话
text = client.chat("你好", thread_id="thread-1")
# 流式对话
for event in client.stream("分析这份文档", thread_id="thread-1"):
print(event)
# 其他 API
client.list_models()
client.upload_files(thread_id, [Path("doc.pdf")])
client.get_memory()
```
返回格式与 Gateway REST API 一致，方便在两种模式下复用代码。
---
## 8. API 接口汇总
所有接口基础路径：`http://localhost:8001`（或通过 Nginx `http://localhost:2026`）
### 8.1 核心接口
| 方法 | 路径 | 说明 |
|------|------|------|
| GET | `/health` | 健康检查 |
| GET | `/docs` | Swagger API 文档 |
### 8.2 线程运行（主要对话接口）
| 方法 | 路径 | 说明 |
|------|------|------|
| POST | `/api/threads/{id}/runs/stream` | 创建运行并 SSE 流式返回 |
| POST | `/api/threads/{id}/runs/wait` | 创建运行并阻塞等待结果 |
| POST | `/api/threads/{id}/runs` | 创建后台运行 |
| GET | `/api/threads/{id}/runs` | 列出线程的所有运行 |
| GET | `/api/threads/{id}/runs/{rid}` | 获取运行详情 |
| POST | `/api/threads/{id}/runs/{rid}/cancel` | 取消运行 |
| GET | `/api/threads/{id}/runs/{rid}/join` | 加入 SSE 流 |
| GET | `/api/threads/{id}/runs/{rid}/messages` | 分页获取消息 |
| GET | `/api/threads/{id}/messages` | 线程所有消息（含反馈） |
### 8.3 无状态运行
| 方法 | 路径 | 说明 |
|------|------|------|
| POST | `/api/runs/stream` | 无状态运行 + SSE 流 |
| POST | `/api/runs/wait` | 无状态运行 + 阻塞 |
### 8.4 文件上传
| 方法 | 路径 | 说明 |
|------|------|------|
| POST | `/api/threads/{id}/uploads` | 上传文件（自动转换 PDF/PPT/Excel/Word） |
| GET | `/api/threads/{id}/uploads/list` | 列出已上传文件 |
| DELETE | `/api/threads/{id}/uploads/{filename}` | 删除文件 |
### 8.5 其他接口
| 方法 | 路径 | 说明 |
|------|------|------|
| GET | `/api/models` | 列出所有可用 LLM 模型 |
| GET | `/api/models/{name}` | 模型详情 |
| GET | `/api/mcp/config` | 获取 MCP 配置 |
| PUT | `/api/mcp/config` | 更新 MCP 配置 |
| GET | `/api/skills` | 列出技能 |
| PUT | `/api/skills/{name}` | 启用/禁用技能 |
| POST | `/api/skills/install` | 安装 .skill 包 |
| GET | `/api/memory` | 获取个人内存数据 |
| POST | `/api/memory/reload` | 强制重新加载内存 |
| GET/POST/PUT/DELETE | `/api/agents` | 自定义 Agent CRUD |
| GET | `/api/threads/{id}/artifacts/{path}` | 下载工件文件 |
| POST | `/api/threads/{id}/suggestions` | 生成后续问题建议 |
| DELETE | `/api/threads/{id}` | 清理线程本地数据 |
| PUT | `/api/threads/{id}/runs/{rid}/feedback` | 提交运行反馈 |
| POST/GET | `/api/scheduled-tasks` | 定时任务管理 |
| POST | `/api/v1/auth/login` | 用户登录 |
| POST | `/api/v1/auth/register` | 用户注册 |
---
## 9. Agent 系统
### Lead Agent
**入口**：`deerflow.agents.lead_agent.agent:make_lead_agent`，注册在 `langgraph.json`。
```python
from deerflow.agents import make_lead_agent
from langgraph.types import RunnableConfig
config = RunnableConfig(configurable={
"thinking_enabled": False, # 是否启用扩展思考
"model_name": "my-model", # 指定 LLM 模型
"is_plan_mode": False, # 是否启用计划模式（TodoList）
"subagent_enabled": True, # 是否启用子 Agent 委托
})
agent = make_lead_agent(config)
```
### ThreadState 数据模型
`deerflow/agents/thread_state.py`：
| 字段 | 类型 | 说明 |
|------|------|------|
| `messages` | list | LangGraph 消息列表 |
| `sandbox` | SandboxInfo | 当前线程沙箱 |
| `thread_data` | dict | 线程本地数据（路径等） |
| `title` | str | 自动生成的对话标题 |
| `artifacts` | list | 生成的工件文件 |
| `todos` | list | 计划模式下的待办事项 |
| `uploaded_files` | list | 已上传的文件 |
| `viewed_images` | list | 已查看的图像（base64） |
### 子 Agent 系统
- **内置子 Agent**：`general-purpose`（所有工具，除 `task`）、`bash`（命令专家）
- **最大并发**：3 个（`SubagentLimitMiddleware` 强制限制）
- **超时时间**：15 分钟
- **触发方式**：调用 `task` 工具委托任务
- **执行流程**：`task()` → `SubagentExecutor` → 后台线程 → 每 5 秒轮询 → SSE 事件
```python
# 委托子 Agent 执行任务（在 Agent 提示词中由 LLM 自动调用）
task(
description="...",
prompt="分析这份数据并生成报告",
subagent_type="general-purpose",
max_turns=10
)
```
---
## 10. 中间件链
Lead Agent 中间件按固定顺序组装，每个中间件负责一个横切关注点：
| 顺序 | 中间件 | 作用 |
|------|--------|------|
| 1 | `ThreadDataMiddleware` | 创建每线程隔离目录（workspace/uploads/outputs） |
| 2 | `UploadsMiddleware` | 注入新上传的文件到对话 |
| 3 | `SandboxMiddleware` | 获取沙箱实例，存储 sandbox_id |
| 4 | `DanglingToolCallMiddleware` | 为中断的工具调用注入占位 ToolMessage |
| 5 | `LLMErrorHandlingMiddleware` | 将 LLM 调用失败归一化为可恢复错误 |
| 6 | `GuardrailMiddleware` | 工具调用前授权检查（可选） |
| 7 | `SandboxAuditMiddleware` | 沙箱操作安全审计日志 |
| 8 | `ToolErrorHandlingMiddleware` | 将工具异常转为错误 ToolMessage，避免中止 |
| 9 | `SummarizationMiddleware` | 接近 Token 上限时压缩上下文（可选） |
| 10 | `TodoListMiddleware` | 计划模式下的任务追踪（可选） |
| 11 | `TokenUsageMiddleware` | 记录 Token 用量指标（可选） |
| 12 | `TitleMiddleware` | 首次完整交互后自动生成标题 |
| 13 | `MemoryMiddleware` | 将对话加入内存更新队列 |
| 14 | `ViewImageMiddleware` | 调用 LLM 前注入 base64 图像（视觉模型） |
| 15 | `DeferredToolFilterMiddleware` | 隐藏延迟工具 Schema（可选） |
| 16 | `SubagentLimitMiddleware` | 限制最大并发子 Agent 数（可选） |
| 17 | `LoopDetectionMiddleware` | 检测工具调用循环并强制终止 |
| 18 | `ClarificationMiddleware` | 拦截澄清请求，中断到 END（必须最后） |
---
## 11. 沙箱执行系统
### 概述
每个线程拥有独立的沙箱环境，Agent 通过沙箱执行命令和读写文件。
### 虚拟路径映射
Agent 看到的虚拟路径 → 物理路径：
| 虚拟路径 | 物理路径 |
|----------|----------|
| `/mnt/user-data/workspace/` | `backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/workspace/` |
| `/mnt/user-data/uploads/` | `backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/uploads/` |
| `/mnt/user-data/outputs/` | `backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/outputs/` |
| `/mnt/skills/` | `deer-flow/skills/` |
### 沙箱工具
| 工具 | 说明 |
|------|------|
| `bash` | 执行 Shell 命令（含路径转换） |
| `ls` | 目录列表（树形格式，最多 2 层） |
| `read_file` | 读取文件内容（支持行范围） |
| `write_file` | 写入/追加文件（自动创建目录） |
| `str_replace` | 字符串替换（单次或全部） |
### 沙箱提供者
- **LocalSandboxProvider**：单例本地文件系统执行（默认）
- **AioSandboxProvider**：基于 Docker 的隔离执行（需要 Docker）
在 `config.yaml` 中切换：
```yaml
sandbox:
use: deerflow.sandbox.local:LocalSandboxProvider
# 或 Docker 沙箱：
# use: deerflow.community.aio_sandbox:AioSandboxProvider
```
---
## 12. 内存系统
### 功能
- 自动从对话中提取用户偏好、知识和上下文，存储为结构化事实
- 下次对话时自动注入内存上下文到系统提示词
- 每用户独立存储，按 Agent 进一步隔离
### 存储路径
```
backend/.deer-flow/users/{user_id}/memory.json # 默认 Agent 内存
backend/.deer-flow/users/{user_id}/agents/{agent_name}/memory.json # 自定义 Agent 内存
```
### 数据结构
```json
{
"workContext": "用户是一名 Python 开发者...",
"personalContext": "...",
"topOfMind": "...",
"recentMonths": "...",
"facts": [
{
"id": "fact-001",
"content": "用户偏好使用 TypeScript",
"category": "preference",
"confidence": 0.9,
"createdAt": "2025-01-01T00:00:00Z"
}
]
}
```
### 工作流程
1. `MemoryMiddleware` 过滤消息（用户输入 + 最终 AI 回复），捕获 `user_id`，加入更新队列
2. 队列去抖（默认 30 秒），按线程去重
3. 后台线程调用 LLM 提取上下文更新和事实
4. 原子写入（临时文件 + rename），跳过重复事实
5. 下次交互注入前 15 条事实 + 上下文到 `<memory>` 标签
### 配置
```yaml
memory:
enabled: true
injection_enabled: true
debounce_seconds: 30 # 去抖等待时间
model_name: null # null 表示使用默认模型
max_facts: 100 # 最大存储事实数
fact_confidence_threshold: 0.7 # 事实置信度阈值
max_injection_tokens: 2000 # 注入 Token 上限
```
---
## 13. 工具系统
### 工具组装
`get_available_tools(groups, include_mcp, model_name, subagent_enabled)` 按顺序组装：
1. **config.yaml 定义的工具**：通过 `resolve_variable()` 反射加载
2. **MCP 工具**：来自已启用的 MCP 服务器（懒加载，带 mtime 失效）
3. **内置工具**：
- `present_files` — 将输出文件呈现给用户（仅 `/mnt/user-data/outputs`）
- `ask_clarification` — 请求澄清（由 `ClarificationMiddleware` 拦截，中断对话）
- `view_image` — 读取图像为 base64（仅视觉模型）
4. **子 Agent 工具**（启用时）：`task` — 委托子 Agent
### 社区工具
| 工具 | 模块 | 说明 |
|------|------|------|
| Tavily 搜索 | `deerflow.community.tavily` | Web 搜索（默认 5 条结果） |
| Tavily 获取 | `deerflow.community.tavily` | Web 页面获取（4KB 限制） |
| Jina AI | `deerflow.community.jina_ai` | 通过 Jina reader API 获取页面 |
| Firecrawl | `deerflow.community.firecrawl` | 网页爬取 |
| 图像搜索 | `deerflow.community.image_search` | DuckDuckGo 图像搜索 |
### MCP 集成
- 使用 `langchain-mcp-adapters` 的 `MultiServerMCPClient`
- 懒加载：首次使用时初始化
- 配置变更自动失效（mtime 检测）
- 支持 stdio、SSE、HTTP 传输
- 支持 OAuth 令牌刷新（HTTP/SSE）
---
## 14. IM 频道集成
### 消息流程
```
外部平台消息
↓
频道实现（feishu.py / slack.py 等）
↓
MessageBus.publish_inbound()
↓
ChannelManager._dispatch_loop()
↓
查找/创建线程（通过 Gateway API）
↓
执行 Agent 运行（stream 或 wait）
↓
OutboundMessage → 平台回复
```
### 各平台特性
| 平台 | 特性 |
|------|------|
| 飞书 | 增量更新，修改同一消息卡片（`update_multi=true`） |
| Slack | 阻塞等待最终结果 |
| Telegram | 阻塞等待最终结果 |
| 钉钉 | 支持 AI 卡片流式更新（需配置 `card_template_id`） |
### 配置（`config.yaml` → `channels`）
```yaml
channels:
langgraph_url: http://localhost:8001/api # LangGraph 兼容 API 基础 URL
gateway_url: http://localhost:8001 # Gateway API URL
feishu:
app_id: $FEISHU_APP_ID
app_secret: $FEISHU_APP_SECRET
slack:
bot_token: $SLACK_BOT_TOKEN
app_token: $SLACK_APP_TOKEN
telegram:
bot_token: $TELEGRAM_BOT_TOKEN
dingtalk:
client_id: $DINGTALK_CLIENT_ID
client_secret: $DINGTALK_CLIENT_SECRET
card_template_id: "" # 可选，启用 AI 卡片流式
```
> **Docker 部署注意**：IM 频道运行在 `gateway` 容器内，应将 `localhost` 改为服务名，例如 `langgraph_url: http://gateway:8001/api`。
---
## 15. 数据持久化
### 数据库
默认使用 SQLite，存放在 `backend/.deer-flow/data/`。可切换为 PostgreSQL：
```yaml
database:
backend: postgres
postgres_url: postgresql://user:pass@localhost:5432/deerflow
```
### 主要数据表
| 表名 | 说明 |
|------|------|
| `agents` | 自定义 Agent 元数据（id、name、user_id、published） |
| `users` | 用户账号 |
| `thread_meta` | 线程元数据（标题、创建时间等） |
| `feedback` | 运行反馈和评分 |
| `scheduled_tasks` | 定时任务配置 |
### 数据库迁移
使用 Alembic 管理 Schema 变更，应用启动时自动执行：
```bash
# 查看迁移历史
PYTHONPATH=. uv run alembic history
# 手动升级到最新
PYTHONPATH=. uv run alembic upgrade head
```
### 运行时文件目录
```
backend/.deer-flow/
├── data/ # SQLite 数据库
└── users/{user_id}/
├── memory.json # 个人内存（默认 Agent）
├── agents/{agent_name}/memory.json # 自定义 Agent 内存
└── threads/{thread_id}/user-data/
├── workspace/ # Agent 工作目录
├── uploads/ # 上传的文件
└── outputs/ # 生成的输出文件
```
---
## 16. 测试
### 测试套件
```bash
# 运行所有测试
make test
# 运行特定测试文件
PYTHONPATH=. uv run pytest tests/test_memory_updater.py -v
# 运行特定测试函数
PYTHONPATH=. uv run pytest tests/test_client.py::TestGatewayConformance -v
```
### 关键测试文件
| 文件 | 说明 |
|------|------|
| `tests/test_harness_boundary.py` | 确保 harness 层不导入 app 层（CI 必须通过） |
| `tests/test_memory_updater.py` | 内存提取和去重逻辑 |
| `tests/test_client.py` | 嵌入式客户端 77 个单元测试，含 Gateway 一致性验证 |
| `tests/test_client_live.py` | 需要 config.yaml 的集成测试 |
| `tests/test_docker_sandbox_mode_detection.py` | Docker 沙箱模式检测回归测试 |
| `tests/test_provisioner_kubeconfig.py` | kubeconfig 处理回归测试 |
| `tests/conftest.py` | pytest fixtures 和通用配置 |
### TDD 原则
**每个新功能或 Bug 修复必须附带单元测试**，无例外。
- 测试文件命名：`tests/test_<feature>.py`
- 轻量配置/工具模块优先写不依赖外部服务的纯单元测试
- 如果模块导致循环导入问题，在 `tests/conftest.py` 中添加 `sys.modules` mock
---
## 17. 常见问题与排错
### 启动失败：找不到 config.yaml
确认 `config.yaml` 在项目根目录或 `backend/` 目录中存在。可用环境变量指定路径：
```bash
export DEER_FLOW_CONFIG_PATH=/path/to/config.yaml
make dev
```
### 模型调用失败
1. 确认 `config.yaml` 中对应模型的 `api_key` 环境变量已设置
2. 确认 `base_url` 可访问
3. 检查模型提供者包是否安装：缺少时启动日志会给出 `uv add langchain-xxx` 提示
### 内存功能不工作
检查 `config.yaml` 中：
```yaml
memory:
enabled: true
injection_enabled: true
```
### 文件上传后 Agent 无法使用
确认文件上传到正确线程，且 `UploadsMiddleware` 处于启用状态（默认启用）。上传文件存放在 `/mnt/user-data/uploads/`，Agent 可直接访问。
### Windows 下端口占用
```bash
# 查找占用 8001 端口的进程
netstat -ano | findstr :8001
# 终止进程（替换 PID）
taskkill /PID <PID> /F
```
详见 `docs/BACKEND_RUN_STOP_ZH.md`（Windows 启停专项指南）。
### 如何新增 API 路由
1. 在 `app/gateway/routers/` 新建路由文件
2. 在 `app/gateway/app.py` 中导入并挂载路由
3. 编写对应测试文件 `tests/test_<feature>.py`
### 如何新增工具
1. 在 `packages/harness/deerflow/community/` 或 `tools/builtins/` 中实现工具函数
2. 在 `config.yaml` 中的 `tools` 列表添加配置，指定 `use` 类路径和 `group`
3. 工具会在下次 Agent 初始化时自动加载
---
*文档最后更新：2026-05-13*