From cb2edeadb46f60622c3e180b1f3a5009c1933fea Mon Sep 17 00:00:00 2001 From: fengxiangdebaozi <10845145+fengxiangdebaozi@user.noreply.gitee.com> Date: Wed, 13 May 2026 16:04:05 +0800 Subject: [PATCH] =?UTF-8?q?=E6=96=87=E6=A1=A3=E4=BF=AE=E6=94=B9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CLAUDE.md | 672 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 672 insertions(+) create mode 100644 CLAUDE.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..7d9748c7 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,672 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +DeerFlow is a full-stack AI agent system combining: +- **Frontend**: React/Vite web interface for interactive agent conversations +- **Backend**: LangGraph-based agent runtime with sandbox execution, persistent memory, and subagent delegation + +This is a monorepo with frontend and backend components running in parallel. + +--- + +## Architecture + +``` +┌────────────────────────────────────────────────────────┐ +│ DeerFlow Application │ +├────────────────────────────────────────────────────────┤ +│ Nginx (Port 2026) │ +│ (Unified reverse proxy) │ +└─────────────────┬──────────────────┬──────────────────┘ + │ │ + /api/langgraph/* (Gateway) / (Frontend + /api/*) + │ │ + ┌──────▼──────────┐ ┌─────────▼────────────┐ + │ Gateway API │ │ Frontend Web UI │ + │ (Port 8001) │ │ (Port 3000) │ + │ │ │ │ + │ - LangGraph │ │ - React 19 + Vite │ + │ - FastAPI │ │ - TypeScript │ + │ - Tools │ │ - Tailwind CSS │ + │ - Memory │ │ │ + └─────────────────┘ └──────────────────────┘ + │ + ┌───────▼──────────────────┐ + │ Agent Runtime (LangGraph)│ + │ ┌──────────────────────┐ │ + │ │ Lead Agent │ │ + │ │ ┌────────────────┐ │ │ + │ │ │ Middlewares │ │ │ + │ │ │ (10 total) │ │ │ + │ │ └────────────────┘ │ │ + │ │ ┌────────────────┐ │ │ + │ │ │ Tools │ │ │ + │ │ │ - Sandbox │ │ │ + │ │ │ - MCP │ │ │ + │ │ │ - Built-in │ │ │ + │ │ │ - Subagents │ │ │ + │ │ └────────────────┘ │ │ + │ └──────────────────────┘ │ + └───────────────────────────┘ +``` + +**Request Routing**: +- `/api/langgraph/*` → Gateway (LangGraph runtime) +- `/api/*` (other) → Gateway (FastAPI REST endpoints) +- `/` → Frontend (React web interface) + +--- + +## Project Structure + +``` +deerflow-server-master/ +├── CLAUDE.md # This file (root guidance) +├── .gitignore # Git ignore rules +│ +├── frontend-web/ # React/Vite web interface +│ ├── package.json # Frontend dependencies +│ ├── package-lock.json +│ ├── vite.config.ts # Vite bundler config +│ ├── tsconfig.json # TypeScript config +│ ├── tailwind.config.ts # Tailwind CSS config +│ ├── postcss.config.js # PostCSS config +│ ├── index.html # HTML entry point +│ └── src/ +│ ├── App.tsx # Root component +│ ├── main.tsx # React DOM mount +│ ├── components/ +│ │ ├── ai-elements/ # Agent UI components (message, artifact, plan, etc.) +│ │ ├── landing/ # Landing page components +│ │ ├── ui/ # Base UI components (shadcn/ui styled) +│ │ ├── workspace/ # Main workspace interface +│ │ ├── query-client-provider.tsx +│ │ └── theme-provider.tsx +│ ├── hooks/ # Custom React hooks +│ ├── lib/ # Utilities (HTTP client, formatting) +│ ├── pages/ # Route pages +│ └── styles/ # Global CSS +│ +└── offline-backend-20260512/backend/ # Python FastAPI + LangGraph backend + ├── CLAUDE.md # Detailed backend guidance + ├── README.md # Backend documentation + ├── Makefile # Backend commands + ├── pyproject.toml # Python dependencies + workspace + ├── langgraph.json # LangGraph Studio configuration + │ + ├── packages/harness/ # deerflow-harness: Publishable agent framework + │ ├── pyproject.toml + │ └── deerflow/ + │ ├── agents/ # LangGraph agent system + │ ├── sandbox/ # Code execution isolation + │ ├── subagents/ # Task delegation system + │ ├── tools/ # Tool ecosystem + │ ├── models/ # LLM model factory + │ ├── mcp/ # Model Context Protocol integration + │ ├── skills/ # Skills discovery & loading + │ ├── config/ # Configuration system + │ ├── community/ # Community tools (Tavily, Jina, Firecrawl) + │ ├── reflection/ # Dynamic module loading + │ └── client.py # Embedded DeerFlowClient + │ + ├── app/ # Unpublished application code + │ ├── gateway/ # FastAPI Gateway API + │ │ ├── app.py # FastAPI application + │ │ └── routers/ # REST endpoints + │ └── channels/ # IM integrations (Feishu, Slack, Telegram, DingTalk) + │ + ├── tests/ # Comprehensive test suite + ├── docs/ # Detailed documentation + └── .deer-flow/ # Runtime directory (created at startup) + ├── agents/ # Custom agent definitions + ├── users/ # Per-user data + │ └── {user_id}/ + │ ├── memory.json # Memory system + │ ├── threads/ # Per-thread data + │ └── agents/ # Per-agent memory + └── threads/ # Thread-specific workspace (legacy) +``` + +--- + +## Setup & Installation + +### Prerequisites + +- Node.js 18+ and pnpm +- Python 3.12+ +- uv package manager +- API keys for LLM providers (OpenAI, Anthropic, etc.) + +### Quick Start (Full Application) + +```bash +cd offline-backend-20260512/backend + +# Install all dependencies +make install + +# Create config from template +cp config.example.yaml config.yaml + +# Set your API keys +export OPENAI_API_KEY="your-key-here" +# ... other API keys as needed + +# Start all services +make dev +``` + +Access at: **http://localhost:2026** + +--- + +## Development Commands + +### Full Application (from backend directory) + +```bash +make install # Install frontend + backend dependencies +make dev # Start Gateway + Frontend + Nginx (port 2026) +make gateway # Run Gateway API only (port 8001) +make test # Run all tests +make lint # Lint code +make format # Format code with ruff +``` + +### Frontend Only (from frontend-web directory) + +```bash +pnpm install # Install frontend dependencies +pnpm dev # Start Vite dev server (port 3000) +pnpm build # Build for production +pnpm preview # Preview production build +pnpm typecheck # TypeScript type checking +``` + +### Backend Only (from backend directory) + +```bash +make install # Install dependencies +make dev # Run Gateway API with auto-reload (port 8001) +make gateway # Run Gateway API (no reload) +make test # Run tests +make lint # Check code style +make format # Auto-format code +``` + +--- + +## Architecture Highlights + +### Frontend (`frontend-web/`) + +- **Framework**: React 19 with TypeScript +- **Bundler**: Vite for fast dev/build +- **Styling**: Tailwind CSS + shadcn/ui components +- **State Management**: React Query (TanStack Query) + local state +- **Routing**: React Router v7 +- **Key Libraries**: + - `@xyflow/react` - Node/edge visualization for agent workflows + - `codemirror` - Code editing with language support + - `sonner` - Toast notifications + - `motion` - Animation library + - `katex` - Math rendering + - `remark-gfm` + `rehype` - Markdown processing + +**Main Components**: +- `components/ai-elements/` - Agent-specific UI (message, artifact, plan, canvas, etc.) +- `components/ui/` - Reusable shadcn/ui-style components +- `components/workspace/` - Main workspace interface +- `components/landing/` - Public landing page + +### Backend (`offline-backend-20260512/backend/`) + +The backend is split into two layers: + +1. **Harness** (`packages/harness/deerflow/`) + - Publishable agent framework package + - Contains: agents, sandbox, tools, models, MCP, skills, config + - Import prefix: `from deerflow.*` + - Can be published as `deerflow-harness` PyPI package + +2. **App** (`app/`) + - Unpublished application code + - Contains: FastAPI Gateway API, IM channel integrations + - Import prefix: `from app.*` + - **Boundary rule**: App imports Harness, but Harness never imports App + +**Key Systems**: +- **Lead Agent** - Single LangGraph agent with 10 middleware components +- **Sandbox** - Per-thread isolated execution (local filesystem or Docker) +- **Subagents** - Background task delegation with 3-concurrent limit +- **Memory** - LLM-powered persistent context across conversations +- **Tools** - Sandbox (bash, read/write), MCP, community (web search), built-in (clarification, image view) +- **Skills** - Domain-specific workflows injected via system prompt +- **MCP** - Model Context Protocol server integration + +**For detailed backend architecture, see `offline-backend-20260512/backend/CLAUDE.md`** + +--- + +## Configuration + +### Main Configuration (`config.yaml`) + +Place in project root (same level as this file). See `offline-backend-20260512/backend/config.example.yaml` for full template. + +Key sections: +- `models` - LLM configurations (OpenAI, Anthropic, etc.) with thinking/vision flags +- `tools` - Tool definitions with module paths +- `sandbox` - Execution environment provider +- `skills` - Skills directory paths +- `memory` - Memory system (enabled, storage, debounce) +- `title` - Auto-title generation +- `summarization` - Context summarization +- `subagents` - Task delegation (enabled/disabled) + +Config values starting with `$` are resolved as environment variables (e.g., `$OPENAI_API_KEY`). + +### Extensions Configuration (`extensions_config.json`) + +Place in project root. Configures MCP servers and skill states: + +```json +{ + "mcpServers": { + "github": { + "enabled": true, + "type": "stdio", + "command": "npx", + "args": ["-y", "@modelcontextprotocol/server-github"], + "env": {"GITHUB_TOKEN": "$GITHUB_TOKEN"} + } + }, + "skills": { + "pdf-processing": {"enabled": true} + } +} +``` + +### Environment Variables + +```bash +# LLM API Keys +export OPENAI_API_KEY="sk-..." +export ANTHROPIC_API_KEY="sk-ant-..." + +# Tool API Keys +export TAVILY_API_KEY="tvly-..." +export GITHUB_TOKEN="ghp_..." + +# Configuration paths +export DEER_FLOW_CONFIG_PATH="./config.yaml" +export DEER_FLOW_EXTENSIONS_CONFIG_PATH="./extensions_config.json" + +# Tracing (optional) +export LANGSMITH_TRACING=true +export LANGSMITH_API_KEY="lsv2_pt_..." +export LANGSMITH_PROJECT="myproject" +``` + +--- + +## Important Development Guidelines + +### Documentation Updates + +**CRITICAL**: Always update relevant documentation after code changes: +- Update `offline-backend-20260512/backend/README.md` for user-facing changes +- Update `offline-backend-20260512/backend/CLAUDE.md` for internal changes +- Keep documentation synchronized with code + +### Testing + +**Test-Driven Development (TDD) is mandatory** for backend: +```bash +cd offline-backend-20260512/backend + +# Run all tests +make test + +# Run specific test +PYTHONPATH=. uv run pytest tests/test_.py -v +``` + +Frontend testing is lighter-weight but should be done for critical components. + +### Code Style + +- **Backend**: `ruff` (line length: 240, Python 3.12+, double quotes) +- **Frontend**: TypeScript with eslint/prettier defaults + +### Architecture Boundaries + +The **Harness ↔ App boundary** (enforced by CI tests): +```python +# ✅ Allowed (App → Harness) +from deerflow.agents import make_lead_agent + +# ❌ Forbidden (Harness → App) +from app.gateway.routers import ... # This will fail CI tests +``` + +--- + +## Key Features + +### 1. Sandbox Execution + +Per-thread isolated code execution with virtual path mapping: +- **Local**: Filesystem-based execution (default) +- **Docker**: Aio sandbox provider for complete isolation +- **Tools**: `bash`, `ls`, `read_file`, `write_file`, `str_replace` +- **Paths**: `/mnt/user-data/{workspace,uploads,outputs}` → thread-specific physical directories + +### 2. Persistent Memory + +LLM-powered context retention across conversations: +- Automatic extraction of facts and preferences +- Structured storage with confidence scoring +- Debounced updates to minimize LLM calls +- Injected into agent system prompt for context awareness + +See `offline-backend-20260512/backend/docs/memory-improvements.md` for details. + +### 3. Subagent Delegation + +Background task execution with concurrent limits: +- Built-in agents: `general-purpose` (full toolset) and `bash` (commands) +- Max 3 concurrent subagents per turn +- 15-minute timeout per task +- Real-time status updates via SSE + +### 4. Tool Ecosystem + +Modular tool system supporting multiple providers: +- **Sandbox tools**: Code execution in isolated environments +- **Built-in tools**: `ask_clarification`, `view_image`, `present_files` +- **MCP tools**: Any Model Context Protocol server (stdio, SSE, HTTP) +- **Community tools**: Tavily (web search), Jina (web fetch), Firecrawl (scraping) +- **Skills**: Domain-specific workflows loaded from SKILL.md files + +### 5. Model Context Protocol (MCP) + +Integrate external tools and services: +- **Transports**: stdio, SSE, HTTP (with OAuth support) +- **Configuration**: `extensions_config.json` for server setup +- **Runtime updates**: Changes detected and reloaded automatically + +### 6. Plan Mode + +TodoList middleware for complex multi-step tasks: +- Controlled via runtime config: `is_plan_mode = True` +- Provides `write_todos` tool for task tracking +- Real-time task status updates + +See `offline-backend-20260512/backend/docs/plan_mode_usage.md`. + +### 7. Vision Support + +For models with `supports_vision: true`: +- `ViewImageMiddleware` processes images automatically +- `view_image_tool` added to agent toolset +- Images converted to base64 and injected into state + +### 8. File Upload + +Multi-file upload with automatic document conversion: +- Supports: PDF, PPT, Excel, Word documents +- Auto-conversion via `markitdown` +- Endpoint: `POST /api/threads/{thread_id}/uploads` +- Files stored in thread-isolated directories + +See `offline-backend-20260512/backend/docs/FILE_UPLOAD.md`. + +### 9. IM Channel Integration + +External messaging platform bridges: +- **Supported**: Feishu, Slack, Telegram, DingTalk +- **Architecture**: Channels use `langgraph-sdk` HTTP client +- **Streaming**: Feishu updates single card in place; Slack/Telegram use `wait()` response +- See `offline-backend-20260512/backend/app/channels/` for implementation + +### 10. Context Summarization + +Automatic conversation compression when approaching token limits: +- Configurable trigger types: tokens, messages, or token fraction +- Keeps recent messages while summarizing older ones +- See `offline-backend-20260512/backend/docs/summarization.md`. + +--- + +## Middleware Chain (Backend) + +The agent runtime executes 18 middleware components in strict order: + +1. **ThreadDataMiddleware** - Per-thread isolated directories +2. **UploadsMiddleware** - Inject uploaded files +3. **SandboxMiddleware** - Acquire sandbox environment +4. **DanglingToolCallMiddleware** - Handle incomplete tool responses +5. **LLMErrorHandlingMiddleware** - Normalize provider errors +6. **GuardrailMiddleware** - Tool call authorization (optional) +7. **SandboxAuditMiddleware** - Security audit logging +8. **ToolErrorHandlingMiddleware** - Convert exceptions to error messages +9. **SummarizationMiddleware** - Context reduction (optional) +10. **TodoListMiddleware** - Task tracking (optional, plan mode) +11. **TokenUsageMiddleware** - Token tracking (optional) +12. **TitleMiddleware** - Auto-generate conversation titles +13. **MemoryMiddleware** - Queue for async memory updates +14. **ViewImageMiddleware** - Inject vision image data (conditional) +15. **DeferredToolFilterMiddleware** - Hide deferred tools (optional) +16. **SubagentLimitMiddleware** - Enforce concurrent subagent limit (optional) +17. **LoopDetectionMiddleware** - Detect and break tool-call loops +18. **ClarificationMiddleware** - Interrupt on clarification requests (must be last) + +See `offline-backend-20260512/backend/CLAUDE.md` for detailed middleware descriptions. + +--- + +## Common Workflows + +### Adding a New Tool + +1. Define tool in `packages/harness/deerflow/tools/` or `packages/harness/deerflow/community/` +2. Add to `config.yaml` `tools` section with module path +3. Write unit tests in `backend/tests/` +4. Update `CLAUDE.md` with tool description +5. Test via `make test` + +### Creating a Custom Agent + +1. Add agent metadata to database (`agents` table) +2. Create agent directory: `.deer-flow/agents/{agent_id}/SOUL.md` +3. Define agent behavior via SOUL.md frontmatter +4. Access via Gateway API: `GET /api/agents/{id}` + +### Configuring MCP Servers + +1. Add server to `extensions_config.json`: +```json +{ + "mcpServers": { + "my-server": { + "enabled": true, + "type": "stdio", + "command": "npx", + "args": ["-y", "my-mcp-package"] + } + } +} +``` + +2. Restart Gateway API - MCP tools auto-loaded via lazy initialization +3. Tools available immediately in agent toolset + +### Testing in Development + +```bash +# Backend unit tests +cd offline-backend-20260512/backend +make test + +# Run specific test +PYTHONPATH=. uv run pytest tests/test_memory_updater.py -v + +# Frontend development +cd frontend-web +pnpm dev # Access at http://localhost:3000 +``` + +--- + +## Port Mapping + +| Service | Port | Purpose | +|---------|------|---------| +| **Frontend** | 3000 | React development server | +| **Gateway API** | 8001 | FastAPI REST endpoints + LangGraph runtime | +| **LangGraph Server** | 2024 | Raw LangGraph protocol (used internally) | +| **Nginx** | 2026 | Unified reverse proxy (public entry point) | +| **Provisioner** | 8002 | Kubernetes sandbox provisioner (optional, Docker dev only) | + +--- + +## Important Files & References + +### Backend Architecture +- `offline-backend-20260512/backend/CLAUDE.md` - Detailed backend guidance +- `offline-backend-20260512/backend/README.md` - User-facing backend documentation +- `offline-backend-20260512/backend/docs/ARCHITECTURE.md` - Deep-dive architecture +- `offline-backend-20260512/backend/docs/API.md` - Gateway API reference +- `offline-backend-20260512/backend/docs/CONFIGURATION.md` - Config schema + +### Frontend Components +- `frontend-web/src/components/ai-elements/` - Agent UI components +- `frontend-web/src/components/ui/` - Reusable UI components +- `frontend-web/src/components/workspace/` - Main workspace + +### Configuration +- `config.example.yaml` - Backend configuration template +- `extensions_config.json` - MCP servers and skills +- `offline-backend-20260512/backend/langgraph.json` - LangGraph Studio config + +### Testing +- `offline-backend-20260512/backend/tests/` - Test suite +- Regression tests: `test_docker_sandbox_mode_detection.py`, `test_provisioner_kubeconfig.py`, `test_harness_boundary.py` + +--- + +## Troubleshooting + +### Port Already in Use + +If a port is already in use, kill the process: +```bash +# Find process on port (e.g., 8001) +lsof -i :8001 + +# Kill process +kill -9 +``` + +### Configuration Issues + +1. Verify `config.yaml` exists in project root +2. Run: `make config-upgrade` to auto-merge missing fields +3. Check `config_version` in `config.yaml` matches `config.example.yaml` + +### Memory System Issues + +Check memory state: +```bash +# List memory files +ls -la offline-backend-20260512/backend/.deer-flow/users/ + +# Reload memory +curl -X POST http://localhost:8001/api/memory/reload +``` + +### Test Failures + +Ensure the test dependency boundary is not violated: +```bash +# Run boundary test specifically +cd offline-backend-20260512/backend +PYTHONPATH=. uv run pytest tests/test_harness_boundary.py -v +``` + +--- + +## Technology Stack + +### Frontend +- **React** 19 - UI framework +- **TypeScript** - Type safety +- **Vite** - Fast bundler +- **Tailwind CSS** - Utility-first styling +- **React Router** - Client routing +- **TanStack Query** - Data fetching +- **XYFlow** - Graph visualization +- **CodeMirror** - Code editor +- **Shiki** - Syntax highlighting +- **Remark/Rehype** - Markdown processing + +### Backend +- **LangGraph** 1.0.6+ - Agent orchestration framework +- **LangChain** 1.2.3+ - LLM abstractions and tools +- **FastAPI** 0.115.0+ - REST API framework +- **Uvicorn** - ASGI server +- **Python 3.12+** - Programming language +- **uv** - Fast Python package manager +- **Ruff** - Linting and formatting +- **Pytest** - Testing framework +- **langchain-mcp-adapters** - Model Context Protocol support +- **markitdown** - Document conversion +- **Tavily/Jina/Firecrawl** - Web tools + +--- + +## Contributing + +1. Create a feature branch from `master` +2. Make changes and write tests +3. Run linting and tests: + ```bash + make lint + make test + ``` +4. Update relevant CLAUDE.md and README.md files +5. Create a pull request + +### CI/CD + +GitHub Actions runs: +- Backend unit tests (including harness boundary test) +- Linting checks +- Type checking + +All must pass before merging to `master`. + +--- + +## Support + +For issues and feature requests: +- Check the detailed docs in `offline-backend-20260512/backend/docs/` +- Review existing tests for usage examples +- Ask Claude Code for help with `/help` command + +--- + +## License & Attribution + +See LICENSE file in project root. + +For more details on specific systems, see: +- Backend: `offline-backend-20260512/backend/CLAUDE.md` +- Backend architecture: `offline-backend-20260512/backend/docs/ARCHITECTURE.md` +- Frontend: Components in `frontend-web/src/components/`