Merge branch 'master' of http://192.168.0.200/knowledge-base-bj/deerflow-server into xuhd
This commit is contained in:
commit
48ddbf0412
423 files changed
+117115
-3214
No files matched your search
@@ -173,6 +173,17 @@ Lead-agent middlewares are assembled in strict append order across `packages/har
|
||||
16. **SubagentLimitMiddleware** - Truncates excess `task` tool calls from model response to enforce `MAX_CONCURRENT_SUBAGENTS` limit (optional, if `subagent_enabled`)
|
||||
17. **LoopDetectionMiddleware** - Detects repeated tool-call loops; hard-stop responses clear both structured `tool_calls` and raw provider tool-call metadata before forcing a final text answer
|
||||
18. **ClarificationMiddleware** - Intercepts `ask_clarification` tool calls, interrupts via `Command(goto=END)` (must be last)
|
||||
9. **PromptPrefixMiddleware** - Injects the admin-configured prompt prefix into the model request only. The prefix is tagged onto the first human message's `additional_kwargs['prompt_prefix']` by the Gateway (`app.gateway.services`) and prepended to content here at call time, so the persisted/displayed message stays prefix-free
|
||||
10. **SummarizationMiddleware** - Context reduction when approaching token limits (optional, if enabled)
|
||||
11. **TodoListMiddleware** - Task tracking with `write_todos` tool (optional, if plan_mode)
|
||||
12. **TokenUsageMiddleware** - Records token usage metrics when token tracking is enabled (optional)
|
||||
13. **TitleMiddleware** - Auto-generates thread title after first complete exchange and normalizes structured message content before prompting the title model
|
||||
14. **MemoryMiddleware** - Queues conversations for async memory update (filters to user + final AI responses)
|
||||
15. **ViewImageMiddleware** - Injects base64 image data before LLM call (conditional on vision support)
|
||||
16. **DeferredToolFilterMiddleware** - Hides deferred tool schemas from the bound model until tool search is enabled (optional)
|
||||
17. **SubagentLimitMiddleware** - Truncates excess `task` tool calls from model response to enforce `MAX_CONCURRENT_SUBAGENTS` limit (optional, if `subagent_enabled`)
|
||||
18. **LoopDetectionMiddleware** - Detects repeated tool-call loops; hard-stop responses clear both structured `tool_calls` and raw provider tool-call metadata before forcing a final text answer
|
||||
19. **ClarificationMiddleware** - Intercepts `ask_clarification` tool calls, interrupts via `Command(goto=END)` (must be last)
|
||||
|
||||
### Configuration System
|
||||
|
||||
@@ -216,6 +227,10 @@ FastAPI application on port 8001 with health check at `GET /health`. Set `GATEWA
|
||||
| **Skills** (`/api/skills`) | `GET /` - list skills; `GET /{name}` - details; `PUT /{name}` - update enabled; `POST /install` - install from .skill archive (accepts standard optional frontmatter like `version`, `author`, `compatibility`) |
|
||||
| **Memory V2** (`/api/memory/v2`) | `GET /` - builtin data (USER.md/MEMORY.md); `POST/PUT/DELETE /entries` - add/replace/remove entry; `GET/PATCH/DELETE /config` - effective config + per-user overrides; `POST /search` - Hindsight recall/reflect proxy |
|
||||
| **Agents** (`/api/agents`) | `GET /` - list built-in, owned, and published agents; `POST /` - create user-owned agent; `GET/PUT/DELETE /{id}` - read/update/delete by stable agent id; `GET /check` - validate non-empty display name (duplicates allowed) |
|
||||
| **Skills** (`/api/skills`) | `GET /` - list skills (optional `search` fuzzy-name + `tag_id` filters; each item carries `tags`); `GET /{name}` - details; `PUT /{name}` - update enabled; `POST /install` - install from .skill archive (accepts standard optional frontmatter like `version`, `author`, `compatibility`) |
|
||||
| **Memory** (`/api/memory`) | `GET /` - memory data; `POST /reload` - force reload; `GET /config` - config; `GET /status` - config + data |
|
||||
| **Agents** (`/api/agents`) | `GET /` - list built-in, owned, and published agents (optional `search` + `tag_id` filters; each item carries `tags`); `POST /` - create user-owned agent; `GET/PUT/DELETE /{id}` - read/update/delete by stable agent id; `GET /check` - validate non-empty display name (duplicates allowed) |
|
||||
| **Tags** (`/api/tags`) | `GET /` - list tags (optional `tag_type` filter; open to all); `POST /` - create tag (admin); `PUT /{id}` / `DELETE /{id}` - update/delete tag (admin); `GET /assignments/{target_type}/{target_id}` - tags on a resource; `PUT /assignments/{target_type}/{target_id}` - replace a resource's tags (admin). Tags are typed `agent`/`skill`/`scheduled_task` and may only tag resources of their own type. |
|
||||
| **Uploads** (`/api/threads/{id}/uploads`) | `POST /` - upload files (auto-converts PDF/PPT/Excel/Word); `GET /list` - list; `DELETE /{filename}` - delete |
|
||||
| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; unexpected failures are logged server-side and return a generic 500 detail |
|
||||
| **Artifacts** (`/api/threads/{id}/artifacts`) | `GET /{path}` - serve artifacts; active content types (`text/html`, `application/xhtml+xml`, `image/svg+xml`) are always forced as download attachments to reduce XSS risk; `?download=true` still forces download for other file types |
|
||||
@@ -300,6 +315,55 @@ Custom agent metadata lives in the `agents` ORM table (`deerflow.persistence.age
|
||||
- **Injection**: Enabled skills listed in agent system prompt with container paths
|
||||
- **Installation**: `POST /api/skills/install` extracts .skill ZIP archive to custom/ directory
|
||||
|
||||
#### Skill Curator (`packages/harness/deerflow/skills/curator.py`)
|
||||
|
||||
Background skill lifecycle manager. The Gateway `lifespan` starts a resident
|
||||
scheduler task (`_curator_scheduler_loop` in `app/gateway/app.py`) that — after a
|
||||
5-minute startup delay — calls `maybe_run_curator()` every hour. A cross-process
|
||||
file lock (`.curator_scheduler.lock`) elects a single leader worker so the
|
||||
scheduler runs once per deployment. Per run:
|
||||
|
||||
- **Master switch** — `skill_evolution.enabled` (default **`false`**) gates the
|
||||
whole feature; `curator.enabled` (default **`false`**) gates the curator
|
||||
specifically. Both must be `true` for the curator to act. `should_run_now()`
|
||||
and `_curator_thread_target()` both enforce this — so manual `POST
|
||||
/api/curator/run` also respects the master switch. Config is
|
||||
`config.yaml.skill_evolution` deep-merged with per-user
|
||||
`users/{user_id}/skill_evolution.json`.
|
||||
- **Per-user run lock** — `.curator_state` carries `running` / `started_at`;
|
||||
`_try_acquire_run_lock()` prevents the same user's curator from running
|
||||
concurrently (scheduler vs. manual trigger, or double manual clicks). A lock
|
||||
whose `started_at` is older than `_CURATOR_STALE_LOCK_HOURS` (6h) is treated as
|
||||
process-crash residue and can be taken over.
|
||||
- **Concurrency cap** — `maybe_run_curator()` collects all due users and hands
|
||||
them to `_dispatch_curator_runs()`, which runs them through a
|
||||
`ThreadPoolExecutor` capped at `_CURATOR_MAX_CONCURRENCY` (5) — avoids a
|
||||
thundering herd of LLM calls when many users come due at once (e.g. first
|
||||
deploy).
|
||||
- **Failure back-off** — `_release_run_lock()` always stamps `last_attempt_at`.
|
||||
When a first run keeps crashing (`last_run_at` never set), `should_run_now()`
|
||||
retries on `_CURATOR_RETRY_HOURS` (6h) instead of every hourly tick.
|
||||
- **`should_run_now()`** is the real gate: the hourly tick only checks "is the
|
||||
per-user `interval_hours` (default 168h) elapsed". Timestamp parsing tolerates
|
||||
naive (tz-less) strings — a corrupt `last_run_at` triggers a re-run rather than
|
||||
a permanent deadlock.
|
||||
- **Published-skill filter** — `SkillStore.list_published_names()` is a batch
|
||||
query; the curator fetches the published-name set once per run instead of one
|
||||
DB round-trip per candidate skill.
|
||||
- **LLM review timeout** — `_run_llm_review()` runs `agent.invoke()` in a worker
|
||||
thread bounded by `curator.review_timeout_seconds` (default 600s). The model's
|
||||
per-request `request_timeout` can't bound a multi-turn React agent loop, so on
|
||||
timeout the run is abandoned (a warning is written to `curator_log`, visible in
|
||||
the Web UI) and the curator finishes cleanly + releases the run lock. The
|
||||
worker context is copied via `contextvars.copy_context()` so tool calls still
|
||||
resolve the correct `user_id`.
|
||||
- **Review model** — `_run_llm_review()` uses `curator.model_name` (a
|
||||
`config.yaml` `models[]` entry name) if set, else the primary model; it is
|
||||
editable from the Web UI evolution panel. The skill security scanner
|
||||
(`scan_skill_content` in `security_scanner.py`) reuses the same
|
||||
`skill_evolution.curator.model_name` setting — review and moderation share one
|
||||
model. There is no separate moderation-model config field.
|
||||
|
||||
### Model Factory (`packages/harness/deerflow/models/factory.py`)
|
||||
|
||||
- `create_chat_model(name, thinking_enabled)` instantiates LLM from config via reflection
|
||||
|
||||
Reference in new issue
Block a user