sync update from LAN repo (source master-yx @ cf505220)
This commit is contained in:
1 parent
e243ec3a46
commit
19d3cb96dc
114 files changed
+14787
-451
No files matched your search
@@ -299,6 +299,8 @@ from deerflow.config import get_app_config
|
||||
|
||||
**Forced-research built-in agent.** `forced-research-responder` (「强制检索输出助手」) is seeded from `app/gateway/routers/_forced_research_seed_assets/` before `_sync_legacy_agents`. `ForcedResearchMiddleware` is mounted only for this stable id. On every visible user turn it requires a successful information-gathering tool result before the model may finish; `tool_choice=required` is backed by an `after_model -> model` retry for gateways that ignore tool choice, with an eight-attempt failure cap that permits only an honest failure response. Skill discovery, reading `SKILL.md`, and writing a helper script do not satisfy the gate; web/knowledge/MCP results, non-skill uploaded-file reads, or successful Python skill-script execution do. Tool filtering removes `present_files`, `skill_manage`, task/schedule/memory/clarification surfaces. The middleware permits `write_file`/`str_replace` only for `/mnt/user-data/workspace/*.py` and permits `bash` only for `.py` execution without redirection/document paths, so skills can run scripts but the agent cannot create Markdown/document artifacts or write `outputs`. Tests: `tests/test_forced_research_agent.py`.
|
||||
|
||||
**Strategy-selection functional agents.** `app/gateway/routers/_strategy_selection_seed.py` seeds nine stable folder-backed agents from `_strategy_selection_seed_assets/` before `_sync_legacy_agents`: intent, governance/diplomacy/economy/security/public-opinion/law-policy angles, judge, and solution editor. Seeding is missing-directory-only and must never overwrite administrator changes. The global `strategy_agent_roles` registry classifies them independently of projects: intent/judge/solution are fixed, unique roles that may be configured but never added, disabled, reordered, or deleted; angle roles are a shared, ordered catalog that administrators may add, edit, disable (soft delete), restore, and reorder through `/api/strategy-agents`. Public `GET /api/strategy-agents` returns enabled identity/description metadata only; `include_disabled=true` or `include_config=true` is admin-only. Admin writes are `POST /api/strategy-agents/angles`, `PUT /api/strategy-agents/{id}`, and `DELETE /api/strategy-agents/{id}`; DELETE retains the folder-backed agent and role row to preserve historical references. Dynamic angles remain built-in folder-backed agents under the reserved `strategy-selection-angle-` id namespace, with `web` + `file:read`; a blank SOUL on create receives a safe role template. All strategy skill writes validate that names exist, are active, and are globally visible; generic agent CRUD cannot create or hard-delete reserved strategy IDs, and parallel execution rejects disabled roles. Angle configs reuse `policy-knowledge`; intent reuses `intent-clarification-policy`; the judge has no retrieval skill; the solution editor gets only `file:write`. Keep the SOUL contracts neutral and evidence-aware, with explicit de-escalation/civilian-protection framing and hard bans on operational combat guidance and targeted political manipulation. Tests: `tests/test_strategy_selection_seed.py`, `tests/test_strategy_agent_management.py`.
|
||||
|
||||
**ThreadState** (`packages/harness/deerflow/agents/thread_state.py`):
|
||||
- Extends `AgentState` with: `sandbox`, `thread_data`, `title`, `artifacts`, `todos`, `uploaded_files`, `viewed_images`
|
||||
- Uses custom reducers: `merge_artifacts` (deduplicate), `merge_viewed_images` (merge/clear)
|
||||
@@ -540,6 +542,8 @@ Self-contained product knowledge base. **All knowledge capability lives in this
|
||||
|
||||
Custom agent metadata lives in the `agents` ORM table (`deerflow.persistence.agents`). `agents.id` is the stable runtime id and the filesystem directory name under `.deer-flow/agents/{id}/`; `agents.name` is display-only, can contain Chinese characters, and is not unique. `user_id IS NULL` marks built-in agents, user-owned agents are visible only to their owner unless `published = true`, and update/delete operations require ownership. The runtime accepts `agent_id` in context/configurable, and non-default `assistant_id` values are treated as agent ids after visibility checks.
|
||||
|
||||
Strategy-selection fan-out (`routers/parallel_agents.py`) must keep hidden reasoning separate per agent: provider reasoning blocks/fields and inline `<think>...</think>` text emit `reasoning_delta`, while visible answer text emits `delta` and only visible text may enter `agent_done.content`. Never concatenate `reasoning_delta` into a selectable strategy or generated deliverable. Inspect the final AI message with the shared `is_llm_error_message` detector and require substantive visible text before constructing a success result. A middleware fallback, empty/thinking-only response, or punctuation-only provider degeneration is a failed seat run: raise so the stream emits `agent_error`; never wrap it in `agent_done`. Regression coverage: `tests/test_roundtable_model_fallback.py`.
|
||||
|
||||
**Listing cache + pagination** ? to keep `GET /api/agents` fast, the gallery's per-card extras (skills + tool_groups) are denormalized into two cache tables instead of reading every agent's `config.yaml` on each request: `agent_skills` (relational, one row per (agent, skill) with `position`) and `agent_extras` (one row per agent: `tool_groups` JSON + `model` + `skills_synced_at`). `config.yaml` remains authoritative for the agent runtime; these tables are a read cache, write-through on create/update and backfilled from disk by `_sync_legacy_agents` (mtime-vs-`skills_synced_at` drift check). The list endpoint enriches via `AgentStore.get_extras_for([ids])` (two batched `IN` queries). Both tables are auto-created by `Base.metadata.create_all` (no migration needed). `GET /api/agents` supports **opt-in server-side pagination**: `scope` (`mine` = owned + favorited / `square` = built-in/published public), `page`, `page_size`, returning `{agents, total, page, page_size}`; omitting `scope`/`page` preserves the legacy "return everything" behavior. Name `search` matches the raw OR desensitized name (the frontend's static term?abbrev map is mirrored server-side in `deerflow/config/agent_name_desensitize.py`); `tag_ids` is an SQL OR-filter; `owner` keeps only agents published by that user id (the card's ??? ? built-in agents never match), powering the gallery's click-???-to-filter. `POST /api/agents/private-square/list` takes the same pagination params (incl. `owner`). Repository entry point: `AgentStore.list_paginated(...)`. Tests: `tests/test_agent_listing_cache.py::test_list_paginated_owner_filter_keeps_one_publisher`.
|
||||
|
||||
**Homepage selector (agent picker on the new-chat page)** ? single admin-curated list backed by `agents.featured_order` (nullable BIGINT). NULL = not featured. Only `published = true` agents can be featured (router enforces on writes); any published agent is eligible regardless of ownership. Cleared with `order: null`. The list rendered to end users is `WHERE featured_order IS NOT NULL` + visibility-filter, sorted by `featured_order ASC`. Admins reorder by sending the desired ids to `PUT /api/agents/featured/order` (assigns indices 0..N-1). BIGINT is used rather than INT so the management UI can seed initial values from `Date.now()` (overflows INT on MySQL).
|
||||
@@ -766,6 +770,7 @@ MySQL index script synchronized.
|
||||
- Physical: `backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/...`, `deer-flow/skills/`
|
||||
- Translation: `replace_virtual_path()` / `replace_virtual_paths_in_command()`
|
||||
- Detection: `is_local_sandbox()` checks `sandbox_id == "local"`
|
||||
- Configured mount portability: `AppConfig._resolve_relative_mount_paths` resolves each relative `sandbox.mounts[*].host_path` against the loaded `config.yaml` directory before validation/provider construction, never against the process cwd. Keep repository-owned knowledge mounts relative (for example `knowledge/knowledge_base`); absolute paths remain unchanged, and sandbox providers always receive an absolute path.
|
||||
- **`{user_id}` for path resolution = `resolve_path_user_id(thread_id)`, using the persisted creator for every tracked thread.** The route's regular owner check still authorizes access first; creator resolution only selects the stable physical directory (`threads_meta.user_id`) and avoids a lost request context falling back to `users/default`. This also preserves cross-account task-deeplink conversations (rwfx/3qfx/xdfx) and roundtable system threads. Positive creator mappings are cached process-wide and DB/memory/missing-row failures fall back to the current login. It is wired at the thread-path consumers including `ThreadDataMiddleware.before_agent`, `present_file_tool`, uploads, artifact serving, and embedded `client.get_artifact`. **Rewrite-history recovery is metadata-only and job-owner scoped** (`list_by_session(..., user_id=current_user)`), so its GET/list endpoints intentionally skip redundant thread-owner checks and return an empty history instead of a history-only 404; the start/commit endpoints retain strict thread write authorization. Tests: `tests/test_thread_paths_owner.py`, `tests/test_thread_paths_roundtable.py`.
|
||||
|
||||
- **Durable document-rewrite commit:** `document_rewrite_pipeline.atomic_write_text` writes through a deliberately short sibling temporary filename so long Windows artifact paths stay below the path-length limit, and guarantees the target `outputs` parent exists. If the sandbox recreates that directory in the narrow commit window, it repairs the directory and retries; the original file is otherwise only changed through `os.replace`.
|
||||
@@ -998,6 +1003,15 @@ scheduler runs once per deployment. Per run:
|
||||
- Supports `supports_vision` flag for image understanding models
|
||||
- Config values starting with `$` resolved as environment variables
|
||||
- Missing provider modules surface actionable install hints from reflection resolvers (for example `uv add langchain-google-genai`)
|
||||
- Generic `langchain_openai:ChatOpenAI` entries are transparently wrapped by
|
||||
`PatchedChatOpenAI`. Besides preserving reasoning/tool-call metadata, the
|
||||
patch recognizes compatible gateways that encode failures inside HTTP 200
|
||||
payloads (`base_resp.status_code != 0` with null/empty `choices`) in both
|
||||
streaming and non-streaming paths. It raises the stable
|
||||
`OpenAICompatibleUpstreamError`; `LLMErrorHandlingMiddleware` retries
|
||||
transient concurrency/inference failures, while genuine quota/balance
|
||||
failures are classified as non-retriable and surfaced immediately. Embedded
|
||||
vendor failures therefore cannot become empty successful assistant turns.
|
||||
|
||||
### Roundtable Model Fault Tolerance (会商模型自动容错 / 换下一个模型)
|
||||
|
||||
@@ -1128,6 +1142,15 @@ Both can be modified at runtime via Gateway API endpoints or `DeerFlowClient` me
|
||||
|
||||
### Conditional WeKnora LLMWiki
|
||||
|
||||
**Embedding ownership:** WeKnora's per-knowledge-base `embedding_model_id`
|
||||
indexes uploaded source documents; `llmwiki.local_wiki_index.embedding` indexes
|
||||
the generated Wiki-page mirror inside DeerFlow. Do not couple these two model
|
||||
choices. `GET /api/llmwiki/embedding-models` returns a strict credential-free
|
||||
projection of WeKnora Embedding models, and KB creation must validate and pass
|
||||
the selected active model id. When the caller omits a choice, retain WeKnora's
|
||||
active-default behavior. Existing KBs cannot change this binding in place, and
|
||||
cross-KB WeKnora searches should use model-compatible knowledge bases.
|
||||
|
||||
Ordinary knowledge-base cards expose an owner/admin-only name/description edit dialog in `WeKnoraKnowledgeWorkspace.tsx`, reusing `useLlmWikiActions().update` and `PATCH /api/llmwiki/knowledge-bases/{id}`. The route trims names, rejects blank or reserved conversation-deposit names, writes to WeKnora before updating local metadata, and retains existing write authorization. A successful remote list is authoritative for every list scope, including `personal`: absent remote IDs are hidden without deleting mappings/references. Upstream failures remain errors. The grid also filters cached `remote_status=missing` rows and no longer renders stale-record cleanup cards. Regression coverage: `tests/test_llmwiki_knowledge_management.py`.
|
||||
|
||||
`llmwiki.weknora.api_base_url` is the provider switch: non-empty enables WeKnora, empty preserves the legacy LLMWiki UI. WeKnora is independently deployed and owns PostgreSQL/Redis/object storage/vector storage/parsers/models; DeerFlow config contains only `api_base_url` and optional `web_base_url`. DeerFlow deliberately sends no WeKnora credential, so the WeKnora endpoint must be restricted to a trusted network and allow DeerFlow traffic itself or through its reverse proxy. There is intentionally no `default_embedding_model_id`: `WeKnoraClient` queries `/api/v1/models` and uses WeKnora's active Embedding model when creating a KB; Wiki creation additionally resolves an active KnowledgeQA model inside WeKnora for synthesis. The native BFF is `app/gateway/routers/llmwiki.py`; ownership/publication mappings live in `deerflow.persistence.llmwiki`; the browser only sees DeerFlow mapping IDs. Knowledge-base owners publish immediately (`private` -> `published`) without a review queue, and can unpublish ordinary bases. The conversation-deposit KB is system-owned, always published, shared by all DeerFlow users, excluded from My Knowledge, and immutable through user/admin detail operations; recognition requires both the reserved name and `system` owner. Existing same-name remote mappings are adopted in place as the system deposit, preserving mapping ids and document links. Its user-facing description, generated Markdown source and tags use `cmzs` branding; list/detail reads synchronize stale remote descriptions, and search results, source previews and embedded-detail JSON recursively replace legacy brand text in historical conversation-deposit payloads. Deposit requests must reference a thread accessible to the current account. Personal knowledge-base list/search views are scoped to the current DeerFlow account even for admins; frontend LLMWiki caches include the live account id, and admin write power does not make other users' private KBs appear as owned. Only `system_role === "admin"` sees the WeKnora connection-config tab. Knowledge cards navigate to a dedicated WeKnora-style detail route. Wiki-enabled bases render and default to `Wiki -> documents -> graph`; standard bases omit Wiki and render `documents -> graph`. The detail page supports sequential multi-file upload with an aggregate result for writable ordinary bases, hides it for public read-only/system-deposit bases, and uses a generic loading label while the embed session starts. It also supports URL and published manual-Markdown imports, browses real Wiki pages/content, proxies authenticated original-file previews, pages through `/chunks/{knowledge_id}`, and renders the Wiki page graph from `/knowledgebase/{kb_id}/wiki/graph`; document details use a right sheet rather than the removed KB modal. The credential-free proxy uses one WeKnora service identity, so per-user separation is enforced by DeerFlow mappings while all remote KBs physically share that identity's WeKnora workspace. WeKnora iframe detail pages first call `/api/llmwiki/knowledge-bases/{id}/weknora/session` through the normal DeerFlow API auth path, then load `/platform/knowledge-bases/{weknora_id}` with a signed `deerflow_weknora_embed` cookie for `/platform`, `/assets`, `/locales`, root assets, and API calls. Injected fetch/XHR rewriting sends all iframe `/api/v1/*`, including colliding `/api/v1/auth/*`, through `/api/llmwiki/weknora-embed/api/v1/*`; Gateway auth/CSRF bypass only applies to those signed iframe proxy requests, the proxy still revalidates the mapped KB, and cookie security honors `X-Forwarded-Proto` for HTTPS reverse proxies. `llmwiki_search` is conditionally registered in `tools.py`, is read-only, and rechecks the current DeerFlow user, agent `llmwiki_knowledge_base_ids`, and thread-context selection on every call. Source previews prefer the cleaned Wiki/LLMWiki page text when a retrieved chunk maps to a Wiki page, with raw split chunks as the fallback. Agent bindings use an inline WeKnora-style searchable/grouped multi-select in the standard agent create/detail UI; new chats use the same interaction pattern as a compact composer dropdown, and selections are written to thread metadata/context and restored on reopen. The DeerFlow frontend is native React and the WeKnora frontend is not modified or embedded. Full deployment and behavior notes: `docs/LLMWIKI_WEKNORA_INTEGRATION_ZH.md`. Tests: `tests/test_llmwiki_weknora.py`.
|
||||
@@ -1324,6 +1347,14 @@ Automatic conversation summarization when approaching token limits:
|
||||
when it has already been folded into the hidden summary. This keeps strict
|
||||
OpenAI-compatible gateways from rejecting a tool-continuation call with
|
||||
`No user query found in message`.
|
||||
- LangChain keeps the agent prompt in `ModelRequest.system_message`, separate
|
||||
from `ModelRequest.messages`. Compression must merge its hidden summary into
|
||||
that dedicated slot and remove every `SystemMessage` from the ordinary
|
||||
message list. Strict Qwen-style adapters accept only one system message at
|
||||
index zero and otherwise return HTTP 400 (`system message must be at the
|
||||
beginning`). Keep this normalization adapter-independent; the payload-level
|
||||
OpenAI-compatible normalizer is a second line of defence and also merges
|
||||
adjacent leading system messages.
|
||||
- Keeps recent messages while summarizing older ones.
|
||||
|
||||
Tests: `tests/test_summarization_toggle.py` (gating + whitelist),
|
||||
@@ -1803,8 +1834,8 @@ Ordinary knowledge detail landing is globally configured by `ordinary_knowledge_
|
||||
pages need backfill, not a forced rebuild of every page.
|
||||
FastEmbed is a standard backend dependency. Encoder configuration, directory
|
||||
upload and warmup live only in the administrator global settings dialog.
|
||||
A runtime `system_settings.json` path override wins over config.yaml; uploaded
|
||||
files are streamed into `.deer-flow/models/bge-m3`, never the source tree.
|
||||
When the effective provider is local, its `system_settings.json` path override wins over config.yaml; uploaded
|
||||
files are streamed into immutable `.deer-flow/models/bge-m3-revisions/<revision>` directories, never the source tree.
|
||||
Encoder warmup is explicitly started from the settings UI/API and must never block application startup
|
||||
or turn a model-load failure into a DeerFlow service failure. Local provider
|
||||
is offline-only: require local_model_path or DEERFLOW_BGE_M3_MODEL_PATH and
|
||||
@@ -1812,6 +1843,21 @@ Ordinary knowledge detail landing is globally configured by `ordinary_knowledge_
|
||||
`local_process_isolation=true` by default: inference runs in the bundled JSONL
|
||||
worker subprocess, and shutdown must terminate that worker without waiting on
|
||||
model teardown.
|
||||
- The administrator encoder override is an explicit persisted mode, not an
|
||||
inference from whether `model_path` happens to be non-empty. `configured`
|
||||
uses the current `config.yaml` provider (including OpenAI-compatible);
|
||||
saving/uploading a local directory sets `local` plus a new model revision,
|
||||
while `/api/llmwiki/local-encoder/use-configured` switches back. Every switch
|
||||
hot-rebuilds embedding/cache/search/sync/retrieval as one graph and updates
|
||||
the harness runtime registration. Status must remain readable for a remote
|
||||
provider. The scheduler stays alive as a lightweight config watcher but only
|
||||
scans when effective `enabled && auto_sync`; keep the default/portable release
|
||||
configs at `auto_sync=true` because asynchronous Wiki generation can finish
|
||||
after the event-triggered first scan. A changed local model revision must
|
||||
change the embedding fingerprint so vectors from replaced weights are never
|
||||
reused. Uploaded weights live in immutable revision directories. A replaced
|
||||
embedding generation gets a bounded grace period for in-flight work, then
|
||||
rejects stragglers, drains real thread work, and releases its model process.
|
||||
- Every successful mapping snapshot is mirrored to the global assistant base,
|
||||
including empty snapshots so deleted source pages are retired. Re-import is
|
||||
source-replacement semantics: absent Wiki/chunk/entity/relation pages must not
|
||||
|
||||
Reference in new issue
Block a user