Merge remote-tracking branch 'origin/master' into ji
# Conflicts: # offline-backend-20260512/backend/app/gateway/app.py
This commit is contained in:
commit
67d12929fc
309 files changed
+49676
-5740
No files matched your search
@@ -206,6 +206,21 @@ The backend is split into two layers with a strict dependency direction:
|
||||
|
||||
**Dependency rule**: App imports deerflow, but deerflow never imports app. This boundary is enforced by `tests/test_harness_boundary.py` which runs in CI.
|
||||
|
||||
### Report Collaboration (AgentScope, in progress)
|
||||
|
||||
Independent of Workflow Studio and the existing roundtable. Plan: repo-root `docs/agentscope-多智能体报告协作工作台-后端实施计划.md`(`RC-BE-000`~`018` 已落地,下一票 `RC-BE-019`)。Spike baseline: `docs/AGENTSCOPE_BASELINE.md`.
|
||||
|
||||
- **Config**: `config.yaml → report_collaboration` (`deerflow.config.report_collaboration_config.ReportCollaborationConfig`). Default **off** (`enabled` / `worker_enabled`).
|
||||
- **Contracts**: `app/report_collaboration/contracts/` — snake_case Pydantic models matching `frontend-web/src/report-collaboration/api/types.ts`. SSE envelope uses camelCase (`eventId` / `sessionId` / `runId`). `node_run_id` is always `${node_id}-attempt{attempt}`.
|
||||
- **Persistence**: `deerflow.persistence.report_collaboration` (`report_collaboration_*` tables, migrations `20260905_01` / `20260906_01` / `20260907_01`). SQL + memory stores; one active run per session via `active_dedupe_key`; command/message/run/version idempotency keys refuse cross-session reuse; event `seq` taken from the run row. Session delete removes children explicitly.
|
||||
- **Gateway**: `/api/report-collaboration` (`app/gateway/routers/report_collaboration.py`). `enabled=false` → 503. Plan/run writes enqueue durable jobs and return immediately. Pre-run `POST /sessions/{id}/messages` runs a rule-based `IntentRouterAgent` + `RequirementResolver` (`app/report_collaboration/requirement/`): writes a requirement revision and clarification messages, never auto-starts planning. `POST /sessions/{id}/plan-requests` then runs `PlanProposalService` (`app/report_collaboration/planning/`): catalog projection (no SOUL/keys) → selector → 2–3 strategy RoleDemand graphs → assembler binds catalog agents/tools/quality gates → validator (acyclic DAG, artifact types, allowlisted tools). Select and start-run stay independent commands. `POST /runs` freezes role models, report-structure template and run budget. Rewrite / apply / restore are live (`reporting/`). `GET /runs/{id}/stream` is a read-only durable SSE tail (`execution/live_hub.py`): persist then publish, replay + `after_seq` / `Last-Event-ID`, heartbeat comments, token-delta coalesce before seq assignment. Disconnect does **not** cancel the run. Lifespan mounts `ReportCollaborationDispatcher` only when `enabled` **and** `worker_enabled` are both true; default both false.
|
||||
- **Runtime**: AgentScope adapters live in `app/report_collaboration/agentscope_runtime/` (not the publishable harness). `ConversationReplica` maps AgentScope 2.0.5 events 1:1 onto collaboration SSE types; TeamSay is `HINT_BLOCK`. `DeerFlowChatModelAdapter` wraps `create_chat_model` (no second Provider config); role models are frozen onto node `agent_snapshot` at `POST /runs` and missing/incapable models fail with 422 before the run is created. Fallback is recorded on the snapshot plus a durable `heartbeat` audit event. Nodes complete only after a validated artifact (`app/report_collaboration/execution/completion.py`). `TaskLedger` (`execution/ledger.py`) owns ready waves, attempts and terminal node status; AgentScope / coordinator suggestions never pick the next node. A writer-bearing run is `completed` only after `ReportFinalizer` publishes a formal version. `WaveExecutor` + `QualityRoleKernel` (`execution/quality_roles.py`) remain for fixtures: Verifier marks stale/contested claims, Analyst/Writer only cite supported IDs, Reviewer `revise` replans only the target write node plus review. Production worker injects `AgentScopeMemberKernel` (`agentscope_runtime/member_kernel.py`) with DeerFlow tools and per-run budget. `QualityGate` (`execution/quality_gate.py`) is the claim-level gate: conflicts/missing units, unverified facts, coverage gaps, and unmapped review targets fail before `validated`; `ArtifactBoard.supersede_previous` hides old attempts from downstream. `ReportCollaborationLiveHub` (`execution/live_hub.py`) fans out persisted envelopes to SSE viewers; `PublishingReportCollaborationStore` sanitizes secrets/privacy/hidden reasoning without truncating visible tool results, and writes redaction + event audits. Dispatcher (`execution/dispatcher.py`) atomically claims queued/recovering/expired-lease runs, heartbeats via `renew_lease`, two-phase cancels (`cancel_requested` → `finalize_cancel`), and recovers orphans (`execution/recovery.py`): keep completed work, mark leftover in-flight `interrupted`/`superseded`, open a new `planned` attempt. Late artifacts after cancel/reclaim are `superseded` and must not change terminal run status. Tests drive `dispatch_once()`; do not rely on the background loop. Per-run budgets (`security/budget.py`) fail-closed; per-user active-run caps are enforced at `create_run`.
|
||||
- **Quality eval**: `app/report_collaboration/eval/` (`RC-BE-018`) — 20 annotated tasks (policy/market/enterprise/event/comparison ×4) with required angles, key facts, forbidden fabrications, templates and review rules. Deterministic scorer (no LLM judge) gates citation/angle/template/directed-mod; gold reports must pass, canned failures (including a roundtable-style essay) must fail and stay reproducible. Recommended runtime knobs live in `eval/thresholds.py` and match current `ReportCollaborationConfig` defaults (`worker_enabled` remains false). Tests: `tests/test_report_collaboration_eval.py`.
|
||||
- **Runtime intervention**: `app/report_collaboration/interventions/` owns run-time natural-language commands. `RuntimeIntentRouterAgent` uses explicit rules first and a tool-free coordinator-model fallback only for ambiguity; its Pydantic decision cannot dispatch or mutate anything. `ImpactAnalyzer` validates server-side node/agent ownership, resolves angle-specific lineage, and enforces confidence/cost/parallel-branch confirmation (`intent_confirmation_threshold`, default `0.8`). `RuntimeCommandService` mirrors the human message, emits frozen command events, CAS-claims accepted commands, and applies them only through `TaskLedger` safe points. Rework persists command overlays, supersedes affected attempts/artifacts, creates monotonic attempts, and prevents late output from entering downstream. Waiting-node answers resume only those nodes. `rerun_node` without a legal target does not expand to the whole graph; requirement patches apply only after rework succeeds; stuck `executing` commands are reclaimed on takeover or lease timeout; `command.cancelled` is a durable SSE event. In-run `POST /sessions/{id}/messages` returns 409 (`RUN_ACTIVE`). Formal rewrite candidates and version apply/restore live in `reporting/` (`RC-BE-015`).
|
||||
- **Dependency**: optional extra `report-collaboration` → `agentscope==2.0.5`. Do not add AgentScope to the harness `pyproject.toml`.
|
||||
|
||||
Tests: `tests/test_report_collaboration_*.py`.
|
||||
|
||||
### Workflow Studio
|
||||
|
||||
Coze Studio is canvas-only; ChatDev is reference-only. Plan + status per phase: `docs/WORKFLOW_STUDIO_BACKEND_DEV_ZH.md`; decisions: `docs/adr/0001-workflow-studio-runtime.md`; contracts: `docs/WORKFLOW_SCHEMA_V1_ZH.md` + `docs/WORKFLOW_SSE_V1_ZH.md` + `docs/WORKFLOW_OPENAPI_DRAFT_V1.yaml`. Master switch + all limits: `config.yaml` → `workflows` (`deerflow.config.workflow_config.WorkflowConfig`). `workflows.agent_recursion_limit` defaults to 250: LangGraph counts each model→tool research exchange as about two super-steps, so workflow agent/skill nodes must not fall back to a chat-sized 60-step limit. In the local Studio proxy, keep Rsbuild `server.compress: false`; dev gzip buffers small SSE frames and makes live planning/run output arrive only at completion despite the backend's `no-transform` / `X-Accel-Buffering: no` headers. The composer’s optional `modelName` must be validated against `AppConfig.models`: it selects the controller model and is copied only into newly generated candidate agent configs (or the projected direct-agent task), never injected into an already-confirmed proposal/run.
|
||||
@@ -280,6 +295,8 @@ from deerflow.config import get_app_config
|
||||
- Tools loaded via `get_available_tools()` - combines sandbox, built-in, MCP, community, and subagent tools
|
||||
- System prompt generated by `apply_prompt_template()` with skills, memory, and subagent instructions
|
||||
|
||||
**agentfx built-in analyst + report convert skills.** `agentfx-analyst` (「任务研判报告助手」) is seeded from `app/gateway/routers/_agentfx_seed_assets/` before `_sync_legacy_agents`. After mandatory `web_search`, the agent writes each tool JSON to `/mnt/user-data/workspace/hits/qN.json` and runs **one** command — `task-report-build` (group `all`) — to fill the 14-category `task-reports.json` for the dashboard slots; the four per-page skills `task-report-{enemy,our,env,judge}` remain for re-running a single dashboard page. All five are thin CLIs over the shared stdlib engine `skills/public/task-report-lib/convert_lib.py` (no SKILL.md on the lib, so it never enters the skill list). The model must not `write_file` that JSON. Because the sandbox maps `/mnt/user-data/...` onto a host dir, a virtual absolute hits path can be `isdir`-true yet list empty on a Windows host — `resolve_hit_files` therefore tries the path as given, then the cwd-mapped sandbox path (`discover_user_data_root` + `resolve_sandbox_path` — bash `cd`s into the thread `workspace`, so `/mnt/user-data/outputs/task-reports.json` must land in that thread's `outputs/`, never `F:\mnt\...` on a Windows host), then the virtual-prefix-stripped relative form, then the bare dir name, and picks the first candidate that actually contains `.json`; a miss returns a `warnings` entry listing every tried form so the agent never has to write a debug script. Assemble of the 14 categories is the same function (`assemble_records`); the model must not copy or merge files. If the model is interrupted after convert, `task-reports.json` is already in outputs and the frontend import card also scans `GET /api/threads/{id}/artifacts` so入库 still works without `present_files`. Import remains `task-report-import`. Existing agent/skill directories are never overwritten. Tests: `tests/test_task_report_convert.py`, `tests/test_agentfx_seed.py`.
|
||||
|
||||
**Forced-research built-in agent.** `forced-research-responder` (「强制检索输出助手」) is seeded from `app/gateway/routers/_forced_research_seed_assets/` before `_sync_legacy_agents`. `ForcedResearchMiddleware` is mounted only for this stable id. On every visible user turn it requires a successful information-gathering tool result before the model may finish; `tool_choice=required` is backed by an `after_model -> model` retry for gateways that ignore tool choice, with an eight-attempt failure cap that permits only an honest failure response. Skill discovery, reading `SKILL.md`, and writing a helper script do not satisfy the gate; web/knowledge/MCP results, non-skill uploaded-file reads, or successful Python skill-script execution do. Tool filtering removes `present_files`, `skill_manage`, task/schedule/memory/clarification surfaces. The middleware permits `write_file`/`str_replace` only for `/mnt/user-data/workspace/*.py` and permits `bash` only for `.py` execution without redirection/document paths, so skills can run scripts but the agent cannot create Markdown/document artifacts or write `outputs`. Tests: `tests/test_forced_research_agent.py`.
|
||||
|
||||
**Strategy-selection functional agents.** `app/gateway/routers/_strategy_selection_seed.py` seeds nine stable folder-backed agents from `_strategy_selection_seed_assets/` before `_sync_legacy_agents`: intent, governance/diplomacy/economy/security/public-opinion/law-policy angles, judge, and solution editor. Seeding is missing-directory-only and must never overwrite administrator changes made through the normal agent API. Angle configs reuse `policy-knowledge` with `web` + `file:read`; intent reuses `intent-clarification-policy`; the judge has no retrieval skill; the solution editor gets only `file:write`. Keep the SOUL contracts neutral and evidence-aware, with explicit de-escalation/civilian-protection framing and hard bans on operational combat guidance and targeted political manipulation. Tests: `tests/test_strategy_selection_seed.py`.
|
||||
@@ -1552,10 +1569,12 @@ detail. Never call `/knowledge-search` from internal search, RAG, or the
|
||||
`deerflow.integrations.weknora.local_index` and
|
||||
`deerflow.persistence.llmwiki_index`; preserve the harness → app import
|
||||
firewall. Gateway composition injects the store/search/sync services through
|
||||
`local_index.runtime`. `enabled` turns on local vector search and manual
|
||||
vectorization; `auto_sync` is the separate opt-in for the lease-protected
|
||||
scheduler. The scheduler sleeps for one full configured interval before its
|
||||
first scan, so application startup must never trigger immediate vectorization.
|
||||
`local_index.runtime`. `enabled` turns on local vector search and `auto_sync`
|
||||
defaults to the lease-protected background scheduler. The scheduler performs
|
||||
an immediate incremental scan at startup, then rescans every configured
|
||||
interval so new WeKnora libraries and asynchronously generated Wiki pages are
|
||||
picked up without a manual vectorization click. Unchanged content must reuse
|
||||
existing vectors.
|
||||
|
||||
Vectors are normalized little-endian float32 BLOBs. A page mirror and all of
|
||||
its vectors are replaced in one transaction only after strict batched
|
||||
@@ -1737,20 +1756,30 @@ snapshot remains authoritative on failure. A binding may have only one active
|
||||
item (`active_dedupe_key` plus atomic lease claims). Skill distillation targets
|
||||
only writable WeKnora Wiki knowledge bases in this product version; reject direct
|
||||
assistant-base targets, non-Wiki/FAQ/conversation/readonly targets, and direct
|
||||
Neo4j writes. Detach/delete operates on source contributions, not canonical rows,
|
||||
Neo4j writes. The create/list/resync/retry/detach flow is available to ordinary
|
||||
users for their own visible skills, writable knowledge bases, jobs and bindings;
|
||||
administrator-only auto-sync/review APIs remain privileged. Detach/delete operates on source contributions, not canonical rows,
|
||||
and shared entities/relations are retired only when their final contribution is
|
||||
gone.
|
||||
|
||||
Assistant knowledge has one global full-library base (`知识梳理总库`) and any
|
||||
number of admin-created custom bases. Ordinary users have read/search access and
|
||||
administrators own all mutations. Assistant bases only accept WeKnora export
|
||||
packages or imports triggered from a WeKnora mapping. Do not add direct skill
|
||||
imports or arbitrary local-file imports to assistant knowledge. Importing into a
|
||||
administrators own all mutations. Assistant bases accept local documents independently
|
||||
of WeKnora via `/api/assistant-knowledge/imports/files`, as well as WeKnora export
|
||||
packages or imports triggered from a WeKnora mapping. Local documents use the
|
||||
Gateway `assistant_file_ingest` pipeline: persisted original files, chunked LLM Wiki
|
||||
extraction, cached body embeddings, and import-job status/retry. Do not add direct skill
|
||||
imports. Importing into a
|
||||
custom assistant base must also mirror the package into the global base for
|
||||
cross-source entity alignment.
|
||||
cross-source entity alignment. Admins may rename any assistant base and delete
|
||||
custom bases. Deletion removes the custom base's Wiki pages, vectors, entities,
|
||||
relations, import audit rows, and package files; the global base is never
|
||||
deletable because it owns full-library aggregation.
|
||||
|
||||
### Wiki package and local encoder invariants (2026-09)
|
||||
|
||||
Ordinary knowledge detail landing is globally configured by `ordinary_knowledge_landing` (`wiki` default / `weknora`), with admin-only PUT `/api/system-settings/ordinary-knowledge-landing`. Explicit document/graph/article links bypass this default. Wiki reading and the embedded WeKnora view provide reciprocal navigation; `?view=weknora` explicitly selects the iframe and must not loop back to the default Wiki route.
|
||||
|
||||
- `knowledge_transfer.sync_to_global` is called after successful explicit local
|
||||
Wiki vectorization; it skips conversation/system mappings. Do not enable a new
|
||||
watcher or automatic remote-write scheduler as a side effect.
|
||||
@@ -1764,16 +1793,42 @@ cross-source entity alignment.
|
||||
- Preserve normalized float32 base64 bytes exactly. Same dimension alone is not
|
||||
model compatibility. Assistant vector retrieval is fingerprint-gated cosine
|
||||
search, and any keyword fallback must be labeled as such.
|
||||
Preserve a valid zero section_index. Export/search vectors belonging to the
|
||||
current Wiki revision's import job, not stale contributions from other sources.
|
||||
Preserve a valid zero section_index. Export/search vectors belonging to each
|
||||
current source snapshot (`source.last_import_job_id`), not merely the page's
|
||||
current revision: one aligned page may intentionally contain current vectors
|
||||
from several sources.
|
||||
- The `local` embedding provider uses FastEmbed with official BAAI/bge-m3 ONNX,
|
||||
1024 dimensions, bounded CPU batches and a per-client async lock. Run inference
|
||||
off the event loop. Existing page vectors are reusable; zero-vector nonempty
|
||||
pages need backfill, not a forced rebuild of every page.
|
||||
FastEmbed is a standard backend dependency. Encoder configuration, directory
|
||||
upload and warmup live only in the administrator global settings dialog.
|
||||
A runtime `system_settings.json` path override wins over config.yaml; uploaded
|
||||
files are streamed into `.deer-flow/models/bge-m3`, never the source tree.
|
||||
Encoder warmup is explicitly started from the settings UI/API and must never block application startup
|
||||
or turn a model-load failure into a DeerFlow service failure. Local provider
|
||||
is offline-only: require local_model_path or DEERFLOW_BGE_M3_MODEL_PATH and
|
||||
never let the application download model weights. Keep
|
||||
`local_process_isolation=true` by default: inference runs in the bundled JSONL
|
||||
worker subprocess, and shutdown must terminate that worker without waiting on
|
||||
model teardown.
|
||||
- Every successful mapping snapshot is mirrored to the global assistant base,
|
||||
including empty snapshots so deleted source pages are retired. Re-import is
|
||||
source-replacement semantics: absent Wiki/chunk/entity/relation pages must not
|
||||
remain visible merely because they existed in an older source snapshot.
|
||||
- Provider graph nodes bind to their original wiki_slug. They must not create
|
||||
duplicate synthetic entity/relation articles. Store original Wiki metadata
|
||||
separately from assistant workflow status. Count Wiki pages separately from
|
||||
raw document chunks.
|
||||
- The global base aligns entities by normalized display name and aliases across
|
||||
source records, and aligns content Wiki pages by normalized title. Do not use
|
||||
source slug as the global canonical identity. Merge unique knowledge blocks
|
||||
while retaining source contributions/origins. Uploaded packages must rebind
|
||||
the temporary upload source to the stable WeKnora knowledge-base id in the
|
||||
package, so re-upload is source-replacement rather than append. Retrieval
|
||||
candidates exclude summary/index/directory pages; keyword matching, vector
|
||||
snippets, citations and prompts must use knowledge body/sections, never page
|
||||
summary as a substitute.
|
||||
- Migration `20260903_01` adds nullable page metadata and merges the pre-existing
|
||||
knowledge/workflow Alembic heads. Regression coverage is in
|
||||
`tests/test_wiki_package_roundtrip.py`, `test_llmwiki_wiki_sync.py` and
|
||||
|
||||
Reference in new issue
Block a user