feat: integrate WeKnora llmwiki knowledge flow
This commit is contained in:
1 parent
a0a75a9964
commit
e0c09f6704
63 files changed
+7045
-197
No files matched your search
@@ -1,13 +1,13 @@
|
||||
# DeerFlow 离线后端部署补充
|
||||
|
||||
本目录提供与原始 `start_backend.sh` 配套的 wheelhouse。启动脚本保持原有结构,只新增一行 Redis Python 客户端的离线安装命令。
|
||||
本目录提供与 `start_backend.sh` 配套的 wheelhouse。启动脚本会在容器启动时检查可选运行时依赖,并在缺包时仅从本地 wheelhouse 安装。
|
||||
|
||||
## 文件
|
||||
|
||||
- `offline-runtime-requirements.txt`:固定版本的完整依赖闭包。
|
||||
- `prepare_wheelhouse.sh`:联网机器下载 Linux/Python 3.12 wheel,并验证传递依赖完整性。
|
||||
- `start_backend.sh`:原始 Docker 启动脚本,加一行 Redis 缺包安装命令。
|
||||
- `wheelhouse/`:22 个 wheel 和 `SHA256SUMS`,需要随发布目录一起复制。
|
||||
- `start_backend.sh`:Docker 启动脚本,包含 `croniter`、`hindsight-client`、`aiomysql` 和 Redis 客户端的缺包补装逻辑。
|
||||
- `wheelhouse/`:24 个 wheel 和 `SHA256SUMS`,需要随发布目录一起复制。
|
||||
|
||||
## 联网机器准备
|
||||
|
||||
@@ -23,8 +23,9 @@ bash prepare_wheelhouse.sh
|
||||
- `croniter==6.2.2`
|
||||
- `hindsight-client==0.6.2`
|
||||
- `redis==5.2.1`
|
||||
- `aiomysql==0.3.2`
|
||||
|
||||
上述依赖在 Python 3.12/Linux x86_64 下的 19 个传递依赖也已锁定在 `offline-runtime-requirements.txt`。
|
||||
上述依赖在 Python 3.12/Linux x86_64 下的 20 个传递依赖(包括 `PyMySQL==1.2.0`)也已锁定在 `offline-runtime-requirements.txt`。
|
||||
|
||||
国内镜像示例:
|
||||
|
||||
@@ -57,7 +58,9 @@ bash start_backend.sh
|
||||
docker logs -f deerflow-backend
|
||||
```
|
||||
|
||||
原脚本已有 `croniter` 和 `hindsight-client` 的离线补装逻辑;新增的一行命令只负责在缺少 `redis` 模块时,从 `/app/wheelhouse` 安装 `redis==5.2.1`。
|
||||
脚本会先尝试导入依赖,仅在缺少模块时从 `/app/wheelhouse` 安装。`aiomysql` 补装默认开启,可通过 `AIOMYSQL_INSTALL_DEPS=0` 禁用,或通过 `AIOMYSQL_PIP_PACKAGES` 覆盖默认的 `aiomysql==0.3.2`。
|
||||
|
||||
MySQL 连接 URL 使用 aiomysql 驱动时应写为 `mysql+aiomysql://user:password@host:3306/database?charset=utf8mb4`。
|
||||
|
||||
## Redis 说明
|
||||
|
||||
|
||||
@@ -257,7 +257,7 @@ Configuration priority:
|
||||
|
||||
### Gateway API (`app/gateway/`)
|
||||
|
||||
FastAPI application on port 8001 with health check at `GET /health`. Set `GATEWAY_ENABLE_DOCS=false` to disable `/docs`, `/redoc`, and `/openapi.json` in production (default: enabled).
|
||||
FastAPI application on port 8001 with health checks at `GET|HEAD /health`, `/api/health`, and `/api/langgraph/health`. These exact paths are always public in `auth_middleware.PUBLIC_HEALTH_PATHS`, including when authentication is enabled; reverse-proxy prefixes and trailing slashes are normalized before the whitelist check. Set `GATEWAY_ENABLE_DOCS=false` to disable `/docs`, `/redoc`, and `/openapi.json` in production (default: enabled). Regression coverage: `tests/test_health_auth_bypass.py`.
|
||||
|
||||
**Routers**:
|
||||
|
||||
@@ -600,7 +600,7 @@ Operators upgrading an existing DB run the whole file (`USE <db>;` then execute)
|
||||
### Tool System (`packages/harness/deerflow/tools/`)
|
||||
|
||||
`get_available_tools(groups, include_mcp, model_name, subagent_enabled)` assembles:
|
||||
1. **Config-defined tools** - Resolved from `config.yaml` via `resolve_variable()`
|
||||
1. **Config-defined tools** - Enabled entries are resolved from `config.yaml` via `resolve_variable()`; entries with `enabled: false` are omitted before import and are not registered with the agent
|
||||
2. **MCP tools** - From enabled MCP servers (lazy initialized, cached with mtime invalidation)
|
||||
3. **Built-in tools**:
|
||||
- `present_files` - Make output files visible to user (only `/mnt/user-data/outputs`)
|
||||
@@ -613,6 +613,7 @@ Operators upgrading an existing DB run the whole file (`USE <db>;` then execute)
|
||||
- `tavily/` - Web search (5 results default) and web fetch (4KB limit)
|
||||
- `jina_ai/` - Web fetch via Jina reader API with readability extraction
|
||||
- `firecrawl/` - Web scraping via Firecrawl API
|
||||
- `configurable_search/` - Config-driven `POST` JSON `web_search` replacement. It deep-copies the configured payload and overwrites only `query`, supports HTTP/HTTPS plus configurable `verify_ssl`, timeout and result limit, normalizes nullable `recUuid`/`content`/`title`/`url` fields, and rewrites a result URL through `result_url_template` only when `recUuid` is non-empty. The required response envelope is `{"results": [...]}`. Tests use `httpx.MockTransport` and never require the deployment endpoint.
|
||||
|
||||
**ACP agent tools**:
|
||||
- `invoke_acp_agent` - Invokes external ACP-compatible agents from `config.yaml`
|
||||
@@ -921,9 +922,9 @@ Both endpoints are whitelisted in `auth_middleware._PUBLIC_EXACT_PATHS`. The sha
|
||||
|
||||
Both can be modified at runtime via Gateway API endpoints or `DeerFlowClient` methods.
|
||||
|
||||
### WeKnora Retrieval Skill
|
||||
### Conditional WeKnora LLMWiki
|
||||
|
||||
`skills/custom/weknora-knowledge-search/` is a read-only integration for a WeKnora tenant. The Skill itself only directs the model to two MCP tools, `weknora_search` and `weknora_list_knowledge_bases`; the credential remains inside the stdio MCP subprocess (`scripts/weknora_retrieval_mcp.py`). Its retrieval core groups document/FAQ knowledge bases by `embedding_model_id` and calls WeKnora's multi-KB `hybrid-search` API once per group, preventing invalid cross-embedding score comparisons. It does not discard a KB merely because the legacy list API reports `chunk_count=0`; hybrid-search decides its actual readiness. Wiki pages are opt-in and use their dedicated Wiki search route. In explicit DeerFlow sharing mode, `WEKNORA_DEERFLOW_SHARE_TOKEN_FILE` selects WeKnora's narrow `/api/v1/deerflow/*` capability API: only owner-published document/FAQ knowledge bases are discoverable and searchable, while Wiki is skipped so the skill cannot fall back to a user bearer-token endpoint. Shared document hits include a credential-free source-preview URL. The frontend source card recognizes it and opens a context dialog; its call to authenticated Gateway route `/api/weknora/source-context` is proxied to WeKnora's token-gated `/api/v1/deerflow/knowledge-bases/:id/chunks/:chunk_id/context`, returning only the matching chunk and two neighbours on either side. Never expose the share token to the browser or add it to a URL. Configure through the bundled `templates/extensions_config.fragment.json`; otherwise prefer `WEKNORA_API_KEY_FILE` for scoped-key deployments and `WEKNORA_BEARER_TOKEN_FILE` for legacy deployments. `WEKNORA_ALLOWED_KB_IDS` remains an optional allow-list. Tests: `tests/test_weknora_knowledge_search_skill.py`, `tests/test_weknora_source_preview.py`.
|
||||
`llmwiki.weknora.api_base_url` is the provider switch: non-empty enables WeKnora, empty preserves the legacy LLMWiki UI. WeKnora is independently deployed and owns PostgreSQL/Redis/object storage/vector storage/parsers/models; DeerFlow config contains only `api_base_url` and optional `web_base_url`. DeerFlow deliberately sends no WeKnora credential, so the WeKnora endpoint must be restricted to a trusted network and allow DeerFlow traffic itself or through its reverse proxy. There is intentionally no `default_embedding_model_id`: `WeKnoraClient` queries `/api/v1/models` and uses WeKnora's active Embedding model when creating a KB; Wiki creation additionally resolves an active KnowledgeQA model inside WeKnora for synthesis. The native BFF is `app/gateway/routers/llmwiki.py`; ownership/publication mappings live in `deerflow.persistence.llmwiki`; the browser only sees DeerFlow mapping IDs. Knowledge-base owners publish immediately (`private` -> `published`) without a review queue, and can unpublish. Knowledge cards navigate to a dedicated WeKnora-style detail route. Wiki-enabled bases render and default to `Wiki -> documents -> graph`; standard bases omit Wiki and render `documents -> graph`. The detail page supports file, URL and published manual-Markdown imports, browses real Wiki pages/content, proxies authenticated original-file previews, pages through `/chunks/{knowledge_id}`, and renders the Wiki page graph from `/knowledgebase/{kb_id}/wiki/graph`; document details use a right sheet rather than the removed KB modal. The credential-free proxy uses one WeKnora service identity, so per-user separation is enforced by DeerFlow mappings while all remote KBs physically share that identity's WeKnora workspace. `llmwiki_search` is conditionally registered in `tools.py`, is read-only, and rechecks the current DeerFlow user, agent `llmwiki_knowledge_base_ids`, and thread-context selection on every call. Agent bindings use an inline WeKnora-style searchable/grouped multi-select in the standard agent create/detail UI; new chats use the same interaction pattern as a compact composer dropdown, and selections are written to thread metadata/context and restored on reopen. The DeerFlow frontend is native React and the WeKnora frontend is not modified or embedded. Full deployment and behavior notes: `docs/LLMWIKI_WEKNORA_INTEGRATION_ZH.md`. Tests: `tests/test_llmwiki_weknora.py`.
|
||||
|
||||
### Embedded Client (`packages/harness/deerflow/client.py`)
|
||||
|
||||
@@ -1133,7 +1134,7 @@ not delete a built-in id because historical chain nodes persist those ids;
|
||||
custom role ids may be added and removed.
|
||||
|
||||
See `docs/` directory for detailed documentation:
|
||||
- [../README.md](../README.md) - Offline Docker wheelhouse preparation and startup workflow. The pinned versions in `../offline-runtime-requirements.txt` and `../start_backend.sh` must stay synchronized. `start_backend.sh` stays aligned with the deployment script and adds only the Redis wheel fallback line. The download target defaults to CPython 3.12 on Linux x86_64 and verifies dependency closure using wheelhouse only.
|
||||
- [../README.md](../README.md) - Offline Docker wheelhouse preparation and startup workflow. The pinned versions in `../offline-runtime-requirements.txt` and `../start_backend.sh` must stay synchronized. Startup checks `croniter`, `hindsight-client`, `aiomysql`, and Redis, installing missing packages from wheelhouse only; `aiomysql==0.3.2` includes the pinned `PyMySQL==1.2.0` transitive dependency. The download target defaults to CPython 3.12 on Linux x86_64 and verifies dependency closure using wheelhouse only.
|
||||
- [CONFIGURATION.md](docs/CONFIGURATION.md) - Configuration options
|
||||
- [ARCHITECTURE.md](docs/ARCHITECTURE.md) - Architecture details
|
||||
- [API.md](docs/API.md) - API reference
|
||||
|
||||
@@ -39,6 +39,8 @@ DeerFlow is a LangGraph-based AI super agent with sandbox execution, persistent
|
||||
- `/api/*` (other) → Gateway API - models, MCP, skills, memory, artifacts, uploads, thread-local cleanup
|
||||
- `/` (non-API) → Frontend - Next.js web interface
|
||||
|
||||
Health probes are always unauthenticated, even when API authentication is enabled: `GET|HEAD /health`, `/api/health`, and `/api/langgraph/health` all return the Gateway health status without requiring a cookie or bearer token.
|
||||
|
||||
---
|
||||
|
||||
## Core Components
|
||||
@@ -78,7 +80,7 @@ Per-thread isolated execution with virtual path translation:
|
||||
- **Virtual paths**: `/mnt/user-data/{workspace,uploads,outputs}` → thread-specific physical directories
|
||||
- **Skills path**: `/mnt/skills` → `deer-flow/skills/` directory
|
||||
- **Skills loading**: Recursively discovers nested `SKILL.md` files under `skills/{public,custom}` and preserves nested container paths
|
||||
- **WeKnora retrieval skill**: `skills/custom/weknora-knowledge-search` pairs a concise Skill with a stdio MCP server. It retrieves tenant-scoped WeKnora document/FAQ passages through the public API, groups multi-KB calls by embedding model, optionally searches Wiki pages, and can use the local reranker. It does not reject a KB solely because the legacy list API reports `chunk_count=0`; hybrid-search remains the source of truth. For WeKnora's explicit DeerFlow sharing mode, configure `WEKNORA_DEERFLOW_SHARE_TOKEN_FILE`: the MCP uses the restricted `/api/v1/deerflow/*` capability API and can only list/search knowledge bases whose owner enabled sharing; it never receives a user bearer token. Shared document results now contain a credential-free `url` pointer; the source card's “查看原文” dialog calls DeerFlow's authenticated `/api/weknora/source-context` proxy, which retrieves the hit plus two adjacent chunks while keeping the share token server-side. Shared mode currently excludes Wiki pages. Otherwise use `WEKNORA_API_KEY_FILE` for scoped-key deployments or `WEKNORA_BEARER_TOKEN_FILE` for legacy deployments; never put any credential in `SKILL.md` or a chat prompt.
|
||||
- **Conditional WeKnora LLMWiki**: a non-empty `llmwiki.weknora.api_base_url` enables the DeerFlow-native WeKnora BFF and UI; an empty address preserves the existing LLMWiki page. WeKnora remains independently deployed and owns its database, Redis, storage, parsers and models. DeerFlow keeps only the API/admin addresses, user/publication mappings, agent knowledge-base bindings and per-conversation selections; it stores and sends no WeKnora credential, so the endpoint must be restricted to a trusted network and allow DeerFlow traffic. Owners publish directly to the public LLMWiki without an approval queue. Knowledge cards open a dedicated WeKnora-style detail route: Wiki-enabled bases default to Wiki, then documents and graph; standard bases hide Wiki and default to documents. It supports file/URL/manual-Markdown imports, Wiki page browsing, document preview, real parsed chunks and Wiki graph nodes/edges. New chats use a compact WeKnora-style searchable/grouped multi-select inside the composer, while agent create/edit pages bind spaces through an inline panel. A single proxy service identity means DeerFlow users are isolated logically while their WeKnora entities share that identity's workspace. The read-only `llmwiki_search` tool reauthorizes the current DeerFlow user on every call and renders credential-free source previews. See [docs/LLMWIKI_WEKNORA_INTEGRATION_ZH.md](docs/LLMWIKI_WEKNORA_INTEGRATION_ZH.md).
|
||||
- **File-write safety**: `str_replace` serializes read-modify-write per `(sandbox.id, path)` so isolated sandboxes keep concurrency even when virtual paths match
|
||||
- **Tools**: `bash`, `ls`, `read_file`, `write_file`, `str_replace` (`bash` is disabled by default when using `LocalSandboxProvider`; use `AioSandboxProvider` for isolated shell access)
|
||||
|
||||
@@ -111,6 +113,19 @@ LLM-powered persistent context retention across conversations:
|
||||
| **MCP** | Any Model Context Protocol server (stdio, SSE, HTTP transports) |
|
||||
| **Skills** | Domain-specific workflows injected via system prompt |
|
||||
|
||||
#### Configurable HTTP web search
|
||||
|
||||
The config-defined `web_search` tool can call a deployment-specific JSON API
|
||||
instead of a public search provider. Set its `endpoint`, static `payload`, SSL
|
||||
verification, timeout, result limit, and `result_url_template` in `config.yaml`.
|
||||
At runtime only `payload.query` is overwritten with the model's search text.
|
||||
The API response must use `{"results": [{"recUuid": "", "content": "",
|
||||
"title": "", "url": ""}]}`; missing or null result fields are normalized to empty strings. When
|
||||
`recUuid` is present, `{recUuid}` in `result_url_template` replaces the original
|
||||
result URL; otherwise the original URL is retained. Set `enabled: false` to keep
|
||||
the tool completely out of the agent's registered tool list. `verify_ssl: false`
|
||||
supports trusted internal HTTPS services with self-signed certificates.
|
||||
|
||||
### Gateway API
|
||||
|
||||
FastAPI application providing REST endpoints for frontend integration:
|
||||
|
||||
@@ -1,26 +1,26 @@
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import queue
|
||||
import sys
|
||||
import threading
|
||||
from collections.abc import AsyncGenerator
|
||||
from contextlib import asynccontextmanager
|
||||
from datetime import datetime
|
||||
from time import perf_counter
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from fastapi import FastAPI
|
||||
import os
|
||||
import queue
|
||||
import sys
|
||||
import threading
|
||||
from collections.abc import AsyncGenerator
|
||||
from contextlib import asynccontextmanager
|
||||
from datetime import datetime
|
||||
from time import perf_counter
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
# Load .env BEFORE any module that reads os.environ at import time.
|
||||
# Without this, GATEWAY_CORS_ORIGINS / AUTH_JWT_SECRET / DEER_FLOW_AUTH_DISABLED
|
||||
# placed in /workspace/.env are silently ignored when the container is started
|
||||
# without explicit `-e` flags, leading to ephemeral JWT secrets, missing CORS
|
||||
# origins, and "Failed to fetch" in the browser.
|
||||
load_dotenv()
|
||||
|
||||
from app.gateway.auth.disabled_mode import is_auth_disabled # noqa: E402
|
||||
# origins, and "Failed to fetch" in the browser.
|
||||
load_dotenv()
|
||||
|
||||
from app.gateway.auth.disabled_mode import is_auth_disabled # noqa: E402
|
||||
from app.gateway.auth_middleware import AuthMiddleware
|
||||
from app.gateway.config import get_gateway_config
|
||||
from app.gateway.csrf_middleware import CSRFMiddleware
|
||||
@@ -43,16 +43,17 @@ from app.gateway.routers import (
|
||||
intent,
|
||||
knowledge,
|
||||
knowledge_ingest,
|
||||
light_apps,
|
||||
llm_metrics,
|
||||
mcp,
|
||||
light_apps,
|
||||
llm_metrics,
|
||||
llmwiki,
|
||||
mcp,
|
||||
memory,
|
||||
menu_overrides,
|
||||
models,
|
||||
multi_agent,
|
||||
notifications,
|
||||
open_3qfx,
|
||||
open_chat,
|
||||
open_chat,
|
||||
parallel_agent_tasks,
|
||||
parallel_agents,
|
||||
position_roundtable,
|
||||
@@ -89,8 +90,7 @@ from app.gateway.routers import (
|
||||
uploads,
|
||||
user_preferences,
|
||||
user_prompts,
|
||||
weknora_sources,
|
||||
writing,
|
||||
writing,
|
||||
)
|
||||
from deerflow.config import app_config as deerflow_app_config
|
||||
from deerflow.config.app_config import apply_logging_level
|
||||
@@ -1430,8 +1430,8 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
|
||||
# MCP API is mounted at /api/mcp
|
||||
app.include_router(mcp.router)
|
||||
|
||||
# Authenticated source-context proxy for the WeKnora retrieval skill.
|
||||
app.include_router(weknora_sources.router)
|
||||
# DeerFlow-native LLMWiki management/search BFF (conditionally backed by WeKnora).
|
||||
app.include_router(llmwiki.router)
|
||||
|
||||
# Memory API is mounted at /api/memory
|
||||
app.include_router(memory.router)
|
||||
@@ -1622,11 +1622,12 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
|
||||
app.include_router(recommend.router) # Step 1?2?/api/recommend/stream?? init?
|
||||
# Step 3 ????/?????? artifacts.router?/api/threads/{id}/artifacts?
|
||||
|
||||
# ?????????(? auth_middleware._PUBLIC_PATH_PREFIXES)?
|
||||
# ????? ``/health`` ?? API ??? ``/api/health``,?????
|
||||
# ?? base URL ????,?????????? 401?
|
||||
@app.get("/health", tags=["health"])
|
||||
@app.get("/api/health", tags=["health"])
|
||||
# Health probes are always unauthenticated (see
|
||||
# auth_middleware.PUBLIC_HEALTH_PATHS). Support the bare Gateway path,
|
||||
# API base URL, and LangGraph proxy base URL for deployment monitors.
|
||||
@app.api_route("/health", methods=["GET", "HEAD"], tags=["health"])
|
||||
@app.api_route("/api/health", methods=["GET", "HEAD"], tags=["health"])
|
||||
@app.api_route("/api/langgraph/health", methods=["GET", "HEAD"], tags=["health"])
|
||||
async def health_check() -> dict:
|
||||
"""Health check endpoint.
|
||||
|
||||
|
||||
@@ -24,15 +24,25 @@ from app.gateway.internal_auth import INTERNAL_AUTH_HEADER_NAME, get_internal_us
|
||||
from app.gateway.proxy_path import app_relative_path
|
||||
from deerflow.runtime.user_context import reset_current_user, set_current_user
|
||||
|
||||
# Paths that never require authentication.
|
||||
# Health probes are deliberately public, including the LangGraph-compatible
|
||||
# gateway alias used by some deployments. Keep these as exact paths: health
|
||||
# checks must bypass authentication, but similarly named application routes
|
||||
# must not inherit that exemption.
|
||||
PUBLIC_HEALTH_PATHS: frozenset[str] = frozenset(
|
||||
{
|
||||
"/health",
|
||||
"/api/health",
|
||||
"/api/langgraph/health",
|
||||
}
|
||||
)
|
||||
|
||||
# Path prefixes that never require authentication.
|
||||
#
|
||||
# ``/api/public/`` is a namespace reserved for intentionally world-readable,
|
||||
# read-only endpoints (e.g. public conversation share snapshots). Only mount
|
||||
# safe read-only routes under it — everything there bypasses auth entirely.
|
||||
_PUBLIC_PATH_PREFIXES: tuple[str, ...] = (
|
||||
"/health",
|
||||
"/api/health",
|
||||
"/docs",
|
||||
_PUBLIC_PATH_PREFIXES: tuple[str, ...] = (
|
||||
"/docs",
|
||||
"/redoc",
|
||||
"/openapi.json",
|
||||
"/api/public/",
|
||||
@@ -43,9 +53,10 @@ _PUBLIC_PATH_PREFIXES: tuple[str, ...] = (
|
||||
|
||||
# Exact auth paths that are public (login/register/status check).
|
||||
# /api/v1/auth/me, /api/v1/auth/change-password etc. are NOT public.
|
||||
_PUBLIC_EXACT_PATHS: frozenset[str] = frozenset(
|
||||
{
|
||||
"/api/v1/auth/login/local",
|
||||
_PUBLIC_EXACT_PATHS: frozenset[str] = frozenset(
|
||||
{
|
||||
*PUBLIC_HEALTH_PATHS,
|
||||
"/api/v1/auth/login/local",
|
||||
"/api/v1/auth/login/username",
|
||||
"/api/v1/auth/login/token",
|
||||
"/api/v1/auth/register",
|
||||
|
||||
@@ -293,10 +293,16 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
|
||||
app.state.dashboard_session_store = make_dashboard_session_store(sf)
|
||||
|
||||
from deerflow.persistence.task_buttons import make_task_button_store
|
||||
|
||||
from deerflow.persistence.task_buttons import make_task_button_store
|
||||
|
||||
app.state.task_button_store = make_task_button_store(sf)
|
||||
|
||||
from deerflow.integrations.weknora.runtime import register_llmwiki_store
|
||||
from deerflow.persistence.llmwiki import make_llmwiki_store
|
||||
|
||||
app.state.llmwiki_store = make_llmwiki_store(sf)
|
||||
register_llmwiki_store(app.state.llmwiki_store)
|
||||
|
||||
from deerflow.persistence.taskcop_tasks import make_taskcop_task_store
|
||||
|
||||
app.state.taskcop_task_store = make_taskcop_task_store(sf)
|
||||
|
||||
@@ -0,0 +1,241 @@
|
||||
"""Asynchronous conversation sedimentation into WeKnora-backed LLMWiki."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from deerflow.integrations.weknora.client import WeKnoraClient, WeKnoraError
|
||||
from deerflow.integrations.weknora.runtime import build_weknora_client, get_resolved_llmwiki_runtime
|
||||
from deerflow.persistence.llmwiki import LlmWikiStore
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
CONVERSATION_DEPOSIT_KB_NAME = "对话沉淀"
|
||||
_DEPOSIT_KB_LOCK = asyncio.Lock()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ConversationTurnDeposit:
|
||||
thread_id: str
|
||||
question: str
|
||||
answer: str
|
||||
human_message_id: str | None
|
||||
assistant_message_id: str
|
||||
assistant_id: str | None
|
||||
knowledge_base_ids: list[str]
|
||||
created_by_user_id: str | None
|
||||
|
||||
|
||||
def _store(app: Any) -> LlmWikiStore | None:
|
||||
return getattr(app.state, "llmwiki_store", None)
|
||||
|
||||
|
||||
def _client(app: Any) -> WeKnoraClient | None:
|
||||
runtime = get_resolved_llmwiki_runtime(getattr(app.state, "config", None))
|
||||
if not runtime.weknora_enabled:
|
||||
return None
|
||||
try:
|
||||
return build_weknora_client(runtime)
|
||||
except RuntimeError:
|
||||
logger.debug("LLMWiki conversation deposit skipped: WeKnora client is unavailable", exc_info=True)
|
||||
return None
|
||||
|
||||
|
||||
def _shorten(text: str, max_chars: int) -> str:
|
||||
value = " ".join(str(text or "").split())
|
||||
if len(value) <= max_chars:
|
||||
return value
|
||||
return f"{value[: max_chars - 1].rstrip()}…"
|
||||
|
||||
|
||||
def _title_for_turn(turn: ConversationTurnDeposit) -> str:
|
||||
question = _shorten(turn.question, 48)
|
||||
return f"对话沉淀 - {question or turn.assistant_message_id[:12]}"
|
||||
|
||||
|
||||
def _format_markdown(turn: ConversationTurnDeposit, title: str) -> str:
|
||||
kb_line = "、".join(turn.knowledge_base_ids) if turn.knowledge_base_ids else "未显式选择知识库"
|
||||
return f"""---
|
||||
source: DeerFlow
|
||||
type: conversation-deposit
|
||||
thread_id: {turn.thread_id}
|
||||
human_message_id: {turn.human_message_id or ""}
|
||||
assistant_message_id: {turn.assistant_message_id}
|
||||
assistant_id: {turn.assistant_id or ""}
|
||||
knowledge_base_ids: {kb_line}
|
||||
created_by_user_id: {turn.created_by_user_id or ""}
|
||||
---
|
||||
|
||||
# {title}
|
||||
|
||||
> 本页由 DeerFlow 在一轮问答结束后自动沉淀,用于后续 LLMWiki 检索、Wiki 索引与图谱构建。
|
||||
|
||||
## 摘要
|
||||
|
||||
{_shorten(turn.answer, 360) or "本轮回答为空。"}
|
||||
|
||||
## 用户问题
|
||||
|
||||
{turn.question.strip() or "(空)"}
|
||||
|
||||
## 助手回答
|
||||
|
||||
{turn.answer.strip() or "(空)"}
|
||||
|
||||
## 结构化线索
|
||||
|
||||
- 来源线程:`{turn.thread_id}`
|
||||
- 用户消息:`{turn.human_message_id or ""}`
|
||||
- 回答消息:`{turn.assistant_message_id}`
|
||||
- 智能体:`{turn.assistant_id or "通用问答"}`
|
||||
- 本轮显式知识库:{kb_line}
|
||||
|
||||
## LLMWiki 标签
|
||||
|
||||
#对话沉淀 #DeerFlow #问答知识
|
||||
"""
|
||||
|
||||
|
||||
async def ensure_conversation_deposit_knowledge_base(
|
||||
app: Any,
|
||||
*,
|
||||
owner_user_id: str | None,
|
||||
) -> dict[str, Any] | None:
|
||||
store = _store(app)
|
||||
client = _client(app)
|
||||
if store is None or client is None:
|
||||
return None
|
||||
|
||||
existing = await store.get_conversation_deposit_mapping()
|
||||
if existing is not None:
|
||||
return existing
|
||||
|
||||
async with _DEPOSIT_KB_LOCK:
|
||||
existing = await store.get_conversation_deposit_mapping()
|
||||
if existing is not None:
|
||||
return existing
|
||||
|
||||
remote_id = ""
|
||||
try:
|
||||
for remote in await client.list_knowledge_bases():
|
||||
if str(remote.get("name") or "").strip() == CONVERSATION_DEPOSIT_KB_NAME and remote.get("id"):
|
||||
remote_id = str(remote["id"])
|
||||
break
|
||||
except WeKnoraError:
|
||||
logger.debug("Could not scan WeKnora knowledge bases before creating deposit base", exc_info=True)
|
||||
|
||||
if remote_id:
|
||||
mapped = await store.get_by_weknora_id(remote_id)
|
||||
if mapped is not None:
|
||||
if mapped.get("publication_status") != "published":
|
||||
return await store.request_publish(
|
||||
str(mapped["id"]),
|
||||
owner_user_id or "system",
|
||||
is_admin=True,
|
||||
)
|
||||
return mapped
|
||||
remote = {"id": remote_id, "name": CONVERSATION_DEPOSIT_KB_NAME, "description": "DeerFlow 全局问答沉淀知识库", "type": "document"}
|
||||
else:
|
||||
try:
|
||||
remote = await client.create_knowledge_base(
|
||||
name=CONVERSATION_DEPOSIT_KB_NAME,
|
||||
description="DeerFlow 全局问答沉淀知识库。所有用户可见,不默认参与问答检索。",
|
||||
kb_type="document",
|
||||
wiki_enabled=True,
|
||||
)
|
||||
except WeKnoraError:
|
||||
logger.warning("Failed to create WeKnora conversation deposit knowledge base", exc_info=True)
|
||||
return None
|
||||
|
||||
try:
|
||||
row = await store.create_mapping(
|
||||
weknora_id=str(remote["id"]),
|
||||
owner_user_id=owner_user_id or "system",
|
||||
name=CONVERSATION_DEPOSIT_KB_NAME,
|
||||
description=str(remote.get("description") or "DeerFlow 全局问答沉淀知识库"),
|
||||
kb_type="document",
|
||||
)
|
||||
published = await store.request_publish(str(row["id"]), owner_user_id or "system", is_admin=True)
|
||||
return published or row
|
||||
except Exception:
|
||||
logger.exception("Failed to persist conversation deposit knowledge-base mapping")
|
||||
return None
|
||||
|
||||
|
||||
async def deposit_conversation_turn(app: Any, turn: ConversationTurnDeposit) -> None:
|
||||
if not turn.thread_id or not turn.assistant_message_id:
|
||||
return
|
||||
if not turn.question.strip() and not turn.answer.strip():
|
||||
return
|
||||
|
||||
store = _store(app)
|
||||
client = _client(app)
|
||||
if store is None or client is None:
|
||||
return
|
||||
|
||||
kb = await ensure_conversation_deposit_knowledge_base(app, owner_user_id=turn.created_by_user_id)
|
||||
if kb is None:
|
||||
return
|
||||
|
||||
previous = await store.get_conversation_deposit(
|
||||
thread_id=turn.thread_id,
|
||||
assistant_message_id=turn.assistant_message_id,
|
||||
)
|
||||
previous_doc_id = str((previous or {}).get("weknora_knowledge_id") or "")
|
||||
if previous_doc_id:
|
||||
try:
|
||||
await client.delete_document(previous_doc_id)
|
||||
except WeKnoraError as exc:
|
||||
if exc.status_code != 404:
|
||||
logger.warning("Failed to delete old conversation deposit document %s", previous_doc_id, exc_info=True)
|
||||
|
||||
title = _title_for_turn(turn)
|
||||
content = _format_markdown(turn, title)
|
||||
try:
|
||||
remote_doc = await client.create_manual_document(
|
||||
str(kb["weknora_id"]),
|
||||
title=title,
|
||||
content=content,
|
||||
)
|
||||
except WeKnoraError:
|
||||
logger.warning("Failed to create conversation deposit document", exc_info=True)
|
||||
return
|
||||
|
||||
await store.upsert_conversation_deposit(
|
||||
thread_id=turn.thread_id,
|
||||
human_message_id=turn.human_message_id,
|
||||
assistant_message_id=turn.assistant_message_id,
|
||||
knowledge_base_mapping_id=str(kb["id"]),
|
||||
weknora_knowledge_id=str(remote_doc.get("id") or remote_doc.get("knowledge_id") or ""),
|
||||
title=title,
|
||||
created_by_user_id=turn.created_by_user_id,
|
||||
)
|
||||
|
||||
|
||||
async def delete_conversation_deposit_documents(
|
||||
app: Any,
|
||||
*,
|
||||
thread_id: str,
|
||||
message_ids: list[str] | None = None,
|
||||
) -> None:
|
||||
store = _store(app)
|
||||
client = _client(app)
|
||||
if store is None or client is None:
|
||||
return
|
||||
try:
|
||||
rows = await store.delete_conversation_deposits(thread_id=thread_id, message_ids=message_ids)
|
||||
except Exception:
|
||||
logger.debug("Failed to delete local conversation deposit rows", exc_info=True)
|
||||
return
|
||||
for row in rows:
|
||||
doc_id = str(row.get("weknora_knowledge_id") or "")
|
||||
if not doc_id:
|
||||
continue
|
||||
try:
|
||||
await client.delete_document(doc_id)
|
||||
except WeKnoraError as exc:
|
||||
if exc.status_code != 404:
|
||||
logger.warning("Failed to delete WeKnora conversation deposit document %s", doc_id, exc_info=True)
|
||||
@@ -0,0 +1,290 @@
|
||||
"""LLMWiki retrieval helpers for normal DeerFlow chat runs.
|
||||
|
||||
The public chat API accepts DeerFlow-local LLMWiki knowledge-base mapping ids.
|
||||
When WeKnora mode is enabled, this module resolves those local ids under the
|
||||
current DeerFlow user's permissions, searches WeKnora once before the model
|
||||
call, and attaches both:
|
||||
|
||||
* a hidden prompt with numbered context snippets, and
|
||||
* structured citation sources for the existing reference panel.
|
||||
|
||||
The helper lives in ``app.gateway`` because it needs Gateway auth state and the
|
||||
DeerFlow-owned mapping store. Harness code remains app-import free.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
from urllib.parse import urlencode
|
||||
|
||||
from fastapi import HTTPException, Request
|
||||
from langchain_core.messages import HumanMessage
|
||||
|
||||
from app.gateway.deps import get_optional_user_from_request
|
||||
from app.gateway.utils import sanitize_log_param
|
||||
from deerflow.config.agents_config import load_agent_config, validate_agent_id
|
||||
from deerflow.integrations.weknora.client import WeKnoraError
|
||||
from deerflow.integrations.weknora.runtime import build_weknora_client, get_resolved_llmwiki_runtime
|
||||
from deerflow.persistence.llmwiki import LlmWikiStore
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_MAX_SOURCES = 8
|
||||
_MAX_SOURCE_CHARS = 1400
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LlmWikiRagContext:
|
||||
knowledge_base_ids: list[str]
|
||||
source_count: int
|
||||
prompt: str
|
||||
sources: list[dict[str, Any]]
|
||||
query: str
|
||||
|
||||
|
||||
def _store(request: Request) -> LlmWikiStore | None:
|
||||
return getattr(request.app.state, "llmwiki_store", None)
|
||||
|
||||
|
||||
async def _actor(request: Request) -> tuple[str, bool]:
|
||||
user = await get_optional_user_from_request(request)
|
||||
if user is None:
|
||||
return get_effective_user_id(), True
|
||||
return str(user.id), getattr(user, "system_role", None) == "admin"
|
||||
|
||||
|
||||
def _as_id_list(value: Any) -> list[str]:
|
||||
if value is None:
|
||||
return []
|
||||
if isinstance(value, str):
|
||||
parts = [value]
|
||||
elif isinstance(value, (list, tuple, set)):
|
||||
parts = list(value)
|
||||
else:
|
||||
return []
|
||||
return list(dict.fromkeys(str(item).strip() for item in parts if str(item).strip()))
|
||||
|
||||
|
||||
def _body_selected_ids(body: Any) -> list[str]:
|
||||
for source in (getattr(body, "context", None), getattr(body, "metadata", None)):
|
||||
if isinstance(source, Mapping) and "llmwiki_knowledge_base_ids" in source:
|
||||
return _as_id_list(source.get("llmwiki_knowledge_base_ids"))
|
||||
return []
|
||||
|
||||
|
||||
def _assistant_agent_id(body: Any) -> str | None:
|
||||
assistant_id = validate_agent_id(getattr(body, "assistant_id", None))
|
||||
if assistant_id:
|
||||
return assistant_id
|
||||
context = getattr(body, "context", None)
|
||||
if isinstance(context, Mapping):
|
||||
return validate_agent_id(str(context.get("agent_id") or "")) if context.get("agent_id") else None
|
||||
metadata = getattr(body, "metadata", None)
|
||||
if isinstance(metadata, Mapping):
|
||||
return validate_agent_id(str(metadata.get("agent_id") or "")) if metadata.get("agent_id") else None
|
||||
return None
|
||||
|
||||
|
||||
def _agent_bound_ids(body: Any) -> list[str]:
|
||||
agent_id = _assistant_agent_id(body)
|
||||
if not agent_id:
|
||||
return []
|
||||
try:
|
||||
config = load_agent_config(agent_id)
|
||||
except (FileNotFoundError, ValueError):
|
||||
return []
|
||||
return _as_id_list(config.llmwiki_knowledge_base_ids if config else None)
|
||||
|
||||
|
||||
def resolve_requested_llmwiki_knowledge_base_ids(body: Any) -> list[str]:
|
||||
"""Return selected ids plus custom-agent defaults, de-duplicated in order."""
|
||||
|
||||
return list(dict.fromkeys([*_agent_bound_ids(body), *_body_selected_ids(body)]))
|
||||
|
||||
|
||||
def llmwiki_selection_requested(body: Any) -> bool:
|
||||
return bool(resolve_requested_llmwiki_knowledge_base_ids(body))
|
||||
|
||||
|
||||
def _latest_human_query(graph_input: dict[str, Any]) -> str:
|
||||
messages = graph_input.get("messages")
|
||||
if not isinstance(messages, list):
|
||||
return ""
|
||||
for message in reversed(messages):
|
||||
if isinstance(message, HumanMessage):
|
||||
return _message_text(getattr(message, "content", ""))
|
||||
if isinstance(message, dict) and str(message.get("type") or message.get("role") or "") in {"human", "user"}:
|
||||
return _message_text(message.get("content"))
|
||||
return ""
|
||||
|
||||
|
||||
def _message_text(content: Any) -> str:
|
||||
if isinstance(content, str):
|
||||
return content.strip()
|
||||
if isinstance(content, list):
|
||||
parts: list[str] = []
|
||||
for item in content:
|
||||
if isinstance(item, str):
|
||||
parts.append(item)
|
||||
elif isinstance(item, Mapping):
|
||||
text = item.get("text") or item.get("content")
|
||||
if isinstance(text, str):
|
||||
parts.append(text)
|
||||
return "\n".join(part.strip() for part in parts if part and part.strip()).strip()
|
||||
return str(content or "").strip()
|
||||
|
||||
|
||||
def _truncate(text: str, max_chars: int) -> str:
|
||||
value = text.strip()
|
||||
if len(value) <= max_chars:
|
||||
return value
|
||||
return f"{value[: max_chars - 1].rstrip()}…"
|
||||
|
||||
|
||||
def _source_from_result(result: dict[str, Any], mapping: dict[str, Any], index: int) -> dict[str, Any]:
|
||||
chunk_id = str(result.get("chunk_id") or "")
|
||||
title = str(result.get("title") or result.get("filename") or mapping.get("name") or f"LLMWiki 片段 {index}")
|
||||
content = str(result.get("content") or "")
|
||||
params = urlencode({"knowledge_base_id": mapping["id"], "chunk_id": chunk_id})
|
||||
raw = {
|
||||
**result,
|
||||
"knowledge_base_id": mapping["id"],
|
||||
"knowledge_base_name": mapping.get("name") or "",
|
||||
"source": mapping.get("name") or "LLMWiki",
|
||||
"url": f"/weknora-source-preview?{params}",
|
||||
}
|
||||
return {
|
||||
"index": index,
|
||||
"id": chunk_id or str(result.get("knowledge_id") or f"{mapping['id']}:{index}"),
|
||||
"title": title,
|
||||
"type": "llmwiki",
|
||||
"snippet": _truncate(content, 360),
|
||||
"content": content,
|
||||
"source": mapping.get("name") or "LLMWiki",
|
||||
"score": float(result.get("score") or 0),
|
||||
"url": f"/weknora-source-preview?{params}",
|
||||
"skillName": "llmwiki_search",
|
||||
"mode": "citation",
|
||||
"raw": raw,
|
||||
}
|
||||
|
||||
|
||||
def _build_prompt(query: str, sources: list[dict[str, Any]]) -> str:
|
||||
if not sources:
|
||||
return (
|
||||
"当前用户选择了 LLMWiki 知识库,但 DeerFlow 未检索到可用片段。"
|
||||
"请直接说明没有检索到可用背景资料,不要编造知识库内容或引用编号。"
|
||||
)
|
||||
|
||||
lines = [
|
||||
"当前用户选择了 LLMWiki 知识库。DeerFlow 已在模型回答前自动检索到以下背景资料。",
|
||||
"请优先依据这些资料回答;如果资料不足,请明确说明不足之处,不要编造。",
|
||||
"引用规则:正文关键结论必须使用 [1]、[2] 这样的编号引用下方资料;只能引用存在的编号;不要在正文末尾重复输出完整参考文献列表,前端会展示引用卡片。",
|
||||
"",
|
||||
f"用户问题:{query}",
|
||||
"",
|
||||
f"<llmwiki_reference_batch source_count=\"{len(sources)}\">",
|
||||
]
|
||||
for source in sources:
|
||||
meta = " / ".join(
|
||||
str(item)
|
||||
for item in (
|
||||
source.get("source"),
|
||||
f"score={source.get('score'):.4f}" if isinstance(source.get("score"), float) else None,
|
||||
)
|
||||
if item
|
||||
)
|
||||
lines.append(f"[{source['index']}] {source.get('title') or 'LLMWiki 片段'}")
|
||||
if meta:
|
||||
lines.append(f"来源:{meta}")
|
||||
lines.append(_truncate(str(source.get("content") or source.get("snippet") or ""), _MAX_SOURCE_CHARS))
|
||||
lines.append("")
|
||||
lines.append("</llmwiki_reference_batch>")
|
||||
return "\n".join(lines).strip()
|
||||
|
||||
|
||||
async def build_llmwiki_rag_context(
|
||||
*,
|
||||
request: Request,
|
||||
body: Any,
|
||||
graph_input: dict[str, Any],
|
||||
) -> LlmWikiRagContext:
|
||||
selected_ids = resolve_requested_llmwiki_knowledge_base_ids(body)
|
||||
if not selected_ids:
|
||||
return LlmWikiRagContext([], 0, "", [], "")
|
||||
|
||||
runtime = get_resolved_llmwiki_runtime(getattr(request.app.state, "config", None))
|
||||
if not runtime.weknora_enabled:
|
||||
return LlmWikiRagContext(selected_ids, 0, "", [], "")
|
||||
store = _store(request)
|
||||
if store is None:
|
||||
return LlmWikiRagContext(selected_ids, 0, "", [], "")
|
||||
|
||||
query = _latest_human_query(graph_input)
|
||||
if not query:
|
||||
return LlmWikiRagContext(selected_ids, 0, "", [], "")
|
||||
|
||||
user_id, is_admin = await _actor(request)
|
||||
rows: list[dict[str, Any]] = []
|
||||
for mapping_id in selected_ids:
|
||||
row = await store.get_authorized(mapping_id, user_id, write=False, is_admin=is_admin)
|
||||
if row is None:
|
||||
raise HTTPException(status_code=404, detail=f"Knowledge base {mapping_id} not found")
|
||||
rows.append(row)
|
||||
remote_to_local = {str(row["weknora_id"]): row for row in rows}
|
||||
|
||||
try:
|
||||
results = await build_weknora_client(runtime).search(query, [str(row["weknora_id"]) for row in rows])
|
||||
except WeKnoraError as exc:
|
||||
logger.warning(
|
||||
"LLMWiki pre-retrieval failed for user=%s kb_count=%s: %s",
|
||||
sanitize_log_param(user_id),
|
||||
len(rows),
|
||||
exc,
|
||||
)
|
||||
return LlmWikiRagContext(
|
||||
selected_ids,
|
||||
0,
|
||||
f"当前用户选择了 LLMWiki 知识库,但自动检索失败:{exc}。请说明暂时无法检索知识库背景资料,不要编造引用。",
|
||||
[],
|
||||
query,
|
||||
)
|
||||
|
||||
sources: list[dict[str, Any]] = []
|
||||
for result in results:
|
||||
mapping = remote_to_local.get(str(result.get("knowledge_base_id") or ""))
|
||||
if mapping is None or not str(result.get("content") or "").strip():
|
||||
continue
|
||||
sources.append(_source_from_result(result, mapping, len(sources) + 1))
|
||||
if len(sources) >= _MAX_SOURCES:
|
||||
break
|
||||
return LlmWikiRagContext(selected_ids, len(sources), _build_prompt(query, sources), sources, query)
|
||||
|
||||
|
||||
def attach_llmwiki_rag_to_graph_input(
|
||||
graph_input: dict[str, Any],
|
||||
rag: LlmWikiRagContext,
|
||||
) -> dict[str, Any]:
|
||||
if not rag.prompt and not rag.sources:
|
||||
return graph_input
|
||||
messages = graph_input.get("messages")
|
||||
if not isinstance(messages, list) or not messages:
|
||||
return graph_input
|
||||
new_messages = list(messages)
|
||||
for index, message in enumerate(new_messages):
|
||||
if not isinstance(message, HumanMessage):
|
||||
continue
|
||||
extra = dict(getattr(message, "additional_kwargs", None) or {})
|
||||
existing = str(extra.get("prompt_prefix") or "").strip()
|
||||
extra["prompt_prefix"] = f"{existing}\n\n{rag.prompt}".strip() if existing else rag.prompt
|
||||
extra["llmwiki_knowledge_base_ids"] = rag.knowledge_base_ids
|
||||
extra["llmwiki_reference_query"] = rag.query
|
||||
if rag.sources:
|
||||
extra["llmwiki_reference_sources"] = rag.sources
|
||||
new_messages[index] = message.model_copy(update={"additional_kwargs": extra})
|
||||
break
|
||||
return {**graph_input, "messages": new_messages}
|
||||
@@ -64,10 +64,15 @@ class AgentResponse(BaseModel):
|
||||
description="入库模板名 (external knowledge-ingest templateName). When set, markdown artifacts produced in this agent's chats can be published to the external knowledge-ingest API.",
|
||||
)
|
||||
|
||||
seat_skill_directive: bool | None = Field(
|
||||
seat_skill_directive: bool | None = Field(
|
||||
default=None,
|
||||
description="会商席位「技能识别强化」开关 (config.yaml). True = 本智能体作为会商席位时强化技能识别 (与业务链条同名开关取 OR). None/False = 关 (默认).",
|
||||
)
|
||||
)
|
||||
|
||||
llmwiki_knowledge_base_ids: list[str] | None = Field(
|
||||
default=None,
|
||||
description="DeerFlow LLMWiki mapping ids this agent may offer for conversation retrieval.",
|
||||
)
|
||||
|
||||
tags: list[dict] = Field(default_factory=list, description="Tags assigned to this agent")
|
||||
|
||||
@@ -120,7 +125,12 @@ class AgentCreateRequest(BaseModel):
|
||||
tool_groups: list[str] | None = Field(default=None, description="Optional tool group whitelist persisted to config.yaml")
|
||||
common_questions: list[CommonQuestion] | None = Field(default=None, description="Recommended questions (常用问题) persisted to config.yaml. Each item is {title, prompt}; legacy strings are coerced.")
|
||||
template_name: str | None = Field(default=None, description="入库模板名 persisted to config.yaml. Enables publishing markdown artifacts to the external knowledge-ingest API.")
|
||||
seat_skill_directive: bool | None = Field(default=None, description="会商席位「技能识别强化」开关 persisted to config.yaml. None/False = off (默认).")
|
||||
seat_skill_directive: bool | None = Field(default=None, description="会商席位「技能识别强化」开关 persisted to config.yaml. None/False = off (默认).")
|
||||
llmwiki_knowledge_base_ids: list[str] | None = Field(
|
||||
default=None,
|
||||
max_length=100,
|
||||
description="Allowed DeerFlow LLMWiki mapping ids persisted to config.yaml.",
|
||||
)
|
||||
|
||||
|
||||
class AgentUpdateRequest(BaseModel):
|
||||
@@ -137,7 +147,12 @@ class AgentUpdateRequest(BaseModel):
|
||||
tool_groups: list[str] | None = Field(default=None, description="Updated tool group whitelist (config.yaml)")
|
||||
common_questions: list[CommonQuestion] | None = Field(default=None, description="Updated recommended questions (常用问题, config.yaml). Each item is {title, prompt}; legacy strings are coerced.")
|
||||
template_name: str | None = Field(default=None, description="Updated 入库模板名 (config.yaml). Empty string clears it.")
|
||||
seat_skill_directive: bool | None = Field(default=None, description="Updated 会商席位「技能识别强化」开关 (config.yaml).")
|
||||
seat_skill_directive: bool | None = Field(default=None, description="Updated 会商席位「技能识别强化」开关 (config.yaml).")
|
||||
llmwiki_knowledge_base_ids: list[str] | None = Field(
|
||||
default=None,
|
||||
max_length=100,
|
||||
description="Updated allowed DeerFlow LLMWiki mapping ids (config.yaml).",
|
||||
)
|
||||
|
||||
|
||||
class AgentSelectorResponse(BaseModel):
|
||||
@@ -178,10 +193,38 @@ def _current_user_id(request: Request) -> str:
|
||||
return get_effective_user_id()
|
||||
|
||||
|
||||
def _is_admin_user(request: Request) -> bool:
|
||||
def _is_admin_user(request: Request) -> bool:
|
||||
"""Return True when the request is authenticated as a system admin."""
|
||||
user = getattr(request.state, "user", None)
|
||||
return getattr(user, "system_role", None) == "admin"
|
||||
return getattr(user, "system_role", None) == "admin"
|
||||
|
||||
|
||||
async def _validate_llmwiki_bindings(
|
||||
request: Request,
|
||||
mapping_ids: list[str] | None,
|
||||
) -> list[str] | None:
|
||||
"""Normalize bindings and require read access for the editing actor."""
|
||||
|
||||
if mapping_ids is None:
|
||||
return None
|
||||
normalized = list(dict.fromkeys(item.strip() for item in mapping_ids if item.strip()))
|
||||
if not normalized:
|
||||
return []
|
||||
store = getattr(request.app.state, "llmwiki_store", None)
|
||||
if store is None:
|
||||
raise HTTPException(status_code=503, detail="LLMWiki metadata store is unavailable")
|
||||
user_id = _current_user_id(request)
|
||||
is_admin = _is_admin_user(request)
|
||||
for mapping_id in normalized:
|
||||
row = await store.get_authorized(
|
||||
mapping_id,
|
||||
user_id,
|
||||
write=False,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
if row is None:
|
||||
raise HTTPException(status_code=422, detail=f"LLMWiki knowledge base '{mapping_id}' is unavailable")
|
||||
return normalized
|
||||
|
||||
|
||||
def _email_local_part(email: str | None) -> str | None:
|
||||
@@ -233,7 +276,7 @@ def _validate_storage_id(agent_id: str) -> str:
|
||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||
|
||||
|
||||
def _write_agent_config(
|
||||
def _write_agent_config(
|
||||
agent_id: str,
|
||||
*,
|
||||
name: str,
|
||||
@@ -243,8 +286,9 @@ def _write_agent_config(
|
||||
tool_groups: list[str] | None = None,
|
||||
common_questions: list[CommonQuestion] | None = None,
|
||||
template_name: str | None = None,
|
||||
seat_skill_directive: bool | None = None,
|
||||
) -> None:
|
||||
seat_skill_directive: bool | None = None,
|
||||
llmwiki_knowledge_base_ids: list[str] | None = None,
|
||||
) -> None:
|
||||
agent_dir = get_paths().agent_dir(agent_id)
|
||||
agent_dir.mkdir(parents=True, exist_ok=True)
|
||||
config_file = agent_dir / "config.yaml"
|
||||
@@ -273,15 +317,19 @@ def _write_agent_config(
|
||||
if template_name is not None and template_name.strip():
|
||||
config_data["template_name"] = template_name.strip()
|
||||
# 会商席位技能识别强化开关:仅当显式设置时写入(None = 不写, 由 AgentConfig 默认为关)。
|
||||
if seat_skill_directive is not None:
|
||||
config_data["seat_skill_directive"] = bool(seat_skill_directive)
|
||||
if seat_skill_directive is not None:
|
||||
config_data["seat_skill_directive"] = bool(seat_skill_directive)
|
||||
if llmwiki_knowledge_base_ids is not None:
|
||||
config_data["llmwiki_knowledge_base_ids"] = list(
|
||||
dict.fromkeys(item.strip() for item in llmwiki_knowledge_base_ids if item.strip())
|
||||
)
|
||||
with open(config_file, "w", encoding="utf-8") as f:
|
||||
yaml.dump(config_data, f, default_flow_style=False, allow_unicode=True)
|
||||
|
||||
|
||||
def _load_config_extras(agent_id: str) -> dict[str, Any]:
|
||||
"""Read skills/model/tool_groups/common_questions from the agent's config.yaml, tolerant of missing files."""
|
||||
empty = {"skills": None, "model": None, "tool_groups": None, "common_questions": None, "template_name": None, "seat_skill_directive": None}
|
||||
empty = {"skills": None, "model": None, "tool_groups": None, "common_questions": None, "template_name": None, "seat_skill_directive": None, "llmwiki_knowledge_base_ids": None}
|
||||
try:
|
||||
cfg = load_agent_config(agent_id)
|
||||
except (FileNotFoundError, ValueError):
|
||||
@@ -294,7 +342,8 @@ def _load_config_extras(agent_id: str) -> dict[str, Any]:
|
||||
"tool_groups": cfg.tool_groups,
|
||||
"common_questions": cfg.common_questions,
|
||||
"template_name": cfg.template_name,
|
||||
"seat_skill_directive": cfg.seat_skill_directive,
|
||||
"seat_skill_directive": cfg.seat_skill_directive,
|
||||
"llmwiki_knowledge_base_ids": cfg.llmwiki_knowledge_base_ids,
|
||||
}
|
||||
|
||||
|
||||
@@ -455,7 +504,7 @@ def _record_to_response(
|
||||
can happen lazily when the agent is actually opened.
|
||||
"""
|
||||
soul: str | None = None
|
||||
extras: dict[str, Any] = {"skills": None, "model": None, "tool_groups": None, "common_questions": None, "template_name": None, "seat_skill_directive": None}
|
||||
extras: dict[str, Any] = {"skills": None, "model": None, "tool_groups": None, "common_questions": None, "template_name": None, "seat_skill_directive": None, "llmwiki_knowledge_base_ids": None}
|
||||
if include_extras:
|
||||
_ensure_agent_dir_from_record(record)
|
||||
extras = _load_config_extras(record["id"])
|
||||
@@ -479,7 +528,8 @@ def _record_to_response(
|
||||
skills=extras["skills"],
|
||||
common_questions=extras.get("common_questions"),
|
||||
template_name=extras.get("template_name"),
|
||||
seat_skill_directive=extras.get("seat_skill_directive"),
|
||||
seat_skill_directive=extras.get("seat_skill_directive"),
|
||||
llmwiki_knowledge_base_ids=extras.get("llmwiki_knowledge_base_ids"),
|
||||
featured_order=record.get("featured_order"),
|
||||
)
|
||||
|
||||
@@ -780,7 +830,7 @@ async def get_agent(agent_id: str, request: Request) -> AgentResponse:
|
||||
summary="Create Agent",
|
||||
description="Create a new user-owned agent. The display name can be Chinese and may duplicate existing names.",
|
||||
)
|
||||
async def create_agent_endpoint(request: AgentCreateRequest, http_request: Request) -> AgentResponse:
|
||||
async def create_agent_endpoint(request: AgentCreateRequest, http_request: Request) -> AgentResponse:
|
||||
_require_agents_api_enabled()
|
||||
name = _validate_display_name(request.name)
|
||||
store = get_agent_store(http_request)
|
||||
@@ -789,9 +839,13 @@ async def create_agent_endpoint(request: AgentCreateRequest, http_request: Reque
|
||||
agent_id = _validate_storage_id(request.id)
|
||||
if await store.get_any(agent_id) is not None:
|
||||
raise HTTPException(status_code=409, detail=f"Agent id '{agent_id}' already exists")
|
||||
else:
|
||||
agent_id = uuid4().hex
|
||||
agent_dir = get_paths().agent_dir(agent_id)
|
||||
else:
|
||||
agent_id = uuid4().hex
|
||||
agent_dir = get_paths().agent_dir(agent_id)
|
||||
llmwiki_ids = await _validate_llmwiki_bindings(
|
||||
http_request,
|
||||
request.llmwiki_knowledge_base_ids,
|
||||
)
|
||||
|
||||
try:
|
||||
if agent_dir.exists():
|
||||
@@ -806,7 +860,8 @@ async def create_agent_endpoint(request: AgentCreateRequest, http_request: Reque
|
||||
tool_groups=request.tool_groups,
|
||||
common_questions=request.common_questions,
|
||||
template_name=request.template_name,
|
||||
seat_skill_directive=request.seat_skill_directive,
|
||||
seat_skill_directive=request.seat_skill_directive,
|
||||
llmwiki_knowledge_base_ids=llmwiki_ids,
|
||||
)
|
||||
_write_agent_soul(agent_id, request.soul)
|
||||
|
||||
@@ -887,9 +942,14 @@ async def update_agent(agent_id: str, request: AgentUpdateRequest, http_request:
|
||||
next_tool_groups = request.tool_groups if "tool_groups" in fields_set else existing_extras["tool_groups"]
|
||||
next_common_questions = request.common_questions if "common_questions" in fields_set else existing_extras["common_questions"]
|
||||
next_template_name = request.template_name if "template_name" in fields_set else existing_extras["template_name"]
|
||||
next_seat_skill_directive = request.seat_skill_directive if "seat_skill_directive" in fields_set else existing_extras["seat_skill_directive"]
|
||||
|
||||
config_dirty = bool(update_data) or any(key in fields_set for key in ("skills", "model", "tool_groups", "common_questions", "template_name", "seat_skill_directive"))
|
||||
next_seat_skill_directive = request.seat_skill_directive if "seat_skill_directive" in fields_set else existing_extras["seat_skill_directive"]
|
||||
next_llmwiki_ids = (
|
||||
await _validate_llmwiki_bindings(http_request, request.llmwiki_knowledge_base_ids)
|
||||
if "llmwiki_knowledge_base_ids" in fields_set
|
||||
else existing_extras["llmwiki_knowledge_base_ids"]
|
||||
)
|
||||
|
||||
config_dirty = bool(update_data) or any(key in fields_set for key in ("skills", "model", "tool_groups", "common_questions", "template_name", "seat_skill_directive", "llmwiki_knowledge_base_ids"))
|
||||
if config_dirty:
|
||||
_write_agent_config(
|
||||
agent_id,
|
||||
@@ -900,7 +960,8 @@ async def update_agent(agent_id: str, request: AgentUpdateRequest, http_request:
|
||||
tool_groups=next_tool_groups,
|
||||
common_questions=next_common_questions,
|
||||
template_name=next_template_name,
|
||||
seat_skill_directive=next_seat_skill_directive,
|
||||
seat_skill_directive=next_seat_skill_directive,
|
||||
llmwiki_knowledge_base_ids=next_llmwiki_ids,
|
||||
)
|
||||
# Keep the listing cache in lock-step with config.yaml, but only
|
||||
# when the cached fields were actually touched — a partial update
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -12,11 +12,12 @@ matching the LangGraph Platform wire format expected by the
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
@@ -26,6 +27,7 @@ from sqlalchemy import select
|
||||
|
||||
from app.gateway.authz import require_permission
|
||||
from app.gateway.deps import get_checkpointer
|
||||
from app.gateway.llmwiki_deposit import delete_conversation_deposit_documents
|
||||
from app.gateway.utils import sanitize_log_param
|
||||
from deerflow.config.extensions_config import SkillDisplayConfig
|
||||
from deerflow.config.paths import Paths, get_paths
|
||||
@@ -34,10 +36,12 @@ from deerflow.persistence.skills.model import SkillDisplayAdapterRow
|
||||
from deerflow.runtime import serialize_channel_values
|
||||
from deerflow.runtime.references import (
|
||||
adapt_reference_batches_with_displays,
|
||||
build_reference_batches as build_runtime_reference_batches,
|
||||
build_reference_debug_payload,
|
||||
build_reference_debug_summary,
|
||||
)
|
||||
from deerflow.runtime.references import (
|
||||
build_reference_batches as build_runtime_reference_batches,
|
||||
)
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
from deerflow.utils.time import coerce_iso, now_iso
|
||||
|
||||
@@ -1250,6 +1254,13 @@ async def clear_thread_messages(thread_id: str, request: Request) -> dict[str, A
|
||||
len(messages),
|
||||
sanitize_log_param(thread_id),
|
||||
)
|
||||
asyncio.create_task(
|
||||
delete_conversation_deposit_documents(
|
||||
request.app,
|
||||
thread_id=thread_id,
|
||||
message_ids=None,
|
||||
)
|
||||
)
|
||||
return {
|
||||
"ok": True,
|
||||
"deleted_ids": deleted_ids,
|
||||
@@ -1345,6 +1356,13 @@ async def delete_thread_message(thread_id: str, message_id: str, request: Reques
|
||||
len(messages) - len(filtered),
|
||||
sanitize_log_param(thread_id),
|
||||
)
|
||||
asyncio.create_task(
|
||||
delete_conversation_deposit_documents(
|
||||
request.app,
|
||||
thread_id=thread_id,
|
||||
message_ids=deleted_ids,
|
||||
)
|
||||
)
|
||||
return {"ok": True, "deleted_ids": deleted_ids, "remaining": len(filtered)}
|
||||
|
||||
|
||||
|
||||
@@ -21,6 +21,11 @@ from fastapi import HTTPException, Request
|
||||
from langchain_core.messages import HumanMessage
|
||||
|
||||
from app.gateway.deps import get_concurrency_gate, get_run_context, get_run_manager, get_stream_bridge
|
||||
from app.gateway.llmwiki_rag import (
|
||||
attach_llmwiki_rag_to_graph_input,
|
||||
build_llmwiki_rag_context,
|
||||
llmwiki_selection_requested,
|
||||
)
|
||||
from app.gateway.utils import sanitize_log_param
|
||||
from deerflow.config import get_app_config
|
||||
from deerflow.config.agents_config import load_agent_config, validate_agent_id
|
||||
@@ -275,6 +280,8 @@ def _prompt_prefix_opted_in(body: Any) -> bool:
|
||||
|
||||
|
||||
def _rag_mode_requested(body: Any) -> bool:
|
||||
if llmwiki_selection_requested(body):
|
||||
return True
|
||||
context = getattr(body, "context", None)
|
||||
if isinstance(context, Mapping) and context.get("rag_mode_enabled"):
|
||||
return True
|
||||
@@ -325,6 +332,7 @@ _CONTEXT_CONFIGURABLE_KEYS: frozenset[str] = frozenset(
|
||||
"thinking_force_disabled",
|
||||
"reasoning_effort",
|
||||
"rag_mode_enabled",
|
||||
"llmwiki_knowledge_base_ids",
|
||||
"writing_mode",
|
||||
"writing_artifact_path",
|
||||
"canvas_artifact_title",
|
||||
@@ -1003,6 +1011,27 @@ async def start_run(
|
||||
prefix_settings = load_system_settings().prompt_prefix
|
||||
if prefix_settings.enabled and prefix_settings.prompt_prefix.strip():
|
||||
graph_input = _prepend_prompt_prefix(graph_input, prefix_settings.prompt_prefix.strip())
|
||||
llmwiki_rag = await build_llmwiki_rag_context(
|
||||
request=request,
|
||||
body=body,
|
||||
graph_input=graph_input,
|
||||
)
|
||||
if llmwiki_rag.prompt or llmwiki_rag.sources:
|
||||
graph_input = attach_llmwiki_rag_to_graph_input(graph_input, llmwiki_rag)
|
||||
effective_rag_mode = True
|
||||
run_metadata["llmwiki_knowledge_base_ids"] = llmwiki_rag.knowledge_base_ids
|
||||
run_metadata["llmwiki_rag_source_count"] = llmwiki_rag.source_count
|
||||
try:
|
||||
await run_ctx.thread_store.update_metadata(
|
||||
thread_id,
|
||||
{
|
||||
"llmwiki_knowledge_base_ids": llmwiki_rag.knowledge_base_ids,
|
||||
"rag_mode_enabled": True,
|
||||
"rag_mode": True,
|
||||
},
|
||||
)
|
||||
except Exception:
|
||||
logger.debug("Failed to persist LLMWiki RAG metadata for %s", sanitize_log_param(thread_id), exc_info=True)
|
||||
if effective_rag_mode:
|
||||
graph_input = _append_invisible_prompt(graph_input, _RAG_CITATION_PROMPT)
|
||||
config = build_run_config(thread_id, body.config, body.metadata, assistant_id=body.assistant_id)
|
||||
@@ -1014,6 +1043,8 @@ async def start_run(
|
||||
body_context = dict(getattr(body, "context", None) or {})
|
||||
if effective_rag_mode:
|
||||
body_context["rag_mode_enabled"] = True
|
||||
if "llmwiki_knowledge_base_ids" not in body_context and llmwiki_rag.knowledge_base_ids:
|
||||
body_context["llmwiki_knowledge_base_ids"] = llmwiki_rag.knowledge_base_ids
|
||||
# Promote whitelisted top-level body fields into the context map so
|
||||
# callers can pass them either nested under ``context`` or as flat
|
||||
# top-level fields (``excluded_tools``, ``skill_stop_names``, …).
|
||||
|
||||
@@ -258,8 +258,20 @@ tool_groups:
|
||||
tools:
|
||||
- name: web_search
|
||||
group: web
|
||||
use: deerflow.community.ddg_search.tools:web_search_tool
|
||||
use: deerflow.community.configurable_search.tools:web_search_tool
|
||||
# Keep disabled until the deployment-specific endpoint is filled in.
|
||||
# When false, web_search is not registered with the agent at all.
|
||||
enabled: false
|
||||
endpoint: "https://xxx.xxx.xxx/searchName"
|
||||
verify_ssl: false
|
||||
timeout: 30
|
||||
max_results: 5
|
||||
result_url_template: "https://xxx.xxx.xxx/mindId/xxx?recUuid={recUuid}"
|
||||
# The runtime query always overwrites payload.query; all other fields are
|
||||
# sent unchanged so deployment-specific request parameters stay in config.
|
||||
payload:
|
||||
query: ""
|
||||
xxx: "xxxx"
|
||||
|
||||
- name: web_fetch
|
||||
group: web
|
||||
@@ -478,6 +490,21 @@ ai_writing:
|
||||
writer: false # 作家(逐章正文写作,逐章流式与严格模式求助逻辑不变)
|
||||
editor: false # 编辑(事实/逻辑/语言三维度审查)
|
||||
|
||||
# Optional independently deployed WeKnora integration. DeerFlow owns only
|
||||
# these connection values; database/Redis/storage/parser/model settings stay
|
||||
# in WeKnora's own docker-compose/.env/config files. A non-empty api_base_url
|
||||
# enables WeKnora mode; leaving it empty preserves the existing LLMWiki iframe.
|
||||
# admin_email/admin_password are used only by DeerFlow's server-side WeKnora
|
||||
# detail-page iframe proxy. Prefer environment variables in production, e.g.
|
||||
# admin_email: "$WEKNORA_ADMIN_EMAIL"
|
||||
# admin_password: "$WEKNORA_ADMIN_PASSWORD"
|
||||
llmwiki:
|
||||
weknora:
|
||||
api_base_url: "http://119.254.86.251:8088/deerflow-api"
|
||||
web_base_url: "http://119.254.86.251:8088"
|
||||
admin_email: "lqq@qq.com"
|
||||
admin_password: "$WEKNORA_ADMIN_PASSWORD"
|
||||
|
||||
log_system:
|
||||
enabled: true
|
||||
url: 'https://your-log-system.example.com'
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
# LLMWiki × WeKnora 集成与部署
|
||||
|
||||
## 目标与边界
|
||||
|
||||
DeerFlow 只把 WeKnora 当作独立部署的知识抽取、向量化和检索服务,不复制或修改 WeKnora 前端,也不接管它的运行配置。
|
||||
|
||||
本版本按 WeKnora `v0.7.2` 的官方 REST API 实现;升级 WeKnora 大版本前应先回归下文列出的主要接口。
|
||||
|
||||
- WeKnora 自己负责 PostgreSQL、Redis、对象存储、解析器、向量库、Embedding/Rerank/LLM 等依赖。
|
||||
- DeerFlow 只保存 WeKnora 连接地址,以及 DeerFlow 用户、发布状态、智能体绑定与 WeKnora 知识库之间的映射。
|
||||
- DeerFlow 不保存也不发送 WeKnora API Key、Bearer Token 或其他 WeKnora 凭证;浏览器拿不到原始 WeKnora 知识库 ID。
|
||||
- 用户权限、智能体所有权和页面风格全部沿用 DeerFlow;库所有者发布后立即进入公共 LLMWiki,不经过审核队列。
|
||||
|
||||
## 模式开关
|
||||
|
||||
`config.yaml` 中 `llmwiki.weknora.api_base_url` 非空时启用 WeKnora 模式;为空时继续使用原有 LLMWiki 页面和流程。
|
||||
|
||||
```yaml
|
||||
llmwiki:
|
||||
weknora:
|
||||
api_base_url: "http://weknora-app:8080"
|
||||
web_base_url: "http://weknora-app:8080"
|
||||
```
|
||||
|
||||
管理员也可在 DeerFlow 原 LLMWiki 页面中的“集成配置”修改地址。页面配置是运行时覆盖项:启用覆盖并填写地址会切换到 WeKnora;启用覆盖但留空会强制回到原模式;“恢复配置文件设置”会重新采用 `config.yaml`。
|
||||
|
||||
这两个地址是 DeerFlow 中全部的 WeKnora 配置,不需要创建密钥文件,也不需要在 DeerFlow 中配置 WeKnora 的数据库、模型或存储参数。
|
||||
|
||||
## WeKnora 侧准备
|
||||
|
||||
先按 WeKnora 官方部署方式完整启动 WeKnora。数据库、Redis、MinIO/S3、解析服务和所有模型均在 WeKnora 自己的 compose、`.env` 或管理页面配置,DeerFlow 无对应字段。
|
||||
|
||||
DeerFlow 对 WeKnora 使用无凭证的内部 HTTP 请求,因此 `api_base_url` 必须指向可信内网服务,不能把匿名接口直接暴露到公网。需要在 WeKnora 自身或前置反向代理中,仅按容器网络、源 IP、安全组或 mTLS 等基础设施边界放行 DeerFlow。若 WeKnora 保持默认接口鉴权且未放行,连接测试会收到 401/403,并提示配置 WeKnora 或反向代理;DeerFlow 不提供凭证回退。
|
||||
|
||||
WeKnora 中至少需要一个可用且标为默认的 Embedding 模型。创建知识库时,DeerFlow 调用 WeKnora `/api/v1/models` 查找 `type=Embedding` 的默认模型,并把它的 ID 传给 WeKnora;因此 DeerFlow 不提供 `default_embedding_model_id` 配置。这个 ID 的含义是“新知识库使用哪个编码模型生成向量”,其生命周期属于 WeKnora。
|
||||
|
||||
WeKnora 侧放行范围需要覆盖本版本调用的模型读取、知识库管理、文档写入和知识检索接口。如果基础设施只放行检索接口,可以检索已有知识库,但无法通过 DeerFlow 创建知识库或上传文档。
|
||||
|
||||
## 数据与权限模型
|
||||
|
||||
DeerFlow 的 `llmwiki_knowledge_bases` 表保存以下控制面数据:DeerFlow 映射 ID、WeKnora ID、所有者用户 ID、名称快照和公开状态。文档、切片、向量和模型参数仍只存在 WeKnora。
|
||||
|
||||
- 个人库:仅所有者和 DeerFlow 管理员可管理。
|
||||
- 公共库:所有者点击发布后立即公开,全部 DeerFlow 用户只读可检索;所有者可随时取消公开。
|
||||
- 删除个人库时先删除 WeKnora 实体,再删除 DeerFlow 映射。
|
||||
- 所有 BFF 接口和智能体工具都会按当前 DeerFlow 登录用户重新鉴权,不能靠前端提交任意知识库 ID 越权。
|
||||
|
||||
当前无凭证/反向代理方案使用一个固定的 WeKnora 服务身份。因此 DeerFlow 的用户隔离是映射层的逻辑隔离:不同 DeerFlow 用户只能看到自己拥有的库和已公开库;但这些库在 WeKnora 物理上都创建于该服务身份所属的同一个空间。上传文件不会落到 DeerFlow 用户目录,而是经 DeerFlow BFF 转发给 WeKnora,由 WeKnora 把文件对象、知识元数据、切片和向量分别写入自己的对象存储、数据库和向量存储。需要在 WeKnora 后台核对时,应登录这个服务身份所属空间,在“知识库管理”中按同名知识库进入查看。
|
||||
|
||||
如果要求 WeKnora 本身也按 DeerFlow 用户划分空间,则需要另一种部署:为每个 DeerFlow 用户建立 WeKnora 用户/租户并安全保存、轮换各自凭证。它与当前“不在 DeerFlow 配置 WeKnora 凭证”的单服务身份方案不同,不能只靠前端切换实现。
|
||||
|
||||
## 智能体和对话选择
|
||||
|
||||
智能体 `config.yaml` 可保存:
|
||||
|
||||
```yaml
|
||||
llmwiki_knowledge_base_ids:
|
||||
- "deerflow-mapping-id-1"
|
||||
- "deerflow-mapping-id-2"
|
||||
```
|
||||
|
||||
这里必须是 DeerFlow 映射 ID,不是 WeKnora 原始 ID。智能体创建页和详情编辑页均提供内嵌的“绑定知识空间”面板,可直接搜索、分组、多选、全选或清空当前用户有权读取的个人库和公共库,不再通过额外弹窗配置。
|
||||
|
||||
新建智能体对话时,页面只展示“智能体已绑定”且“当前登录用户仍有权读取”的交集,默认全选,可在发送第一条消息前调整或清空。普通新建对话也可选择知识库;选中了智能体时会自动切换成该智能体的绑定范围。问答选择器仿照 WeKnora 的交互结构,作为输入框工具栏的锚定下拉呈现,包含搜索、个人/公共分组、文档数量、多选、全选和清空,不再在输入框外单独占据一行。选择结果写入 LangGraph thread metadata 和运行 context,重新打开对话会恢复,开始对话后不可中途扩大权限范围。
|
||||
|
||||
`llmwiki_search` 是只读工具,只在 WeKnora 模式注册。每次调用会同时校验:
|
||||
|
||||
1. 当前对话选择;
|
||||
2. 智能体允许的绑定范围;
|
||||
3. 当前 DeerFlow 用户对映射的实时读取权限。
|
||||
|
||||
明确选择空列表表示本次对话不使用 LLMWiki;旧对话没有选择字段时,智能体对话默认使用其绑定范围,通用 Lead Agent 默认使用当前用户全部可见知识库。
|
||||
|
||||
## 前端实现
|
||||
|
||||
WeKnora 前端不做任何修改。DeerFlow 原“知识库资源管理”入口根据运行模式渲染:
|
||||
|
||||
- 原模式:保留现有 LLMWiki iframe 页面;
|
||||
- WeKnora 模式:渲染 DeerFlow 原生 React 页面,提供个人库、公共库、直接发布和检索。点击卡片进入独立知识库路由,内容区复刻 WeKnora 的详情页结构,同时保留 DeerFlow 的导航、登录和权限壳。Wiki 库按 **Wiki → 文档 → 图谱** 排列并默认打开 Wiki;标准库不显示 Wiki 标签,按 **文档 → 图谱** 排列。Wiki 视图包含页面目录、搜索、页面正文和引用摘要;文档以卡片网格展示,点击后从右侧抽屉查看元数据、原文件与真实解析分块;图谱视图渲染 WeKnora Wiki 页面节点和边。新建知识库可选择“Wiki 知识库”或“标准知识库”,详情页支持文件、网页 URL、手工 Markdown 三种导入方式。问答选择器沿用 WeKnora 的紧凑搜索/分组/多选交互,颜色、间距和组件沿用 DeerFlow 设计系统;智能体创建和编辑页使用同一数据源的内嵌绑定面板。
|
||||
|
||||
问答检索命中以 DeerFlow 来源卡片展示。“查看原文”调用 DeerFlow 的 `/api/llmwiki/sources/context`,由服务端在重新校验 DeerFlow 用户权限后向 WeKnora 读取命中切片及相邻上下文,不向浏览器暴露 WeKnora 原始资源 ID。
|
||||
|
||||
## 主要接口
|
||||
|
||||
- `GET /api/llmwiki/runtime`:当前原模式/WeKnora 模式。
|
||||
- `/api/system-settings/llmwiki`:管理员地址配置、重置和连通性测试。
|
||||
- `/api/llmwiki/knowledge-bases`:知识库 CRUD、标准/Wiki 模式创建、详情、文档上传、解析、直接发布和取消公开。
|
||||
- `POST /api/llmwiki/knowledge-bases/{id}/documents/url`:导入网页 URL。
|
||||
- `POST /api/llmwiki/knowledge-bases/{id}/documents/manual`:发布手工 Markdown 内容。
|
||||
- `GET /api/llmwiki/knowledge-bases/{id}/documents/{document_id}`:具体文档元数据。
|
||||
- `GET /api/llmwiki/knowledge-bases/{id}/documents/{document_id}/preview`:鉴权后代理原文件预览。
|
||||
- `GET /api/llmwiki/knowledge-bases/{id}/documents/{document_id}/chunks`:分页读取解析后的真实分块索引。
|
||||
- `GET /api/llmwiki/knowledge-bases/{id}/graph`:读取 Wiki 页面图谱的节点和边;未开启 Wiki 索引时返回明确的空状态。
|
||||
- `GET /api/llmwiki/knowledge-bases/{id}/wiki/pages` 与 `.../wiki/pages/{slug}`:读取 Wiki 页面目录和正文。
|
||||
- `GET /api/llmwiki/agents/{agent_id}/knowledge-bases`:当前用户可用于该智能体的绑定交集。
|
||||
- `POST /api/llmwiki/search`:页面检索。
|
||||
- `GET /api/llmwiki/sources/context`:引用原文上下文。
|
||||
|
||||
## 验收建议
|
||||
|
||||
1. 地址留空启动,确认原 LLMWiki 页面和原有流程不变。
|
||||
2. 在可信网络中放行 DeerFlow,配置地址,测试连接并创建个人库、上传文档、等待解析完成;确认 DeerFlow 请求没有认证头。
|
||||
3. 普通用户直接发布,换另一用户确认能立即读取和检索、但不能修改;再由所有者取消公开并确认访问随即失效。
|
||||
4. 给智能体绑定两个库,新建对话取消其中一个,确认引用只来自保留的库;重新打开对话确认选择恢复。
|
||||
5. 撤销公共发布或删除用户权限后继续提问,确认工具实时拒绝已失效的知识库。
|
||||
@@ -1,24 +1,5 @@
|
||||
{
|
||||
"mcpServers": {
|
||||
"weknora-retrieval": {
|
||||
"enabled": true,
|
||||
"type": "stdio",
|
||||
"command": "uv",
|
||||
"args": [
|
||||
"run",
|
||||
"--no-sync",
|
||||
"python",
|
||||
"skills/custom/weknora-knowledge-search/scripts/weknora_retrieval_mcp.py"
|
||||
],
|
||||
"env": {
|
||||
"WEKNORA_BASE_URL": "http://[::1]:8081",
|
||||
"WEKNORA_DEERFLOW_SHARE_TOKEN_FILE": "F:\\react01\\deeflow-code\\deerflow-server\\offline-backend-20260512\\backend\\.runtime\\secrets\\weknora_deerflow_share_token",
|
||||
"WEKNORA_ALLOWED_KB_IDS": "",
|
||||
"WEKNORA_REQUEST_TIMEOUT_SECONDS": "30",
|
||||
"WEKNORA_RERANK_URL": "http://47.88.25.99:7022"
|
||||
}
|
||||
}
|
||||
},
|
||||
"mcpServers": {},
|
||||
"skills": {
|
||||
"web-research": {
|
||||
"enabled": true
|
||||
@@ -49,9 +30,6 @@
|
||||
},
|
||||
"browser_act": {
|
||||
"enabled": true
|
||||
},
|
||||
"weknora-knowledge-search": {
|
||||
"enabled": true
|
||||
}
|
||||
}
|
||||
}
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
from .tools import web_search_tool
|
||||
|
||||
__all__ = ["web_search_tool"]
|
||||
+202
@@ -0,0 +1,202 @@
|
||||
"""Configurable JSON-over-HTTP implementation of the ``web_search`` tool."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from copy import deepcopy
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
from urllib.parse import quote
|
||||
|
||||
import httpx
|
||||
from langchain.tools import tool
|
||||
|
||||
from deerflow.config import get_app_config
|
||||
from deerflow.config.tool_config import ToolConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ConfigurableSearchSettings:
|
||||
"""Validated runtime settings read from the ``web_search`` tool entry."""
|
||||
|
||||
enabled: bool
|
||||
endpoint: str
|
||||
payload: dict[str, Any]
|
||||
verify_ssl: bool
|
||||
timeout: float
|
||||
max_results: int
|
||||
result_url_template: str
|
||||
|
||||
@classmethod
|
||||
def from_tool_config(cls, config: ToolConfig | None) -> ConfigurableSearchSettings:
|
||||
if config is None:
|
||||
raise ValueError("web_search is not configured")
|
||||
|
||||
extra = config.model_extra or {}
|
||||
endpoint = str(extra.get("endpoint") or "").strip()
|
||||
if not endpoint.startswith(("http://", "https://")):
|
||||
raise ValueError("web_search.endpoint must use http:// or https://")
|
||||
|
||||
raw_payload = extra.get("payload", {})
|
||||
if not isinstance(raw_payload, dict):
|
||||
raise ValueError("web_search.payload must be an object")
|
||||
|
||||
try:
|
||||
timeout = float(extra.get("timeout", 30))
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise ValueError("web_search.timeout must be a number") from exc
|
||||
if timeout <= 0:
|
||||
raise ValueError("web_search.timeout must be greater than zero")
|
||||
|
||||
try:
|
||||
max_results = int(extra.get("max_results", 5))
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise ValueError("web_search.max_results must be an integer") from exc
|
||||
if max_results <= 0:
|
||||
raise ValueError("web_search.max_results must be greater than zero")
|
||||
|
||||
result_url_template = str(extra.get("result_url_template") or "").strip()
|
||||
if result_url_template and "{recUuid}" not in result_url_template:
|
||||
raise ValueError("web_search.result_url_template must contain {recUuid}")
|
||||
|
||||
verify_ssl = extra.get("verify_ssl", True)
|
||||
if not isinstance(verify_ssl, bool):
|
||||
raise ValueError("web_search.verify_ssl must be true or false")
|
||||
|
||||
return cls(
|
||||
enabled=config.enabled,
|
||||
endpoint=endpoint,
|
||||
payload=deepcopy(raw_payload),
|
||||
verify_ssl=verify_ssl,
|
||||
timeout=timeout,
|
||||
max_results=max_results,
|
||||
result_url_template=result_url_template,
|
||||
)
|
||||
|
||||
|
||||
def _text(value: Any) -> str:
|
||||
"""Normalize optional scalar result fields without leaking ``None``."""
|
||||
|
||||
if value is None:
|
||||
return ""
|
||||
if isinstance(value, str):
|
||||
return value
|
||||
return str(value)
|
||||
|
||||
|
||||
def _error_result(query: str, message: str) -> str:
|
||||
return json.dumps(
|
||||
{
|
||||
"query": query,
|
||||
"total_results": 0,
|
||||
"results": [],
|
||||
"error": message,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
|
||||
|
||||
def _normalize_results(data: Any, settings: ConfigurableSearchSettings) -> list[dict[str, str]]:
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError("search API response must be a JSON object")
|
||||
|
||||
raw_results = data.get("results")
|
||||
if not isinstance(raw_results, list):
|
||||
raise ValueError("search API response field 'results' must be an array")
|
||||
|
||||
normalized: list[dict[str, str]] = []
|
||||
for item in raw_results[: settings.max_results]:
|
||||
if not isinstance(item, dict):
|
||||
logger.warning("Ignoring non-object search result: %r", item)
|
||||
continue
|
||||
|
||||
rec_uuid = _text(item.get("recUuid")).strip()
|
||||
original_url = _text(item.get("url")).strip()
|
||||
result_url = original_url
|
||||
if rec_uuid and settings.result_url_template:
|
||||
result_url = settings.result_url_template.replace(
|
||||
"{recUuid}",
|
||||
quote(rec_uuid, safe=""),
|
||||
)
|
||||
|
||||
normalized.append(
|
||||
{
|
||||
"recUuid": rec_uuid,
|
||||
"content": _text(item.get("content")),
|
||||
"title": _text(item.get("title")),
|
||||
"url": result_url,
|
||||
}
|
||||
)
|
||||
return normalized
|
||||
|
||||
|
||||
async def execute_search(
|
||||
query: str,
|
||||
settings: ConfigurableSearchSettings,
|
||||
*,
|
||||
transport: httpx.AsyncBaseTransport | None = None,
|
||||
) -> str:
|
||||
"""Execute a search request; ``transport`` exists for network-free tests."""
|
||||
|
||||
normalized_query = query.strip()
|
||||
if not normalized_query:
|
||||
return _error_result(query, "Search query must not be empty")
|
||||
if not settings.enabled:
|
||||
return _error_result(normalized_query, "web_search is disabled")
|
||||
|
||||
payload = deepcopy(settings.payload)
|
||||
payload["query"] = normalized_query
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(
|
||||
timeout=settings.timeout,
|
||||
verify=settings.verify_ssl,
|
||||
transport=transport,
|
||||
) as client:
|
||||
response = await client.post(settings.endpoint, json=payload)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
results = _normalize_results(data, settings)
|
||||
except httpx.TimeoutException:
|
||||
logger.warning("Configured web search timed out for endpoint %s", settings.endpoint)
|
||||
return _error_result(normalized_query, "Search API request timed out")
|
||||
except httpx.HTTPStatusError as exc:
|
||||
logger.warning(
|
||||
"Configured web search returned HTTP %s for endpoint %s",
|
||||
exc.response.status_code,
|
||||
settings.endpoint,
|
||||
)
|
||||
return _error_result(normalized_query, f"Search API returned HTTP {exc.response.status_code}")
|
||||
except (httpx.HTTPError, json.JSONDecodeError) as exc:
|
||||
logger.warning("Configured web search request failed: %s", exc)
|
||||
return _error_result(normalized_query, f"Search API request failed: {type(exc).__name__}")
|
||||
except ValueError as exc:
|
||||
logger.warning("Configured web search returned an invalid response: %s", exc)
|
||||
return _error_result(normalized_query, str(exc))
|
||||
|
||||
return json.dumps(
|
||||
{
|
||||
"query": normalized_query,
|
||||
"total_results": len(results),
|
||||
"results": results,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
|
||||
|
||||
@tool("web_search", parse_docstring=True)
|
||||
async def web_search_tool(query: str) -> str:
|
||||
"""Search the configured information service for current facts and source passages.
|
||||
|
||||
Args:
|
||||
query: Focused search keywords or a natural-language retrieval question.
|
||||
"""
|
||||
|
||||
try:
|
||||
settings = ConfigurableSearchSettings.from_tool_config(get_app_config().get_tool_config("web_search"))
|
||||
except ValueError as exc:
|
||||
return _error_result(query, str(exc))
|
||||
return await execute_search(query, settings)
|
||||
@@ -7,7 +7,7 @@ import threading
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
from pydantic import BaseModel, model_validator
|
||||
from pydantic import BaseModel, field_validator, model_validator
|
||||
|
||||
from deerflow.config.paths import get_paths
|
||||
|
||||
@@ -82,7 +82,7 @@ class CommonQuestion(BaseModel):
|
||||
return value
|
||||
|
||||
|
||||
class AgentConfig(BaseModel):
|
||||
class AgentConfig(BaseModel):
|
||||
"""Configuration for a custom agent."""
|
||||
|
||||
id: str | None = None
|
||||
@@ -108,7 +108,20 @@ class AgentConfig(BaseModel):
|
||||
# 拼进本轮任务,避免弱模型无视系统提示里的技能块。None/False = 关(默认);True = 开。
|
||||
# 与「业务链条配置」里的同名开关取 OR:任一打开即对该席位生效。见
|
||||
# ``app/gateway/roundtable_seat_skills.py``。
|
||||
seat_skill_directive: bool | None = None
|
||||
seat_skill_directive: bool | None = None
|
||||
# DeerFlow mapping ids this agent may offer in its new-conversation
|
||||
# knowledge selector. WeKnora ids and credentials never enter agent files.
|
||||
llmwiki_knowledge_base_ids: list[str] | None = None
|
||||
|
||||
@field_validator("llmwiki_knowledge_base_ids")
|
||||
@classmethod
|
||||
def _normalize_llmwiki_ids(cls, value: list[str] | None) -> list[str] | None:
|
||||
if value is None:
|
||||
return None
|
||||
normalized = list(dict.fromkeys(str(item).strip() for item in value if str(item).strip()))
|
||||
if len(normalized) > 100:
|
||||
raise ValueError("An agent can bind at most 100 LLMWiki knowledge bases")
|
||||
return normalized
|
||||
|
||||
|
||||
def load_agent_config(agent_id: str | None) -> AgentConfig | None:
|
||||
|
||||
@@ -17,10 +17,11 @@ from deerflow.config.database_config import DatabaseConfig
|
||||
from deerflow.config.extensions_config import ExtensionsConfig
|
||||
from deerflow.config.gateway_config import GatewayConfig
|
||||
from deerflow.config.guardrails_config import GuardrailsConfig, load_guardrails_config_from_dict
|
||||
from deerflow.config.knowledge_config import KnowledgeConfig
|
||||
from deerflow.config.knowledge_ingest_config import KnowledgeIngestConfig
|
||||
from deerflow.config.log_system_config import LogSystemConfig
|
||||
from deerflow.config.login_config import AuthLoginConfig
|
||||
from deerflow.config.knowledge_config import KnowledgeConfig
|
||||
from deerflow.config.knowledge_ingest_config import KnowledgeIngestConfig
|
||||
from deerflow.config.llmwiki_config import LlmWikiConfig
|
||||
from deerflow.config.log_system_config import LogSystemConfig
|
||||
from deerflow.config.login_config import AuthLoginConfig
|
||||
from deerflow.config.memory_config import MemoryConfig, load_memory_config_from_dict
|
||||
from deerflow.config.model_config import ModelConfig
|
||||
from deerflow.config.run_events_config import RunEventsConfig
|
||||
@@ -42,7 +43,8 @@ from deerflow.config.tool_search_config import ToolSearchConfig, load_tool_searc
|
||||
from deerflow.config.vision_config import VisionConfig
|
||||
from deerflow.config.y_log_config import YLogConfig
|
||||
|
||||
load_dotenv()
|
||||
load_dotenv()
|
||||
load_dotenv(Path(__file__).resolve().parents[4] / ".env.local", override=True)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -115,7 +117,8 @@ class AppConfig(BaseModel):
|
||||
models: list[ModelConfig] = Field(default_factory=list, description="Available models")
|
||||
vision: VisionConfig = Field(default_factory=VisionConfig, description="Image recognition (识图) configuration")
|
||||
auth_login: AuthLoginConfig = Field(default_factory=AuthLoginConfig, description="Custom login modes (username password gate, token exchange)")
|
||||
log_system: LogSystemConfig = Field(default_factory=LogSystemConfig, description="Admin-only external log-system entry shown in the settings menu")
|
||||
log_system: LogSystemConfig = Field(default_factory=LogSystemConfig, description="Admin-only external log-system entry shown in the settings menu")
|
||||
llmwiki: LlmWikiConfig = Field(default_factory=LlmWikiConfig, description="Conditional legacy/WeKnora LLMWiki integration")
|
||||
sandbox: SandboxConfig = Field(description="Sandbox configuration")
|
||||
tools: list[ToolConfig] = Field(default_factory=list, description="Available tools")
|
||||
tool_groups: list[ToolGroupConfig] = Field(default_factory=list, description="Available tool groups")
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
"""Conditional LLMWiki provider configuration.
|
||||
|
||||
DeerFlow owns only the connection to WeKnora. Database, Redis, document
|
||||
parsing, storage and model configuration stay entirely in WeKnora's own
|
||||
deployment files. An effective non-empty API URL is the sole provider switch.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from urllib.parse import urlsplit, urlunsplit
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
|
||||
|
||||
def normalize_service_url(value: str | None) -> str:
|
||||
"""Normalize an optional HTTP(S) service URL and reject embedded secrets."""
|
||||
|
||||
raw = str(value or "").strip()
|
||||
if not raw:
|
||||
return ""
|
||||
parsed = urlsplit(raw)
|
||||
if parsed.scheme not in {"http", "https"} or not parsed.hostname:
|
||||
raise ValueError("WeKnora addresses must be absolute http(s) URLs")
|
||||
if parsed.username is not None or parsed.password is not None:
|
||||
raise ValueError("WeKnora addresses must not contain credentials")
|
||||
if parsed.query or parsed.fragment:
|
||||
raise ValueError("WeKnora addresses must not contain query strings or fragments")
|
||||
path = parsed.path.rstrip("/")
|
||||
return urlunsplit((parsed.scheme, parsed.netloc, path, "", ""))
|
||||
|
||||
|
||||
class WeKnoraConnectionConfig(BaseModel):
|
||||
"""The complete DeerFlow-owned connection surface for WeKnora."""
|
||||
|
||||
api_base_url: str = Field(default="", description="WeKnora app API base URL; non-empty enables WeKnora mode")
|
||||
web_base_url: str = Field(default="", description="Optional internal WeKnora administration URL")
|
||||
admin_email: str = Field(
|
||||
default="",
|
||||
description=(
|
||||
"Optional WeKnora administrator email used only by DeerFlow's "
|
||||
"server-side detail-page iframe proxy. Prefer $ENV references."
|
||||
),
|
||||
)
|
||||
admin_password: str = Field(
|
||||
default="",
|
||||
repr=False,
|
||||
description=(
|
||||
"Optional WeKnora administrator password used only by DeerFlow's "
|
||||
"server-side detail-page iframe proxy. Prefer $ENV references."
|
||||
),
|
||||
)
|
||||
model_config = ConfigDict(extra="ignore")
|
||||
|
||||
@field_validator("api_base_url", "web_base_url")
|
||||
@classmethod
|
||||
def _validate_url(cls, value: str) -> str:
|
||||
return normalize_service_url(value)
|
||||
|
||||
|
||||
class LlmWikiConfig(BaseModel):
|
||||
"""Top-level ``config.yaml`` section for conditional WeKnora mode."""
|
||||
|
||||
weknora: WeKnoraConnectionConfig = Field(default_factory=WeKnoraConnectionConfig)
|
||||
model_config = ConfigDict(extra="ignore")
|
||||
|
||||
|
||||
class LlmWikiRuntimeOverride(BaseModel):
|
||||
"""Administrator-written URL override stored in system settings.
|
||||
|
||||
``enabled=False`` means "use config.yaml". When enabled, an empty API URL
|
||||
deliberately forces legacy mode even if the source file has an address.
|
||||
Secrets are never stored here.
|
||||
"""
|
||||
|
||||
enabled: bool = False
|
||||
api_base_url: str = ""
|
||||
web_base_url: str = ""
|
||||
model_config = ConfigDict(extra="ignore")
|
||||
|
||||
@field_validator("api_base_url", "web_base_url")
|
||||
@classmethod
|
||||
def _validate_url(cls, value: str) -> str:
|
||||
return normalize_service_url(value)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ResolvedLlmWikiRuntime:
|
||||
provider: str
|
||||
api_base_url: str
|
||||
web_base_url: str
|
||||
admin_email: str
|
||||
admin_password: str
|
||||
override_enabled: bool
|
||||
|
||||
@property
|
||||
def weknora_enabled(self) -> bool:
|
||||
return self.provider == "weknora"
|
||||
|
||||
|
||||
def resolve_llmwiki_runtime(
|
||||
config: LlmWikiConfig,
|
||||
override: LlmWikiRuntimeOverride | None = None,
|
||||
) -> ResolvedLlmWikiRuntime:
|
||||
"""Resolve the deterministic provider from file config plus URL override."""
|
||||
|
||||
use_override = bool(override and override.enabled)
|
||||
api_base_url = override.api_base_url if use_override and override else config.weknora.api_base_url
|
||||
web_base_url = override.web_base_url if use_override and override else config.weknora.web_base_url
|
||||
return ResolvedLlmWikiRuntime(
|
||||
provider="weknora" if api_base_url else "legacy",
|
||||
api_base_url=api_base_url,
|
||||
web_base_url=web_base_url,
|
||||
admin_email=config.weknora.admin_email,
|
||||
admin_password=config.weknora.admin_password,
|
||||
override_enabled=use_override,
|
||||
)
|
||||
@@ -23,6 +23,7 @@ from uuid import uuid4
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
|
||||
from deerflow.config.llmwiki_config import LlmWikiRuntimeOverride
|
||||
from deerflow.config.runtime_paths import runtime_home
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -499,6 +500,7 @@ class SystemSettings(BaseModel):
|
||||
agent_modules: AgentModuleSettings = Field(default_factory=AgentModuleSettings)
|
||||
concurrency_monitor: ConcurrencyMonitorSettings = Field(default_factory=ConcurrencyMonitorSettings)
|
||||
model_concurrency: ModelConcurrencySettings = Field(default_factory=ModelConcurrencySettings)
|
||||
llmwiki: LlmWikiRuntimeOverride = Field(default_factory=LlmWikiRuntimeOverride)
|
||||
model_config = ConfigDict(extra="ignore")
|
||||
|
||||
|
||||
|
||||
@@ -1,20 +1,21 @@
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
|
||||
class ToolGroupConfig(BaseModel):
|
||||
"""Config section for a tool group"""
|
||||
|
||||
name: str = Field(..., description="Unique name for the tool group")
|
||||
model_config = ConfigDict(extra="allow")
|
||||
|
||||
|
||||
class ToolConfig(BaseModel):
|
||||
"""Config section for a tool"""
|
||||
|
||||
name: str = Field(..., description="Unique name for the tool")
|
||||
group: str = Field(..., description="Group name for the tool")
|
||||
use: str = Field(
|
||||
...,
|
||||
description="Variable name of the tool provider(e.g. deerflow.sandbox.tools:bash_tool)",
|
||||
)
|
||||
model_config = ConfigDict(extra="allow")
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
|
||||
class ToolGroupConfig(BaseModel):
|
||||
"""Config section for a tool group"""
|
||||
|
||||
name: str = Field(..., description="Unique name for the tool group")
|
||||
model_config = ConfigDict(extra="allow")
|
||||
|
||||
|
||||
class ToolConfig(BaseModel):
|
||||
"""Config section for a tool"""
|
||||
|
||||
name: str = Field(..., description="Unique name for the tool")
|
||||
group: str = Field(..., description="Group name for the tool")
|
||||
enabled: bool = Field(default=True, description="Whether the tool is registered with the agent")
|
||||
use: str = Field(
|
||||
...,
|
||||
description="Variable name of the tool provider(e.g. deerflow.sandbox.tools:bash_tool)",
|
||||
)
|
||||
model_config = ConfigDict(extra="allow")
|
||||
@@ -0,0 +1,2 @@
|
||||
"""External service integrations owned by the DeerFlow harness."""
|
||||
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
"""WeKnora REST integration."""
|
||||
|
||||
from deerflow.integrations.weknora.client import WeKnoraClient, WeKnoraError
|
||||
|
||||
__all__ = ["WeKnoraClient", "WeKnoraError"]
|
||||
+443
@@ -0,0 +1,443 @@
|
||||
"""Small, version-pinned adapter over WeKnora's official REST API."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping, Sequence
|
||||
from typing import Any
|
||||
from urllib.parse import quote
|
||||
|
||||
import httpx
|
||||
|
||||
|
||||
class WeKnoraError(RuntimeError):
|
||||
"""Normalized upstream failure that is safe for DeerFlow to translate."""
|
||||
|
||||
def __init__(self, message: str, *, status_code: int = 502) -> None:
|
||||
super().__init__(message)
|
||||
self.status_code = status_code
|
||||
|
||||
|
||||
def _data(payload: Any) -> Any:
|
||||
if not isinstance(payload, dict):
|
||||
raise WeKnoraError("WeKnora returned an invalid response")
|
||||
if payload.get("success") is False:
|
||||
message = payload.get("message") or payload.get("error") or "WeKnora rejected the request"
|
||||
raise WeKnoraError(str(message), status_code=400)
|
||||
return payload.get("data", payload)
|
||||
|
||||
|
||||
def _as_list(value: Any) -> list[dict[str, Any]]:
|
||||
if isinstance(value, list):
|
||||
return [dict(item) for item in value if isinstance(item, Mapping)]
|
||||
if isinstance(value, dict):
|
||||
for key in ("items", "list", "knowledge_bases", "knowledge", "records"):
|
||||
nested = value.get(key)
|
||||
if isinstance(nested, list):
|
||||
return [dict(item) for item in nested if isinstance(item, Mapping)]
|
||||
return []
|
||||
|
||||
|
||||
class WeKnoraClient:
|
||||
"""Async client for a trusted-network WeKnora endpoint.
|
||||
|
||||
The caller may inject an ``httpx.AsyncClient`` for connection pooling and
|
||||
tests. When omitted, each public method creates and closes a short-lived
|
||||
client. DeerFlow deliberately sends no WeKnora credential; the independently
|
||||
deployed service or its reverse proxy must allow DeerFlow's internal traffic.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
base_url: str,
|
||||
timeout_seconds: float = 30.0,
|
||||
http_client: httpx.AsyncClient | None = None,
|
||||
) -> None:
|
||||
self.base_url = base_url.rstrip("/")
|
||||
self.timeout_seconds = timeout_seconds
|
||||
self._http_client = http_client
|
||||
if not self.base_url:
|
||||
raise ValueError("WeKnora base URL is required")
|
||||
|
||||
async def _request(
|
||||
self,
|
||||
method: str,
|
||||
path: str,
|
||||
*,
|
||||
json: Any = None,
|
||||
params: Mapping[str, Any] | None = None,
|
||||
files: Any = None,
|
||||
data: Mapping[str, Any] | None = None,
|
||||
) -> Any:
|
||||
headers = {"Accept": "application/json"}
|
||||
owns_client = self._http_client is None
|
||||
client = self._http_client or httpx.AsyncClient(timeout=httpx.Timeout(self.timeout_seconds), follow_redirects=False)
|
||||
try:
|
||||
response = await client.request(
|
||||
method,
|
||||
f"{self.base_url}{path}",
|
||||
headers=headers,
|
||||
json=json,
|
||||
params=params,
|
||||
files=files,
|
||||
data=data,
|
||||
)
|
||||
except httpx.TimeoutException as exc:
|
||||
raise WeKnoraError("WeKnora request timed out", status_code=504) from exc
|
||||
except httpx.HTTPError as exc:
|
||||
raise WeKnoraError("WeKnora is unavailable", status_code=502) from exc
|
||||
finally:
|
||||
if owns_client:
|
||||
await client.aclose()
|
||||
if response.status_code == 404:
|
||||
raise WeKnoraError("WeKnora resource was not found", status_code=404)
|
||||
if response.status_code in {401, 403}:
|
||||
raise WeKnoraError(
|
||||
"WeKnora rejected DeerFlow's anonymous internal request; allow DeerFlow in WeKnora or its reverse proxy",
|
||||
status_code=502,
|
||||
)
|
||||
if response.status_code >= 400:
|
||||
detail = ""
|
||||
try:
|
||||
body = response.json()
|
||||
if isinstance(body, dict):
|
||||
detail = str(body.get("message") or body.get("error") or body.get("detail") or "")
|
||||
except ValueError:
|
||||
detail = ""
|
||||
raise WeKnoraError(detail or f"WeKnora returned HTTP {response.status_code}", status_code=502)
|
||||
if response.status_code == 204 or not response.content:
|
||||
return {"success": True}
|
||||
try:
|
||||
return response.json()
|
||||
except ValueError as exc:
|
||||
raise WeKnoraError("WeKnora returned invalid JSON") from exc
|
||||
|
||||
async def health(self) -> dict[str, Any]:
|
||||
payload = await self._request("GET", "/health")
|
||||
return dict(payload) if isinstance(payload, dict) else {"status": "ok"}
|
||||
|
||||
async def list_knowledge_bases(self) -> list[dict[str, Any]]:
|
||||
return _as_list(_data(await self._request("GET", "/api/v1/knowledge-bases")))
|
||||
|
||||
async def get_knowledge_base(self, knowledge_base_id: str) -> dict[str, Any]:
|
||||
value = _data(await self._request("GET", f"/api/v1/knowledge-bases/{quote(knowledge_base_id, safe='')}"))
|
||||
if not isinstance(value, dict):
|
||||
raise WeKnoraError("WeKnora returned an invalid knowledge base")
|
||||
return dict(value)
|
||||
|
||||
async def list_models(self) -> list[dict[str, Any]]:
|
||||
"""List models from WeKnora; model credentials are never returned."""
|
||||
|
||||
return _as_list(_data(await self._request("GET", "/api/v1/models")))
|
||||
|
||||
async def get_default_embedding_model_id(self) -> str:
|
||||
models = [item for item in await self.list_models() if str(item.get("type") or "").strip().lower() == "embedding"]
|
||||
preferred = next((item for item in models if item.get("is_default") is True), None)
|
||||
selected = preferred or next(
|
||||
(item for item in models if str(item.get("status") or "active").lower() == "active"),
|
||||
None,
|
||||
)
|
||||
model_id = str((selected or {}).get("id") or "")
|
||||
if not model_id:
|
||||
raise WeKnoraError(
|
||||
"WeKnora has no available embedding model; configure its default model in WeKnora",
|
||||
status_code=400,
|
||||
)
|
||||
return model_id
|
||||
|
||||
async def get_default_knowledge_qa_model_id(self) -> str:
|
||||
"""Resolve WeKnora's own active synthesis model for Wiki indexing."""
|
||||
|
||||
models = [item for item in await self.list_models() if str(item.get("type") or "").strip().lower() == "knowledgeqa"]
|
||||
preferred = next((item for item in models if item.get("is_default") is True), None)
|
||||
selected = preferred or next(
|
||||
(item for item in models if str(item.get("status") or "active").lower() == "active"),
|
||||
None,
|
||||
)
|
||||
model_id = str((selected or {}).get("id") or "")
|
||||
if not model_id:
|
||||
raise WeKnoraError(
|
||||
"WeKnora has no available KnowledgeQA model; configure one in WeKnora before creating a Wiki",
|
||||
status_code=400,
|
||||
)
|
||||
return model_id
|
||||
|
||||
async def create_knowledge_base(
|
||||
self,
|
||||
*,
|
||||
name: str,
|
||||
description: str = "",
|
||||
kb_type: str = "document",
|
||||
embedding_model_id: str | None = None,
|
||||
wiki_enabled: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
model_id = embedding_model_id or await self.get_default_embedding_model_id()
|
||||
create_payload: dict[str, Any] = {
|
||||
"name": name,
|
||||
"description": description,
|
||||
"type": kb_type,
|
||||
"embedding_model_id": model_id,
|
||||
}
|
||||
if wiki_enabled:
|
||||
synthesis_model_id = await self.get_default_knowledge_qa_model_id()
|
||||
create_payload.update(
|
||||
{
|
||||
"summary_model_id": synthesis_model_id,
|
||||
"indexing_strategy": {
|
||||
"vector_enabled": True,
|
||||
"keyword_enabled": True,
|
||||
"wiki_enabled": True,
|
||||
"graph_enabled": False,
|
||||
},
|
||||
"wiki_config": {
|
||||
"synthesis_model_id": synthesis_model_id,
|
||||
"extraction_granularity": "standard",
|
||||
},
|
||||
}
|
||||
)
|
||||
payload = await self._request(
|
||||
"POST",
|
||||
"/api/v1/knowledge-bases",
|
||||
json=create_payload,
|
||||
)
|
||||
value = _data(payload)
|
||||
if not isinstance(value, dict) or not value.get("id"):
|
||||
raise WeKnoraError("WeKnora did not return the created knowledge base")
|
||||
return dict(value)
|
||||
|
||||
async def update_knowledge_base(self, knowledge_base_id: str, changes: Mapping[str, Any]) -> dict[str, Any]:
|
||||
value = _data(
|
||||
await self._request(
|
||||
"PUT",
|
||||
f"/api/v1/knowledge-bases/{quote(knowledge_base_id, safe='')}",
|
||||
json=dict(changes),
|
||||
)
|
||||
)
|
||||
return dict(value) if isinstance(value, dict) else {}
|
||||
|
||||
async def delete_knowledge_base(self, knowledge_base_id: str) -> None:
|
||||
await self._request("DELETE", f"/api/v1/knowledge-bases/{quote(knowledge_base_id, safe='')}")
|
||||
|
||||
async def list_documents(self, knowledge_base_id: str, *, page: int = 1, page_size: int = 100) -> dict[str, Any]:
|
||||
payload = await self._request(
|
||||
"GET",
|
||||
f"/api/v1/knowledge-bases/{quote(knowledge_base_id, safe='')}/knowledge",
|
||||
params={"page": page, "page_size": page_size},
|
||||
)
|
||||
rows = _as_list(_data(payload))
|
||||
total = payload.get("total", len(rows)) if isinstance(payload, dict) else len(rows)
|
||||
return {"items": rows, "total": int(total or 0), "page": page, "page_size": page_size}
|
||||
|
||||
async def upload_document(
|
||||
self,
|
||||
knowledge_base_id: str,
|
||||
*,
|
||||
filename: str,
|
||||
content: bytes,
|
||||
content_type: str = "application/octet-stream",
|
||||
) -> dict[str, Any]:
|
||||
value = _data(
|
||||
await self._request(
|
||||
"POST",
|
||||
f"/api/v1/knowledge-bases/{quote(knowledge_base_id, safe='')}/knowledge/file",
|
||||
files={"file": (filename, content, content_type)},
|
||||
)
|
||||
)
|
||||
return dict(value) if isinstance(value, dict) else {}
|
||||
|
||||
async def import_document_url(self, knowledge_base_id: str, *, url: str) -> dict[str, Any]:
|
||||
value = _data(
|
||||
await self._request(
|
||||
"POST",
|
||||
f"/api/v1/knowledge-bases/{quote(knowledge_base_id, safe='')}/knowledge/url",
|
||||
json={"url": url},
|
||||
)
|
||||
)
|
||||
return dict(value) if isinstance(value, dict) else {}
|
||||
|
||||
async def create_manual_document(
|
||||
self,
|
||||
knowledge_base_id: str,
|
||||
*,
|
||||
title: str,
|
||||
content: str,
|
||||
) -> dict[str, Any]:
|
||||
value = _data(
|
||||
await self._request(
|
||||
"POST",
|
||||
f"/api/v1/knowledge-bases/{quote(knowledge_base_id, safe='')}/knowledge/manual",
|
||||
json={"title": title, "content": content, "status": "publish"},
|
||||
)
|
||||
)
|
||||
return dict(value) if isinstance(value, dict) else {}
|
||||
|
||||
async def delete_document(self, knowledge_id: str) -> None:
|
||||
await self._request("DELETE", f"/api/v1/knowledge/{quote(knowledge_id, safe='')}")
|
||||
|
||||
async def get_document(self, knowledge_id: str) -> dict[str, Any]:
|
||||
value = _data(await self._request("GET", f"/api/v1/knowledge/{quote(knowledge_id, safe='')}"))
|
||||
if not isinstance(value, dict):
|
||||
raise WeKnoraError("WeKnora returned an invalid document")
|
||||
return dict(value)
|
||||
|
||||
async def get_document_preview(self, knowledge_id: str) -> tuple[bytes, str]:
|
||||
"""Return the original document preview bytes and its media type."""
|
||||
|
||||
owns_client = self._http_client is None
|
||||
client = self._http_client or httpx.AsyncClient(
|
||||
timeout=httpx.Timeout(self.timeout_seconds),
|
||||
follow_redirects=True,
|
||||
)
|
||||
try:
|
||||
response = await client.request(
|
||||
"GET",
|
||||
f"{self.base_url}/api/v1/knowledge/{quote(knowledge_id, safe='')}/preview",
|
||||
headers={"Accept": "*/*"},
|
||||
follow_redirects=True,
|
||||
)
|
||||
except httpx.TimeoutException as exc:
|
||||
raise WeKnoraError("WeKnora preview request timed out", status_code=504) from exc
|
||||
except httpx.HTTPError as exc:
|
||||
raise WeKnoraError("WeKnora preview is unavailable", status_code=502) from exc
|
||||
finally:
|
||||
if owns_client:
|
||||
await client.aclose()
|
||||
if response.status_code == 404:
|
||||
raise WeKnoraError("WeKnora document preview was not found", status_code=404)
|
||||
if response.status_code in {401, 403}:
|
||||
raise WeKnoraError(
|
||||
"WeKnora rejected DeerFlow's anonymous internal request; allow DeerFlow in WeKnora or its reverse proxy",
|
||||
status_code=502,
|
||||
)
|
||||
if response.status_code >= 400:
|
||||
raise WeKnoraError(f"WeKnora returned HTTP {response.status_code}", status_code=502)
|
||||
media_type = response.headers.get("content-type", "application/octet-stream").split(";", 1)[0].strip()
|
||||
return response.content, media_type or "application/octet-stream"
|
||||
|
||||
async def reprocess_document(self, knowledge_id: str) -> dict[str, Any]:
|
||||
value = _data(await self._request("POST", f"/api/v1/knowledge/{quote(knowledge_id, safe='')}/reparse"))
|
||||
return dict(value) if isinstance(value, dict) else {}
|
||||
|
||||
async def list_chunks(self, knowledge_id: str, *, page: int = 1, page_size: int = 20) -> dict[str, Any]:
|
||||
payload = await self._request(
|
||||
"GET",
|
||||
f"/api/v1/chunks/{quote(knowledge_id, safe='')}",
|
||||
params={"page": page, "page_size": page_size},
|
||||
)
|
||||
rows = _as_list(_data(payload))
|
||||
total = payload.get("total", len(rows)) if isinstance(payload, dict) else len(rows)
|
||||
return {"items": rows, "total": int(total or 0), "page": page, "page_size": page_size}
|
||||
|
||||
async def get_wiki_graph(self, knowledge_base_id: str, *, limit: int = 200) -> dict[str, Any]:
|
||||
value = _data(
|
||||
await self._request(
|
||||
"GET",
|
||||
f"/api/v1/knowledgebase/{quote(knowledge_base_id, safe='')}/wiki/graph",
|
||||
params={"mode": "overview", "limit": max(1, min(limit, 500))},
|
||||
)
|
||||
)
|
||||
if not isinstance(value, dict):
|
||||
raise WeKnoraError("WeKnora returned an invalid wiki graph")
|
||||
return dict(value)
|
||||
|
||||
async def list_wiki_pages(
|
||||
self,
|
||||
knowledge_base_id: str,
|
||||
*,
|
||||
page: int = 1,
|
||||
page_size: int = 50,
|
||||
query: str = "",
|
||||
) -> dict[str, Any]:
|
||||
payload = await self._request(
|
||||
"GET",
|
||||
f"/api/v1/knowledgebase/{quote(knowledge_base_id, safe='')}/wiki/pages",
|
||||
params={"page": page, "page_size": page_size, "query": query or None},
|
||||
)
|
||||
value = _data(payload)
|
||||
source = value if isinstance(value, dict) else payload if isinstance(payload, dict) else {}
|
||||
rows = _as_list(source.get("pages") if isinstance(source, dict) else value)
|
||||
return {
|
||||
"pages": rows,
|
||||
"total": int(source.get("total") or len(rows)),
|
||||
"page": int(source.get("page") or page),
|
||||
"page_size": int(source.get("page_size") or page_size),
|
||||
"total_pages": int(source.get("total_pages") or max(1, (int(source.get("total") or len(rows)) + page_size - 1) // page_size)),
|
||||
}
|
||||
|
||||
async def get_wiki_page(self, knowledge_base_id: str, slug: str) -> dict[str, Any]:
|
||||
safe_slug = "/".join(quote(part, safe="") for part in slug.split("/"))
|
||||
value = _data(
|
||||
await self._request(
|
||||
"GET",
|
||||
f"/api/v1/knowledgebase/{quote(knowledge_base_id, safe='')}/wiki/pages/{safe_slug}",
|
||||
)
|
||||
)
|
||||
if not isinstance(value, dict):
|
||||
raise WeKnoraError("WeKnora returned an invalid Wiki page")
|
||||
return dict(value)
|
||||
|
||||
async def search(self, query: str, knowledge_base_ids: Sequence[str]) -> list[dict[str, Any]]:
|
||||
ids = [str(value) for value in knowledge_base_ids if str(value)]
|
||||
if not ids:
|
||||
return []
|
||||
rows = _as_list(
|
||||
_data(
|
||||
await self._request(
|
||||
"POST",
|
||||
"/api/v1/knowledge-search",
|
||||
json={"query": query, "knowledge_base_ids": ids},
|
||||
)
|
||||
)
|
||||
)
|
||||
normalized: list[dict[str, Any]] = []
|
||||
for row in rows:
|
||||
normalized.append(
|
||||
{
|
||||
"chunk_id": str(row.get("id") or row.get("chunk_id") or ""),
|
||||
"content": str(row.get("content") or ""),
|
||||
"knowledge_id": str(row.get("knowledge_id") or ""),
|
||||
"knowledge_base_id": str(row.get("knowledge_base_id") or ""),
|
||||
"title": str(row.get("knowledge_title") or row.get("title") or row.get("knowledge_filename") or ""),
|
||||
"filename": str(row.get("knowledge_filename") or ""),
|
||||
"score": float(row.get("score") or 0),
|
||||
"chunk_index": int(row.get("chunk_index") or 0),
|
||||
"metadata": row.get("metadata") if isinstance(row.get("metadata"), dict) else {},
|
||||
}
|
||||
)
|
||||
return normalized
|
||||
|
||||
async def get_chunk(self, chunk_id: str) -> dict[str, Any]:
|
||||
value = _data(await self._request("GET", f"/api/v1/chunks/by-id/{quote(chunk_id, safe='')}"))
|
||||
if not isinstance(value, dict):
|
||||
raise WeKnoraError("WeKnora returned an invalid chunk")
|
||||
return dict(value)
|
||||
|
||||
async def get_chunk_context(self, chunk_id: str, *, expected_knowledge_base_id: str, radius: int = 2) -> list[dict[str, Any]]:
|
||||
center = await self.get_chunk(chunk_id)
|
||||
if str(center.get("knowledge_base_id") or "") != expected_knowledge_base_id:
|
||||
raise WeKnoraError("Chunk does not belong to the authorized knowledge base", status_code=404)
|
||||
|
||||
before: list[dict[str, Any]] = []
|
||||
cursor = center
|
||||
for _ in range(max(0, min(radius, 4))):
|
||||
previous_id = str(cursor.get("pre_chunk_id") or "")
|
||||
if not previous_id:
|
||||
break
|
||||
previous = await self.get_chunk(previous_id)
|
||||
if str(previous.get("knowledge_base_id") or "") != expected_knowledge_base_id:
|
||||
break
|
||||
before.append(previous)
|
||||
cursor = previous
|
||||
|
||||
after: list[dict[str, Any]] = []
|
||||
cursor = center
|
||||
for _ in range(max(0, min(radius, 4))):
|
||||
next_id = str(cursor.get("next_chunk_id") or "")
|
||||
if not next_id:
|
||||
break
|
||||
following = await self.get_chunk(next_id)
|
||||
if str(following.get("knowledge_base_id") or "") != expected_knowledge_base_id:
|
||||
break
|
||||
after.append(following)
|
||||
cursor = following
|
||||
return [*reversed(before), center, *after]
|
||||
+43
@@ -0,0 +1,43 @@
|
||||
"""Runtime composition shared by Gateway routes and the read-only agent tool."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import httpx
|
||||
|
||||
from deerflow.config import get_app_config
|
||||
from deerflow.config.app_config import AppConfig
|
||||
from deerflow.config.llmwiki_config import ResolvedLlmWikiRuntime, resolve_llmwiki_runtime
|
||||
from deerflow.config.system_settings import load_system_settings
|
||||
from deerflow.integrations.weknora.client import WeKnoraClient
|
||||
from deerflow.persistence.llmwiki import LlmWikiStore
|
||||
|
||||
_default_store: LlmWikiStore | None = None
|
||||
|
||||
|
||||
def register_llmwiki_store(store: LlmWikiStore | None) -> None:
|
||||
global _default_store
|
||||
_default_store = store
|
||||
|
||||
|
||||
def get_llmwiki_store() -> LlmWikiStore:
|
||||
if _default_store is None:
|
||||
raise RuntimeError("LLMWiki store is not initialized")
|
||||
return _default_store
|
||||
|
||||
|
||||
def get_resolved_llmwiki_runtime(config: AppConfig | None = None) -> ResolvedLlmWikiRuntime:
|
||||
current = config or get_app_config()
|
||||
return resolve_llmwiki_runtime(current.llmwiki, load_system_settings().llmwiki)
|
||||
|
||||
|
||||
def build_weknora_client(
|
||||
runtime: ResolvedLlmWikiRuntime,
|
||||
*,
|
||||
http_client: httpx.AsyncClient | None = None,
|
||||
) -> WeKnoraClient:
|
||||
if not runtime.weknora_enabled:
|
||||
raise RuntimeError("WeKnora mode is not enabled")
|
||||
return WeKnoraClient(
|
||||
base_url=runtime.api_base_url,
|
||||
http_client=http_client,
|
||||
)
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
"""DeerFlow-owned LLMWiki authorization metadata."""
|
||||
|
||||
from deerflow.persistence.llmwiki.base import LlmWikiScope, LlmWikiStore
|
||||
from deerflow.persistence.llmwiki.memory import MemoryLlmWikiStore
|
||||
from deerflow.persistence.llmwiki.model import LlmWikiConversationDepositRow, LlmWikiKnowledgeBaseRow
|
||||
from deerflow.persistence.llmwiki.sql import LlmWikiRepository
|
||||
|
||||
__all__ = [
|
||||
"LlmWikiKnowledgeBaseRow",
|
||||
"LlmWikiConversationDepositRow",
|
||||
"LlmWikiRepository",
|
||||
"LlmWikiScope",
|
||||
"LlmWikiStore",
|
||||
"MemoryLlmWikiStore",
|
||||
"make_llmwiki_store",
|
||||
]
|
||||
|
||||
|
||||
def make_llmwiki_store(session_factory):
|
||||
if session_factory is None:
|
||||
return MemoryLlmWikiStore()
|
||||
return LlmWikiRepository(session_factory)
|
||||
+72
@@ -0,0 +1,72 @@
|
||||
"""Persistence contract for DeerFlow-owned LLMWiki authorization metadata."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Any, Literal
|
||||
|
||||
LlmWikiScope = Literal["personal", "public", "all"]
|
||||
|
||||
|
||||
class LlmWikiStore(ABC):
|
||||
@abstractmethod
|
||||
async def create_mapping(self, *, weknora_id: str, owner_user_id: str, name: str, description: str, kb_type: str) -> dict[str, Any]: ...
|
||||
|
||||
@abstractmethod
|
||||
async def list_visible(self, user_id: str, *, scope: LlmWikiScope = "all", is_admin: bool = False) -> list[dict[str, Any]]: ...
|
||||
|
||||
@abstractmethod
|
||||
async def get_authorized(self, mapping_id: str, user_id: str, *, write: bool, is_admin: bool) -> dict[str, Any] | None: ...
|
||||
|
||||
@abstractmethod
|
||||
async def get_by_weknora_id(self, weknora_id: str) -> dict[str, Any] | None: ...
|
||||
|
||||
@abstractmethod
|
||||
async def update_mapping(self, mapping_id: str, *, name: str | None = None, description: str | None = None) -> dict[str, Any] | None: ...
|
||||
|
||||
@abstractmethod
|
||||
async def delete_mapping(self, mapping_id: str) -> bool: ...
|
||||
|
||||
@abstractmethod
|
||||
async def request_publish(self, mapping_id: str, actor_user_id: str, *, is_admin: bool) -> dict[str, Any] | None: ...
|
||||
|
||||
@abstractmethod
|
||||
async def review_publish(self, mapping_id: str, reviewer_user_id: str, *, approved: bool) -> dict[str, Any] | None: ...
|
||||
|
||||
@abstractmethod
|
||||
async def unpublish(self, mapping_id: str, actor_user_id: str, *, is_admin: bool) -> dict[str, Any] | None: ...
|
||||
|
||||
@abstractmethod
|
||||
async def list_publications(self, *, status: str | None = None) -> list[dict[str, Any]]: ...
|
||||
|
||||
@abstractmethod
|
||||
async def get_conversation_deposit_mapping(self) -> dict[str, Any] | None: ...
|
||||
|
||||
@abstractmethod
|
||||
async def get_conversation_deposit(
|
||||
self,
|
||||
*,
|
||||
thread_id: str,
|
||||
assistant_message_id: str,
|
||||
) -> dict[str, Any] | None: ...
|
||||
|
||||
@abstractmethod
|
||||
async def upsert_conversation_deposit(
|
||||
self,
|
||||
*,
|
||||
thread_id: str,
|
||||
human_message_id: str | None,
|
||||
assistant_message_id: str,
|
||||
knowledge_base_mapping_id: str,
|
||||
weknora_knowledge_id: str | None,
|
||||
title: str,
|
||||
created_by_user_id: str | None,
|
||||
) -> dict[str, Any]: ...
|
||||
|
||||
@abstractmethod
|
||||
async def delete_conversation_deposits(
|
||||
self,
|
||||
*,
|
||||
thread_id: str,
|
||||
message_ids: list[str] | None = None,
|
||||
) -> list[dict[str, Any]]: ...
|
||||
+190
@@ -0,0 +1,190 @@
|
||||
"""In-memory LLMWiki metadata store for tests and DB-less deployments."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
|
||||
from deerflow.persistence.llmwiki.base import LlmWikiScope, LlmWikiStore
|
||||
|
||||
|
||||
def _now() -> str:
|
||||
return datetime.now(UTC).isoformat()
|
||||
|
||||
|
||||
class MemoryLlmWikiStore(LlmWikiStore):
|
||||
def __init__(self) -> None:
|
||||
self._rows: dict[str, dict[str, Any]] = {}
|
||||
self._conversation_deposits: dict[str, dict[str, Any]] = {}
|
||||
|
||||
async def create_mapping(self, *, weknora_id: str, owner_user_id: str, name: str, description: str, kb_type: str) -> dict[str, Any]:
|
||||
if any(row["weknora_id"] == weknora_id for row in self._rows.values()):
|
||||
raise ValueError("WeKnora knowledge base is already mapped")
|
||||
now = _now()
|
||||
row = {
|
||||
"id": str(uuid4()),
|
||||
"weknora_id": weknora_id,
|
||||
"owner_user_id": owner_user_id,
|
||||
"name": name,
|
||||
"description": description,
|
||||
"kb_type": kb_type,
|
||||
"publication_status": "private",
|
||||
"published_by": None,
|
||||
"reviewed_by": None,
|
||||
"published_at": None,
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
self._rows[row["id"]] = row
|
||||
return dict(row)
|
||||
|
||||
async def list_visible(self, user_id: str, *, scope: LlmWikiScope = "all", is_admin: bool = False) -> list[dict[str, Any]]:
|
||||
rows = list(self._rows.values())
|
||||
if scope == "personal":
|
||||
rows = rows if is_admin else [row for row in rows if row["owner_user_id"] == user_id]
|
||||
elif scope == "public":
|
||||
rows = [row for row in rows if row["publication_status"] == "published"]
|
||||
else:
|
||||
rows = rows if is_admin else [row for row in rows if row["owner_user_id"] == user_id or row["publication_status"] == "published"]
|
||||
return [dict(row) for row in sorted(rows, key=lambda row: row["created_at"], reverse=True)]
|
||||
|
||||
async def get_authorized(self, mapping_id: str, user_id: str, *, write: bool, is_admin: bool) -> dict[str, Any] | None:
|
||||
row = self._rows.get(mapping_id)
|
||||
if row is None:
|
||||
return None
|
||||
if is_admin or row["owner_user_id"] == user_id:
|
||||
return dict(row)
|
||||
if not write and row["publication_status"] == "published":
|
||||
return dict(row)
|
||||
return None
|
||||
|
||||
async def get_by_weknora_id(self, weknora_id: str) -> dict[str, Any] | None:
|
||||
for row in self._rows.values():
|
||||
if row["weknora_id"] == weknora_id:
|
||||
return dict(row)
|
||||
return None
|
||||
|
||||
async def update_mapping(self, mapping_id: str, *, name: str | None = None, description: str | None = None) -> dict[str, Any] | None:
|
||||
row = self._rows.get(mapping_id)
|
||||
if row is None:
|
||||
return None
|
||||
if name is not None:
|
||||
row["name"] = name
|
||||
if description is not None:
|
||||
row["description"] = description
|
||||
row["updated_at"] = _now()
|
||||
return dict(row)
|
||||
|
||||
async def delete_mapping(self, mapping_id: str) -> bool:
|
||||
return self._rows.pop(mapping_id, None) is not None
|
||||
|
||||
async def request_publish(self, mapping_id: str, actor_user_id: str, *, is_admin: bool) -> dict[str, Any] | None:
|
||||
row = self._rows.get(mapping_id)
|
||||
if row is None or (not is_admin and row["owner_user_id"] != actor_user_id):
|
||||
return None
|
||||
row["publication_status"] = "published"
|
||||
row["published_by"] = actor_user_id
|
||||
row["reviewed_by"] = None
|
||||
row["published_at"] = _now()
|
||||
row["updated_at"] = _now()
|
||||
return dict(row)
|
||||
|
||||
async def review_publish(self, mapping_id: str, reviewer_user_id: str, *, approved: bool) -> dict[str, Any] | None:
|
||||
row = self._rows.get(mapping_id)
|
||||
if row is None or row["publication_status"] != "pending":
|
||||
return None
|
||||
row["publication_status"] = "published" if approved else "rejected"
|
||||
row["reviewed_by"] = reviewer_user_id
|
||||
row["published_at"] = _now() if approved else None
|
||||
row["updated_at"] = _now()
|
||||
return dict(row)
|
||||
|
||||
async def unpublish(self, mapping_id: str, actor_user_id: str, *, is_admin: bool) -> dict[str, Any] | None:
|
||||
row = self._rows.get(mapping_id)
|
||||
if row is None or (not is_admin and row["owner_user_id"] != actor_user_id):
|
||||
return None
|
||||
row["publication_status"] = "private"
|
||||
row["reviewed_by"] = actor_user_id if is_admin else row.get("reviewed_by")
|
||||
row["published_at"] = None
|
||||
row["updated_at"] = _now()
|
||||
return dict(row)
|
||||
|
||||
async def list_publications(self, *, status: str | None = None) -> list[dict[str, Any]]:
|
||||
rows = [row for row in self._rows.values() if row["publication_status"] != "private"]
|
||||
if status:
|
||||
rows = [row for row in rows if row["publication_status"] == status]
|
||||
return [dict(row) for row in sorted(rows, key=lambda row: row["updated_at"], reverse=True)]
|
||||
|
||||
async def get_conversation_deposit_mapping(self) -> dict[str, Any] | None:
|
||||
for row in self._rows.values():
|
||||
if row.get("name") == "对话沉淀" and row.get("publication_status") == "published":
|
||||
return dict(row)
|
||||
return None
|
||||
|
||||
async def get_conversation_deposit(
|
||||
self,
|
||||
*,
|
||||
thread_id: str,
|
||||
assistant_message_id: str,
|
||||
) -> dict[str, Any] | None:
|
||||
key = f"{thread_id}:{assistant_message_id}"
|
||||
row = self._conversation_deposits.get(key)
|
||||
return dict(row) if row else None
|
||||
|
||||
async def upsert_conversation_deposit(
|
||||
self,
|
||||
*,
|
||||
thread_id: str,
|
||||
human_message_id: str | None,
|
||||
assistant_message_id: str,
|
||||
knowledge_base_mapping_id: str,
|
||||
weknora_knowledge_id: str | None,
|
||||
title: str,
|
||||
created_by_user_id: str | None,
|
||||
) -> dict[str, Any]:
|
||||
key = f"{thread_id}:{assistant_message_id}"
|
||||
now = _now()
|
||||
row = self._conversation_deposits.get(key)
|
||||
if row is None:
|
||||
row = {
|
||||
"id": str(uuid4()),
|
||||
"thread_id": thread_id,
|
||||
"human_message_id": human_message_id,
|
||||
"assistant_message_id": assistant_message_id,
|
||||
"knowledge_base_mapping_id": knowledge_base_mapping_id,
|
||||
"weknora_knowledge_id": weknora_knowledge_id,
|
||||
"title": title,
|
||||
"created_by_user_id": created_by_user_id,
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
self._conversation_deposits[key] = row
|
||||
else:
|
||||
row.update(
|
||||
{
|
||||
"human_message_id": human_message_id,
|
||||
"knowledge_base_mapping_id": knowledge_base_mapping_id,
|
||||
"weknora_knowledge_id": weknora_knowledge_id,
|
||||
"title": title,
|
||||
"updated_at": now,
|
||||
}
|
||||
)
|
||||
return dict(row)
|
||||
|
||||
async def delete_conversation_deposits(
|
||||
self,
|
||||
*,
|
||||
thread_id: str,
|
||||
message_ids: list[str] | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
ids = set(message_ids or [])
|
||||
deleted: list[dict[str, Any]] = []
|
||||
for key, row in list(self._conversation_deposits.items()):
|
||||
if row.get("thread_id") != thread_id:
|
||||
continue
|
||||
if ids and row.get("human_message_id") not in ids and row.get("assistant_message_id") not in ids:
|
||||
continue
|
||||
deleted.append(dict(row))
|
||||
self._conversation_deposits.pop(key, None)
|
||||
return deleted
|
||||
+63
@@ -0,0 +1,63 @@
|
||||
"""ORM mapping between DeerFlow ownership and WeKnora knowledge bases."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from sqlalchemy import String, UniqueConstraint
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from deerflow.persistence.base import Base
|
||||
from deerflow.persistence.types import BeijingDateTime
|
||||
|
||||
|
||||
class LlmWikiKnowledgeBaseRow(Base):
|
||||
__tablename__ = "llmwiki_knowledge_bases"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True)
|
||||
weknora_id: Mapped[str] = mapped_column(String(128), unique=True, index=True)
|
||||
owner_user_id: Mapped[str] = mapped_column(String(64), index=True)
|
||||
name: Mapped[str] = mapped_column(String(255), default="")
|
||||
description: Mapped[str] = mapped_column(String(2048), default="")
|
||||
kb_type: Mapped[str] = mapped_column(String(32), default="document")
|
||||
publication_status: Mapped[str] = mapped_column(String(16), default="private", index=True)
|
||||
published_by: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
reviewed_by: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
published_at: Mapped[datetime | None] = mapped_column(BeijingDateTime(), nullable=True)
|
||||
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), default=lambda: datetime.now(UTC))
|
||||
updated_at: Mapped[datetime] = mapped_column(BeijingDateTime(), default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC))
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {column.name: getattr(self, column.name) for column in self.__table__.columns}
|
||||
|
||||
|
||||
class LlmWikiConversationDepositRow(Base):
|
||||
"""Mapping from one DeerFlow answer turn to its WeKnora manual document.
|
||||
|
||||
The actual content lives in WeKnora. DeerFlow keeps only the stable ids it
|
||||
needs to rewrite/delete the document when the user regenerates or removes a
|
||||
conversation turn.
|
||||
"""
|
||||
|
||||
__tablename__ = "llmwiki_conversation_deposits"
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"thread_id",
|
||||
"assistant_message_id",
|
||||
name="uq_llmwiki_conversation_deposit_answer",
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True)
|
||||
thread_id: Mapped[str] = mapped_column(String(128), index=True)
|
||||
human_message_id: Mapped[str | None] = mapped_column(String(128), nullable=True, index=True)
|
||||
assistant_message_id: Mapped[str] = mapped_column(String(128), index=True)
|
||||
knowledge_base_mapping_id: Mapped[str] = mapped_column(String(36), index=True)
|
||||
weknora_knowledge_id: Mapped[str | None] = mapped_column(String(128), nullable=True)
|
||||
title: Mapped[str] = mapped_column(String(255), default="")
|
||||
created_by_user_id: Mapped[str | None] = mapped_column(String(64), nullable=True, index=True)
|
||||
created_at: Mapped[datetime] = mapped_column(BeijingDateTime(), default=lambda: datetime.now(UTC))
|
||||
updated_at: Mapped[datetime] = mapped_column(BeijingDateTime(), default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC))
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {column.name: getattr(self, column.name) for column in self.__table__.columns}
|
||||
+244
@@ -0,0 +1,244 @@
|
||||
"""SQLAlchemy LLMWiki metadata repository."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
|
||||
from sqlalchemy import delete, or_, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
from deerflow.persistence.llmwiki.base import LlmWikiScope, LlmWikiStore
|
||||
from deerflow.persistence.llmwiki.model import LlmWikiConversationDepositRow, LlmWikiKnowledgeBaseRow
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(UTC)
|
||||
|
||||
|
||||
def _to_dict(row: LlmWikiKnowledgeBaseRow | LlmWikiConversationDepositRow) -> dict[str, Any]:
|
||||
data = row.to_dict()
|
||||
for key in ("published_at", "created_at", "updated_at"):
|
||||
value = data.get(key)
|
||||
if isinstance(value, datetime):
|
||||
data[key] = value.isoformat()
|
||||
return data
|
||||
|
||||
|
||||
class LlmWikiRepository(LlmWikiStore):
|
||||
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
|
||||
self._sf = session_factory
|
||||
|
||||
async def create_mapping(self, *, weknora_id: str, owner_user_id: str, name: str, description: str, kb_type: str) -> dict[str, Any]:
|
||||
row = LlmWikiKnowledgeBaseRow(
|
||||
id=str(uuid4()),
|
||||
weknora_id=weknora_id,
|
||||
owner_user_id=owner_user_id,
|
||||
name=name,
|
||||
description=description,
|
||||
kb_type=kb_type,
|
||||
publication_status="private",
|
||||
created_at=_now(),
|
||||
updated_at=_now(),
|
||||
)
|
||||
async with self._sf() as session:
|
||||
session.add(row)
|
||||
await session.commit()
|
||||
return _to_dict(row)
|
||||
|
||||
async def list_visible(self, user_id: str, *, scope: LlmWikiScope = "all", is_admin: bool = False) -> list[dict[str, Any]]:
|
||||
stmt = select(LlmWikiKnowledgeBaseRow)
|
||||
if scope == "personal" and not is_admin:
|
||||
stmt = stmt.where(LlmWikiKnowledgeBaseRow.owner_user_id == user_id)
|
||||
elif scope == "public":
|
||||
stmt = stmt.where(LlmWikiKnowledgeBaseRow.publication_status == "published")
|
||||
elif scope == "all" and not is_admin:
|
||||
stmt = stmt.where(
|
||||
or_(
|
||||
LlmWikiKnowledgeBaseRow.owner_user_id == user_id,
|
||||
LlmWikiKnowledgeBaseRow.publication_status == "published",
|
||||
)
|
||||
)
|
||||
stmt = stmt.order_by(LlmWikiKnowledgeBaseRow.created_at.desc())
|
||||
async with self._sf() as session:
|
||||
result = await session.execute(stmt)
|
||||
return [_to_dict(row) for row in result.scalars()]
|
||||
|
||||
async def get_authorized(self, mapping_id: str, user_id: str, *, write: bool, is_admin: bool) -> dict[str, Any] | None:
|
||||
stmt = select(LlmWikiKnowledgeBaseRow).where(LlmWikiKnowledgeBaseRow.id == mapping_id)
|
||||
if not is_admin:
|
||||
condition = LlmWikiKnowledgeBaseRow.owner_user_id == user_id
|
||||
if not write:
|
||||
condition = or_(condition, LlmWikiKnowledgeBaseRow.publication_status == "published")
|
||||
stmt = stmt.where(condition)
|
||||
async with self._sf() as session:
|
||||
row = (await session.execute(stmt)).scalar_one_or_none()
|
||||
return _to_dict(row) if row else None
|
||||
|
||||
async def get_by_weknora_id(self, weknora_id: str) -> dict[str, Any] | None:
|
||||
stmt = select(LlmWikiKnowledgeBaseRow).where(LlmWikiKnowledgeBaseRow.weknora_id == weknora_id)
|
||||
async with self._sf() as session:
|
||||
row = (await session.execute(stmt)).scalar_one_or_none()
|
||||
return _to_dict(row) if row else None
|
||||
|
||||
async def update_mapping(self, mapping_id: str, *, name: str | None = None, description: str | None = None) -> dict[str, Any] | None:
|
||||
async with self._sf() as session:
|
||||
row = await session.get(LlmWikiKnowledgeBaseRow, mapping_id)
|
||||
if row is None:
|
||||
return None
|
||||
if name is not None:
|
||||
row.name = name
|
||||
if description is not None:
|
||||
row.description = description
|
||||
row.updated_at = _now()
|
||||
await session.commit()
|
||||
return _to_dict(row)
|
||||
|
||||
async def delete_mapping(self, mapping_id: str) -> bool:
|
||||
async with self._sf() as session:
|
||||
row = await session.get(LlmWikiKnowledgeBaseRow, mapping_id)
|
||||
if row is None:
|
||||
return False
|
||||
await session.delete(row)
|
||||
await session.commit()
|
||||
return True
|
||||
|
||||
async def request_publish(self, mapping_id: str, actor_user_id: str, *, is_admin: bool) -> dict[str, Any] | None:
|
||||
async with self._sf() as session:
|
||||
row = await session.get(LlmWikiKnowledgeBaseRow, mapping_id)
|
||||
if row is None or (not is_admin and row.owner_user_id != actor_user_id):
|
||||
return None
|
||||
row.publication_status = "published"
|
||||
row.published_by = actor_user_id
|
||||
row.reviewed_by = None
|
||||
row.published_at = _now()
|
||||
row.updated_at = _now()
|
||||
await session.commit()
|
||||
return _to_dict(row)
|
||||
|
||||
async def review_publish(self, mapping_id: str, reviewer_user_id: str, *, approved: bool) -> dict[str, Any] | None:
|
||||
async with self._sf() as session:
|
||||
row = await session.get(LlmWikiKnowledgeBaseRow, mapping_id)
|
||||
if row is None or row.publication_status != "pending":
|
||||
return None
|
||||
row.publication_status = "published" if approved else "rejected"
|
||||
row.reviewed_by = reviewer_user_id
|
||||
row.published_at = _now() if approved else None
|
||||
row.updated_at = _now()
|
||||
await session.commit()
|
||||
return _to_dict(row)
|
||||
|
||||
async def unpublish(self, mapping_id: str, actor_user_id: str, *, is_admin: bool) -> dict[str, Any] | None:
|
||||
async with self._sf() as session:
|
||||
row = await session.get(LlmWikiKnowledgeBaseRow, mapping_id)
|
||||
if row is None or (not is_admin and row.owner_user_id != actor_user_id):
|
||||
return None
|
||||
row.publication_status = "private"
|
||||
if is_admin:
|
||||
row.reviewed_by = actor_user_id
|
||||
row.published_at = None
|
||||
row.updated_at = _now()
|
||||
await session.commit()
|
||||
return _to_dict(row)
|
||||
|
||||
async def list_publications(self, *, status: str | None = None) -> list[dict[str, Any]]:
|
||||
stmt = select(LlmWikiKnowledgeBaseRow).where(LlmWikiKnowledgeBaseRow.publication_status != "private")
|
||||
if status:
|
||||
stmt = stmt.where(LlmWikiKnowledgeBaseRow.publication_status == status)
|
||||
stmt = stmt.order_by(LlmWikiKnowledgeBaseRow.updated_at.desc())
|
||||
async with self._sf() as session:
|
||||
result = await session.execute(stmt)
|
||||
return [_to_dict(row) for row in result.scalars()]
|
||||
|
||||
async def get_conversation_deposit_mapping(self) -> dict[str, Any] | None:
|
||||
stmt = (
|
||||
select(LlmWikiKnowledgeBaseRow)
|
||||
.where(
|
||||
LlmWikiKnowledgeBaseRow.name == "对话沉淀",
|
||||
LlmWikiKnowledgeBaseRow.publication_status == "published",
|
||||
)
|
||||
.order_by(LlmWikiKnowledgeBaseRow.created_at.asc())
|
||||
.limit(1)
|
||||
)
|
||||
async with self._sf() as session:
|
||||
row = (await session.execute(stmt)).scalar_one_or_none()
|
||||
return _to_dict(row) if row else None
|
||||
|
||||
async def get_conversation_deposit(
|
||||
self,
|
||||
*,
|
||||
thread_id: str,
|
||||
assistant_message_id: str,
|
||||
) -> dict[str, Any] | None:
|
||||
stmt = select(LlmWikiConversationDepositRow).where(
|
||||
LlmWikiConversationDepositRow.thread_id == thread_id,
|
||||
LlmWikiConversationDepositRow.assistant_message_id == assistant_message_id,
|
||||
)
|
||||
async with self._sf() as session:
|
||||
row = (await session.execute(stmt)).scalar_one_or_none()
|
||||
return _to_dict(row) if row else None
|
||||
|
||||
async def upsert_conversation_deposit(
|
||||
self,
|
||||
*,
|
||||
thread_id: str,
|
||||
human_message_id: str | None,
|
||||
assistant_message_id: str,
|
||||
knowledge_base_mapping_id: str,
|
||||
weknora_knowledge_id: str | None,
|
||||
title: str,
|
||||
created_by_user_id: str | None,
|
||||
) -> dict[str, Any]:
|
||||
now = _now()
|
||||
async with self._sf() as session:
|
||||
stmt = select(LlmWikiConversationDepositRow).where(
|
||||
LlmWikiConversationDepositRow.thread_id == thread_id,
|
||||
LlmWikiConversationDepositRow.assistant_message_id == assistant_message_id,
|
||||
)
|
||||
row = (await session.execute(stmt)).scalar_one_or_none()
|
||||
if row is None:
|
||||
row = LlmWikiConversationDepositRow(
|
||||
id=str(uuid4()),
|
||||
thread_id=thread_id,
|
||||
human_message_id=human_message_id,
|
||||
assistant_message_id=assistant_message_id,
|
||||
knowledge_base_mapping_id=knowledge_base_mapping_id,
|
||||
weknora_knowledge_id=weknora_knowledge_id,
|
||||
title=title,
|
||||
created_by_user_id=created_by_user_id,
|
||||
created_at=now,
|
||||
updated_at=now,
|
||||
)
|
||||
session.add(row)
|
||||
else:
|
||||
row.human_message_id = human_message_id
|
||||
row.knowledge_base_mapping_id = knowledge_base_mapping_id
|
||||
row.weknora_knowledge_id = weknora_knowledge_id
|
||||
row.title = title
|
||||
row.updated_at = now
|
||||
await session.commit()
|
||||
return _to_dict(row)
|
||||
|
||||
async def delete_conversation_deposits(
|
||||
self,
|
||||
*,
|
||||
thread_id: str,
|
||||
message_ids: list[str] | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
conditions = [LlmWikiConversationDepositRow.thread_id == thread_id]
|
||||
if message_ids:
|
||||
conditions.append(
|
||||
or_(
|
||||
LlmWikiConversationDepositRow.human_message_id.in_(message_ids),
|
||||
LlmWikiConversationDepositRow.assistant_message_id.in_(message_ids),
|
||||
)
|
||||
)
|
||||
stmt = select(LlmWikiConversationDepositRow).where(*conditions)
|
||||
async with self._sf() as session:
|
||||
rows = (await session.execute(stmt)).scalars().all()
|
||||
deleted = [_to_dict(row) for row in rows]
|
||||
if rows:
|
||||
await session.execute(delete(LlmWikiConversationDepositRow).where(LlmWikiConversationDepositRow.id.in_([row.id for row in rows])))
|
||||
await session.commit()
|
||||
return deleted
|
||||
+3
@@ -42,6 +42,7 @@ from deerflow.persistence.embed_sessions.model import EmbedSessionRow
|
||||
from deerflow.persistence.feedback.model import FeedbackRow
|
||||
from deerflow.persistence.light_apps.model import LightAppRow
|
||||
from deerflow.persistence.llm_metrics.model import LlmCallMetricRow
|
||||
from deerflow.persistence.llmwiki.model import LlmWikiConversationDepositRow, LlmWikiKnowledgeBaseRow
|
||||
from deerflow.persistence.memory_config.model import UserMemoryConfigRow
|
||||
from deerflow.persistence.menu_overrides.model import MenuOverrideRow
|
||||
from deerflow.persistence.models.run_event import RunEventRow
|
||||
@@ -108,6 +109,8 @@ __all__ = [
|
||||
"KnowledgeTagRow",
|
||||
"LightAppRow",
|
||||
"LlmCallMetricRow",
|
||||
"LlmWikiKnowledgeBaseRow",
|
||||
"LlmWikiConversationDepositRow",
|
||||
"MenuOverrideRow",
|
||||
"NotificationRow",
|
||||
"ParallelAgentTaskRow",
|
||||
|
||||
@@ -13,7 +13,6 @@ from deerflow.config.extensions_config import (
|
||||
get_extensions_config,
|
||||
)
|
||||
|
||||
|
||||
_SKILL_PATH_RE = re.compile(r"[\\/]skills[\\/](?:public|custom)[\\/]([^\\/]+)(?:[\\/]|$)")
|
||||
|
||||
|
||||
@@ -27,7 +26,11 @@ def build_reference_batches(messages: list[Any]) -> list[dict[str, Any]]:
|
||||
if _message_type(messages[cursor]) == "human":
|
||||
end = cursor
|
||||
break
|
||||
sources = _collect_sources(messages[index + 1 : end])
|
||||
preloaded_sources = _collect_preloaded_sources(message, start_index=1)
|
||||
sources = [
|
||||
*preloaded_sources,
|
||||
*_collect_sources(messages[index + 1 : end], start_index=len(preloaded_sources) + 1),
|
||||
]
|
||||
if sources:
|
||||
anchor = _find_reference_batch_anchor(messages, index + 1, end)
|
||||
message_id = _message_id(anchor) or _message_id(message) or str(index)
|
||||
@@ -46,7 +49,11 @@ def build_latest_reference_batch(messages: list[Any]) -> dict[str, Any] | None:
|
||||
human_index = _find_latest_human_index(messages)
|
||||
if human_index is None:
|
||||
return None
|
||||
sources = _collect_sources(messages[human_index + 1 :])
|
||||
preloaded_sources = _collect_preloaded_sources(messages[human_index], start_index=1)
|
||||
sources = [
|
||||
*preloaded_sources,
|
||||
*_collect_sources(messages[human_index + 1 :], start_index=len(preloaded_sources) + 1),
|
||||
]
|
||||
if not sources:
|
||||
return None
|
||||
assistant = _find_reference_batch_anchor(messages, human_index + 1, len(messages))
|
||||
@@ -365,7 +372,7 @@ def _source_debug_item(source: dict[str, Any]) -> dict[str, Any]:
|
||||
}
|
||||
|
||||
|
||||
def _collect_sources(messages: list[Any]) -> list[dict[str, Any]]:
|
||||
def _collect_sources(messages: list[Any], *, start_index: int = 1) -> list[dict[str, Any]]:
|
||||
sources: list[dict[str, Any]] = []
|
||||
pending_tool_contexts: list[dict[str, Any]] = []
|
||||
tool_contexts_by_id: dict[str, dict[str, Any]] = {}
|
||||
@@ -393,7 +400,7 @@ def _collect_sources(messages: list[Any]) -> list[dict[str, Any]]:
|
||||
)
|
||||
if configured_sources is not None:
|
||||
for normalized in configured_sources:
|
||||
normalized["index"] = len(sources) + 1
|
||||
normalized["index"] = start_index + len(sources)
|
||||
sources.append(normalized)
|
||||
continue
|
||||
for raw_item in _first_list(payload):
|
||||
@@ -405,11 +412,37 @@ def _collect_sources(messages: list[Any]) -> list[dict[str, Any]]:
|
||||
)
|
||||
if not normalized:
|
||||
continue
|
||||
normalized["index"] = len(sources) + 1
|
||||
normalized["index"] = start_index + len(sources)
|
||||
sources.append(normalized)
|
||||
return sources
|
||||
|
||||
|
||||
def _collect_preloaded_sources(message: Any, *, start_index: int = 1) -> list[dict[str, Any]]:
|
||||
extra = _message_additional_kwargs(message)
|
||||
raw_sources = extra.get("llmwiki_reference_sources")
|
||||
if not isinstance(raw_sources, list):
|
||||
return []
|
||||
sources: list[dict[str, Any]] = []
|
||||
for raw in raw_sources:
|
||||
if not isinstance(raw, dict):
|
||||
continue
|
||||
source = dict(raw)
|
||||
index = start_index + len(sources)
|
||||
source["index"] = index
|
||||
source.setdefault("skillName", "llmwiki_search")
|
||||
source.setdefault("mode", "citation")
|
||||
source.setdefault("type", "llmwiki")
|
||||
title = str(source.get("title") or "").strip()
|
||||
if not title:
|
||||
source["title"] = f"LLMWiki 片段 {index}"
|
||||
snippet = str(source.get("snippet") or source.get("content") or "").strip()
|
||||
source["snippet"] = snippet[:360]
|
||||
if "raw" not in source or not isinstance(source.get("raw"), dict):
|
||||
source["raw"] = {key: value for key, value in source.items() if key != "raw"}
|
||||
sources.append(source)
|
||||
return sources
|
||||
|
||||
|
||||
def _collect_configured_sources(
|
||||
payload: Any,
|
||||
fallback_index: int,
|
||||
@@ -486,6 +519,14 @@ def _message_content(message: Any) -> Any:
|
||||
return message.get("content") if isinstance(message, dict) else getattr(message, "content", None)
|
||||
|
||||
|
||||
def _message_additional_kwargs(message: Any) -> dict[str, Any]:
|
||||
if isinstance(message, dict):
|
||||
value = message.get("additional_kwargs") or message.get("additionalKwargs")
|
||||
else:
|
||||
value = getattr(message, "additional_kwargs", None)
|
||||
return value if isinstance(value, dict) else {}
|
||||
|
||||
|
||||
def _message_name(message: Any) -> str:
|
||||
if isinstance(message, dict):
|
||||
return str(message.get("name") or (message.get("additional_kwargs") or {}).get("name") or "tool")
|
||||
|
||||
+132
@@ -0,0 +1,132 @@
|
||||
"""Authorized read-only WeKnora retrieval for DeerFlow Q&A."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Mapping
|
||||
from typing import Any
|
||||
from urllib.parse import urlencode
|
||||
|
||||
from langchain.tools import ToolRuntime, tool
|
||||
from langgraph.typing import ContextT
|
||||
|
||||
from deerflow.agents.thread_state import ThreadState
|
||||
from deerflow.config.agents_config import load_agent_config
|
||||
from deerflow.integrations.weknora.client import WeKnoraError
|
||||
from deerflow.integrations.weknora.runtime import build_weknora_client, get_llmwiki_store, get_resolved_llmwiki_runtime
|
||||
from deerflow.runtime.user_context import get_current_user, get_effective_user_id
|
||||
|
||||
|
||||
def selected_knowledge_base_ids(context: Mapping[str, Any] | None) -> list[str] | None:
|
||||
"""Read the conversation selection; ``None`` means legacy/no selection key."""
|
||||
|
||||
if not context or "llmwiki_knowledge_base_ids" not in context:
|
||||
return None
|
||||
raw = context.get("llmwiki_knowledge_base_ids")
|
||||
if not isinstance(raw, list):
|
||||
return []
|
||||
return list(dict.fromkeys(str(item).strip() for item in raw if str(item).strip()))[:100]
|
||||
|
||||
|
||||
def _agent_allowed_ids(context: Mapping[str, Any] | None) -> list[str] | None:
|
||||
agent_id = str((context or {}).get("agent_id") or "").strip()
|
||||
if not agent_id or agent_id in {"default", "lead_agent"}:
|
||||
return None
|
||||
try:
|
||||
config = load_agent_config(agent_id)
|
||||
except (FileNotFoundError, ValueError):
|
||||
return []
|
||||
return list(config.llmwiki_knowledge_base_ids or []) if config else []
|
||||
|
||||
|
||||
@tool("llmwiki_search", parse_docstring=True)
|
||||
async def llmwiki_search_tool(
|
||||
query: str,
|
||||
runtime: ToolRuntime[ContextT, ThreadState],
|
||||
) -> str:
|
||||
"""Search the LLMWiki knowledge selected for this conversation.
|
||||
|
||||
Use this tool when an answer can benefit from the user's managed knowledge
|
||||
bases or the shared public LLMWiki. It only retrieves passages; it cannot
|
||||
upload, edit, delete, publish or change permissions.
|
||||
|
||||
Args:
|
||||
query: Focused natural-language retrieval query.
|
||||
"""
|
||||
|
||||
resolved = get_resolved_llmwiki_runtime()
|
||||
if not resolved.weknora_enabled:
|
||||
return "LLMWiki retrieval is unavailable because WeKnora mode is not enabled."
|
||||
|
||||
store = get_llmwiki_store()
|
||||
user_id = get_effective_user_id()
|
||||
is_admin = getattr(get_current_user(), "system_role", None) == "admin"
|
||||
context = runtime.context if runtime.context else {}
|
||||
selected_ids = selected_knowledge_base_ids(context)
|
||||
allowed_ids = _agent_allowed_ids(context)
|
||||
if selected_ids is None:
|
||||
selected_ids = allowed_ids
|
||||
elif allowed_ids is not None and not set(selected_ids).issubset(set(allowed_ids)):
|
||||
return "The conversation requested knowledge outside this agent's configured LLMWiki scope."
|
||||
|
||||
if selected_ids == []:
|
||||
return "No LLMWiki knowledge base is selected for this conversation."
|
||||
if selected_ids is not None:
|
||||
rows = []
|
||||
for mapping_id in selected_ids:
|
||||
row = await store.get_authorized(
|
||||
mapping_id,
|
||||
user_id,
|
||||
write=False,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
if row is None:
|
||||
return f"Knowledge base {mapping_id!r} is not available to the current user."
|
||||
rows.append(row)
|
||||
else:
|
||||
rows = await store.list_visible(user_id, scope="all", is_admin=is_admin)
|
||||
if not rows:
|
||||
return "No authorized LLMWiki knowledge bases are available."
|
||||
|
||||
try:
|
||||
results = await build_weknora_client(resolved).search(
|
||||
query.strip(),
|
||||
[str(row["weknora_id"]) for row in rows],
|
||||
)
|
||||
except (RuntimeError, WeKnoraError) as exc:
|
||||
return f"LLMWiki retrieval failed: {exc}"
|
||||
|
||||
local_by_remote = {str(row["weknora_id"]): row for row in rows}
|
||||
citations = []
|
||||
for result in results:
|
||||
mapping = local_by_remote.get(str(result.get("knowledge_base_id") or ""))
|
||||
if mapping is None:
|
||||
continue
|
||||
params = urlencode(
|
||||
{
|
||||
"knowledge_base_id": mapping["id"],
|
||||
"chunk_id": result.get("chunk_id") or "",
|
||||
}
|
||||
)
|
||||
citations.append(
|
||||
{
|
||||
"title": result.get("title") or result.get("filename") or mapping.get("name") or "LLMWiki source",
|
||||
"content": result.get("content") or "",
|
||||
"score": result.get("score") or 0,
|
||||
"knowledge_base_id": mapping["id"],
|
||||
"knowledge_base_name": mapping.get("name") or "",
|
||||
"source": mapping.get("name") or "LLMWiki",
|
||||
"knowledge_id": result.get("knowledge_id") or "",
|
||||
"chunk_id": result.get("chunk_id") or "",
|
||||
"url": f"/weknora-source-preview?{params}",
|
||||
}
|
||||
)
|
||||
return json.dumps(
|
||||
{
|
||||
"query": query,
|
||||
"selection": [row["id"] for row in rows],
|
||||
"knowledge_base_count": len(rows),
|
||||
"results": citations,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
@@ -75,7 +75,11 @@ def get_available_tools(
|
||||
List of available tools.
|
||||
"""
|
||||
config = app_config or get_app_config()
|
||||
tool_configs = [tool for tool in config.tools if groups is None or tool.group in groups]
|
||||
selected_tool_configs = [tool for tool in config.tools if groups is None or tool.group in groups]
|
||||
disabled_tool_names = [tool.name for tool in selected_tool_configs if not tool.enabled]
|
||||
if disabled_tool_names:
|
||||
logger.info("Skipping disabled configured tool(s): %s", disabled_tool_names)
|
||||
tool_configs = [tool for tool in selected_tool_configs if tool.enabled]
|
||||
|
||||
browser_context_config = getattr(config, "browser_context", None)
|
||||
if not getattr(browser_context_config, "enabled", False):
|
||||
@@ -154,9 +158,21 @@ def get_available_tools(
|
||||
# read_peer_delivery —— 圆桌席位「按需读取前序席位完整交付」的只读工具(从 run context
|
||||
# 取网关注入的全文,非圆桌场景为空 → 工具自述「无」)。始终提供(只读、无副作用);圆桌
|
||||
# 总控/报告角色由 run policy 显式排除,避免干扰它们。
|
||||
from deerflow.tools.builtins.roundtable_peers_tool import read_peer_delivery_tool
|
||||
|
||||
builtin_tools.append(read_peer_delivery_tool)
|
||||
from deerflow.tools.builtins.roundtable_peers_tool import read_peer_delivery_tool
|
||||
|
||||
builtin_tools.append(read_peer_delivery_tool)
|
||||
|
||||
# Conditional LLMWiki mode: the tool is absent when no effective WeKnora
|
||||
# address is configured, keeping legacy deployments behavior-identical.
|
||||
try:
|
||||
from deerflow.integrations.weknora.runtime import get_resolved_llmwiki_runtime
|
||||
|
||||
if get_resolved_llmwiki_runtime(config).weknora_enabled:
|
||||
from deerflow.tools.builtins.llmwiki_search_tool import llmwiki_search_tool
|
||||
|
||||
builtin_tools.append(llmwiki_search_tool)
|
||||
except Exception: # noqa: BLE001 - configuration/store failures must not block agent creation
|
||||
logger.debug("Failed to evaluate the conditional LLMWiki search tool", exc_info=True)
|
||||
|
||||
# skill_manage 能创建/修改/删除技能,属于"自我进化"写能力,按 skill_evolution.enabled
|
||||
# 门控。该开关用户可在 per-user 配置里覆盖,所以按"生效配置(含 per-user 覆盖)"判断 ——
|
||||
|
||||
+229
@@ -0,0 +1,229 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from deerflow.community.configurable_search.tools import (
|
||||
ConfigurableSearchSettings,
|
||||
execute_search,
|
||||
)
|
||||
from deerflow.config.app_config import AppConfig
|
||||
from deerflow.config.tool_config import ToolConfig
|
||||
from deerflow.tools.tools import get_available_tools
|
||||
|
||||
|
||||
def _settings(**overrides: object) -> ConfigurableSearchSettings:
|
||||
values = {
|
||||
"enabled": True,
|
||||
"endpoint": "https://search.example.test/searchName",
|
||||
"payload": {"query": "configured value", "xxx": "xxxx"},
|
||||
"verify_ssl": False,
|
||||
"timeout": 30.0,
|
||||
"max_results": 5,
|
||||
"result_url_template": "https://search.example.test/mindId/xxx?recUuid={recUuid}",
|
||||
}
|
||||
values.update(overrides)
|
||||
return ConfigurableSearchSettings(**values) # type: ignore[arg-type]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_configurable_search_overrides_query_and_normalizes_nullable_results() -> None:
|
||||
requests: list[httpx.Request] = []
|
||||
|
||||
async def handler(request: httpx.Request) -> httpx.Response:
|
||||
requests.append(request)
|
||||
assert request.method == "POST"
|
||||
assert json.loads(request.content) == {"query": "测试检索词", "xxx": "xxxx"}
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"results": [
|
||||
{
|
||||
"recUuid": "record/123",
|
||||
"content": "检索内容",
|
||||
"title": "标题",
|
||||
"url": "https://original.example.test/one",
|
||||
},
|
||||
{
|
||||
"recUuid": None,
|
||||
"content": None,
|
||||
"title": None,
|
||||
"url": "https://original.example.test/two",
|
||||
},
|
||||
"invalid row",
|
||||
]
|
||||
},
|
||||
)
|
||||
|
||||
settings = _settings()
|
||||
result = json.loads(
|
||||
await execute_search(
|
||||
" 测试检索词 ",
|
||||
settings,
|
||||
transport=httpx.MockTransport(handler),
|
||||
)
|
||||
)
|
||||
|
||||
assert len(requests) == 1
|
||||
assert settings.payload == {"query": "configured value", "xxx": "xxxx"}
|
||||
assert result == {
|
||||
"query": "测试检索词",
|
||||
"total_results": 2,
|
||||
"results": [
|
||||
{
|
||||
"recUuid": "record/123",
|
||||
"content": "检索内容",
|
||||
"title": "标题",
|
||||
"url": "https://search.example.test/mindId/xxx?recUuid=record%2F123",
|
||||
},
|
||||
{
|
||||
"recUuid": "",
|
||||
"content": "",
|
||||
"title": "",
|
||||
"url": "https://original.example.test/two",
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_configurable_search_handles_empty_results() -> None:
|
||||
async def handler(_: httpx.Request) -> httpx.Response:
|
||||
return httpx.Response(200, json={"results": []})
|
||||
|
||||
result = json.loads(
|
||||
await execute_search(
|
||||
"nothing",
|
||||
_settings(),
|
||||
transport=httpx.MockTransport(handler),
|
||||
)
|
||||
)
|
||||
|
||||
assert result == {"query": "nothing", "total_results": 0, "results": []}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
("response_factory", "expected_error"),
|
||||
[
|
||||
(lambda request: httpx.Response(503, request=request), "Search API returned HTTP 503"),
|
||||
(
|
||||
lambda request: httpx.Response(200, content=b"not-json", request=request),
|
||||
"Search API request failed: JSONDecodeError",
|
||||
),
|
||||
(
|
||||
lambda request: httpx.Response(200, json={"results": None}, request=request),
|
||||
"search API response field 'results' must be an array",
|
||||
),
|
||||
],
|
||||
)
|
||||
async def test_configurable_search_returns_safe_errors(
|
||||
response_factory,
|
||||
expected_error: str,
|
||||
) -> None:
|
||||
async def handler(request: httpx.Request) -> httpx.Response:
|
||||
return response_factory(request)
|
||||
|
||||
result = json.loads(
|
||||
await execute_search(
|
||||
"query",
|
||||
_settings(),
|
||||
transport=httpx.MockTransport(handler),
|
||||
)
|
||||
)
|
||||
|
||||
assert result["results"] == []
|
||||
assert result["total_results"] == 0
|
||||
assert result["error"] == expected_error
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_configurable_search_handles_timeout() -> None:
|
||||
async def handler(request: httpx.Request) -> httpx.Response:
|
||||
raise httpx.ReadTimeout("timed out", request=request)
|
||||
|
||||
result = json.loads(
|
||||
await execute_search(
|
||||
"query",
|
||||
_settings(),
|
||||
transport=httpx.MockTransport(handler),
|
||||
)
|
||||
)
|
||||
|
||||
assert result["error"] == "Search API request timed out"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_configurable_search_passes_verify_ssl_to_httpx(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
captured: dict[str, object] = {}
|
||||
|
||||
class FakeAsyncClient:
|
||||
def __init__(self, **kwargs: object) -> None:
|
||||
captured.update(kwargs)
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *_: object) -> None:
|
||||
return None
|
||||
|
||||
async def post(self, url: str, *, json: dict[str, object]) -> httpx.Response:
|
||||
request = httpx.Request("POST", url, json=json)
|
||||
return httpx.Response(200, json={"results": []}, request=request)
|
||||
|
||||
monkeypatch.setattr(
|
||||
"deerflow.community.configurable_search.tools.httpx.AsyncClient",
|
||||
FakeAsyncClient,
|
||||
)
|
||||
|
||||
await execute_search("query", _settings(verify_ssl=False))
|
||||
|
||||
assert captured["verify"] is False
|
||||
assert captured["timeout"] == 30.0
|
||||
|
||||
|
||||
def test_disabled_configured_tool_is_not_registered() -> None:
|
||||
config = AppConfig(
|
||||
sandbox={"use": "deerflow.sandbox.local:LocalSandboxProvider"},
|
||||
tools=[
|
||||
ToolConfig(
|
||||
name="web_search",
|
||||
group="web",
|
||||
enabled=False,
|
||||
use="deerflow.community.configurable_search.tools:web_search_tool",
|
||||
endpoint="https://search.example.test/searchName",
|
||||
)
|
||||
],
|
||||
)
|
||||
|
||||
available = get_available_tools(
|
||||
groups=["web"],
|
||||
include_mcp=False,
|
||||
app_config=config,
|
||||
)
|
||||
|
||||
assert "web_search" not in {item.name for item in available}
|
||||
|
||||
config.tools[0].enabled = True
|
||||
enabled_tools = get_available_tools(
|
||||
groups=["web"],
|
||||
include_mcp=False,
|
||||
app_config=config,
|
||||
)
|
||||
|
||||
assert "web_search" in {item.name for item in enabled_tools}
|
||||
|
||||
|
||||
def test_config_validation_requires_rec_uuid_url_placeholder() -> None:
|
||||
config = ToolConfig(
|
||||
name="web_search",
|
||||
group="web",
|
||||
use="deerflow.community.configurable_search.tools:web_search_tool",
|
||||
endpoint="https://search.example.test/searchName",
|
||||
result_url_template="https://search.example.test/mindId/xxx",
|
||||
)
|
||||
|
||||
with pytest.raises(ValueError, match="must contain"):
|
||||
ConfigurableSearchSettings.from_tool_config(config)
|
||||
@@ -0,0 +1,100 @@
|
||||
"""Regression coverage for the unauthenticated Gateway health probes."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from app.gateway import auth_middleware
|
||||
from app.gateway.app import app as gateway_app
|
||||
from app.gateway.auth_middleware import PUBLIC_HEALTH_PATHS, AuthMiddleware
|
||||
|
||||
|
||||
def _health_only_app() -> FastAPI:
|
||||
app = FastAPI()
|
||||
|
||||
async def health_check() -> dict[str, str]:
|
||||
return {"status": "healthy", "service": "deer-flow-gateway"}
|
||||
|
||||
for path in PUBLIC_HEALTH_PATHS:
|
||||
app.add_api_route(path, health_check, methods=["GET", "HEAD"])
|
||||
app.add_middleware(AuthMiddleware)
|
||||
return app
|
||||
|
||||
|
||||
@pytest.mark.parametrize("path", sorted(PUBLIC_HEALTH_PATHS))
|
||||
@pytest.mark.parametrize("method", ["GET", "HEAD"])
|
||||
def test_health_probe_is_public_when_auth_is_enabled(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
path: str,
|
||||
method: str,
|
||||
) -> None:
|
||||
monkeypatch.setattr(auth_middleware, "is_auth_disabled", lambda: False)
|
||||
|
||||
with TestClient(_health_only_app()) as client:
|
||||
client.cookies.set("access_token", "invalid-token")
|
||||
response = client.request(
|
||||
method,
|
||||
path,
|
||||
headers={"Authorization": "Bearer invalid-token"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
if method == "GET":
|
||||
assert response.json() == {"status": "healthy", "service": "deer-flow-gateway"}
|
||||
else:
|
||||
assert response.content == b""
|
||||
|
||||
|
||||
@pytest.mark.parametrize("path", sorted(PUBLIC_HEALTH_PATHS))
|
||||
def test_health_probe_allows_a_trailing_slash(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
path: str,
|
||||
) -> None:
|
||||
monkeypatch.setattr(auth_middleware, "is_auth_disabled", lambda: False)
|
||||
|
||||
with TestClient(_health_only_app(), follow_redirects=False) as client:
|
||||
response = client.get(f"{path}/")
|
||||
|
||||
# FastAPI redirects to the canonical route; importantly, auth must not turn
|
||||
# the request into a 401 before routing can perform that redirect.
|
||||
assert response.status_code == 307
|
||||
|
||||
|
||||
@pytest.mark.parametrize("path", sorted(PUBLIC_HEALTH_PATHS))
|
||||
def test_health_probe_is_public_behind_a_reverse_proxy_prefix(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
path: str,
|
||||
) -> None:
|
||||
monkeypatch.setattr(auth_middleware, "is_auth_disabled", lambda: False)
|
||||
|
||||
with TestClient(_health_only_app(), root_path="/deerflow") as client:
|
||||
response = client.get(f"/deerflow{path}")
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
|
||||
def test_gateway_registers_every_public_health_path_for_get_and_head() -> None:
|
||||
methods_by_path = {
|
||||
route.path: route.methods
|
||||
for route in gateway_app.routes
|
||||
if hasattr(route, "methods")
|
||||
}
|
||||
|
||||
for path in PUBLIC_HEALTH_PATHS:
|
||||
assert {"GET", "HEAD"} <= methods_by_path[path]
|
||||
|
||||
|
||||
def test_non_health_route_still_requires_auth(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setattr(auth_middleware, "is_auth_disabled", lambda: False)
|
||||
app = _health_only_app()
|
||||
|
||||
@app.get("/private")
|
||||
async def private() -> dict[str, bool]:
|
||||
return {"ok": True}
|
||||
|
||||
with TestClient(app) as client:
|
||||
response = client.get("/private")
|
||||
|
||||
assert response.status_code == 401
|
||||
@@ -0,0 +1,418 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from fastapi import HTTPException
|
||||
from starlette.datastructures import Headers, QueryParams
|
||||
|
||||
from deerflow.config.agents_config import AgentConfig
|
||||
from deerflow.config.llmwiki_config import (
|
||||
LlmWikiConfig,
|
||||
LlmWikiRuntimeOverride,
|
||||
resolve_llmwiki_runtime,
|
||||
)
|
||||
from deerflow.integrations.weknora.client import WeKnoraClient, WeKnoraError
|
||||
from deerflow.persistence.llmwiki import MemoryLlmWikiStore
|
||||
from deerflow.tools.builtins.llmwiki_search_tool import selected_knowledge_base_ids
|
||||
|
||||
|
||||
def test_empty_weknora_address_keeps_legacy_mode() -> None:
|
||||
runtime = resolve_llmwiki_runtime(LlmWikiConfig())
|
||||
|
||||
assert runtime.provider == "legacy"
|
||||
assert runtime.api_base_url == ""
|
||||
|
||||
|
||||
def test_configured_weknora_address_enables_weknora_mode() -> None:
|
||||
config = LlmWikiConfig.model_validate(
|
||||
{
|
||||
"weknora": {
|
||||
"api_base_url": "http://weknora-app:8080/",
|
||||
"web_base_url": "https://weknora.example.test/",
|
||||
"admin_email": "admin@example.test",
|
||||
"admin_password": "secret",
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
runtime = resolve_llmwiki_runtime(config)
|
||||
|
||||
assert runtime.provider == "weknora"
|
||||
assert runtime.api_base_url == "http://weknora-app:8080"
|
||||
assert runtime.web_base_url == "https://weknora.example.test"
|
||||
assert runtime.admin_email == "admin@example.test"
|
||||
assert runtime.admin_password == "secret"
|
||||
|
||||
|
||||
def test_runtime_override_can_force_legacy_or_reset_to_file() -> None:
|
||||
config = LlmWikiConfig.model_validate({"weknora": {"api_base_url": "http://weknora:8080"}})
|
||||
|
||||
forced_legacy = resolve_llmwiki_runtime(
|
||||
config,
|
||||
LlmWikiRuntimeOverride(enabled=True, api_base_url=""),
|
||||
)
|
||||
reset_to_file = resolve_llmwiki_runtime(
|
||||
config,
|
||||
LlmWikiRuntimeOverride(enabled=False, api_base_url=""),
|
||||
)
|
||||
|
||||
assert forced_legacy.provider == "legacy"
|
||||
assert reset_to_file.provider == "weknora"
|
||||
|
||||
|
||||
def test_weknora_url_rejects_credentials_and_non_http_schemes() -> None:
|
||||
with pytest.raises(ValueError):
|
||||
LlmWikiConfig.model_validate({"weknora": {"api_base_url": "ftp://weknora.local"}})
|
||||
with pytest.raises(ValueError):
|
||||
LlmWikiConfig.model_validate({"weknora": {"api_base_url": "http://user:pass@weknora.local"}})
|
||||
|
||||
|
||||
def test_agent_config_tracks_allowed_llmwiki_knowledge_bases() -> None:
|
||||
config = AgentConfig(
|
||||
name="Research agent",
|
||||
llmwiki_knowledge_base_ids=["kb-one", "kb-two", "kb-one"],
|
||||
)
|
||||
|
||||
assert config.llmwiki_knowledge_base_ids == ["kb-one", "kb-two"]
|
||||
|
||||
|
||||
def test_conversation_selection_is_deduplicated_and_absence_is_distinct() -> None:
|
||||
assert selected_knowledge_base_ids({}) is None
|
||||
assert selected_knowledge_base_ids({"llmwiki_knowledge_base_ids": []}) == []
|
||||
assert selected_knowledge_base_ids({"llmwiki_knowledge_base_ids": ["kb-one", "kb-one", "", "kb-two"]}) == ["kb-one", "kb-two"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_weknora_client_sends_no_credentials_and_normalizes_search() -> None:
|
||||
requests: list[httpx.Request] = []
|
||||
|
||||
async def handler(request: httpx.Request) -> httpx.Response:
|
||||
requests.append(request)
|
||||
assert "X-API-Key" not in request.headers
|
||||
assert "Authorization" not in request.headers
|
||||
assert json.loads(request.content) == {
|
||||
"query": "deployment guide",
|
||||
"knowledge_base_ids": ["kb-one", "kb-two"],
|
||||
}
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"success": True,
|
||||
"data": [
|
||||
{
|
||||
"id": "chunk-one",
|
||||
"content": "matched passage",
|
||||
"knowledge_id": "doc-one",
|
||||
"knowledge_base_id": "kb-one",
|
||||
"knowledge_title": "Guide",
|
||||
"score": 0.93,
|
||||
}
|
||||
],
|
||||
},
|
||||
)
|
||||
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as http_client:
|
||||
client = WeKnoraClient(
|
||||
base_url="http://weknora.local:8080",
|
||||
http_client=http_client,
|
||||
)
|
||||
results = await client.search("deployment guide", ["kb-one", "kb-two"])
|
||||
|
||||
assert requests[0].url.path == "/api/v1/knowledge-search"
|
||||
assert results[0]["chunk_id"] == "chunk-one"
|
||||
assert results[0]["title"] == "Guide"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_weknora_client_resolves_default_embedding_model_inside_weknora() -> None:
|
||||
async def handler(request: httpx.Request) -> httpx.Response:
|
||||
if request.method == "GET" and request.url.path == "/api/v1/models":
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"success": True,
|
||||
"data": [
|
||||
{"id": "embed-default", "type": "Embedding", "is_default": True},
|
||||
],
|
||||
},
|
||||
)
|
||||
assert request.method == "POST"
|
||||
assert request.url.path == "/api/v1/knowledge-bases"
|
||||
assert json.loads(request.content)["embedding_model_id"] == "embed-default"
|
||||
return httpx.Response(201, json={"success": True, "data": {"id": "kb-created", "name": "Docs"}})
|
||||
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as http_client:
|
||||
client = WeKnoraClient(
|
||||
base_url="http://weknora.local:8080",
|
||||
http_client=http_client,
|
||||
)
|
||||
created = await client.create_knowledge_base(name="Docs")
|
||||
|
||||
assert created["id"] == "kb-created"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_weknora_client_creates_wiki_with_weknora_owned_models() -> None:
|
||||
model_reads = 0
|
||||
|
||||
async def handler(request: httpx.Request) -> httpx.Response:
|
||||
nonlocal model_reads
|
||||
if request.method == "GET" and request.url.path == "/api/v1/models":
|
||||
model_reads += 1
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"data": [
|
||||
{"id": "embed-one", "type": "Embedding", "is_default": True},
|
||||
{"id": "qa-one", "type": "KnowledgeQA", "status": "active"},
|
||||
]
|
||||
},
|
||||
)
|
||||
payload = json.loads(request.content)
|
||||
assert payload["embedding_model_id"] == "embed-one"
|
||||
assert payload["summary_model_id"] == "qa-one"
|
||||
assert payload["indexing_strategy"] == {
|
||||
"vector_enabled": True,
|
||||
"keyword_enabled": True,
|
||||
"wiki_enabled": True,
|
||||
"graph_enabled": False,
|
||||
}
|
||||
assert payload["wiki_config"]["synthesis_model_id"] == "qa-one"
|
||||
return httpx.Response(201, json={"data": {"id": "wiki-one", "name": "Wiki"}})
|
||||
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as http_client:
|
||||
client = WeKnoraClient(base_url="http://weknora.local:8080", http_client=http_client)
|
||||
created = await client.create_knowledge_base(name="Wiki", wiki_enabled=True)
|
||||
|
||||
assert created["id"] == "wiki-one"
|
||||
assert model_reads == 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_weknora_client_imports_url_manual_content_and_lists_wiki_pages() -> None:
|
||||
async def handler(request: httpx.Request) -> httpx.Response:
|
||||
if request.url.path.endswith("/knowledge/url"):
|
||||
assert json.loads(request.content) == {"url": "https://example.test/guide"}
|
||||
return httpx.Response(201, json={"data": {"id": "url-one"}})
|
||||
if request.url.path.endswith("/knowledge/manual"):
|
||||
assert json.loads(request.content) == {
|
||||
"title": "Guide",
|
||||
"content": "# Guide",
|
||||
"status": "publish",
|
||||
}
|
||||
return httpx.Response(201, json={"data": {"id": "manual-one"}})
|
||||
assert request.url.path == "/api/v1/knowledgebase/kb-one/wiki/pages"
|
||||
assert request.url.params["query"] == "guide"
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={"pages": [{"slug": "guide", "title": "Guide"}], "total": 1, "page": 1, "page_size": 50},
|
||||
)
|
||||
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as http_client:
|
||||
client = WeKnoraClient(base_url="http://weknora.local:8080", http_client=http_client)
|
||||
await client.import_document_url("kb-one", url="https://example.test/guide")
|
||||
await client.create_manual_document("kb-one", title="Guide", content="# Guide")
|
||||
pages = await client.list_wiki_pages("kb-one", query="guide")
|
||||
|
||||
assert pages["total"] == 1
|
||||
assert pages["pages"][0]["slug"] == "guide"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_weknora_client_reads_document_chunks_and_wiki_graph() -> None:
|
||||
async def handler(request: httpx.Request) -> httpx.Response:
|
||||
if request.url.path == "/api/v1/chunks/doc-one":
|
||||
assert request.url.params["page"] == "2"
|
||||
assert request.url.params["page_size"] == "10"
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"success": True,
|
||||
"data": [{"id": "chunk-one", "content": "parsed content", "chunk_index": 10}],
|
||||
"total": 31,
|
||||
"page": 2,
|
||||
"page_size": 10,
|
||||
},
|
||||
)
|
||||
assert request.url.path == "/api/v1/knowledgebase/kb-one/wiki/graph"
|
||||
assert request.url.params["mode"] == "overview"
|
||||
assert request.url.params["limit"] == "50"
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"success": True,
|
||||
"data": {
|
||||
"nodes": [{"slug": "entity/one", "title": "Entity One", "page_type": "entity"}],
|
||||
"edges": [],
|
||||
"meta": {"mode": "overview", "total": 1, "returned": 1},
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as http_client:
|
||||
client = WeKnoraClient(base_url="http://weknora.local:8080", http_client=http_client)
|
||||
chunks = await client.list_chunks("doc-one", page=2, page_size=10)
|
||||
graph = await client.get_wiki_graph("kb-one", limit=50)
|
||||
|
||||
assert chunks["total"] == 31
|
||||
assert chunks["items"][0]["content"] == "parsed content"
|
||||
assert graph["nodes"][0]["slug"] == "entity/one"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_weknora_client_proxies_original_document_preview_without_credentials() -> None:
|
||||
async def handler(request: httpx.Request) -> httpx.Response:
|
||||
assert request.url.path == "/api/v1/knowledge/doc-one/preview"
|
||||
assert "X-API-Key" not in request.headers
|
||||
assert "Authorization" not in request.headers
|
||||
return httpx.Response(200, content=b"preview", headers={"Content-Type": "text/plain; charset=utf-8"})
|
||||
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as http_client:
|
||||
client = WeKnoraClient(base_url="http://weknora.local:8080", http_client=http_client)
|
||||
content, media_type = await client.get_document_preview("doc-one")
|
||||
|
||||
assert content == b"preview"
|
||||
assert media_type == "text/plain"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_memory_store_enforces_owner_and_public_visibility() -> None:
|
||||
store = MemoryLlmWikiStore()
|
||||
created = await store.create_mapping(
|
||||
weknora_id="wk-private",
|
||||
owner_user_id="user-a",
|
||||
name="Private notes",
|
||||
description="",
|
||||
kb_type="document",
|
||||
)
|
||||
|
||||
assert [row["id"] for row in await store.list_visible("user-a", scope="personal")] == [created["id"]]
|
||||
assert await store.list_visible("user-b", scope="personal") == []
|
||||
assert await store.get_authorized(created["id"], "user-b", write=False, is_admin=False) is None
|
||||
|
||||
published = await store.request_publish(created["id"], "user-a", is_admin=False)
|
||||
assert published and published["publication_status"] == "published"
|
||||
assert [row["id"] for row in await store.list_visible("user-b", scope="public")] == [created["id"]]
|
||||
assert await store.get_authorized(created["id"], "user-b", write=False, is_admin=False) is not None
|
||||
assert await store.get_authorized(created["id"], "user-b", write=True, is_admin=False) is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_delete_cleans_owner_mapping_when_remote_is_already_missing(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
from app.gateway.routers import llmwiki as llmwiki_router
|
||||
|
||||
deleted: list[str] = []
|
||||
|
||||
class MissingClient:
|
||||
async def delete_knowledge_base(self, _: str) -> None:
|
||||
raise WeKnoraError("WeKnora resource was not found", status_code=404)
|
||||
|
||||
class Store:
|
||||
async def delete_mapping(self, mapping_id: str) -> None:
|
||||
deleted.append(mapping_id)
|
||||
|
||||
async def authorized_mapping(*_: object, **__: object) -> tuple[dict[str, str], str, bool]:
|
||||
return {"weknora_id": "remote-missing"}, "owner", False
|
||||
|
||||
monkeypatch.setattr(llmwiki_router, "_authorized_mapping", authorized_mapping)
|
||||
monkeypatch.setattr(llmwiki_router, "_client_or_503", lambda: MissingClient())
|
||||
monkeypatch.setattr(llmwiki_router, "_store", lambda _: Store())
|
||||
|
||||
await llmwiki_router.delete_knowledge_base(object(), "mapping-one")
|
||||
|
||||
assert deleted == ["mapping-one"]
|
||||
|
||||
|
||||
def test_weknora_iframe_cookie_is_signed_and_short_lived() -> None:
|
||||
from app.gateway.routers import llmwiki as llmwiki_router
|
||||
|
||||
raw = llmwiki_router._sign_embed_payload(
|
||||
{
|
||||
"mapping_id": "local-kb",
|
||||
"weknora_id": "remote-kb",
|
||||
"user_id": "user-a",
|
||||
"is_admin": False,
|
||||
"can_write": True,
|
||||
"exp": 4_102_444_800,
|
||||
}
|
||||
)
|
||||
|
||||
ctx = llmwiki_router._verify_embed_cookie(raw)
|
||||
|
||||
assert ctx.mapping_id == "local-kb"
|
||||
assert ctx.weknora_id == "remote-kb"
|
||||
assert ctx.can_write is True
|
||||
|
||||
with pytest.raises(HTTPException):
|
||||
llmwiki_router._verify_embed_cookie(raw + "tampered")
|
||||
|
||||
|
||||
def test_weknora_iframe_filters_kb_list_to_current_mapping() -> None:
|
||||
from app.gateway.routers import llmwiki as llmwiki_router
|
||||
|
||||
ctx = llmwiki_router.WeKnoraEmbedContext(
|
||||
mapping_id="local-kb",
|
||||
weknora_id="remote-kb",
|
||||
user_id="user-a",
|
||||
is_admin=False,
|
||||
can_write=False,
|
||||
exp=4_102_444_800,
|
||||
)
|
||||
response = httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"success": True,
|
||||
"data": [
|
||||
{"id": "remote-kb", "name": "Allowed"},
|
||||
{"id": "other-kb", "name": "Blocked"},
|
||||
],
|
||||
},
|
||||
)
|
||||
|
||||
filtered = llmwiki_router._filter_knowledge_base_list_response(response, ctx)
|
||||
|
||||
assert filtered is not None
|
||||
payload = json.loads(filtered)
|
||||
assert payload["data"] == [{"id": "remote-kb", "name": "Allowed"}]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_weknora_iframe_proxy_blocks_other_knowledge_bases() -> None:
|
||||
from app.gateway.routers import llmwiki as llmwiki_router
|
||||
|
||||
class FakeRequest:
|
||||
method = "GET"
|
||||
headers = Headers({})
|
||||
query_params = QueryParams("")
|
||||
|
||||
ctx = llmwiki_router.WeKnoraEmbedContext(
|
||||
mapping_id="local-kb",
|
||||
weknora_id="remote-kb",
|
||||
user_id="user-a",
|
||||
is_admin=False,
|
||||
can_write=False,
|
||||
exp=4_102_444_800,
|
||||
)
|
||||
|
||||
await llmwiki_router._authorize_weknora_api_proxy(
|
||||
FakeRequest(),
|
||||
ctx,
|
||||
object(),
|
||||
"/api/v1/knowledge-bases/remote-kb",
|
||||
b"",
|
||||
"token",
|
||||
)
|
||||
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
await llmwiki_router._authorize_weknora_api_proxy(
|
||||
FakeRequest(),
|
||||
ctx,
|
||||
object(),
|
||||
"/api/v1/knowledge-bases/other-kb",
|
||||
b"",
|
||||
"token",
|
||||
)
|
||||
|
||||
assert exc_info.value.status_code == 403
|
||||
@@ -1,8 +1,9 @@
|
||||
# Direct runtime packages layered into the prebuilt offline backend image.
|
||||
# Keep these three pins synchronized with start_backend.sh.
|
||||
# Keep these four pins synchronized with start_backend.sh.
|
||||
croniter==6.2.2
|
||||
hindsight-client==0.6.2
|
||||
redis==5.2.1
|
||||
aiomysql==0.3.2
|
||||
|
||||
# Fully pinned dependency closure for CPython 3.12 on Linux. Keeping the
|
||||
# transitive packages here makes a newly generated wheelhouse reproducible.
|
||||
@@ -25,3 +26,4 @@ typing-extensions==4.16.0
|
||||
typing-inspection==0.4.2
|
||||
urllib3==2.7.0
|
||||
yarl==1.24.5
|
||||
PyMySQL==1.2.0
|
||||
@@ -59,6 +59,8 @@ echo " Python/ABI: ${TARGET_PYTHON_VERSION}/${TARGET_ABI}"
|
||||
require_wheel 'croniter-6.2.2-*.whl'
|
||||
require_wheel 'hindsight_client-0.6.2-*.whl'
|
||||
require_wheel 'redis-5.2.1-*.whl'
|
||||
require_wheel 'aiomysql-0.3.2-*.whl'
|
||||
require_wheel 'pymysql-1.2.0-*.whl'
|
||||
|
||||
# Resolve the same requirements a second time using only wheelhouse. This
|
||||
# catches a missing transitive dependency before the bundle reaches offline.
|
||||
|
||||
@@ -130,6 +130,12 @@ if [ "${HINDSIGHT_INSTALL_DEPS:-1}" != "0" ]; then
|
||||
uv pip install --python "${PYTHON_BIN}" --no-index --find-links /app/wheelhouse ${HINDSIGHT_PIP_PACKAGES:-hindsight-client==0.6.2}
|
||||
}
|
||||
fi
|
||||
if [ "${AIOMYSQL_INSTALL_DEPS:-1}" != "0" ]; then
|
||||
uv run --no-sync python -c "import aiomysql" >/dev/null 2>&1 || {
|
||||
PYTHON_BIN="$(uv run --no-sync python -c "import sys; print(sys.executable)")"
|
||||
uv pip install --python "${PYTHON_BIN}" --no-index --find-links /app/wheelhouse ${AIOMYSQL_PIP_PACKAGES:-aiomysql==0.3.2}
|
||||
}
|
||||
fi
|
||||
uv run --no-sync python -c "import redis" >/dev/null 2>&1 || uv pip install --python "$(uv run --no-sync python -c "import sys; print(sys.executable)")" --no-index --find-links /app/wheelhouse ${REDIS_PIP_PACKAGES:-redis==5.2.1}
|
||||
PYTHONPATH=. uv run --no-sync python -m uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 --workers ${GATEWAY_WORKERS:-2}'
|
||||
|
||||
|
||||
Reference in new issue
Block a user