feat: add compact toolbox and public wiki vector search

This commit is contained in:
吉鑫升 committed 2026-08-26 14:40:57 +08:00
1 parent 9f29ecae9e
commit d35d075669
29 files changed
+757 -128

No files matched your search

+30 -8
View File
@@ -1024,10 +1024,14 @@ Both can be modified at runtime via Gateway API endpoints or `DeerFlowClient` me
The embedded detail proxy hides WeKnora's outer `.main > .aside_box` application sidebar and expands the route outlet, but deliberately keeps the Wiki page's internal directory/index sidebar. Do not add an HTML `sandbox` attribute to this trusted, permission-filtered detail iframe: a sandboxed ancestor causes Chromium's built-in Blob PDF viewer to render the browser-blocked page. The Gateway follows upstream redirects only for `/knowledge/{id}/preview` and `/knowledge-bases/{id}/files`, keeping PDF bytes and protected Wiki images inside the signed proxy instead of sending the browser to WeKnora object storage.
Native/private and public Wiki readers must rewrite protected Markdown image
sources to their mapping-scoped DeerFlow `/files?file_path=` endpoints and load
them as Blob URLs. The embedded guard also rewrites protected fetch/XHR values
and hydrates late `img[data-protected-src]` placeholders; never expose the
service token or relax the current-KB authorization to make images work.
sources to their mapping-scoped DeerFlow `/files?file_path=` endpoints and keep
real HTTP URLs on rendered image elements. The embedded guard also rewrites
protected fetch/XHR values and hydrates late `img[data-protected-src]`
placeholders with the signed iframe HTTP proxy URL. Observe both
`data-protected-src` and `src`, because WeKnora may replace the URL after first
paint. Never create/revoke a temporary object URL here: its lightbox reuses the
thumbnail `src` after click. Never expose the service token or relax the
current-KB authorization to make images work.
**Embedded native knowledge-base switching:** The original WeKnora breadcrumb picker remains available inside the detail iframe. A signed 15-minute embed cookie carries only the current DeerFlow user's visible mappings (their own plus published bases); the proxied `/api/v1/knowledge-bases` response is filtered to that allowlist. Choosing another entry performs a full iframe navigation, rechecks the DeerFlow mapping and read/write permission, then reissues the cookie for the selected base. Never expose the upstream service-admin list directly or permit an arbitrary remote knowledge-base ID.
@@ -1509,7 +1513,10 @@ authenticated readers obtain a short-lived signed capability from
`/files/access-url` and load `/api/public/llmwiki/files/{token}`. The signed
payload is restricted to `kind=wiki_file`, mapping id, remote id, validated
protected path, and expiry, and responses provide an inline UTF-8 filename for
new-tab viewing and browser Save As.
new-tab viewing and browser Save As. The proxied WeKnora detail iframe must
likewise leave its scoped HTTP file-proxy URL on every protected image and
restore it if upstream rendering replaces `src`; its click preview reads that
same DOM value, so a fetched-and-revoked object URL is not acceptable.
Third-party pages embed published Wiki content through the chrome-free route
`/#/embed/knowledge/{mapping_id}` (optional exact-page `wikiSlug`). It defaults
@@ -1558,11 +1565,26 @@ Internal search authorization always starts from DeerFlow mapping ids and is
rechecked by the Gateway. External endpoints use the isolated
`/api/external/llmwiki/wiki` namespace and `X-API-Key`; require published +
external-enabled + index-enabled on every request and response, and hard-deny
the `system` / `对话沉淀` mapping. Never return remote WeKnora ids, provenance
refs, credentials, or vector bytes externally. Regression tests:
the `system` / `对话沉淀` mapping. Those API-key endpoints never return remote
WeKnora ids, provenance refs, credentials, or vector bytes.
`POST /api/knowledge/vector-search` is a deliberately separate anonymous API.
It has no token/API-key requirement or rate limit, applies route-scoped
`Access-Control-Allow-Origin: *`, searches every published local Wiki index
regardless of `external_search_enabled`, and still hard-denies the `system` /
`对话沉淀` mapping before and after search. Its contract is section-level (one
result per matched vector, so Wiki-page fields and URLs may repeat), default
`top_k=10`, with optional `include_page_content`. Unlike the protected legacy
external namespace, this endpoint intentionally returns DeerFlow/WeKnora ids
and raw source/chunk provenance, but never credentials or vector bytes. Wiki
links use `llmwiki.local_wiki_index.frontend_base_url` (development default
`http://127.0.0.1:5174`) and point to the native
`/#/page/workspace/knowledge/{mapping_id}/wiki/view?wikiSlug=...` route.
Regression tests:
`tests/test_llmwiki_local_index_repository.py`,
`tests/test_llmwiki_wiki_sync.py`, `tests/test_llmwiki_vector_search.py`,
`tests/test_external_llmwiki_search.py`, and
`tests/test_external_llmwiki_search.py`,
`tests/test_public_knowledge_vector_search.py`, and
`tests/test_llmwiki_wiki_retrieval.py`.
The sandbox whole-report rewrite may carry an optional `reportOutline` Markdown
+19 -1
View File
@@ -837,7 +837,10 @@ URLs. Public images use the published file proxy directly; private images first
obtain a 24-hour signed capability and then load through
`GET /api/public/llmwiki/files/{token}` with a real HTTP URL and inline
filename, allowing zoom, new-tab viewing, and browser Save As without exposing
the user's bearer token.
the user's bearer token. The embedded WeKnora detail page follows the same
rule: late `img[data-protected-src]` placeholders keep their scoped HTTP proxy
URL on the DOM node, so WeKnora's click-to-enlarge viewer never receives an
already-revoked `blob:` URL.
## Leaderboard query and schema startup safety
@@ -894,6 +897,21 @@ Authenticated operations are available below `/api/llmwiki/wiki-index` and
`/api/external/llmwiki/wiki` require `X-API-Key`, are rate-limited, and only
expose mappings that are both internally published and explicitly marked
`external_search_enabled`; the system `对话沉淀` mapping is always denied.
A separate browser-friendly endpoint, `POST /api/knowledge/vector-search`, is
fully anonymous: it requires neither a DeerFlow token nor `X-API-Key`, has no
rate limit, and responds with `Access-Control-Allow-Origin: *`. It searches the
local vectors of every published knowledge base regardless of
`external_search_enabled`, while still excluding the system `对话沉淀` base.
The minimal request is `{"query":"如何部署"}`; optional fields are `top_k`
(default `10`, maximum `100`) and `include_page_content` (default `false`). Each
matched vector section is returned as its own ranked result, including the
matched text, score, DeerFlow and WeKnora knowledge/Wiki ids, page metadata,
raw source/chunk references, best-effort document/chunk ids, and a directly
usable `wiki_url`. Set `llmwiki.local_wiki_index.frontend_base_url` to the
deployed frontend origin/base path; its development default is
`http://127.0.0.1:5174`.
See `docs/LLMWIKI_DEERFLOW_LOCAL_VECTOR_INDEX_ZH.md` for configuration,
security boundaries, rollout, and API contracts.
@@ -24,8 +24,9 @@ from app.gateway.auth.disabled_mode import is_auth_disabled # noqa: E402
from app.gateway.auth_middleware import AuthMiddleware
from app.gateway.config import get_gateway_config
from app.gateway.csrf_middleware import CSRFMiddleware
from app.gateway.deps import langgraph_runtime
from app.gateway.routers import (
from app.gateway.deps import langgraph_runtime
from app.gateway.public_knowledge_cors import PublicKnowledgeCORSMiddleware
from app.gateway.routers import (
ai_writing,
admin_active_runs,
admin_users,
@@ -66,8 +67,9 @@ from app.gateway.routers import (
positions,
position_roles,
public_config,
public_embed,
public_leaderboard,
public_embed,
public_knowledge_vector_search,
public_leaderboard,
public_scheduled_tasks,
recommend,
flow_extract,
@@ -1473,11 +1475,16 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
"Set GATEWAY_CORS_ORIGINS or GATEWAY_CORS_ALLOW_ALL for deployment.",
", ".join(dev_origins),
)
app.add_middleware(
CORSMiddleware,
allow_origins=dev_origins,
**cors_kwargs,
)
app.add_middleware(
CORSMiddleware,
allow_origins=dev_origins,
**cors_kwargs,
)
# This one anonymous API is intentionally callable by browser code from
# every origin. Keep the override route-scoped so authenticated APIs retain
# the deployment's normal CORS allow-list.
app.add_middleware(PublicKnowledgeCORSMiddleware)
# Include routers
# Models API is mounted at /api/models
@@ -1626,8 +1633,11 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
app.include_router(public_config.router)
app.include_router(html_page_favorites.router)
# Product knowledge base API is mounted at /api/knowledge
app.include_router(knowledge.router)
# Product knowledge base API is mounted at /api/knowledge
app.include_router(knowledge.router)
# Anonymous, section-level vector search over every published Wiki base
# except the system conversation-deposit base.
app.include_router(public_knowledge_vector_search.router)
# External knowledge ingest (publish md artifacts) at /api/knowledge-ingest
app.include_router(knowledge_ingest.router)
@@ -68,9 +68,11 @@ _PUBLIC_EXACT_PATHS: frozenset[str] = frozenset(
"/api/v1/auth/login/token",
"/api/v1/auth/register",
"/api/v1/auth/logout",
"/api/v1/auth/setup-status",
"/api/v1/auth/initialize",
# 并行多智能体面板专用的「绕过常规登录」取 token 端点(内网部署限制了标准登录时用)。
"/api/v1/auth/setup-status",
"/api/v1/auth/initialize",
# Anonymous vector search over published local Wiki indexes.
"/api/knowledge/vector-search",
# 并行多智能体面板专用的「绕过常规登录」取 token 端点(内网部署限制了标准登录时用)。
"/api/parallel-agents/auth/token",
}
)
@@ -48,8 +48,10 @@ def should_check_csrf(request: Request) -> bool:
path = app_relative_path(request)
stripped = path.rstrip("/")
# Exempt /api/v1/auth/me endpoint
if stripped == "/api/v1/auth/me":
return False
if stripped == "/api/v1/auth/me":
return False
if stripped == "/api/knowledge/vector-search":
return False
# Anonymous public endpoints (mirror auth_middleware._PUBLIC_PATH_PREFIXES).
# These intentionally have no cookie session, so the Double-Submit Cookie
# pattern is both impossible (no token to compare) and unnecessary (no
@@ -0,0 +1,49 @@
"""Route-scoped permissive CORS for the anonymous knowledge vector API."""
from __future__ import annotations
from collections.abc import Callable
from fastapi import Request, Response
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.types import ASGIApp
from app.gateway.proxy_path import app_relative_path
PUBLIC_KNOWLEDGE_VECTOR_SEARCH_PATH = "/api/knowledge/vector-search"
def is_public_knowledge_vector_search(request: Request) -> bool:
return app_relative_path(request).rstrip("/") == PUBLIC_KNOWLEDGE_VECTOR_SEARCH_PATH
def _apply_public_cors(response: Response) -> Response:
response.headers["Access-Control-Allow-Origin"] = "*"
response.headers["Access-Control-Allow-Methods"] = "POST, OPTIONS"
response.headers["Access-Control-Allow-Headers"] = "*"
response.headers["Access-Control-Allow-Private-Network"] = "true"
response.headers["Access-Control-Max-Age"] = "600"
if "Access-Control-Allow-Credentials" in response.headers:
del response.headers["Access-Control-Allow-Credentials"]
return response
class PublicKnowledgeCORSMiddleware(BaseHTTPMiddleware):
"""Allow every browser origin for one credential-free public route only."""
def __init__(self, app: ASGIApp) -> None:
super().__init__(app)
async def dispatch(self, request: Request, call_next: Callable) -> Response:
if not is_public_knowledge_vector_search(request):
return await call_next(request)
if request.method.upper() == "OPTIONS":
return _apply_public_cors(Response(status_code=204))
return _apply_public_cors(await call_next(request))
__all__ = [
"PUBLIC_KNOWLEDGE_VECTOR_SEARCH_PATH",
"PublicKnowledgeCORSMiddleware",
"is_public_knowledge_vector_search",
]
@@ -17,8 +17,8 @@ from pydantic import BaseModel, Field
from app.gateway.deps import get_current_user, get_optional_user_from_request
from deerflow.persistence.compact_menu_overrides import (
CompactMenuOverrideStore,
DEFAULT_COMPACT_MENU_TYPE,
CompactMenuOverrideStore,
normalize_compact_menu_type,
)
@@ -27,7 +27,7 @@ router = APIRouter(prefix="/api/compact-menu-overrides", tags=["compact-menu-ove
class CompactMenuOverride(BaseModel):
node_id: str = Field(..., min_length=1, max_length=128)
placement: Literal["sidebar", "settings"] = "sidebar"
placement: Literal["sidebar", "settings", "toolbox"] = "sidebar"
sort_order: int = 0
label: str | None = Field(default=None, max_length=255)
@@ -1151,35 +1151,15 @@ html.deerflow-source-detail-open [data-deerflow-source-detail-panel="true"] [cla
if (!proxyUrl) return;
if (
image.dataset.deerflowProtectedSource === protectedSrc &&
String(image.getAttribute("src") || "").startsWith("blob:")
String(image.getAttribute("src") || "") === proxyUrl
) return;
const attempts = Number(image.dataset.deerflowProtectedAttempts || "0");
if (attempts >= 3 || image.dataset.deerflowProtectedLoading === "1") return;
image.dataset.deerflowProtectedLoading = "1";
image.dataset.deerflowProtectedAttempts = String(attempts + 1);
nativeFetch(proxyUrl, {{ credentials: "same-origin", headers: {{ Accept: "image/*" }} }})
.then((response) => {{
if (!response.ok) throw new Error(`protected image request failed (${{response.status}})`);
return response.blob();
}})
.then((blob) => {{
const objectUrl = URL.createObjectURL(blob);
image.dataset.deerflowProtectedSource = protectedSrc;
image.removeAttribute("data-img-loading");
image.removeAttribute("data-wiki-image-error");
image.src = objectUrl;
image.addEventListener("load", () => URL.revokeObjectURL(objectUrl), {{ once: true }});
}})
.catch(() => {{
image.dataset.wikiImageError = "true";
window.setTimeout(() => {{
delete image.dataset.deerflowProtectedLoading;
hydrateProtectedImage(image);
}}, 1200 * (attempts + 1));
}})
.finally(() => {{
delete image.dataset.deerflowProtectedLoading;
}});
// Keep the real HTTP proxy URL on the DOM node. WeKnora's preview/lightbox
// reuses the thumbnail's src when the user clicks it; a short-lived blob URL
// may render once and then become invalid before the preview opens.
image.dataset.deerflowProtectedSource = protectedSrc;
image.removeAttribute("data-img-loading");
image.removeAttribute("data-wiki-image-error");
image.src = proxyUrl;
}}
function hydrateProtectedImages(scope = document) {{
if (scope?.matches?.("img[data-protected-src]")) hydrateProtectedImage(scope);
@@ -1197,7 +1177,7 @@ html.deerflow-source-detail-open [data-deerflow-source-detail-panel="true"] [cla
childList: true,
subtree: true,
attributes: true,
attributeFilter: ["data-protected-src"]
attributeFilter: ["data-protected-src", "src"]
}});
hydrateProtectedImages();
@@ -0,0 +1,152 @@
"""Anonymous vector search over published, locally indexed WeKnora Wiki data."""
from __future__ import annotations
from typing import Any
from urllib.parse import quote, urlencode
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel, Field
from deerflow.integrations.weknora.local_index.search import WikiIndexError
router = APIRouter(tags=["public-knowledge-vector-search"])
class PublicKnowledgeVectorSearchRequest(BaseModel):
query: str = Field(..., min_length=1, max_length=10_000)
top_k: int = Field(default=10, ge=1, le=100)
include_page_content: bool = False
def _is_conversation_deposit(mapping: dict[str, Any]) -> bool:
return str(mapping.get("owner_user_id") or "") == "system" and str(mapping.get("name") or "").strip() == "对话沉淀"
async def _published_mappings(request: Request) -> list[dict[str, Any]]:
store = getattr(request.app.state, "llmwiki_store", None)
if store is None:
raise HTTPException(status_code=503, detail={"code": "LLMWIKI_STORE_UNAVAILABLE"})
rows = await store.list_visible("system", scope="public", is_admin=False)
return [row for row in rows if row.get("publication_status") == "published" and not _is_conversation_deposit(row)]
def _unique_strings(values: list[str]) -> list[str]:
return list(dict.fromkeys(value for value in values if value))
def _reference_metadata(item: dict[str, Any]) -> dict[str, Any]:
source_refs = item.get("source_refs") if isinstance(item.get("source_refs"), list) else []
chunk_refs = item.get("chunk_refs") if isinstance(item.get("chunk_refs"), list) else []
document_ids: list[str] = []
document_names: list[str] = []
chunk_ids: list[str] = []
source_ref_ids: list[str] = []
for ref in source_refs:
if isinstance(ref, dict):
document_id = str(ref.get("document_id") or ref.get("knowledge_id") or ref.get("documentId") or ref.get("knowledgeId") or "").strip()
document_name = str(
ref.get("document_name") or ref.get("knowledge_title") or ref.get("documentName") or ref.get("knowledgeTitle") or ref.get("file_name") or ref.get("filename") or ref.get("title") or ref.get("name") or ""
).strip()
chunk_id = str(ref.get("chunk_id") or ref.get("chunkId") or "").strip()
if document_id:
document_ids.append(document_id)
if document_name:
document_names.append(document_name)
if chunk_id:
chunk_ids.append(chunk_id)
elif isinstance(ref, (str, int)) and str(ref).strip():
source_ref_ids.append(str(ref).strip())
for ref in chunk_refs:
if isinstance(ref, dict):
chunk_id = str(ref.get("chunk_id") or ref.get("chunkId") or ref.get("id") or "").strip()
else:
chunk_id = str(ref or "").strip()
if chunk_id:
chunk_ids.append(chunk_id)
document_ids = _unique_strings(document_ids)
document_names = _unique_strings(document_names)
chunk_ids = _unique_strings(chunk_ids)
source_ref_ids = _unique_strings(source_ref_ids)
return {
"document_id": document_ids[0] if document_ids else None,
"document_name": document_names[0] if document_names else None,
"document_ids": document_ids,
"document_names": document_names,
"chunk_id": chunk_ids[0] if chunk_ids else None,
"chunk_ids": chunk_ids,
"source_ref_ids": source_ref_ids,
}
def _wiki_link(frontend_base_url: str, mapping_id: str, wiki_slug: str) -> tuple[str, str]:
query = urlencode({"wikiSlug": wiki_slug})
route = f"/#/page/workspace/knowledge/{quote(mapping_id, safe='')}/wiki/view?{query}"
return route, f"{frontend_base_url.rstrip('/')}{route}"
@router.post("/api/knowledge/vector-search")
async def public_knowledge_vector_search(
request: Request,
body: PublicKnowledgeVectorSearchRequest,
) -> dict[str, Any]:
mappings = await _published_mappings(request)
search = getattr(request.app.state, "llmwiki_vector_search", None)
if search is None:
raise HTTPException(status_code=503, detail={"code": "WIKI_INDEX_UNAVAILABLE"})
try:
payload = await search.search_sections(
body.query,
mappings,
top_k_sections=body.top_k,
external=True,
)
except WikiIndexError as exc:
raise HTTPException(
status_code=exc.status_code,
detail={"code": exc.code, "message": str(exc)},
) from None
# Publication can change while an embedding/search call is in flight. A
# second read ensures an unpublished or newly reserved result is never
# emitted by this anonymous endpoint.
still_published = {str(row["id"]) for row in await _published_mappings(request)}
frontend_base_url = request.app.state.config.llmwiki.local_wiki_index.frontend_base_url
results: list[dict[str, Any]] = []
for raw_item in payload.get("results") or []:
if str(raw_item.get("knowledge_base_id") or "") not in still_published:
continue
item = dict(raw_item)
wiki_route, wiki_url = _wiki_link(
frontend_base_url,
str(item["knowledge_base_id"]),
str(item["wiki_slug"]),
)
item.update(_reference_metadata(item))
item["wiki_route"] = wiki_route
item["wiki_url"] = wiki_url
if not body.include_page_content:
item.pop("page_content", None)
results.append(item)
return {
"query": body.query,
"top_k": body.top_k,
"mode": "vector",
"results": results,
"count": len(results),
"knowledge_base_count": payload.get("knowledge_base_count", len(mappings)),
"index_fingerprint": payload.get("index_fingerprint"),
"partial": bool(payload.get("partial", False)),
"warnings": payload.get("warnings") or [],
}
__all__ = [
"PublicKnowledgeVectorSearchRequest",
"public_knowledge_vector_search",
"router",
]
@@ -561,6 +561,8 @@ llmwiki:
# 且服务启动时只启动定时器,不会立刻全量向量化。
local_wiki_index:
enabled: false
# 免登录 POST /api/knowledge/vector-search 返回的 Wiki 一键跳转链接前缀。
frontend_base_url: "http://127.0.0.1:5174"
auto_sync: false
database_batch_size: 50
embedding:
@@ -1132,6 +1132,35 @@ Content-Type: application/json
}
```
#### 13.4.1 免鉴权浏览器向量搜索
本部署另行提供一个与上述 API-Key 命名空间隔离的新接口:
```http
POST /api/knowledge/vector-search
Content-Type: application/json
```
最小请求只需检索词:
```json
{"query": "任正非早期创业经历"}
```
可选参数只有 `top_k`(默认 `10`,最大 `100`)和
`include_page_content`(默认 `false`)。接口不需要登录 Token 或
`X-API-Key`,不做应用内限流,并针对这一条路由返回
`Access-Control-Allow-Origin: *`。检索范围是全部 `published` 知识库,
不检查 `external_search_enabled`,但在检索前后都排除系统“对话沉淀”库。
响应按向量分段展开:同一 Wiki 页命中多个分段时会返回多条结果。每条结果
包含命中文本和分数、DeerFlow/WeKnora 知识库与 Wiki 页 ID、页面元数据、
原始 `source_refs`/`chunk_refs`、可提取到的文档/分块 ID,以及可直接打开的
`wiki_url`。当 `include_page_content=true` 时额外返回完整 `page_content`。
`wiki_url` 的前缀配置为
`llmwiki.local_wiki_index.frontend_base_url`,开发默认值是
`http://127.0.0.1:5174`。
### 13.5 外部页面详情
```http
@@ -1156,6 +1185,9 @@ X-API-Key: <external key>
如果为了少改 AuthMiddleware 而把接口挂到 `/api/public/`,路由自身也必须强制校验 API Key;但长期推荐独立 `/api/external/` 语义。
上面的原则仍适用于通用服务间接口;13.4.1 是经过明确产品确认的例外,并通过
精确路径白名单和路由级 CORS 隔离,不能据此放宽其他 API。
### 13.7 状态码约定
| 状态码 | 场景 |
@@ -108,6 +108,10 @@ class LocalWikiIndexConfig(BaseModel):
"""DeerFlow-owned mirror and exact-vector index for WeKnora Wiki pages."""
enabled: bool = False
frontend_base_url: str = Field(
default="http://127.0.0.1:5174",
description="DeerFlow frontend base URL used to build directly usable Wiki links in public vector-search results.",
)
auto_sync: bool = False
sync_interval_seconds: int = Field(default=120, ge=10, le=86400)
sync_page_size: int = Field(default=100, ge=1, le=500)
@@ -135,6 +139,14 @@ class LocalWikiIndexConfig(BaseModel):
external_api: LocalWikiExternalApiConfig = Field(default_factory=LocalWikiExternalApiConfig)
model_config = ConfigDict(extra="ignore")
@field_validator("frontend_base_url")
@classmethod
def _validate_frontend_base_url(cls, value: str) -> str:
normalized = normalize_service_url(value)
if not normalized:
raise ValueError("local Wiki index frontend_base_url is required")
return normalized
@model_validator(mode="before")
@classmethod
def _migrate_legacy_flags(cls, value):
@@ -46,29 +46,12 @@ class WikiVectorSearchService:
raise WikiIndexError("WIKI_QUERY_EMPTY", "Search query is empty", status_code=400)
if not mappings:
return {"query": normalized_query, "mode": "vector", "results": [], "knowledge_base_count": 0, "index_fingerprint": self.embedding.fingerprint[:12], "partial": False, "warnings": []}
try:
query_vector = await self._embed_query(normalized_query)
except (WikiEmbeddingError, ValueError) as exc:
raise WikiIndexError("WIKI_EMBEDDING_UNAVAILABLE", str(exc), status_code=503) from None
fingerprint = self.embedding.fingerprint
tasks = [self._search_mapping(normalized_query, query_vector, mapping, fingerprint, include_drafts=include_drafts, external=external) for mapping in mappings]
batches = await asyncio.gather(*tasks, return_exceptions=True)
candidates: list[dict[str, Any]] = []
warnings: list[str] = []
for mapping, batch in zip(mappings, batches, strict=True):
if isinstance(batch, Exception):
warnings.append(f"{mapping.get('id')}: {batch}")
else:
candidates.extend(batch)
if warnings and not candidates and len(warnings) == len(mappings):
if any("FINGERPRINT" in item for item in warnings):
code = "WIKI_INDEX_FINGERPRINT_MISMATCH"
elif any("rebuild" in item.lower() for item in warnings):
code = "WIKI_INDEX_REBUILDING"
else:
code = "WIKI_INDEX_NOT_READY"
raise WikiIndexError(code, "Local Wiki index is not ready", status_code=409)
candidates.sort(key=lambda item: item["score"], reverse=True)
fingerprint, candidates, warnings = await self._collect_candidates(
normalized_query,
mappings,
include_drafts=include_drafts,
external=external,
)
grouped: dict[tuple[str, str], list[dict[str, Any]]] = defaultdict(list)
for candidate in candidates:
page = candidate["page"]
@@ -111,6 +94,141 @@ class WikiVectorSearchService:
limit = min(top_k_pages or self.config.top_k_pages, self.config.top_k_pages if top_k_pages is None else 50)
return {"query": normalized_query, "mode": "vector", "results": results[:limit], "knowledge_base_count": len(mappings), "index_fingerprint": fingerprint[:12], "partial": bool(warnings), "warnings": warnings}
async def search_sections(
self,
query: str,
mappings: list[dict[str, Any]],
*,
top_k_sections: int = 10,
external: bool = False,
) -> dict[str, Any]:
"""Return one result per matching Wiki vector section.
The regular ``search`` method intentionally groups several sections
under one Wiki page for citations. Public consumers of the standalone
vector API need the opposite contract: every matched vector is a
separate ranked item, while the complete page and knowledge-base
metadata remains attached to that item.
"""
normalized_query = query.strip()
if not normalized_query:
raise WikiIndexError("WIKI_QUERY_EMPTY", "Search query is empty", status_code=400)
limit = max(1, min(int(top_k_sections), 100))
if not mappings:
return {
"query": normalized_query,
"mode": "vector",
"results": [],
"knowledge_base_count": 0,
"index_fingerprint": self.embedding.fingerprint[:12],
"partial": False,
"warnings": [],
}
fingerprint, candidates, warnings = await self._collect_candidates(
normalized_query,
mappings,
include_drafts=False,
external=external,
section_limit=limit,
)
results: list[dict[str, Any]] = []
for candidate in candidates[:limit]:
page = candidate["page"]
mapping = candidate["mapping"]
results.append(
{
"kind": "wiki_section",
"id": candidate.get("id") or f"{page['id']}:{candidate['section_index']}",
"section_id": candidate.get("id") or f"{page['id']}:{candidate['section_index']}",
"section_index": int(candidate["section_index"]),
"heading": candidate.get("heading"),
"content": str(candidate.get("section_content") or ""),
"section_content_hash": candidate.get("content_hash") or "",
"score": float(candidate["score"]),
"knowledge_base_id": mapping["id"],
"weknora_knowledge_base_id": mapping.get("weknora_id") or "",
"knowledge_base_name": mapping.get("name") or "",
"knowledge_base_description": mapping.get("description") or "",
"knowledge_base_type": mapping.get("kb_type") or "document",
"publication_status": mapping.get("publication_status") or "",
"published_at": mapping.get("published_at"),
"wiki_page_id": page["id"],
"weknora_wiki_page_id": page.get("remote_page_id"),
"wiki_slug": page["slug"],
"title": page.get("title") or page["slug"],
"summary": page.get("summary") or "",
"page_content": page.get("content_md") or "",
"page_type": page.get("page_type") or "article",
"page_status": page.get("status") or "published",
"page_version": int(page.get("page_version") or 0),
"aliases": page.get("aliases") or [],
"category_path": page.get("category_path") or [],
"parent_slug": page.get("parent_slug"),
"wiki_path": page.get("wiki_path"),
"source_refs": page.get("source_refs") or [],
"chunk_refs": page.get("chunk_refs") or [],
"page_content_hash": page.get("content_hash") or "",
"remote_created_at": page.get("remote_created_at"),
"remote_updated_at": page.get("remote_updated_at"),
"indexed_at": page.get("indexed_at"),
}
)
return {
"query": normalized_query,
"mode": "vector",
"results": results,
"knowledge_base_count": len(mappings),
"index_fingerprint": fingerprint[:12],
"partial": bool(warnings),
"warnings": warnings,
}
async def _collect_candidates(
self,
query: str,
mappings: list[dict[str, Any]],
*,
include_drafts: bool,
external: bool,
section_limit: int | None = None,
) -> tuple[str, list[dict[str, Any]], list[str]]:
try:
query_vector = await self._embed_query(query)
except (WikiEmbeddingError, ValueError) as exc:
raise WikiIndexError("WIKI_EMBEDDING_UNAVAILABLE", str(exc), status_code=503) from None
fingerprint = self.embedding.fingerprint
tasks = [
self._search_mapping(
query,
query_vector,
mapping,
fingerprint,
include_drafts=include_drafts,
external=external,
section_limit=section_limit,
)
for mapping in mappings
]
batches = await asyncio.gather(*tasks, return_exceptions=True)
candidates: list[dict[str, Any]] = []
warnings: list[str] = []
for mapping, batch in zip(mappings, batches, strict=True):
if isinstance(batch, Exception):
warnings.append(f"{mapping.get('id')}: {batch}")
else:
candidates.extend(batch)
if warnings and not candidates and len(warnings) == len(mappings):
if any("FINGERPRINT" in item for item in warnings):
code = "WIKI_INDEX_FINGERPRINT_MISMATCH"
elif any("rebuild" in item.lower() for item in warnings):
code = "WIKI_INDEX_REBUILDING"
else:
code = "WIKI_INDEX_NOT_READY"
raise WikiIndexError(code, "Local Wiki index is not ready", status_code=409)
candidates.sort(key=lambda item: item["score"], reverse=True)
return fingerprint, candidates, warnings
async def _embed_query(self, query: str) -> np.ndarray:
digest = hashlib.sha256(query.encode("utf-8")).hexdigest()
key = f"{self.embedding.fingerprint}:{digest}"
@@ -126,7 +244,17 @@ class WikiVectorSearchService:
self._query_vectors.popitem(last=False)
return vector
async def _search_mapping(self, query: str, vector: np.ndarray, mapping: dict[str, Any], fingerprint: str, *, include_drafts: bool, external: bool) -> list[dict[str, Any]]:
async def _search_mapping(
self,
query: str,
vector: np.ndarray,
mapping: dict[str, Any],
fingerprint: str,
*,
include_drafts: bool,
external: bool,
section_limit: int | None = None,
) -> list[dict[str, Any]]:
mapping_id = str(mapping["id"])
try:
snapshot = await self.cache.get(mapping_id, fingerprint)
@@ -148,7 +276,13 @@ class WikiVectorSearchService:
)
if eligible.size == 0:
return []
indices, scores = await asyncio.to_thread(self._top_indices, snapshot, vector, eligible)
indices, scores = await asyncio.to_thread(
self._top_indices,
snapshot,
vector,
eligible,
section_limit,
)
terms = {term.lower() for term in re.findall(r"[\w\u4e00-\u9fff]{2,}", query)}
candidates = []
for index, score in zip(indices, scores, strict=True):
@@ -166,9 +300,15 @@ class WikiVectorSearchService:
candidates.append({**row, "page": page, "mapping": mapping, "score": final_score})
return candidates
def _top_indices(self, snapshot: WikiVectorSnapshot, vector: np.ndarray, eligible: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
def _top_indices(
self,
snapshot: WikiVectorSnapshot,
vector: np.ndarray,
eligible: np.ndarray,
section_limit: int | None = None,
) -> tuple[np.ndarray, np.ndarray]:
scores = snapshot.matrix[eligible] @ vector
count = min(self.config.top_k_sections, scores.size)
count = min(section_limit or self.config.top_k_sections, scores.size)
if count <= 0:
return np.asarray([], dtype=int), np.asarray([], dtype=np.float32)
positions = np.argpartition(scores, -count)[-count:]
@@ -19,7 +19,7 @@ def _normalize(data: dict[str, Any], menu_type: str, updated_by: str | None) ->
return {
"node_id": storage_node_id(menu_type, node_id),
"menu_type": menu_type,
"placement": placement if placement in {"sidebar", "settings"} else "sidebar",
"placement": placement if placement in {"sidebar", "settings", "toolbox"} else "sidebar",
"sort_order": int(data.get("sort_order", 0)),
"label": (data.get("label") or None),
"updated_by": updated_by,
@@ -35,7 +35,7 @@ def _row_from(data: dict[str, Any], menu_type: str, updated_by: str | None) -> C
return CompactMenuOverrideRow(
node_id=storage_node_id(menu_type, node_id),
menu_type=menu_type,
placement=placement if placement in {"sidebar", "settings"} else "sidebar",
placement=placement if placement in {"sidebar", "settings", "toolbox"} else "sidebar",
sort_order=int(data.get("sort_order", 0)),
label=(data.get("label") or None),
updated_by=updated_by,
@@ -12,8 +12,8 @@ from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
from deerflow.persistence.base import Base
from deerflow.persistence.compact_menu_overrides import (
MemoryCompactMenuOverrideStore,
CompactMenuOverrideRepository,
MemoryCompactMenuOverrideStore,
normalize_compact_menu_type,
)
from deerflow.persistence.compact_menu_overrides.keys import public_node_id, storage_node_id
@@ -53,15 +53,17 @@ async def test_replace_all_roundtrips(store: CompactMenuOverrideRepository):
[
{"node_id": "workspace-chats", "placement": "settings", "sort_order": 1, "label": "会话"},
{"node_id": "workspace-agents", "placement": "sidebar", "sort_order": 0, "label": None},
{"node_id": "workspace-skills", "placement": "toolbox", "sort_order": 0, "label": "技能工具"},
],
updated_by="admin-1",
)
assert len(out) == 2
assert len(out) == 3
rows = {r["node_id"]: r for r in await store.list_overrides()}
assert rows["workspace-chats"]["label"] == "会话"
assert rows["workspace-chats"]["placement"] == "settings"
assert rows["workspace-chats"]["menu_type"] == "A"
assert rows["workspace-agents"]["updated_by"] == "admin-1"
assert rows["workspace-skills"]["placement"] == "toolbox"
@pytest.mark.asyncio
@@ -56,6 +56,13 @@ async def test_exact_search_groups_sections_by_wiki_page_and_filters_drafts() ->
with_drafts = await service.search("alice", [mapping], include_drafts=True)
assert {item["wiki_slug"] for item in with_drafts["results"]} == {"alice", "secret"}
section_result = await service.search_sections("alice", [mapping], top_k_sections=2, external=True)
assert [item["content"] for item in section_result["results"]] == ["Alpha", "Beta"]
assert all(item["wiki_slug"] == "alice" for item in section_result["results"])
assert all(item["kind"] == "wiki_section" for item in section_result["results"])
assert section_result["results"][0]["wiki_page_id"]
assert "page_content" in section_result["results"][0]
narrow_service = WikiVectorSearchService(
LocalWikiIndexConfig(min_similarity=0, top_k_sections=1, top_k_pages=1),
FakeEmbedding(),
@@ -1379,7 +1379,9 @@ def test_weknora_iframe_html_rewrites_public_prefix_paths() -> None:
assert "function protectedImageApiUrl" in injected
assert 'img[data-protected-src]' in injected
assert 'encodeURIComponent(cfg.weknoraId)' in injected
assert 'credentials: "same-origin"' in injected
assert "image.src = proxyUrl;" in injected
assert "URL.createObjectURL(blob)" not in injected
assert 'attributeFilter: ["data-protected-src", "src"]' in injected
def test_weknora_iframe_filters_kb_list_to_signed_visible_mappings() -> None: