Merge branch 'master' of http://192.168.0.200/knowledge-base-bj/deerflow-code into master-nj
# Conflicts: # AGENTS.md # frontend-web/src/pages/AgentChatPage.tsx
This commit is contained in:
董宏凯 committed
commit
38c3de39f0
41 files changed
+2113
-737
No files matched your search
@@ -324,6 +324,7 @@ FastAPI application on port 8001 with health checks at `GET|HEAD /health`, `/api
|
||||
| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; unexpected failures are logged server-side and return a generic 500 detail |
|
||||
| **Threads search** (`POST /threads/search`, LangGraph-compat) | Conversation listing. **System threads are excluded by default**: rows whose `metadata.system` is truthy (scheduler runs, roundtable workers) never crowd the user's list; a caller whose `metadata` filter explicitly references `thread_type`/`system` (e.g. studio notebooks) opts out of the exclusion. New `query` field = case-insensitive title (`display_name`) substring search, used by the chats page's server-side pagination. `POST /threads/count` takes the same request body (limit/offset ignored) and returns `{total}` under identical filtering/exclusion semantics ? powers the chats page's ??/??? display without fetching rows (`ThreadMetaStore.count`). `ThreadMetaStore.search` gained `exclude_system`/`query` params; JSON-blob filters batch-scan until the requested `limit`/`offset` window is full (so pagination over a heavily-filtered set never under-fills), with a `thread_id` tie-breaker on the `updated_at DESC` ordering for stable OFFSET paging. Tests: `tests/test_thread_meta_search.py` |
|
||||
| **Artifacts** (`/api/threads/{id}/artifacts`) | `GET /{path}` - serve artifacts; active content types (`text/html`, `application/xhtml+xml`, `image/svg+xml`) are always forced as download attachments to reduce XSS risk; `?download=true` still forces download for other file types |
|
||||
| **Word export** (`POST /api/writing/export/docx`) | Shared Markdown → DOCX generator used by conversation, artifact, scheduled-task, Deep Research, and AI-writing download surfaces. `app/gateway/word_export.py` builds OOXML with no Python conversion dependency: A4 portrait, 3.7/3.5/2.8/2.6 cm margins, real multilevel heading numbering, formal body/table styles, page numbers beginning at 1 with odd pages left-aligned and even pages right-aligned, and embedded `app/gateway/assets/fonts/方正小标宋简体.ttf`. The font loader validates its internal family, SHA-256, and OpenType embedding permission (`OS/2.fsType`) before export. Tests: `tests/test_word_export.py`. |
|
||||
| **Deep Research** (`/api/deep-research`) | Durable, user-owned research sessions/jobs with replayable SSE events and hidden-thread artifacts. `basic`/`quick` use the compact runner; `detailed` creates bounded subtopics, searches/writes sections concurrently, and assembles a report; `deep` recursively searches follow-up questions under a deterministic breadth/depth query budget. `multi_agent` is a LangGraph 1.x editor/researcher/writer/reviewer graph: its initial plan emits `awaiting_input`, the recorder stores the interrupt and releases the lease, and `POST /jobs/{id}/resume` rebuilds the graph from the persisted approved/revised plan. A bounded `custom_outline` is frozen into the session; headings/list entries deterministically supply the detailed and multi-agent plan, and the shared writer receives it only as a guarded content constraint. Completed reports support owner-scoped follow-up (`GET /sessions/{id}/messages`, `POST /sessions/{id}/chat`): context is only the current report, selected sources and recent messages; source-id citations are persisted, while `allow_new_research=true` is explicitly rejected. Owners may change that evidence set only after the job stops via `PATCH /sessions/{id}/sources/{source_id}`; this applies to future follow-ups rather than rewriting a completed report. Optional report illustrations are gated by `config.yaml -> deep_research.image_generation`: the Harness `OpenAICompatibleImageProvider` requires base64 output, the shared runner step writes only bounded `research-image-01..04.(png|jpg|webp)` artifacts and Markdown uses private `deep-research://` links. `GET /sessions/{id}/artifacts/{name}` checks session ownership before serving an image; an unavailable provider yields a warning but never fails the text report. Runners live in `packages/harness/deerflow/agents/deep_research/runners/`, may never import `app.*`, and use only the `AdapterBundle`. |
|
||||
| **Gateway auth** | Owner-scoped Deep Research routes await the platform's asynchronous `get_current_user` resolver before they read or write persistence, ensuring each session and job operation receives a concrete user ID. Deep Research artifacts are always resolved from a whitelisted filename via `/mnt/user-data/outputs/<name>` before mapping to a host path, preserving the sandbox guard on native Windows and Linux hosts. |
|
||||
| **Suggestions** (`/api/threads/{id}/suggestions`) | `POST /` - generate follow-up questions; rich list/block model content is normalized before JSON parsing |
|
||||
@@ -1027,10 +1028,14 @@ Both can be modified at runtime via Gateway API endpoints or `DeerFlowClient` me
|
||||
|
||||
The embedded detail proxy hides WeKnora's outer `.main > .aside_box` application sidebar and expands the route outlet, but deliberately keeps the Wiki page's internal directory/index sidebar. Do not add an HTML `sandbox` attribute to this trusted, permission-filtered detail iframe: a sandboxed ancestor causes Chromium's built-in Blob PDF viewer to render the browser-blocked page. The Gateway follows upstream redirects only for `/knowledge/{id}/preview` and `/knowledge-bases/{id}/files`, keeping PDF bytes and protected Wiki images inside the signed proxy instead of sending the browser to WeKnora object storage.
|
||||
Native/private and public Wiki readers must rewrite protected Markdown image
|
||||
sources to their mapping-scoped DeerFlow `/files?file_path=` endpoints and load
|
||||
them as Blob URLs. The embedded guard also rewrites protected fetch/XHR values
|
||||
and hydrates late `img[data-protected-src]` placeholders; never expose the
|
||||
service token or relax the current-KB authorization to make images work.
|
||||
sources to their mapping-scoped DeerFlow `/files?file_path=` endpoints and keep
|
||||
real HTTP URLs on rendered image elements. The embedded guard also rewrites
|
||||
protected fetch/XHR values and hydrates late `img[data-protected-src]`
|
||||
placeholders with the signed iframe HTTP proxy URL. Observe both
|
||||
`data-protected-src` and `src`, because WeKnora may replace the URL after first
|
||||
paint. Never create/revoke a temporary object URL here: its lightbox reuses the
|
||||
thumbnail `src` after click. Never expose the service token or relax the
|
||||
current-KB authorization to make images work.
|
||||
|
||||
**Embedded native knowledge-base switching:** The original WeKnora breadcrumb picker remains available inside the detail iframe. A signed 15-minute embed cookie carries only the current DeerFlow user's visible mappings (their own plus published bases); the proxied `/api/v1/knowledge-bases` response is filtered to that allowlist. Choosing another entry performs a full iframe navigation, rechecks the DeerFlow mapping and read/write permission, then reissues the cookie for the selected base. Never expose the upstream service-admin list directly or permit an arbitrary remote knowledge-base ID.
|
||||
|
||||
@@ -1339,10 +1344,11 @@ the message-list thought trace, while the sandbox receives answer Markdown
|
||||
only. The post-report digest (`summarize_report`) must likewise forward
|
||||
provider reasoning through live-only `summary_thinking` frames so the
|
||||
chat timeline can render thought while waiting for the first `summary_delta`.
|
||||
The Vite
|
||||
page owns Word export through
|
||||
`frontend-web/src/core/scheduled-tasks/markdown-docx.ts`; do not add Python
|
||||
document-conversion dependencies merely to export text.
|
||||
The Vite page starts Word downloads through
|
||||
`frontend-web/src/core/scheduled-tasks/markdown-docx.ts`, but the Gateway owns
|
||||
generation through `POST /api/writing/export/docx`. Keep the standard-library
|
||||
OOXML builder and its embedded-font validation; do not add a Python document-
|
||||
conversion dependency merely to export text.
|
||||
All non-streaming Deep Research control-plane calls (`complete()` for query /
|
||||
section / plan / review JSON) run through `asyncio.wait_for` using the frozen
|
||||
`DeepResearchConfig.control_plane_timeout_seconds` (45 seconds by default,
|
||||
@@ -1512,7 +1518,10 @@ authenticated readers obtain a short-lived signed capability from
|
||||
`/files/access-url` and load `/api/public/llmwiki/files/{token}`. The signed
|
||||
payload is restricted to `kind=wiki_file`, mapping id, remote id, validated
|
||||
protected path, and expiry, and responses provide an inline UTF-8 filename for
|
||||
new-tab viewing and browser Save As.
|
||||
new-tab viewing and browser Save As. The proxied WeKnora detail iframe must
|
||||
likewise leave its scoped HTTP file-proxy URL on every protected image and
|
||||
restore it if upstream rendering replaces `src`; its click preview reads that
|
||||
same DOM value, so a fetched-and-revoked object URL is not acceptable.
|
||||
|
||||
Third-party pages embed published Wiki content through the chrome-free route
|
||||
`/#/embed/knowledge/{mapping_id}` (optional exact-page `wikiSlug`). It defaults
|
||||
@@ -1561,11 +1570,26 @@ Internal search authorization always starts from DeerFlow mapping ids and is
|
||||
rechecked by the Gateway. External endpoints use the isolated
|
||||
`/api/external/llmwiki/wiki` namespace and `X-API-Key`; require published +
|
||||
external-enabled + index-enabled on every request and response, and hard-deny
|
||||
the `system` / `对话沉淀` mapping. Never return remote WeKnora ids, provenance
|
||||
refs, credentials, or vector bytes externally. Regression tests:
|
||||
the `system` / `对话沉淀` mapping. Those API-key endpoints never return remote
|
||||
WeKnora ids, provenance refs, credentials, or vector bytes.
|
||||
|
||||
`POST /api/knowledge/vector-search` is a deliberately separate anonymous API.
|
||||
It has no token/API-key requirement or rate limit, applies route-scoped
|
||||
`Access-Control-Allow-Origin: *`, searches every published local Wiki index
|
||||
regardless of `external_search_enabled`, and still hard-denies the `system` /
|
||||
`对话沉淀` mapping before and after search. Its contract is section-level (one
|
||||
result per matched vector, so Wiki-page fields and URLs may repeat), default
|
||||
`top_k=10`, with optional `include_page_content`. Unlike the protected legacy
|
||||
external namespace, this endpoint intentionally returns DeerFlow/WeKnora ids
|
||||
and raw source/chunk provenance, but never credentials or vector bytes. Wiki
|
||||
links use `llmwiki.local_wiki_index.frontend_base_url` (development default
|
||||
`http://127.0.0.1:5174`) and point to the native
|
||||
`/#/page/workspace/knowledge/{mapping_id}/wiki/view?wikiSlug=...` route.
|
||||
Regression tests:
|
||||
`tests/test_llmwiki_local_index_repository.py`,
|
||||
`tests/test_llmwiki_wiki_sync.py`, `tests/test_llmwiki_vector_search.py`,
|
||||
`tests/test_external_llmwiki_search.py`, and
|
||||
`tests/test_external_llmwiki_search.py`,
|
||||
`tests/test_public_knowledge_vector_search.py`, and
|
||||
`tests/test_llmwiki_wiki_retrieval.py`.
|
||||
|
||||
The sandbox whole-report rewrite may carry an optional `reportOutline` Markdown
|
||||
|
||||
@@ -156,6 +156,7 @@ FastAPI application providing REST endpoints for frontend integration:
|
||||
| `GET/POST /api/deep-research/*` | Durable Deep Research sessions and jobs. `quick`/`basic` run the compact report flow; `detailed` plans subtopics and writes independent sections; `deep` recursively investigates bounded follow-up branches. All modes reuse DeerFlow's configured material provider, LLM factory, hidden-thread artifacts, and SSE event log. Deep Research session/job/event/source/message writes serialise their rows in the writer transaction, rather than doing a post-commit ORM refresh or readback; this prevents MySQL read/write-splitting replica lag from failing “start writing”, dispatcher claim, report persistence, or follow-up history. Whole-document artifact/report rewrites likewise create their reversible version snapshots from writer-local ORM data and never refresh after commit, so replica lag cannot restore an otherwise successful rewrite. A report is written to the session and confirmed before its Job becomes `completed`; an unconfirmed write is retried and never exposed as a false completion. Retrying an already-created queued job re-nudges the dispatcher, recovering an interrupted HTTP response without creating a second report. If progress-event persistence, durable replay, or live fan-out is briefly unavailable, the worker/stream logs the degradation and continues or reconnects rather than failing the whole job. Whole-report rewrite accepts an optional `reportOutline` Markdown template; the durable job persists it and applies it as the chapter structure constraint. Optional report illustrations use a separately configured OpenAI-compatible image endpoint and are served only through the owning session's authenticated artifact route. |
|
||||
| `/api/enterprise-research/*` | Temporarily disabled. Its router and persistence initialization are not registered, so these endpoints return 404 and no enterprise-research tables are created or migrated during startup. |
|
||||
| `POST /api/ai-writing/sample/extract` | Extract text from Word/PDF/Markdown/TXT samples for AI-writing imitation mode |
|
||||
| `POST /api/writing/export/docx` | Generate the shared formal Word download from Markdown. Uses the standard-library OOXML builder in `app/gateway/word_export.py`, applies the prescribed A4/margin/heading/body/table/footer profile, and embeds the server-bundled `方正小标宋简体.ttf`; the client computer does not need that font installed. |
|
||||
| `POST /cop/saveSuperiorTask` | TaskCOP compatibility API (DeerFlow Bearer/session authentication required): create a situation-overview task with a sequential four-digit id (`0001`, `0002`, ...) and return the legacy `{state, msg, data}` envelope |
|
||||
| `POST /api/taskcop/import/tasks` | TaskCOP compatibility API (authenticated): import tasks from a JSON array or `tasks`/`records` envelope; every row's supplied `id`/`taskId` is authoritative and a duplicate id completely replaces the stored task |
|
||||
| `DELETE /cop/tasks/{task_id}` | TaskCOP compatibility API (authenticated): delete one task and its task-scoped situation-report document |
|
||||
@@ -288,9 +289,10 @@ The Deep Research page is a conversation workspace: the original research
|
||||
question, observable retrieval steps, completion summary, and report-grounded
|
||||
follow-ups appear in chronological order. The report itself is written in the
|
||||
adjacent sandbox as `/mnt/user-data/outputs/report.md`, with Word, Markdown,
|
||||
and HTML export actions. Word export uses the frontend's existing `docx`
|
||||
dependency and includes the collected reference list, without changing backend
|
||||
Python dependencies. In chat-collection sessions the report configuration card
|
||||
and HTML export actions. Word export calls the shared Gateway
|
||||
`POST /api/writing/export/docx` generator, embeds the licensed title font, and
|
||||
includes the collected reference list without adding a backend Python package.
|
||||
In chat-collection sessions the report configuration card
|
||||
shows the unique material count inferred from the collector's visible tool
|
||||
results before writing begins. The write lifecycle itself is rendered as
|
||||
ordinary workspace message-list tool steps (not a second timeline component),
|
||||
@@ -840,7 +842,10 @@ URLs. Public images use the published file proxy directly; private images first
|
||||
obtain a 24-hour signed capability and then load through
|
||||
`GET /api/public/llmwiki/files/{token}` with a real HTTP URL and inline
|
||||
filename, allowing zoom, new-tab viewing, and browser Save As without exposing
|
||||
the user's bearer token.
|
||||
the user's bearer token. The embedded WeKnora detail page follows the same
|
||||
rule: late `img[data-protected-src]` placeholders keep their scoped HTTP proxy
|
||||
URL on the DOM node, so WeKnora's click-to-enlarge viewer never receives an
|
||||
already-revoked `blob:` URL.
|
||||
|
||||
## Leaderboard query and schema startup safety
|
||||
|
||||
@@ -897,6 +902,21 @@ Authenticated operations are available below `/api/llmwiki/wiki-index` and
|
||||
`/api/external/llmwiki/wiki` require `X-API-Key`, are rate-limited, and only
|
||||
expose mappings that are both internally published and explicitly marked
|
||||
`external_search_enabled`; the system `对话沉淀` mapping is always denied.
|
||||
|
||||
A separate browser-friendly endpoint, `POST /api/knowledge/vector-search`, is
|
||||
fully anonymous: it requires neither a DeerFlow token nor `X-API-Key`, has no
|
||||
rate limit, and responds with `Access-Control-Allow-Origin: *`. It searches the
|
||||
local vectors of every published knowledge base regardless of
|
||||
`external_search_enabled`, while still excluding the system `对话沉淀` base.
|
||||
The minimal request is `{"query":"如何部署"}`; optional fields are `top_k`
|
||||
(default `10`, maximum `100`) and `include_page_content` (default `false`). Each
|
||||
matched vector section is returned as its own ranked result, including the
|
||||
matched text, score, DeerFlow and WeKnora knowledge/Wiki ids, page metadata,
|
||||
raw source/chunk references, best-effort document/chunk ids, and a directly
|
||||
usable `wiki_url`. Set `llmwiki.local_wiki_index.frontend_base_url` to the
|
||||
deployed frontend origin/base path; its development default is
|
||||
`http://127.0.0.1:5174`.
|
||||
|
||||
See `docs/LLMWIKI_DEERFLOW_LOCAL_VECTOR_INDEX_ZH.md` for configuration,
|
||||
security boundaries, rollout, and API contracts.
|
||||
|
||||
|
||||
@@ -24,8 +24,9 @@ from app.gateway.auth.disabled_mode import is_auth_disabled # noqa: E402
|
||||
from app.gateway.auth_middleware import AuthMiddleware
|
||||
from app.gateway.config import get_gateway_config
|
||||
from app.gateway.csrf_middleware import CSRFMiddleware
|
||||
from app.gateway.deps import langgraph_runtime
|
||||
from app.gateway.routers import (
|
||||
from app.gateway.deps import langgraph_runtime
|
||||
from app.gateway.public_knowledge_cors import PublicKnowledgeCORSMiddleware
|
||||
from app.gateway.routers import (
|
||||
ai_writing,
|
||||
admin_active_runs,
|
||||
admin_users,
|
||||
@@ -66,8 +67,9 @@ from app.gateway.routers import (
|
||||
positions,
|
||||
position_roles,
|
||||
public_config,
|
||||
public_embed,
|
||||
public_leaderboard,
|
||||
public_embed,
|
||||
public_knowledge_vector_search,
|
||||
public_leaderboard,
|
||||
public_scheduled_tasks,
|
||||
recommend,
|
||||
flow_extract,
|
||||
@@ -1488,11 +1490,16 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
|
||||
"Set GATEWAY_CORS_ORIGINS or GATEWAY_CORS_ALLOW_ALL for deployment.",
|
||||
", ".join(dev_origins),
|
||||
)
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=dev_origins,
|
||||
**cors_kwargs,
|
||||
)
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=dev_origins,
|
||||
**cors_kwargs,
|
||||
)
|
||||
|
||||
# This one anonymous API is intentionally callable by browser code from
|
||||
# every origin. Keep the override route-scoped so authenticated APIs retain
|
||||
# the deployment's normal CORS allow-list.
|
||||
app.add_middleware(PublicKnowledgeCORSMiddleware)
|
||||
|
||||
# Include routers
|
||||
# Models API is mounted at /api/models
|
||||
@@ -1644,8 +1651,11 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
|
||||
app.include_router(public_config.router)
|
||||
app.include_router(html_page_favorites.router)
|
||||
|
||||
# Product knowledge base API is mounted at /api/knowledge
|
||||
app.include_router(knowledge.router)
|
||||
# Product knowledge base API is mounted at /api/knowledge
|
||||
app.include_router(knowledge.router)
|
||||
# Anonymous, section-level vector search over every published Wiki base
|
||||
# except the system conversation-deposit base.
|
||||
app.include_router(public_knowledge_vector_search.router)
|
||||
|
||||
# External knowledge ingest (publish md artifacts) at /api/knowledge-ingest
|
||||
app.include_router(knowledge_ingest.router)
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
# Word 导出字体资源
|
||||
|
||||
`方正小标宋简体.ttf` 是 Word 正式格式导出的标题字体，由部署方提供并授权本系统用于文档嵌入。
|
||||
|
||||
- 字体家族：方正小标宋简体
|
||||
- SHA-256：`C52D577DD3AA719EF5FB9FF2043B7F7CA36FAE468EF34A47628A3A555D006A49`
|
||||
- OpenType `OS/2.fsType`：`0x0008`（允许可编辑嵌入）
|
||||
|
||||
后端启动后不会把字体安装到操作系统；导出时会将字体以 OOXML 规定的混淆字体部件嵌入 `.docx`。因此下载者电脑无需预装此字体，也不会触发客户端字体回退。
|
||||
|
||||
如部署时需要从外部密钥卷提供字体，可设置 `DEERFLOW_WORD_TITLE_FONT_PATH` 指向同一字体文件。后端仍会校验字体内部家族名和嵌入许可。
|
||||
Binary file not shown.
@@ -68,9 +68,13 @@ _PUBLIC_EXACT_PATHS: frozenset[str] = frozenset(
|
||||
"/api/v1/auth/login/token",
|
||||
"/api/v1/auth/register",
|
||||
"/api/v1/auth/logout",
|
||||
"/api/v1/auth/setup-status",
|
||||
"/api/v1/auth/initialize",
|
||||
# 并行多智能体面板专用的「绕过常规登录」取 token 端点（内网部署限制了标准登录时用）。
|
||||
"/api/v1/auth/setup-status",
|
||||
"/api/v1/auth/initialize",
|
||||
# Anonymous vector search over published local Wiki indexes.
|
||||
"/api/knowledge/vector-search",
|
||||
# Stateless Markdown -> DOCX conversion; required by public report pages.
|
||||
"/api/writing/export/docx",
|
||||
# 并行多智能体面板专用的「绕过常规登录」取 token 端点（内网部署限制了标准登录时用）。
|
||||
"/api/parallel-agents/auth/token",
|
||||
}
|
||||
)
|
||||
|
||||
@@ -4,9 +4,14 @@ Per RFC-001:
|
||||
State-changing operations require CSRF protection.
|
||||
"""
|
||||
|
||||
import secrets
|
||||
from collections.abc import Callable
|
||||
|
||||
import secrets
|
||||
from collections.abc import Callable
|
||||
|
||||
from fastapi import Request, Response
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.responses import JSONResponse
|
||||
from starlette.types import ASGIApp
|
||||
|
||||
from app.gateway.auth.disabled_mode import is_auth_disabled
|
||||
from app.gateway.proxy_path import app_relative_path
|
||||
from app.gateway.weknora_embed import (
|
||||
@@ -14,11 +19,7 @@ from app.gateway.weknora_embed import (
|
||||
has_valid_weknora_embed_cookie,
|
||||
is_weknora_embed_proxy_path,
|
||||
)
|
||||
from fastapi import Request, Response
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.responses import JSONResponse
|
||||
from starlette.types import ASGIApp
|
||||
|
||||
|
||||
CSRF_COOKIE_NAME = "csrf_token"
|
||||
CSRF_HEADER_NAME = "X-CSRF-Token"
|
||||
CSRF_TOKEN_LENGTH = 64 # bytes
|
||||
@@ -48,8 +49,12 @@ def should_check_csrf(request: Request) -> bool:
|
||||
path = app_relative_path(request)
|
||||
stripped = path.rstrip("/")
|
||||
# Exempt /api/v1/auth/me endpoint
|
||||
if stripped == "/api/v1/auth/me":
|
||||
return False
|
||||
if stripped == "/api/v1/auth/me":
|
||||
return False
|
||||
if stripped == "/api/knowledge/vector-search":
|
||||
return False
|
||||
if stripped == "/api/writing/export/docx":
|
||||
return False
|
||||
# Anonymous public endpoints (mirror auth_middleware._PUBLIC_PATH_PREFIXES).
|
||||
# These intentionally have no cookie session, so the Double-Submit Cookie
|
||||
# pattern is both impossible (no token to compare) and unnecessary (no
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
"""Route-scoped permissive CORS for the anonymous knowledge vector API."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable
|
||||
|
||||
from fastapi import Request, Response
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.types import ASGIApp
|
||||
|
||||
from app.gateway.proxy_path import app_relative_path
|
||||
|
||||
PUBLIC_KNOWLEDGE_VECTOR_SEARCH_PATH = "/api/knowledge/vector-search"
|
||||
|
||||
|
||||
def is_public_knowledge_vector_search(request: Request) -> bool:
|
||||
return app_relative_path(request).rstrip("/") == PUBLIC_KNOWLEDGE_VECTOR_SEARCH_PATH
|
||||
|
||||
|
||||
def _apply_public_cors(response: Response) -> Response:
|
||||
response.headers["Access-Control-Allow-Origin"] = "*"
|
||||
response.headers["Access-Control-Allow-Methods"] = "POST, OPTIONS"
|
||||
response.headers["Access-Control-Allow-Headers"] = "*"
|
||||
response.headers["Access-Control-Allow-Private-Network"] = "true"
|
||||
response.headers["Access-Control-Max-Age"] = "600"
|
||||
if "Access-Control-Allow-Credentials" in response.headers:
|
||||
del response.headers["Access-Control-Allow-Credentials"]
|
||||
return response
|
||||
|
||||
|
||||
class PublicKnowledgeCORSMiddleware(BaseHTTPMiddleware):
|
||||
"""Allow every browser origin for one credential-free public route only."""
|
||||
|
||||
def __init__(self, app: ASGIApp) -> None:
|
||||
super().__init__(app)
|
||||
|
||||
async def dispatch(self, request: Request, call_next: Callable) -> Response:
|
||||
if not is_public_knowledge_vector_search(request):
|
||||
return await call_next(request)
|
||||
if request.method.upper() == "OPTIONS":
|
||||
return _apply_public_cors(Response(status_code=204))
|
||||
return _apply_public_cors(await call_next(request))
|
||||
|
||||
|
||||
__all__ = [
|
||||
"PUBLIC_KNOWLEDGE_VECTOR_SEARCH_PATH",
|
||||
"PublicKnowledgeCORSMiddleware",
|
||||
"is_public_knowledge_vector_search",
|
||||
]
|
||||
@@ -17,8 +17,8 @@ from pydantic import BaseModel, Field
|
||||
|
||||
from app.gateway.deps import get_current_user, get_optional_user_from_request
|
||||
from deerflow.persistence.compact_menu_overrides import (
|
||||
CompactMenuOverrideStore,
|
||||
DEFAULT_COMPACT_MENU_TYPE,
|
||||
CompactMenuOverrideStore,
|
||||
normalize_compact_menu_type,
|
||||
)
|
||||
|
||||
@@ -27,7 +27,7 @@ router = APIRouter(prefix="/api/compact-menu-overrides", tags=["compact-menu-ove
|
||||
|
||||
class CompactMenuOverride(BaseModel):
|
||||
node_id: str = Field(..., min_length=1, max_length=128)
|
||||
placement: Literal["sidebar", "settings"] = "sidebar"
|
||||
placement: Literal["sidebar", "settings", "toolbox"] = "sidebar"
|
||||
sort_order: int = 0
|
||||
label: str | None = Field(default=None, max_length=255)
|
||||
|
||||
|
||||
@@ -1151,35 +1151,15 @@ html.deerflow-source-detail-open [data-deerflow-source-detail-panel="true"] [cla
|
||||
if (!proxyUrl) return;
|
||||
if (
|
||||
image.dataset.deerflowProtectedSource === protectedSrc &&
|
||||
String(image.getAttribute("src") || "").startsWith("blob:")
|
||||
String(image.getAttribute("src") || "") === proxyUrl
|
||||
) return;
|
||||
const attempts = Number(image.dataset.deerflowProtectedAttempts || "0");
|
||||
if (attempts >= 3 || image.dataset.deerflowProtectedLoading === "1") return;
|
||||
image.dataset.deerflowProtectedLoading = "1";
|
||||
image.dataset.deerflowProtectedAttempts = String(attempts + 1);
|
||||
nativeFetch(proxyUrl, {{ credentials: "same-origin", headers: {{ Accept: "image/*" }} }})
|
||||
.then((response) => {{
|
||||
if (!response.ok) throw new Error(`protected image request failed (${{response.status}})`);
|
||||
return response.blob();
|
||||
}})
|
||||
.then((blob) => {{
|
||||
const objectUrl = URL.createObjectURL(blob);
|
||||
image.dataset.deerflowProtectedSource = protectedSrc;
|
||||
image.removeAttribute("data-img-loading");
|
||||
image.removeAttribute("data-wiki-image-error");
|
||||
image.src = objectUrl;
|
||||
image.addEventListener("load", () => URL.revokeObjectURL(objectUrl), {{ once: true }});
|
||||
}})
|
||||
.catch(() => {{
|
||||
image.dataset.wikiImageError = "true";
|
||||
window.setTimeout(() => {{
|
||||
delete image.dataset.deerflowProtectedLoading;
|
||||
hydrateProtectedImage(image);
|
||||
}}, 1200 * (attempts + 1));
|
||||
}})
|
||||
.finally(() => {{
|
||||
delete image.dataset.deerflowProtectedLoading;
|
||||
}});
|
||||
// Keep the real HTTP proxy URL on the DOM node. WeKnora's preview/lightbox
|
||||
// reuses the thumbnail's src when the user clicks it; a short-lived blob URL
|
||||
// may render once and then become invalid before the preview opens.
|
||||
image.dataset.deerflowProtectedSource = protectedSrc;
|
||||
image.removeAttribute("data-img-loading");
|
||||
image.removeAttribute("data-wiki-image-error");
|
||||
image.src = proxyUrl;
|
||||
}}
|
||||
function hydrateProtectedImages(scope = document) {{
|
||||
if (scope?.matches?.("img[data-protected-src]")) hydrateProtectedImage(scope);
|
||||
@@ -1197,7 +1177,7 @@ html.deerflow-source-detail-open [data-deerflow-source-detail-panel="true"] [cla
|
||||
childList: true,
|
||||
subtree: true,
|
||||
attributes: true,
|
||||
attributeFilter: ["data-protected-src"]
|
||||
attributeFilter: ["data-protected-src", "src"]
|
||||
}});
|
||||
hydrateProtectedImages();
|
||||
|
||||
|
||||
+152
@@ -0,0 +1,152 @@
|
||||
"""Anonymous vector search over published, locally indexed WeKnora Wiki data."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
from urllib.parse import quote, urlencode
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from deerflow.integrations.weknora.local_index.search import WikiIndexError
|
||||
|
||||
router = APIRouter(tags=["public-knowledge-vector-search"])
|
||||
|
||||
|
||||
class PublicKnowledgeVectorSearchRequest(BaseModel):
|
||||
query: str = Field(..., min_length=1, max_length=10_000)
|
||||
top_k: int = Field(default=10, ge=1, le=100)
|
||||
include_page_content: bool = False
|
||||
|
||||
|
||||
def _is_conversation_deposit(mapping: dict[str, Any]) -> bool:
|
||||
return str(mapping.get("owner_user_id") or "") == "system" and str(mapping.get("name") or "").strip() == "对话沉淀"
|
||||
|
||||
|
||||
async def _published_mappings(request: Request) -> list[dict[str, Any]]:
|
||||
store = getattr(request.app.state, "llmwiki_store", None)
|
||||
if store is None:
|
||||
raise HTTPException(status_code=503, detail={"code": "LLMWIKI_STORE_UNAVAILABLE"})
|
||||
rows = await store.list_visible("system", scope="public", is_admin=False)
|
||||
return [row for row in rows if row.get("publication_status") == "published" and not _is_conversation_deposit(row)]
|
||||
|
||||
|
||||
def _unique_strings(values: list[str]) -> list[str]:
|
||||
return list(dict.fromkeys(value for value in values if value))
|
||||
|
||||
|
||||
def _reference_metadata(item: dict[str, Any]) -> dict[str, Any]:
|
||||
source_refs = item.get("source_refs") if isinstance(item.get("source_refs"), list) else []
|
||||
chunk_refs = item.get("chunk_refs") if isinstance(item.get("chunk_refs"), list) else []
|
||||
document_ids: list[str] = []
|
||||
document_names: list[str] = []
|
||||
chunk_ids: list[str] = []
|
||||
source_ref_ids: list[str] = []
|
||||
|
||||
for ref in source_refs:
|
||||
if isinstance(ref, dict):
|
||||
document_id = str(ref.get("document_id") or ref.get("knowledge_id") or ref.get("documentId") or ref.get("knowledgeId") or "").strip()
|
||||
document_name = str(
|
||||
ref.get("document_name") or ref.get("knowledge_title") or ref.get("documentName") or ref.get("knowledgeTitle") or ref.get("file_name") or ref.get("filename") or ref.get("title") or ref.get("name") or ""
|
||||
).strip()
|
||||
chunk_id = str(ref.get("chunk_id") or ref.get("chunkId") or "").strip()
|
||||
if document_id:
|
||||
document_ids.append(document_id)
|
||||
if document_name:
|
||||
document_names.append(document_name)
|
||||
if chunk_id:
|
||||
chunk_ids.append(chunk_id)
|
||||
elif isinstance(ref, (str, int)) and str(ref).strip():
|
||||
source_ref_ids.append(str(ref).strip())
|
||||
|
||||
for ref in chunk_refs:
|
||||
if isinstance(ref, dict):
|
||||
chunk_id = str(ref.get("chunk_id") or ref.get("chunkId") or ref.get("id") or "").strip()
|
||||
else:
|
||||
chunk_id = str(ref or "").strip()
|
||||
if chunk_id:
|
||||
chunk_ids.append(chunk_id)
|
||||
|
||||
document_ids = _unique_strings(document_ids)
|
||||
document_names = _unique_strings(document_names)
|
||||
chunk_ids = _unique_strings(chunk_ids)
|
||||
source_ref_ids = _unique_strings(source_ref_ids)
|
||||
return {
|
||||
"document_id": document_ids[0] if document_ids else None,
|
||||
"document_name": document_names[0] if document_names else None,
|
||||
"document_ids": document_ids,
|
||||
"document_names": document_names,
|
||||
"chunk_id": chunk_ids[0] if chunk_ids else None,
|
||||
"chunk_ids": chunk_ids,
|
||||
"source_ref_ids": source_ref_ids,
|
||||
}
|
||||
|
||||
|
||||
def _wiki_link(frontend_base_url: str, mapping_id: str, wiki_slug: str) -> tuple[str, str]:
|
||||
query = urlencode({"wikiSlug": wiki_slug})
|
||||
route = f"/#/page/workspace/knowledge/{quote(mapping_id, safe='')}/wiki/view?{query}"
|
||||
return route, f"{frontend_base_url.rstrip('/')}{route}"
|
||||
|
||||
|
||||
@router.post("/api/knowledge/vector-search")
|
||||
async def public_knowledge_vector_search(
|
||||
request: Request,
|
||||
body: PublicKnowledgeVectorSearchRequest,
|
||||
) -> dict[str, Any]:
|
||||
mappings = await _published_mappings(request)
|
||||
search = getattr(request.app.state, "llmwiki_vector_search", None)
|
||||
if search is None:
|
||||
raise HTTPException(status_code=503, detail={"code": "WIKI_INDEX_UNAVAILABLE"})
|
||||
try:
|
||||
payload = await search.search_sections(
|
||||
body.query,
|
||||
mappings,
|
||||
top_k_sections=body.top_k,
|
||||
external=True,
|
||||
)
|
||||
except WikiIndexError as exc:
|
||||
raise HTTPException(
|
||||
status_code=exc.status_code,
|
||||
detail={"code": exc.code, "message": str(exc)},
|
||||
) from None
|
||||
|
||||
# Publication can change while an embedding/search call is in flight. A
|
||||
# second read ensures an unpublished or newly reserved result is never
|
||||
# emitted by this anonymous endpoint.
|
||||
still_published = {str(row["id"]) for row in await _published_mappings(request)}
|
||||
frontend_base_url = request.app.state.config.llmwiki.local_wiki_index.frontend_base_url
|
||||
results: list[dict[str, Any]] = []
|
||||
for raw_item in payload.get("results") or []:
|
||||
if str(raw_item.get("knowledge_base_id") or "") not in still_published:
|
||||
continue
|
||||
item = dict(raw_item)
|
||||
wiki_route, wiki_url = _wiki_link(
|
||||
frontend_base_url,
|
||||
str(item["knowledge_base_id"]),
|
||||
str(item["wiki_slug"]),
|
||||
)
|
||||
item.update(_reference_metadata(item))
|
||||
item["wiki_route"] = wiki_route
|
||||
item["wiki_url"] = wiki_url
|
||||
if not body.include_page_content:
|
||||
item.pop("page_content", None)
|
||||
results.append(item)
|
||||
|
||||
return {
|
||||
"query": body.query,
|
||||
"top_k": body.top_k,
|
||||
"mode": "vector",
|
||||
"results": results,
|
||||
"count": len(results),
|
||||
"knowledge_base_count": payload.get("knowledge_base_count", len(mappings)),
|
||||
"index_fingerprint": payload.get("index_fingerprint"),
|
||||
"partial": bool(payload.get("partial", False)),
|
||||
"warnings": payload.get("warnings") or [],
|
||||
}
|
||||
|
||||
|
||||
__all__ = [
|
||||
"PublicKnowledgeVectorSearchRequest",
|
||||
"public_knowledge_vector_search",
|
||||
"router",
|
||||
]
|
||||
@@ -15,11 +15,13 @@ import hashlib
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from urllib.parse import quote
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request
|
||||
from fastapi.responses import StreamingResponse
|
||||
from fastapi.responses import Response, StreamingResponse
|
||||
from langchain_core.messages import AIMessage, HumanMessage, SystemMessage
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
@@ -57,6 +59,12 @@ from app.gateway.document_rewrite_pipeline import (
|
||||
)
|
||||
from app.gateway.document_rewrite_summary import build_document_rewrite_summary
|
||||
from app.gateway.path_utils import aresolve_thread_virtual_path
|
||||
from app.gateway.word_export import (
|
||||
DOCX_MEDIA_TYPE,
|
||||
WordExportError,
|
||||
WordExportFontError,
|
||||
build_markdown_docx,
|
||||
)
|
||||
from deerflow.config.app_config import AppConfig
|
||||
from deerflow.models import create_chat_model
|
||||
from deerflow.utils.stream_text import InlineThinkTagFilter
|
||||
@@ -130,6 +138,13 @@ class RewriteRequest(BaseModel):
|
||||
model_config = {"populate_by_name": True}
|
||||
|
||||
|
||||
class MarkdownDocxExportRequest(BaseModel):
|
||||
"""Markdown source for the server-rendered formal Word download."""
|
||||
|
||||
title: str | None = Field(default=None, max_length=300)
|
||||
markdown: str = Field(..., min_length=1, max_length=2_000_000)
|
||||
|
||||
|
||||
class DocumentRewriteRequest(BaseModel):
|
||||
"""Request for a complete Markdown artifact rewrite.
|
||||
|
||||
@@ -363,6 +378,40 @@ def build_rewrite_messages(body: RewriteRequest) -> tuple[str, list]:
|
||||
# ── Endpoint ──────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.post(
|
||||
"/export/docx",
|
||||
summary="Export Markdown as a formally formatted Word document",
|
||||
response_class=Response,
|
||||
)
|
||||
async def export_markdown_docx(body: MarkdownDocxExportRequest) -> Response:
|
||||
"""Generate DOCX on the server and embed the required title font.
|
||||
|
||||
This route is intentionally usable by both authenticated workspaces and
|
||||
public scheduled-task pages. It performs no external fetches and does not
|
||||
persist the supplied Markdown.
|
||||
"""
|
||||
try:
|
||||
content = await asyncio.to_thread(build_markdown_docx, body.markdown, body.title)
|
||||
except WordExportFontError as exc:
|
||||
raise HTTPException(status_code=503, detail=str(exc)) from exc
|
||||
except WordExportError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
|
||||
fallback_name = (body.title or "document").strip() or "document"
|
||||
fallback_name = re.sub(r'[<>:"/\\|?*\x00-\x1f]+', "_", fallback_name).strip(" .")[:120] or "document"
|
||||
filename = f"{fallback_name}.docx"
|
||||
return Response(
|
||||
content=content,
|
||||
media_type=DOCX_MEDIA_TYPE,
|
||||
headers={
|
||||
"Content-Disposition": (
|
||||
f'attachment; filename="document.docx"; filename*=UTF-8\'\'{quote(filename, safe="")}'
|
||||
),
|
||||
"Cache-Control": "no-store",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.post("/document-rewrite/jobs")
|
||||
@require_permission("threads", "write", owner_check=True)
|
||||
async def start_durable_document_rewrite_job(
|
||||
|
||||
@@ -0,0 +1,994 @@
|
||||
"""Deterministic Markdown -> DOCX export with an embedded title font.
|
||||
|
||||
The implementation intentionally uses Python's standard library only. It
|
||||
builds the small set of OOXML parts required by Word instead of depending on a
|
||||
desktop Office installation or a document-conversion service. The bundled
|
||||
方正小标宋简体 font is validated and embedded according to ECMA-376 so the
|
||||
recipient's computer does not need the font installed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import io
|
||||
import os
|
||||
import re
|
||||
import struct
|
||||
import uuid
|
||||
import zipfile
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from xml.sax.saxutils import escape
|
||||
|
||||
DOCX_MEDIA_TYPE = "application/vnd.openxmlformats-officedocument.wordprocessingml.document"
|
||||
|
||||
TITLE_FONT_FAMILY = "方正小标宋简体"
|
||||
BODY_FONT_FAMILY = "仿宋"
|
||||
HEADING_1_FONT_FAMILY = "黑体"
|
||||
HEADING_2_FONT_FAMILY = "楷体"
|
||||
TABLE_FONT_FAMILY = "宋体"
|
||||
|
||||
TITLE_SIZE_HALF_POINTS = 44 # 二号 / 22 pt
|
||||
BODY_SIZE_HALF_POINTS = 32 # 三号 / 16 pt
|
||||
TABLE_SIZE_HALF_POINTS = 28 # 四号 / 14 pt
|
||||
EXACT_LINE_TWIPS = 579 # 28.95 pt * 20
|
||||
|
||||
A4_WIDTH_TWIPS = 11_906
|
||||
A4_HEIGHT_TWIPS = 16_838
|
||||
MARGIN_TOP_TWIPS = 2_098 # 3.7 cm
|
||||
MARGIN_BOTTOM_TWIPS = 1_984 # 3.5 cm
|
||||
MARGIN_LEFT_TWIPS = 1_587 # 2.8 cm
|
||||
MARGIN_RIGHT_TWIPS = 1_474 # 2.6 cm
|
||||
CONTENT_WIDTH_TWIPS = A4_WIDTH_TWIPS - MARGIN_LEFT_TWIPS - MARGIN_RIGHT_TWIPS
|
||||
|
||||
_XML_DECL = '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
|
||||
_W_NS = "http://schemas.openxmlformats.org/wordprocessingml/2006/main"
|
||||
_R_NS = "http://schemas.openxmlformats.org/officeDocument/2006/relationships"
|
||||
_PACKAGE_REL_NS = "http://schemas.openxmlformats.org/package/2006/relationships"
|
||||
|
||||
_FONT_DIR = Path(__file__).resolve().parent / "assets" / "fonts"
|
||||
_BUNDLED_FONT_PATH = _FONT_DIR / "方正小标宋简体.ttf"
|
||||
_BUNDLED_FONT_SHA256 = "C52D577DD3AA719EF5FB9FF2043B7F7CA36FAE468EF34A47628A3A555D006A49"
|
||||
|
||||
|
||||
class WordExportError(RuntimeError):
|
||||
"""Base error returned to the Word export API."""
|
||||
|
||||
|
||||
class WordExportFontError(WordExportError):
|
||||
"""The required server-side font is missing, invalid, or not embeddable."""
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class FontAsset:
|
||||
data: bytes
|
||||
path: Path
|
||||
names: frozenset[str]
|
||||
fs_type: int
|
||||
sha256: str
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class ParagraphBlock:
|
||||
kind: str
|
||||
text: str
|
||||
level: int = 0
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class TableBlock:
|
||||
header: list[str]
|
||||
rows: list[list[str]]
|
||||
|
||||
|
||||
Block = ParagraphBlock | TableBlock
|
||||
|
||||
|
||||
def _safe_xml_text(value: str) -> str:
|
||||
"""Remove XML-illegal controls and invisible Word-breaking selectors."""
|
||||
value = re.sub(r"[\u0000-\u0008\u000b\u000c\u000e-\u001f]", "", value)
|
||||
value = re.sub(r"[\u200b-\u200d\ufeff\ufe00-\ufe0f]", "", value)
|
||||
value = value.replace("\u00ad", "").replace("\ufffc", "")
|
||||
value = re.sub(r"</?(?:details|summary)\b[^>]*>", "", value, flags=re.I)
|
||||
return value
|
||||
|
||||
|
||||
def _xml_attr(value: str) -> str:
|
||||
return escape(_safe_xml_text(value), {'"': """, "'": "'"})
|
||||
|
||||
|
||||
def _decode_name_record(platform_id: int, raw: bytes) -> str | None:
|
||||
try:
|
||||
if platform_id in {0, 3}:
|
||||
return raw.decode("utf-16-be").strip("\x00")
|
||||
if platform_id == 1:
|
||||
return raw.decode("mac_roman").strip("\x00")
|
||||
except (UnicodeDecodeError, LookupError):
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def _read_sfnt_metadata(data: bytes) -> tuple[frozenset[str], int]:
|
||||
if len(data) < 12:
|
||||
raise WordExportFontError("Word 标题字体文件不是有效的 OpenType/TrueType 字体。")
|
||||
try:
|
||||
table_count = struct.unpack_from(">H", data, 4)[0]
|
||||
tables: dict[str, tuple[int, int]] = {}
|
||||
for index in range(table_count):
|
||||
position = 12 + index * 16
|
||||
tag, _checksum, offset, length = struct.unpack_from(">4sIII", data, position)
|
||||
tag_text = tag.decode("latin-1")
|
||||
if offset + length > len(data):
|
||||
raise WordExportFontError("Word 标题字体的表目录已损坏。")
|
||||
tables[tag_text] = (offset, length)
|
||||
|
||||
if "OS/2" not in tables or "name" not in tables:
|
||||
raise WordExportFontError("Word 标题字体缺少 OS/2 或 name 元数据表。")
|
||||
|
||||
os2_offset, os2_length = tables["OS/2"]
|
||||
if os2_length < 10:
|
||||
raise WordExportFontError("Word 标题字体的 OS/2 元数据无效。")
|
||||
fs_type = struct.unpack_from(">H", data, os2_offset + 8)[0]
|
||||
|
||||
name_offset, name_length = tables["name"]
|
||||
if name_length < 6:
|
||||
raise WordExportFontError("Word 标题字体的 name 元数据无效。")
|
||||
name_count, string_offset = struct.unpack_from(">HH", data, name_offset + 2)
|
||||
names: set[str] = set()
|
||||
for index in range(name_count):
|
||||
record_offset = name_offset + 6 + index * 12
|
||||
if record_offset + 12 > name_offset + name_length:
|
||||
break
|
||||
platform_id, _encoding, _language, name_id, length, offset = struct.unpack_from(
|
||||
">HHHHHH", data, record_offset
|
||||
)
|
||||
if name_id not in {1, 4, 6}:
|
||||
continue
|
||||
value_start = name_offset + string_offset + offset
|
||||
value_end = value_start + length
|
||||
if value_end > name_offset + name_length:
|
||||
continue
|
||||
decoded = _decode_name_record(platform_id, data[value_start:value_end])
|
||||
if decoded:
|
||||
names.add(decoded)
|
||||
return frozenset(names), fs_type
|
||||
except struct.error as exc:
|
||||
raise WordExportFontError("Word 标题字体文件结构不完整。") from exc
|
||||
|
||||
|
||||
def resolve_title_font_path() -> Path:
|
||||
configured = os.getenv("DEERFLOW_WORD_TITLE_FONT_PATH", "").strip()
|
||||
return Path(configured).expanduser() if configured else _BUNDLED_FONT_PATH
|
||||
|
||||
|
||||
def load_title_font(path: Path | None = None) -> FontAsset:
|
||||
font_path = (path or resolve_title_font_path()).resolve()
|
||||
if not font_path.is_file():
|
||||
raise WordExportFontError(
|
||||
f"服务器缺少“{TITLE_FONT_FAMILY}”字体资源：{font_path}。"
|
||||
"请将已授权字体放入后端字体目录后再导出。"
|
||||
)
|
||||
data = font_path.read_bytes()
|
||||
if not data or len(data) > 25 * 1024 * 1024:
|
||||
raise WordExportFontError("Word 标题字体文件为空或超过 25 MB 限制。")
|
||||
|
||||
names, fs_type = _read_sfnt_metadata(data)
|
||||
accepted_name = TITLE_FONT_FAMILY in names or any("FZXiaoBiaoSong" in name for name in names)
|
||||
if not accepted_name:
|
||||
readable_names = "、".join(sorted(names)[:6]) or "无法读取"
|
||||
raise WordExportFontError(
|
||||
f"服务器字体不是“{TITLE_FONT_FAMILY}”（检测到：{readable_names}），已阻止导出。"
|
||||
)
|
||||
if fs_type & 0x0002:
|
||||
raise WordExportFontError(f"“{TITLE_FONT_FAMILY}”授权标志禁止文档嵌入，已阻止导出。")
|
||||
if fs_type & 0x0200:
|
||||
raise WordExportFontError(f"“{TITLE_FONT_FAMILY}”仅允许位图嵌入，不能用于本次 Word 导出。")
|
||||
|
||||
digest = hashlib.sha256(data).hexdigest().upper()
|
||||
if font_path == _BUNDLED_FONT_PATH.resolve() and digest != _BUNDLED_FONT_SHA256:
|
||||
raise WordExportFontError("代码仓库中的 Word 标题字体校验和不匹配，已阻止导出。")
|
||||
return FontAsset(data=data, path=font_path, names=names, fs_type=fs_type, sha256=digest)
|
||||
|
||||
|
||||
def _obfuscate_font(data: bytes, font_key: str) -> bytes:
|
||||
"""Apply the ECMA-376 ODTTF XOR obfuscation to the first 32 bytes."""
|
||||
key_hex = font_key.replace("-", "")
|
||||
if len(key_hex) != 32:
|
||||
raise WordExportFontError("生成 Word 嵌入字体密钥失败。")
|
||||
key = bytes.fromhex(key_hex)[::-1]
|
||||
output = bytearray(data)
|
||||
for index in range(min(32, len(output))):
|
||||
output[index] ^= key[index % len(key)]
|
||||
return bytes(output)
|
||||
|
||||
|
||||
def _strip_inline_markdown(text: str) -> str:
|
||||
text = _safe_xml_text(text)
|
||||
text = re.sub(r"^\s*#+\s*", "", text)
|
||||
text = re.sub(r"!\[([^\]]*)\]\([^)]*\)", r"\1", text)
|
||||
text = re.sub(r"\[([^\]]+)\]\([^)]*\)", r"\1", text)
|
||||
text = re.sub(r"(?:\*\*|__)(.*?)(?:\*\*|__)", r"\1", text)
|
||||
return re.sub(r"[`*_~]", "", text).strip()
|
||||
|
||||
|
||||
def _strip_manual_heading_number(text: str) -> str:
|
||||
return re.sub(
|
||||
r"^\s*(?:[一二三四五六七八九十百千]+、|[（(][一二三四五六七八九十百千]+[）)]|\d+[.、]|[（(]\d+[）)]|\d+[）)])\s*",
|
||||
"",
|
||||
text,
|
||||
).strip()
|
||||
|
||||
|
||||
def _split_table_row(line: str) -> list[str]:
|
||||
value = re.sub(r"^\s*\|", "", line)
|
||||
value = re.sub(r"\|\s*$", "", value)
|
||||
return [part.replace(r"\|", "|").strip() for part in re.split(r"(?<!\\)\|", value)]
|
||||
|
||||
|
||||
def _is_table_separator(line: str) -> bool:
|
||||
cells = _split_table_row(line)
|
||||
return bool(cells) and all(re.fullmatch(r":?-{3,}:?", cell.replace(" ", "")) for cell in cells)
|
||||
|
||||
|
||||
def _looks_like_table_row(line: str) -> bool:
|
||||
return bool(re.match(r"^\s*\|.+", line))
|
||||
|
||||
|
||||
def _is_table_caption(text: str) -> bool:
|
||||
return bool(re.match(r"^\s*表\s*(?:\d+|[一二三四五六七八九十百]+|[：:])", _strip_inline_markdown(text)))
|
||||
|
||||
|
||||
def _extract_title(markdown: str, explicit_title: str | None) -> tuple[str | None, list[str]]:
|
||||
lines = markdown.replace("\r\n", "\n").replace("\r", "\n").split("\n")
|
||||
in_fence = False
|
||||
for index, line in enumerate(lines):
|
||||
if line.strip().startswith("```"):
|
||||
in_fence = not in_fence
|
||||
continue
|
||||
if not in_fence:
|
||||
match = re.match(r"^#\s+(.+)$", line)
|
||||
if match:
|
||||
del lines[index]
|
||||
return _strip_inline_markdown(match.group(1)) or (explicit_title or "").strip() or None, lines
|
||||
return (explicit_title or "").strip() or None, lines
|
||||
|
||||
|
||||
def _parse_markdown(markdown: str, explicit_title: str | None) -> tuple[str | None, list[Block]]:
|
||||
title, lines = _extract_title(markdown, explicit_title)
|
||||
blocks: list[Block] = []
|
||||
paragraph_buffer: list[str] = []
|
||||
in_fence = False
|
||||
|
||||
def flush_paragraph() -> None:
|
||||
if paragraph_buffer:
|
||||
blocks.append(ParagraphBlock("body", " ".join(paragraph_buffer)))
|
||||
paragraph_buffer.clear()
|
||||
|
||||
index = 0
|
||||
while index < len(lines):
|
||||
raw = lines[index].replace("\t", " ")
|
||||
if raw.strip().startswith("```"):
|
||||
flush_paragraph()
|
||||
in_fence = not in_fence
|
||||
index += 1
|
||||
continue
|
||||
if in_fence:
|
||||
blocks.append(ParagraphBlock("code", raw or " "))
|
||||
index += 1
|
||||
continue
|
||||
if not raw.strip():
|
||||
flush_paragraph()
|
||||
index += 1
|
||||
continue
|
||||
|
||||
heading = re.match(r"^(#{1,6})\s+(.+)$", raw)
|
||||
if heading:
|
||||
flush_paragraph()
|
||||
markdown_level = max(2, len(heading.group(1)))
|
||||
section_level = max(0, min(markdown_level - 2, 4))
|
||||
blocks.append(ParagraphBlock("heading", _strip_manual_heading_number(heading.group(2)), section_level))
|
||||
index += 1
|
||||
continue
|
||||
|
||||
if _looks_like_table_row(raw) and index + 1 < len(lines) and _is_table_separator(lines[index + 1]):
|
||||
caption: str | None = None
|
||||
if paragraph_buffer and _is_table_caption(paragraph_buffer[-1]):
|
||||
caption = paragraph_buffer.pop()
|
||||
flush_paragraph()
|
||||
if caption:
|
||||
blocks.append(ParagraphBlock("table_caption", caption))
|
||||
header = _split_table_row(raw)
|
||||
index += 2
|
||||
rows: list[list[str]] = []
|
||||
while index < len(lines) and _looks_like_table_row(lines[index]):
|
||||
rows.append(_split_table_row(lines[index]))
|
||||
index += 1
|
||||
blocks.append(TableBlock(header, rows))
|
||||
continue
|
||||
|
||||
image = re.match(r'^\s*!\[([^\]]*)\]\(([^)\s]+)(?:\s+["\'][^"\']*["\'])?\)\s*$', raw)
|
||||
if image:
|
||||
flush_paragraph()
|
||||
alt = image.group(1).strip() or "未命名图片"
|
||||
blocks.append(ParagraphBlock("image", f"图片：{alt}|{image.group(2)}"))
|
||||
index += 1
|
||||
continue
|
||||
|
||||
quote = re.match(r"^>\s?(.*)$", raw)
|
||||
if quote:
|
||||
flush_paragraph()
|
||||
blocks.append(ParagraphBlock("quote", quote.group(1)))
|
||||
index += 1
|
||||
continue
|
||||
|
||||
ordered = re.match(r"^(\s*)(\d+)[.)、]\s+(.+)$", raw)
|
||||
if ordered:
|
||||
flush_paragraph()
|
||||
blocks.append(ParagraphBlock("ordered_list", ordered.group(3), min(len(ordered.group(1)) // 2, 8)))
|
||||
index += 1
|
||||
continue
|
||||
|
||||
unordered = re.match(r"^(\s*)[-*+]\s+(.+)$", raw)
|
||||
if unordered:
|
||||
flush_paragraph()
|
||||
blocks.append(ParagraphBlock("unordered_list", unordered.group(2), min(len(unordered.group(1)) // 2, 8)))
|
||||
index += 1
|
||||
continue
|
||||
|
||||
if re.fullmatch(r"\s*(?:---|\*\*\*|___)\s*", raw):
|
||||
flush_paragraph()
|
||||
blocks.append(ParagraphBlock("horizontal_rule", "────────"))
|
||||
index += 1
|
||||
continue
|
||||
|
||||
paragraph_buffer.append(raw.strip())
|
||||
index += 1
|
||||
|
||||
flush_paragraph()
|
||||
return title, blocks
|
||||
|
||||
|
||||
class _DocumentRelationships:
|
||||
def __init__(self) -> None:
|
||||
self._hyperlinks: list[tuple[str, str]] = []
|
||||
|
||||
def add_hyperlink(self, target: str) -> str:
|
||||
relation_id = f"rId{100 + len(self._hyperlinks)}"
|
||||
self._hyperlinks.append((relation_id, target))
|
||||
return relation_id
|
||||
|
||||
def xml(self) -> str:
|
||||
fixed = [
|
||||
("rId1", "http://schemas.openxmlformats.org/officeDocument/2006/relationships/styles", "styles.xml"),
|
||||
("rId2", "http://schemas.openxmlformats.org/officeDocument/2006/relationships/numbering", "numbering.xml"),
|
||||
("rId3", "http://schemas.openxmlformats.org/officeDocument/2006/relationships/settings", "settings.xml"),
|
||||
("rId4", "http://schemas.openxmlformats.org/officeDocument/2006/relationships/fontTable", "fontTable.xml"),
|
||||
("rId5", "http://schemas.openxmlformats.org/officeDocument/2006/relationships/footer", "footer1.xml"),
|
||||
("rId6", "http://schemas.openxmlformats.org/officeDocument/2006/relationships/footer", "footer2.xml"),
|
||||
]
|
||||
relationships = "".join(
|
||||
f'<Relationship Id="{relation_id}" Type="{rel_type}" Target="{target}"/>'
|
||||
for relation_id, rel_type, target in fixed
|
||||
)
|
||||
relationships += "".join(
|
||||
f'<Relationship Id="{relation_id}" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/hyperlink" Target="{_xml_attr(target)}" TargetMode="External"/>'
|
||||
for relation_id, target in self._hyperlinks
|
||||
)
|
||||
return f'{_XML_DECL}<Relationships xmlns="{_PACKAGE_REL_NS}">{relationships}</Relationships>'
|
||||
|
||||
|
||||
_INLINE_PATTERN = re.compile(
|
||||
r"\[([^\]]+)\]\(([^)\s]+)(?:\s+[\"\'][^\"\']*[\"\'])?\)|(`+)([^`]+)\3|\*\*([^*]+)\*\*|__([^_]+)__|\*([^*]+)\*|_([^_]+)_"
|
||||
)
|
||||
|
||||
|
||||
def _run_xml(
|
||||
text: str,
|
||||
*,
|
||||
font: str = BODY_FONT_FAMILY,
|
||||
size: int = BODY_SIZE_HALF_POINTS,
|
||||
bold: bool = False,
|
||||
italic: bool = False,
|
||||
underline: bool = False,
|
||||
) -> str:
|
||||
text = _safe_xml_text(text)
|
||||
preserve = text.startswith(" ") or text.endswith(" ") or " " in text
|
||||
properties = (
|
||||
f'<w:rFonts w:ascii="{_xml_attr(font)}" w:hAnsi="{_xml_attr(font)}" '
|
||||
f'w:eastAsia="{_xml_attr(font)}" w:cs="{_xml_attr(font)}"/>'
|
||||
'<w:color w:val="000000"/>'
|
||||
f'<w:sz w:val="{size}"/><w:szCs w:val="{size}"/>'
|
||||
)
|
||||
if bold:
|
||||
properties += "<w:b/><w:bCs/>"
|
||||
if italic:
|
||||
properties += "<w:i/><w:iCs/>"
|
||||
if underline:
|
||||
properties += '<w:u w:val="single"/>'
|
||||
space = ' xml:space="preserve"' if preserve else ""
|
||||
return f"<w:r><w:rPr>{properties}</w:rPr><w:t{space}>{escape(text)}</w:t></w:r>"
|
||||
|
||||
|
||||
def _inline_xml(
|
||||
text: str,
|
||||
relationships: _DocumentRelationships,
|
||||
*,
|
||||
font: str = BODY_FONT_FAMILY,
|
||||
size: int = BODY_SIZE_HALF_POINTS,
|
||||
) -> str:
|
||||
cleaned = _safe_xml_text(text)
|
||||
output: list[str] = []
|
||||
cursor = 0
|
||||
for match in _INLINE_PATTERN.finditer(cleaned):
|
||||
if match.start() > cursor:
|
||||
output.append(_run_xml(cleaned[cursor : match.start()], font=font, size=size))
|
||||
if match.group(1) is not None and match.group(2) is not None:
|
||||
relation_id = relationships.add_hyperlink(match.group(2))
|
||||
output.append(
|
||||
f'<w:hyperlink r:id="{relation_id}" w:history="1">'
|
||||
f'{_run_xml(match.group(1), font=font, size=size, underline=True)}</w:hyperlink>'
|
||||
)
|
||||
elif match.group(4) is not None:
|
||||
output.append(_run_xml(match.group(4), font=font, size=size))
|
||||
elif match.group(5) is not None:
|
||||
output.append(_run_xml(match.group(5), font=font, size=size, bold=True))
|
||||
elif match.group(6) is not None:
|
||||
output.append(_run_xml(match.group(6), font=font, size=size, bold=True))
|
||||
elif match.group(7) is not None:
|
||||
output.append(_run_xml(match.group(7), font=font, size=size, italic=True))
|
||||
elif match.group(8) is not None:
|
||||
output.append(_run_xml(match.group(8), font=font, size=size, italic=True))
|
||||
cursor = match.end()
|
||||
if cursor < len(cleaned):
|
||||
output.append(_run_xml(cleaned[cursor:], font=font, size=size))
|
||||
return "".join(output) or _run_xml(" ", font=font, size=size)
|
||||
|
||||
|
||||
def _paragraph_xml(
|
||||
style: str,
|
||||
children: str,
|
||||
*,
|
||||
numbering_level: int | None = None,
|
||||
numbering_id: int = 1,
|
||||
keep_next: bool = False,
|
||||
alignment: str | None = None,
|
||||
) -> str:
|
||||
properties = f'<w:pStyle w:val="{style}"/>'
|
||||
if numbering_level is not None:
|
||||
properties += (
|
||||
f'<w:numPr><w:ilvl w:val="{numbering_level}"/><w:numId w:val="{numbering_id}"/></w:numPr>'
|
||||
)
|
||||
if keep_next:
|
||||
properties += "<w:keepNext/>"
|
||||
if alignment:
|
||||
properties += f'<w:jc w:val="{alignment}"/>'
|
||||
return f"<w:p><w:pPr>{properties}</w:pPr>{children}</w:p>"
|
||||
|
||||
|
||||
def _image_paragraph_xml(text: str, relationships: _DocumentRelationships) -> str:
|
||||
label, _, target = text.partition("|")
|
||||
children = _run_xml(label)
|
||||
if target:
|
||||
relation_id = relationships.add_hyperlink(target)
|
||||
children += (
|
||||
f'<w:hyperlink r:id="{relation_id}" w:history="1">'
|
||||
f'{_run_xml(f"（{target}）", underline=True)}</w:hyperlink>'
|
||||
)
|
||||
return _paragraph_xml("ImageCaption", children, alignment="center")
|
||||
|
||||
|
||||
def _cell_xml(text: str, width: int, relationships: _DocumentRelationships, *, header: bool) -> str:
|
||||
style = "TableHeader" if header else "TableBody"
|
||||
paragraph = _paragraph_xml(
|
||||
style,
|
||||
_inline_xml(text, relationships, font=TABLE_FONT_FAMILY, size=TABLE_SIZE_HALF_POINTS),
|
||||
)
|
||||
return (
|
||||
"<w:tc>"
|
||||
f'<w:tcPr><w:tcW w:w="{width}" w:type="dxa"/><w:vAlign w:val="center"/>'
|
||||
'<w:tcMar><w:top w:w="80" w:type="dxa"/><w:left w:w="100" w:type="dxa"/>'
|
||||
'<w:bottom w:w="80" w:type="dxa"/><w:right w:w="100" w:type="dxa"/></w:tcMar></w:tcPr>'
|
||||
f"{paragraph}</w:tc>"
|
||||
)
|
||||
|
||||
|
||||
def _table_xml(block: TableBlock, relationships: _DocumentRelationships) -> str:
|
||||
column_count = max(1, len(block.header))
|
||||
base_width = CONTENT_WIDTH_TWIPS // column_count
|
||||
widths = [base_width] * column_count
|
||||
widths[-1] = CONTENT_WIDTH_TWIPS - base_width * (column_count - 1)
|
||||
grid = "".join(f'<w:gridCol w:w="{width}"/>' for width in widths)
|
||||
border = (
|
||||
'<w:top w:val="single" w:sz="4" w:color="000000"/>'
|
||||
'<w:left w:val="single" w:sz="4" w:color="000000"/>'
|
||||
'<w:bottom w:val="single" w:sz="4" w:color="000000"/>'
|
||||
'<w:right w:val="single" w:sz="4" w:color="000000"/>'
|
||||
'<w:insideH w:val="single" w:sz="4" w:color="000000"/>'
|
||||
'<w:insideV w:val="single" w:sz="4" w:color="000000"/>'
|
||||
)
|
||||
properties = (
|
||||
f'<w:tblPr><w:tblW w:w="{CONTENT_WIDTH_TWIPS}" w:type="dxa"/>'
|
||||
'<w:jc w:val="center"/><w:tblLayout w:type="fixed"/>'
|
||||
f"<w:tblBorders>{border}</w:tblBorders></w:tblPr>"
|
||||
)
|
||||
|
||||
header = list(block.header[:column_count]) + [""] * max(0, column_count - len(block.header))
|
||||
header_cells = "".join(
|
||||
_cell_xml(header[index], widths[index], relationships, header=True) for index in range(column_count)
|
||||
)
|
||||
rows = [f"<w:tr><w:trPr><w:tblHeader/><w:cantSplit/></w:trPr>{header_cells}</w:tr>"]
|
||||
for source_row in block.rows:
|
||||
normalized = list(source_row[:column_count]) + [""] * max(0, column_count - len(source_row))
|
||||
cells = "".join(
|
||||
_cell_xml(normalized[index], widths[index], relationships, header=False)
|
||||
for index in range(column_count)
|
||||
)
|
||||
rows.append(f"<w:tr><w:trPr><w:cantSplit/></w:trPr>{cells}</w:tr>")
|
||||
return f"<w:tbl>{properties}<w:tblGrid>{grid}</w:tblGrid>{''.join(rows)}</w:tbl>"
|
||||
|
||||
|
||||
def _blocks_xml(blocks: list[Block], relationships: _DocumentRelationships) -> str:
|
||||
output: list[str] = []
|
||||
for block in blocks:
|
||||
if isinstance(block, TableBlock):
|
||||
output.append(_table_xml(block, relationships))
|
||||
continue
|
||||
if block.kind == "heading":
|
||||
style = f"Heading{block.level + 1}"
|
||||
font = (
|
||||
HEADING_1_FONT_FAMILY
|
||||
if block.level == 0
|
||||
else HEADING_2_FONT_FAMILY
|
||||
if block.level == 1
|
||||
else BODY_FONT_FAMILY
|
||||
)
|
||||
output.append(
|
||||
_paragraph_xml(
|
||||
style,
|
||||
_inline_xml(block.text, relationships, font=font),
|
||||
numbering_level=block.level,
|
||||
numbering_id=1,
|
||||
keep_next=True,
|
||||
)
|
||||
)
|
||||
elif block.kind == "table_caption":
|
||||
output.append(
|
||||
_paragraph_xml(
|
||||
"TableCaption",
|
||||
_inline_xml(block.text, relationships, font=HEADING_1_FONT_FAMILY, size=TABLE_SIZE_HALF_POINTS),
|
||||
keep_next=True,
|
||||
)
|
||||
)
|
||||
elif block.kind == "ordered_list":
|
||||
output.append(
|
||||
_paragraph_xml(
|
||||
"ListText",
|
||||
_inline_xml(block.text, relationships),
|
||||
numbering_level=block.level,
|
||||
numbering_id=2,
|
||||
)
|
||||
)
|
||||
elif block.kind == "unordered_list":
|
||||
output.append(
|
||||
_paragraph_xml(
|
||||
"ListText",
|
||||
_inline_xml(block.text, relationships),
|
||||
numbering_level=block.level,
|
||||
numbering_id=3,
|
||||
)
|
||||
)
|
||||
elif block.kind == "code":
|
||||
output.append(_paragraph_xml("CodeText", _run_xml(block.text)))
|
||||
elif block.kind == "quote":
|
||||
output.append(_paragraph_xml("QuoteText", _inline_xml(block.text, relationships)))
|
||||
elif block.kind == "image":
|
||||
output.append(_image_paragraph_xml(block.text, relationships))
|
||||
elif block.kind == "horizontal_rule":
|
||||
output.append(_paragraph_xml("BodyText", _run_xml(block.text), alignment="center"))
|
||||
else:
|
||||
output.append(_paragraph_xml("BodyText", _inline_xml(block.text, relationships)))
|
||||
return "".join(output)
|
||||
|
||||
|
||||
def _font_run_properties(font: str, size: int, *, bold: bool = False) -> str:
|
||||
value = (
|
||||
f'<w:rFonts w:ascii="{_xml_attr(font)}" w:hAnsi="{_xml_attr(font)}" '
|
||||
f'w:eastAsia="{_xml_attr(font)}" w:cs="{_xml_attr(font)}"/>'
|
||||
'<w:color w:val="000000"/>'
|
||||
f'<w:sz w:val="{size}"/><w:szCs w:val="{size}"/>'
|
||||
)
|
||||
return value + ("<w:b/><w:bCs/>" if bold else '<w:b w:val="0"/><w:bCs w:val="0"/>')
|
||||
|
||||
|
||||
def _style_xml(
|
||||
style_id: str,
|
||||
name: str,
|
||||
*,
|
||||
font: str,
|
||||
size: int,
|
||||
based_on: str | None = "Normal",
|
||||
next_style: str = "BodyText",
|
||||
alignment: str = "both",
|
||||
first_line_chars: int = 0,
|
||||
left: int = 0,
|
||||
hanging: int = 0,
|
||||
keep_next: bool = False,
|
||||
keep_lines: bool = False,
|
||||
outline_level: int | None = None,
|
||||
bold: bool = False,
|
||||
) -> str:
|
||||
paragraph = (
|
||||
f'<w:jc w:val="{alignment}"/><w:spacing w:before="0" w:after="0" '
|
||||
f'w:line="{EXACT_LINE_TWIPS}" w:lineRule="exact"/>'
|
||||
f'<w:ind w:left="{left}" w:hanging="{hanging}" w:firstLineChars="{first_line_chars}"/>'
|
||||
)
|
||||
if keep_next:
|
||||
paragraph += "<w:keepNext/>"
|
||||
if keep_lines:
|
||||
paragraph += "<w:keepLines/>"
|
||||
if outline_level is not None:
|
||||
paragraph += f'<w:outlineLvl w:val="{outline_level}"/>'
|
||||
based_on_xml = f'<w:basedOn w:val="{based_on}"/>' if based_on else ""
|
||||
return (
|
||||
f'<w:style w:type="paragraph" w:styleId="{style_id}"><w:name w:val="{_xml_attr(name)}"/>'
|
||||
f'{based_on_xml}<w:next w:val="{next_style}"/><w:qFormat/>'
|
||||
f"<w:pPr>{paragraph}</w:pPr><w:rPr>{_font_run_properties(font, size, bold=bold)}</w:rPr></w:style>"
|
||||
)
|
||||
|
||||
|
||||
def _styles_xml() -> str:
|
||||
defaults = (
|
||||
"<w:docDefaults><w:rPrDefault><w:rPr>"
|
||||
f"{_font_run_properties(BODY_FONT_FAMILY, BODY_SIZE_HALF_POINTS)}"
|
||||
'<w:lang w:val="zh-CN" w:eastAsia="zh-CN"/></w:rPr></w:rPrDefault>'
|
||||
'<w:pPrDefault><w:pPr><w:jc w:val="both"/>'
|
||||
f'<w:spacing w:before="0" w:after="0" w:line="{EXACT_LINE_TWIPS}" w:lineRule="exact"/>'
|
||||
'<w:ind w:firstLineChars="200"/></w:pPr></w:pPrDefault></w:docDefaults>'
|
||||
)
|
||||
styles = [
|
||||
_style_xml(
|
||||
"Normal",
|
||||
"Normal",
|
||||
font=BODY_FONT_FAMILY,
|
||||
size=BODY_SIZE_HALF_POINTS,
|
||||
based_on=None,
|
||||
next_style="BodyText",
|
||||
first_line_chars=200,
|
||||
),
|
||||
_style_xml(
|
||||
"DocumentTitle",
|
||||
"Document Title",
|
||||
font=TITLE_FONT_FAMILY,
|
||||
size=TITLE_SIZE_HALF_POINTS,
|
||||
alignment="center",
|
||||
keep_next=True,
|
||||
),
|
||||
_style_xml(
|
||||
"BodyText",
|
||||
"Body Text",
|
||||
font=BODY_FONT_FAMILY,
|
||||
size=BODY_SIZE_HALF_POINTS,
|
||||
first_line_chars=200,
|
||||
),
|
||||
_style_xml(
|
||||
"Heading1",
|
||||
"heading 1",
|
||||
font=HEADING_1_FONT_FAMILY,
|
||||
size=BODY_SIZE_HALF_POINTS,
|
||||
alignment="left",
|
||||
keep_next=True,
|
||||
keep_lines=True,
|
||||
outline_level=0,
|
||||
),
|
||||
_style_xml(
|
||||
"Heading2",
|
||||
"heading 2",
|
||||
font=HEADING_2_FONT_FAMILY,
|
||||
size=BODY_SIZE_HALF_POINTS,
|
||||
alignment="left",
|
||||
keep_next=True,
|
||||
keep_lines=True,
|
||||
outline_level=1,
|
||||
),
|
||||
]
|
||||
for level in range(3, 6):
|
||||
styles.append(
|
||||
_style_xml(
|
||||
f"Heading{level}",
|
||||
f"heading {level}",
|
||||
font=BODY_FONT_FAMILY,
|
||||
size=BODY_SIZE_HALF_POINTS,
|
||||
alignment="left",
|
||||
keep_next=True,
|
||||
keep_lines=True,
|
||||
outline_level=level - 1,
|
||||
)
|
||||
)
|
||||
styles.extend(
|
||||
[
|
||||
_style_xml(
|
||||
"ListText",
|
||||
"List Text",
|
||||
font=BODY_FONT_FAMILY,
|
||||
size=BODY_SIZE_HALF_POINTS,
|
||||
alignment="both",
|
||||
),
|
||||
_style_xml(
|
||||
"QuoteText",
|
||||
"Quote Text",
|
||||
font=BODY_FONT_FAMILY,
|
||||
size=BODY_SIZE_HALF_POINTS,
|
||||
alignment="both",
|
||||
first_line_chars=200,
|
||||
left=640,
|
||||
),
|
||||
_style_xml(
|
||||
"CodeText",
|
||||
"Code Text",
|
||||
font=BODY_FONT_FAMILY,
|
||||
size=BODY_SIZE_HALF_POINTS,
|
||||
alignment="left",
|
||||
),
|
||||
_style_xml(
|
||||
"TableCaption",
|
||||
"Table Caption",
|
||||
font=HEADING_1_FONT_FAMILY,
|
||||
size=TABLE_SIZE_HALF_POINTS,
|
||||
alignment="center",
|
||||
keep_next=True,
|
||||
),
|
||||
_style_xml(
|
||||
"TableHeader",
|
||||
"Table Header",
|
||||
font=TABLE_FONT_FAMILY,
|
||||
size=TABLE_SIZE_HALF_POINTS,
|
||||
alignment="center",
|
||||
keep_lines=True,
|
||||
),
|
||||
_style_xml(
|
||||
"TableBody",
|
||||
"Table Body",
|
||||
font=TABLE_FONT_FAMILY,
|
||||
size=TABLE_SIZE_HALF_POINTS,
|
||||
alignment="both",
|
||||
keep_lines=True,
|
||||
),
|
||||
_style_xml(
|
||||
"ImageCaption",
|
||||
"Image Caption",
|
||||
font=BODY_FONT_FAMILY,
|
||||
size=BODY_SIZE_HALF_POINTS,
|
||||
alignment="center",
|
||||
keep_lines=True,
|
||||
),
|
||||
]
|
||||
)
|
||||
return f'{_XML_DECL}<w:styles xmlns:w="{_W_NS}">{defaults}{"".join(styles)}</w:styles>'
|
||||
|
||||
|
||||
def _heading_level_xml(level: int, format_value: str, text: str, font: str) -> str:
|
||||
restart = f'<w:lvlRestart w:val="{level}"/>' if level > 0 else ""
|
||||
return (
|
||||
f'<w:lvl w:ilvl="{level}"><w:start w:val="1"/>{restart}<w:numFmt w:val="{format_value}"/>'
|
||||
f'<w:lvlText w:val="{_xml_attr(text)}"/><w:lvlJc w:val="left"/>'
|
||||
'<w:pPr><w:ind w:left="0" w:hanging="0"/></w:pPr>'
|
||||
f'<w:rPr>{_font_run_properties(font, BODY_SIZE_HALF_POINTS)}</w:rPr></w:lvl>'
|
||||
)
|
||||
|
||||
|
||||
def _list_abstract_xml(abstract_id: int, *, ordered: bool) -> str:
|
||||
levels: list[str] = []
|
||||
bullets = ["●", "○", "▪"]
|
||||
for level in range(9):
|
||||
format_value = "decimal" if ordered else "bullet"
|
||||
text = f"%{level + 1}." if ordered else bullets[level % len(bullets)]
|
||||
left = 720 * (level + 1)
|
||||
levels.append(
|
||||
f'<w:lvl w:ilvl="{level}"><w:start w:val="1"/><w:numFmt w:val="{format_value}"/>'
|
||||
f'<w:lvlText w:val="{_xml_attr(text)}"/><w:lvlJc w:val="left"/>'
|
||||
f'<w:pPr><w:tabs><w:tab w:val="num" w:pos="{left}"/></w:tabs>'
|
||||
f'<w:ind w:left="{left}" w:hanging="360"/></w:pPr>'
|
||||
f'<w:rPr>{_font_run_properties(BODY_FONT_FAMILY, BODY_SIZE_HALF_POINTS)}</w:rPr></w:lvl>'
|
||||
)
|
||||
return f'<w:abstractNum w:abstractNumId="{abstract_id}"><w:multiLevelType w:val="multilevel"/>{"".join(levels)}</w:abstractNum>'
|
||||
|
||||
|
||||
def _numbering_xml() -> str:
|
||||
heading_levels = "".join(
|
||||
[
|
||||
_heading_level_xml(0, "chineseCounting", "%1、", HEADING_1_FONT_FAMILY),
|
||||
_heading_level_xml(1, "chineseCounting", "（%2）", HEADING_2_FONT_FAMILY),
|
||||
_heading_level_xml(2, "decimal", "%3.", BODY_FONT_FAMILY),
|
||||
_heading_level_xml(3, "decimal", "（%4）", BODY_FONT_FAMILY),
|
||||
_heading_level_xml(4, "decimal", "%5）", BODY_FONT_FAMILY),
|
||||
]
|
||||
)
|
||||
return (
|
||||
f'{_XML_DECL}<w:numbering xmlns:w="{_W_NS}">'
|
||||
f'<w:abstractNum w:abstractNumId="0"><w:multiLevelType w:val="multilevel">'
|
||||
f"</w:multiLevelType>{heading_levels}</w:abstractNum>"
|
||||
f'{_list_abstract_xml(1, ordered=True)}{_list_abstract_xml(2, ordered=False)}'
|
||||
'<w:num w:numId="1"><w:abstractNumId w:val="0"/></w:num>'
|
||||
'<w:num w:numId="2"><w:abstractNumId w:val="1"/></w:num>'
|
||||
'<w:num w:numId="3"><w:abstractNumId w:val="2"/></w:num>'
|
||||
"</w:numbering>"
|
||||
)
|
||||
|
||||
|
||||
def _footer_xml(alignment: str) -> str:
|
||||
run_properties = _font_run_properties(TABLE_FONT_FAMILY, TABLE_SIZE_HALF_POINTS)
|
||||
return (
|
||||
f'{_XML_DECL}<w:ftr xmlns:w="{_W_NS}" xmlns:r="{_R_NS}">'
|
||||
f'<w:p><w:pPr><w:jc w:val="{alignment}"/><w:spacing w:before="0" w:after="0"/></w:pPr>'
|
||||
f'<w:r><w:rPr>{run_properties}</w:rPr><w:fldChar w:fldCharType="begin"/></w:r>'
|
||||
f'<w:r><w:rPr>{run_properties}</w:rPr><w:instrText xml:space="preserve"> PAGE </w:instrText></w:r>'
|
||||
f'<w:r><w:rPr>{run_properties}</w:rPr><w:fldChar w:fldCharType="separate"/></w:r>'
|
||||
f'<w:r><w:rPr>{run_properties}</w:rPr><w:t>1</w:t></w:r>'
|
||||
f'<w:r><w:rPr>{run_properties}</w:rPr><w:fldChar w:fldCharType="end"/></w:r>'
|
||||
"</w:p></w:ftr>"
|
||||
)
|
||||
|
||||
|
||||
def _font_table_xml(font_key: str) -> str:
|
||||
def font(name: str, *, embedded: bool = False) -> str:
|
||||
embed = f'<w:embedRegular r:id="rId1" w:fontKey="{{{font_key}}}"/>' if embedded else ""
|
||||
return (
|
||||
f'<w:font w:name="{_xml_attr(name)}"><w:family w:val="roman"/>'
|
||||
f'<w:charset w:val="86"/>{embed}</w:font>'
|
||||
)
|
||||
|
||||
fonts = "".join(
|
||||
[
|
||||
font(TITLE_FONT_FAMILY, embedded=True),
|
||||
font(BODY_FONT_FAMILY),
|
||||
font(HEADING_1_FONT_FAMILY),
|
||||
font(HEADING_2_FONT_FAMILY),
|
||||
font(TABLE_FONT_FAMILY),
|
||||
]
|
||||
)
|
||||
return f'{_XML_DECL}<w:fonts xmlns:w="{_W_NS}" xmlns:r="{_R_NS}">{fonts}</w:fonts>'
|
||||
|
||||
|
||||
def _font_table_relationships_xml() -> str:
|
||||
return (
|
||||
f'{_XML_DECL}<Relationships xmlns="{_PACKAGE_REL_NS}">'
|
||||
'<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/font" Target="fonts/font1.odttf"/>'
|
||||
"</Relationships>"
|
||||
)
|
||||
|
||||
|
||||
def _settings_xml() -> str:
|
||||
return (
|
||||
f'{_XML_DECL}<w:settings xmlns:w="{_W_NS}">'
|
||||
'<w:zoom w:percent="100"/><w:evenAndOddHeaders/><w:updateFields w:val="true"/>'
|
||||
'<w:compat><w:compatSetting w:name="compatibilityMode" '
|
||||
'w:uri="http://schemas.microsoft.com/office/word" w:val="15"/></w:compat>'
|
||||
"</w:settings>"
|
||||
)
|
||||
|
||||
|
||||
def _document_xml(title: str | None, blocks: list[Block], relationships: _DocumentRelationships) -> str:
|
||||
body: list[str] = []
|
||||
if title:
|
||||
body.append(
|
||||
_paragraph_xml(
|
||||
"DocumentTitle",
|
||||
_run_xml(title, font=TITLE_FONT_FAMILY, size=TITLE_SIZE_HALF_POINTS),
|
||||
keep_next=True,
|
||||
alignment="center",
|
||||
)
|
||||
)
|
||||
body.append(_blocks_xml(blocks, relationships))
|
||||
section = (
|
||||
'<w:sectPr><w:footerReference w:type="default" r:id="rId5"/>'
|
||||
'<w:footerReference w:type="even" r:id="rId6"/>'
|
||||
f'<w:pgSz w:w="{A4_WIDTH_TWIPS}" w:h="{A4_HEIGHT_TWIPS}" w:orient="portrait"/>'
|
||||
f'<w:pgMar w:top="{MARGIN_TOP_TWIPS}" w:right="{MARGIN_RIGHT_TWIPS}" '
|
||||
f'w:bottom="{MARGIN_BOTTOM_TWIPS}" w:left="{MARGIN_LEFT_TWIPS}" '
|
||||
'w:header="720" w:footer="992" w:gutter="0"/>'
|
||||
'<w:pgNumType w:start="1" w:fmt="decimal"/><w:cols w:space="720"/></w:sectPr>'
|
||||
)
|
||||
return (
|
||||
f'{_XML_DECL}<w:document xmlns:w="{_W_NS}" xmlns:r="{_R_NS}">'
|
||||
f"<w:body>{''.join(body)}{section}</w:body></w:document>"
|
||||
)
|
||||
|
||||
|
||||
def _content_types_xml() -> str:
|
||||
return (
|
||||
f'{_XML_DECL}<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">'
|
||||
'<Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/>'
|
||||
'<Default Extension="xml" ContentType="application/xml"/>'
|
||||
'<Default Extension="odttf" ContentType="application/vnd.openxmlformats-officedocument.obfuscatedFont"/>'
|
||||
'<Override PartName="/word/document.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.document.main+xml"/>'
|
||||
'<Override PartName="/word/styles.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.styles+xml"/>'
|
||||
'<Override PartName="/word/numbering.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.numbering+xml"/>'
|
||||
'<Override PartName="/word/settings.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.settings+xml"/>'
|
||||
'<Override PartName="/word/fontTable.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.fontTable+xml"/>'
|
||||
'<Override PartName="/word/footer1.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.footer+xml"/>'
|
||||
'<Override PartName="/word/footer2.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.footer+xml"/>'
|
||||
'<Override PartName="/docProps/core.xml" ContentType="application/vnd.openxmlformats-package.core-properties+xml"/>'
|
||||
'<Override PartName="/docProps/app.xml" ContentType="application/vnd.openxmlformats-officedocument.extended-properties+xml"/>'
|
||||
"</Types>"
|
||||
)
|
||||
|
||||
|
||||
def _package_relationships_xml() -> str:
|
||||
return (
|
||||
f'{_XML_DECL}<Relationships xmlns="{_PACKAGE_REL_NS}">'
|
||||
'<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument" Target="word/document.xml"/>'
|
||||
'<Relationship Id="rId2" Type="http://schemas.openxmlformats.org/package/2006/relationships/metadata/core-properties" Target="docProps/core.xml"/>'
|
||||
'<Relationship Id="rId3" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/extended-properties" Target="docProps/app.xml"/>'
|
||||
"</Relationships>"
|
||||
)
|
||||
|
||||
|
||||
def _core_properties_xml(title: str | None) -> str:
|
||||
now = datetime.now(UTC).replace(microsecond=0).isoformat().replace("+00:00", "Z")
|
||||
safe_title = escape(title or "Word 文档")
|
||||
return (
|
||||
f'{_XML_DECL}<cp:coreProperties xmlns:cp="http://schemas.openxmlformats.org/package/2006/metadata/core-properties" '
|
||||
'xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:dcterms="http://purl.org/dc/terms/" '
|
||||
'xmlns:dcmitype="http://purl.org/dc/dcmitype/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">'
|
||||
f"<dc:title>{safe_title}</dc:title><dc:creator>DeerFlow</dc:creator>"
|
||||
f'<dcterms:created xsi:type="dcterms:W3CDTF">{now}</dcterms:created>'
|
||||
f'<dcterms:modified xsi:type="dcterms:W3CDTF">{now}</dcterms:modified></cp:coreProperties>'
|
||||
)
|
||||
|
||||
|
||||
def _app_properties_xml() -> str:
|
||||
return (
|
||||
f'{_XML_DECL}<Properties xmlns="http://schemas.openxmlformats.org/officeDocument/2006/extended-properties" '
|
||||
'xmlns:vt="http://schemas.openxmlformats.org/officeDocument/2006/docPropsVTypes">'
|
||||
"<Application>DeerFlow</Application><AppVersion>1.0</AppVersion></Properties>"
|
||||
)
|
||||
|
||||
|
||||
def build_markdown_docx(markdown: str, title: str | None = None, *, font_path: Path | None = None) -> bytes:
|
||||
"""Build a formal DOCX and embed the validated server-side title font."""
|
||||
if not markdown or not markdown.strip():
|
||||
raise WordExportError("文档内容为空。")
|
||||
if len(markdown) > 2_000_000:
|
||||
raise WordExportError("文档内容超过 200 万字符，无法导出。")
|
||||
|
||||
font = load_title_font(font_path)
|
||||
resolved_title, blocks = _parse_markdown(markdown, title)
|
||||
relationships = _DocumentRelationships()
|
||||
document_xml = _document_xml(resolved_title, blocks, relationships)
|
||||
font_key = str(uuid.uuid4()).upper()
|
||||
|
||||
output = io.BytesIO()
|
||||
with zipfile.ZipFile(output, mode="w", compression=zipfile.ZIP_DEFLATED, compresslevel=6) as archive:
|
||||
parts: dict[str, str | bytes] = {
|
||||
"[Content_Types].xml": _content_types_xml(),
|
||||
"_rels/.rels": _package_relationships_xml(),
|
||||
"docProps/core.xml": _core_properties_xml(resolved_title),
|
||||
"docProps/app.xml": _app_properties_xml(),
|
||||
"word/document.xml": document_xml,
|
||||
"word/_rels/document.xml.rels": relationships.xml(),
|
||||
"word/styles.xml": _styles_xml(),
|
||||
"word/numbering.xml": _numbering_xml(),
|
||||
"word/settings.xml": _settings_xml(),
|
||||
"word/fontTable.xml": _font_table_xml(font_key),
|
||||
"word/_rels/fontTable.xml.rels": _font_table_relationships_xml(),
|
||||
# Word applies the default footer to odd pages and the even footer
|
||||
# to even pages. The requested placement is odd-left/even-right.
|
||||
"word/footer1.xml": _footer_xml("left"),
|
||||
"word/footer2.xml": _footer_xml("right"),
|
||||
"word/fonts/font1.odttf": _obfuscate_font(font.data, font_key),
|
||||
}
|
||||
for name, value in parts.items():
|
||||
archive.writestr(name, value.encode("utf-8") if isinstance(value, str) else value)
|
||||
return output.getvalue()
|
||||
|
||||
|
||||
__all__ = [
|
||||
"DOCX_MEDIA_TYPE",
|
||||
"TITLE_FONT_FAMILY",
|
||||
"WordExportError",
|
||||
"WordExportFontError",
|
||||
"build_markdown_docx",
|
||||
"load_title_font",
|
||||
"resolve_title_font_path",
|
||||
]
|
||||
@@ -561,6 +561,8 @@ llmwiki:
|
||||
# 且服务启动时只启动定时器，不会立刻全量向量化。
|
||||
local_wiki_index:
|
||||
enabled: false
|
||||
# 免登录 POST /api/knowledge/vector-search 返回的 Wiki 一键跳转链接前缀。
|
||||
frontend_base_url: "http://127.0.0.1:5174"
|
||||
auto_sync: false
|
||||
database_batch_size: 50
|
||||
embedding:
|
||||
|
||||
@@ -1132,6 +1132,35 @@ Content-Type: application/json
|
||||
}
|
||||
```
|
||||
|
||||
#### 13.4.1 免鉴权浏览器向量搜索
|
||||
|
||||
本部署另行提供一个与上述 API-Key 命名空间隔离的新接口：
|
||||
|
||||
```http
|
||||
POST /api/knowledge/vector-search
|
||||
Content-Type: application/json
|
||||
```
|
||||
|
||||
最小请求只需检索词：
|
||||
|
||||
```json
|
||||
{"query": "任正非早期创业经历"}
|
||||
```
|
||||
|
||||
可选参数只有 `top_k`（默认 `10`，最大 `100`）和
|
||||
`include_page_content`（默认 `false`）。接口不需要登录 Token 或
|
||||
`X-API-Key`，不做应用内限流，并针对这一条路由返回
|
||||
`Access-Control-Allow-Origin: *`。检索范围是全部 `published` 知识库，
|
||||
不检查 `external_search_enabled`，但在检索前后都排除系统“对话沉淀”库。
|
||||
|
||||
响应按向量分段展开：同一 Wiki 页命中多个分段时会返回多条结果。每条结果
|
||||
包含命中文本和分数、DeerFlow/WeKnora 知识库与 Wiki 页 ID、页面元数据、
|
||||
原始 `source_refs`/`chunk_refs`、可提取到的文档/分块 ID，以及可直接打开的
|
||||
`wiki_url`。当 `include_page_content=true` 时额外返回完整 `page_content`。
|
||||
`wiki_url` 的前缀配置为
|
||||
`llmwiki.local_wiki_index.frontend_base_url`，开发默认值是
|
||||
`http://127.0.0.1:5174`。
|
||||
|
||||
### 13.5 外部页面详情
|
||||
|
||||
```http
|
||||
@@ -1156,6 +1185,9 @@ X-API-Key: <external key>
|
||||
|
||||
如果为了少改 AuthMiddleware 而把接口挂到 `/api/public/`，路由自身也必须强制校验 API Key；但长期推荐独立 `/api/external/` 语义。
|
||||
|
||||
上面的原则仍适用于通用服务间接口；13.4.1 是经过明确产品确认的例外，并通过
|
||||
精确路径白名单和路由级 CORS 隔离，不能据此放宽其他 API。
|
||||
|
||||
### 13.7 状态码约定
|
||||
|
||||
| 状态码 | 场景 |
|
||||
|
||||
@@ -108,6 +108,10 @@ class LocalWikiIndexConfig(BaseModel):
|
||||
"""DeerFlow-owned mirror and exact-vector index for WeKnora Wiki pages."""
|
||||
|
||||
enabled: bool = False
|
||||
frontend_base_url: str = Field(
|
||||
default="http://127.0.0.1:5174",
|
||||
description="DeerFlow frontend base URL used to build directly usable Wiki links in public vector-search results.",
|
||||
)
|
||||
auto_sync: bool = False
|
||||
sync_interval_seconds: int = Field(default=120, ge=10, le=86400)
|
||||
sync_page_size: int = Field(default=100, ge=1, le=500)
|
||||
@@ -135,6 +139,14 @@ class LocalWikiIndexConfig(BaseModel):
|
||||
external_api: LocalWikiExternalApiConfig = Field(default_factory=LocalWikiExternalApiConfig)
|
||||
model_config = ConfigDict(extra="ignore")
|
||||
|
||||
@field_validator("frontend_base_url")
|
||||
@classmethod
|
||||
def _validate_frontend_base_url(cls, value: str) -> str:
|
||||
normalized = normalize_service_url(value)
|
||||
if not normalized:
|
||||
raise ValueError("local Wiki index frontend_base_url is required")
|
||||
return normalized
|
||||
|
||||
@model_validator(mode="before")
|
||||
@classmethod
|
||||
def _migrate_legacy_flags(cls, value):
|
||||
|
||||
+167
-27
@@ -46,29 +46,12 @@ class WikiVectorSearchService:
|
||||
raise WikiIndexError("WIKI_QUERY_EMPTY", "Search query is empty", status_code=400)
|
||||
if not mappings:
|
||||
return {"query": normalized_query, "mode": "vector", "results": [], "knowledge_base_count": 0, "index_fingerprint": self.embedding.fingerprint[:12], "partial": False, "warnings": []}
|
||||
try:
|
||||
query_vector = await self._embed_query(normalized_query)
|
||||
except (WikiEmbeddingError, ValueError) as exc:
|
||||
raise WikiIndexError("WIKI_EMBEDDING_UNAVAILABLE", str(exc), status_code=503) from None
|
||||
fingerprint = self.embedding.fingerprint
|
||||
tasks = [self._search_mapping(normalized_query, query_vector, mapping, fingerprint, include_drafts=include_drafts, external=external) for mapping in mappings]
|
||||
batches = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
candidates: list[dict[str, Any]] = []
|
||||
warnings: list[str] = []
|
||||
for mapping, batch in zip(mappings, batches, strict=True):
|
||||
if isinstance(batch, Exception):
|
||||
warnings.append(f"{mapping.get('id')}: {batch}")
|
||||
else:
|
||||
candidates.extend(batch)
|
||||
if warnings and not candidates and len(warnings) == len(mappings):
|
||||
if any("FINGERPRINT" in item for item in warnings):
|
||||
code = "WIKI_INDEX_FINGERPRINT_MISMATCH"
|
||||
elif any("rebuild" in item.lower() for item in warnings):
|
||||
code = "WIKI_INDEX_REBUILDING"
|
||||
else:
|
||||
code = "WIKI_INDEX_NOT_READY"
|
||||
raise WikiIndexError(code, "Local Wiki index is not ready", status_code=409)
|
||||
candidates.sort(key=lambda item: item["score"], reverse=True)
|
||||
fingerprint, candidates, warnings = await self._collect_candidates(
|
||||
normalized_query,
|
||||
mappings,
|
||||
include_drafts=include_drafts,
|
||||
external=external,
|
||||
)
|
||||
grouped: dict[tuple[str, str], list[dict[str, Any]]] = defaultdict(list)
|
||||
for candidate in candidates:
|
||||
page = candidate["page"]
|
||||
@@ -111,6 +94,141 @@ class WikiVectorSearchService:
|
||||
limit = min(top_k_pages or self.config.top_k_pages, self.config.top_k_pages if top_k_pages is None else 50)
|
||||
return {"query": normalized_query, "mode": "vector", "results": results[:limit], "knowledge_base_count": len(mappings), "index_fingerprint": fingerprint[:12], "partial": bool(warnings), "warnings": warnings}
|
||||
|
||||
async def search_sections(
|
||||
self,
|
||||
query: str,
|
||||
mappings: list[dict[str, Any]],
|
||||
*,
|
||||
top_k_sections: int = 10,
|
||||
external: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
"""Return one result per matching Wiki vector section.
|
||||
|
||||
The regular ``search`` method intentionally groups several sections
|
||||
under one Wiki page for citations. Public consumers of the standalone
|
||||
vector API need the opposite contract: every matched vector is a
|
||||
separate ranked item, while the complete page and knowledge-base
|
||||
metadata remains attached to that item.
|
||||
"""
|
||||
|
||||
normalized_query = query.strip()
|
||||
if not normalized_query:
|
||||
raise WikiIndexError("WIKI_QUERY_EMPTY", "Search query is empty", status_code=400)
|
||||
limit = max(1, min(int(top_k_sections), 100))
|
||||
if not mappings:
|
||||
return {
|
||||
"query": normalized_query,
|
||||
"mode": "vector",
|
||||
"results": [],
|
||||
"knowledge_base_count": 0,
|
||||
"index_fingerprint": self.embedding.fingerprint[:12],
|
||||
"partial": False,
|
||||
"warnings": [],
|
||||
}
|
||||
fingerprint, candidates, warnings = await self._collect_candidates(
|
||||
normalized_query,
|
||||
mappings,
|
||||
include_drafts=False,
|
||||
external=external,
|
||||
section_limit=limit,
|
||||
)
|
||||
results: list[dict[str, Any]] = []
|
||||
for candidate in candidates[:limit]:
|
||||
page = candidate["page"]
|
||||
mapping = candidate["mapping"]
|
||||
results.append(
|
||||
{
|
||||
"kind": "wiki_section",
|
||||
"id": candidate.get("id") or f"{page['id']}:{candidate['section_index']}",
|
||||
"section_id": candidate.get("id") or f"{page['id']}:{candidate['section_index']}",
|
||||
"section_index": int(candidate["section_index"]),
|
||||
"heading": candidate.get("heading"),
|
||||
"content": str(candidate.get("section_content") or ""),
|
||||
"section_content_hash": candidate.get("content_hash") or "",
|
||||
"score": float(candidate["score"]),
|
||||
"knowledge_base_id": mapping["id"],
|
||||
"weknora_knowledge_base_id": mapping.get("weknora_id") or "",
|
||||
"knowledge_base_name": mapping.get("name") or "",
|
||||
"knowledge_base_description": mapping.get("description") or "",
|
||||
"knowledge_base_type": mapping.get("kb_type") or "document",
|
||||
"publication_status": mapping.get("publication_status") or "",
|
||||
"published_at": mapping.get("published_at"),
|
||||
"wiki_page_id": page["id"],
|
||||
"weknora_wiki_page_id": page.get("remote_page_id"),
|
||||
"wiki_slug": page["slug"],
|
||||
"title": page.get("title") or page["slug"],
|
||||
"summary": page.get("summary") or "",
|
||||
"page_content": page.get("content_md") or "",
|
||||
"page_type": page.get("page_type") or "article",
|
||||
"page_status": page.get("status") or "published",
|
||||
"page_version": int(page.get("page_version") or 0),
|
||||
"aliases": page.get("aliases") or [],
|
||||
"category_path": page.get("category_path") or [],
|
||||
"parent_slug": page.get("parent_slug"),
|
||||
"wiki_path": page.get("wiki_path"),
|
||||
"source_refs": page.get("source_refs") or [],
|
||||
"chunk_refs": page.get("chunk_refs") or [],
|
||||
"page_content_hash": page.get("content_hash") or "",
|
||||
"remote_created_at": page.get("remote_created_at"),
|
||||
"remote_updated_at": page.get("remote_updated_at"),
|
||||
"indexed_at": page.get("indexed_at"),
|
||||
}
|
||||
)
|
||||
return {
|
||||
"query": normalized_query,
|
||||
"mode": "vector",
|
||||
"results": results,
|
||||
"knowledge_base_count": len(mappings),
|
||||
"index_fingerprint": fingerprint[:12],
|
||||
"partial": bool(warnings),
|
||||
"warnings": warnings,
|
||||
}
|
||||
|
||||
async def _collect_candidates(
|
||||
self,
|
||||
query: str,
|
||||
mappings: list[dict[str, Any]],
|
||||
*,
|
||||
include_drafts: bool,
|
||||
external: bool,
|
||||
section_limit: int | None = None,
|
||||
) -> tuple[str, list[dict[str, Any]], list[str]]:
|
||||
try:
|
||||
query_vector = await self._embed_query(query)
|
||||
except (WikiEmbeddingError, ValueError) as exc:
|
||||
raise WikiIndexError("WIKI_EMBEDDING_UNAVAILABLE", str(exc), status_code=503) from None
|
||||
fingerprint = self.embedding.fingerprint
|
||||
tasks = [
|
||||
self._search_mapping(
|
||||
query,
|
||||
query_vector,
|
||||
mapping,
|
||||
fingerprint,
|
||||
include_drafts=include_drafts,
|
||||
external=external,
|
||||
section_limit=section_limit,
|
||||
)
|
||||
for mapping in mappings
|
||||
]
|
||||
batches = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
candidates: list[dict[str, Any]] = []
|
||||
warnings: list[str] = []
|
||||
for mapping, batch in zip(mappings, batches, strict=True):
|
||||
if isinstance(batch, Exception):
|
||||
warnings.append(f"{mapping.get('id')}: {batch}")
|
||||
else:
|
||||
candidates.extend(batch)
|
||||
if warnings and not candidates and len(warnings) == len(mappings):
|
||||
if any("FINGERPRINT" in item for item in warnings):
|
||||
code = "WIKI_INDEX_FINGERPRINT_MISMATCH"
|
||||
elif any("rebuild" in item.lower() for item in warnings):
|
||||
code = "WIKI_INDEX_REBUILDING"
|
||||
else:
|
||||
code = "WIKI_INDEX_NOT_READY"
|
||||
raise WikiIndexError(code, "Local Wiki index is not ready", status_code=409)
|
||||
candidates.sort(key=lambda item: item["score"], reverse=True)
|
||||
return fingerprint, candidates, warnings
|
||||
|
||||
async def _embed_query(self, query: str) -> np.ndarray:
|
||||
digest = hashlib.sha256(query.encode("utf-8")).hexdigest()
|
||||
key = f"{self.embedding.fingerprint}:{digest}"
|
||||
@@ -126,7 +244,17 @@ class WikiVectorSearchService:
|
||||
self._query_vectors.popitem(last=False)
|
||||
return vector
|
||||
|
||||
async def _search_mapping(self, query: str, vector: np.ndarray, mapping: dict[str, Any], fingerprint: str, *, include_drafts: bool, external: bool) -> list[dict[str, Any]]:
|
||||
async def _search_mapping(
|
||||
self,
|
||||
query: str,
|
||||
vector: np.ndarray,
|
||||
mapping: dict[str, Any],
|
||||
fingerprint: str,
|
||||
*,
|
||||
include_drafts: bool,
|
||||
external: bool,
|
||||
section_limit: int | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
mapping_id = str(mapping["id"])
|
||||
try:
|
||||
snapshot = await self.cache.get(mapping_id, fingerprint)
|
||||
@@ -148,7 +276,13 @@ class WikiVectorSearchService:
|
||||
)
|
||||
if eligible.size == 0:
|
||||
return []
|
||||
indices, scores = await asyncio.to_thread(self._top_indices, snapshot, vector, eligible)
|
||||
indices, scores = await asyncio.to_thread(
|
||||
self._top_indices,
|
||||
snapshot,
|
||||
vector,
|
||||
eligible,
|
||||
section_limit,
|
||||
)
|
||||
terms = {term.lower() for term in re.findall(r"[\w\u4e00-\u9fff]{2,}", query)}
|
||||
candidates = []
|
||||
for index, score in zip(indices, scores, strict=True):
|
||||
@@ -166,9 +300,15 @@ class WikiVectorSearchService:
|
||||
candidates.append({**row, "page": page, "mapping": mapping, "score": final_score})
|
||||
return candidates
|
||||
|
||||
def _top_indices(self, snapshot: WikiVectorSnapshot, vector: np.ndarray, eligible: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
|
||||
def _top_indices(
|
||||
self,
|
||||
snapshot: WikiVectorSnapshot,
|
||||
vector: np.ndarray,
|
||||
eligible: np.ndarray,
|
||||
section_limit: int | None = None,
|
||||
) -> tuple[np.ndarray, np.ndarray]:
|
||||
scores = snapshot.matrix[eligible] @ vector
|
||||
count = min(self.config.top_k_sections, scores.size)
|
||||
count = min(section_limit or self.config.top_k_sections, scores.size)
|
||||
if count <= 0:
|
||||
return np.asarray([], dtype=int), np.asarray([], dtype=np.float32)
|
||||
positions = np.argpartition(scores, -count)[-count:]
|
||||
|
||||
+1
-1
@@ -19,7 +19,7 @@ def _normalize(data: dict[str, Any], menu_type: str, updated_by: str | None) ->
|
||||
return {
|
||||
"node_id": storage_node_id(menu_type, node_id),
|
||||
"menu_type": menu_type,
|
||||
"placement": placement if placement in {"sidebar", "settings"} else "sidebar",
|
||||
"placement": placement if placement in {"sidebar", "settings", "toolbox"} else "sidebar",
|
||||
"sort_order": int(data.get("sort_order", 0)),
|
||||
"label": (data.get("label") or None),
|
||||
"updated_by": updated_by,
|
||||
|
||||
+1
-1
@@ -35,7 +35,7 @@ def _row_from(data: dict[str, Any], menu_type: str, updated_by: str | None) -> C
|
||||
return CompactMenuOverrideRow(
|
||||
node_id=storage_node_id(menu_type, node_id),
|
||||
menu_type=menu_type,
|
||||
placement=placement if placement in {"sidebar", "settings"} else "sidebar",
|
||||
placement=placement if placement in {"sidebar", "settings", "toolbox"} else "sidebar",
|
||||
sort_order=int(data.get("sort_order", 0)),
|
||||
label=(data.get("label") or None),
|
||||
updated_by=updated_by,
|
||||
|
||||
@@ -12,8 +12,8 @@ from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
|
||||
|
||||
from deerflow.persistence.base import Base
|
||||
from deerflow.persistence.compact_menu_overrides import (
|
||||
MemoryCompactMenuOverrideStore,
|
||||
CompactMenuOverrideRepository,
|
||||
MemoryCompactMenuOverrideStore,
|
||||
normalize_compact_menu_type,
|
||||
)
|
||||
from deerflow.persistence.compact_menu_overrides.keys import public_node_id, storage_node_id
|
||||
@@ -53,15 +53,17 @@ async def test_replace_all_roundtrips(store: CompactMenuOverrideRepository):
|
||||
[
|
||||
{"node_id": "workspace-chats", "placement": "settings", "sort_order": 1, "label": "会话"},
|
||||
{"node_id": "workspace-agents", "placement": "sidebar", "sort_order": 0, "label": None},
|
||||
{"node_id": "workspace-skills", "placement": "toolbox", "sort_order": 0, "label": "技能工具"},
|
||||
],
|
||||
updated_by="admin-1",
|
||||
)
|
||||
assert len(out) == 2
|
||||
assert len(out) == 3
|
||||
rows = {r["node_id"]: r for r in await store.list_overrides()}
|
||||
assert rows["workspace-chats"]["label"] == "会话"
|
||||
assert rows["workspace-chats"]["placement"] == "settings"
|
||||
assert rows["workspace-chats"]["menu_type"] == "A"
|
||||
assert rows["workspace-agents"]["updated_by"] == "admin-1"
|
||||
assert rows["workspace-skills"]["placement"] == "toolbox"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
@@ -56,6 +56,13 @@ async def test_exact_search_groups_sections_by_wiki_page_and_filters_drafts() ->
|
||||
with_drafts = await service.search("alice", [mapping], include_drafts=True)
|
||||
assert {item["wiki_slug"] for item in with_drafts["results"]} == {"alice", "secret"}
|
||||
|
||||
section_result = await service.search_sections("alice", [mapping], top_k_sections=2, external=True)
|
||||
assert [item["content"] for item in section_result["results"]] == ["Alpha", "Beta"]
|
||||
assert all(item["wiki_slug"] == "alice" for item in section_result["results"])
|
||||
assert all(item["kind"] == "wiki_section" for item in section_result["results"])
|
||||
assert section_result["results"][0]["wiki_page_id"]
|
||||
assert "page_content" in section_result["results"][0]
|
||||
|
||||
narrow_service = WikiVectorSearchService(
|
||||
LocalWikiIndexConfig(min_similarity=0, top_k_sections=1, top_k_pages=1),
|
||||
FakeEmbedding(),
|
||||
|
||||
@@ -1379,7 +1379,9 @@ def test_weknora_iframe_html_rewrites_public_prefix_paths() -> None:
|
||||
assert "function protectedImageApiUrl" in injected
|
||||
assert 'img[data-protected-src]' in injected
|
||||
assert 'encodeURIComponent(cfg.weknoraId)' in injected
|
||||
assert 'credentials: "same-origin"' in injected
|
||||
assert "image.src = proxyUrl;" in injected
|
||||
assert "URL.createObjectURL(blob)" not in injected
|
||||
assert 'attributeFilter: ["data-protected-src", "src"]' in injected
|
||||
|
||||
|
||||
def test_weknora_iframe_filters_kb_list_to_signed_visible_mappings() -> None:
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import re
|
||||
import sys
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
from xml.etree import ElementTree
|
||||
|
||||
import pytest
|
||||
|
||||
BACKEND_ROOT = Path(__file__).resolve().parents[1]
|
||||
MODULE_PATH = BACKEND_ROOT / "app" / "gateway" / "word_export.py"
|
||||
|
||||
|
||||
def _load_word_export_module():
|
||||
"""Load the leaf module without executing app.gateway.__init__."""
|
||||
spec = importlib.util.spec_from_file_location("word_export_test_module", MODULE_PATH)
|
||||
assert spec is not None and spec.loader is not None
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
sys.modules[spec.name] = module
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
word_export = _load_word_export_module()
|
||||
|
||||
|
||||
def test_bundled_font_is_exact_and_allows_editable_embedding() -> None:
|
||||
font = word_export.load_title_font()
|
||||
|
||||
assert font.sha256 == "C52D577DD3AA719EF5FB9FF2043B7F7CA36FAE468EF34A47628A3A555D006A49"
|
||||
assert font.fs_type == 0x0008
|
||||
assert any("FZXiaoBiaoSong" in name for name in font.names)
|
||||
|
||||
|
||||
def test_formal_docx_contains_required_layout_numbering_and_embedded_font(tmp_path: Path) -> None:
|
||||
markdown = """# 样例标题
|
||||
|
||||
## 一、一级标题
|
||||
这是正文，用于验证首行缩进、两端对齐和固定行距。
|
||||
|
||||
### （一）二级标题
|
||||
#### 1. 三级标题
|
||||
|
||||
表1 测试表格
|
||||
| 项目 | 数值 |
|
||||
|---|---:|
|
||||
| 页面 | A4 纵向 |
|
||||
"""
|
||||
docx_path = tmp_path / "sample.docx"
|
||||
docx_path.write_bytes(word_export.build_markdown_docx(markdown))
|
||||
|
||||
with zipfile.ZipFile(docx_path) as archive:
|
||||
names = set(archive.namelist())
|
||||
assert "word/fonts/font1.odttf" in names
|
||||
assert "word/_rels/fontTable.xml.rels" in names
|
||||
for name in names:
|
||||
if name.endswith(".xml") or name.endswith(".rels"):
|
||||
ElementTree.fromstring(archive.read(name))
|
||||
|
||||
document = archive.read("word/document.xml").decode("utf-8")
|
||||
styles = archive.read("word/styles.xml").decode("utf-8")
|
||||
numbering = archive.read("word/numbering.xml").decode("utf-8")
|
||||
settings = archive.read("word/settings.xml").decode("utf-8")
|
||||
font_table = archive.read("word/fontTable.xml").decode("utf-8")
|
||||
default_footer = archive.read("word/footer1.xml").decode("utf-8")
|
||||
even_footer = archive.read("word/footer2.xml").decode("utf-8")
|
||||
|
||||
assert '<w:pgSz w:w="11906" w:h="16838" w:orient="portrait"/>' in document
|
||||
assert 'w:top="2098"' in document
|
||||
assert 'w:bottom="1984"' in document
|
||||
assert 'w:left="1587"' in document
|
||||
assert 'w:right="1474"' in document
|
||||
assert '<w:pgNumType w:start="1" w:fmt="decimal"/>' in document
|
||||
assert '<w:pStyle w:val="DocumentTitle"/>' in document
|
||||
assert "方正小标宋简体" in document
|
||||
assert "一、一级标题" not in document
|
||||
assert "（一）二级标题" not in document
|
||||
assert "1. 三级标题" not in document
|
||||
|
||||
assert 'w:line="579" w:lineRule="exact"' in styles
|
||||
assert '<w:ind w:left="0" w:hanging="0" w:firstLineChars="200"/>' in styles
|
||||
assert "方正小标宋简体" in styles
|
||||
assert 'w:val="44"' in styles
|
||||
assert "仿宋" in styles and 'w:val="32"' in styles
|
||||
assert "宋体" in styles and 'w:val="28"' in styles
|
||||
|
||||
assert '<w:numFmt w:val="chineseCounting"/>' in numbering
|
||||
assert '<w:lvlText w:val="%1、"/>' in numbering
|
||||
assert '<w:lvlText w:val="（%2）"/>' in numbering
|
||||
assert '<w:lvlText w:val="%3."/>' in numbering
|
||||
assert '<w:evenAndOddHeaders/>' in settings
|
||||
assert '<w:updateFields w:val="true"/>' in settings
|
||||
|
||||
assert "方正小标宋简体" in font_table
|
||||
key_match = re.search(r'w:fontKey="\{([0-9A-F-]{36})\}"', font_table)
|
||||
assert key_match is not None
|
||||
font_key = key_match.group(1)
|
||||
restored = word_export._obfuscate_font(archive.read("word/fonts/font1.odttf"), font_key)
|
||||
assert hashlib.sha256(restored).hexdigest().upper() == word_export._BUNDLED_FONT_SHA256
|
||||
|
||||
# Word's default footer applies to odd pages; the separate even footer
|
||||
# applies to even pages. The requested alignment is odd-left/even-right.
|
||||
assert '<w:jc w:val="left"/>' in default_footer
|
||||
assert '<w:jc w:val="right"/>' in even_footer
|
||||
assert " PAGE " in default_footer and " PAGE " in even_footer
|
||||
|
||||
|
||||
def test_export_rejects_wrong_font(tmp_path: Path) -> None:
|
||||
wrong_font = tmp_path / "wrong.ttf"
|
||||
wrong_font.write_bytes(b"not-a-font")
|
||||
|
||||
with pytest.raises(word_export.WordExportFontError):
|
||||
word_export.build_markdown_docx("正文", font_path=wrong_font)
|
||||
Reference in new issue
Block a user