fix: update offline backend llmwiki support

This commit is contained in:
吉鑫升 committed 2026-08-13 11:17:11 +08:00
1 parent a4a1e73b8e
commit 4809ee7db3
9 files changed
+160 -9

No files matched your search

+1 -1
View File
@@ -56,7 +56,7 @@ ELEMENT_STATE_FILE=/app/backend/.deer-flow/element-bot-token
# PYTHON_PACKAGE_INDEX_URL=http://your-python-mirror/simple
# PYTHON_EXTRA_INDEX_URL=
# PYTHON_TRUSTED_HOST=your-python-mirror
MYSQL_DATABASE_URL=mysql+asyncmy://deerflow:deerflow123@192.168.0.123:3306/deerflow?charset=utf8mb4
MYSQL_DATABASE_URL=mysql+aiomysql://deerflow:deerflow123@192.168.0.123:3306/deerflow?charset=utf8mb4
DEER_FLOW_NOTEBOOK_API_BASE=http://127.0.0.1:8003
# 圆桌后台作业网关:缺省即 inprocess(进程内驱动真模型,后台挂起开箱即用);
# 仅前端联调时显式设 simulated(真实进度但不调真模型)。本行可删,留作文档。
+5 -4
View File
@@ -6,8 +6,8 @@
- `offline-runtime-requirements.txt`:固定版本的完整依赖闭包。
- `prepare_wheelhouse.sh`:联网机器下载 Linux/Python 3.12 wheel,并验证传递依赖完整性。
- `start_backend.sh`:Docker 启动脚本,包含 `croniter`、`hindsight-client`、`aiomysql` 和 Redis 客户端的缺包补装逻辑。
- `wheelhouse/`:24 个 wheel 和 `SHA256SUMS`,需要随发布目录一起复制。
- `start_backend.sh`:Docker 启动脚本,包含 `croniter`、`hindsight-client`、aiomysql 兼容组和 Redis 客户端的缺包补装逻辑。
- `wheelhouse/`:26 个 wheel 和 `SHA256SUMS`,需要随发布目录一起复制。
## 联网机器准备
@@ -24,8 +24,9 @@ bash prepare_wheelhouse.sh
- `hindsight-client==0.6.2`
- `redis==5.2.1`
- `aiomysql==0.3.2`
- `SQLAlchemy==2.0.51`
上述依赖在 Python 3.12/Linux x86_64 下的 20 个传递依赖(包括 `PyMySQL==1.2.0`)也已锁定在 `offline-runtime-requirements.txt`。
上述依赖在 Python 3.12/Linux x86_64 下的 21 个传递依赖(包括 `PyMySQL==1.2.0` 和 `greenlet==3.2.5`)也已锁定在 `offline-runtime-requirements.txt`。
国内镜像示例:
@@ -58,7 +59,7 @@ bash start_backend.sh
docker logs -f deerflow-backend
```
脚本会先尝试导入依赖,仅在缺少模块时从 `/app/wheelhouse` 安装。`aiomysql` 补装默认开启,可通过 `AIOMYSQL_INSTALL_DEPS=0` 禁用,或通过 `AIOMYSQL_PIP_PACKAGES` 覆盖默认的 `aiomysql==0.3.2`。
脚本会先尝试导入依赖,仅在缺少模块或 SQLAlchemy 的 aiomysql `ping(reconnect)` 适配接口不兼容时从 `/app/wheelhouse` 安装。aiomysql 补装默认开启,并在一次命令中安装兼容组 `aiomysql==0.3.2 PyMySQL==1.2.0 SQLAlchemy==2.0.51`,避免旧版 SQLAlchemy 启动时出现 `ping() missing 1 required positional argument: reconnect`。可通过 `AIOMYSQL_INSTALL_DEPS=0` 禁用,或通过 `AIOMYSQL_PIP_PACKAGES` 覆盖整个兼容组。
MySQL 连接 URL 使用 aiomysql 驱动时应写为 `mysql+aiomysql://user:password@host:3306/database?charset=utf8mb4`。
+3 -1
View File
@@ -956,6 +956,8 @@ Both can be modified at runtime via Gateway API endpoints or `DeerFlowClient` me
`llmwiki.weknora.api_base_url` is the provider switch: non-empty enables WeKnora, empty preserves the legacy LLMWiki UI. WeKnora is independently deployed and owns PostgreSQL/Redis/object storage/vector storage/parsers/models; DeerFlow config contains only `api_base_url` and optional `web_base_url`. DeerFlow deliberately sends no WeKnora credential, so the WeKnora endpoint must be restricted to a trusted network and allow DeerFlow traffic itself or through its reverse proxy. There is intentionally no `default_embedding_model_id`: `WeKnoraClient` queries `/api/v1/models` and uses WeKnora's active Embedding model when creating a KB; Wiki creation additionally resolves an active KnowledgeQA model inside WeKnora for synthesis. The native BFF is `app/gateway/routers/llmwiki.py`; ownership/publication mappings live in `deerflow.persistence.llmwiki`; the browser only sees DeerFlow mapping IDs. Knowledge-base owners publish immediately (`private` -> `published`) without a review queue, and can unpublish ordinary bases. The conversation-deposit KB is system-owned, always published, shared by all DeerFlow users, excluded from My Knowledge, and immutable through user/admin detail operations; recognition requires both the reserved name and `system` owner. Existing same-name remote mappings are adopted in place as the system deposit, preserving mapping ids and document links. Its user-facing description, generated Markdown source and tags use `cmzs` branding; list/detail reads synchronize stale remote descriptions, and search results, source previews and embedded-detail JSON recursively replace legacy brand text in historical conversation-deposit payloads. Deposit requests must reference a thread accessible to the current account. Personal knowledge-base list/search views are scoped to the current DeerFlow account even for admins; frontend LLMWiki caches include the live account id, and admin write power does not make other users' private KBs appear as owned. Only `system_role === "admin"` sees the WeKnora connection-config tab. Knowledge cards navigate to a dedicated WeKnora-style detail route. Wiki-enabled bases render and default to `Wiki -> documents -> graph`; standard bases omit Wiki and render `documents -> graph`. The detail page supports sequential multi-file upload with an aggregate result for writable ordinary bases, hides it for public read-only/system-deposit bases, and uses a generic loading label while the embed session starts. It also supports URL and published manual-Markdown imports, browses real Wiki pages/content, proxies authenticated original-file previews, pages through `/chunks/{knowledge_id}`, and renders the Wiki page graph from `/knowledgebase/{kb_id}/wiki/graph`; document details use a right sheet rather than the removed KB modal. The credential-free proxy uses one WeKnora service identity, so per-user separation is enforced by DeerFlow mappings while all remote KBs physically share that identity's WeKnora workspace. WeKnora iframe detail pages first call `/api/llmwiki/knowledge-bases/{id}/weknora/session` through the normal DeerFlow API auth path, then load `/platform/knowledge-bases/{weknora_id}` with a signed `deerflow_weknora_embed` cookie for `/platform`, `/assets`, `/locales`, root assets, and API calls. Injected fetch/XHR rewriting sends all iframe `/api/v1/*`, including colliding `/api/v1/auth/*`, through `/api/llmwiki/weknora-embed/api/v1/*`; Gateway auth/CSRF bypass only applies to those signed iframe proxy requests, the proxy still revalidates the mapped KB, and cookie security honors `X-Forwarded-Proto` for HTTPS reverse proxies. `llmwiki_search` is conditionally registered in `tools.py`, is read-only, and rechecks the current DeerFlow user, agent `llmwiki_knowledge_base_ids`, and thread-context selection on every call. Source previews prefer the cleaned Wiki/LLMWiki page text when a retrieved chunk maps to a Wiki page, with raw split chunks as the fallback. Agent bindings use an inline WeKnora-style searchable/grouped multi-select in the standard agent create/detail UI; new chats use the same interaction pattern as a compact composer dropdown, and selections are written to thread metadata/context and restored on reopen. The DeerFlow frontend is native React and the WeKnora frontend is not modified or embedded. Full deployment and behavior notes: `docs/LLMWIKI_WEKNORA_INTEGRATION_ZH.md`. Tests: `tests/test_llmwiki_weknora.py`.
The embedded detail proxy hides WeKnora's outer `.main > .aside_box` application sidebar and expands the route outlet, but deliberately keeps the Wiki page's internal directory/index sidebar.
### Embedded Client (`packages/harness/deerflow/client.py`)
`DeerFlowClient` provides direct in-process access to all DeerFlow capabilities without HTTP services. All return types align with the Gateway API response schemas, so consumer code works identically in HTTP and embedded modes.
@@ -1184,7 +1186,7 @@ and timestamps; do not persist Markdown content or call a formal delivery
interface. Previews continue to read the DeerFlow artifact by `threadId + path`.
See `docs/` directory for detailed documentation:
- [../README.md](../README.md) - Offline Docker wheelhouse preparation and startup workflow. The pinned versions in `../offline-runtime-requirements.txt` and `../start_backend.sh` must stay synchronized. Startup checks `croniter`, `hindsight-client`, `aiomysql`, and Redis, installing missing packages from wheelhouse only; `aiomysql==0.3.2` includes the pinned `PyMySQL==1.2.0` transitive dependency. The download target defaults to CPython 3.12 on Linux x86_64 and verifies dependency closure using wheelhouse only.
- [../README.md](../README.md) - Offline Docker wheelhouse preparation and startup workflow. The pinned versions in `../offline-runtime-requirements.txt` and `../start_backend.sh` must stay synchronized. Startup checks `croniter`, `hindsight-client`, the aiomysql compatibility bundle, and Redis, installing missing or incompatible packages from wheelhouse only. The bundle installs `aiomysql==0.3.2`, `PyMySQL==1.2.0`, and `SQLAlchemy==2.0.51` together; the startup probe also verifies that SQLAlchemy's adapted `ping(reconnect=False)` signature is present, preventing the old-adapter missing-`reconnect` failure. The download target defaults to CPython 3.12 on Linux x86_64 and verifies dependency closure using wheelhouse only.
- [CONFIGURATION.md](docs/CONFIGURATION.md) - Configuration options
- [ARCHITECTURE.md](docs/ARCHITECTURE.md) - Architecture details
- [API.md](docs/API.md) - API reference
@@ -81,6 +81,7 @@ Per-thread isolated execution with virtual path translation:
- **Skills path**: `/mnt/skills` → `deer-flow/skills/` directory
- **Skills loading**: Recursively discovers nested `SKILL.md` files under `skills/{public,custom}` and preserves nested container paths
- **Conditional WeKnora LLMWiki**: a non-empty `llmwiki.weknora.api_base_url` enables the DeerFlow-native WeKnora BFF and UI; an empty address preserves the existing LLMWiki page. WeKnora remains independently deployed and owns its database, Redis, storage, parsers and models. DeerFlow keeps only the API/admin addresses, user/publication mappings, agent knowledge-base bindings and per-conversation selections; it stores and sends no WeKnora credential, so the endpoint must be restricted to a trusted network and allow DeerFlow traffic. Owners publish directly to the public LLMWiki without an approval queue for ordinary bases; the system-owned conversation-deposit base is always published, shared by all users, excluded from My Knowledge, immutable through user/admin detail operations, and recognized by both its reserved name and `system` owner. An existing same-name remote mapping is adopted in place as the system deposit so its mapping id and document links stay valid. Its user-facing knowledge-base description and generated document metadata use the `cmzs` brand; list/detail reads synchronize stale remote descriptions, while conversation-deposit search results, source previews and embedded-detail JSON rewrite legacy brand text to `cmzs`. Deposit requests must reference a thread accessible to the current account. Personal knowledge-base lists are scoped to the current DeerFlow account even for admins, while frontend LLMWiki query caches also include the active account id so switching accounts cannot reuse another account's list/detail data. Admins remain the only role that sees the WeKnora connection-config tab. Knowledge cards open a dedicated WeKnora-style detail route: Wiki-enabled bases default to Wiki, then documents and graph; standard bases hide Wiki and default to documents. Writable ordinary knowledge-base details expose a native multi-file upload action that submits files sequentially and reports aggregate success/failure; read-only public bases and the system deposit base do not. It supports file/URL/manual-Markdown imports, Wiki page browsing, document preview, real parsed chunks and Wiki graph nodes/edges. New chats use a compact WeKnora-style searchable/grouped multi-select inside the composer, while agent create/edit pages bind spaces through an inline panel. A single proxy service identity means DeerFlow users are isolated logically while their WeKnora entities share that identity's workspace. The read-only `llmwiki_search` tool reauthorizes the current DeerFlow user on every call and renders credential-free source previews; source previews prefer cleaned Wiki/LLMWiki text when a retrieved chunk maps to a Wiki page, with raw chunks as the fallback. The detail page first creates a signed iframe session through the normal DeerFlow API auth path, then points the iframe directly at the proxied WeKnora `/platform/...` route using the signed `deerflow_weknora_embed` cookie; global auth/CSRF bypass is limited to those signed iframe proxy requests, and HTTPS cookie attributes honor `X-Forwarded-Proto` behind reverse proxies. The iframe proxy keeps WeKnora's root-relative `/platform`, `/assets`, `/locales`, and `/api/v1/*` paths, but is registered last through `llmwiki.proxy_router` so it cannot intercept first-party Gateway APIs such as `POST /api/v1/auth/login/username`. See [docs/LLMWIKI_WEKNORA_INTEGRATION_ZH.md](docs/LLMWIKI_WEKNORA_INTEGRATION_ZH.md).
- **Embedded WeKnora detail chrome**: the proxy hides WeKnora's application sidebar (`.main > .aside_box`) and expands the detail outlet to the full iframe width, while preserving the Wiki page's own index/navigation panel.
- **File-write safety**: `str_replace` serializes read-modify-write per `(sandbox.id, path)` so isolated sandboxes keep concurrency even when virtual paths match
- **Tools**: `bash`, `ls`, `read_file`, `write_file`, `str_replace` (`bash` is disabled by default when using `LocalSandboxProvider`; use `AioSandboxProvider` for isolated shell access)
@@ -548,6 +548,42 @@ html.deerflow-weknora-embed .knowledge-layout {{
min-width: 0 !important;
background: var(--td-bg-color-page) !important;
}}
html.deerflow-weknora-embed [data-deerflow-weknora-chrome="hidden"],
html.deerflow-weknora-embed #app > .main > .aside_box,
html.deerflow-weknora-embed #app > aside,
html.deerflow-weknora-embed #app > .sidebar,
html.deerflow-weknora-embed #app > .side-bar,
html.deerflow-weknora-embed #app > .side-menu,
html.deerflow-weknora-embed #app > .t-layout__sider,
html.deerflow-weknora-embed #app > .t-layout-sider,
html.deerflow-weknora-embed #app > .layout-sider,
html.deerflow-weknora-embed #app > .layout-sidebar,
html.deerflow-weknora-embed #app > .platform-sidebar,
html.deerflow-weknora-embed #app > .platform-sider,
html.deerflow-weknora-embed #app > .app-sidebar,
html.deerflow-weknora-embed #app > .app-sider,
html.deerflow-weknora-embed #app > .t-layout > aside,
html.deerflow-weknora-embed #app > .t-layout > .t-layout__sider,
html.deerflow-weknora-embed #app > .t-layout > .t-layout-sider,
html.deerflow-weknora-embed #app > .layout > aside,
html.deerflow-weknora-embed #app > .layout > .layout-sider,
html.deerflow-weknora-embed #app > .layout > .layout-sidebar {{
display: none !important;
visibility: hidden !important;
pointer-events: none !important;
}}
html.deerflow-weknora-embed [data-deerflow-weknora-content="true"],
html.deerflow-weknora-embed #app,
html.deerflow-weknora-embed .t-layout,
html.deerflow-weknora-embed .t-layout__content,
html.deerflow-weknora-embed .layout-content,
html.deerflow-weknora-embed .platform-content {{
margin-left: 0 !important;
padding-left: 0 !important;
left: 0 !important;
width: 100% !important;
max-width: none !important;
}}
html.deerflow-weknora-embed .document-header {{
padding-left: 32px !important;
padding-right: 32px !important;
@@ -743,6 +779,86 @@ html.deerflow-weknora-embed [class*=" guide__"] {{
applyTheme(window.__deerflowWeKnoraTheme);
}}, 80);
}}
const shellSidebarSelector = [
".main > .aside_box",
"aside",
".sidebar",
".side-bar",
".side-menu",
".nav-sidebar",
".navigation-sidebar",
".menu-sidebar",
".t-layout__sider",
".t-layout-sider",
".layout-sider",
".layout-sidebar",
".platform-sidebar",
".platform-sider",
".app-sidebar",
".app-sider",
"[class*='Sidebar']",
"[class*='sidebar']",
"[class*='SideBar']",
"[class*='sideBar']",
"[class*='Sider']",
"[class*='sider']"
].join(",");
function isInsideKnowledgeDetail(node) {{
return Boolean(
node.closest(
".knowledge-layout,.document-layout,.document-detail,.wiki-layout,.wiki-page,.graph-layout,.graph-page,[class*='knowledge-detail'],[class*='wiki-detail']"
)
);
}}
function isLikelyOuterSidebar(node) {{
if (!node || node === document.body || node === document.documentElement) return false;
if (isInsideKnowledgeDetail(node)) return false;
const rect = node.getBoundingClientRect();
if (!rect.width || !rect.height) return false;
const style = getComputedStyle(node);
const className = String(node.className || "").toLowerCase();
const namedLikeShell =
/sidebar|side-bar|side_menu|side-menu|sider|layout-sider|platform-sider|platform-sidebar|app-sider|app-sidebar|nav-sidebar|menu-sidebar/.test(
className
);
const nearLeft = rect.left <= 4;
const sidebarWidth = rect.width >= 40 && rect.width <= Math.min(360, Math.max(120, window.innerWidth * 0.42));
const tallEnough = rect.height >= Math.max(240, window.innerHeight * 0.45);
const topShellChild =
node.parentElement === document.body ||
node.parentElement === document.getElementById("app") ||
node.parentElement?.parentElement === document.getElementById("app") ||
node.closest("#app") === node.parentElement?.closest("#app");
const positionedShell = style.position === "fixed" || style.position === "sticky";
const text = (node.textContent || "").slice(0, 400);
const containsAppNav = /知识库|工作台|空间|租户|模型|智能体|设置|Knowledge|Workspace|Agent|Model|Tenant/.test(text);
return namedLikeShell && nearLeft && sidebarWidth && tallEnough && (topShellChild || positionedShell || containsAppNav);
}}
function hideWeKnoraShell() {{
try {{
document.querySelectorAll(shellSidebarSelector).forEach((node) => {{
if (isLikelyOuterSidebar(node)) {{
node.dataset.deerflowWeknoraChrome = "hidden";
node.setAttribute("aria-hidden", "true");
}}
}});
document
.querySelectorAll(
"#app,.t-layout,.t-layout__content,.layout-content,.platform-content,.platform-route-outlet,.main"
)
.forEach((node) => {{
if (!isInsideKnowledgeDetail(node)) {{
node.dataset.deerflowWeknoraContent = "true";
const element = node;
element.style.marginLeft = "0";
element.style.paddingLeft = "0";
element.style.left = "0";
element.style.width = "100%";
element.style.maxWidth = "none";
}}
}});
}} catch (_) {{}}
}}
const rawPushState = history.pushState.bind(history);
const rawReplaceState = history.replaceState.bind(history);
history.pushState = (state, title, url) => rawPushState(state, title, url ? allowedUrl(url) : location.href);
@@ -760,6 +876,7 @@ html.deerflow-weknora-embed [class*=" guide__"] {{
}}, true);
enforceLocation();
purgeGuides();
hideWeKnoraShell();
let ready = false;
function hasDetailShell() {{
return Boolean(document.querySelector(".knowledge-layout") || document.querySelector(".document-header"));
@@ -767,6 +884,7 @@ html.deerflow-weknora-embed [class*=" guide__"] {{
function markReady(force = false) {{
enforceLocation();
purgeGuides();
hideWeKnoraShell();
if (ready) return;
if (!force && (location.pathname !== allowedPath || !hasDetailShell())) return;
ready = true;
@@ -785,6 +903,7 @@ html.deerflow-weknora-embed [class*=" guide__"] {{
setInterval(() => {{
enforceLocation();
purgeGuides();
hideWeKnoraShell();
}}, 500);
setTimeout(() => markReady(true), 2200);
}})();
@@ -890,6 +890,28 @@ def test_weknora_iframe_proxy_is_fallback_after_deerflow_api_routes() -> None:
assert _first_gateway_match("GET", "/platform/knowledge-bases/remote-kb") == "/platform/{proxied_path:path}"
def test_weknora_iframe_hides_current_outer_sidebar_shell() -> None:
from app.gateway.routers import llmwiki as llmwiki_router
ctx = llmwiki_router.WeKnoraEmbedContext(
mapping_id="local-kb",
weknora_id="remote-kb",
user_id="user-a",
is_admin=False,
can_write=False,
exp=4_102_444_800,
)
injection = llmwiki_router._weknora_embed_injection(
allowed_path="/platform/knowledge-bases/remote-kb",
tab="wiki",
session={},
ctx=ctx,
)
assert "#app > .main > .aside_box" in injection
assert '".main > .aside_box"' in injection
def test_weknora_iframe_filters_kb_list_to_current_mapping() -> None:
from app.gateway.routers import llmwiki as llmwiki_router
@@ -1,9 +1,12 @@
# Direct runtime packages layered into the prebuilt offline backend image.
# Keep these four pins synchronized with start_backend.sh.
# Keep the aiomysql/PyMySQL/SQLAlchemy compatibility bundle synchronized with
# start_backend.sh. Installing aiomysql alone can leave the prebuilt image's
# older SQLAlchemy adapter with an incompatible ping(reconnect) signature.
croniter==6.2.2
hindsight-client==0.6.2
redis==5.2.1
aiomysql==0.3.2
SQLAlchemy==2.0.51
# Fully pinned dependency closure for CPython 3.12 on Linux. Keeping the
# transitive packages here makes a newly generated wheelhouse reproducible.
@@ -15,6 +18,7 @@ annotated-types==0.7.0
async-timeout==5.0.1
attrs==26.1.0
frozenlist==1.8.0
greenlet==3.2.5
idna==3.18
multidict==6.7.1
propcache==0.5.2
@@ -61,6 +61,8 @@ require_wheel 'hindsight_client-0.6.2-*.whl'
require_wheel 'redis-5.2.1-*.whl'
require_wheel 'aiomysql-0.3.2-*.whl'
require_wheel 'pymysql-1.2.0-*.whl'
require_wheel 'sqlalchemy-2.0.51-*.whl'
require_wheel 'greenlet-3.2.5-*.whl'
# Resolve the same requirements a second time using only wheelhouse. This
# catches a missing transitive dependency before the bundle reaches offline.
+2 -2
View File
@@ -131,9 +131,9 @@ if [ "${HINDSIGHT_INSTALL_DEPS:-1}" != "0" ]; then
}
fi
if [ "${AIOMYSQL_INSTALL_DEPS:-1}" != "0" ]; then
uv run --no-sync python -c "import aiomysql" >/dev/null 2>&1 || {
uv run --no-sync python -c "import aiomysql, pymysql; import inspect; from sqlalchemy.dialects.mysql.aiomysql import AsyncAdapt_aiomysql_connection; assert inspect.signature(AsyncAdapt_aiomysql_connection.ping).parameters.get(\"reconnect\").default is False" >/dev/null 2>&1 || {
PYTHON_BIN="$(uv run --no-sync python -c "import sys; print(sys.executable)")"
uv pip install --python "${PYTHON_BIN}" --no-index --find-links /app/wheelhouse ${AIOMYSQL_PIP_PACKAGES:-aiomysql==0.3.2}
uv pip install --python "${PYTHON_BIN}" --no-index --find-links /app/wheelhouse ${AIOMYSQL_PIP_PACKAGES:-aiomysql==0.3.2 PyMySQL==1.2.0 SQLAlchemy==2.0.51}
}
fi
uv run --no-sync python -c "import redis" >/dev/null 2>&1 || uv pip install --python "$(uv run --no-sync python -c "import sys; print(sys.executable)")" --no-index --find-links /app/wheelhouse ${REDIS_PIP_PACKAGES:-redis==5.2.1}