feat: preserve Wiki vectors across knowledge transfers and add local BGE-M3
Unify Wiki reading, preserve metadata and graph links, add asynchronous file-backed transfers, and reuse compatible vectors. Includes regression coverage and live acceptance notes.
This commit is contained in:
吉鑫升 committed
1 parent
49d9ced04f
commit
60e2c72514
38 files changed
+2536
-581
No files matched your search
+5
-2
@@ -91,6 +91,7 @@ offline-backend-20260512/backend/tests/*
|
||||
# Keep the openGauss URL/dialect/schema compatibility regression coverage versioned.
|
||||
!offline-backend-20260512/backend/tests/test_gauss_database.py
|
||||
!offline-backend-20260512/backend/tests/test_llmwiki_weknora.py
|
||||
!offline-backend-20260512/backend/tests/test_llmwiki_knowledge_management.py
|
||||
!offline-backend-20260512/backend/tests/test_health_auth_bypass.py
|
||||
# Keep formal Word OOXML and embedded-font regression coverage versioned.
|
||||
!offline-backend-20260512/backend/tests/test_word_export.py
|
||||
@@ -172,8 +173,10 @@ docs/backups
|
||||
# Frontend skills components must be tracked (excluded from the broad skills/ ignore above)
|
||||
!frontend-web/src/components/workspace/skills/
|
||||
|
||||
# uv lockfile (not tracked per project preference)
|
||||
offline-backend-20260512/backend/uv.lock
|
||||
# uv lockfile (not tracked per project preference)
|
||||
offline-backend-20260512/backend/uv.lock
|
||||
|
||||
!offline-backend-20260512/backend/tests/test_wiki_package_roundtrip.py
|
||||
|
||||
# Local-only scratch directory (not committed)
|
||||
0622/
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
# Wiki 数据流转与本地编码验收（2026-09-03）
|
||||
|
||||
## 结论
|
||||
|
||||
本地 BGE-M3 小批量编码已安装、配置并实际运行。已有 Wiki 的向量化、自动沉淀、
|
||||
普通库文件导出、助手库异步导入、助手库文件导出、空普通库回流及两端向量检索已实测通过。
|
||||
**技能自动生成 Wiki 尚未完成验收**：WeKnora 的两个生成模型都返回 HTTP 402
|
||||
`Insufficient Balance`。不能把后半段通过等同于全链路通过。
|
||||
|
||||
本次修改运行在本地 DeerFlow 开发服务（前端 5175、后端 8001），连接用户提供的远端
|
||||
WeKnora；没有把业务代码发布到服务器，没有提交 Git，也没有直接写 WeKnora 图数据库。
|
||||
|
||||
## 本地编码配置
|
||||
|
||||
- 实现：FastEmbed / ONNX Runtime CPU，官方 `BAAI/bge-m3` 权重。
|
||||
- 配置名：`bge-embedding-m3`，1024 维，每批 2 条，2 个 CPU 推理线程。
|
||||
- 运行依赖：后端 `local-embedding` extra；当前本地虚拟环境已安装。
|
||||
- 权重：约 2.3 GB，持久缓存 `C:\Users\ji\.cache\deerflow\embeddings`，不进入业务仓库。
|
||||
- 配置：`llmwiki.local_wiki_index.embedding.provider: local`；可用
|
||||
`local_model_path` 指定离线权重目录，详见后端 README。
|
||||
- 查询向量在本地实时编码。已有 Wiki 在内容哈希、模型指纹匹配且向量非空时直接复用。
|
||||
不同模型或维度不能混用；导入不静默重新编码。
|
||||
- 原始文档分段向量与模型生成后的 Wiki 文本不同，不能相互冒用。本次新建的是 Wiki
|
||||
正文的 BGE-M3 索引，不是把既有 512 维文档向量改名为 1024 维。
|
||||
|
||||
## 实际数据验收
|
||||
|
||||
来源是已有普通知识库“图片测试111”，没有改写其 Wiki 内容。
|
||||
|
||||
| 阶段 | Wiki | Wiki 向量 | 实体 | 引用关系 |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| 普通知识库导出 | 19 | 101 | 19 | 108 |
|
||||
| 助手知识库导入后导出 | 19 | 101 | 19 | 108 |
|
||||
| 新建普通知识库回流后导出 | 19 | 101 | 19 | 108 |
|
||||
| 知识梳理总库导出（7 个来源记录） | 19 | 101 | 19 | 108 |
|
||||
|
||||
逐项断言通过：
|
||||
|
||||
1. 三份包的 Wiki slug、标题、正文、摘要、页面类型、别名、分类路径、出链一致。
|
||||
2. 101 条向量的原始 float32 字节、分段编号、维度和模型指纹完全一致。
|
||||
3. 将导入前后实体 ID 映射回原 Wiki slug 后，108 条关系的端点及谓词集合完全一致。
|
||||
4. 总库导出没有因多个来源重复同步而重复导出同一当前页面的分段向量。
|
||||
5. 元数据回归测试另覆盖 tags、目录恢复、原始文档片段不计入 Wiki 数量、原生技能无
|
||||
ownership 记录时可被选中、短 Wiki 不漏向量、模型指纹不兼容时不假装向量检索。
|
||||
|
||||
普通包约 1.01 MB，助手包约 0.79 MB，回流包约 0.83 MB。普通包还带 2 个原始文档及
|
||||
108 个原始片段；助手导出提供 Wiki、Wiki 向量和关系，不重新上传原始文档触发二次生成。
|
||||
|
||||
### 检索
|
||||
|
||||
实际查询：“军方用什么工具评估极端天气带来的风险？”
|
||||
|
||||
- 回流普通库 `/api/llmwiki/search` 返回 `mode=vector`，首条“国防气候评估工具”。
|
||||
- 助手库 `/api/assistant-knowledge/search` 返回 `retrieval_mode=vector`，首条同上；
|
||||
首个命中分段余弦相似度约 0.6343。
|
||||
- 浏览器实际提交总库检索，显示“向量检索 · 余弦相似度”和相同命中；不是只检查返回字段。
|
||||
- 普通库采用已有页面级聚合评分，因此页面最终分数与单分段余弦值不必相等。
|
||||
- 回流的向量写入 **DeerFlow 管理的本地 Wiki 索引**。未写入 WeKnora 原生向量数据库，
|
||||
本次结论适用于本系统问答检索，不能声称原生 WeKnora 自身的检索也已完成该向量导入。
|
||||
|
||||
### 页面
|
||||
|
||||
普通与助手库复用同一 Wiki 阅读组件：目录树、类型/标签筛选、分类面包屑、别名标签、
|
||||
正文内链、出链、反向链接及邻接关系视图。窄窗口可收起目录。
|
||||
|
||||
浏览器验证了“索引 → 国防气候评估工具”、展开关系视图、助手 Wiki 跳回普通库来源，
|
||||
以及普通 Wiki 的“在知识梳理中查看”反向跳转。
|
||||
|
||||
### 验收数据位置
|
||||
|
||||
- 原有普通库：`48f669dc-433e-4aab-b5ea-897b6f7a3d2f`
|
||||
- 知识梳理总库：`c3fd9841-0718-44dc-860e-aae3020fd485`
|
||||
- 最终助手测试库：`489be905-08c8-40fe-8d60-42ac21dd945a`
|
||||
- 最终普通回流测试库：`428162aa-03f6-4c61-8994-360f2f39e0c9`
|
||||
|
||||
新建测试库名称均以“Wiki链路验收-0903-1826”开头，保留用于复核；未删除用户已有库。
|
||||
本地 `.runtime/wiki-live-result.json` 保存验收结果，`.runtime/wiki_live_acceptance.py`
|
||||
保存本次验收流程，均为忽略的运行产物，不包含认证口令。
|
||||
|
||||
## 这次实际修复的问题
|
||||
|
||||
- 内置技能只有磁盘目录、没有归属记录时，被错误判定为不存在。
|
||||
- 技能应上传去代码后的知识正文，让 WeKnora 生成 Wiki，而不是伪造一个 `skill` 类型页面。
|
||||
- Wiki 生成等待使用实际 stats 接口；零页结果明确失败，不无期限等待。
|
||||
- 向量化完成后自动导出沉淀到总库，并阻断导出回调重复触发全量同步。
|
||||
- 短摘要和短正文同时存在时，旧分段阈值会将整页过滤为零向量。
|
||||
- 分类路径必须先创建 WeKnora folder 并绑定 folder_id；只传 category_path 会被服务端清空。
|
||||
- Wiki 的标签/分类/别名/原始引用保留；图节点绑定真实 Wiki，避免生成重复的空实体页面。
|
||||
- 保留归一化 float32 原始字节；导入中不能用 `section_index or index`，因为合法编号 0 会被改写。
|
||||
- 总库多个来源保留审计，但检索与导出只使用当前 Wiki 版本对应的向量，避免旧贡献污染检索。
|
||||
- 包解析改为同盘临时 SQLite，上传先落盘再异步导入，导出完成后才提供文件下载。
|
||||
|
||||
## 阻塞与未通过的范围
|
||||
|
||||
1. 技能 `knowledge-base-ingest` 已送入专门创建的普通测试库，但远端
|
||||
`deepseek-v4-flash` 和 `qwen3.6-flash` 均因余额不足失败。WeKnora 即使失败也可能把
|
||||
文档 parse_status 标为 completed。需恢复可用生成模型后重新运行技能全链路验收。
|
||||
2. 已采用磁盘暂存和有界批处理，但本次只有约 1 MB 实测，**未做数 GB 压测、断点续传、
|
||||
进程中断恢复验收**。大包部署仍需磁盘容量、代理 body size 和超时配置。
|
||||
3. 当前实体去重主要按类型及规范化名称；这次没有完成跨别名、歧义实体的模型对齐及
|
||||
多来源事实冲突合并验收，不能视为这一历史需求已全面完成。
|
||||
4. 重复上传仍会保留独立来源记录。向量导出已消除旧版本重复，但上传身份幂等、部分远端
|
||||
写入失败后的无损重试、跨实例图片资源打包仍需独立验证。
|
||||
5. 自动沉淀挂在本系统成功向量化完成后；未新增文件监听或定时远程写入。直接在外部
|
||||
WeKnora 修改的内容仍需通过本系统同步/向量化入口接入。
|
||||
6. 回流当前只允许新建空 Wiki 普通库，避免覆盖已有页面。普通库现有页面列表的超大规模
|
||||
分页与助手问答的最终生成回答不属于这轮通过结论。
|
||||
|
||||
## 自动化检查
|
||||
|
||||
- 后端相关测试：46 passed（assistant repository、Wiki sync、package roundtrip、技能扫描、
|
||||
技能关系仓储、知识库管理）。
|
||||
- 修改的 Python 功能文件 Ruff 检查通过；app.py 既有启动顺序导致的 E402 不在此次改动中处理。
|
||||
- `git diff --check` 通过（只有仓库换行提示）。
|
||||
- 前端完整 `pnpm typecheck` 未通过：57 个既有/本轮范围外错误，涉及 nextra 缺失、Markdown
|
||||
插件类型及其他模块；本轮新建/修改的 Wiki 组件、助手 API、技能面板没有出现在错误清单中。
|
||||
浏览器实际页面与跳转已验证，但不能声称全项目构建通过。
|
||||
@@ -25,6 +25,10 @@ const PHASE_LABELS: Record<string, string> = {
|
||||
awaiting_review: "等待审核",
|
||||
merging: "合并实体关系",
|
||||
writing_wiki: "写入 Wiki",
|
||||
writing_sources: "上传知识资料",
|
||||
generating_wiki: "分析并生成分类 Wiki",
|
||||
vectorizing_wiki: "Wiki 向量化",
|
||||
syncing_assistant: "同步知识梳理总库",
|
||||
activating: "激活新版本",
|
||||
completed: "完成",
|
||||
};
|
||||
@@ -35,7 +39,7 @@ export function SkillKnowledgePanel() {
|
||||
const [selectedSkills, setSelectedSkills] = useState<string[]>([]);
|
||||
const [skillQuery, setSkillQuery] = useState("");
|
||||
const [selectedTargetKey, setSelectedTargetKey] = useState<string | null>(null);
|
||||
const [reviewMode, setReviewMode] = useState<"auto_high_confidence" | "off">("auto_high_confidence");
|
||||
const [reviewMode, setReviewMode] = useState<"auto_high_confidence" | "off">("off");
|
||||
const runtime = useLlmWikiRuntime();
|
||||
const knowledgeBases = useLlmWikiKnowledgeBases("all", runtime.data?.provider === "weknora");
|
||||
const bindings = useSkillKnowledgeBindings();
|
||||
|
||||
@@ -1,443 +1,425 @@
|
||||
import { BookOpen, Database, Download, Loader2, Plus, RefreshCw, Search, ShieldCheck, Trash2, Upload } from "lucide-react";
|
||||
import {
|
||||
ArrowLeft,
|
||||
BookOpenText,
|
||||
Database,
|
||||
Loader2,
|
||||
Plus,
|
||||
RefreshCw,
|
||||
Search,
|
||||
Upload,
|
||||
} from "lucide-react";
|
||||
import { useEffect, useMemo, useRef, useState } from "react";
|
||||
import { useSearchParams } from "react-router-dom";
|
||||
import { useQueryClient } from "@tanstack/react-query";
|
||||
import { toast } from "sonner";
|
||||
|
||||
import { Badge } from "@/components/ui/badge";
|
||||
import { Button } from "@/components/ui/button";
|
||||
import {
|
||||
Dialog,
|
||||
DialogContent,
|
||||
DialogDescription,
|
||||
DialogFooter,
|
||||
DialogHeader,
|
||||
DialogTitle,
|
||||
DialogFooter,
|
||||
} from "@/components/ui/dialog";
|
||||
import { Input } from "@/components/ui/input";
|
||||
import { ScrollArea } from "@/components/ui/scroll-area";
|
||||
import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs";
|
||||
import { Textarea } from "@/components/ui/textarea";
|
||||
import { KnowledgeProductNav } from "@/components/workspace/knowledge-product-nav";
|
||||
import { MarkdownContent } from "@/components/workspace/messages/markdown-content";
|
||||
import { getStoredAuth } from "@/core/auth";
|
||||
import { buildAppHashUrl } from "@/core/llmwiki/source-links";
|
||||
import {
|
||||
type AssistantKnowledgeBase,
|
||||
type AssistantWikiPage,
|
||||
searchAssistantWiki,
|
||||
useAssistantImportJobs,
|
||||
useAssistantKnowledgeActions,
|
||||
useAssistantKnowledgeBases,
|
||||
useAssistantKnowledgeSources,
|
||||
useAssistantWikiPages,
|
||||
} from "@/strategy-components/api/assistant-knowledge";
|
||||
|
||||
function baseSubtitle(base: AssistantKnowledgeBase) {
|
||||
if (base.is_global) return "全库沉淀总库 · 所有 WeKnora 导入数据都会同步进入这里并做实体对齐";
|
||||
return base.description || "自建助手知识库 · 可上传普通知识库导出的 Wiki 向量包";
|
||||
}
|
||||
|
||||
function EmptyInitializeCard({
|
||||
isAdmin,
|
||||
onInitialize,
|
||||
pending,
|
||||
}: {
|
||||
isAdmin: boolean;
|
||||
onInitialize: () => void;
|
||||
pending: boolean;
|
||||
}) {
|
||||
return (
|
||||
<div className="flex flex-1 items-center justify-center p-8">
|
||||
<div className="max-w-lg rounded-xl border bg-card p-8 text-center">
|
||||
<Database className="mx-auto mb-4 size-10 text-primary" />
|
||||
<h1 className="text-xl font-semibold">初始化助手知识库</h1>
|
||||
<p className="mt-2 text-sm text-muted-foreground">
|
||||
未配置 WeKnora 时会默认展示助手知识库。首次使用需要创建“知识梳理总库”，后续从 WeKnora 导入的数据都会沉淀到这里。
|
||||
</p>
|
||||
{isAdmin ? (
|
||||
<Button className="mt-5" disabled={pending} onClick={onInitialize}>
|
||||
{pending ? <Loader2 className="mr-2 size-4 animate-spin" /> : <Database className="mr-2 size-4" />}
|
||||
创建知识梳理总库
|
||||
</Button>
|
||||
) : (
|
||||
<p className="mt-5 text-sm">请联系管理员初始化。</p>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
import {
|
||||
WikiLibraryReader,
|
||||
type LibraryWikiPage,
|
||||
} from "@/strategy-components/components/resource-management/llmwiki/WikiLibraryReader";
|
||||
import { WikiPackageTransfers } from "@/strategy-components/components/resource-management/llmwiki/WikiPackageTransfers";
|
||||
|
||||
export default function AssistantKnowledgePage() {
|
||||
const isAdmin = getStoredAuth()?.system_role === "admin";
|
||||
const [query, setQuery] = useState("");
|
||||
const [selectedBaseId, setSelectedBaseId] = useState<string | null>(null);
|
||||
const [selected, setSelected] = useState<AssistantWikiPage | null>(null);
|
||||
const [createOpen, setCreateOpen] = useState(false);
|
||||
const [createDraft, setCreateDraft] = useState({ name: "", description: "" });
|
||||
const packageInputRef = useRef<HTMLInputElement | null>(null);
|
||||
const [params, setParams] = useSearchParams();
|
||||
const baseId = params.get("base_id");
|
||||
const slug = params.get("wikiSlug");
|
||||
const bases = useAssistantKnowledgeBases();
|
||||
const actions = useAssistantKnowledgeActions();
|
||||
|
||||
useEffect(() => {
|
||||
const values = bases.data ?? [];
|
||||
if (!values.length) {
|
||||
setSelectedBaseId(null);
|
||||
return;
|
||||
}
|
||||
if (selectedBaseId && values.some((base) => base.id === selectedBaseId)) return;
|
||||
setSelectedBaseId((values.find((base) => base.is_global) ?? values[0]).id);
|
||||
}, [bases.data, selectedBaseId]);
|
||||
|
||||
useEffect(() => {
|
||||
setSelected(null);
|
||||
}, [selectedBaseId]);
|
||||
|
||||
const selectedBase = useMemo(
|
||||
() => (bases.data ?? []).find((base) => base.id === selectedBaseId) ?? null,
|
||||
[bases.data, selectedBaseId],
|
||||
const pages = useAssistantWikiPages("", baseId);
|
||||
const sources = useAssistantKnowledgeSources(baseId);
|
||||
const jobs = useAssistantImportJobs(baseId);
|
||||
const client = useQueryClient();
|
||||
const [tab, setTab] = useState("wiki");
|
||||
const [createOpen, setCreateOpen] = useState(false);
|
||||
const [name, setName] = useState("");
|
||||
const [query, setQuery] = useState("");
|
||||
const [searching, setSearching] = useState(false);
|
||||
const [searchResult, setSearchResult] = useState<Awaited<
|
||||
ReturnType<typeof searchAssistantWiki>
|
||||
> | null>(null);
|
||||
const upload = useRef<HTMLInputElement>(null);
|
||||
const base = bases.data?.find((b) => b.id === baseId);
|
||||
const libraryPages = useMemo<LibraryWikiPage[]>(
|
||||
() =>
|
||||
(pages.data || []).map((p) => ({
|
||||
...p,
|
||||
content: p.content_markdown,
|
||||
status: "published",
|
||||
aliases: p.aliases || [],
|
||||
source_refs: p.source_refs || [],
|
||||
in_links: p.in_links || [],
|
||||
out_links: p.out_links || [],
|
||||
version: 1,
|
||||
})),
|
||||
[pages.data],
|
||||
);
|
||||
const pages = useAssistantWikiPages(query, selectedBaseId);
|
||||
const sources = useAssistantKnowledgeSources(selectedBaseId);
|
||||
const jobs = useAssistantImportJobs(selectedBaseId);
|
||||
const activePage = selected ?? pages.data?.[0] ?? null;
|
||||
const counts = useMemo(
|
||||
() => ({
|
||||
pages: pages.data?.length ?? selectedBase?.page_count ?? 0,
|
||||
sources: sources.data?.length ?? selectedBase?.source_count ?? 0,
|
||||
jobs: jobs.data?.length ?? 0,
|
||||
}),
|
||||
[jobs.data, pages.data, selectedBase?.page_count, selectedBase?.source_count, sources.data],
|
||||
);
|
||||
|
||||
const initializeGlobal = () => {
|
||||
actions.initialize.mutate("知识梳理总库", {
|
||||
onSuccess: (base) => {
|
||||
setSelectedBaseId(base.id);
|
||||
toast.success("知识梳理总库已创建");
|
||||
},
|
||||
onError: (error) => toast.error(error.message),
|
||||
const active =
|
||||
libraryPages.find((p) => p.slug === slug) ||
|
||||
libraryPages.find((p) => p.page_type === "index") ||
|
||||
libraryPages[0] ||
|
||||
null;
|
||||
const navigatePage = (nextSlug: string) => {
|
||||
setParams({ base_id: baseId!, wikiSlug: nextSlug });
|
||||
setTab("wiki");
|
||||
};
|
||||
const finishedJobs = (jobs.data || [])
|
||||
.filter((j) => j.status === "completed")
|
||||
.map((j) => j.id)
|
||||
.join(",");
|
||||
useEffect(() => {
|
||||
void client.invalidateQueries({
|
||||
queryKey: ["assistant-knowledge", "pages"],
|
||||
});
|
||||
};
|
||||
|
||||
const createBase = async () => {
|
||||
const name = createDraft.name.trim();
|
||||
if (!name) {
|
||||
toast.error("请输入助手知识库名称");
|
||||
return;
|
||||
}
|
||||
void client.invalidateQueries({
|
||||
queryKey: ["assistant-knowledge", "bases"],
|
||||
});
|
||||
}, [finishedJobs, client]);
|
||||
useEffect(() => {
|
||||
setSearchResult(null);
|
||||
setQuery("");
|
||||
}, [baseId]);
|
||||
const create = async () => {
|
||||
if (!name.trim()) return;
|
||||
try {
|
||||
const base = await actions.createBase.mutateAsync({
|
||||
name,
|
||||
description: createDraft.description.trim(),
|
||||
});
|
||||
setSelectedBaseId(base.id);
|
||||
setCreateDraft({ name: "", description: "" });
|
||||
const b = await actions.createBase.mutateAsync({ name: name.trim() });
|
||||
setCreateOpen(false);
|
||||
toast.success("助手知识库已创建");
|
||||
} catch (error) {
|
||||
toast.error(error instanceof Error ? error.message : String(error));
|
||||
setName("");
|
||||
setParams({ base_id: b.id });
|
||||
} catch (e) {
|
||||
toast.error((e as Error).message);
|
||||
}
|
||||
};
|
||||
|
||||
const uploadPackage = async (file: File | null | undefined) => {
|
||||
if (!file) return;
|
||||
const search = async () => {
|
||||
if (!query.trim() || !baseId) return;
|
||||
setSearching(true);
|
||||
try {
|
||||
await actions.uploadWeKnoraPackage.mutateAsync({
|
||||
file,
|
||||
assistantBaseId: selectedBaseId,
|
||||
});
|
||||
toast.success("导出包已上传，后台正在异步导入");
|
||||
} catch (error) {
|
||||
toast.error(error instanceof Error ? error.message : String(error));
|
||||
setSearchResult(await searchAssistantWiki(query, baseId));
|
||||
} catch (e) {
|
||||
toast.error((e as Error).message);
|
||||
} finally {
|
||||
if (packageInputRef.current) packageInputRef.current.value = "";
|
||||
setSearching(false);
|
||||
}
|
||||
};
|
||||
|
||||
if (bases.isLoading) {
|
||||
return (
|
||||
<div className="flex size-full flex-col">
|
||||
<KnowledgeProductNav />
|
||||
<div className="flex flex-1 items-center justify-center">
|
||||
<Loader2 className="animate-spin" />
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
if (!bases.data?.length) {
|
||||
return (
|
||||
<div className="flex size-full flex-col">
|
||||
<KnowledgeProductNav />
|
||||
<EmptyInitializeCard
|
||||
isAdmin={isAdmin}
|
||||
pending={actions.initialize.isPending}
|
||||
onInitialize={initializeGlobal}
|
||||
/>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
const refresh = () =>
|
||||
void client.invalidateQueries({ queryKey: ["assistant-knowledge"] });
|
||||
return (
|
||||
<div className="flex size-full min-h-0 flex-col bg-background">
|
||||
<KnowledgeProductNav />
|
||||
<div className="grid min-h-0 flex-1 grid-cols-[280px_1fr]">
|
||||
<aside className="flex min-h-0 flex-col border-r bg-muted/20">
|
||||
<div className="border-b p-4">
|
||||
<div className="flex items-center justify-between">
|
||||
<div>
|
||||
<h2 className="text-sm font-semibold">助手知识库</h2>
|
||||
<p className="mt-1 text-xs text-muted-foreground">总库置顶，自建库可上传 WeKnora 导出包</p>
|
||||
</div>
|
||||
{!baseId ? (
|
||||
<main className="flex-1 overflow-auto p-6 lg:p-8">
|
||||
<header className="mb-8 flex items-center justify-between">
|
||||
<div>
|
||||
<h1 className="text-2xl font-semibold">助手知识库</h1>
|
||||
<p className="mt-2 text-sm text-muted-foreground">
|
||||
浏览分类
|
||||
Wiki、关联知识与来源。普通知识库向量化完成后自动沉淀到知识梳理。
|
||||
</p>
|
||||
</div>
|
||||
{isAdmin && (
|
||||
<Button onClick={() => setCreateOpen(true)}>
|
||||
<Plus className="mr-2 size-4" />
|
||||
新建知识库
|
||||
</Button>
|
||||
)}
|
||||
</header>
|
||||
{bases.isLoading && <Loader2 className="animate-spin" />}
|
||||
{bases.error && (
|
||||
<p className="text-destructive">{bases.error.message}</p>
|
||||
)}
|
||||
<div className="grid gap-5 md:grid-cols-2 xl:grid-cols-3">
|
||||
{bases.data?.map((b) => (
|
||||
<button
|
||||
key={b.id}
|
||||
onClick={() => setParams({ base_id: b.id })}
|
||||
className="group rounded-2xl border bg-card p-6 text-left shadow-sm transition hover:border-primary/40 hover:shadow-md"
|
||||
>
|
||||
<div className="mb-5 flex items-center justify-between">
|
||||
<div className="rounded-xl bg-emerald-100 p-3 text-emerald-700 dark:bg-emerald-950">
|
||||
<Database className="size-6" />
|
||||
</div>
|
||||
{b.is_global && (
|
||||
<Badge variant="secondary">全库知识梳理</Badge>
|
||||
)}
|
||||
</div>
|
||||
<h2 className="text-lg font-semibold group-hover:text-primary">
|
||||
{b.name}
|
||||
</h2>
|
||||
<p className="mt-2 line-clamp-2 min-h-10 text-sm text-muted-foreground">
|
||||
{b.is_global
|
||||
? "汇聚普通知识库的分析成果，保留分类、标签和知识关联。"
|
||||
: b.description ||
|
||||
"独立的 Wiki 知识空间，支持导入、导出与向量检索。"}
|
||||
</p>
|
||||
<div className="mt-6 flex gap-5 text-xs text-muted-foreground">
|
||||
<span>{b.page_count || 0} Wiki</span>
|
||||
<span>{b.source_count || 0} 来源</span>
|
||||
<span>{b.relation_count || 0} 关系</span>
|
||||
</div>
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
{!bases.isLoading && !bases.data?.length && isAdmin && (
|
||||
<Button
|
||||
className="mt-5"
|
||||
onClick={() =>
|
||||
actions.initialize.mutate("知识梳理总库", {
|
||||
onError: (e) => toast.error(e.message),
|
||||
})
|
||||
}
|
||||
>
|
||||
初始化知识梳理总库
|
||||
</Button>
|
||||
)}
|
||||
</main>
|
||||
) : (
|
||||
<>
|
||||
<header className="flex shrink-0 flex-wrap items-center gap-3 border-b px-5 py-4">
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
aria-label="返回助手知识库"
|
||||
onClick={() => setParams({})}
|
||||
>
|
||||
<ArrowLeft className="size-4" />
|
||||
</Button>
|
||||
<div className="rounded-xl bg-emerald-100 p-2.5 text-emerald-700 dark:bg-emerald-950">
|
||||
<BookOpenText className="size-5" />
|
||||
</div>
|
||||
<div className="min-w-0">
|
||||
<h1 className="text-lg font-semibold">
|
||||
{base?.name || "助手知识库"}
|
||||
</h1>
|
||||
<p className="mt-1 text-xs text-muted-foreground">
|
||||
{base?.is_global
|
||||
? "全库知识梳理 · 自动同步分析成果"
|
||||
: "Wiki 知识空间"}
|
||||
</p>
|
||||
</div>
|
||||
<div className="ml-auto flex gap-2">
|
||||
<Button variant="outline" size="sm" onClick={refresh}>
|
||||
<RefreshCw className="mr-2 size-4" />
|
||||
刷新
|
||||
</Button>
|
||||
{isAdmin && (
|
||||
<Button size="icon" variant="outline" onClick={() => setCreateOpen(true)}>
|
||||
<Plus className="size-4" />
|
||||
<Button
|
||||
variant="outline"
|
||||
size="sm"
|
||||
disabled={actions.uploadWeKnoraPackage.isPending}
|
||||
onClick={() => upload.current?.click()}
|
||||
>
|
||||
<Upload className="mr-2 size-4" />
|
||||
导入数据包
|
||||
</Button>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
<ScrollArea className="min-h-0 flex-1">
|
||||
<div className="space-y-2 p-3">
|
||||
{(bases.data ?? []).map((base) => (
|
||||
<button
|
||||
key={base.id}
|
||||
type="button"
|
||||
onClick={() => setSelectedBaseId(base.id)}
|
||||
className={`w-full rounded-lg border px-3 py-3 text-left transition ${
|
||||
selectedBaseId === base.id ? "border-primary bg-primary/10 text-primary" : "bg-card hover:bg-muted"
|
||||
}`}
|
||||
>
|
||||
<div className="flex items-center gap-2">
|
||||
<Database className="size-4 shrink-0" />
|
||||
<span className="min-w-0 flex-1 truncate text-sm font-medium">{base.name}</span>
|
||||
{base.is_global && <Badge variant="outline" className="text-[10px]">总库</Badge>}
|
||||
</div>
|
||||
<div className="mt-2 flex gap-3 text-[11px] text-muted-foreground">
|
||||
<span>{base.page_count ?? 0} 页</span>
|
||||
<span>{base.source_count ?? 0} 来源</span>
|
||||
<span>{base.entity_count ?? 0} 实体</span>
|
||||
</div>
|
||||
</button>
|
||||
))}
|
||||
<input
|
||||
ref={upload}
|
||||
type="file"
|
||||
accept=".jsonl,.ndjson"
|
||||
className="hidden"
|
||||
onChange={async (e) => {
|
||||
const file = e.target.files?.[0];
|
||||
e.target.value = "";
|
||||
if (!file) return;
|
||||
try {
|
||||
await actions.uploadWeKnoraPackage.mutateAsync({
|
||||
file,
|
||||
assistantBaseId: baseId,
|
||||
});
|
||||
toast.success("文件已上传，后台正在导入");
|
||||
} catch (err) {
|
||||
toast.error((err as Error).message);
|
||||
}
|
||||
}}
|
||||
/>
|
||||
</header>
|
||||
<Tabs
|
||||
value={tab}
|
||||
onValueChange={setTab}
|
||||
className="flex min-h-0 flex-1 flex-col gap-0"
|
||||
>
|
||||
<div className="shrink-0 border-b px-5 py-2">
|
||||
<TabsList>
|
||||
<TabsTrigger value="wiki">
|
||||
Wiki（{base?.page_count || 0}）
|
||||
</TabsTrigger>
|
||||
<TabsTrigger value="search">向量检索</TabsTrigger>
|
||||
<TabsTrigger value="sources">来源与同步</TabsTrigger>
|
||||
{isAdmin && (
|
||||
<TabsTrigger value="transfer">导入 / 导出记录</TabsTrigger>
|
||||
)}
|
||||
</TabsList>
|
||||
</div>
|
||||
</ScrollArea>
|
||||
</aside>
|
||||
|
||||
<main className="flex min-h-0 flex-col">
|
||||
<header className="flex items-center gap-4 border-b px-6 py-4">
|
||||
<div className="min-w-0">
|
||||
<h1 className="truncate text-xl font-semibold">{selectedBase?.name}</h1>
|
||||
<p className="mt-1 truncate text-xs text-muted-foreground">
|
||||
{selectedBase ? baseSubtitle(selectedBase) : "请选择助手知识库"}
|
||||
</p>
|
||||
</div>
|
||||
<Badge variant="outline">
|
||||
<ShieldCheck className="mr-1 size-3" />
|
||||
实体对齐
|
||||
</Badge>
|
||||
{isAdmin && (
|
||||
<div className="ml-auto flex items-center gap-2">
|
||||
<input
|
||||
ref={packageInputRef}
|
||||
type="file"
|
||||
accept=".jsonl,.ndjson,application/x-ndjson,application/json"
|
||||
className="hidden"
|
||||
onChange={(event) => void uploadPackage(event.target.files?.[0])}
|
||||
<TabsContent value="wiki" className="mt-0 flex min-h-0 flex-1">
|
||||
<WikiLibraryReader
|
||||
pages={libraryPages}
|
||||
activePage={active}
|
||||
onNavigate={navigatePage}
|
||||
pageHref={(s) =>
|
||||
buildAppHashUrl(
|
||||
`/page/workspace/knowledge/assistant?${new URLSearchParams({ base_id: baseId, wikiSlug: s })}`,
|
||||
)
|
||||
}
|
||||
mappingId={
|
||||
active?.origins?.find((o) => o.mapping_id)?.mapping_id
|
||||
}
|
||||
/>
|
||||
</TabsContent>
|
||||
<TabsContent value="search" className="overflow-auto p-6">
|
||||
<div className="mb-5 flex max-w-3xl gap-2">
|
||||
<Input
|
||||
placeholder="用自然语言检索知识，例如：如何处理设备过热？"
|
||||
value={query}
|
||||
onChange={(e) => setQuery(e.target.value)}
|
||||
onKeyDown={(e) => {
|
||||
if (e.key === "Enter") void search();
|
||||
}}
|
||||
/>
|
||||
<Button
|
||||
variant="outline"
|
||||
disabled={actions.uploadWeKnoraPackage.isPending || !selectedBaseId}
|
||||
onClick={() => packageInputRef.current?.click()}
|
||||
>
|
||||
{actions.uploadWeKnoraPackage.isPending ? (
|
||||
<Loader2 className="mr-2 size-4 animate-spin" />
|
||||
<Button disabled={searching} onClick={() => void search()}>
|
||||
{searching ? (
|
||||
<Loader2 className="size-4 animate-spin" />
|
||||
) : (
|
||||
<Upload className="mr-2 size-4" />
|
||||
<Search className="size-4" />
|
||||
)}
|
||||
上传 WeKnora 导出包
|
||||
检索
|
||||
</Button>
|
||||
</div>
|
||||
)}
|
||||
</header>
|
||||
|
||||
<Tabs defaultValue="wiki" className="flex min-h-0 flex-1 flex-col">
|
||||
<div className="flex items-center gap-4 border-b px-6">
|
||||
<TabsList className="my-2">
|
||||
<TabsTrigger value="wiki">Wiki（{counts.pages}）</TabsTrigger>
|
||||
<TabsTrigger value="sources">来源（{counts.sources}）</TabsTrigger>
|
||||
<TabsTrigger value="jobs">导入记录（{counts.jobs}）</TabsTrigger>
|
||||
</TabsList>
|
||||
<div className="relative ml-auto w-72">
|
||||
<Search className="absolute left-2.5 top-1/2 size-4 -translate-y-1/2 text-muted-foreground" />
|
||||
<Input
|
||||
value={query}
|
||||
onChange={(event) => setQuery(event.target.value)}
|
||||
placeholder="搜索标题、实体别名和正文"
|
||||
className="pl-8"
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<TabsContent value="wiki" className="mt-0 grid min-h-0 flex-1 grid-cols-[300px_1fr]">
|
||||
<ScrollArea className="border-r">
|
||||
<div className="space-y-1 p-3">
|
||||
{pages.isLoading && <Loader2 className="m-4 size-5 animate-spin" />}
|
||||
{pages.data?.map((page) => (
|
||||
<button
|
||||
key={page.id}
|
||||
type="button"
|
||||
onClick={() => setSelected(page)}
|
||||
className={`w-full rounded-md px-3 py-2 text-left ${
|
||||
activePage?.id === page.id ? "bg-primary/10 text-primary" : "hover:bg-muted"
|
||||
}`}
|
||||
>
|
||||
<div className="truncate text-sm font-medium">{page.title}</div>
|
||||
<div className="mt-1 flex items-center gap-2 text-[11px] text-muted-foreground">
|
||||
<span>{page.page_type}</span>
|
||||
<span>{new Date(page.updated_at).toLocaleString()}</span>
|
||||
</div>
|
||||
</button>
|
||||
))}
|
||||
{!pages.isLoading && !pages.data?.length && (
|
||||
<div className="p-4 text-xs text-muted-foreground">
|
||||
暂无 Wiki 页面。请先到普通知识库详情页下载向量化数据包，再在当前助手知识库上传。
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</ScrollArea>
|
||||
<ScrollArea>
|
||||
<article className="mx-auto max-w-5xl p-8">
|
||||
{activePage ? (
|
||||
<>
|
||||
<h1 className="mb-1 flex items-center gap-2 text-2xl font-semibold">
|
||||
<BookOpen className="size-5 text-primary" />
|
||||
{activePage.title}
|
||||
</h1>
|
||||
<p className="mb-6 text-xs text-muted-foreground">{activePage.slug}</p>
|
||||
<MarkdownContent content={activePage.content_markdown} isLoading={false} rehypePlugins={[]} />
|
||||
</>
|
||||
) : (
|
||||
<p className="text-sm text-muted-foreground">暂无 Wiki 页面。</p>
|
||||
)}
|
||||
</article>
|
||||
</ScrollArea>
|
||||
{searchResult && (
|
||||
<>
|
||||
<Badge variant="outline">
|
||||
{searchResult.retrieval_mode === "vector"
|
||||
? "向量检索 · 余弦相似度"
|
||||
: searchResult.retrieval_mode === "empty"
|
||||
? "没有匹配结果"
|
||||
: "关键词检索（暂无匹配模型的向量）"}
|
||||
</Badge>
|
||||
<div className="mt-4 space-y-3">
|
||||
{searchResult.results.map((p) => (
|
||||
<button
|
||||
key={p.id}
|
||||
onClick={() => navigatePage(p.slug)}
|
||||
className="block w-full max-w-4xl rounded-xl border p-5 text-left hover:bg-muted"
|
||||
>
|
||||
<div className="font-medium">
|
||||
{p.title}
|
||||
<span className="float-right text-xs text-muted-foreground">
|
||||
{p.score?.toFixed(4)}
|
||||
</span>
|
||||
</div>
|
||||
<p className="mt-2 line-clamp-3 text-sm text-muted-foreground">
|
||||
{p.summary || p.content_markdown}
|
||||
</p>
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
</>
|
||||
)}
|
||||
</TabsContent>
|
||||
|
||||
<TabsContent value="sources" className="mt-0 min-h-0 flex-1 overflow-auto p-6">
|
||||
<div className="grid gap-3 md:grid-cols-2 xl:grid-cols-3">
|
||||
{sources.data?.map((source) => (
|
||||
<div key={source.id} className="rounded-lg border bg-card p-4">
|
||||
<div className="flex items-center gap-2">
|
||||
<Database className="size-4" />
|
||||
<span className="truncate font-medium">{source.source_name}</span>
|
||||
<Badge variant="outline" className="ml-auto">{source.source_type}</Badge>
|
||||
</div>
|
||||
<p className="mt-3 text-xs text-muted-foreground">业务键：{source.source_key}</p>
|
||||
<TabsContent value="sources" className="overflow-auto p-6">
|
||||
<div className="grid gap-4 lg:grid-cols-2">
|
||||
{sources.data?.map((s) => (
|
||||
<div className="rounded-xl border p-5" key={s.id}>
|
||||
<h3 className="font-semibold">{s.source_name}</h3>
|
||||
<p className="mt-2 text-xs text-muted-foreground">
|
||||
上次同步：{new Date(s.updated_at).toLocaleString()}
|
||||
</p>
|
||||
{isAdmin && (
|
||||
<div className="mt-4 flex justify-end gap-2">
|
||||
<Button
|
||||
size="sm"
|
||||
variant="outline"
|
||||
disabled={source.source_type !== "weknora" || actions.reimportSource.isPending}
|
||||
onClick={() =>
|
||||
actions.reimportSource.mutate(source.id, {
|
||||
onSuccess: () => toast.success("已发起重新导入"),
|
||||
onError: (error) => toast.error(error.message),
|
||||
})
|
||||
}
|
||||
>
|
||||
<RefreshCw className="mr-2 size-4" />
|
||||
重新导入
|
||||
</Button>
|
||||
<Button
|
||||
size="sm"
|
||||
variant="destructive"
|
||||
disabled={actions.deleteSource.isPending}
|
||||
onClick={() => {
|
||||
if (window.confirm(`确认删除来源“${source.source_name}”的贡献？仍被其他来源引用的知识会保留。`)) {
|
||||
actions.deleteSource.mutate(source.id, {
|
||||
onSuccess: () => toast.success("来源贡献已删除"),
|
||||
onError: (error) => toast.error(error.message),
|
||||
});
|
||||
}
|
||||
}}
|
||||
>
|
||||
<Trash2 className="mr-2 size-4" />
|
||||
删除来源
|
||||
</Button>
|
||||
</div>
|
||||
<Button
|
||||
className="mt-4"
|
||||
size="sm"
|
||||
variant="outline"
|
||||
disabled={actions.reimportSource.isPending}
|
||||
onClick={() =>
|
||||
actions.reimportSource.mutate(s.id, {
|
||||
onSuccess: () => toast.success("已提交同步任务"),
|
||||
onError: (e) => toast.error(e.message),
|
||||
})
|
||||
}
|
||||
>
|
||||
<RefreshCw className="mr-2 size-4" />
|
||||
同步最新数据
|
||||
</Button>
|
||||
)}
|
||||
</div>
|
||||
))}
|
||||
{!sources.data?.length && <p className="text-sm text-muted-foreground">暂无来源。</p>}
|
||||
</div>
|
||||
</TabsContent>
|
||||
|
||||
<TabsContent value="jobs" className="mt-0 min-h-0 flex-1 overflow-auto p-6">
|
||||
<div className="space-y-3">
|
||||
{jobs.data?.map((job) => (
|
||||
<div key={job.id} className="rounded-lg border bg-card p-4">
|
||||
{isAdmin && (
|
||||
<TabsContent
|
||||
value="transfer"
|
||||
className="space-y-5 overflow-auto p-6"
|
||||
>
|
||||
<WikiPackageTransfers id={baseId} assistant />
|
||||
<h2 className="text-base font-semibold">导入与同步记录</h2>
|
||||
{jobs.data?.map((j) => (
|
||||
<div key={j.id} className="rounded-xl border p-4">
|
||||
<div className="flex items-center gap-3">
|
||||
<Badge>{job.status}</Badge>
|
||||
<span className="text-sm">{job.trigger}</span>
|
||||
<span className="text-xs text-muted-foreground">阶段：{job.phase}</span>
|
||||
<span className="ml-auto text-xs">{new Date(job.created_at).toLocaleString()}</span>
|
||||
{isAdmin && job.package_download_url && (
|
||||
<Button asChild size="sm" variant="outline">
|
||||
<a href={job.package_download_url} target="_blank" rel="noreferrer">
|
||||
<Download className="mr-2 size-4" />
|
||||
下载导出包
|
||||
</a>
|
||||
</Button>
|
||||
)}
|
||||
<Badge variant="outline">
|
||||
{{
|
||||
completed: "已完成",
|
||||
failed: "失败",
|
||||
running: "处理中",
|
||||
queued: "排队中",
|
||||
}[j.status] || j.status}
|
||||
</Badge>
|
||||
<span className="text-xs text-muted-foreground">
|
||||
{new Date(j.created_at).toLocaleString()}
|
||||
</span>
|
||||
</div>
|
||||
<div className="mt-2 text-xs text-muted-foreground">
|
||||
{Object.entries(job.counts || {}).map(([key, value]) => `${key} ${value}`).join(" · ") || "暂无统计"}
|
||||
</div>
|
||||
{job.last_error_message && (
|
||||
<div className="mt-2 rounded-md bg-destructive/10 px-3 py-2 text-xs text-destructive">
|
||||
{job.last_error_message}
|
||||
</div>
|
||||
<p className="mt-3 text-sm">
|
||||
{j.counts?.wiki_pages || 0} Wiki ·{" "}
|
||||
{j.counts?.vectors || 0} 向量 · {j.counts?.entities || 0}{" "}
|
||||
实体 · {j.counts?.relations || 0} 关系
|
||||
</p>
|
||||
{j.last_error_message && (
|
||||
<p className="mt-2 text-sm text-destructive">
|
||||
{j.last_error_message}
|
||||
</p>
|
||||
)}
|
||||
</div>
|
||||
))}
|
||||
{!jobs.data?.length && <p className="text-sm text-muted-foreground">暂无导入记录。</p>}
|
||||
</div>
|
||||
</TabsContent>
|
||||
</TabsContent>
|
||||
)}
|
||||
</Tabs>
|
||||
</main>
|
||||
</div>
|
||||
|
||||
</>
|
||||
)}
|
||||
<Dialog open={createOpen} onOpenChange={setCreateOpen}>
|
||||
<DialogContent>
|
||||
<DialogHeader>
|
||||
<DialogTitle>新建助手知识库</DialogTitle>
|
||||
<DialogDescription>
|
||||
自建助手知识库用于承接从 WeKnora 普通知识库下载的 Wiki 向量包；导入时同一份数据也会沉淀到“知识梳理总库”。
|
||||
</DialogDescription>
|
||||
</DialogHeader>
|
||||
<div className="space-y-4">
|
||||
<div className="space-y-1.5">
|
||||
<label className="text-sm font-medium">名称</label>
|
||||
<Input
|
||||
value={createDraft.name}
|
||||
onChange={(event) => setCreateDraft((prev) => ({ ...prev, name: event.target.value }))}
|
||||
placeholder="例如：政策资料库"
|
||||
/>
|
||||
</div>
|
||||
<div className="space-y-1.5">
|
||||
<label className="text-sm font-medium">描述</label>
|
||||
<Textarea
|
||||
value={createDraft.description}
|
||||
onChange={(event) => setCreateDraft((prev) => ({ ...prev, description: event.target.value }))}
|
||||
placeholder="可选，用于说明这个助手知识库的范围"
|
||||
rows={3}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
<Input
|
||||
placeholder="知识库名称"
|
||||
value={name}
|
||||
onChange={(e) => setName(e.target.value)}
|
||||
/>
|
||||
<DialogFooter>
|
||||
<Button variant="outline" onClick={() => setCreateOpen(false)}>
|
||||
取消
|
||||
</Button>
|
||||
<Button disabled={actions.createBase.isPending} onClick={() => void createBase()}>
|
||||
{actions.createBase.isPending ? <Loader2 className="mr-2 size-4 animate-spin" /> : <Plus className="mr-2 size-4" />}
|
||||
<Button
|
||||
disabled={!name.trim() || actions.createBase.isPending}
|
||||
onClick={() => void create()}
|
||||
>
|
||||
创建
|
||||
</Button>
|
||||
</DialogFooter>
|
||||
|
||||
@@ -30,6 +30,15 @@ export interface AssistantWikiPage {
|
||||
summary: string;
|
||||
current_revision_id?: string | null;
|
||||
updated_at: string;
|
||||
aliases?: string[];
|
||||
tags?: string[];
|
||||
category_path?: string[];
|
||||
parent_slug?: string;
|
||||
wiki_path?: string;
|
||||
in_links?: string[];
|
||||
out_links?: string[];
|
||||
source_refs?: string[];
|
||||
origins?: { mapping_id?: string; source_name: string; slug: string }[];
|
||||
}
|
||||
|
||||
export interface AssistantWikiRevision {
|
||||
@@ -88,10 +97,13 @@ export interface LlmWikiExportPackageJob {
|
||||
async function readJson<T>(response: Response): Promise<T> {
|
||||
if (!response.ok) {
|
||||
const data = await response.json().catch(() => null);
|
||||
const detail = data && typeof data === "object" && "detail" in data
|
||||
? (data as { detail: unknown }).detail
|
||||
: `请求失败 (${response.status})`;
|
||||
throw new Error(typeof detail === "string" ? detail : JSON.stringify(detail));
|
||||
const detail =
|
||||
data && typeof data === "object" && "detail" in data
|
||||
? (data as { detail: unknown }).detail
|
||||
: `请求失败 (${response.status})`;
|
||||
throw new Error(
|
||||
typeof detail === "string" ? detail : JSON.stringify(detail),
|
||||
);
|
||||
}
|
||||
return (await response.json()) as T;
|
||||
}
|
||||
@@ -101,12 +113,16 @@ function normalizeImportJob(job: AssistantImportJob): AssistantImportJob {
|
||||
return url ? { ...job, package_download_url: url } : job;
|
||||
}
|
||||
|
||||
function normalizeBackendUrl(url: string | undefined | null): string | undefined {
|
||||
function normalizeBackendUrl(
|
||||
url: string | undefined | null,
|
||||
): string | undefined {
|
||||
if (!url || /^https?:\/\//i.test(url)) return url || undefined;
|
||||
return `${getBackendBaseURL()}${url.startsWith("/") ? url : `/${url}`}`;
|
||||
}
|
||||
|
||||
function normalizeExportPackageJob(job: LlmWikiExportPackageJob): LlmWikiExportPackageJob {
|
||||
function normalizeExportPackageJob(
|
||||
job: LlmWikiExportPackageJob,
|
||||
): LlmWikiExportPackageJob {
|
||||
return {
|
||||
...job,
|
||||
package_download_url: normalizeBackendUrl(job.package_download_url),
|
||||
@@ -115,7 +131,9 @@ function normalizeExportPackageJob(job: LlmWikiExportPackageJob): LlmWikiExportP
|
||||
}
|
||||
|
||||
export async function getAssistantKnowledgeBase() {
|
||||
const data = await readJson<{ base: AssistantKnowledgeBase | null }>(await apiFetch(`${BASE()}/base`));
|
||||
const data = await readJson<{ base: AssistantKnowledgeBase | null }>(
|
||||
await apiFetch(`${BASE()}/base`),
|
||||
);
|
||||
return data.base;
|
||||
}
|
||||
|
||||
@@ -127,7 +145,11 @@ export async function getAssistantKnowledgeBaseById(baseId: string) {
|
||||
}
|
||||
|
||||
export async function listAssistantKnowledgeBases() {
|
||||
return (await readJson<{ bases: AssistantKnowledgeBase[] }>(await apiFetch(`${BASE()}/bases`))).bases;
|
||||
return (
|
||||
await readJson<{ bases: AssistantKnowledgeBase[] }>(
|
||||
await apiFetch(`${BASE()}/bases`),
|
||||
)
|
||||
).bases;
|
||||
}
|
||||
|
||||
export async function initializeAssistantKnowledgeBase(name = "知识梳理总库") {
|
||||
@@ -141,7 +163,10 @@ export async function initializeAssistantKnowledgeBase(name = "知识梳理总
|
||||
return data.base;
|
||||
}
|
||||
|
||||
export async function createAssistantKnowledgeBase(input: { name: string; description?: string }) {
|
||||
export async function createAssistantKnowledgeBase(input: {
|
||||
name: string;
|
||||
description?: string;
|
||||
}) {
|
||||
const data = await readJson<{ base: AssistantKnowledgeBase }>(
|
||||
await apiFetch(`${BASE()}/bases`, {
|
||||
method: "POST",
|
||||
@@ -160,30 +185,96 @@ function withBaseParam(baseId?: string | null) {
|
||||
|
||||
export async function listAssistantWikiPages(q = "", baseId?: string | null) {
|
||||
const params = withBaseParam(baseId);
|
||||
params.set("limit", "500");
|
||||
if (q.trim()) params.set("q", q.trim());
|
||||
return (await readJson<{ pages: AssistantWikiPage[] }>(await apiFetch(`${BASE()}/wiki-pages?${params}`))).pages;
|
||||
const pages: AssistantWikiPage[] = [];
|
||||
let offset: number | null = 0;
|
||||
do {
|
||||
params.set("offset", String(offset));
|
||||
const batch: { pages: AssistantWikiPage[]; next_offset?: number | null } =
|
||||
await readJson(await apiFetch(`${BASE()}/wiki-pages?${params}`));
|
||||
pages.push(...batch.pages);
|
||||
offset = batch.next_offset ?? null;
|
||||
} while (offset !== null);
|
||||
return pages;
|
||||
}
|
||||
|
||||
export async function searchAssistantWiki(q: string, baseId: string) {
|
||||
return readJson<{
|
||||
results: (AssistantWikiPage & { score: number; retrieval_mode: string })[];
|
||||
retrieval_mode: string;
|
||||
}>(
|
||||
await apiFetch(
|
||||
`${BASE()}/search?${new URLSearchParams({ q, base_id: baseId })}`,
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
export async function listWikiTransfers(id: string, assistant: boolean) {
|
||||
const url = assistant
|
||||
? `${BASE()}/bases/${encodeURIComponent(id)}/export-package`
|
||||
: `${getBackendBaseURL()}/api/llmwiki/knowledge-bases/${encodeURIComponent(id)}/import-package`;
|
||||
return (
|
||||
await readJson<{ jobs: LlmWikiExportPackageJob[] }>(await apiFetch(url))
|
||||
).jobs.map(normalizeExportPackageJob);
|
||||
}
|
||||
|
||||
export async function createAssistantExportPackage(id: string) {
|
||||
return readJson<LlmWikiExportPackageJob>(
|
||||
await apiFetch(`${BASE()}/bases/${encodeURIComponent(id)}/export-package`, {
|
||||
method: "POST",
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
export async function importOrdinaryWikiPackage(id: string, file: File) {
|
||||
const body = new FormData();
|
||||
body.set("file", file);
|
||||
return readJson<LlmWikiExportPackageJob>(
|
||||
await apiFetch(
|
||||
`${getBackendBaseURL()}/api/llmwiki/knowledge-bases/${encodeURIComponent(id)}/import-package`,
|
||||
{ method: "POST", body },
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
export async function listAssistantKnowledgeSources(baseId?: string | null) {
|
||||
const params = withBaseParam(baseId);
|
||||
return (await readJson<{ sources: AssistantKnowledgeSource[] }>(await apiFetch(`${BASE()}/sources?${params}`))).sources;
|
||||
return (
|
||||
await readJson<{ sources: AssistantKnowledgeSource[] }>(
|
||||
await apiFetch(`${BASE()}/sources?${params}`),
|
||||
)
|
||||
).sources;
|
||||
}
|
||||
|
||||
export async function listAssistantImportJobs(baseId?: string | null) {
|
||||
const params = withBaseParam(baseId);
|
||||
return (
|
||||
await readJson<{ jobs: AssistantImportJob[] }>(await apiFetch(`${BASE()}/import-jobs?${params}`))
|
||||
await readJson<{ jobs: AssistantImportJob[] }>(
|
||||
await apiFetch(`${BASE()}/import-jobs?${params}`),
|
||||
)
|
||||
).jobs.map(normalizeImportJob);
|
||||
}
|
||||
|
||||
export async function listAssistantWikiRevisions(slug: string, baseId?: string | null) {
|
||||
export async function listAssistantWikiRevisions(
|
||||
slug: string,
|
||||
baseId?: string | null,
|
||||
) {
|
||||
const params = withBaseParam(baseId);
|
||||
return (await readJson<{ revisions: AssistantWikiRevision[] }>(
|
||||
await apiFetch(`${BASE()}/wiki-pages/${encodeURIComponent(slug)}/revisions?${params}`),
|
||||
)).revisions;
|
||||
return (
|
||||
await readJson<{ revisions: AssistantWikiRevision[] }>(
|
||||
await apiFetch(
|
||||
`${BASE()}/wiki-pages/${encodeURIComponent(slug)}/revisions?${params}`,
|
||||
),
|
||||
)
|
||||
).revisions;
|
||||
}
|
||||
|
||||
export async function rollbackAssistantWikiRevision(slug: string, revisionId: string, baseId?: string | null) {
|
||||
export async function rollbackAssistantWikiRevision(
|
||||
slug: string,
|
||||
revisionId: string,
|
||||
baseId?: string | null,
|
||||
) {
|
||||
const params = withBaseParam(baseId);
|
||||
return readJson<AssistantWikiRevision>(
|
||||
await apiFetch(
|
||||
@@ -193,17 +284,22 @@ export async function rollbackAssistantWikiRevision(slug: string, revisionId: st
|
||||
);
|
||||
}
|
||||
|
||||
export async function exportLlmWikiToAssistant(mappingId: string, assistantBaseId?: string | null) {
|
||||
return normalizeImportJob(await readJson<AssistantImportJob>(
|
||||
await apiFetch(
|
||||
`${getBackendBaseURL()}/api/llmwiki/knowledge-bases/${encodeURIComponent(mappingId)}/export-to-assistant`,
|
||||
{
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ assistant_base_id: assistantBaseId || null }),
|
||||
},
|
||||
export async function exportLlmWikiToAssistant(
|
||||
mappingId: string,
|
||||
assistantBaseId?: string | null,
|
||||
) {
|
||||
return normalizeImportJob(
|
||||
await readJson<AssistantImportJob>(
|
||||
await apiFetch(
|
||||
`${getBackendBaseURL()}/api/llmwiki/knowledge-bases/${encodeURIComponent(mappingId)}/export-to-assistant`,
|
||||
{
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ assistant_base_id: assistantBaseId || null }),
|
||||
},
|
||||
),
|
||||
),
|
||||
));
|
||||
);
|
||||
}
|
||||
|
||||
export async function createLlmWikiExportPackage(mappingId: string) {
|
||||
@@ -227,7 +323,10 @@ export async function listLlmWikiExportPackages(mappingId: string) {
|
||||
).jobs.map(normalizeExportPackageJob);
|
||||
}
|
||||
|
||||
export async function getLlmWikiExportPackageStatus(mappingId: string, jobId: string) {
|
||||
export async function getLlmWikiExportPackageStatus(
|
||||
mappingId: string,
|
||||
jobId: string,
|
||||
) {
|
||||
return normalizeExportPackageJob(
|
||||
await readJson<LlmWikiExportPackageJob>(
|
||||
await apiFetch(
|
||||
@@ -238,33 +337,47 @@ export async function getLlmWikiExportPackageStatus(mappingId: string, jobId: st
|
||||
}
|
||||
|
||||
export async function reimportAssistantKnowledgeSource(sourceId: string) {
|
||||
return normalizeImportJob(await readJson<AssistantImportJob>(
|
||||
await apiFetch(`${BASE()}/sources/${encodeURIComponent(sourceId)}/reimport`, {
|
||||
method: "POST",
|
||||
}),
|
||||
));
|
||||
return normalizeImportJob(
|
||||
await readJson<AssistantImportJob>(
|
||||
await apiFetch(
|
||||
`${BASE()}/sources/${encodeURIComponent(sourceId)}/reimport`,
|
||||
{
|
||||
method: "POST",
|
||||
},
|
||||
),
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
export function getAssistantImportPackageUrl(jobId: string) {
|
||||
return `${BASE()}/import-jobs/${encodeURIComponent(jobId)}/package`;
|
||||
}
|
||||
|
||||
export async function uploadAssistantWeKnoraPackage(input: { file: File; assistantBaseId?: string | null }) {
|
||||
export async function uploadAssistantWeKnoraPackage(input: {
|
||||
file: File;
|
||||
assistantBaseId?: string | null;
|
||||
}) {
|
||||
const body = new FormData();
|
||||
body.set("file", input.file);
|
||||
if (input.assistantBaseId) body.set("assistant_base_id", input.assistantBaseId);
|
||||
return normalizeImportJob(await readJson<AssistantImportJob>(
|
||||
await apiFetch(`${BASE()}/imports/weknora-package`, {
|
||||
method: "POST",
|
||||
body,
|
||||
}),
|
||||
));
|
||||
if (input.assistantBaseId)
|
||||
body.set("assistant_base_id", input.assistantBaseId);
|
||||
return normalizeImportJob(
|
||||
await readJson<AssistantImportJob>(
|
||||
await apiFetch(`${BASE()}/imports/weknora-package`, {
|
||||
method: "POST",
|
||||
body,
|
||||
}),
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
export async function deleteAssistantKnowledgeSource(sourceId: string) {
|
||||
const response = await apiFetch(`${BASE()}/sources/${encodeURIComponent(sourceId)}`, {
|
||||
method: "DELETE",
|
||||
});
|
||||
const response = await apiFetch(
|
||||
`${BASE()}/sources/${encodeURIComponent(sourceId)}`,
|
||||
{
|
||||
method: "DELETE",
|
||||
},
|
||||
);
|
||||
if (!response.ok) await readJson(response);
|
||||
}
|
||||
|
||||
@@ -273,12 +386,19 @@ const KEY = "assistant-knowledge";
|
||||
export function useAssistantKnowledgeBase(baseId?: string | null) {
|
||||
return useQuery({
|
||||
queryKey: [KEY, "base", baseId ?? "global"],
|
||||
queryFn: () => (baseId ? getAssistantKnowledgeBaseById(baseId) : getAssistantKnowledgeBase()),
|
||||
queryFn: () =>
|
||||
baseId
|
||||
? getAssistantKnowledgeBaseById(baseId)
|
||||
: getAssistantKnowledgeBase(),
|
||||
});
|
||||
}
|
||||
|
||||
export function useAssistantKnowledgeBases() {
|
||||
return useQuery({ queryKey: [KEY, "bases"], queryFn: listAssistantKnowledgeBases });
|
||||
export function useAssistantKnowledgeBases(enabled = true) {
|
||||
return useQuery({
|
||||
queryKey: [KEY, "bases"],
|
||||
queryFn: listAssistantKnowledgeBases,
|
||||
enabled,
|
||||
});
|
||||
}
|
||||
|
||||
export function useAssistantWikiPages(query = "", baseId?: string | null) {
|
||||
@@ -288,10 +408,11 @@ export function useAssistantWikiPages(query = "", baseId?: string | null) {
|
||||
});
|
||||
}
|
||||
|
||||
export function useAssistantKnowledgeSources(baseId?: string | null) {
|
||||
export function useAssistantKnowledgeSources(baseId?: string | null, enabled = true) {
|
||||
return useQuery({
|
||||
queryKey: [KEY, "sources", baseId ?? "global"],
|
||||
queryFn: () => listAssistantKnowledgeSources(baseId),
|
||||
enabled,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -300,7 +421,9 @@ export function useAssistantImportJobs(baseId?: string | null) {
|
||||
queryKey: [KEY, "jobs", baseId ?? "global"],
|
||||
queryFn: () => listAssistantImportJobs(baseId),
|
||||
refetchInterval: (query) =>
|
||||
(query.state.data ?? []).some((job) => ["queued", "running"].includes(job.status))
|
||||
(query.state.data ?? []).some((job) =>
|
||||
["queued", "running"].includes(job.status),
|
||||
)
|
||||
? 3000
|
||||
: false,
|
||||
});
|
||||
@@ -310,14 +433,36 @@ export function useAssistantKnowledgeActions() {
|
||||
const client = useQueryClient();
|
||||
const invalidate = () => client.invalidateQueries({ queryKey: [KEY] });
|
||||
return {
|
||||
initialize: useMutation({ mutationFn: initializeAssistantKnowledgeBase, onSuccess: invalidate }),
|
||||
createBase: useMutation({ mutationFn: createAssistantKnowledgeBase, onSuccess: invalidate }),
|
||||
uploadWeKnoraPackage: useMutation({ mutationFn: uploadAssistantWeKnoraPackage, onSuccess: invalidate }),
|
||||
reimportSource: useMutation({ mutationFn: reimportAssistantKnowledgeSource, onSuccess: invalidate }),
|
||||
deleteSource: useMutation({ mutationFn: deleteAssistantKnowledgeSource, onSuccess: invalidate }),
|
||||
initialize: useMutation({
|
||||
mutationFn: initializeAssistantKnowledgeBase,
|
||||
onSuccess: invalidate,
|
||||
}),
|
||||
createBase: useMutation({
|
||||
mutationFn: createAssistantKnowledgeBase,
|
||||
onSuccess: invalidate,
|
||||
}),
|
||||
uploadWeKnoraPackage: useMutation({
|
||||
mutationFn: uploadAssistantWeKnoraPackage,
|
||||
onSuccess: invalidate,
|
||||
}),
|
||||
reimportSource: useMutation({
|
||||
mutationFn: reimportAssistantKnowledgeSource,
|
||||
onSuccess: invalidate,
|
||||
}),
|
||||
deleteSource: useMutation({
|
||||
mutationFn: deleteAssistantKnowledgeSource,
|
||||
onSuccess: invalidate,
|
||||
}),
|
||||
rollback: useMutation({
|
||||
mutationFn: ({ slug, revisionId, baseId }: { slug: string; revisionId: string; baseId?: string | null }) =>
|
||||
rollbackAssistantWikiRevision(slug, revisionId, baseId),
|
||||
mutationFn: ({
|
||||
slug,
|
||||
revisionId,
|
||||
baseId,
|
||||
}: {
|
||||
slug: string;
|
||||
revisionId: string;
|
||||
baseId?: string | null;
|
||||
}) => rollbackAssistantWikiRevision(slug, revisionId, baseId),
|
||||
onSuccess: invalidate,
|
||||
}),
|
||||
};
|
||||
|
||||
@@ -167,6 +167,7 @@ export interface LlmWikiGraph {
|
||||
}
|
||||
|
||||
export interface LlmWikiWikiPage {
|
||||
page_metadata?: { tags?: string[]; [key: string]: unknown };
|
||||
id: string;
|
||||
slug: string;
|
||||
title: string;
|
||||
|
||||
+2
@@ -9,6 +9,7 @@ import { Dialog, DialogContent, DialogHeader, DialogTitle } from "@/components/u
|
||||
import { Input } from "@/components/ui/input";
|
||||
import { Textarea } from "@/components/ui/textarea";
|
||||
import { MarkdownCodeEditor } from "@/components/wiki/MarkdownCodeEditor";
|
||||
import { WikiPackageTransfers } from "./WikiPackageTransfers";
|
||||
import { getBackendBaseURL } from "@/core/config";
|
||||
import { getStoredAuth } from "@/core/auth";
|
||||
import { buildWeKnoraWikiPagePath } from "@/core/llmwiki/source-links";
|
||||
@@ -323,6 +324,7 @@ export function WeKnoraKnowledgeDetailPage() {
|
||||
|
||||
return (
|
||||
<div className="flex h-full min-h-0 flex-col bg-background">
|
||||
{canManageWiki && isAdmin && <details className="shrink-0 border-b px-4 py-2"><summary className="cursor-pointer text-sm">从助手知识库导入数据</summary><div className="py-3"><WikiPackageTransfers id={knowledgeBaseId} /></div></details>}
|
||||
{canManageWiki && (
|
||||
<div className="flex min-h-12 shrink-0 flex-wrap items-center justify-between gap-2 border-b px-4 py-2">
|
||||
<div className="min-w-0 flex-1">
|
||||
|
||||
+85
-23
@@ -3,6 +3,7 @@ import {
|
||||
BookOpenText,
|
||||
ChevronRight,
|
||||
Database,
|
||||
Edit3,
|
||||
FileText,
|
||||
Globe2,
|
||||
Loader2,
|
||||
@@ -109,6 +110,7 @@ export function WeKnoraKnowledgeWorkspace() {
|
||||
const knowledge = useLlmWikiKnowledgeBases(scope, tab !== "settings");
|
||||
const actions = useLlmWikiActions();
|
||||
const [createOpen, setCreateOpen] = useState(false);
|
||||
const [editingKnowledge, setEditingKnowledge] = useState<LlmWikiKnowledgeBase | null>(null);
|
||||
const [searchText, setSearchText] = useState("");
|
||||
const [searching, setSearching] = useState(false);
|
||||
const [searchResponse, setSearchResponse] = useState<LlmWikiSearchResponse | null>(null);
|
||||
@@ -119,12 +121,8 @@ export function WeKnoraKnowledgeWorkspace() {
|
||||
setSelectedSearchResult(null);
|
||||
}, [tab]);
|
||||
|
||||
const busy = actions.create.isPending || actions.remove.isPending || actions.publication.isPending;
|
||||
const busy = actions.create.isPending || actions.update.isPending || actions.remove.isPending || actions.publication.isPending;
|
||||
const openDetails = (item: LlmWikiKnowledgeBase) => {
|
||||
if (item.remote_status === "missing") {
|
||||
toast.error("该知识库已从知识服务删除，只能清理当前记录");
|
||||
return;
|
||||
}
|
||||
navigate(`${LIST_PATH}/${encodeURIComponent(item.id)}`);
|
||||
};
|
||||
|
||||
@@ -142,10 +140,7 @@ export function WeKnoraKnowledgeWorkspace() {
|
||||
};
|
||||
|
||||
const removeKnowledgeBase = async (item: LlmWikiKnowledgeBase) => {
|
||||
const warning = item.remote_status === "missing"
|
||||
? `知识库“${item.name}”已不在知识服务中。确定清理 cmzs 中的这条失效记录吗？`
|
||||
: `确定删除知识库“${item.name}”吗？知识服务中的文档和索引也会删除。`;
|
||||
if (!window.confirm(warning)) return;
|
||||
if (!window.confirm(`确定删除知识库“${item.name}”吗？知识服务中的文档和索引也会删除。`)) return;
|
||||
try {
|
||||
await actions.remove.mutateAsync(item.id);
|
||||
toast.success("知识库已删除");
|
||||
@@ -207,8 +202,8 @@ export function WeKnoraKnowledgeWorkspace() {
|
||||
)}
|
||||
</div>
|
||||
|
||||
<TabsContent value="personal" className="mt-5"><KnowledgeGrid rows={knowledge.data ?? []} loading={knowledge.isLoading} error={knowledge.error} emptyText="还没有个人知识库，点击右上角创建。" busy={busy} onOpen={openDetails} onPublish={togglePublication} onDelete={removeKnowledgeBase} /></TabsContent>
|
||||
<TabsContent value="public" className="mt-5"><KnowledgeGrid rows={knowledge.data ?? []} loading={knowledge.isLoading} error={knowledge.error} emptyText="公共 LLMWiki 暂无已发布知识库。" busy={busy} onOpen={openDetails} onPublish={togglePublication} onDelete={removeKnowledgeBase} /></TabsContent>
|
||||
<TabsContent value="personal" className="mt-5"><KnowledgeGrid rows={knowledge.data ?? []} loading={knowledge.isLoading} error={knowledge.error} emptyText="还没有个人知识库，点击右上角创建。" busy={busy} onOpen={openDetails} onEdit={setEditingKnowledge} onPublish={togglePublication} onDelete={removeKnowledgeBase} /></TabsContent>
|
||||
<TabsContent value="public" className="mt-5"><KnowledgeGrid rows={knowledge.data ?? []} loading={knowledge.isLoading} error={knowledge.error} emptyText="公共 LLMWiki 暂无已发布知识库。" busy={busy} onOpen={openDetails} onEdit={setEditingKnowledge} onPublish={togglePublication} onDelete={removeKnowledgeBase} /></TabsContent>
|
||||
{isAdmin && <TabsContent value="settings" className="mt-5"><ConnectionSettings /></TabsContent>}
|
||||
</Tabs>
|
||||
|
||||
@@ -301,6 +296,24 @@ export function WeKnoraKnowledgeWorkspace() {
|
||||
}}
|
||||
/>
|
||||
|
||||
{editingKnowledge && (
|
||||
<EditKnowledgeDialog
|
||||
key={editingKnowledge.id}
|
||||
knowledgeBase={editingKnowledge}
|
||||
pending={actions.update.isPending}
|
||||
onClose={() => setEditingKnowledge(null)}
|
||||
onSave={async (input) => {
|
||||
try {
|
||||
await actions.update.mutateAsync({ id: editingKnowledge.id, ...input });
|
||||
setEditingKnowledge(null);
|
||||
toast.success("知识库信息已保存");
|
||||
} catch (error) {
|
||||
toast.error(messageOf(error));
|
||||
}
|
||||
}}
|
||||
/>
|
||||
)}
|
||||
|
||||
<CreateKnowledgeDialog open={createOpen} pending={actions.create.isPending} onOpenChange={setCreateOpen} onCreate={async (input) => {
|
||||
try {
|
||||
const created = await actions.create.mutateAsync(input);
|
||||
@@ -322,6 +335,7 @@ function KnowledgeGrid({
|
||||
emptyText,
|
||||
busy,
|
||||
onOpen,
|
||||
onEdit,
|
||||
onPublish,
|
||||
onDelete,
|
||||
}: {
|
||||
@@ -331,29 +345,27 @@ function KnowledgeGrid({
|
||||
emptyText: string;
|
||||
busy: boolean;
|
||||
onOpen: (item: LlmWikiKnowledgeBase) => void;
|
||||
onEdit: (item: LlmWikiKnowledgeBase) => void;
|
||||
onPublish: (item: LlmWikiKnowledgeBase) => void;
|
||||
onDelete: (item: LlmWikiKnowledgeBase) => void;
|
||||
}) {
|
||||
if (loading) return <div className="flex h-64 items-center justify-center text-muted-foreground"><Loader2 className="mr-2 size-5 animate-spin" />正在读取知识库</div>;
|
||||
if (error) return <div className="rounded-lg border border-destructive/30 bg-destructive/5 p-6 text-sm text-destructive">{messageOf(error)}</div>;
|
||||
if (!rows.length) return <div className="flex h-64 items-center justify-center rounded-xl border border-dashed text-sm text-muted-foreground">{emptyText}</div>;
|
||||
const visibleRows = rows.filter((item) => item.remote_status !== "missing");
|
||||
if (!visibleRows.length) return <div className="flex h-64 items-center justify-center rounded-xl border border-dashed text-sm text-muted-foreground">{emptyText}</div>;
|
||||
return (
|
||||
<div className="grid gap-4 md:grid-cols-2 xl:grid-cols-3">
|
||||
{rows.map((item) => (
|
||||
{visibleRows.map((item) => (
|
||||
<Card
|
||||
key={item.id}
|
||||
className={`group overflow-hidden rounded-2xl transition-all hover:-translate-y-0.5 hover:shadow-lg ${
|
||||
item.remote_status === "missing"
|
||||
? "border-amber-500/40 bg-amber-500/[0.03]"
|
||||
: "border-border/70 bg-card/95 hover:border-primary/30"
|
||||
}`}
|
||||
className="group overflow-hidden rounded-2xl border-border/70 bg-card/95 transition-all hover:-translate-y-0.5 hover:border-primary/30 hover:shadow-lg"
|
||||
>
|
||||
<div className={`h-1 ${item.publication_status === "published" ? "bg-primary" : "bg-muted"}`} />
|
||||
<button type="button" disabled={item.remote_status === "missing"} className="w-full text-left outline-none transition-colors enabled:hover:bg-muted/25 focus-visible:ring-2 focus-visible:ring-primary focus-visible:ring-offset-2 disabled:cursor-default" onClick={() => onOpen(item)}>
|
||||
<button type="button" className="w-full text-left outline-none transition-colors hover:bg-muted/25 focus-visible:ring-2 focus-visible:ring-primary focus-visible:ring-offset-2" onClick={() => onOpen(item)}>
|
||||
<CardHeader className="pb-3 pt-5">
|
||||
<div className="flex items-start justify-between gap-3">
|
||||
<div className="min-w-0"><CardTitle className="truncate text-base">{item.name}</CardTitle><CardDescription className="mt-1 line-clamp-2 min-h-10">{item.description || "暂无描述"}</CardDescription></div>
|
||||
<div className="flex shrink-0 flex-col items-end gap-1.5">{item.remote_status === "missing" ? <Badge variant="outline" className="border-amber-500/50 text-amber-700 dark:text-amber-300">远端已删除</Badge> : <Badge variant={item.publication_status === "published" ? "default" : "secondary"}>{publicationLabels[item.publication_status] ?? "仅自己"}</Badge>}{item.remote_status !== "missing" && item.indexing_strategy?.wiki_enabled && <Badge className="bg-violet-100 text-violet-700 hover:bg-violet-100 dark:bg-violet-950 dark:text-violet-300">Wiki</Badge>}</div>
|
||||
<div className="flex shrink-0 flex-col items-end gap-1.5"><Badge variant={item.publication_status === "published" ? "default" : "secondary"}>{publicationLabels[item.publication_status] ?? "仅自己"}</Badge>{item.indexing_strategy?.wiki_enabled && <Badge className="bg-violet-100 text-violet-700 hover:bg-violet-100 dark:bg-violet-950 dark:text-violet-300">Wiki</Badge>}</div>
|
||||
</div>
|
||||
</CardHeader>
|
||||
<div className="px-6 pb-4"><div className="grid grid-cols-3 gap-2 rounded-xl border bg-muted/35 p-3 text-center text-xs"><div><div className="text-base font-semibold text-foreground">{item.knowledge_count}</div><span className="text-muted-foreground">文档</span></div><div><div className="text-base font-semibold text-foreground">{item.chunk_count}</div><span className="text-muted-foreground">分块</span></div><div><div className="text-base font-semibold text-foreground">{item.processing_count}</div><span className="text-muted-foreground">处理中</span></div></div></div>
|
||||
@@ -361,10 +373,11 @@ function KnowledgeGrid({
|
||||
<CardContent className="border-t bg-muted/20 px-6 py-4">
|
||||
<div className="flex items-center justify-between gap-3">
|
||||
<div className="flex min-w-0 flex-wrap items-center gap-2">
|
||||
{item.remote_status === "missing" ? <span className="text-xs text-muted-foreground">知识服务资源不存在，可清理本地记录</span> : <Button size="sm" variant="outline" className="rounded-full" onClick={() => onOpen(item)}><FileText className="mr-2 size-4" />查看详情</Button>}
|
||||
{item.remote_status !== "missing" && item.is_owner && !item.is_conversation_deposit && <Button size="sm" variant="secondary" className="rounded-full" disabled={busy} onClick={() => void onPublish(item)}>{item.publication_status === "published" ? "取消公开" : "发布公共库"}</Button>}
|
||||
<Button size="sm" variant="outline" className="rounded-full" onClick={() => onOpen(item)}><FileText className="mr-2 size-4" />查看详情</Button>
|
||||
{item.can_write && !item.is_conversation_deposit && <Button size="sm" variant="outline" className="rounded-full" disabled={busy} onClick={() => onEdit(item)} aria-label={`编辑知识库 ${item.name}`}><Edit3 className="mr-2 size-4" />编辑</Button>}
|
||||
{item.is_owner && !item.is_conversation_deposit && <Button size="sm" variant="secondary" className="rounded-full" disabled={busy} onClick={() => void onPublish(item)}>{item.publication_status === "published" ? "取消公开" : "发布公共库"}</Button>}
|
||||
</div>
|
||||
{item.can_write && !item.is_conversation_deposit && <Button size="icon" variant="ghost" className="size-8 shrink-0 rounded-full text-destructive hover:bg-destructive/10 hover:text-destructive" aria-label={item.remote_status === "missing" ? `清理失效记录 ${item.name}` : `删除知识库 ${item.name}`} title={item.remote_status === "missing" ? "清理失效记录" : "删除知识库"} disabled={busy} onClick={() => void onDelete(item)}><Trash2 className="size-4" /></Button>}
|
||||
{item.can_write && !item.is_conversation_deposit && <Button size="icon" variant="ghost" className="size-8 shrink-0 rounded-full text-destructive hover:bg-destructive/10 hover:text-destructive" aria-label={`删除知识库 ${item.name}`} title="删除知识库" disabled={busy} onClick={() => void onDelete(item)}><Trash2 className="size-4" /></Button>}
|
||||
</div>
|
||||
</CardContent>
|
||||
</Card>
|
||||
@@ -373,6 +386,55 @@ function KnowledgeGrid({
|
||||
);
|
||||
}
|
||||
|
||||
function EditKnowledgeDialog({
|
||||
knowledgeBase,
|
||||
pending,
|
||||
onClose,
|
||||
onSave,
|
||||
}: {
|
||||
knowledgeBase: LlmWikiKnowledgeBase;
|
||||
pending: boolean;
|
||||
onClose: () => void;
|
||||
onSave: (input: { name: string; description: string }) => Promise<void>;
|
||||
}) {
|
||||
const [name, setName] = useState(knowledgeBase.name);
|
||||
const [description, setDescription] = useState(knowledgeBase.description);
|
||||
|
||||
return (
|
||||
<Dialog open onOpenChange={(open) => { if (!open && !pending) onClose(); }}>
|
||||
<DialogContent>
|
||||
<form onSubmit={(event) => {
|
||||
event.preventDefault();
|
||||
if (!pending && name.trim()) {
|
||||
void onSave({ name: name.trim(), description: description.trim() });
|
||||
}
|
||||
}}>
|
||||
<DialogHeader>
|
||||
<DialogTitle>编辑知识库</DialogTitle>
|
||||
<DialogDescription>修改知识库名称和描述，保存后同步到知识服务。</DialogDescription>
|
||||
</DialogHeader>
|
||||
<div className="space-y-4 py-4">
|
||||
<div className="space-y-2">
|
||||
<label className="text-sm font-medium" htmlFor="llmwiki-edit-name">名称</label>
|
||||
<Input id="llmwiki-edit-name" value={name} onChange={(event) => setName(event.target.value)} maxLength={255} required disabled={pending} />
|
||||
</div>
|
||||
<div className="space-y-2">
|
||||
<label className="text-sm font-medium" htmlFor="llmwiki-edit-description">描述</label>
|
||||
<Textarea id="llmwiki-edit-description" value={description} onChange={(event) => setDescription(event.target.value)} rows={4} maxLength={2048} disabled={pending} />
|
||||
</div>
|
||||
</div>
|
||||
<DialogFooter>
|
||||
<Button type="button" variant="outline" onClick={onClose} disabled={pending}>取消</Button>
|
||||
<Button type="submit" disabled={pending || !name.trim()}>
|
||||
{pending && <Loader2 className="mr-2 size-4 animate-spin" />}保存
|
||||
</Button>
|
||||
</DialogFooter>
|
||||
</form>
|
||||
</DialogContent>
|
||||
</Dialog>
|
||||
);
|
||||
}
|
||||
|
||||
function CreateKnowledgeDialog({
|
||||
open,
|
||||
pending,
|
||||
|
||||
+9
-2
@@ -52,6 +52,8 @@ import {
|
||||
usePublicLlmWikiWikiPages,
|
||||
} from "@/strategy-components/api/llmwiki";
|
||||
import { WikiProtectedImage } from "./WikiProtectedImage";
|
||||
import { WikiLibraryReader } from "./WikiLibraryReader";
|
||||
import { useAssistantKnowledgeBases, useAssistantKnowledgeSources } from "@/strategy-components/api/assistant-knowledge";
|
||||
|
||||
const LIST_PATH = "/page/workspace/knowledge";
|
||||
const GENERATED_PUBLIC_INDEX_SLUG = "__index__";
|
||||
@@ -121,6 +123,10 @@ export function WeKnoraWikiPageView({
|
||||
const navigate = useNavigate();
|
||||
const publicBackground = usePublicBackground();
|
||||
const requestedWikiSlug = searchParams.get("wikiSlug")?.trim() || "";
|
||||
const assistantBases = useAssistantKnowledgeBases(!publicEmbed);
|
||||
const assistantSources = useAssistantKnowledgeSources(null, !publicEmbed);
|
||||
const globalBase = assistantBases.data?.find(base => base.is_global);
|
||||
const isDeposited = assistantSources.data?.some(source => source.source_key === noteId && source.status === "active");
|
||||
const privateKnowledgeBase = useLlmWikiKnowledgeBase(
|
||||
noteId || null,
|
||||
!!noteId && !publicEmbed,
|
||||
@@ -486,6 +492,7 @@ export function WeKnoraWikiPageView({
|
||||
</div>
|
||||
</div>
|
||||
<div className="flex items-center gap-2">
|
||||
{globalBase && isDeposited && <Button variant="outline" size="sm" onClick={() => navigate(`/page/workspace/knowledge/assistant?${new URLSearchParams({ base_id: globalBase.id, wikiSlug: pageData.slug })}`)}>在知识梳理中查看</Button>}
|
||||
<Button variant="outline" size="sm" onClick={() => void download("md")} disabled={downloading !== null}>
|
||||
{downloading === "md" ? <Loader2 className="size-4 animate-spin" /> : <Download className="size-4" />}
|
||||
下载 MD
|
||||
@@ -498,7 +505,7 @@ export function WeKnoraWikiPageView({
|
||||
</header>
|
||||
)}
|
||||
|
||||
<div className="flex min-h-0 flex-1 overflow-hidden">
|
||||
{!publicEmbed ? <WikiLibraryReader pages={pagesList} activePage={pageData} mappingId={noteId} onNavigate={slug => navigate(buildWeKnoraWikiPagePath(noteId, slug))} pageHref={slug => buildAppHashUrl(buildWeKnoraWikiPagePath(noteId, slug))} /> : <div className="flex min-h-0 flex-1 overflow-hidden">
|
||||
<aside className="flex w-64 shrink-0 flex-col border-r bg-background/90 lg:w-72">
|
||||
<div className="shrink-0 border-b px-3 py-3">
|
||||
<div className="flex items-center justify-between gap-3 px-1">
|
||||
@@ -583,7 +590,7 @@ export function WeKnoraWikiPageView({
|
||||
</section>
|
||||
</article>
|
||||
</main>
|
||||
</div>
|
||||
</div>}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
+449
@@ -0,0 +1,449 @@
|
||||
import {
|
||||
BookOpenText,
|
||||
ChevronDown,
|
||||
ChevronRight,
|
||||
FileText,
|
||||
Folder,
|
||||
Link2,
|
||||
Network,
|
||||
Search,
|
||||
X,
|
||||
} from "lucide-react";
|
||||
import { useEffect, useMemo, useState } from "react";
|
||||
import { Badge } from "@/components/ui/badge";
|
||||
import { Button } from "@/components/ui/button";
|
||||
import { Input } from "@/components/ui/input";
|
||||
import { MarkdownContent } from "@/components/workspace/messages/markdown-content";
|
||||
import {
|
||||
buildWikiTree,
|
||||
filterWikiPagesWithAncestors,
|
||||
type WikiTreeNode,
|
||||
} from "@/core/llmwiki/wiki-tree";
|
||||
import {
|
||||
renderableWikiMarkdown,
|
||||
rewriteProtectedWikiImageSources,
|
||||
} from "@/core/llmwiki/wiki-page-markdown";
|
||||
import { reasoningPlugins } from "@/core/streamdown";
|
||||
import type { LlmWikiWikiPage } from "@/strategy-components/api/llmwiki";
|
||||
import { WikiProtectedImage } from "./WikiProtectedImage";
|
||||
|
||||
export type LibraryWikiPage = LlmWikiWikiPage & {
|
||||
tags?: string[];
|
||||
origins?: { mapping_id?: string; source_name: string; slug: string }[];
|
||||
};
|
||||
const TYPES: Record<string, string> = {
|
||||
index: "索引",
|
||||
summary: "摘要",
|
||||
entity: "实体",
|
||||
concept: "概念",
|
||||
synthesis: "综合分析",
|
||||
comparison: "对比分析",
|
||||
article: "文章",
|
||||
source: "资料",
|
||||
relation: "关系",
|
||||
};
|
||||
|
||||
export function WikiLibraryReader({
|
||||
pages,
|
||||
activePage,
|
||||
onNavigate,
|
||||
pageHref,
|
||||
mappingId,
|
||||
}: {
|
||||
pages: LibraryWikiPage[];
|
||||
activePage: LibraryWikiPage | null;
|
||||
onNavigate: (slug: string) => void;
|
||||
pageHref: (slug: string) => string;
|
||||
mappingId?: string;
|
||||
}) {
|
||||
const [query, setQuery] = useState("");
|
||||
const [kind, setKind] = useState("");
|
||||
const [tag, setTag] = useState("");
|
||||
const [collapsed, setCollapsed] = useState<Set<string>>(new Set());
|
||||
const [graph, setGraph] = useState(false);
|
||||
const [directoryOpen, setDirectoryOpen] = useState(false);
|
||||
const navigate = (slug: string) => {
|
||||
onNavigate(slug);
|
||||
setDirectoryOpen(false);
|
||||
};
|
||||
const groups = useMemo(
|
||||
() => [...new Set(pages.map((p) => p.page_type))],
|
||||
[pages],
|
||||
);
|
||||
const tags = useMemo(
|
||||
() =>
|
||||
[
|
||||
...new Set(
|
||||
pages.flatMap((p) => [
|
||||
...(p.tags || p.page_metadata?.tags || []),
|
||||
...(p.aliases || []),
|
||||
]),
|
||||
),
|
||||
].sort(),
|
||||
[pages],
|
||||
);
|
||||
const tree = useMemo(
|
||||
() =>
|
||||
buildWikiTree(
|
||||
filterWikiPagesWithAncestors(
|
||||
pages.filter(
|
||||
(p) =>
|
||||
(!kind || p.page_type === kind) &&
|
||||
(!tag ||
|
||||
[
|
||||
...(p.tags || p.page_metadata?.tags || []),
|
||||
...(p.aliases || []),
|
||||
].includes(tag)),
|
||||
),
|
||||
query,
|
||||
),
|
||||
),
|
||||
[pages, kind, tag, query],
|
||||
);
|
||||
const bySlug = useMemo(() => new Map(pages.map((p) => [p.slug, p])), [pages]);
|
||||
useEffect(() => {
|
||||
const ancestors: string[] = [];
|
||||
const visit = (nodes: WikiTreeNode[], parents: string[]): boolean =>
|
||||
nodes.some((node) => {
|
||||
if (node.page?.slug === activePage?.slug) {
|
||||
ancestors.push(...parents);
|
||||
return true;
|
||||
}
|
||||
return visit(node.children, [...parents, node.key]);
|
||||
});
|
||||
visit(tree.roots, []);
|
||||
setCollapsed(
|
||||
(current) =>
|
||||
new Set([...current].filter((key) => !ancestors.includes(key))),
|
||||
);
|
||||
}, [activePage?.slug, tree]);
|
||||
const links = useMemo(() => {
|
||||
const outgoing = new Set(activePage?.out_links || []);
|
||||
for (const match of (activePage?.content || "").matchAll(
|
||||
/\[\[([^\]|]+)(?:\|[^\]]+)?\]\]/g,
|
||||
))
|
||||
outgoing.add(match[1].trim());
|
||||
const incoming = new Set(activePage?.in_links || []);
|
||||
for (const p of pages)
|
||||
if (
|
||||
p.slug !== activePage?.slug &&
|
||||
((p.out_links || []).includes(activePage?.slug || "") ||
|
||||
p.content?.includes(`[[${activePage?.slug}`))
|
||||
)
|
||||
incoming.add(p.slug);
|
||||
return {
|
||||
outgoing: [...outgoing].filter((s) => bySlug.has(s)),
|
||||
incoming: [...incoming].filter((s) => bySlug.has(s)),
|
||||
};
|
||||
}, [activePage, pages, bySlug]);
|
||||
const renderNode = (node: WikiTreeNode, depth = 0): React.ReactNode => {
|
||||
const active = activePage?.slug === node.page?.slug;
|
||||
const open = !!query || !collapsed.has(node.key);
|
||||
const toggle = () =>
|
||||
setCollapsed((current) => {
|
||||
const next = new Set(current);
|
||||
if (next.has(node.key)) next.delete(node.key);
|
||||
else next.add(node.key);
|
||||
return next;
|
||||
});
|
||||
return (
|
||||
<div key={node.key}>
|
||||
<div
|
||||
style={{ paddingLeft: 8 + depth * 14 }}
|
||||
className={`flex items-center gap-1 rounded-lg pr-2 text-sm ${active ? "bg-emerald-100 text-emerald-800 dark:bg-emerald-950 dark:text-emerald-200" : "text-muted-foreground hover:bg-muted"}`}
|
||||
>
|
||||
{node.children.length ? (
|
||||
<button
|
||||
aria-label={open ? "收起分类" : "展开分类"}
|
||||
onClick={toggle}
|
||||
className="p-1"
|
||||
>
|
||||
{open ? (
|
||||
<ChevronDown className="size-4" />
|
||||
) : (
|
||||
<ChevronRight className="size-4" />
|
||||
)}
|
||||
</button>
|
||||
) : (
|
||||
<span className="w-6" />
|
||||
)}
|
||||
<button
|
||||
onClick={() => (node.page ? navigate(node.page.slug) : toggle())}
|
||||
className="flex min-w-0 flex-1 items-center gap-2 py-2.5 text-left"
|
||||
aria-current={active ? "page" : undefined}
|
||||
>
|
||||
{node.page ? (
|
||||
<FileText className="size-4 shrink-0" />
|
||||
) : (
|
||||
<Folder className="size-4 shrink-0 text-amber-500" />
|
||||
)}
|
||||
<span className="truncate">{TYPES[node.label] || node.label}</span>
|
||||
</button>
|
||||
</div>
|
||||
{open && node.children.map((child) => renderNode(child, depth + 1))}
|
||||
</div>
|
||||
);
|
||||
};
|
||||
const renderLinks = (values: string[]) =>
|
||||
values.map((slug) => (
|
||||
<button
|
||||
key={slug}
|
||||
onClick={() => navigate(slug)}
|
||||
className="block w-full rounded-lg px-2 py-2 text-left text-sm text-primary hover:bg-muted"
|
||||
>
|
||||
<Link2 className="mr-2 inline size-3.5" />
|
||||
{bySlug.get(slug)?.title || slug}
|
||||
</button>
|
||||
));
|
||||
return (
|
||||
<div className="relative flex min-h-0 flex-1 overflow-hidden bg-muted/20">
|
||||
{directoryOpen && (
|
||||
<button
|
||||
aria-label="关闭 Wiki 目录"
|
||||
className="absolute inset-0 z-20 bg-black/30 md:hidden"
|
||||
onClick={() => setDirectoryOpen(false)}
|
||||
/>
|
||||
)}
|
||||
<aside
|
||||
className={`${directoryOpen ? "flex" : "hidden"} absolute inset-y-0 left-0 z-30 w-64 shrink-0 flex-col border-r bg-background md:static md:z-auto md:flex lg:w-72`}
|
||||
>
|
||||
<div className="space-y-3 border-b p-4">
|
||||
<div className="flex items-center gap-2 font-semibold">
|
||||
<BookOpenText className="size-4 text-emerald-600" />
|
||||
Wiki 目录
|
||||
<span className="ml-auto text-xs font-normal text-muted-foreground">
|
||||
{pages.length} 篇
|
||||
</span>
|
||||
<button
|
||||
className="md:hidden"
|
||||
aria-label="收起 Wiki 目录"
|
||||
onClick={() => setDirectoryOpen(false)}
|
||||
>
|
||||
<X className="size-4" />
|
||||
</button>
|
||||
</div>
|
||||
<div className="relative">
|
||||
<Search className="absolute left-2 top-3 size-4 text-muted-foreground" />
|
||||
<Input
|
||||
value={query}
|
||||
onChange={(e) => setQuery(e.target.value)}
|
||||
placeholder="搜索标题、分类或标签"
|
||||
className="pl-8"
|
||||
/>
|
||||
</div>
|
||||
<select
|
||||
aria-label="Wiki 分类"
|
||||
className="h-9 w-full rounded-md border bg-background px-2 text-sm"
|
||||
value={kind}
|
||||
onChange={(e) => setKind(e.target.value)}
|
||||
>
|
||||
<option value="">全部类型</option>
|
||||
{groups.map((k) => (
|
||||
<option key={k} value={k}>
|
||||
{TYPES[k] || k}（{pages.filter((p) => p.page_type === k).length}
|
||||
）
|
||||
</option>
|
||||
))}
|
||||
</select>
|
||||
{tags.length > 0 && (
|
||||
<select
|
||||
aria-label="Wiki 标签"
|
||||
className="h-9 w-full rounded-md border bg-background px-2 text-sm"
|
||||
value={tag}
|
||||
onChange={(e) => setTag(e.target.value)}
|
||||
>
|
||||
<option value="">全部标签</option>
|
||||
{tags.map((t) => (
|
||||
<option key={t}>{t}</option>
|
||||
))}
|
||||
</select>
|
||||
)}
|
||||
</div>
|
||||
<nav
|
||||
className="min-h-0 flex-1 overflow-auto p-2"
|
||||
aria-label="Wiki 目录"
|
||||
>
|
||||
{tree.roots.map((node) => renderNode(node))}
|
||||
{!tree.roots.length && (
|
||||
<p className="p-4 text-sm text-muted-foreground">暂无匹配页面</p>
|
||||
)}
|
||||
</nav>
|
||||
</aside>
|
||||
<main className="min-w-0 flex-1 overflow-auto p-3 md:p-5 lg:p-8">
|
||||
<Button
|
||||
variant="outline"
|
||||
size="sm"
|
||||
className="mb-3 md:hidden"
|
||||
onClick={() => setDirectoryOpen(true)}
|
||||
>
|
||||
<BookOpenText className="mr-2 size-4" />
|
||||
目录与分类（{pages.length}）
|
||||
</Button>
|
||||
{activePage ? (
|
||||
<article className="mx-auto max-w-5xl rounded-2xl border bg-background shadow-sm">
|
||||
<header className="space-y-3 border-b p-4 md:px-7 md:py-6">
|
||||
<div className="text-xs text-muted-foreground">
|
||||
{(activePage.category_path || []).join(" / ") || "知识库 Wiki"}
|
||||
</div>
|
||||
<div className="flex items-start justify-between gap-4">
|
||||
<h1 className="text-2xl font-semibold tracking-tight">
|
||||
{activePage.title}
|
||||
</h1>
|
||||
<Badge variant="outline">
|
||||
{TYPES[activePage.page_type] || activePage.page_type}
|
||||
</Badge>
|
||||
</div>
|
||||
<div className="flex flex-wrap gap-2">
|
||||
{[
|
||||
...new Set([
|
||||
...(activePage.tags ||
|
||||
activePage.page_metadata?.tags ||
|
||||
[]),
|
||||
...(activePage.aliases || []),
|
||||
]),
|
||||
].map((t) => (
|
||||
<button
|
||||
key={t}
|
||||
onClick={() => {
|
||||
setTag(t);
|
||||
setDirectoryOpen(true);
|
||||
}}
|
||||
>
|
||||
<Badge variant="secondary">{t}</Badge>
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
{activePage.summary && (
|
||||
<p className="text-sm leading-7 text-muted-foreground">
|
||||
{activePage.summary}
|
||||
</p>
|
||||
)}
|
||||
<div className="flex flex-wrap gap-3 text-xs text-muted-foreground">
|
||||
{activePage.updated_at && (
|
||||
<span>
|
||||
更新于 {new Date(activePage.updated_at).toLocaleString()}
|
||||
</span>
|
||||
)}
|
||||
<span>
|
||||
{links.outgoing.length} 个引用 · {links.incoming.length}{" "}
|
||||
个反向链接
|
||||
</span>
|
||||
</div>
|
||||
</header>
|
||||
<section className="px-4 py-6 md:px-7 md:py-8 lg:px-10">
|
||||
<MarkdownContent
|
||||
content={rewriteProtectedWikiImageSources(
|
||||
renderableWikiMarkdown(
|
||||
activePage.title,
|
||||
activePage.content,
|
||||
mappingId || "assistant",
|
||||
(_id, slug) => pageHref(slug),
|
||||
pages,
|
||||
).replace(/^# [^\n]*(?:\n|$)/, ""),
|
||||
)}
|
||||
isLoading={false}
|
||||
remarkPlugins={reasoningPlugins.remarkPlugins}
|
||||
rehypePlugins={reasoningPlugins.rehypePlugins}
|
||||
components={
|
||||
mappingId
|
||||
? {
|
||||
img: (props) => (
|
||||
<WikiProtectedImage
|
||||
{...props}
|
||||
knowledgeBaseId={mappingId}
|
||||
/>
|
||||
),
|
||||
}
|
||||
: undefined
|
||||
}
|
||||
linkClassName={(href) =>
|
||||
pages.some((p) => pageHref(p.slug) === href)
|
||||
? "wiki-content-link !text-primary border-b border-dashed"
|
||||
: undefined
|
||||
}
|
||||
onLinkClick={(event, href) => {
|
||||
const target = pages.find(
|
||||
(p) =>
|
||||
pageHref(p.slug) === href ||
|
||||
href === p.slug ||
|
||||
href === `/${p.slug}`,
|
||||
);
|
||||
if (target) {
|
||||
event.preventDefault();
|
||||
navigate(target.slug);
|
||||
}
|
||||
}}
|
||||
/>
|
||||
</section>
|
||||
<section className="border-t px-4 py-5 md:px-7">
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="sm"
|
||||
onClick={() => setGraph(!graph)}
|
||||
>
|
||||
<Network className="mr-2 size-4" />
|
||||
{graph ? "收起关系图谱" : "查看关系图谱"}
|
||||
</Button>
|
||||
{graph && (
|
||||
<div className="my-4 grid grid-cols-3 items-center gap-3 rounded-xl bg-muted/40 p-4">
|
||||
<div>
|
||||
{renderLinks(links.incoming)}
|
||||
{!links.incoming.length && (
|
||||
<span className="text-xs text-muted-foreground">
|
||||
暂无反向链接
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
<div className="rounded-xl border border-primary/30 bg-primary/10 p-4 text-center text-sm font-medium">
|
||||
→ {activePage.title} →
|
||||
</div>
|
||||
<div>
|
||||
{renderLinks(links.outgoing)}
|
||||
{!links.outgoing.length && (
|
||||
<span className="text-xs text-muted-foreground">
|
||||
暂无引用
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
<div className="grid gap-5 sm:grid-cols-2">
|
||||
<div>
|
||||
<h3 className="my-2 text-sm font-medium">引用页面</h3>
|
||||
{renderLinks(links.outgoing)}
|
||||
</div>
|
||||
<div>
|
||||
<h3 className="my-2 text-sm font-medium">被哪些页面引用</h3>
|
||||
{renderLinks(links.incoming)}
|
||||
</div>
|
||||
</div>
|
||||
{(activePage.origins || [])
|
||||
.filter(
|
||||
(o, index, all) =>
|
||||
o.mapping_id &&
|
||||
all.findIndex(
|
||||
(value) =>
|
||||
value.mapping_id === o.mapping_id &&
|
||||
value.slug === o.slug,
|
||||
) === index,
|
||||
)
|
||||
.map((o) => (
|
||||
<a
|
||||
key={`${o.mapping_id}:${o.slug}`}
|
||||
className="mt-3 block text-xs text-primary hover:underline"
|
||||
href={`#/page/workspace/knowledge/${encodeURIComponent(o.mapping_id!)}/wiki/view?wikiSlug=${encodeURIComponent(o.slug)}`}
|
||||
>
|
||||
查看普通知识库来源 · {o.source_name}
|
||||
</a>
|
||||
))}
|
||||
</section>
|
||||
</article>
|
||||
) : (
|
||||
<div className="py-20 text-center text-sm text-muted-foreground">
|
||||
选择一个 Wiki 页面开始阅读。完成向量化的数据会自动沉淀到知识梳理。
|
||||
</div>
|
||||
)}
|
||||
</main>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
+126
@@ -0,0 +1,126 @@
|
||||
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import { Download, Loader2, Upload } from "lucide-react";
|
||||
import { useRef } from "react";
|
||||
import { toast } from "sonner";
|
||||
import { Button } from "@/components/ui/button";
|
||||
import { Badge } from "@/components/ui/badge";
|
||||
import {
|
||||
createAssistantExportPackage,
|
||||
importOrdinaryWikiPackage,
|
||||
listWikiTransfers,
|
||||
} from "@/strategy-components/api/assistant-knowledge";
|
||||
|
||||
export function WikiPackageTransfers({
|
||||
id,
|
||||
assistant = false,
|
||||
}: {
|
||||
id: string;
|
||||
assistant?: boolean;
|
||||
}) {
|
||||
const input = useRef<HTMLInputElement>(null);
|
||||
const client = useQueryClient();
|
||||
const key = ["wiki-transfers", assistant, id];
|
||||
const jobs = useQuery({
|
||||
queryKey: key,
|
||||
queryFn: () => listWikiTransfers(id, assistant),
|
||||
refetchInterval: (q) =>
|
||||
q.state.data?.some((j) => ["queued", "running"].includes(j.status))
|
||||
? 3000
|
||||
: false,
|
||||
});
|
||||
const action = useMutation({
|
||||
mutationFn: (file?: File) =>
|
||||
assistant
|
||||
? createAssistantExportPackage(id)
|
||||
: importOrdinaryWikiPackage(id, file!),
|
||||
onSuccess: () => {
|
||||
void client.invalidateQueries({ queryKey: key });
|
||||
toast.success(
|
||||
assistant
|
||||
? "已创建导出记录，整理完成后可下载"
|
||||
: "文件已上传，后台正在校验并导入原有 Wiki 和向量",
|
||||
);
|
||||
},
|
||||
onError: (e) => toast.error(e.message),
|
||||
});
|
||||
return (
|
||||
<div className="space-y-3 rounded-xl border bg-card p-4">
|
||||
<div className="flex flex-wrap items-center justify-between gap-3">
|
||||
<div>
|
||||
<h3 className="text-sm font-semibold">
|
||||
{assistant ? "导出 Wiki 向量数据包" : "导入 Wiki 向量数据包"}
|
||||
</h3>
|
||||
<p className="mt-1 text-xs text-muted-foreground">
|
||||
{assistant
|
||||
? "保留正文、分类、标签、链接和原始向量；后台整理完成后下载。"
|
||||
: "选择助手知识库导出的文件，导入当前空库。模型一致时直接复用向量。"}
|
||||
</p>
|
||||
</div>
|
||||
<input
|
||||
type="file"
|
||||
accept=".jsonl,.ndjson"
|
||||
ref={input}
|
||||
className="hidden"
|
||||
onChange={(e) => {
|
||||
const f = e.target.files?.[0];
|
||||
if (f) action.mutate(f);
|
||||
e.target.value = "";
|
||||
}}
|
||||
/>
|
||||
<Button
|
||||
variant="outline"
|
||||
size="sm"
|
||||
disabled={action.isPending}
|
||||
onClick={() =>
|
||||
assistant ? action.mutate(undefined) : input.current?.click()
|
||||
}
|
||||
>
|
||||
{action.isPending ? (
|
||||
<Loader2 className="mr-2 size-4 animate-spin" />
|
||||
) : assistant ? (
|
||||
<Download className="mr-2 size-4" />
|
||||
) : (
|
||||
<Upload className="mr-2 size-4" />
|
||||
)}
|
||||
{assistant ? "生成导出包" : "上传并导入"}
|
||||
</Button>
|
||||
</div>
|
||||
{jobs.error && (
|
||||
<p className="text-sm text-destructive">{jobs.error.message}</p>
|
||||
)}
|
||||
{jobs.data?.map((job) => (
|
||||
<div
|
||||
key={job.id}
|
||||
className="flex flex-wrap items-center gap-3 border-t pt-3 text-xs"
|
||||
>
|
||||
<Badge variant="outline">
|
||||
{{
|
||||
queued: "排队中",
|
||||
running: "处理中",
|
||||
failed: "失败",
|
||||
completed: "已完成",
|
||||
}[job.status] || job.status}
|
||||
</Badge>
|
||||
<span>
|
||||
{job.created_at && new Date(job.created_at).toLocaleString()}
|
||||
</span>
|
||||
<span>
|
||||
{job.counts?.wiki_page || 0} Wiki · {job.counts?.vector || 0} 向量
|
||||
</span>
|
||||
{job.file_size_bytes ? (
|
||||
<span>{(job.file_size_bytes / 1024 / 1024).toFixed(2)} MB</span>
|
||||
) : null}
|
||||
{job.error && <span className="text-destructive">{job.error}</span>}
|
||||
{assistant &&
|
||||
job.status === "completed" &&
|
||||
job.file_exists &&
|
||||
job.package_download_url && (
|
||||
<Button asChild size="sm" variant="outline" className="ml-auto">
|
||||
<a href={job.package_download_url}>下载</a>
|
||||
</Button>
|
||||
)}
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -1109,6 +1109,8 @@ Both can be modified at runtime via Gateway API endpoints or `DeerFlowClient` me
|
||||
|
||||
### Conditional WeKnora LLMWiki
|
||||
|
||||
Ordinary knowledge-base cards expose an owner/admin-only name/description edit dialog in `WeKnoraKnowledgeWorkspace.tsx`, reusing `useLlmWikiActions().update` and `PATCH /api/llmwiki/knowledge-bases/{id}`. The route trims names, rejects blank or reserved conversation-deposit names, writes to WeKnora before updating local metadata, and retains existing write authorization. A successful remote list is authoritative for every list scope, including `personal`: absent remote IDs are hidden without deleting mappings/references. Upstream failures remain errors. The grid also filters cached `remote_status=missing` rows and no longer renders stale-record cleanup cards. Regression coverage: `tests/test_llmwiki_knowledge_management.py`.
|
||||
|
||||
`llmwiki.weknora.api_base_url` is the provider switch: non-empty enables WeKnora, empty preserves the legacy LLMWiki UI. WeKnora is independently deployed and owns PostgreSQL/Redis/object storage/vector storage/parsers/models; DeerFlow config contains only `api_base_url` and optional `web_base_url`. DeerFlow deliberately sends no WeKnora credential, so the WeKnora endpoint must be restricted to a trusted network and allow DeerFlow traffic itself or through its reverse proxy. There is intentionally no `default_embedding_model_id`: `WeKnoraClient` queries `/api/v1/models` and uses WeKnora's active Embedding model when creating a KB; Wiki creation additionally resolves an active KnowledgeQA model inside WeKnora for synthesis. The native BFF is `app/gateway/routers/llmwiki.py`; ownership/publication mappings live in `deerflow.persistence.llmwiki`; the browser only sees DeerFlow mapping IDs. Knowledge-base owners publish immediately (`private` -> `published`) without a review queue, and can unpublish ordinary bases. The conversation-deposit KB is system-owned, always published, shared by all DeerFlow users, excluded from My Knowledge, and immutable through user/admin detail operations; recognition requires both the reserved name and `system` owner. Existing same-name remote mappings are adopted in place as the system deposit, preserving mapping ids and document links. Its user-facing description, generated Markdown source and tags use `cmzs` branding; list/detail reads synchronize stale remote descriptions, and search results, source previews and embedded-detail JSON recursively replace legacy brand text in historical conversation-deposit payloads. Deposit requests must reference a thread accessible to the current account. Personal knowledge-base list/search views are scoped to the current DeerFlow account even for admins; frontend LLMWiki caches include the live account id, and admin write power does not make other users' private KBs appear as owned. Only `system_role === "admin"` sees the WeKnora connection-config tab. Knowledge cards navigate to a dedicated WeKnora-style detail route. Wiki-enabled bases render and default to `Wiki -> documents -> graph`; standard bases omit Wiki and render `documents -> graph`. The detail page supports sequential multi-file upload with an aggregate result for writable ordinary bases, hides it for public read-only/system-deposit bases, and uses a generic loading label while the embed session starts. It also supports URL and published manual-Markdown imports, browses real Wiki pages/content, proxies authenticated original-file previews, pages through `/chunks/{knowledge_id}`, and renders the Wiki page graph from `/knowledgebase/{kb_id}/wiki/graph`; document details use a right sheet rather than the removed KB modal. The credential-free proxy uses one WeKnora service identity, so per-user separation is enforced by DeerFlow mappings while all remote KBs physically share that identity's WeKnora workspace. WeKnora iframe detail pages first call `/api/llmwiki/knowledge-bases/{id}/weknora/session` through the normal DeerFlow API auth path, then load `/platform/knowledge-bases/{weknora_id}` with a signed `deerflow_weknora_embed` cookie for `/platform`, `/assets`, `/locales`, root assets, and API calls. Injected fetch/XHR rewriting sends all iframe `/api/v1/*`, including colliding `/api/v1/auth/*`, through `/api/llmwiki/weknora-embed/api/v1/*`; Gateway auth/CSRF bypass only applies to those signed iframe proxy requests, the proxy still revalidates the mapped KB, and cookie security honors `X-Forwarded-Proto` for HTTPS reverse proxies. `llmwiki_search` is conditionally registered in `tools.py`, is read-only, and rechecks the current DeerFlow user, agent `llmwiki_knowledge_base_ids`, and thread-context selection on every call. Source previews prefer the cleaned Wiki/LLMWiki page text when a retrieved chunk maps to a Wiki page, with raw split chunks as the fallback. Agent bindings use an inline WeKnora-style searchable/grouped multi-select in the standard agent create/detail UI; new chats use the same interaction pattern as a compact composer dropdown, and selections are written to thread metadata/context and restored on reopen. The DeerFlow frontend is native React and the WeKnora frontend is not modified or embedded. Full deployment and behavior notes: `docs/LLMWIKI_WEKNORA_INTEGRATION_ZH.md`. Tests: `tests/test_llmwiki_weknora.py`.
|
||||
|
||||
The embedded detail proxy hides WeKnora's outer `.main > .aside_box` application sidebar and expands the route outlet, but deliberately keeps the Wiki page's internal directory/index sidebar. Do not add an HTML `sandbox` attribute to this trusted, permission-filtered detail iframe: a sandboxed ancestor causes Chromium's built-in Blob PDF viewer to render the browser-blocked page. The Gateway follows upstream redirects only for `/knowledge/{id}/preview` and `/knowledge-bases/{id}/files`, keeping PDF bytes and protected Wiki images inside the signed proxy instead of sending the browser to WeKnora object storage.
|
||||
@@ -1745,6 +1747,36 @@ imports or arbitrary local-file imports to assistant knowledge. Importing into a
|
||||
custom assistant base must also mirror the package into the global base for
|
||||
cross-source entity alignment.
|
||||
|
||||
### Wiki package and local encoder invariants (2026-09)
|
||||
|
||||
- `knowledge_transfer.sync_to_global` is called after successful explicit local
|
||||
Wiki vectorization; it skips conversation/system mappings. Do not enable a new
|
||||
watcher or automatic remote-write scheduler as a side effect.
|
||||
- `wiki_packages` exposes file-backed assistant exports and restores into empty
|
||||
ordinary Wiki libraries. Validate vectors before any remote mutation; recreate
|
||||
folder IDs from category paths using provider APIs (a raw category_path is
|
||||
discarded by WeKnora without a matching folder_id). Preserve page metadata and
|
||||
rebuild backlinks after restoring all pages.
|
||||
- `PackageArchive` is a replayable, disk-backed JSONL staging index. Always close
|
||||
it in finally. Runtime imports must not use the legacy in-memory read_package.
|
||||
- Preserve normalized float32 base64 bytes exactly. Same dimension alone is not
|
||||
model compatibility. Assistant vector retrieval is fingerprint-gated cosine
|
||||
search, and any keyword fallback must be labeled as such.
|
||||
Preserve a valid zero section_index. Export/search vectors belonging to the
|
||||
current Wiki revision's import job, not stale contributions from other sources.
|
||||
- The `local` embedding provider uses FastEmbed with official BAAI/bge-m3 ONNX,
|
||||
1024 dimensions, bounded CPU batches and a per-client async lock. Run inference
|
||||
off the event loop. Existing page vectors are reusable; zero-vector nonempty
|
||||
pages need backfill, not a forced rebuild of every page.
|
||||
- Provider graph nodes bind to their original wiki_slug. They must not create
|
||||
duplicate synthetic entity/relation articles. Store original Wiki metadata
|
||||
separately from assistant workflow status. Count Wiki pages separately from
|
||||
raw document chunks.
|
||||
- Migration `20260903_01` adds nullable page metadata and merges the pre-existing
|
||||
knowledge/workflow Alembic heads. Regression coverage is in
|
||||
`tests/test_wiki_package_roundtrip.py`, `test_llmwiki_wiki_sync.py` and
|
||||
`test_assistant_knowledge_repository.py`.
|
||||
|
||||
## openGauss / GaussDB persistence invariant
|
||||
|
||||
`database.backend=gauss` is an application-ORM backend, parallel to `mysql`.
|
||||
|
||||
@@ -900,6 +900,12 @@ without deleting the audit row. DeerFlow stores only the artifact reference
|
||||
never the Markdown body and never a task-system delivery. Continue to preview
|
||||
the file through the existing thread-artifact endpoint.
|
||||
|
||||
## 普通知识库管理
|
||||
|
||||
在「普通知识库」的「我的知识库」或「公共 LLMWiki」中，知识库所有者和管理员可以点击卡片上的「编辑」，修改名称和描述（描述可清空）。保存后同步更新知识服务和本地记录，并刷新列表；系统「对话沉淀」库不提供编辑入口。
|
||||
|
||||
远端已经删除的知识库会在下次列表刷新时自动隐藏，包括个人、公共列表和知识库选择器；页面每 15 秒自动刷新，也可以手动点击「刷新」。远端服务请求失败时展示错误，不把连接故障当成删除，不自动删除本地映射或已有引用。
|
||||
|
||||
## DeerFlow-local LLMWiki Wiki vector index
|
||||
|
||||
LLMWiki can mirror processed WeKnora Wiki pages into DeerFlow and search only
|
||||
@@ -1113,6 +1119,51 @@ assistant base also mirrors the same package into the global base, where
|
||||
canonical entities and relations are aligned across all imported sources.
|
||||
The database schema is introduced by Alembic revision `20260901_01`.
|
||||
|
||||
### Wiki 流转与本地小批量编码
|
||||
|
||||
技能归纳现在上传清理后的知识正文，等待 WeKnora 完成 Wiki 分析，再建立本地
|
||||
Wiki 向量并通过落盘导出包自动沉淀到「知识梳理」。内置技能没有数据库所有权记录
|
||||
也可以归纳。若解析结束却没有 Wiki，会提示检查生成模型和资料；文档解析完成
|
||||
不等于 Wiki 生成成功，例如生成服务余额不足仍可能表现为文档已完成。
|
||||
|
||||
管理员可以在普通知识库直接生成向量数据包，每次操作有独立后台记录和下载入口。
|
||||
助手知识库支持上传包、异步导入和导出；新建空的普通 Wiki 库可上传助手导出包。
|
||||
包保留正文、别名/标签、目录分类、双向链接、实体关系及 float32 向量。
|
||||
回流时通过 WeKnora API 重建目录/页面，向量进入 DeerFlow 本地 Wiki 索引，
|
||||
**不直接写 WeKnora 的底层向量或图数据库**。普通与助手 Wiki 使用同一个阅读组件。
|
||||
|
||||
小批量 CPU 编码可配置如下，无需额外部署 HTTP 编码服务：
|
||||
|
||||
```bash
|
||||
uv sync --extra local-embedding
|
||||
```
|
||||
|
||||
```yaml
|
||||
llmwiki:
|
||||
local_wiki_index:
|
||||
enabled: true
|
||||
auto_sync: false
|
||||
embedding:
|
||||
provider: local
|
||||
model: bge-embedding-m3
|
||||
dimensions: 1024
|
||||
batch_size: 2
|
||||
local_threads: 2
|
||||
# 离线部署时可指定已下载的 BAAI/bge-m3 模型目录：
|
||||
# local_model_path: /opt/models/bge-m3
|
||||
```
|
||||
|
||||
首次使用需要预下载官方 ONNX 权重（约 2.3 GB），默认持久缓存位于
|
||||
`~/.cache/deerflow/embeddings`，也可设置 `local_cache_dir`。权重不进入业务源码仓库。
|
||||
相同模型指纹、相同内容的页面复用现有向量；短正文/摘要至少保留一个非空分段。
|
||||
数据包导入不重新编码，模型指纹或维度不一致时明确拒绝，不能只按维度混用模型。
|
||||
问答编码查询本身仍需可用的同一模型。WeKnora 的原始文档向量不能当作生成后
|
||||
Wiki 正文的向量使用，两种文本需要分别建立各自的首次索引。
|
||||
|
||||
大包采用流式 JSONL 落盘、同盘临时 SQLite 分阶段读取及异步导入，避免一次性将
|
||||
整个包加载进内存。部署需预留包、暂存索引和业务数据库空间，并配置代理上传大小
|
||||
与超时。当前不宣称断点续传或 GB 级压力验收已完成。
|
||||
|
||||
## openGauss / GaussDB application database
|
||||
|
||||
The application ORM can use a PG-compatible openGauss/GaussDB database through
|
||||
|
||||
@@ -1586,6 +1586,9 @@ This gateway provides custom endpoints for models, MCP configuration, skills, an
|
||||
app.include_router(skill_knowledge.router)
|
||||
app.include_router(assistant_knowledge.router)
|
||||
app.include_router(assistant_knowledge.export_router)
|
||||
from app.gateway.routers import wiki_packages
|
||||
|
||||
app.include_router(wiki_packages.router)
|
||||
|
||||
# Artifacts API is mounted at /api/threads/{thread_id}/artifacts
|
||||
app.include_router(artifacts.router)
|
||||
|
||||
@@ -381,7 +381,7 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
embedding_config = local_index_config.embedding
|
||||
embedding_configured = bool(
|
||||
embedding_config.enabled
|
||||
and embedding_config.base_url
|
||||
and (embedding_config.base_url or embedding_config.provider == "local")
|
||||
and embedding_config.model.strip()
|
||||
)
|
||||
if embedding_configured:
|
||||
@@ -404,13 +404,19 @@ async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
app.state.llmwiki_vector_cache,
|
||||
)
|
||||
if weknora_client is not None:
|
||||
app.state.llmwiki_sync_service = WikiSyncService(
|
||||
app.state.llmwiki_sync_service = WikiSyncService(
|
||||
local_index_config,
|
||||
app.state.llmwiki_index_store,
|
||||
app.state.llmwiki_embedding,
|
||||
app.state.llmwiki_vector_cache,
|
||||
weknora_client,
|
||||
)
|
||||
weknora_client,
|
||||
)
|
||||
from app.gateway.knowledge_transfer import sync_to_global
|
||||
|
||||
async def on_wiki_vectorized(mapping):
|
||||
return await sync_to_global(app, mapping)
|
||||
|
||||
app.state.llmwiki_sync_service.on_completed = on_wiki_vectorized
|
||||
if weknora_client is not None or app.state.llmwiki_vector_search is not None:
|
||||
from deerflow.integrations.weknora.local_index.retrieval import WikiRetrievalService
|
||||
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
"""Wiki package flow orchestration; all remote writes use WeKnora APIs."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from starlette.requests import Request
|
||||
|
||||
|
||||
async def sync_to_global(app, mapping):
|
||||
from app.gateway.routers.assistant_knowledge import _export_weknora_then_import_for_job
|
||||
|
||||
if mapping.get("is_conversation_deposit") or mapping.get("owner_user_id") == "system":
|
||||
return {"status": "excluded"}
|
||||
store = app.state.assistant_knowledge_store
|
||||
revision, fingerprint, _ = await app.state.llmwiki_index_store.get_index_revision(str(mapping["id"]))
|
||||
base = await store.initialize(created_by=None)
|
||||
for source in await store.list_sources(base_id=base["id"]):
|
||||
meta = source.get("metadata") or {}
|
||||
if source.get("source_key") == mapping["id"] and meta.get("index_revision") == revision:
|
||||
job = await store.get_job(source.get("last_import_job_id")) if source.get("last_import_job_id") else None
|
||||
if job and job.get("status") == "completed":
|
||||
return {"status": "unchanged", "job_id": job["id"], "base_id": base["id"]}
|
||||
source = {"provider": "weknora", "type": "weknora", "knowledge_base_id": mapping["id"], "knowledge_base_name": mapping["name"], "index_revision": revision, "embedding_fingerprint": fingerprint}
|
||||
job = await store.create_queued_import_job(source_type="weknora", source_key=mapping["id"], source_name=mapping["name"], trigger="vectorization_completed", created_by=None, metadata=source, mapping_id=mapping["id"], base_id=base["id"])
|
||||
await _export_weknora_then_import_for_job(request=Request({"type": "http", "app": app}), job_id=job["id"], mapping=mapping, remote_id=mapping["weknora_id"], source=source, actor=None, index_fresh=True)
|
||||
result = await store.get_job(job["id"])
|
||||
if not result or result.get("status") != "completed":
|
||||
raise RuntimeError(f"向量化已完成，但知识梳理同步失败：{(result or {}).get('last_error_message') or job['id']}")
|
||||
return {"status": "completed", "job_id": job["id"], "base_id": base["id"], "counts": result.get("counts")}
|
||||
@@ -6,6 +6,7 @@ import asyncio
|
||||
import base64
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from collections.abc import AsyncIterable, AsyncIterator
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
@@ -17,7 +18,8 @@ from fastapi.responses import FileResponse
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.gateway.deps import get_optional_user_from_request
|
||||
from deerflow.assistant_knowledge.package_io import PACKAGE_VERSION, package_path, read_package, write_jsonl_record
|
||||
from deerflow.assistant_knowledge.archive import PackageArchive
|
||||
from deerflow.assistant_knowledge.package_io import PACKAGE_VERSION, package_path, write_jsonl_record
|
||||
from deerflow.config.runtime_paths import runtime_home
|
||||
from deerflow.integrations.weknora.client import WeKnoraError
|
||||
from deerflow.integrations.weknora.runtime import build_weknora_client, get_resolved_llmwiki_runtime
|
||||
@@ -114,7 +116,9 @@ def _write_export_manifest(
|
||||
}
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
payload = _normalize_export_manifest(payload)
|
||||
path.write_text(json.dumps(payload, ensure_ascii=False), encoding="utf-8")
|
||||
temp = path.with_suffix(".tmp")
|
||||
temp.write_text(json.dumps(payload, ensure_ascii=False), encoding="utf-8")
|
||||
temp.replace(path)
|
||||
return payload
|
||||
|
||||
|
||||
@@ -227,10 +231,11 @@ async def list_wiki_pages(
|
||||
q: str | None = Query(default=None),
|
||||
page_type: str | None = Query(default=None),
|
||||
limit: int = Query(default=100, ge=1, le=500),
|
||||
offset: int = Query(default=0, ge=0),
|
||||
) -> dict[str, Any]:
|
||||
await _actor(request)
|
||||
pages = await _store(request).list_pages(base_id=base_id, query=q, page_type=page_type, limit=limit)
|
||||
return {"pages": pages, "total": len(pages)}
|
||||
pages = await _store(request).list_pages(base_id=base_id, query=q, page_type=page_type, limit=limit + 1, offset=offset)
|
||||
return {"pages": pages[:limit], "total": min(len(pages), limit), "next_offset": offset + limit if len(pages) > limit else None}
|
||||
|
||||
|
||||
@router.get("/wiki-pages/{slug:path}/revisions/compare")
|
||||
@@ -279,8 +284,11 @@ async def search(
|
||||
page_type: str | None = Query(default=None),
|
||||
) -> dict[str, Any]:
|
||||
await _actor(request)
|
||||
rows = await _store(request).list_pages(base_id=base_id, query=q, page_type=page_type, limit=100)
|
||||
return {"results": rows, "total": len(rows)}
|
||||
store = _store(request)
|
||||
base = await store.get_base(base_id)
|
||||
embedding = getattr(request.app.state, "llmwiki_embedding", None)
|
||||
rows = await store.search_pages(base_ids=[base["id"]] if base else [], query=q, limit=20, include_chunks=False, embedding_client=embedding)
|
||||
return {"results": rows, "total": len(rows), "retrieval_mode": rows[0].get("retrieval_mode", "keyword") if rows else "empty"}
|
||||
|
||||
|
||||
@router.get("/sources")
|
||||
@@ -399,7 +407,10 @@ async def import_weknora_package_file(
|
||||
if not chunk:
|
||||
break
|
||||
size += len(chunk)
|
||||
handle.write(chunk)
|
||||
await asyncio.to_thread(handle.write, chunk)
|
||||
except Exception as exc:
|
||||
await _store(request).update_import_job(str(job["id"]), status="failed", phase="upload_failed", error=str(exc), completed=True)
|
||||
raise
|
||||
finally:
|
||||
await file.close()
|
||||
if size <= 0:
|
||||
@@ -454,6 +465,7 @@ async def _write_weknora_export_package(
|
||||
"source",
|
||||
{
|
||||
**source,
|
||||
"provider": "weknora",
|
||||
"package_version": PACKAGE_VERSION,
|
||||
"export_format": "assistant-knowledge-jsonl",
|
||||
},
|
||||
@@ -462,6 +474,8 @@ async def _write_weknora_export_package(
|
||||
# Wiki pages first: these are the human-readable pages that the
|
||||
# assistant knowledge base presents in wiki form.
|
||||
page_no = 1
|
||||
graph_nodes = []
|
||||
graph_edges = set()
|
||||
while True:
|
||||
page = await client.list_wiki_pages(remote_id, page=page_no, page_size=100)
|
||||
batch = page.get("pages") or []
|
||||
@@ -475,14 +489,16 @@ async def _write_weknora_export_package(
|
||||
if isinstance(detail, dict):
|
||||
payload = {**row, **detail}
|
||||
except WeKnoraError:
|
||||
logger.warning(
|
||||
"Could not fetch full WeKnora wiki page during export: kb=%s slug=%s",
|
||||
remote_id,
|
||||
slug,
|
||||
exc_info=True,
|
||||
)
|
||||
# A list summary is not a replacement for the full
|
||||
# article. An incomplete package must fail visibly.
|
||||
raise
|
||||
write_jsonl_record(handle, "wiki_page", payload)
|
||||
counts["wiki_page"] += 1
|
||||
graph_nodes.append({"slug": slug, "title": payload.get("title"), "page_type": payload.get("page_type"), "aliases": payload.get("aliases") or []})
|
||||
for target in payload.get("out_links") or []:
|
||||
graph_edges.add((slug, str(target)))
|
||||
for match in re.finditer(r"\[\[([^\]|]+)(?:\|[^\]]+)?\]\]", str(payload.get("content") or "")):
|
||||
graph_edges.add((slug, match.group(1).strip()))
|
||||
if counts["wiki_page"] >= int(page.get("total") or counts["wiki_page"]) or not batch:
|
||||
break
|
||||
page_no += 1
|
||||
@@ -526,23 +542,25 @@ async def _write_weknora_export_package(
|
||||
write_jsonl_record(handle, "vector", row)
|
||||
counts["vector"] += 1
|
||||
|
||||
try:
|
||||
graph = await client.get_wiki_graph(remote_id, limit=10000)
|
||||
except WeKnoraError:
|
||||
graph = {"nodes": [], "edges": [], "capability_warning": "graph API unavailable"}
|
||||
# Build the complete link graph from the paginated full pages. The
|
||||
# provider's overview graph endpoint is capped and can omit nodes.
|
||||
known_slugs = {row["slug"] for row in graph_nodes}
|
||||
graph = {"nodes": graph_nodes, "edges": [{"source": a, "target": b} for a, b in sorted(graph_edges) if a in known_slugs and b in known_slugs]}
|
||||
nodes = [row for row in graph.get("nodes") or [] if isinstance(row, dict)]
|
||||
edges = [row for row in graph.get("edges") or [] if isinstance(row, dict)]
|
||||
write_jsonl_record(handle, "graph", {**graph, "nodes": nodes, "edges": edges})
|
||||
counts["graph"] += 1
|
||||
for row in nodes:
|
||||
if row.get("id") and (row.get("title") or row.get("name")):
|
||||
if (row.get("slug") or row.get("id")) and (row.get("title") or row.get("name")):
|
||||
write_jsonl_record(
|
||||
handle,
|
||||
"entity",
|
||||
{
|
||||
"id": row.get("id"),
|
||||
"id": row.get("slug") or row.get("id"),
|
||||
"name": row.get("title") or row.get("name"),
|
||||
"type": row.get("type") or "concept",
|
||||
"type": row.get("page_type") or row.get("type") or "concept",
|
||||
"source_wiki_slug": row.get("slug") or row.get("id"),
|
||||
"aliases": row.get("aliases") or [],
|
||||
"confidence": 1.0,
|
||||
},
|
||||
)
|
||||
@@ -555,6 +573,7 @@ async def _write_weknora_export_package(
|
||||
"source_entity_id": row.get("source"),
|
||||
"target_entity_id": row.get("target"),
|
||||
"predicate": row.get("predicate") or "references",
|
||||
"source_wiki_slug": row.get("source"),
|
||||
"confidence": 1.0,
|
||||
"evidence": [{"source": "weknora_graph"}],
|
||||
},
|
||||
@@ -627,8 +646,9 @@ async def _iter_local_wiki_vectors_for_export(
|
||||
payload = _local_wiki_vector_export_payload(row, model=model, fingerprint=fingerprint)
|
||||
if payload is not None:
|
||||
yield payload
|
||||
except Exception: # noqa: BLE001
|
||||
logger.debug("No local wiki vector snapshot available for %s", mapping_id, exc_info=True)
|
||||
except Exception:
|
||||
logger.exception("Wiki vector export failed for %s", mapping_id)
|
||||
raise
|
||||
|
||||
|
||||
async def _has_local_wiki_vectors_for_export(request: Request, mapping_id: str) -> bool:
|
||||
@@ -646,12 +666,11 @@ async def _ensure_local_wiki_vectors_for_export(
|
||||
mapping: dict[str, Any],
|
||||
) -> None:
|
||||
mapping_id = str(mapping.get("id") or mapping.get("knowledge_base_id") or "")
|
||||
if not mapping_id or await _has_local_wiki_vectors_for_export(request, mapping_id):
|
||||
return
|
||||
|
||||
service = getattr(request.app.state, "llmwiki_sync_service", None)
|
||||
if service is None or not mapping_id:
|
||||
return
|
||||
raise RuntimeError("请先配置 Wiki 编码模型并启用本地向量索引，再生成向量数据包")
|
||||
# Refresh remote metadata/content hashes even when the previous generation
|
||||
# was complete. Unchanged articles reuse their existing vectors.
|
||||
sync_mapping = dict(mapping)
|
||||
sync_mapping.setdefault("id", mapping_id)
|
||||
sync_mapping.setdefault(
|
||||
@@ -660,15 +679,9 @@ async def _ensure_local_wiki_vectors_for_export(
|
||||
)
|
||||
if not sync_mapping.get("weknora_id"):
|
||||
return
|
||||
try:
|
||||
logger.info("Building local wiki vectors before export: mapping=%s", mapping_id)
|
||||
await service.sync_mapping(sync_mapping, force=False)
|
||||
except Exception: # noqa: BLE001
|
||||
logger.warning(
|
||||
"Could not build local wiki vectors before export: mapping=%s",
|
||||
mapping_id,
|
||||
exc_info=True,
|
||||
)
|
||||
result = await service.sync_mapping(sync_mapping, force=False, notify=False)
|
||||
if result.get("status") != "completed":
|
||||
raise RuntimeError("Wiki 向量化尚未全部完成，请查看向量化进度后重试")
|
||||
|
||||
|
||||
async def _import_package_file_for_job(
|
||||
@@ -679,9 +692,10 @@ async def _import_package_file_for_job(
|
||||
actor: str | None,
|
||||
) -> None:
|
||||
store = _store(request)
|
||||
package = None
|
||||
try:
|
||||
await store.update_import_job(job_id, status="running", phase="loading_package")
|
||||
package = await asyncio.to_thread(read_package, path)
|
||||
package = await asyncio.to_thread(PackageArchive, path)
|
||||
await store.update_import_job(job_id, status="running", phase="importing")
|
||||
await store.import_package_for_job(
|
||||
job_id,
|
||||
@@ -698,6 +712,9 @@ async def _import_package_file_for_job(
|
||||
error=str(exc),
|
||||
completed=True,
|
||||
)
|
||||
finally:
|
||||
if package is not None:
|
||||
await asyncio.to_thread(package.close)
|
||||
|
||||
|
||||
async def _export_weknora_then_import_for_job(
|
||||
@@ -708,6 +725,7 @@ async def _export_weknora_then_import_for_job(
|
||||
remote_id: str,
|
||||
source: dict[str, Any],
|
||||
actor: str | None,
|
||||
index_fresh: bool = False,
|
||||
) -> None:
|
||||
store = _store(request)
|
||||
path = package_path(_package_root(), job_id)
|
||||
@@ -717,7 +735,8 @@ async def _export_weknora_then_import_for_job(
|
||||
if not runtime.weknora_enabled:
|
||||
raise RuntimeError("未配置普通知识库服务")
|
||||
mapping_id = str(mapping.get("id") or mapping.get("knowledge_base_id") or "")
|
||||
await _ensure_local_wiki_vectors_for_export(request, mapping)
|
||||
if not index_fresh:
|
||||
await _ensure_local_wiki_vectors_for_export(request, mapping)
|
||||
counts = await _write_weknora_export_package(
|
||||
build_weknora_client(runtime),
|
||||
remote_id,
|
||||
|
||||
@@ -2041,6 +2041,7 @@ def _wiki_page_view(page: dict[str, Any]) -> dict[str, Any] | None:
|
||||
if not slug and not title and not content and not summary:
|
||||
return None
|
||||
return {
|
||||
"page_metadata": page.get("page_metadata") if isinstance(page.get("page_metadata"), dict) else {},
|
||||
"id": str(page.get("id") or ""),
|
||||
"slug": slug,
|
||||
"title": title,
|
||||
@@ -2987,12 +2988,11 @@ async def list_knowledge_bases(
|
||||
knowledge_bases: list[dict[str, Any]] = []
|
||||
for row in rows:
|
||||
remote_item = remote.get(str(row["weknora_id"]))
|
||||
row, remote_item = await _sync_conversation_deposit_view(_store(request), client, row, remote_item)
|
||||
# Owners need to see stale personal mappings so they can clean them up.
|
||||
# Public/selector views must never expose a resource that no longer exists
|
||||
# in WeKnora.
|
||||
if remote_item is None and scope != "personal":
|
||||
# A successful remote list is authoritative for every scope. Keep the
|
||||
# mapping for existing references, but never display a deleted base.
|
||||
if remote_item is None:
|
||||
continue
|
||||
row, remote_item = await _sync_conversation_deposit_view(_store(request), client, row, remote_item)
|
||||
knowledge_bases.append(_mapping_view(row, remote_item, actor_user_id=user_id, is_admin=is_admin))
|
||||
return {"knowledge_bases": knowledge_bases}
|
||||
|
||||
@@ -3077,13 +3077,19 @@ async def get_knowledge_base(request: Request, mapping_id: str) -> dict[str, Any
|
||||
async def update_knowledge_base(request: Request, mapping_id: str, body: KnowledgeBaseUpdate) -> dict[str, Any]:
|
||||
row, user_id, is_admin = await _authorized_mapping(request, mapping_id, write=True)
|
||||
changes = body.model_dump(exclude_none=True)
|
||||
if "name" in changes:
|
||||
changes["name"] = changes["name"].strip()
|
||||
if not changes["name"]:
|
||||
raise HTTPException(status_code=422, detail="知识库名称不能为空")
|
||||
if changes["name"] == CONVERSATION_DEPOSIT_KB_NAME:
|
||||
raise HTTPException(status_code=409, detail="对话沉淀是 cmzs 系统保留知识库")
|
||||
if not changes:
|
||||
return _mapping_view(row, None, actor_user_id=user_id, is_admin=is_admin)
|
||||
try:
|
||||
remote = await _client_or_503().update_knowledge_base(str(row["weknora_id"]), changes)
|
||||
except WeKnoraError as exc:
|
||||
_raise_upstream(exc)
|
||||
updated = await _store(request).update_mapping(mapping_id, name=body.name, description=body.description)
|
||||
updated = await _store(request).update_mapping(mapping_id, name=changes.get("name"), description=changes.get("description"))
|
||||
return _mapping_view(updated or row, remote, actor_user_id=user_id, is_admin=is_admin)
|
||||
|
||||
|
||||
|
||||
@@ -313,7 +313,7 @@ async def vectorize_knowledge_base(request: Request, mapping_id: str) -> dict[st
|
||||
if not mapping.get("wiki_index_enabled", True):
|
||||
mapping = await mapping_store.update_mapping(mapping_id, wiki_index_enabled=True)
|
||||
job_id = str(uuid4())
|
||||
spawn_local_wiki_index_job(request.app, _run_one_sync(request.app, mapping, force=True))
|
||||
spawn_local_wiki_index_job(request.app, _run_one_sync(request.app, mapping, force=False))
|
||||
return {"job_id": job_id, "status": "accepted", "knowledge_base_id": mapping_id}
|
||||
|
||||
|
||||
|
||||
@@ -52,9 +52,9 @@ class SyncTarget(BaseModel):
|
||||
|
||||
class CreateSyncJobRequest(BaseModel):
|
||||
skill_names: list[str] = Field(min_length=1, max_length=200)
|
||||
targets: list[SyncTarget] = Field(min_length=1, max_length=50)
|
||||
targets: list[SyncTarget] = Field(min_length=1, max_length=1)
|
||||
force: bool = False
|
||||
review_mode: Literal["required", "auto_high_confidence", "off"] = "auto_high_confidence"
|
||||
review_mode: Literal["required", "auto_high_confidence", "off"] = "off"
|
||||
|
||||
@field_validator("skill_names")
|
||||
@classmethod
|
||||
@@ -165,8 +165,14 @@ async def disable_binding_auto_sync(request: Request, binding_id: str) -> dict[s
|
||||
@router.post("/sync-jobs", status_code=202)
|
||||
async def create_sync_job(request: Request, body: CreateSyncJobRequest) -> dict[str, Any]:
|
||||
actor = await _require_admin(request)
|
||||
skill_store = request.app.state.skill_store
|
||||
missing = [name for name in body.skill_names if await skill_store.get_any(name) is None]
|
||||
# Built-in and legacy skills need not have an ownership row. Validate
|
||||
# against the same running filesystem used by the skills list/executor.
|
||||
missing = []
|
||||
for name in body.skill_names:
|
||||
try:
|
||||
await asyncio.to_thread(_skill_directory, request.app.state.config, name)
|
||||
except (FileNotFoundError, ValueError):
|
||||
missing.append(name)
|
||||
if missing:
|
||||
raise HTTPException(status_code=404, detail=f"技能不存在：{', '.join(missing[:10])}")
|
||||
targets: list[dict[str, Any]] = []
|
||||
|
||||
@@ -0,0 +1,193 @@
|
||||
"""Asynchronous, file-backed assistant export and ordinary Wiki restore."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import APIRouter, File, HTTPException, Request, UploadFile
|
||||
from fastapi.responses import FileResponse
|
||||
|
||||
from app.gateway.llmwiki_index_scheduler import spawn_local_wiki_index_job
|
||||
from app.gateway.routers.assistant_knowledge import _actor, _list_export_manifests, _package_root, _read_export_manifest, _store, _write_export_manifest
|
||||
from deerflow.assistant_knowledge.archive import PackageArchive
|
||||
from deerflow.assistant_knowledge.package_io import package_path, write_jsonl_record
|
||||
from deerflow.integrations.weknora.local_index.normalize import normalize_wiki_page
|
||||
from deerflow.integrations.weknora.runtime import build_weknora_client, get_resolved_llmwiki_runtime
|
||||
from deerflow.persistence.assistant_knowledge.store import _vector_blob_from_payload
|
||||
|
||||
router = APIRouter(tags=["wiki-packages"])
|
||||
|
||||
|
||||
def job_view(row):
|
||||
if row["mapping_id"].startswith("assistant:"):
|
||||
base_id = row["mapping_id"].split(":", 1)[1]
|
||||
row["package_download_url"] = f"/api/assistant-knowledge/bases/{base_id}/export-package/{row['id']}/download"
|
||||
elif row["mapping_id"].startswith("restore:"):
|
||||
row["package_download_url"] = None
|
||||
return row
|
||||
|
||||
|
||||
async def export_assistant(app, base_id, job_id):
|
||||
scope = f"assistant:{base_id}"
|
||||
path = package_path(_package_root(), job_id)
|
||||
temp = path.with_suffix(".tmp")
|
||||
counts = {}
|
||||
try:
|
||||
_write_export_manifest(job_id=job_id, mapping_id=scope, status="running", phase="exporting")
|
||||
with temp.open("w", encoding="utf-8", newline="\n") as handle:
|
||||
async for kind, payload in app.state.assistant_knowledge_store.iter_export_records(base_id):
|
||||
await asyncio.to_thread(write_jsonl_record, handle, kind, payload)
|
||||
counts[kind] = counts.get(kind, 0) + 1
|
||||
if counts.get("wiki_page") and not counts.get("vector"):
|
||||
raise ValueError("该助手知识库尚无可导出的 Wiki 向量，请先从已向量化的普通知识库同步")
|
||||
await asyncio.to_thread(temp.replace, path)
|
||||
_write_export_manifest(job_id=job_id, mapping_id=scope, status="completed", phase="package_ready", counts=counts)
|
||||
except Exception as exc:
|
||||
_write_export_manifest(job_id=job_id, mapping_id=scope, status="failed", phase="failed", counts=counts, error=str(exc))
|
||||
|
||||
|
||||
@router.post("/api/assistant-knowledge/bases/{base_id}/export-package", status_code=202)
|
||||
async def create_export(request: Request, base_id: str):
|
||||
await _actor(request, admin=True)
|
||||
if not await _store(request).get_base(base_id):
|
||||
raise HTTPException(404, "助手知识库不存在")
|
||||
job_id = str(uuid4())
|
||||
row = _write_export_manifest(job_id=job_id, mapping_id=f"assistant:{base_id}", status="queued", phase="queued")
|
||||
spawn_local_wiki_index_job(request.app, export_assistant(request.app, base_id, job_id))
|
||||
return job_view(row)
|
||||
|
||||
|
||||
@router.get("/api/assistant-knowledge/bases/{base_id}/export-package")
|
||||
async def list_exports(request: Request, base_id: str):
|
||||
await _actor(request, admin=True)
|
||||
return {"jobs": [job_view(row) for row in await asyncio.to_thread(_list_export_manifests, f"assistant:{base_id}", limit=100)]}
|
||||
|
||||
|
||||
@router.get("/api/assistant-knowledge/bases/{base_id}/export-package/{job_id}/download")
|
||||
async def download_export(request: Request, base_id: str, job_id: str):
|
||||
await _actor(request, admin=True)
|
||||
row = _read_export_manifest(job_id)
|
||||
if not row or row["mapping_id"] != f"assistant:{base_id}" or row["status"] != "completed" or not row["file_exists"]:
|
||||
raise HTTPException(404, "导出包尚未就绪")
|
||||
return FileResponse(package_path(_package_root(), job_id), media_type="application/x-ndjson", filename=f"wiki-vector-{job_id}.jsonl")
|
||||
|
||||
|
||||
async def restore_package(app, mapping, job_id, path: Path):
|
||||
scope = f"restore:{mapping['id']}"
|
||||
archive = None
|
||||
counts = {"wiki_page": 0, "vector": 0}
|
||||
store = app.state.llmwiki_index_store
|
||||
owner = f"package:{job_id}"
|
||||
leased = False
|
||||
try:
|
||||
_write_export_manifest(job_id=job_id, mapping_id=scope, status="running", phase="validating")
|
||||
archive = await asyncio.to_thread(PackageArchive, path)
|
||||
embedding = app.state.llmwiki_embedding
|
||||
if embedding is None:
|
||||
raise ValueError("请配置与数据包相同的 Wiki 编码模型，以便复用向量进行检索")
|
||||
if embedding.dimensions is None:
|
||||
await embedding.embed_query("Wiki embedding dimension probe")
|
||||
# Validate the complete package before the first remote mutation.
|
||||
slugs = {p["slug"] for p in archive["wiki_pages"]}
|
||||
vector_slugs = set()
|
||||
for raw in archive["vectors"]:
|
||||
slug = str(raw.get("wiki_slug") or raw.get("page_slug") or raw.get("slug") or "")
|
||||
if slug not in slugs or _vector_blob_from_payload(raw) is None:
|
||||
raise ValueError(f"向量数据或 Wiki 引用无效：{slug}")
|
||||
if raw.get("embedding_fingerprint") != embedding.fingerprint or raw.get("embedding_dimensions") != embedding.dimensions:
|
||||
raise ValueError("数据包编码模型指纹与当前配置不一致；请使用同一模型配置，禁止静默重新编码")
|
||||
vector_slugs.add(slug)
|
||||
if not vector_slugs:
|
||||
raise ValueError("数据包中没有 Wiki 向量")
|
||||
for page in archive["wiki_pages"]:
|
||||
if page["slug"] not in vector_slugs and page.get("page_type") not in {"index", "directory"} and (page.get("content") or "").strip():
|
||||
raise ValueError(f"Wiki 缺少向量：{page['slug']}")
|
||||
leased = await store.try_acquire_sync_lease(mapping["id"], owner=owner, sync_id=job_id, lease_seconds=900, fingerprint=embedding.fingerprint)
|
||||
if not leased:
|
||||
raise ValueError("目标知识库正在同步，请稍后重试")
|
||||
client = build_weknora_client(get_resolved_llmwiki_runtime(app.state.config))
|
||||
_write_export_manifest(job_id=job_id, mapping_id=scope, status="running", phase="restoring")
|
||||
folders = {(): ""}
|
||||
for raw in archive["wiki_pages"]:
|
||||
if not await store.renew_sync_lease(mapping["id"], owner=owner, lease_seconds=900):
|
||||
raise RuntimeError("导入租约已失效，请重新发起")
|
||||
# Restore the provider's Wiki schema, never raw documents; raw
|
||||
# uploads would trigger generation and embedding a second time.
|
||||
categories = raw.get("category_path") or []
|
||||
if isinstance(categories, str):
|
||||
categories = [part for part in categories.split("/") if part]
|
||||
parent = ()
|
||||
for name in categories:
|
||||
path_key = (*parent, name)
|
||||
if path_key not in folders:
|
||||
folder = await client.create_wiki_folder(mapping["weknora_id"], name=name, parent_id=folders[parent])
|
||||
folders[path_key] = folder["id"]
|
||||
parent = path_key
|
||||
payload = {key: value for key, value in raw.items() if key not in {"id", "tenant_id", "knowledge_base_id", "folder_id", "origins"}}
|
||||
payload["folder_id"] = folders[parent]
|
||||
if raw.get("tags"):
|
||||
payload["page_metadata"] = {**(raw.get("page_metadata") or {}), "tags": raw["tags"]}
|
||||
remote = await client.create_wiki_page(mapping["weknora_id"], payload)
|
||||
page = normalize_wiki_page(mapping["id"], {**raw, **remote})
|
||||
vectors = []
|
||||
for value in archive.vectors_for(raw["slug"]):
|
||||
blob, dimensions = _vector_blob_from_payload(value)
|
||||
vectors.append({**value, "vector_blob": blob, "embedding_dimensions": dimensions})
|
||||
await store.replace_page_vectors(page=page, vectors=vectors, fingerprint=embedding.fingerprint, sync_id=job_id)
|
||||
counts["wiki_page"] += 1
|
||||
counts["vector"] += len(vectors)
|
||||
_write_export_manifest(job_id=job_id, mapping_id=scope, status="running", phase="restoring", counts=counts)
|
||||
await client.rebuild_wiki_links(mapping["weknora_id"])
|
||||
await store.finish_sync(mapping["id"], owner=owner, sync_id=job_id, success=True, remote_page_count=counts["wiki_page"])
|
||||
leased = False
|
||||
if app.state.llmwiki_vector_cache:
|
||||
app.state.llmwiki_vector_cache.invalidate(mapping["id"])
|
||||
from app.gateway.knowledge_transfer import sync_to_global
|
||||
|
||||
await sync_to_global(app, mapping)
|
||||
_write_export_manifest(job_id=job_id, mapping_id=scope, status="completed", phase="completed", counts=counts)
|
||||
except Exception as exc:
|
||||
if leased:
|
||||
await store.finish_sync(mapping["id"], owner=owner, sync_id=job_id, success=False, remote_page_count=counts["wiki_page"], error=str(exc))
|
||||
_write_export_manifest(job_id=job_id, mapping_id=scope, status="failed", phase="failed", counts=counts, error=str(exc))
|
||||
finally:
|
||||
if archive is not None:
|
||||
await asyncio.to_thread(archive.close)
|
||||
|
||||
|
||||
@router.post("/api/llmwiki/knowledge-bases/{mapping_id}/import-package", status_code=202)
|
||||
async def upload_restore(request: Request, mapping_id: str, file: UploadFile = File(...)):
|
||||
actor = await _actor(request, admin=True)
|
||||
mapping = await request.app.state.llmwiki_store.get_authorized(mapping_id, actor or "system", write=True, is_admin=True)
|
||||
if not mapping or mapping.get("is_conversation_deposit") or mapping.get("owner_user_id") == "system":
|
||||
raise HTTPException(404, "目标普通知识库不存在或不可导入")
|
||||
client = build_weknora_client(get_resolved_llmwiki_runtime(request.app.state.config))
|
||||
remote = await client.get_knowledge_base(mapping["weknora_id"])
|
||||
if not (remote.get("indexing_strategy") or {}).get("wiki_enabled"):
|
||||
raise HTTPException(422, "请先开启目标知识库的 Wiki")
|
||||
existing = await client.list_wiki_pages(mapping["weknora_id"], page_size=1)
|
||||
if existing.get("total"):
|
||||
raise HTTPException(409, "请导入到新建的空 Wiki 知识库，避免覆盖已有页面")
|
||||
job_id = str(uuid4())
|
||||
scope = f"restore:{mapping_id}"
|
||||
row = _write_export_manifest(job_id=job_id, mapping_id=scope, status="queued", phase="uploading")
|
||||
path = package_path(_package_root(), job_id)
|
||||
try:
|
||||
with path.open("wb") as handle:
|
||||
while data := await file.read(1024 * 1024):
|
||||
await asyncio.to_thread(handle.write, data)
|
||||
except Exception as exc:
|
||||
_write_export_manifest(job_id=job_id, mapping_id=scope, status="failed", phase="upload_failed", error=str(exc))
|
||||
raise
|
||||
finally:
|
||||
await file.close()
|
||||
spawn_local_wiki_index_job(request.app, restore_package(request.app, mapping, job_id, path))
|
||||
return job_view(row)
|
||||
|
||||
|
||||
@router.get("/api/llmwiki/knowledge-bases/{mapping_id}/import-package")
|
||||
async def list_restores(request: Request, mapping_id: str):
|
||||
await _actor(request, admin=True)
|
||||
return {"jobs": [job_view(row) for row in await asyncio.to_thread(_list_export_manifests, f"restore:{mapping_id}", limit=100)]}
|
||||
@@ -27,7 +27,7 @@ def _skill_directory(app_config: Any, skill_name: str) -> Path:
|
||||
root = storage.get_skills_root_path().resolve()
|
||||
for category in SkillCategory:
|
||||
direct = root / category.value / skill_name
|
||||
if (direct / "SKILL.md").is_file():
|
||||
if direct.resolve().is_relative_to((root / category.value).resolve()) and (direct / "SKILL.md").is_file():
|
||||
return direct
|
||||
for category in SkillCategory:
|
||||
category_root = root / category.value
|
||||
@@ -235,6 +235,9 @@ class SkillKnowledgeJobExecutor:
|
||||
scan = await asyncio.to_thread(scan_skill, str(item["skill_name"]), root)
|
||||
scan = await enrich_converted_documents(scan)
|
||||
await self._assert_source_unchanged(item, expected_digest)
|
||||
if item["target_type"] == "weknora" and item["target_mode"] == "wiki":
|
||||
await self._generate_vectorize_and_deposit(item, snapshot_id, ir, scan)
|
||||
return
|
||||
await self.store.update_item(str(item["id"]), phase="writing_wiki", progress=65)
|
||||
target_type = str(item["target_type"])
|
||||
artifacts: list[dict[str, Any]] = []
|
||||
@@ -401,3 +404,63 @@ class SkillKnowledgeJobExecutor:
|
||||
await self._assert_source_unchanged(item, expected_digest)
|
||||
await self.store.update_item(str(item["id"]), phase="activating", progress=90)
|
||||
await self.store.stage_and_activate(item_id=str(item["id"]), snapshot_id=snapshot_id, ir=ir, artifacts=artifacts, status=binding_status)
|
||||
|
||||
async def _generate_vectorize_and_deposit(self, item, snapshot_id, ir, scan):
|
||||
mapping = await self.app.state.llmwiki_store.get_authorized(str(item["target_id"]), "system", write=True, is_admin=True)
|
||||
if mapping is None:
|
||||
raise ValueError("目标普通知识库不存在")
|
||||
sync = getattr(self.app.state, "llmwiki_sync_service", None)
|
||||
if sync is None:
|
||||
raise ValueError("请先配置 Wiki 编码模型并启用本地向量索引；归纳需要完成向量化后同步知识梳理")
|
||||
client = build_weknora_client(get_resolved_llmwiki_runtime(self.app.state.config))
|
||||
remote_id = mapping["weknora_id"]
|
||||
remote = await client.get_knowledge_base(remote_id)
|
||||
if not (remote.get("indexing_strategy") or {}).get("wiki_enabled"):
|
||||
raise ValueError("目标普通知识库尚未开启 Wiki")
|
||||
artifacts = []
|
||||
document_ids = []
|
||||
await self.store.update_item(str(item["id"]), phase="writing_sources", progress=55)
|
||||
for file in scan.files:
|
||||
if await self._cancel_if_requested(item):
|
||||
return
|
||||
if not file.text:
|
||||
continue
|
||||
document = await client.create_manual_document(remote_id, title=f"{ir['skill'].get('display_name') or item['skill_name']} · {file.path}", content=file.text)
|
||||
doc_id = str(document.get("id") or document.get("knowledge_id") or "")
|
||||
if not doc_id:
|
||||
raise ValueError("资料上传未返回文档标识")
|
||||
document_ids.append(doc_id)
|
||||
artifacts.append({"kind": "raw_source", "slug": file.path, "remote_id": doc_id})
|
||||
if not document_ids:
|
||||
raise ValueError("该技能没有可归纳的知识正文，代码、脚本和代码块不参与归纳")
|
||||
await self.store.update_item(str(item["id"]), phase="generating_wiki", progress=65)
|
||||
# WeKnora performs model extraction, categorization and linking. Wait
|
||||
# for both parsing and Wiki's debounced finalize before vectorization.
|
||||
stable = 0
|
||||
for _ in range(720):
|
||||
if await self._cancel_if_requested(item):
|
||||
return
|
||||
docs = [await client.get_document(doc_id) for doc_id in document_ids]
|
||||
if any(doc.get("parse_status") in {"failed", "error"} for doc in docs):
|
||||
raise RuntimeError("WeKnora 资料解析失败，请查看普通知识库中的文档状态")
|
||||
stats = await client.get_wiki_stats(remote_id)
|
||||
done = all(doc.get("parse_status") == "completed" for doc in docs) and not stats.get("is_active") and not stats.get("pending_tasks")
|
||||
stable = stable + 1 if done else 0
|
||||
if stable >= 3:
|
||||
if not int(stats.get("total_pages") or 0):
|
||||
raise RuntimeError("WeKnora 解析已结束，但未生成 Wiki；请检查 Wiki 生成模型的调用错误、余额及资料是否含实质知识，再重试")
|
||||
break
|
||||
await asyncio.sleep(5)
|
||||
else:
|
||||
raise TimeoutError("Wiki 分析尚未完成；请检查普通知识库的生成任务后重新归纳")
|
||||
await self.store.update_item(str(item["id"]), phase="vectorizing_wiki", progress=80)
|
||||
result = await sync.sync_mapping(mapping, force=False, notify=False)
|
||||
if result.get("status") != "completed":
|
||||
raise RuntimeError("Wiki 向量化未全部完成，请检查向量化进度")
|
||||
await self.store.update_item(str(item["id"]), phase="syncing_assistant", progress=92)
|
||||
from app.gateway.knowledge_transfer import sync_to_global
|
||||
|
||||
deposited = await sync_to_global(self.app, mapping)
|
||||
artifacts.append({"kind": "assistant_deposit", "slug": "knowledge-summary", "metadata": deposited})
|
||||
await self._assert_source_unchanged(item, str(ir["skill"]["source_digest"]))
|
||||
await self.store.stage_and_activate(item_id=str(item["id"]), snapshot_id=snapshot_id, ir=ir, artifacts=artifacts, status="synced")
|
||||
@@ -589,16 +589,20 @@ llmwiki:
|
||||
timeout_seconds: 30
|
||||
# 本地 Wiki 向量化：用于普通知识库“下载向量化数据包”和助手知识库向量检索。
|
||||
# 已有相同内容哈希 + 相同 embedding fingerprint 的 Wiki 向量会直接复用，不会重复编码。
|
||||
# 如需导出时自动补建向量，把 enabled 改为 true，并填写下面 embedding 服务地址。
|
||||
# local 模式使用 CPU BGE-M3，需安装 local-embedding 依赖并预下载官方 ONNX 权重。
|
||||
# auto_sync=true 才会后台自动增量同步；服务启动时只启动定时器，不会立刻全量向量化。
|
||||
local_wiki_index:
|
||||
enabled: false
|
||||
enabled: true
|
||||
# 免登录 POST /api/knowledge/vector-search 返回的 Wiki 一键跳转链接前缀。
|
||||
frontend_base_url: "http://127.0.0.1:5174"
|
||||
auto_sync: false
|
||||
database_batch_size: 50
|
||||
embedding:
|
||||
provider: local
|
||||
model: bge-embedding-m3
|
||||
dimensions: 1024
|
||||
batch_size: 2
|
||||
local_threads: 2
|
||||
base_url: ""
|
||||
api_key: ""
|
||||
|
||||
|
||||
+79
@@ -0,0 +1,79 @@
|
||||
"""Disk-backed, replayable package records. Memory is bounded by one JSONL record.
|
||||
|
||||
The staging index is local SQLite, never a remote provider's database. It also
|
||||
allows restoration to fetch only the vectors belonging to the current page.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
import tempfile
|
||||
from collections.abc import Iterator
|
||||
from contextlib import closing
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
class Records:
|
||||
def __init__(self, path: Path, kind: str, slug: str | None = None):
|
||||
self.path, self.kind, self.slug = path, kind, slug
|
||||
|
||||
def _where(self):
|
||||
return ("kind=?", (self.kind,)) if self.slug is None else ("kind=? AND slug=?", (self.kind, self.slug))
|
||||
|
||||
def __iter__(self) -> Iterator[dict[str, Any]]:
|
||||
where, args = self._where()
|
||||
with closing(sqlite3.connect(self.path)) as db:
|
||||
for (payload,) in db.execute(f"SELECT payload FROM records WHERE {where} ORDER BY seq", args):
|
||||
yield json.loads(payload)
|
||||
|
||||
def __len__(self) -> int:
|
||||
where, args = self._where()
|
||||
with closing(sqlite3.connect(self.path)) as db:
|
||||
return db.execute(f"SELECT count(*) FROM records WHERE {where}", args).fetchone()[0]
|
||||
|
||||
|
||||
class PackageArchive(dict):
|
||||
def __init__(self, path: Path):
|
||||
super().__init__()
|
||||
temporary = tempfile.NamedTemporaryFile(prefix="wiki-package-", suffix=".sqlite", dir=path.parent, delete=False)
|
||||
self.path = Path(temporary.name)
|
||||
temporary.close()
|
||||
try:
|
||||
with closing(sqlite3.connect(self.path)) as db, path.open(encoding="utf-8-sig") as handle:
|
||||
db.execute("CREATE TABLE records(seq INTEGER PRIMARY KEY, kind TEXT, slug TEXT, payload TEXT)")
|
||||
for line_no, line in enumerate(handle, 1):
|
||||
if not line.strip():
|
||||
continue
|
||||
record = json.loads(line)
|
||||
kind, payload = record.get("record_type"), record.get("payload")
|
||||
if not isinstance(payload, dict):
|
||||
raise ValueError(f"导出包第 {line_no} 行格式错误")
|
||||
if kind == "source":
|
||||
self["source"] = {**self.get("source", {}), **payload}
|
||||
elif kind == "graph":
|
||||
for key, subkind in (("nodes", "graph_node"), ("edges", "graph_edge")):
|
||||
db.executemany("INSERT INTO records(kind, payload) VALUES (?, ?)", [(subkind, json.dumps(v, ensure_ascii=False)) for v in payload.get(key) or []])
|
||||
elif kind in {"wiki_page", "document", "chunk", "vector", "entity", "relation"}:
|
||||
slug = str(payload.get("wiki_slug") or payload.get("page_slug") or payload.get("slug") or "").strip("/")
|
||||
db.execute("INSERT INTO records(kind, slug, payload) VALUES (?, ?, ?)", (kind, slug, json.dumps(payload, ensure_ascii=False)))
|
||||
else:
|
||||
raise ValueError(f"不支持的导出记录类型：{kind}")
|
||||
db.execute("CREATE INDEX by_kind_slug ON records(kind, slug)")
|
||||
db.commit()
|
||||
source = self.get("source", {})
|
||||
if source.get("package_version") != 1 or (source.get("provider") or source.get("type")) not in {"weknora", "assistant"}:
|
||||
raise ValueError("仅接受 WeKnora / 助手知识库生成的兼容 Wiki 向量包（v1）")
|
||||
for key, kind in (("wiki_pages", "wiki_page"), ("documents", "document"), ("chunks", "chunk"), ("vectors", "vector"), ("entities", "entity"), ("relations", "relation")):
|
||||
self[key] = Records(self.path, kind)
|
||||
self["graph"] = {"nodes": Records(self.path, "graph_node"), "edges": Records(self.path, "graph_edge")}
|
||||
except BaseException:
|
||||
self.close()
|
||||
raise
|
||||
|
||||
def vectors_for(self, slug: str) -> Records:
|
||||
return Records(self.path, "vector", slug)
|
||||
|
||||
def close(self):
|
||||
self.path.unlink(missing_ok=True)
|
||||
@@ -76,6 +76,9 @@ class LocalWikiEmbeddingConfig(BaseModel):
|
||||
model: str = "bge-embedding-m3"
|
||||
base_url: str = ""
|
||||
api_key: str = Field(default="", repr=False)
|
||||
local_model_path: str = ""
|
||||
local_cache_dir: str = ""
|
||||
local_threads: int = Field(default=2, ge=1, le=32)
|
||||
dimensions: int | None = Field(
|
||||
default=None,
|
||||
ge=1,
|
||||
@@ -91,8 +94,8 @@ class LocalWikiEmbeddingConfig(BaseModel):
|
||||
@classmethod
|
||||
def _validate_provider(cls, value: str) -> str:
|
||||
normalized = value.strip().lower()
|
||||
if normalized not in {"openai", "openai_compatible"}:
|
||||
raise ValueError("local Wiki embedding provider must be openai or openai_compatible")
|
||||
if normalized not in {"openai", "openai_compatible", "local"}:
|
||||
raise ValueError("Wiki embedding provider must be openai, openai_compatible or local")
|
||||
return normalized
|
||||
|
||||
@field_validator("base_url")
|
||||
@@ -198,7 +201,7 @@ class LocalWikiIndexConfig(BaseModel):
|
||||
raise ValueError("Only the numpy_exact local Wiki vector engine is currently supported")
|
||||
if self.enabled and not self.embedding.enabled:
|
||||
raise ValueError("local Wiki index embedding must be enabled when the index is enabled")
|
||||
if self.enabled and (not self.embedding.base_url or not self.embedding.model.strip()):
|
||||
if self.enabled and ((not self.embedding.base_url and self.embedding.provider != "local") or not self.embedding.model.strip()):
|
||||
raise ValueError("local Wiki index embedding base_url and model are required when enabled")
|
||||
return self
|
||||
|
||||
|
||||
+15
@@ -501,6 +501,21 @@ class WeKnoraClient:
|
||||
raise WeKnoraError("WeKnora returned an invalid wiki graph")
|
||||
return dict(value)
|
||||
|
||||
async def get_wiki_stats(self, knowledge_base_id: str) -> dict[str, Any]:
|
||||
value = _data(await self._request("GET", f"/api/v1/knowledgebase/{quote(knowledge_base_id, safe='')}/wiki/stats"))
|
||||
if not isinstance(value, dict):
|
||||
raise WeKnoraError("Invalid Wiki generation status")
|
||||
return value
|
||||
|
||||
async def create_wiki_folder(self, knowledge_base_id: str, *, name: str, parent_id: str = "") -> dict[str, Any]:
|
||||
value = _data(await self._request("POST", f"/api/v1/knowledgebase/{quote(knowledge_base_id, safe='')}/wiki/folders", json={"name": name, "parent_id": parent_id}))
|
||||
if not isinstance(value, dict) or not value.get("id"):
|
||||
raise WeKnoraError("Invalid Wiki folder response")
|
||||
return value
|
||||
|
||||
async def rebuild_wiki_links(self, knowledge_base_id: str) -> None:
|
||||
await self._request("POST", f"/api/v1/knowledgebase/{quote(knowledge_base_id, safe='')}/wiki/rebuild-links")
|
||||
|
||||
async def list_wiki_pages(
|
||||
self,
|
||||
knowledge_base_id: str,
|
||||
|
||||
+8
@@ -105,4 +105,12 @@ def chunk_wiki_page(
|
||||
context.append(f"章节：{heading}")
|
||||
embedding_text = "\n".join([*context, "", chunk]).strip()
|
||||
result.append(WikiSection(len(result), heading, chunk, embedding_text, digest))
|
||||
# A short article with a short summary must not disappear merely because
|
||||
# both sections fall below min_chars. Keep one combined, useful vector.
|
||||
if not result and (content or summary):
|
||||
fallback = "\n\n".join(dict.fromkeys(text for text in (summary, content) if text))
|
||||
normalized = re.sub(r"\s+", " ", fallback).strip()
|
||||
digest = hashlib.sha256(normalized.encode("utf-8")).hexdigest()
|
||||
prefix = prefix_lines if title_prefix_enabled else []
|
||||
result.append(WikiSection(0, None, fallback, "\n".join([*prefix, "", fallback]).strip(), digest))
|
||||
return result
|
||||
+47
@@ -5,6 +5,7 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import hashlib
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from time import monotonic
|
||||
|
||||
import httpx
|
||||
@@ -43,6 +44,8 @@ class StrictWikiEmbeddingClient:
|
||||
self._client = http_client
|
||||
self._owns_client = http_client is None
|
||||
self._resolved_dimensions = config.dimensions
|
||||
self._local_model = None
|
||||
self._local_lock = asyncio.Lock()
|
||||
|
||||
@property
|
||||
def dimensions(self) -> int | None:
|
||||
@@ -64,6 +67,11 @@ class StrictWikiEmbeddingClient:
|
||||
return self._client
|
||||
|
||||
async def embed_texts(self, texts: list[str]) -> list[list[float]]:
|
||||
if self.config.provider == "local":
|
||||
if not texts:
|
||||
return []
|
||||
async with self._local_lock:
|
||||
return await asyncio.to_thread(self._embed_local, texts)
|
||||
if not self.config.enabled or not self.config.base_url or not self.config.model:
|
||||
raise WikiEmbeddingError("Wiki embedding is not configured", category="configuration", retryable=False)
|
||||
if not texts:
|
||||
@@ -76,6 +84,45 @@ class StrictWikiEmbeddingClient:
|
||||
raise WikiEmbeddingError("Embedding response count does not match request", category="response", retryable=False)
|
||||
return vectors
|
||||
|
||||
def _embed_local(self, texts: list[str]) -> list[list[float]]:
|
||||
if not self.config.enabled:
|
||||
raise WikiEmbeddingError("本地编码已关闭", retryable=False)
|
||||
if self._local_model is None:
|
||||
try:
|
||||
from fastembed import TextEmbedding
|
||||
except ImportError as exc:
|
||||
raise WikiEmbeddingError("请安装本地编码依赖：uv sync --extra local-embedding", retryable=False) from exc
|
||||
name = "BAAI/bge-m3" if self.config.model.lower() in {"bge-m3", "bge-embedding-m3"} else self.config.model
|
||||
if name == "BAAI/bge-m3" and not any(m["model"] == name for m in TextEmbedding.list_supported_models()):
|
||||
from fastembed.common.model_description import ModelSource, PoolingType
|
||||
|
||||
TextEmbedding.add_custom_model(
|
||||
model=name,
|
||||
pooling=PoolingType.CLS,
|
||||
normalization=True,
|
||||
dim=1024,
|
||||
sources=ModelSource(hf="BAAI/bge-m3"),
|
||||
model_file="onnx/model.onnx",
|
||||
additional_files=["onnx/model.onnx_data", "onnx/Constant_7_attr__value"],
|
||||
license="mit",
|
||||
size_in_gb=2.29,
|
||||
)
|
||||
cache_dir = Path(self.config.local_cache_dir).expanduser() if self.config.local_cache_dir else Path.home() / ".cache" / "deerflow" / "embeddings"
|
||||
options = {"model_name": name, "threads": self.config.local_threads, "cache_dir": str(cache_dir), "providers": ["CPUExecutionProvider"]}
|
||||
if self.config.local_model_path:
|
||||
options["specific_model_path"] = str(Path(self.config.local_model_path).expanduser())
|
||||
options["local_files_only"] = True
|
||||
self._local_model = TextEmbedding(**options)
|
||||
values = []
|
||||
for vector in self._local_model.embed(texts, batch_size=min(self.config.batch_size, 4)):
|
||||
normalized = normalize_vector(vector, dimensions=self._resolved_dimensions)
|
||||
if self._resolved_dimensions is None:
|
||||
self._resolved_dimensions = int(normalized.size)
|
||||
values.append(normalized.tolist())
|
||||
if len(values) != len(texts):
|
||||
raise WikiEmbeddingError("本地编码返回数量不符", retryable=False)
|
||||
return values
|
||||
|
||||
async def _embed_batch(self, texts: list[str]) -> list[list[float]]:
|
||||
url = self.config.base_url.rstrip("/") + "/embeddings"
|
||||
headers = {"Content-Type": "application/json"}
|
||||
|
||||
+7
-2
@@ -36,8 +36,9 @@ class WikiSyncService:
|
||||
self.cache = cache
|
||||
self.client = client
|
||||
self.owner = f"{platform.node()}:{uuid4()}"
|
||||
self.on_completed = None
|
||||
|
||||
async def sync_mapping(self, mapping: dict[str, Any], *, force: bool = False) -> dict[str, Any]:
|
||||
async def sync_mapping(self, mapping: dict[str, Any], *, force: bool = False, notify: bool = True) -> dict[str, Any]:
|
||||
mapping_id = str(mapping["id"])
|
||||
sync_id = str(uuid4())
|
||||
if self.embedding.dimensions is None:
|
||||
@@ -111,6 +112,7 @@ class WikiSyncService:
|
||||
and existing.get("content_hash") == page["content_hash"]
|
||||
and existing.get("indexed_fingerprint") == active_fingerprint
|
||||
and existing.get("index_status") == "ready"
|
||||
and (existing.get("vector_count", 1) > 0 or page["status"] == "archived" or not (page["content_md"] or page["summary"]))
|
||||
and not existing.get("is_remote_deleted")
|
||||
):
|
||||
await self.store.touch_page(str(existing["id"]), sync_id)
|
||||
@@ -160,7 +162,10 @@ class WikiSyncService:
|
||||
remote_page_count=remote_count,
|
||||
error=f"{failed} page(s) failed" if failed else None,
|
||||
)
|
||||
return {"mapping_id": mapping_id, "status": "partial" if failed else "completed", "remote_pages": remote_count, "changed_pages": changed, "deleted_pages": deleted, "failed_pages": failed}
|
||||
result = {"mapping_id": mapping_id, "status": "partial" if failed else "completed", "remote_pages": remote_count, "changed_pages": changed, "deleted_pages": deleted, "failed_pages": failed}
|
||||
if not failed and remote_count and notify and self.on_completed:
|
||||
result["assistant_sync"] = await self.on_completed(mapping)
|
||||
return result
|
||||
except asyncio.CancelledError:
|
||||
await self.store.finish_sync(
|
||||
mapping_id,
|
||||
|
||||
+1
@@ -65,6 +65,7 @@ class AssistantKnowledgeImportJobRow(Base):
|
||||
|
||||
class AssistantKnowledgeWikiPageRow(Base):
|
||||
__tablename__ = "assistant_knowledge_wiki_pages"
|
||||
metadata_json: Mapped[dict | None] = mapped_column(PortableJSON(), default=dict, nullable=True)
|
||||
__table_args__ = (UniqueConstraint("base_id", "slug", name="uq_assistant_wiki_slug"),)
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True)
|
||||
base_id: Mapped[str] = mapped_column(String(36), index=True)
|
||||
|
||||
+136
-61
@@ -2,6 +2,7 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import binascii
|
||||
import difflib
|
||||
@@ -60,6 +61,12 @@ def _dict(row: Any) -> dict[str, Any]:
|
||||
data["base_kind"] = data.get("base_kind") or "custom"
|
||||
data["is_global"] = data["base_kind"] == "global"
|
||||
data["description"] = data.get("description") or ""
|
||||
if isinstance(row, AssistantKnowledgeWikiPageRow):
|
||||
metadata = data.get("metadata") or {}
|
||||
for key in ("aliases", "tags", "category_path", "parent_slug", "wiki_path", "in_links", "out_links", "source_refs", "origins"):
|
||||
data[key] = metadata.get(key, [] if key not in {"parent_slug", "wiki_path"} else None)
|
||||
data["tags"] = metadata.get("tags") or (metadata.get("page_metadata") or {}).get("tags") or []
|
||||
data["content"] = data.get("content_markdown", "")
|
||||
return data
|
||||
|
||||
|
||||
@@ -95,7 +102,11 @@ def _vector_blob_from_payload(raw: dict[str, Any]) -> tuple[bytes, int] | None:
|
||||
try:
|
||||
decoded = base64.b64decode(str(blob64), validate=True)
|
||||
vector = decode_vector(decoded, int(dimensions))
|
||||
return encode_vector(vector)
|
||||
# Our packages contain normalized float32 bytes. Preserve those
|
||||
# bytes exactly: another normalization causes round-trip drift.
|
||||
if not np.isclose(np.linalg.norm(vector), 1.0, rtol=1e-5, atol=1e-6):
|
||||
return None
|
||||
return decoded, int(dimensions)
|
||||
except (ValueError, TypeError, binascii.Error):
|
||||
return None
|
||||
value = _first_vector_value(raw)
|
||||
@@ -115,6 +126,66 @@ class AssistantKnowledgeRepository:
|
||||
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
|
||||
self._sf = session_factory
|
||||
|
||||
async def iter_export_records(self, base_id: str):
|
||||
"""Export current Wiki articles and their original vectors, without encoding."""
|
||||
base = await self.get_base(base_id)
|
||||
if not base:
|
||||
raise ValueError("助手知识库不存在")
|
||||
yield "source", {"package_version": 1, "provider": "assistant", "type": "assistant", "knowledge_base_id": base_id, "knowledge_base_name": base["name"], "export_format": "assistant-knowledge-jsonl"}
|
||||
async with self._sf() as session:
|
||||
pages = select(AssistantKnowledgeWikiPageRow).where(AssistantKnowledgeWikiPageRow.base_id == base_id, AssistantKnowledgeWikiPageRow.status == "active", AssistantKnowledgeWikiPageRow.page_type != "chunk")
|
||||
stream = await session.stream_scalars(pages.execution_options(yield_per=100))
|
||||
async for page in stream:
|
||||
data = _dict(page)
|
||||
yield "wiki_page", {**(page.metadata_json or {}), **{k: data[k] for k in ("slug", "title", "page_type", "summary", "content")}, "status": "published"}
|
||||
vectors = (
|
||||
select(AssistantKnowledgeVectorRow, AssistantKnowledgeWikiPageRow.slug)
|
||||
.join(AssistantKnowledgeWikiPageRow, AssistantKnowledgeWikiPageRow.id == AssistantKnowledgeVectorRow.page_id)
|
||||
.join(AssistantKnowledgeWikiRevisionRow, AssistantKnowledgeWikiRevisionRow.id == AssistantKnowledgeWikiPageRow.current_revision_id)
|
||||
.where(
|
||||
AssistantKnowledgeVectorRow.base_id == base_id,
|
||||
AssistantKnowledgeWikiPageRow.status == "active",
|
||||
AssistantKnowledgeWikiPageRow.page_type != "chunk",
|
||||
AssistantKnowledgeVectorRow.import_job_id == AssistantKnowledgeWikiRevisionRow.import_job_id,
|
||||
)
|
||||
)
|
||||
stream = await session.stream(vectors.execution_options(yield_per=100))
|
||||
async for vector, slug in stream:
|
||||
yield (
|
||||
"vector",
|
||||
{
|
||||
"wiki_slug": slug,
|
||||
"section_index": vector.section_index,
|
||||
"heading": vector.heading,
|
||||
"section_content": vector.section_content,
|
||||
"content_hash": vector.content_hash,
|
||||
"embedding_model": vector.embedding_model,
|
||||
"embedding_fingerprint": vector.embedding_fingerprint,
|
||||
"embedding_dimensions": vector.embedding_dimensions,
|
||||
"vector_blob_base64": base64.b64encode(vector.vector_blob).decode("ascii"),
|
||||
},
|
||||
)
|
||||
stream = await session.stream(
|
||||
select(AssistantKnowledgeEntityRow, AssistantKnowledgeWikiPageRow.slug)
|
||||
.outerjoin(AssistantKnowledgeWikiPageRow, AssistantKnowledgeWikiPageRow.id == AssistantKnowledgeEntityRow.wiki_page_id)
|
||||
.where(AssistantKnowledgeEntityRow.base_id == base_id, AssistantKnowledgeEntityRow.status == "active")
|
||||
.execution_options(yield_per=100)
|
||||
)
|
||||
async for entity, slug in stream:
|
||||
yield "entity", {"id": entity.id, "name": entity.display_name, "type": entity.entity_type, "aliases": entity.aliases_json or [], "attributes": entity.attributes_json or {}, "source_wiki_slug": slug}
|
||||
stream = await session.stream(
|
||||
select(AssistantKnowledgeRelationRow, AssistantKnowledgeWikiPageRow.slug)
|
||||
.join(AssistantKnowledgeEntityRow, AssistantKnowledgeEntityRow.id == AssistantKnowledgeRelationRow.source_entity_id)
|
||||
.outerjoin(AssistantKnowledgeWikiPageRow, AssistantKnowledgeWikiPageRow.id == AssistantKnowledgeEntityRow.wiki_page_id)
|
||||
.where(AssistantKnowledgeRelationRow.base_id == base_id, AssistantKnowledgeRelationRow.status == "active")
|
||||
.execution_options(yield_per=100)
|
||||
)
|
||||
async for relation, slug in stream:
|
||||
yield (
|
||||
"relation",
|
||||
{"source_entity_id": relation.source_entity_id, "target_entity_id": relation.target_entity_id, "predicate": relation.predicate, "description": relation.description, "source_wiki_slug": slug},
|
||||
)
|
||||
|
||||
async def _base_with_stats(
|
||||
self,
|
||||
session: AsyncSession,
|
||||
@@ -138,6 +209,7 @@ class AssistantKnowledgeRepository:
|
||||
select(func.count(AssistantKnowledgeWikiPageRow.id)).where(
|
||||
AssistantKnowledgeWikiPageRow.base_id == row.id,
|
||||
AssistantKnowledgeWikiPageRow.status == "active",
|
||||
AssistantKnowledgeWikiPageRow.page_type != "chunk",
|
||||
)
|
||||
)
|
||||
).scalar_one()
|
||||
@@ -195,14 +267,7 @@ class AssistantKnowledgeRepository:
|
||||
# Backward compatibility: earlier builds had a single active row
|
||||
# and no ``base_kind``. Promote the first surviving base into the
|
||||
# global full-library aggregation base.
|
||||
row = (
|
||||
await session.execute(
|
||||
select(AssistantKnowledgeBaseRow)
|
||||
.where(AssistantKnowledgeBaseRow.status != "deleted")
|
||||
.order_by(AssistantKnowledgeBaseRow.created_at.asc())
|
||||
.limit(1)
|
||||
)
|
||||
).scalar_one_or_none()
|
||||
row = (await session.execute(select(AssistantKnowledgeBaseRow).where(AssistantKnowledgeBaseRow.status != "deleted").order_by(AssistantKnowledgeBaseRow.created_at.asc()).limit(1))).scalar_one_or_none()
|
||||
if row is not None:
|
||||
row.base_kind = "global"
|
||||
if not row.description:
|
||||
@@ -655,7 +720,7 @@ class AssistantKnowledgeRepository:
|
||||
|
||||
entity_map: dict[str, str] = {}
|
||||
entity_rows: dict[str, tuple[AssistantKnowledgeEntityRow, dict[str, Any]]] = {}
|
||||
raw_entities = list(package.get("entities") or [])
|
||||
raw_entities = package.get("entities") or []
|
||||
for raw in raw_entities:
|
||||
display_name = str(raw.get("name") or raw.get("display_name") or "").strip()
|
||||
entity_type = str(raw.get("type") or raw.get("entity_type") or "concept")
|
||||
@@ -698,7 +763,7 @@ class AssistantKnowledgeRepository:
|
||||
)
|
||||
)
|
||||
|
||||
raw_relations = list(package.get("relations") or [])
|
||||
raw_relations = package.get("relations") or []
|
||||
relation_rows: dict[str, tuple[AssistantKnowledgeRelationRow, dict[str, Any]]] = {}
|
||||
for raw in raw_relations:
|
||||
source_entity_id = entity_map.get(str(raw.get("source_entity_id") or raw.get("source_local_id") or raw.get("source")))
|
||||
@@ -741,6 +806,10 @@ class AssistantKnowledgeRepository:
|
||||
)
|
||||
|
||||
for entity, raw in entity_rows.values():
|
||||
# Provider Wiki graph nodes already have real articles. Do not
|
||||
# turn each node into a second, empty 'entity' article.
|
||||
if raw.get("source_wiki_slug"):
|
||||
continue
|
||||
aliases = list(dict.fromkeys([*list(entity.aliases_json or []), *list(raw.get("aliases") or [])]))
|
||||
entity.aliases_json = aliases
|
||||
content = "\n".join(
|
||||
@@ -775,6 +844,8 @@ class AssistantKnowledgeRepository:
|
||||
entity.wiki_page_id = entity_page.id
|
||||
|
||||
for relation, raw in relation_rows.values():
|
||||
if raw.get("source_wiki_slug"):
|
||||
continue
|
||||
source_entity = await session.get(AssistantKnowledgeEntityRow, relation.source_entity_id)
|
||||
target_entity = await session.get(AssistantKnowledgeEntityRow, relation.target_entity_id)
|
||||
source_name_display = source_entity.display_name if source_entity else relation.source_entity_id
|
||||
@@ -796,7 +867,7 @@ class AssistantKnowledgeRepository:
|
||||
)
|
||||
relation.wiki_page_id = relation_page.id
|
||||
|
||||
pages = list(package.get("wiki_pages") or [])
|
||||
pages = package.get("wiki_pages") or []
|
||||
if not pages and package.get("wiki"):
|
||||
wiki = package["wiki"]
|
||||
page = wiki.get("skill_page") if isinstance(wiki, dict) else None
|
||||
@@ -807,7 +878,7 @@ class AssistantKnowledgeRepository:
|
||||
continue
|
||||
slug = str(raw.get("slug") or f"assistant/sources/{source.id}").strip("/")
|
||||
title = str(raw.get("title") or source_name)
|
||||
content = str(raw.get("content") or raw.get("content_markdown") or "")
|
||||
content = str(raw.get("content") or raw.get("content_md") or raw.get("content_markdown") or "")
|
||||
summary = str(raw.get("summary") or "")
|
||||
stmt = select(AssistantKnowledgeWikiPageRow).where(AssistantKnowledgeWikiPageRow.base_id == base["id"], AssistantKnowledgeWikiPageRow.slug == slug)
|
||||
page = (await session.execute(stmt)).scalar_one_or_none()
|
||||
@@ -850,32 +921,35 @@ class AssistantKnowledgeRepository:
|
||||
page.current_revision_id = revision.id
|
||||
page.status = "active"
|
||||
page.updated_at = now
|
||||
metadata = {key: raw[key] for key in ("aliases", "tags", "page_metadata", "category_path", "parent_slug", "wiki_path", "in_links", "out_links", "source_refs", "chunk_refs", "version") if key in raw}
|
||||
source_meta = package.get("source") or {}
|
||||
origin_mapping = mapping_id
|
||||
if not origin_mapping and (source_meta.get("provider") or source_meta.get("type")) == "weknora":
|
||||
origin_mapping = source_meta.get("knowledge_base_id")
|
||||
origins = [*((page.metadata_json or {}).get("origins") or []), *(raw.get("origins") or [])]
|
||||
origin = {"mapping_id": origin_mapping, "source_key": source_key, "source_name": source_name, "slug": slug}
|
||||
origins = [value for value in origins if isinstance(value, dict) and value.get("source_key") != source_key]
|
||||
metadata["origins"] = list({(v.get("mapping_id"), v.get("source_key"), v.get("slug")): v for v in [*origins, origin]}.values())
|
||||
page.metadata_json = metadata
|
||||
page_ids_by_slug[slug] = page.id
|
||||
for entity, raw_entity in entity_rows.values():
|
||||
if raw_entity.get("source_wiki_slug") == slug:
|
||||
entity.wiki_page_id = page.id
|
||||
if len(page_ids_by_slug) % 50 == 0:
|
||||
await session.flush()
|
||||
|
||||
chunks = list(package.get("chunks") or [])
|
||||
chunks = package.get("chunks") or []
|
||||
chunk_pages = 0
|
||||
imported_vectors = 0
|
||||
for index, raw in enumerate(chunks):
|
||||
if not isinstance(raw, dict):
|
||||
continue
|
||||
content = str(
|
||||
raw.get("content")
|
||||
or raw.get("section_content")
|
||||
or raw.get("text")
|
||||
or raw.get("chunk_content")
|
||||
or ""
|
||||
).strip()
|
||||
content = str(raw.get("content") or raw.get("section_content") or raw.get("text") or raw.get("chunk_content") or "").strip()
|
||||
if not content:
|
||||
continue
|
||||
raw_chunk_id = str(raw.get("id") or raw.get("chunk_id") or raw.get("remote_id") or f"{source.id}:{index}")
|
||||
document_id = str(raw.get("source_document_id") or raw.get("knowledge_id") or raw.get("document_id") or "")
|
||||
title = str(
|
||||
raw.get("title")
|
||||
or raw.get("knowledge_title")
|
||||
or raw.get("filename")
|
||||
or raw.get("knowledge_filename")
|
||||
or f"{source_name} 片段 {index + 1}"
|
||||
)
|
||||
title = str(raw.get("title") or raw.get("knowledge_title") or raw.get("filename") or raw.get("knowledge_filename") or f"{source_name} 片段 {index + 1}")
|
||||
heading = str(raw.get("heading") or raw.get("section_title") or "")
|
||||
summary = content[:240]
|
||||
page_content = "\n".join(
|
||||
@@ -919,7 +993,7 @@ class AssistantKnowledgeRepository:
|
||||
import_job_id=job.id,
|
||||
source_vector_id=str(raw.get("vector_id") or raw.get("embedding_id") or raw_chunk_id),
|
||||
source_chunk_id=raw_chunk_id,
|
||||
section_index=int(raw.get("section_index") or raw.get("chunk_index") or index),
|
||||
section_index=int(next((raw[key] for key in ("section_index", "chunk_index") if raw.get(key) is not None), index)),
|
||||
heading=heading or None,
|
||||
section_content=content,
|
||||
content_hash=str(raw.get("content_hash") or _content_hash(content)),
|
||||
@@ -933,19 +1007,19 @@ class AssistantKnowledgeRepository:
|
||||
imported_vectors += 1
|
||||
chunk_pages += 1
|
||||
|
||||
for index, raw in enumerate(list(package.get("vectors") or [])):
|
||||
for index, raw in enumerate(package.get("vectors") or []):
|
||||
if not isinstance(raw, dict):
|
||||
continue
|
||||
raise ValueError("向量记录必须为对象")
|
||||
slug = str(raw.get("wiki_slug") or raw.get("page_slug") or raw.get("slug") or "").strip("/")
|
||||
page_id = page_ids_by_slug.get(slug)
|
||||
if not page_id:
|
||||
continue
|
||||
raise ValueError(f"向量引用不存在的 Wiki：{slug}")
|
||||
content = str(raw.get("section_content") or raw.get("content") or raw.get("text") or "").strip()
|
||||
if not content:
|
||||
continue
|
||||
vector_blob = _vector_blob_from_payload(raw)
|
||||
if vector_blob is None:
|
||||
continue
|
||||
raise ValueError(f"Wiki 向量格式无效：{slug}")
|
||||
blob, dimensions = vector_blob
|
||||
raw_chunk_id = str(raw.get("source_chunk_id") or raw.get("chunk_id") or "")
|
||||
session.add(
|
||||
@@ -957,7 +1031,7 @@ class AssistantKnowledgeRepository:
|
||||
import_job_id=job.id,
|
||||
source_vector_id=str(raw.get("id") or raw.get("vector_id") or f"{slug}:{index}"),
|
||||
source_chunk_id=raw_chunk_id or None,
|
||||
section_index=int(raw.get("section_index") or index),
|
||||
section_index=int(raw["section_index"]) if raw.get("section_index") is not None else index,
|
||||
heading=str(raw.get("heading") or "") or None,
|
||||
section_content=content,
|
||||
content_hash=str(raw.get("content_hash") or _content_hash(content)),
|
||||
@@ -969,9 +1043,11 @@ class AssistantKnowledgeRepository:
|
||||
)
|
||||
)
|
||||
imported_vectors += 1
|
||||
if imported_vectors % 50 == 0:
|
||||
await session.flush()
|
||||
|
||||
counts = {
|
||||
"wiki_pages": len(pages) + len(entity_rows) + len(relation_rows),
|
||||
"wiki_pages": len(pages) + sum(not raw.get("source_wiki_slug") for _, raw in entity_rows.values()) + sum(not raw.get("source_wiki_slug") for _, raw in relation_rows.values()),
|
||||
"documents": len(package.get("documents") or []),
|
||||
"chunks": len(chunks),
|
||||
"chunk_pages": chunk_pages,
|
||||
@@ -1008,6 +1084,7 @@ class AssistantKnowledgeRepository:
|
||||
query: str | None = None,
|
||||
page_type: str | None = None,
|
||||
limit: int = 100,
|
||||
offset: int = 0,
|
||||
include_chunks: bool = False,
|
||||
) -> list[dict[str, Any]]:
|
||||
base = await self.get_base(base_id)
|
||||
@@ -1024,7 +1101,7 @@ class AssistantKnowledgeRepository:
|
||||
stmt = stmt.where(or_(AssistantKnowledgeWikiPageRow.title.ilike(needle), AssistantKnowledgeWikiPageRow.search_text.ilike(needle)))
|
||||
if page_type:
|
||||
stmt = stmt.where(AssistantKnowledgeWikiPageRow.page_type == page_type)
|
||||
stmt = stmt.order_by(AssistantKnowledgeWikiPageRow.updated_at.desc()).limit(limit)
|
||||
stmt = stmt.order_by(AssistantKnowledgeWikiPageRow.slug, AssistantKnowledgeWikiPageRow.id).offset(offset).limit(limit)
|
||||
async with self._sf() as session:
|
||||
return [_dict(row) for row in (await session.execute(stmt)).scalars()]
|
||||
|
||||
@@ -1070,10 +1147,7 @@ class AssistantKnowledgeRepository:
|
||||
or_(
|
||||
AssistantKnowledgeWikiPageRow.title.ilike(needle),
|
||||
AssistantKnowledgeWikiPageRow.search_text.ilike(needle),
|
||||
*[
|
||||
AssistantKnowledgeWikiPageRow.search_text.ilike(f"%{token}%")
|
||||
for token in tokens[:8]
|
||||
],
|
||||
*[AssistantKnowledgeWikiPageRow.search_text.ilike(f"%{token}%") for token in tokens[:8]],
|
||||
)
|
||||
)
|
||||
stmt = stmt.order_by(AssistantKnowledgeWikiPageRow.updated_at.desc()).limit(max(limit * 8, limit))
|
||||
@@ -1098,6 +1172,7 @@ class AssistantKnowledgeRepository:
|
||||
results = rows[:limit]
|
||||
for row in results:
|
||||
row["score"] = score(row)
|
||||
row["retrieval_mode"] = "keyword"
|
||||
return results
|
||||
|
||||
async def _search_pages_by_vector(
|
||||
@@ -1119,31 +1194,36 @@ class AssistantKnowledgeRepository:
|
||||
AssistantKnowledgeWikiPageRow,
|
||||
AssistantKnowledgeWikiPageRow.id == AssistantKnowledgeVectorRow.page_id,
|
||||
)
|
||||
.join(AssistantKnowledgeWikiRevisionRow, AssistantKnowledgeWikiRevisionRow.id == AssistantKnowledgeWikiPageRow.current_revision_id)
|
||||
.where(
|
||||
AssistantKnowledgeVectorRow.base_id.in_(base_ids),
|
||||
AssistantKnowledgeVectorRow.import_job_id == AssistantKnowledgeWikiRevisionRow.import_job_id,
|
||||
AssistantKnowledgeVectorRow.embedding_dimensions == dimensions,
|
||||
AssistantKnowledgeVectorRow.embedding_fingerprint == embedding_client.fingerprint,
|
||||
AssistantKnowledgeWikiPageRow.status == "active",
|
||||
)
|
||||
.order_by(AssistantKnowledgeVectorRow.created_at.desc())
|
||||
)
|
||||
if not include_chunks:
|
||||
stmt = stmt.where(AssistantKnowledgeWikiPageRow.page_type != "chunk")
|
||||
async with self._sf() as session:
|
||||
rows = list((await session.execute(stmt)).all())
|
||||
if not rows:
|
||||
return []
|
||||
tokens = [token for token in query.strip().lower().split() if token]
|
||||
ranked: list[tuple[float, AssistantKnowledgeVectorRow, AssistantKnowledgeWikiPageRow]] = []
|
||||
for vector, page in rows:
|
||||
try:
|
||||
stored = decode_vector(vector.vector_blob, int(vector.embedding_dimensions))
|
||||
except ValueError:
|
||||
continue
|
||||
score = float(np.dot(query_vector, stored))
|
||||
if tokens:
|
||||
haystack = f"{page.title}\n{page.summary}\n{vector.heading or ''}\n{vector.section_content}".lower()
|
||||
score += min(sum(1 for token in tokens if token in haystack), 4) * 0.02
|
||||
ranked.append((score, vector, page))
|
||||
# Bounded scan: keep only the best section per page, and at most limit
|
||||
# candidates. Never load a multi-GB vector library into one matrix.
|
||||
async with self._sf() as session:
|
||||
stream = await session.stream(stmt.execution_options(yield_per=256))
|
||||
async for batch in stream.partitions(256):
|
||||
|
||||
def score_batch():
|
||||
values = []
|
||||
for vector, page in batch:
|
||||
values.append((float(np.dot(query_vector, decode_vector(vector.vector_blob, dimensions))), vector, page))
|
||||
return values
|
||||
|
||||
candidates = ranked + await asyncio.to_thread(score_batch)
|
||||
best = {}
|
||||
for item in sorted(candidates, key=lambda item: item[0], reverse=True):
|
||||
best.setdefault(item[2].id, item)
|
||||
ranked = list(best.values())[:limit]
|
||||
ranked.sort(key=lambda item: item[0], reverse=True)
|
||||
|
||||
results: list[dict[str, Any]] = []
|
||||
@@ -1320,12 +1400,7 @@ class AssistantKnowledgeRepository:
|
||||
if base is None:
|
||||
return []
|
||||
async with self._sf() as session:
|
||||
stmt = (
|
||||
select(AssistantKnowledgeImportJobRow)
|
||||
.where(AssistantKnowledgeImportJobRow.base_id == base["id"])
|
||||
.order_by(AssistantKnowledgeImportJobRow.created_at.desc())
|
||||
.limit(limit)
|
||||
)
|
||||
stmt = select(AssistantKnowledgeImportJobRow).where(AssistantKnowledgeImportJobRow.base_id == base["id"]).order_by(AssistantKnowledgeImportJobRow.created_at.desc()).limit(limit)
|
||||
return [_dict(row) for row in (await session.execute(stmt)).scalars()]
|
||||
|
||||
async def get_job(self, job_id: str) -> dict[str, Any] | None:
|
||||
|
||||
+1
-1
@@ -24,7 +24,7 @@ class MemoryLlmWikiIndexStore(LlmWikiIndexStore):
|
||||
|
||||
async def get_page_by_slug_hash(self, mapping_id: str, slug_hash: str) -> dict[str, Any] | None:
|
||||
row = next((p for p in self.pages.values() if p["knowledge_base_mapping_id"] == mapping_id and p["slug_hash"] == slug_hash), None)
|
||||
return dict(row) if row else None
|
||||
return {**row, "vector_count": len(self.vectors.get(row["id"], []))} if row else None
|
||||
|
||||
async def get_page(self, mapping_id: str, slug: str) -> dict[str, Any] | None:
|
||||
row = next((p for p in self.pages.values() if p["knowledge_base_mapping_id"] == mapping_id and p["slug"] == slug), None)
|
||||
|
||||
+4
-3
@@ -41,10 +41,11 @@ class SqlLlmWikiIndexStore(LlmWikiIndexStore):
|
||||
return row
|
||||
|
||||
async def get_page_by_slug_hash(self, mapping_id: str, slug_hash: str) -> dict[str, Any] | None:
|
||||
stmt = select(LlmWikiWikiPageRow).where(LlmWikiWikiPageRow.knowledge_base_mapping_id == mapping_id, LlmWikiWikiPageRow.slug_hash == slug_hash)
|
||||
count = select(func.count(LlmWikiWikiVectorRow.id)).where(LlmWikiWikiVectorRow.page_id == LlmWikiWikiPageRow.id).correlate(LlmWikiWikiPageRow).scalar_subquery()
|
||||
stmt = select(LlmWikiWikiPageRow, count).where(LlmWikiWikiPageRow.knowledge_base_mapping_id == mapping_id, LlmWikiWikiPageRow.slug_hash == slug_hash)
|
||||
async with self._sf() as session:
|
||||
row = (await session.execute(stmt)).scalar_one_or_none()
|
||||
return _dict(row) if row else None
|
||||
result = (await session.execute(stmt)).one_or_none()
|
||||
return {**_dict(result[0]), "vector_count": result[1]} if result else None
|
||||
|
||||
async def get_page(self, mapping_id: str, slug: str) -> dict[str, Any] | None:
|
||||
stmt = select(LlmWikiWikiPageRow).where(LlmWikiWikiPageRow.knowledge_base_mapping_id == mapping_id, LlmWikiWikiPageRow.slug == slug)
|
||||
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
"""Preserve source Wiki metadata during package round trips."""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
from deerflow.persistence.types import PortableJSON
|
||||
|
||||
revision = "20260903_01"
|
||||
down_revision = ("20260902_01", "20260831_01")
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
columns = {col["name"] for col in sa.inspect(op.get_bind()).get_columns("assistant_knowledge_wiki_pages")}
|
||||
if "metadata_json" not in columns:
|
||||
op.add_column("assistant_knowledge_wiki_pages", sa.Column("metadata_json", PortableJSON(), nullable=True))
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_column("assistant_knowledge_wiki_pages", "metadata_json")
|
||||
@@ -31,6 +31,7 @@ dependencies = [
|
||||
postgres = ["deerflow-harness[postgres]"]
|
||||
mysql = ["deerflow-harness[mysql]"]
|
||||
gauss = ["deerflow-harness[gauss]"]
|
||||
local-embedding = ["fastembed>=0.7,<0.9"]
|
||||
|
||||
[dependency-groups]
|
||||
dev = [
|
||||
|
||||
@@ -13,6 +13,7 @@ from deerflow.persistence.base import Base
|
||||
|
||||
class FakeEmbeddingClient:
|
||||
dimensions = 3
|
||||
fingerprint = "test-bge-m3"
|
||||
|
||||
async def embed_query(self, _query: str) -> list[float]:
|
||||
return [1.0, 0.0, 0.0]
|
||||
@@ -154,6 +155,7 @@ async def test_imported_vectors_are_reused_for_assistant_search(
|
||||
"section_index": 0,
|
||||
"section_content": "张三负责向量化检索。",
|
||||
"embedding_model": "bge-embedding-m3",
|
||||
"embedding_fingerprint": "test-bge-m3",
|
||||
"embedding_dimensions": 3,
|
||||
"vector": [1.0, 0.0, 0.0],
|
||||
},
|
||||
@@ -162,6 +164,7 @@ async def test_imported_vectors_are_reused_for_assistant_search(
|
||||
"section_index": 1,
|
||||
"section_content": "无关片段。",
|
||||
"embedding_model": "bge-embedding-m3",
|
||||
"embedding_fingerprint": "test-bge-m3",
|
||||
"embedding_dimensions": 3,
|
||||
"vector": [0.0, 1.0, 0.0],
|
||||
},
|
||||
|
||||
@@ -0,0 +1,180 @@
|
||||
"""Ordinary knowledge-base editing and remote-deletion reconciliation."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from types import SimpleNamespace
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from fastapi import FastAPI
|
||||
|
||||
from deerflow.integrations.weknora.client import WeKnoraClient
|
||||
from deerflow.persistence.llmwiki import MemoryLlmWikiStore
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def knowledge_api(monkeypatch: pytest.MonkeyPatch):
|
||||
from app.gateway.routers import llmwiki
|
||||
|
||||
store = MemoryLlmWikiStore()
|
||||
row = await store.create_mapping(weknora_id="remote-one", owner_user_id="owner", name="原名称", description="原描述", kb_type="document")
|
||||
await store.request_publish(row["id"], "owner", is_admin=False)
|
||||
state = SimpleNamespace(
|
||||
store=store,
|
||||
mapping_id=row["id"],
|
||||
remote={"remote-one": {"id": "remote-one", "name": "原名称", "description": "原描述", "knowledge_count": 3}},
|
||||
user_id="owner",
|
||||
is_admin=False,
|
||||
upstream_status=200,
|
||||
writes=[],
|
||||
)
|
||||
|
||||
async def upstream(request: httpx.Request) -> httpx.Response:
|
||||
if state.upstream_status != 200:
|
||||
return httpx.Response(state.upstream_status, json={"message": "知识服务暂不可用"})
|
||||
if request.url.path == "/api/v1/knowledge-bases":
|
||||
assert request.method == "GET"
|
||||
return httpx.Response(200, json={"data": list(state.remote.values())})
|
||||
remote_id = request.url.path.rsplit("/", 1)[-1]
|
||||
if remote_id not in state.remote:
|
||||
return httpx.Response(404)
|
||||
if request.method == "PUT":
|
||||
changes = json.loads(request.content)
|
||||
state.writes.append(changes)
|
||||
state.remote[remote_id].update(changes)
|
||||
else:
|
||||
assert request.method == "GET"
|
||||
return httpx.Response(200, json={"data": state.remote[remote_id]})
|
||||
|
||||
async def actor(_: object) -> tuple[str, bool]:
|
||||
return state.user_id, state.is_admin
|
||||
|
||||
app = FastAPI()
|
||||
app.include_router(llmwiki.router)
|
||||
monkeypatch.setattr(llmwiki, "_actor", actor)
|
||||
monkeypatch.setattr(llmwiki, "_store", lambda _: store)
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(upstream)) as remote_client:
|
||||
client = WeKnoraClient(base_url="http://weknora.test", http_client=remote_client)
|
||||
monkeypatch.setattr(llmwiki, "_client_or_503", lambda: client)
|
||||
async with httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url="http://gateway.test") as api:
|
||||
state.api = api
|
||||
yield state
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("scope", ["personal", "public", "all"])
|
||||
@pytest.mark.parametrize("is_admin", [False, True])
|
||||
async def test_lists_hide_remote_deleted_bases_without_deleting_mappings(knowledge_api, scope: str, is_admin: bool) -> None:
|
||||
state = knowledge_api
|
||||
state.is_admin = is_admin
|
||||
live = await state.store.create_mapping(weknora_id="remote-live", owner_user_id="owner", name="保留知识库", description="", kb_type="document")
|
||||
await state.store.request_publish(live["id"], "owner", is_admin=False)
|
||||
state.remote["remote-live"] = {"id": "remote-live", "name": "保留知识库"}
|
||||
deleted = state.remote.pop("remote-one")
|
||||
|
||||
response = await state.api.get("/api/llmwiki/knowledge-bases", params={"scope": scope})
|
||||
|
||||
assert response.status_code == 200
|
||||
assert [item["id"] for item in response.json()["knowledge_bases"]] == [live["id"]]
|
||||
assert await state.store.get_by_weknora_id("remote-one") is not None
|
||||
|
||||
state.remote.clear()
|
||||
response = await state.api.get("/api/llmwiki/knowledge-bases", params={"scope": scope})
|
||||
assert response.json()["knowledge_bases"] == []
|
||||
|
||||
state.remote["remote-one"] = deleted
|
||||
response = await state.api.get("/api/llmwiki/knowledge-bases", params={"scope": scope})
|
||||
assert [item["id"] for item in response.json()["knowledge_bases"]] == [state.mapping_id]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("status", [502, 504])
|
||||
async def test_list_upstream_failure_is_not_treated_as_remote_deletion(knowledge_api, status: int) -> None:
|
||||
state = knowledge_api
|
||||
state.upstream_status = status
|
||||
|
||||
response = await state.api.get("/api/llmwiki/knowledge-bases", params={"scope": "personal"})
|
||||
|
||||
assert response.status_code >= 500
|
||||
assert await state.store.get_by_weknora_id("remote-one") is not None
|
||||
state.upstream_status = 200
|
||||
response = await state.api.get("/api/llmwiki/knowledge-bases", params={"scope": "personal"})
|
||||
assert [item["id"] for item in response.json()["knowledge_bases"]] == [state.mapping_id]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
("changes", "expected_name", "expected_description"),
|
||||
[
|
||||
({"name": " 新名称 ", "description": "新描述"}, "新名称", "新描述"),
|
||||
({"name": "仅修改名称"}, "仅修改名称", "原描述"),
|
||||
({"description": "仅修改描述"}, "原名称", "仅修改描述"),
|
||||
({"name": "清空描述", "description": ""}, "清空描述", ""),
|
||||
],
|
||||
)
|
||||
async def test_edit_synchronizes_remote_and_local_metadata(knowledge_api, changes, expected_name: str, expected_description: str) -> None:
|
||||
state = knowledge_api
|
||||
response = await state.api.patch(f"/api/llmwiki/knowledge-bases/{state.mapping_id}", json=changes)
|
||||
|
||||
assert response.status_code == 200
|
||||
item = response.json()
|
||||
assert (item["name"], item["description"]) == (expected_name, expected_description)
|
||||
assert item["remote_status"] == "available"
|
||||
assert item["knowledge_count"] == 3
|
||||
assert item["publication_status"] == "published"
|
||||
local = await state.store.get_by_weknora_id("remote-one")
|
||||
assert (local["name"], local["description"]) == (expected_name, expected_description)
|
||||
assert state.writes == [{**changes, "name": expected_name}]
|
||||
listing = await state.api.get("/api/llmwiki/knowledge-bases", params={"scope": "personal"})
|
||||
assert listing.json()["knowledge_bases"][0]["name"] == expected_name
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("name", ["", " \t ", "x" * 256, "对话沉淀", " 对话沉淀 "])
|
||||
async def test_edit_rejects_invalid_or_reserved_names(knowledge_api, name: str) -> None:
|
||||
state = knowledge_api
|
||||
response = await state.api.patch(f"/api/llmwiki/knowledge-bases/{state.mapping_id}", json={"name": name})
|
||||
|
||||
assert response.status_code in {409, 422}
|
||||
assert state.writes == []
|
||||
assert (await state.store.get_by_weknora_id("remote-one"))["name"] == "原名称"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("is_admin", [False, True])
|
||||
async def test_only_owner_or_admin_can_edit_a_public_base(knowledge_api, is_admin: bool) -> None:
|
||||
state = knowledge_api
|
||||
state.user_id = "other-user"
|
||||
state.is_admin = is_admin
|
||||
response = await state.api.patch(f"/api/llmwiki/knowledge-bases/{state.mapping_id}", json={"name": "新名称"})
|
||||
|
||||
assert response.status_code == (200 if is_admin else 404)
|
||||
assert len(state.writes) == (1 if is_admin else 0)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_admin_cannot_edit_system_conversation_deposit(knowledge_api) -> None:
|
||||
state = knowledge_api
|
||||
state.is_admin = True
|
||||
await state.store.adopt_conversation_deposit_mapping(state.mapping_id, name="对话沉淀", description="系统知识库")
|
||||
response = await state.api.patch(f"/api/llmwiki/knowledge-bases/{state.mapping_id}", json={"name": "新名称"})
|
||||
|
||||
assert response.status_code == 403
|
||||
assert state.writes == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("status", [404, 502, 504])
|
||||
async def test_failed_remote_edit_leaves_local_metadata_unchanged(knowledge_api, status: int) -> None:
|
||||
state = knowledge_api
|
||||
state.upstream_status = status
|
||||
response = await state.api.patch(f"/api/llmwiki/knowledge-bases/{state.mapping_id}", json={"name": "新名称", "description": "新描述"})
|
||||
|
||||
if status == 404:
|
||||
assert response.status_code == 404
|
||||
else:
|
||||
assert response.status_code >= 500
|
||||
local = await state.store.get_by_weknora_id("remote-one")
|
||||
assert (local["name"], local["description"]) == ("原名称", "原描述")
|
||||
@@ -0,0 +1,207 @@
|
||||
import base64
|
||||
import sys
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
|
||||
|
||||
from app.gateway.routers import wiki_packages
|
||||
from deerflow.assistant_knowledge.archive import PackageArchive
|
||||
from deerflow.assistant_knowledge.package_io import write_package
|
||||
from deerflow.config.llmwiki_config import LocalWikiEmbeddingConfig
|
||||
from deerflow.integrations.weknora.local_index.chunking import chunk_wiki_page
|
||||
from deerflow.integrations.weknora.local_index.embedding import StrictWikiEmbeddingClient
|
||||
from deerflow.integrations.weknora.local_index.normalize import normalize_wiki_page
|
||||
from deerflow.integrations.weknora.local_index.vector_codec import encode_vector
|
||||
from deerflow.persistence.assistant_knowledge import AssistantKnowledgeRepository
|
||||
from deerflow.persistence.base import Base
|
||||
from deerflow.persistence.llmwiki_index.memory import MemoryLlmWikiIndexStore
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_metadata_vectors_and_links_survive_roundtrip_without_encoding(tmp_path, monkeypatch):
|
||||
engine = create_async_engine("sqlite+aiosqlite:///:memory:")
|
||||
async with engine.begin() as conn:
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
store = AssistantKnowledgeRepository(async_sessionmaker(engine, expire_on_commit=False))
|
||||
blob, _ = encode_vector([0.3, 0.5, 0.7])
|
||||
pages = [
|
||||
{
|
||||
"slug": "concept/cooling",
|
||||
"title": "散热",
|
||||
"page_type": "concept",
|
||||
"content": "散热管理用于降低设备温度。参见 [[entity/fan]]",
|
||||
"summary": "设备散热",
|
||||
"tags": ["设备", "维护"],
|
||||
"category_path": ["运维", "硬件"],
|
||||
"aliases": ["降温"],
|
||||
"out_links": ["entity/fan"],
|
||||
"in_links": [],
|
||||
},
|
||||
{"slug": "entity/fan", "title": "风扇", "page_type": "entity", "content": "风扇通过气流帮助散热，参见 [[concept/cooling]]。", "aliases": [], "out_links": ["concept/cooling"], "in_links": ["concept/cooling"]},
|
||||
]
|
||||
vectors = [
|
||||
{
|
||||
"wiki_slug": p["slug"],
|
||||
"section_index": 0,
|
||||
"section_content": p["content"],
|
||||
"content_hash": "test",
|
||||
"embedding_model": "test-model",
|
||||
"embedding_fingerprint": "test-fp",
|
||||
"embedding_dimensions": 3,
|
||||
"vector_blob_base64": base64.b64encode(blob).decode(),
|
||||
}
|
||||
for p in pages
|
||||
]
|
||||
original = tmp_path / "original.jsonl"
|
||||
entities = [{"id": p["slug"], "name": p["title"], "type": p["page_type"], "source_wiki_slug": p["slug"]} for p in pages]
|
||||
relations = [{"source_entity_id": pages[0]["slug"], "target_entity_id": pages[1]["slug"], "source_wiki_slug": pages[0]["slug"], "predicate": "references"}]
|
||||
write_package(original, [
|
||||
("source", {"provider": "weknora", "package_version": 1}),
|
||||
*[("wiki_page", p) for p in pages],
|
||||
*[("vector", v) for v in vectors],
|
||||
*[("entity", e) for e in entities],
|
||||
*[("relation", r) for r in relations],
|
||||
("chunk", {"id": "raw-chunk", "content": "原始文档分段不是 Wiki 页面"}),
|
||||
])
|
||||
archive = PackageArchive(original)
|
||||
base = await store.initialize(created_by="admin")
|
||||
job = await store.import_package(package=archive, source_type="weknora", source_key="source", source_name="来源", trigger="test", created_by="admin")
|
||||
assert job["counts"]["vectors"] == 2
|
||||
assert job["counts"]["wiki_pages"] == 2
|
||||
assert job["counts"]["chunk_pages"] == 1
|
||||
imported = await store.get_page(pages[0]["slug"])
|
||||
for field in ("category_path", "tags", "aliases", "out_links"):
|
||||
assert imported[field] == pages[0][field]
|
||||
assert imported["content_markdown"] == pages[0]["content"]
|
||||
# The global base may receive the same Wiki through multiple libraries.
|
||||
# Only the current article's import supplies vectors; older contributions
|
||||
# remain auditable but must not duplicate/stale the restored index.
|
||||
await store.import_package(package=archive, source_type="weknora", source_key="second-source", source_name="第二来源", trigger="test", created_by="admin")
|
||||
exported = tmp_path / "export.jsonl"
|
||||
write_package(exported, [record async for record in store.iter_export_records(base["id"])])
|
||||
restored_archive = PackageArchive(exported)
|
||||
assert {v["vector_blob_base64"] for v in restored_archive["vectors"]} == {base64.b64encode(blob).decode()}
|
||||
assert sorted((v["wiki_slug"], v["section_index"]) for v in restored_archive["vectors"]) == sorted((v["wiki_slug"], v["section_index"]) for v in vectors)
|
||||
assert all(r["source_wiki_slug"] in {p["slug"] for p in pages} for r in restored_archive["relations"])
|
||||
index = MemoryLlmWikiIndexStore()
|
||||
received = []
|
||||
|
||||
class Remote:
|
||||
async def create_wiki_folder(self, kb, *, name, parent_id=""):
|
||||
return {"id": f"{parent_id}/{name}"}
|
||||
|
||||
async def create_wiki_page(self, kb, page):
|
||||
received.append(page)
|
||||
return page
|
||||
|
||||
async def rebuild_wiki_links(self, kb):
|
||||
pass
|
||||
|
||||
class Embedding:
|
||||
dimensions = 3
|
||||
fingerprint = "test-fp"
|
||||
|
||||
async def embed_query(self, query):
|
||||
raise AssertionError("Existing package must not be encoded")
|
||||
|
||||
app = FastAPI()
|
||||
app.state.llmwiki_index_store = index
|
||||
app.state.llmwiki_embedding = Embedding()
|
||||
app.state.llmwiki_vector_cache = None
|
||||
app.state.config = None
|
||||
monkeypatch.setattr(wiki_packages, "get_resolved_llmwiki_runtime", lambda c: None)
|
||||
monkeypatch.setattr(wiki_packages, "build_weknora_client", lambda c: Remote())
|
||||
monkeypatch.setattr(wiki_packages, "_package_root", lambda: tmp_path)
|
||||
from app.gateway import knowledge_transfer
|
||||
|
||||
async def sync(app, mapping):
|
||||
return {"status": "completed"}
|
||||
|
||||
monkeypatch.setattr(knowledge_transfer, "sync_to_global", sync)
|
||||
await wiki_packages.restore_package(app, {"id": "target", "weknora_id": "remote"}, "job-test", exported)
|
||||
for raw in received:
|
||||
assert raw["content"] == next(p["content"] for p in pages if p["slug"] == raw["slug"])
|
||||
local = await index.get_page("target", raw["slug"])
|
||||
assert local["content_hash"] == normalize_wiki_page("target", raw)["content_hash"]
|
||||
snapshot = await index.load_vector_snapshot("target", "test-fp")
|
||||
assert len(snapshot["rows"]) == 2
|
||||
assert all(r["vector_blob"] == blob for r in snapshot["rows"])
|
||||
|
||||
class SearchEmbedding:
|
||||
dimensions = 3
|
||||
fingerprint = "test-fp"
|
||||
|
||||
async def embed_query(self, query):
|
||||
return [1, 0, 0]
|
||||
|
||||
found = await store.search_pages(base_ids=[base["id"]], query="没有关键词重合的查询", embedding_client=SearchEmbedding())
|
||||
assert found and all(r["retrieval_mode"] == "vector" for r in found)
|
||||
SearchEmbedding.fingerprint = "different-model-same-dimension"
|
||||
assert await store.search_pages(base_ids=[base["id"]], query="没有关键词重合的查询", embedding_client=SearchEmbedding()) == []
|
||||
archive.close()
|
||||
restored_archive.close()
|
||||
await engine.dispose()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_local_encoder_is_bounded_cpu_and_does_not_need_http(monkeypatch):
|
||||
observed = {}
|
||||
|
||||
class LocalModel:
|
||||
@staticmethod
|
||||
def list_supported_models():
|
||||
return [{"model": "BAAI/bge-m3"}]
|
||||
|
||||
def __init__(self, **kwargs):
|
||||
observed.update(kwargs)
|
||||
|
||||
def embed(self, texts, *, batch_size):
|
||||
observed["batch_size"] = batch_size
|
||||
return [[1.0, *([0.0] * 1023)] for _ in texts]
|
||||
|
||||
monkeypatch.setitem(sys.modules, "fastembed", SimpleNamespace(TextEmbedding=LocalModel))
|
||||
config = LocalWikiEmbeddingConfig(provider="local", model="bge-embedding-m3", dimensions=1024, batch_size=2, local_threads=2)
|
||||
client = StrictWikiEmbeddingClient(config)
|
||||
assert len(await client.embed_query("本地查询")) == 1024
|
||||
assert observed["model_name"] == "BAAI/bge-m3"
|
||||
assert observed["providers"] == ["CPUExecutionProvider"]
|
||||
assert observed["batch_size"] == observed["threads"] == 2
|
||||
assert client._client is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_builtin_skill_without_ownership_record_can_be_distilled(monkeypatch, tmp_path):
|
||||
from starlette.requests import Request
|
||||
|
||||
from app.gateway.routers import skill_knowledge
|
||||
|
||||
async def admin(_):
|
||||
return "admin"
|
||||
|
||||
class Mappings:
|
||||
async def get_authorized(self, *args, **kwargs):
|
||||
return {"name": "测试库", "weknora_id": "remote"}
|
||||
|
||||
class Jobs:
|
||||
async def create_job(self, **kwargs):
|
||||
return kwargs
|
||||
|
||||
app = FastAPI()
|
||||
app.state.config = None
|
||||
app.state.llmwiki_store = Mappings()
|
||||
app.state.skill_knowledge_store = Jobs()
|
||||
# Deliberately no skill_store ownership row: built-ins live on disk.
|
||||
monkeypatch.setattr(skill_knowledge, "_require_admin", admin)
|
||||
monkeypatch.setattr(skill_knowledge, "_skill_directory", lambda *args: tmp_path)
|
||||
body = skill_knowledge.CreateSyncJobRequest(skill_names=["builtin"], targets=[{"target_type": "weknora", "target_mode": "wiki", "target_id": "mapping"}])
|
||||
job = await skill_knowledge.create_sync_job(Request({"type": "http", "app": app}), body)
|
||||
assert job["skill_names"] == ["builtin"]
|
||||
assert job["review_mode"] == "off"
|
||||
|
||||
|
||||
def test_short_wiki_summary_and_body_still_have_a_vector_section():
|
||||
sections = chunk_wiki_page({"title": "风扇", "summary": "通过气流帮助散热。", "content_md": "# 风扇\n用于降低设备温度。"}, max_chars=1000, overlap=100, min_chars=80, title_prefix_enabled=True)
|
||||
assert len(sections) == 1
|
||||
assert "降低设备温度" in sections[0].embedding_text
|
||||
Reference in new issue
Block a user