feat(knowledge): 支持文件导入沉淀知识，知识图谱可视化随主题切换
- 后端新增 /api/knowledge/ingest/file：上传 md/txt/pdf/word/ppt/excel， 转 Markdown 后走 wiki-capture LLM 蒸馏为陈述式知识笔记，失败回退逐字草稿 - service.capture_file + 文档专用蒸馏提示词，按主题拆分 1~4 页 - 知识图谱导出 HTML 支持 ?theme=light|dark，布局稳定后冻结， 按度数缩放节点，悬停/点击聚焦邻居 - 前端知识库页新增「导入文件」对话框与 ingestKnowledgeFile API - 智能体问答页与简介侧边栏 UI 优化；批量人物蒸馏脚本提取正文逻辑修正 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
吉鑫升andClaude Opus 4.8 committed
1 parent
17bb3715d1
commit
3845d5090b
14 files changed
+1040
-352
No files matched your search
@@ -250,7 +250,7 @@ FastAPI application on port 8001 with health check at `GET /health`. Set `GATEWA
|
||||
| **Scheduled Tasks** (`/api/scheduled-tasks`) | Per-user cron-like agent tasks. `GET/POST /` - list/create owned tasks; `PATCH/DELETE /{id}` - update/delete (owner); `POST /{id}/{pause,resume,trigger}`; `GET /{id}/runs`, `GET /runs/{rid}`, run-file preview/edit/download; `GET /runs/{rid}/messages` - agent messages for the run's underlying agent run (used to inspect a stuck/running run; authorized via `_load_run`); `GET /runs/{rid}/html` - generated HTML document for an `html_page` run; `DELETE /runs/{rid}` - owner-scoped, with an admin fallback so admins can delete any user's run; publish/subscribe under `/{id}/{publish,unpublish,subscribe}` and `GET /published`. **Admin (cross-user):** `GET /admin/all` - every task; `PATCH/DELETE /admin/{id}` - edit/delete any task; `POST /admin/{id}/trigger`; `GET /admin/{id}/runs`. Admin routes require `system_role == "admin"`. |
|
||||
| **HTML Page Favorites** (`/api/html-page-favorites`) | Bookmarked generated HTML pages, owner-scoped. `GET /` - list; `POST /` `{page_id, title?, description?, tags?}` - bookmark a `scheduled_html_pages` row (extracts a reusable style/layout summary via GLM5 at favorite time); a page can only be favorited once per user (`DuplicateFavoriteError` → HTTP 409). `GET/DELETE /{id}` - read/delete (the create-task reference picker deletes favorites here). A favorite can be selected as a style reference when creating a new `html_page` task. |
|
||||
| **Public Scheduled Tasks** (`/api/public/scheduled-tasks`) | **Unauthenticated** mirror for embedding. Adds `GET /runs/{rid}/html` and `GET /{task_id}/latest-html` for the `html_page` preview; the run result keeps `result.html` (only file paths are redacted). |
|
||||
| **Knowledge** (`/api/knowledge`) | Product knowledge base, **global/shared** (no user isolation; writes record `created_by`/`updated_by` for audit). `POST /ingest/thread/{thread_id}` - sediment one thread into a note (reads messages from the checkpointer, `threads:read` owner-checked); `POST /ingest/threads/batch` - batch-sediment the caller's recent threads (`skip_existing` via `.manifest.json` dedup). **Optional model selection**: both ingest endpoints accept `model_name` (overrides `knowledge.extract_model_name` for that capture; threaded through `KnowledgeService.capture_thread` → `extract_thread_llm_multi`). **Model-error surfacing**: when the chosen model produces nothing usable (timeout / API error / non-JSON) capture still **falls back to rule-based extraction and lands a draft** (`llm_fallback:true`); the single-thread (non-background) response and the batch response (`llm_fallback` count) carry this so the UI tells the user "已改用规则抽取". The "保存到知识库" chat button stays background/fire-and-forget (passes `model_name`, no per-save error toast); the batch-import dialog is synchronous and shows the fallback count. `POST /ingest/search` - sediment search/tool results as a reference note; `POST /notes` - manual note; `GET /notes` (keyword/tag/source_type/status + pagination); `GET/PUT/DELETE /notes/{id}` (DELETE is soft = archive, `?hard=true` removes the vault file); `POST /search` - `mode=keyword\|vector\|hybrid` (vector→keyword fallback when no embedding endpoint); `GET /notes/{id}/versions` - edit history; `GET /duplicates` + `POST /merge` - dedup; `POST /reindex` - rebuild vector index; `GET /graph` - product graph data (notes + entities); `POST /export/graph` + `GET /export/files/{graph.json\|graph.html\|graph.graphml\|cypher.txt}` - obsidian-wiki `wiki-export` artifacts (self-contained `graph.html` for iframe); `POST /resolve` `{targets:[…]}` + `GET /notes/{id}/wikilinks` - resolve `[[wikilink]]` targets to a note id / vault page / missing (powers clickable wikilinks); `GET /vault-page?path=` - read a raw vault meta/hub Markdown page (no DB note). Backed by the `deerflow.knowledge` module (see below). |
|
||||
| **Knowledge** (`/api/knowledge`) | Product knowledge base, **global/shared** (no user isolation; writes record `created_by`/`updated_by` for audit). `POST /ingest/thread/{thread_id}` - sediment one thread into a note (reads messages from the checkpointer, `threads:read` owner-checked); `POST /ingest/threads/batch` - batch-sediment the caller's recent threads (`skip_existing` via `.manifest.json` dedup). **Optional model selection**: both ingest endpoints accept `model_name` (overrides `knowledge.extract_model_name` for that capture; threaded through `KnowledgeService.capture_thread` → `extract_thread_llm_multi`). **Model-error surfacing**: when the chosen model produces nothing usable (timeout / API error / non-JSON) capture still **falls back to rule-based extraction and lands a draft** (`llm_fallback:true`); the single-thread (non-background) response and the batch response (`llm_fallback` count) carry this so the UI tells the user "已改用规则抽取". The "保存到知识库" chat button stays background/fire-and-forget (passes `model_name`, no per-save error toast); the batch-import dialog is synchronous and shows the fallback count. `POST /ingest/file` - **import an uploaded file** (multipart: `file` + optional `title`/`tags`/`status`/`model_name`) into wiki note(s): the file is converted to Markdown (md/txt read directly, pdf/word/ppt/excel via `convert_file_to_markdown`), distilled by the same wiki-capture LLM pipeline as thread capture (`source_type="file"`), and falls back to a verbatim **draft** note when the model returns nothing usable (`llm_fallback:true`); `POST /ingest/search` - sediment search/tool results as a reference note; `POST /notes` - manual note; `GET /notes` (keyword/tag/source_type/status + pagination); `GET/PUT/DELETE /notes/{id}` (DELETE is soft = archive, `?hard=true` removes the vault file); `POST /search` - `mode=keyword\|vector\|hybrid` (vector→keyword fallback when no embedding endpoint); `GET /notes/{id}/versions` - edit history; `GET /duplicates` + `POST /merge` - dedup; `POST /reindex` - rebuild vector index; `GET /graph` - product graph data (notes + entities); `POST /export/graph` + `GET /export/files/{graph.json\|graph.html\|graph.graphml\|cypher.txt}` - obsidian-wiki `wiki-export` artifacts (self-contained `graph.html` for iframe); `POST /resolve` `{targets:[…]}` + `GET /notes/{id}/wikilinks` - resolve `[[wikilink]]` targets to a note id / vault page / missing (powers clickable wikilinks); `GET /vault-page?path=` - read a raw vault meta/hub Markdown page (no DB note). Backed by the `deerflow.knowledge` module (see below). |
|
||||
|
||||
Proxied through nginx: `/api/langgraph/*` → LangGraph, all other `/api/*` → Gateway.
|
||||
|
||||
@@ -290,7 +290,7 @@ Self-contained product knowledge base. **All knowledge capability lives in this
|
||||
- **DB mirror** — tables `knowledge_notes` / `knowledge_sources` / `knowledge_tags` / `knowledge_note_tags` (`knowledge/models.py`, registered in `persistence/models/__init__.py`, auto-created by `Base.metadata.create_all`; portable types `PortableLongText`/`BeijingDateTime`). Powers listing/filter/search/audit. Global/shared — no `user_id` column; `created_by`/`updated_by` are audit-only (resolved via `get_effective_user_id()`).
|
||||
- **Markdown vault** — obsidian-wiki compatible, default `{$DEER_FLOW_HOME}/data/knowledge/obsidian-vault`. `ObsidianWikiAdapter` (`obsidian_adapter.py`) writes wiki-capture frontmatter+body, and after **every** write syncs `index.md` / `log.md` (CAPTURE/INGEST/UPDATE/ARCHIVE/EXPORT) / `hot.md` / `.manifest.json` (source-hash dedup for `skip_existing`). `markdown_vault.py` handles frontmatter build/parse, slugging, wikilink extraction and **path-traversal-proof** resolution.
|
||||
|
||||
**Module layout:** `models.py` (ORM: notes/sources/tags/note_tags + embeddings/versions/entities/relations) · `repository.py` (DB only, no LLM) · `schemas.py` (NoteDraft/SourceDraft DTOs) · `skill_templates.py` · `markdown_vault.py` · `manifest.py` · `extractor.py` (rule-based fallback) · `llm_extractor.py` (**default**: distills declarative knowledge per wiki-capture, structured JSON incl. entities/relations; falls back to `extractor.py`) · `sources.py` · `exporter.py` (`graph.json`/`graph.graphml`/`cypher.txt` + self-contained vanilla-JS `graph.html`; entity nodes + relation edges) · `chunking.py` + `embeddings.py` (OpenAI-compatible embedding client, keyword fallback) + `search.py` (keyword/vector/hybrid) · `sensitive.py` (secret/PII redaction) · `dedup.py` (Jaccard + hash dedup) · `auto_ingest.py` (background queue/worker) · `obsidian_adapter.py` · `service.py` (`KnowledgeService` orchestrator — the only thing the router touches). Tests: `tests/test_knowledge.py` (20 cases). Frontend: `frontend-web/src/core/knowledge/*` + `pages/KnowledgeBasePage.tsx` / `KnowledgeGraphPage.tsx` + chat `knowledge-save-trigger.tsx` / `knowledge-rag-toggle.tsx`; spec at `frontend-web/docs/product-knowledge-base-dev.md`.
|
||||
**Module layout:** `models.py` (ORM: notes/sources/tags/note_tags + embeddings/versions/entities/relations) · `repository.py` (DB only, no LLM) · `schemas.py` (NoteDraft/SourceDraft DTOs) · `skill_templates.py` · `markdown_vault.py` · `manifest.py` · `extractor.py` (rule-based fallback) · `llm_extractor.py` (**default**: distills declarative knowledge per wiki-capture, structured JSON incl. entities/relations; falls back to `extractor.py`) · `sources.py` · `exporter.py` (`graph.json`/`graph.graphml`/`cypher.txt` + self-contained vanilla-JS `graph.html`; entity nodes + relation edges. **`graph.html` is theme-aware** — it reads `?theme=light|dark` from its own URL since the embedding iframe can't inherit the app's CSS; `KnowledgeGraphPage.tsx` appends the active day/night mode and re-skins on toggle. The layout cools down and **freezes** so dense graphs stay clickable, node radius scales with degree, and hovering/clicking a node focuses it + its direct neighbours while dimming the rest + a detail panel — taming heavy node/edge clutter) · `chunking.py` + `embeddings.py` (OpenAI-compatible embedding client, keyword fallback) + `search.py` (keyword/vector/hybrid) · `sensitive.py` (secret/PII redaction) · `dedup.py` (Jaccard + hash dedup) · `auto_ingest.py` (background queue/worker) · `obsidian_adapter.py` · `service.py` (`KnowledgeService` orchestrator — the only thing the router touches; capture entry points `capture_thread` / `capture_search` / `capture_file` / `create_manual_note`, all converging on `_persist_draft`). Tests: `tests/test_knowledge.py` (20 cases). Frontend: `frontend-web/src/core/knowledge/*` + `pages/KnowledgeBasePage.tsx` / `KnowledgeGraphPage.tsx` + chat `knowledge-save-trigger.tsx` / `knowledge-rag-toggle.tsx`; spec at `frontend-web/docs/product-knowledge-base-dev.md`.
|
||||
|
||||
**Wikilinks & multi-note capture** (`knowledge.{related_notes_enabled,related_notes_limit,multi_note_enabled}`, all default-on; `project_name` defaults to `zncm`):
|
||||
- **Navigable `[[wikilinks]]`** — `service.resolve_wikilink(s)` maps a `[[target]]` to a knowledge note (by id → `vault_path` → title) or a vault meta page (`read_vault_page`, path-traversal-proof), else `missing`. `markdown_vault.normalize_wikilink_target` canonicalizes targets (drops `|alias`, slashes, `.md`). The frontend `MarkdownRenderer` renders non-numeric `[[...]]` as clickable links from the resolver map (numeric `[[123]]` stay citation badges); note links navigate, vault links open a `VaultPageDialog`, missing links are greyed.
|
||||
|
||||
@@ -13,9 +13,12 @@ from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import mimetypes
|
||||
import re
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, BackgroundTasks, HTTPException, Request
|
||||
from fastapi import APIRouter, BackgroundTasks, File, Form, HTTPException, Request, UploadFile
|
||||
from fastapi.responses import FileResponse
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
@@ -26,6 +29,12 @@ from deerflow.knowledge.service import EmptyThreadError
|
||||
from deerflow.runtime import serialize_channel_values
|
||||
from deerflow.runtime.references import build_reference_batches
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
from deerflow.utils.file_conversion import CONVERTIBLE_EXTENSIONS, convert_file_to_markdown
|
||||
|
||||
# Plain-text/markdown files are read directly; binary documents are converted.
|
||||
_TEXT_EXTENSIONS = {".md", ".markdown", ".txt", ".text", ""}
|
||||
# Bound the in-memory read so a knowledge import can't OOM the gateway.
|
||||
_MAX_IMPORT_BYTES = 25 * 1024 * 1024
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/api/knowledge", tags=["knowledge"])
|
||||
@@ -299,6 +308,80 @@ async def ingest_threads_batch(body: BatchIngestRequest, request: Request) -> di
|
||||
return {"ok": True, "processed": len(items), "created": created, "skipped": skipped, "failed": failed, "llm_fallback": llm_fallback, "items": items}
|
||||
|
||||
|
||||
async def _extract_upload_text(file: UploadFile) -> tuple[str, str]:
|
||||
"""Read an uploaded file and return ``(filename, markdown_text)``.
|
||||
|
||||
Plain-text/markdown is decoded directly; PDF/Office documents are converted
|
||||
to Markdown via the shared converter. Raises ``HTTPException`` on empty /
|
||||
oversized / unsupported / unparseable files.
|
||||
"""
|
||||
name = file.filename or "document"
|
||||
ext = Path(name).suffix.lower()
|
||||
raw = await file.read()
|
||||
if not raw:
|
||||
raise HTTPException(status_code=400, detail="文件为空")
|
||||
if len(raw) > _MAX_IMPORT_BYTES:
|
||||
raise HTTPException(status_code=413, detail=f"文件过大，最大支持 {_MAX_IMPORT_BYTES // (1024 * 1024)}MB")
|
||||
|
||||
if ext in _TEXT_EXTENSIONS:
|
||||
text = raw.decode("utf-8", errors="ignore")
|
||||
elif ext in CONVERTIBLE_EXTENSIONS:
|
||||
# Convert in an isolated temp dir; a fixed safe name avoids path issues
|
||||
# while preserving the extension so the converter detects the type.
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
tmp_path = Path(tmp) / f"import{ext}"
|
||||
tmp_path.write_bytes(raw)
|
||||
md_path = await convert_file_to_markdown(tmp_path)
|
||||
if md_path is None:
|
||||
raise HTTPException(status_code=422, detail=f"无法解析文件：{name}")
|
||||
text = md_path.read_text(encoding="utf-8", errors="ignore")
|
||||
else:
|
||||
raise HTTPException(status_code=415, detail=f"不支持的文件类型：{ext or '未知'}（支持 md/txt/pdf/word/ppt/excel）")
|
||||
|
||||
if not text.strip():
|
||||
raise HTTPException(status_code=422, detail="文件内容为空或无法提取文本")
|
||||
return name, text
|
||||
|
||||
|
||||
@router.post("/ingest/file")
|
||||
@require_auth
|
||||
async def ingest_file(
|
||||
request: Request,
|
||||
file: UploadFile = File(...),
|
||||
title: str | None = Form(default=None),
|
||||
tags: str | None = Form(default=None),
|
||||
status: str = Form(default="approved"),
|
||||
model_name: str | None = Form(default=None),
|
||||
) -> dict[str, Any]:
|
||||
"""Import an uploaded file into the knowledge base as wiki note(s).
|
||||
|
||||
Converts the file to Markdown, distills it via the wiki-capture LLM pipeline
|
||||
(same as thread capture), and writes the resulting declarative note(s) into
|
||||
the shared vault. ``tags`` is a comma/space-separated string.
|
||||
"""
|
||||
service = get_knowledge_service(request)
|
||||
name, text = await _extract_upload_text(file)
|
||||
tag_list = [t for t in re.split(r"[,，\s]+", tags or "") if t]
|
||||
try:
|
||||
result = await service.capture_file(
|
||||
content_md=text,
|
||||
filename=name,
|
||||
title=(title or None),
|
||||
tags=tag_list,
|
||||
status=status,
|
||||
created_by=get_effective_user_id(),
|
||||
model_name=(model_name or None),
|
||||
)
|
||||
except EmptyThreadError:
|
||||
raise HTTPException(status_code=422, detail="文件内容无可沉淀的知识")
|
||||
return {
|
||||
"note": result["note"],
|
||||
"created": result["created"],
|
||||
"notes": result.get("notes", [result["note"]]),
|
||||
"llm_fallback": result.get("llm_fallback", False),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/ingest/search")
|
||||
@require_auth
|
||||
async def ingest_search(body: IngestSearchRequest, request: Request) -> dict[str, Any]:
|
||||
|
||||
@@ -135,7 +135,15 @@ def to_cypher(graph: dict[str, Any]) -> str:
|
||||
|
||||
|
||||
def to_html(graph: dict[str, Any]) -> str:
|
||||
"""Self-contained force-directed graph (canvas + vanilla JS, no CDN)."""
|
||||
"""Self-contained, theme-aware force-directed graph (canvas + vanilla JS, no CDN).
|
||||
|
||||
The page reads ``?theme=light|dark`` from its own URL so the embedding iframe
|
||||
can stay in sync with the app's day/night mode (an iframe cannot inherit the
|
||||
parent's CSS). The layout settles and then **freezes** so a dense graph stops
|
||||
drifting and stays clickable; node size scales with degree; and hovering or
|
||||
clicking a node focuses it + its direct neighbours while dimming everything
|
||||
else, which cuts through heavy node/edge clutter.
|
||||
"""
|
||||
data_json = json.dumps(graph, ensure_ascii=False)
|
||||
return _HTML_TEMPLATE.replace("__GRAPH_DATA__", data_json)
|
||||
|
||||
@@ -178,50 +186,246 @@ _HTML_TEMPLATE = """<!DOCTYPE html>
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1"/>
|
||||
<title>Knowledge Graph</title>
|
||||
<style>
|
||||
html,body{margin:0;height:100%;background:#0b1020;color:#e7ecff;font-family:system-ui,Segoe UI,Arial,sans-serif;overflow:hidden}
|
||||
#hud{position:fixed;top:10px;left:12px;font-size:12px;opacity:.8;z-index:2}
|
||||
#tip{position:fixed;padding:4px 8px;background:#111a36;border:1px solid #2a3a66;border-radius:4px;font-size:12px;pointer-events:none;display:none;z-index:3}
|
||||
canvas{display:block}
|
||||
#empty{position:fixed;inset:0;display:flex;align-items:center;justify-content:center;opacity:.6}
|
||||
:root{
|
||||
--bg:#0b1020; --fg:#e7ecff; --sub:rgba(231,236,255,.78);
|
||||
--panel:rgba(17,26,54,.92); --panel-border:#2a3a66;
|
||||
}
|
||||
html.light{
|
||||
--bg:#f6f8fc; --fg:#1f2433; --sub:rgba(31,36,51,.72);
|
||||
--panel:rgba(255,255,255,.94); --panel-border:#d7deec;
|
||||
}
|
||||
html,body{margin:0;height:100%;background:var(--bg);color:var(--fg);font-family:system-ui,Segoe UI,Arial,sans-serif;overflow:hidden}
|
||||
canvas{display:block;cursor:grab}
|
||||
canvas.grabbing{cursor:grabbing}
|
||||
.card{position:fixed;background:var(--panel);border:1px solid var(--panel-border);border-radius:8px;font-size:12px;z-index:3;backdrop-filter:blur(6px)}
|
||||
#hud{top:10px;left:12px;padding:6px 10px;color:var(--sub);z-index:2}
|
||||
#legend{top:10px;right:12px;padding:8px 10px;display:flex;flex-direction:column;gap:6px}
|
||||
#legend .row{display:flex;align-items:center;gap:6px;color:var(--sub)}
|
||||
#legend .dot{width:10px;height:10px;border-radius:50%}
|
||||
#tip{padding:4px 8px;pointer-events:none;display:none}
|
||||
#panel{left:12px;bottom:12px;max-width:300px;padding:12px 14px;display:none}
|
||||
#panel h3{margin:0 0 6px;font-size:14px;line-height:1.3;word-break:break-word}
|
||||
#panel .meta{color:var(--sub);margin-bottom:8px}
|
||||
#panel .nb{display:flex;flex-wrap:wrap;gap:5px;max-height:120px;overflow:auto}
|
||||
#panel .nb span{padding:2px 7px;border:1px solid var(--panel-border);border-radius:999px;line-height:1.4;max-width:100%;overflow:hidden;text-overflow:ellipsis;white-space:nowrap}
|
||||
#panel .close{position:absolute;top:8px;right:10px;cursor:pointer;color:var(--sub);font-size:14px}
|
||||
#reset{bottom:12px;right:12px;padding:7px 12px;cursor:pointer;color:var(--fg);user-select:none}
|
||||
#reset:hover{filter:brightness(1.1)}
|
||||
#empty{position:fixed;inset:0;display:flex;align-items:center;justify-content:center;opacity:.55}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div id="hud"></div>
|
||||
<div id="tip"></div>
|
||||
<div id="hud" class="card"></div>
|
||||
<div id="legend" class="card">
|
||||
<div class="row"><span class="dot" id="lg-note"></span>知识笔记</div>
|
||||
<div class="row"><span class="dot" id="lg-entity"></span>实体</div>
|
||||
</div>
|
||||
<div id="tip" class="card"></div>
|
||||
<div id="panel" class="card"><span class="close" id="panel-close">×</span><h3></h3><div class="meta"></div><div class="nb"></div></div>
|
||||
<div id="reset" class="card">重置视图</div>
|
||||
<canvas id="c"></canvas>
|
||||
<script>
|
||||
const GRAPH = __GRAPH_DATA__;
|
||||
// Theme is supplied by the embedding iframe (it can't read the parent's CSS).
|
||||
const THEME = (new URLSearchParams(location.search).get('theme') === 'light') ? 'light' : 'dark';
|
||||
const PAL = {
|
||||
dark: {note:'#5b8cff', entity:'#ffb454', label:'rgba(231,236,255,.92)', edge:[140,165,255], halo:'rgba(11,16,32,.55)'},
|
||||
light: {note:'#2f6df6', entity:'#d97706', label:'rgba(31,36,51,.92)', edge:[70,110,200], halo:'rgba(255,255,255,.65)'},
|
||||
}[THEME];
|
||||
document.documentElement.classList.toggle('light', THEME === 'light');
|
||||
document.getElementById('lg-note').style.background = PAL.note;
|
||||
document.getElementById('lg-entity').style.background = PAL.entity;
|
||||
|
||||
const canvas = document.getElementById('c'), ctx = canvas.getContext('2d');
|
||||
const hud = document.getElementById('hud'), tip = document.getElementById('tip');
|
||||
let W=0,H=0;
|
||||
function resize(){W=canvas.width=innerWidth;H=canvas.height=innerHeight;}
|
||||
addEventListener('resize',resize);resize();
|
||||
const panel = document.getElementById('panel');
|
||||
let W=0,H=0,DPR=Math.min(window.devicePixelRatio||1,2);
|
||||
function resize(){W=innerWidth;H=innerHeight;canvas.width=W*DPR;canvas.height=H*DPR;canvas.style.width=W+'px';canvas.style.height=H+'px';}
|
||||
addEventListener('resize',()=>{resize();bump();});resize();
|
||||
|
||||
hud.textContent = GRAPH.nodes.length + ' 个知识节点 · ' + GRAPH.edges.length + ' 条关系';
|
||||
if(!GRAPH.nodes.length){document.body.insertAdjacentHTML('beforeend','<div id=\"empty\">暂无知识图谱数据</div>');}
|
||||
const idx={};GRAPH.nodes.forEach((n,i)=>{idx[n.id]=n;n.x=W/2+Math.cos(i)*200*Math.random()+(Math.random()-.5)*200;n.y=H/2+Math.sin(i)*200*Math.random()+(Math.random()-.5)*200;n.vx=0;n.vy=0;});
|
||||
const edges=GRAPH.edges.map(e=>({s:idx[e.source],t:idx[e.target],w:e.w||e.weight||1})).filter(e=>e.s&&e.t);
|
||||
let cam={x:0,y:0,k:1},drag=null,pan=null;
|
||||
function step(){
|
||||
for(const n of GRAPH.nodes){for(const m of GRAPH.nodes){if(n===m)continue;let dx=n.x-m.x,dy=n.y-m.y,d2=dx*dx+dy*dy+0.01,f=1200/d2;n.vx+=dx*f*0.0006;n.vy+=dy*f*0.0006;}}
|
||||
for(const e of edges){let dx=e.t.x-e.s.x,dy=e.t.y-e.s.y,d=Math.sqrt(dx*dx+dy*dy)||1,f=(d-110)*0.01;e.s.vx+=dx/d*f;e.s.vy+=dy/d*f;e.t.vx-=dx/d*f;e.t.vy-=dy/d*f;}
|
||||
for(const n of GRAPH.nodes){n.vx+=(W/2-n.x)*0.0008;n.vy+=(H/2-n.y)*0.0008;n.x+=n.vx*=0.85;n.y+=n.vy*=0.85;}
|
||||
if(!GRAPH.nodes.length){document.body.insertAdjacentHTML('beforeend','<div id="empty">暂无知识图谱数据</div>');}
|
||||
|
||||
// --- index, adjacency, degree, node sizing -------------------------------
|
||||
const idx={}, adj={};
|
||||
GRAPH.nodes.forEach(n=>{idx[n.id]=n;adj[n.id]=new Set();});
|
||||
const edges=[];
|
||||
for(const e of GRAPH.edges){
|
||||
const s=idx[e.source], t=idx[e.target];
|
||||
if(!s||!t||s===t) continue;
|
||||
edges.push({s,t,w:e.weight||e.w||1,label:e.label||''});
|
||||
adj[e.source].add(e.target); adj[e.target].add(e.source);
|
||||
}
|
||||
function draw(){
|
||||
ctx.setTransform(cam.k,0,0,cam.k,cam.x,cam.y);ctx.clearRect(-cam.x/cam.k,-cam.y/cam.k,W/cam.k,H/cam.k);
|
||||
ctx.strokeStyle='rgba(120,150,255,.25)';
|
||||
for(const e of edges){ctx.lineWidth=e.w*0.6;ctx.beginPath();ctx.moveTo(e.s.x,e.s.y);ctx.lineTo(e.t.x,e.t.y);ctx.stroke();}
|
||||
for(const n of GRAPH.nodes){const r=6;ctx.fillStyle='#5b8cff';ctx.beginPath();ctx.arc(n.x,n.y,r,0,7);ctx.fill();
|
||||
ctx.fillStyle='rgba(231,236,255,.85)';ctx.font='11px system-ui';ctx.fillText((n.label||'').slice(0,18),n.x+9,n.y+3);}
|
||||
const N=GRAPH.nodes.length;
|
||||
GRAPH.nodes.forEach((n,i)=>{
|
||||
n.deg=adj[n.id].size;
|
||||
n.r=4.5+Math.min(11,Math.sqrt(n.deg)*2.4); // hubs are visibly bigger → easier to hit
|
||||
const a=i*2.399963, rad=18*Math.sqrt(i); // phyllotaxis spread = no initial overlap
|
||||
n.x=Math.cos(a)*rad; n.y=Math.sin(a)*rad; n.vx=0; n.vy=0;
|
||||
});
|
||||
// "Hub" nodes keep their label even when zoomed out; the rest declutter.
|
||||
const degs=GRAPH.nodes.map(n=>n.deg).sort((a,b)=>b-a);
|
||||
const HUB=Math.max(3,(degs[Math.floor(N*0.08)]||3));
|
||||
|
||||
// --- force layout that cools down and FREEZES (frozen graph = clickable) --
|
||||
const REP=9000, LINK=92, CENTER=0.0016;
|
||||
let alpha=1, drag=null;
|
||||
function physics(){
|
||||
for(let i=0;i<N;i++){
|
||||
const n=GRAPH.nodes[i];
|
||||
for(let j=i+1;j<N;j++){
|
||||
const m=GRAPH.nodes[j];
|
||||
let dx=n.x-m.x,dy=n.y-m.y,d2=dx*dx+dy*dy+0.01,f=REP/d2;
|
||||
let fx=dx*f,fy=dy*f; n.vx+=fx;n.vy+=fy;m.vx-=fx;m.vy-=fy;
|
||||
}
|
||||
}
|
||||
for(const e of edges){
|
||||
let dx=e.t.x-e.s.x,dy=e.t.y-e.s.y,d=Math.hypot(dx,dy)||1,f=(d-LINK)*0.02;
|
||||
e.s.vx+=dx/d*f;e.s.vy+=dy/d*f;e.t.vx-=dx/d*f;e.t.vy-=dy/d*f;
|
||||
}
|
||||
for(const n of GRAPH.nodes){
|
||||
if(n===drag){n.vx=n.vy=0;continue;}
|
||||
n.vx-=n.x*CENTER; n.vy-=n.y*CENTER;
|
||||
n.x+=(n.vx*=0.82)*alpha; n.y+=(n.vy*=0.82)*alpha;
|
||||
}
|
||||
alpha*=0.985;
|
||||
}
|
||||
function loop(){step();draw();requestAnimationFrame(loop);}loop();
|
||||
|
||||
// --- camera / fit-to-view -------------------------------------------------
|
||||
let cam={x:0,y:0,k:1};
|
||||
function toWorld(x,y){return {x:(x-cam.x)/cam.k,y:(y-cam.y)/cam.k};}
|
||||
canvas.addEventListener('mousedown',ev=>{const p=toWorld(ev.clientX,ev.clientY);let hit=null;
|
||||
for(const n of GRAPH.nodes){if((n.x-p.x)**2+(n.y-p.y)**2<100){hit=n;break;}}if(hit)drag=hit;else pan={x:ev.clientX,y:ev.clientY,cx:cam.x,cy:cam.y};});
|
||||
addEventListener('mousemove',ev=>{const p=toWorld(ev.clientX,ev.clientY);if(drag){drag.x=p.x;drag.y=p.y;drag.vx=drag.vy=0;}else if(pan){cam.x=pan.cx+ev.clientX-pan.x;cam.y=pan.cy+ev.clientY-pan.y;}
|
||||
let hov=null;for(const n of GRAPH.nodes){if((n.x-p.x)**2+(n.y-p.y)**2<100){hov=n;break;}}
|
||||
if(hov){tip.style.display='block';tip.style.left=(ev.clientX+12)+'px';tip.style.top=(ev.clientY+12)+'px';tip.textContent=hov.label;}else{tip.style.display='none';}});
|
||||
addEventListener('mouseup',()=>{drag=null;pan=null;});
|
||||
canvas.addEventListener('wheel',ev=>{ev.preventDefault();const s=ev.deltaY<0?1.1:0.9;cam.k*=s;cam.x=ev.clientX-(ev.clientX-cam.x)*s;cam.y=ev.clientY-(ev.clientY-cam.y)*s;},{passive:false});
|
||||
function fitView(){
|
||||
if(!N){return;}
|
||||
let a=1e9,b=1e9,c=-1e9,d=-1e9;
|
||||
for(const n of GRAPH.nodes){a=Math.min(a,n.x);b=Math.min(b,n.y);c=Math.max(c,n.x);d=Math.max(d,n.y);}
|
||||
const gw=Math.max(c-a,1),gh=Math.max(d-b,1),pad=90;
|
||||
cam.k=Math.max(0.12,Math.min(2.2,Math.min((W-pad)/gw,(H-pad)/gh)));
|
||||
cam.x=W/2-((a+c)/2)*cam.k; cam.y=H/2-((b+d)/2)*cam.k;
|
||||
}
|
||||
let fitted=false;
|
||||
function bump(){alpha=Math.max(alpha,0.35);} // re-warm the sim on interaction
|
||||
|
||||
// --- focus / hover state --------------------------------------------------
|
||||
let hoverNode=null, focusNode=null;
|
||||
function active(){return focusNode||hoverNode;}
|
||||
function neighbors(n){return n?adj[n.id]:null;}
|
||||
|
||||
function draw(){
|
||||
ctx.setTransform(DPR,0,0,DPR,0,0);
|
||||
ctx.clearRect(0,0,W,H);
|
||||
ctx.translate(cam.x,cam.y); ctx.scale(cam.k,cam.k);
|
||||
const act=active(), nb=neighbors(act), zoomed=cam.k>1.35;
|
||||
const [er,eg,eb]=PAL.edge;
|
||||
|
||||
// edges: dim by default; when a node is focused, only its edges stay bright.
|
||||
for(const e of edges){
|
||||
const lit = act && (e.s===act||e.t===act);
|
||||
const op = act ? (lit?0.9:0.05) : Math.min(0.5,0.12+e.w*0.05);
|
||||
ctx.strokeStyle='rgba('+er+','+eg+','+eb+','+op+')';
|
||||
ctx.lineWidth=(lit?1.6:0.8)*Math.max(0.6,e.w*0.55)/Math.sqrt(cam.k);
|
||||
ctx.beginPath();ctx.moveTo(e.s.x,e.s.y);ctx.lineTo(e.t.x,e.t.y);ctx.stroke();
|
||||
}
|
||||
// nodes
|
||||
for(const n of GRAPH.nodes){
|
||||
const dim = act && n!==act && !(nb&&nb.has(n.id));
|
||||
ctx.globalAlpha = dim?0.22:1;
|
||||
ctx.fillStyle = n.type==='entity'?PAL.entity:PAL.note;
|
||||
ctx.beginPath();ctx.arc(n.x,n.y,n.r,0,6.2832);ctx.fill();
|
||||
if(n===act){ctx.lineWidth=2/cam.k;ctx.strokeStyle=PAL.label;ctx.stroke();}
|
||||
ctx.globalAlpha=1;
|
||||
}
|
||||
// labels: only for focus/neighbors, hubs, or when zoomed in — kills clutter.
|
||||
ctx.textBaseline='middle';
|
||||
for(const n of GRAPH.nodes){
|
||||
const show = act ? (n===act||(nb&&nb.has(n.id))) : (zoomed||n.deg>=HUB);
|
||||
if(!show) continue;
|
||||
const txt=(n.label||'').slice(0,20);
|
||||
if(!txt) continue;
|
||||
ctx.font=(n===act?'bold ':'')+(11/cam.k)+'px system-ui';
|
||||
const tx=n.x+n.r+4/cam.k, ty=n.y;
|
||||
ctx.lineWidth=3/cam.k; ctx.strokeStyle=PAL.halo;
|
||||
ctx.strokeText(txt,tx,ty);
|
||||
ctx.fillStyle=PAL.label; ctx.fillText(txt,tx,ty);
|
||||
}
|
||||
}
|
||||
|
||||
function loop(){
|
||||
if(alpha>0.02 || drag){physics();} else if(!fitted){fitView();fitted=true;}
|
||||
draw();
|
||||
requestAnimationFrame(loop);
|
||||
}
|
||||
// Pre-settle a few iterations so the first paint is already roughly laid out.
|
||||
for(let i=0;i<60;i++)physics();
|
||||
fitView();
|
||||
loop();
|
||||
|
||||
// --- picking & interaction ------------------------------------------------
|
||||
function pick(p){
|
||||
let best=null,bd=1e9;
|
||||
for(const n of GRAPH.nodes){
|
||||
const dx=n.x-p.x,dy=n.y-p.y,d=dx*dx+dy*dy, hit=(n.r+8/cam.k);
|
||||
if(d<hit*hit && d<bd){bd=d;best=n;}
|
||||
}
|
||||
return best;
|
||||
}
|
||||
let pan=null, downXY=null, moved=false;
|
||||
canvas.addEventListener('mousedown',ev=>{
|
||||
const p=toWorld(ev.clientX,ev.clientY), hit=pick(p);
|
||||
downXY={x:ev.clientX,y:ev.clientY,hit}; moved=false;
|
||||
if(hit){drag=hit;} else {pan={x:ev.clientX,y:ev.clientY,cx:cam.x,cy:cam.y};}
|
||||
canvas.classList.add('grabbing');
|
||||
});
|
||||
addEventListener('mousemove',ev=>{
|
||||
const p=toWorld(ev.clientX,ev.clientY);
|
||||
if(downXY && (Math.abs(ev.clientX-downXY.x)>3||Math.abs(ev.clientY-downXY.y)>3)) moved=true;
|
||||
if(drag){drag.x=p.x;drag.y=p.y;drag.vx=drag.vy=0;bump();}
|
||||
else if(pan){cam.x=pan.cx+ev.clientX-pan.x;cam.y=pan.cy+ev.clientY-pan.y;}
|
||||
hoverNode=drag||pan?hoverNode:pick(p);
|
||||
if(hoverNode && !drag && !pan){
|
||||
tip.style.display='block';
|
||||
tip.style.left=Math.min(ev.clientX+14,W-220)+'px';
|
||||
tip.style.top=(ev.clientY+14)+'px';
|
||||
tip.textContent=hoverNode.label||'';
|
||||
} else { tip.style.display='none'; }
|
||||
});
|
||||
addEventListener('mouseup',ev=>{
|
||||
if(downXY && !moved){ // a click, not a drag → toggle focus / detail panel
|
||||
focusNode = (downXY.hit && downXY.hit!==focusNode) ? downXY.hit : null;
|
||||
renderPanel();
|
||||
}
|
||||
drag=null;pan=null;downXY=null;canvas.classList.remove('grabbing');
|
||||
});
|
||||
canvas.addEventListener('wheel',ev=>{
|
||||
ev.preventDefault();
|
||||
const s=ev.deltaY<0?1.12:0.89;
|
||||
const nk=Math.max(0.08,Math.min(6,cam.k*s)), r=nk/cam.k;
|
||||
cam.x=ev.clientX-(ev.clientX-cam.x)*r; cam.y=ev.clientY-(ev.clientY-cam.y)*r; cam.k=nk;
|
||||
},{passive:false});
|
||||
|
||||
// --- detail panel ---------------------------------------------------------
|
||||
const TYPE_LABEL={note:'知识笔记',entity:'实体'};
|
||||
function renderPanel(){
|
||||
if(!focusNode){panel.style.display='none';return;}
|
||||
const n=focusNode;
|
||||
panel.querySelector('h3').textContent=n.label||n.id;
|
||||
const cat=n.category?' · '+n.category:'';
|
||||
panel.querySelector('.meta').textContent=(TYPE_LABEL[n.type]||n.type||'节点')+cat+' · '+n.deg+' 条关系';
|
||||
const nb=panel.querySelector('.nb'); nb.innerHTML='';
|
||||
let count=0;
|
||||
for(const id of adj[n.id]){
|
||||
if(count++>=24)break;
|
||||
const m=idx[id]; if(!m)continue;
|
||||
const s=document.createElement('span'); s.textContent=m.label||id;
|
||||
s.title=m.label||id;
|
||||
s.style.cursor='pointer';
|
||||
s.onclick=()=>{focusNode=m;renderPanel();};
|
||||
nb.appendChild(s);
|
||||
}
|
||||
if(!adj[n.id].size){const s=document.createElement('span');s.textContent='（无关联）';s.style.opacity=.6;nb.appendChild(s);}
|
||||
panel.style.display='block';
|
||||
}
|
||||
document.getElementById('panel-close').onclick=()=>{focusNode=null;renderPanel();};
|
||||
document.getElementById('reset').onclick=()=>{focusNode=null;renderPanel();fitView();};
|
||||
</script>
|
||||
</body>
|
||||
</html>"""
|
||||
+146
-13
@@ -84,6 +84,46 @@ references（外部来源汇总）、decision（架构/设计决策）、journal
|
||||
}"""
|
||||
|
||||
|
||||
_DOC_SYSTEM_PROMPT = """\
|
||||
你是知识库整理专家，遵循 obsidian-wiki 的 wiki-capture 规范，把一篇**文档**提炼成\
|
||||
**陈述式知识（declarative knowledge）**，而不是照抄原文。
|
||||
|
||||
要求：
|
||||
- 只保留高价值内容：核心结论、关键事实、定义、流程、数据、决策与其依据。
|
||||
- 丢弃：目录、页眉页脚、版权声明、重复的样板文字、无意义的排版噪声。
|
||||
- 用自己的话重组为结构化知识，不要逐段复制原文。
|
||||
- 推断内容用 ^[inferred] 标记，不确定/冲突用 ^[ambiguous] 标记。
|
||||
- 分类 category 取值之一：synthesis（分析/方案/结论）、concepts（概念/框架）、\
|
||||
references（外部来源汇总）、decision（架构/设计决策）、journal（纪要）。
|
||||
- 用中文输出。
|
||||
|
||||
一篇文档可能只讲一个主题，也可能覆盖多个相互独立的主题。请把它拆成 1~4 页知识：
|
||||
- 默认只产出 **1 页**；只有当文档明确包含**多个彼此独立、可各自成文**的主题时才拆成多页。
|
||||
- 每一页聚焦单一主题，标题与 summary 必须彼此区分。
|
||||
- 文档没有可沉淀的知识时返回空的 notes 数组。
|
||||
|
||||
只输出一个 JSON 对象，不要任何额外文字或 Markdown 代码块，结构与对话提炼一致：
|
||||
{
|
||||
"should_ingest": true,
|
||||
"reason": "为什么值得/不值得沉淀",
|
||||
"notes": [
|
||||
{
|
||||
"title": "简洁的知识标题（不超过 40 字）",
|
||||
"summary": "1-2 句话概括这页知识",
|
||||
"category": "synthesis",
|
||||
"context": "文档背景或适用场景，1-3 句",
|
||||
"findings": ["结论/要点（陈述式，可含 ^[inferred]）", "..."],
|
||||
"reasoning": "为什么成立、取舍、不确定性",
|
||||
"implications": ["后续建议/风险/下一步", "..."],
|
||||
"tags": ["标签1", "标签2"],
|
||||
"entities": [{"name": "实体名", "type": "technology|concept|person|org|product|other"}],
|
||||
"relations": [{"from": "实体A", "to": "实体B", "type": "uses|relates_to|part_of|depends_on"}],
|
||||
"confidence": 0.0
|
||||
}
|
||||
]
|
||||
}"""
|
||||
|
||||
|
||||
def _build_transcript(turns: list[dict[str, str]], sources: list[SourceDraft], *, max_chars: int) -> str:
|
||||
parts: list[str] = []
|
||||
for i, t in enumerate(turns, 1):
|
||||
@@ -171,21 +211,27 @@ def _notes_from_data(data: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
def _build_note_draft(
|
||||
note_data: dict[str, Any],
|
||||
*,
|
||||
thread_id: str,
|
||||
thread_title: str | None,
|
||||
turns: list[dict[str, str]],
|
||||
source_type: str,
|
||||
source_id: str | None,
|
||||
source_key: str,
|
||||
fallback_title: str,
|
||||
title_override: str | None,
|
||||
tags: list[str] | None,
|
||||
project_name: str | None,
|
||||
sources: list[SourceDraft],
|
||||
source_key_suffix: str = "",
|
||||
) -> NoteDraft:
|
||||
"""Render one note dict into a :class:`NoteDraft`."""
|
||||
"""Render one note dict into a :class:`NoteDraft`.
|
||||
|
||||
Source-agnostic: the caller supplies ``source_type`` / ``source_id`` /
|
||||
``source_key`` and a ``fallback_title`` (used when neither the user override
|
||||
nor the LLM produced a usable title), so both thread and document capture
|
||||
share one renderer.
|
||||
"""
|
||||
category = str(note_data.get("category") or "synthesis").strip().lower()
|
||||
if category not in _VALID_CATEGORIES:
|
||||
category = "synthesis"
|
||||
|
||||
final_title = (title_override or "").strip() or str(note_data.get("title") or "").strip() or extractor._auto_title(thread_title, turns)
|
||||
final_title = (title_override or "").strip() or str(note_data.get("title") or "").strip() or fallback_title
|
||||
summary = str(note_data.get("summary") or "").strip()
|
||||
related = extractor.related_links(project_name)
|
||||
body = _render_body(note_data, final_title, related, sources)
|
||||
@@ -210,15 +256,15 @@ def _build_note_draft(
|
||||
summary=summary or final_title,
|
||||
content_md=body,
|
||||
category=category,
|
||||
source_type="thread",
|
||||
source_id=thread_id,
|
||||
source_type=source_type,
|
||||
source_id=source_id,
|
||||
tags=base_tags,
|
||||
confidence=max(0.0, min(confidence, 1.0)),
|
||||
related=related,
|
||||
sources=sources,
|
||||
entities=[e for e in entities if isinstance(e, dict) and e.get("name")],
|
||||
relations=[r for r in relations if isinstance(r, dict) and r.get("from") and r.get("to")],
|
||||
source_key=f"thread:{thread_id}{source_key_suffix}",
|
||||
source_key=source_key,
|
||||
)
|
||||
|
||||
|
||||
@@ -293,6 +339,7 @@ async def extract_thread_llm_multi(
|
||||
else:
|
||||
note_dicts = note_dicts[: max(1, max_notes)]
|
||||
|
||||
fallback_title = extractor._auto_title(thread_title, turns)
|
||||
drafts: list[NoteDraft] = []
|
||||
for i, nd in enumerate(note_dicts):
|
||||
# The user-supplied title override only applies to a single-note result;
|
||||
@@ -302,14 +349,100 @@ async def extract_thread_llm_multi(
|
||||
drafts.append(
|
||||
_build_note_draft(
|
||||
nd,
|
||||
thread_id=thread_id,
|
||||
thread_title=thread_title,
|
||||
turns=turns,
|
||||
source_type="thread",
|
||||
source_id=thread_id,
|
||||
source_key=f"thread:{thread_id}{suffix}",
|
||||
fallback_title=fallback_title,
|
||||
title_override=override,
|
||||
tags=tags,
|
||||
project_name=project_name,
|
||||
sources=sources,
|
||||
)
|
||||
)
|
||||
return drafts
|
||||
|
||||
|
||||
async def extract_document_llm_multi(
|
||||
text: str,
|
||||
*,
|
||||
doc_id: str,
|
||||
doc_title: str | None = None,
|
||||
title: str | None = None,
|
||||
tags: list[str] | None = None,
|
||||
project_name: str | None = "zncm",
|
||||
sources: list[SourceDraft] | None = None,
|
||||
model_name: str | None = None,
|
||||
max_input_chars: int = 12000,
|
||||
allow_multi: bool = True,
|
||||
max_notes: int = 4,
|
||||
) -> list[NoteDraft]:
|
||||
"""Distill an imported **document** into one *or several* knowledge notes.
|
||||
|
||||
Mirrors :func:`extract_thread_llm_multi` but takes raw document text (already
|
||||
converted to Markdown/plain text) instead of a conversation. Returns ``[]``
|
||||
when the LLM path fails (callers fall back to a verbatim note); raises
|
||||
:class:`LLMRejectedIngest` when the model deliberately judges the document
|
||||
worthless. Resulting notes carry ``source_type="file"``.
|
||||
"""
|
||||
text = (text or "").strip()
|
||||
if not text:
|
||||
return []
|
||||
sources = sources or []
|
||||
|
||||
try:
|
||||
from langchain_core.messages import HumanMessage, SystemMessage
|
||||
|
||||
from deerflow.models import create_chat_model
|
||||
|
||||
model = create_chat_model(name=model_name or None, thinking_enabled=False)
|
||||
body = text if len(text) <= max_input_chars else text[:max_input_chars] + "\n…（已截断）"
|
||||
header = f"文档标题：{doc_title}\n\n" if doc_title else ""
|
||||
prompt = [
|
||||
SystemMessage(content=_DOC_SYSTEM_PROMPT + "\n\n" + WIKI_CAPTURE_GUIDANCE),
|
||||
HumanMessage(content=f"以下是一篇文档，请按规范提炼为知识 JSON：\n\n{header}{body}"),
|
||||
]
|
||||
response = await model.ainvoke(prompt)
|
||||
raw = response.content if isinstance(response.content, str) else str(response.content)
|
||||
data = _parse_json(raw)
|
||||
except Exception:
|
||||
logger.exception("[knowledge] document %s — LLM extraction call failed; using rule-based fallback", doc_id)
|
||||
return []
|
||||
|
||||
if not data:
|
||||
logger.warning("[knowledge] document %s — LLM returned no parsable JSON; using rule-based fallback", doc_id)
|
||||
return []
|
||||
|
||||
note_dicts = _notes_from_data(data)
|
||||
if not note_dicts:
|
||||
reason = str(data.get("reason") or "")[:200]
|
||||
if _is_falsey(data.get("should_ingest")):
|
||||
logger.warning("[knowledge] document %s SKIPPED — model judged not worth ingesting: %s", doc_id, reason)
|
||||
raise LLMRejectedIngest(reason or "no knowledge value")
|
||||
logger.warning(
|
||||
"[knowledge] document %s — LLM produced no notes (should_ingest=%s, reason=%r); using rule-based fallback",
|
||||
doc_id,
|
||||
data.get("should_ingest"),
|
||||
reason,
|
||||
)
|
||||
return []
|
||||
note_dicts = note_dicts[:1] if not allow_multi else note_dicts[: max(1, max_notes)]
|
||||
|
||||
fallback_title = (doc_title or title or "文档知识").strip() or "文档知识"
|
||||
drafts: list[NoteDraft] = []
|
||||
for i, nd in enumerate(note_dicts):
|
||||
override = title if (len(note_dicts) == 1) else None
|
||||
suffix = "" if len(note_dicts) == 1 else f"#{i}"
|
||||
drafts.append(
|
||||
_build_note_draft(
|
||||
nd,
|
||||
source_type="file",
|
||||
source_id=doc_id,
|
||||
source_key=f"file:{doc_id}{suffix}",
|
||||
fallback_title=fallback_title,
|
||||
title_override=override,
|
||||
tags=tags,
|
||||
project_name=project_name,
|
||||
sources=sources,
|
||||
source_key_suffix=suffix,
|
||||
)
|
||||
)
|
||||
return drafts
|
||||
|
||||
@@ -394,6 +394,129 @@ class KnowledgeService:
|
||||
)
|
||||
return await self._persist_draft(draft, created_by=created_by)
|
||||
|
||||
def _verbatim_file_draft(self, body: str, filename: str, *, title: str | None, tags: list[str] | None, doc_id: str) -> NoteDraft:
|
||||
"""Build a verbatim fallback note from a file when LLM distillation fails.
|
||||
|
||||
Lands as a ``draft`` (like the thread fallback) since it is the raw
|
||||
document, not distilled knowledge.
|
||||
"""
|
||||
base_tags = list(tags or [])
|
||||
for default in (self.project_name, "knowledge-base"):
|
||||
if default and default not in base_tags:
|
||||
base_tags.append(default)
|
||||
name = (title or Path(filename).stem or "文档知识").strip() or "文档知识"
|
||||
summary = next((ln.strip() for ln in body.splitlines() if ln.strip()), name)[:280]
|
||||
return NoteDraft(
|
||||
title=name,
|
||||
summary=summary,
|
||||
content_md=body,
|
||||
category="references",
|
||||
source_type="file",
|
||||
source_id=doc_id,
|
||||
status="draft",
|
||||
tags=base_tags,
|
||||
confidence=0.4,
|
||||
related=extractor.related_links(self.project_name),
|
||||
source_key=f"file:{doc_id}",
|
||||
)
|
||||
|
||||
async def capture_file(
|
||||
self,
|
||||
*,
|
||||
content_md: str,
|
||||
filename: str,
|
||||
title: str | None = None,
|
||||
tags: list[str] | None = None,
|
||||
status: str = "approved",
|
||||
created_by: str | None = None,
|
||||
model_name: str | None = None,
|
||||
skip_existing: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
"""Import a file (already converted to Markdown/text) into knowledge notes.
|
||||
|
||||
Runs the same wiki-capture LLM distillation as thread capture, so an
|
||||
uploaded document becomes one or more declarative wiki notes in the vault.
|
||||
Falls back to a verbatim ``draft`` note when the LLM is disabled or returns
|
||||
nothing usable. Returns ``{"note", "created", "notes", "llm_fallback"}``;
|
||||
raises :class:`EmptyThreadError` when there is nothing to keep.
|
||||
"""
|
||||
import hashlib
|
||||
|
||||
text = (content_md or "").strip()
|
||||
if not text:
|
||||
raise EmptyThreadError(f"File {filename} is empty")
|
||||
# Strip any frontmatter the document carries; the vault writer adds its own.
|
||||
_, parsed = MarkdownVault.parse(text)
|
||||
body = (parsed or text).strip()
|
||||
if not body:
|
||||
raise EmptyThreadError(f"File {filename} has no body content")
|
||||
# Stable id from name + size so re-importing the same file can be skipped.
|
||||
doc_id = hashlib.sha1(f"{filename}:{len(body)}".encode("utf-8")).hexdigest()[:16]
|
||||
|
||||
logger.info(
|
||||
"[knowledge] importing file '%s' (chars=%d, doc_id=%s, llm_extract=%s, skip_existing=%s)",
|
||||
filename,
|
||||
len(body),
|
||||
doc_id,
|
||||
self.config.llm_extract_enabled,
|
||||
skip_existing,
|
||||
)
|
||||
|
||||
if skip_existing:
|
||||
existing = await self.repo.find_by_source("file", doc_id)
|
||||
if existing:
|
||||
logger.info("[knowledge] file '%s' already imported (note %s); skipping", filename, existing.get("id"))
|
||||
return {"note": existing, "created": False, "notes": [existing], "llm_fallback": False}
|
||||
|
||||
drafts: list[NoteDraft] = []
|
||||
llm_fallback = False
|
||||
if self.config.llm_extract_enabled:
|
||||
try:
|
||||
drafts = await llm_extractor.extract_document_llm_multi(
|
||||
body,
|
||||
doc_id=doc_id,
|
||||
doc_title=title or Path(filename).stem,
|
||||
title=title,
|
||||
tags=tags,
|
||||
project_name=self.project_name,
|
||||
model_name=model_name or self.config.extract_model_name,
|
||||
max_input_chars=self.config.extract_max_input_chars,
|
||||
allow_multi=self.config.multi_note_enabled,
|
||||
)
|
||||
except llm_extractor.LLMRejectedIngest as exc:
|
||||
raise EmptyThreadError(f"File {filename} judged not worth ingesting: {exc}") from exc
|
||||
if not drafts:
|
||||
llm_fallback = True
|
||||
if not drafts:
|
||||
drafts = [self._verbatim_file_draft(body, filename, title=title, tags=tags, doc_id=doc_id)]
|
||||
|
||||
effective_status = status
|
||||
if llm_fallback and normalize_status(status) == "approved":
|
||||
effective_status = "draft"
|
||||
normalized = normalize_status(effective_status)
|
||||
for draft in drafts:
|
||||
draft.status = normalized
|
||||
|
||||
# Cross-link sibling notes split from the same document (resolved by title
|
||||
# once all are persisted), mirroring multi-note thread capture.
|
||||
if len(drafts) > 1:
|
||||
for i, draft in enumerate(drafts):
|
||||
siblings = [f"[[{o.title}]]" for j, o in enumerate(drafts) if j != i and o.title]
|
||||
if siblings:
|
||||
draft.content_md = _inject_related_notes(draft.content_md, siblings)
|
||||
draft.related = list(dict.fromkeys([*siblings, *draft.related]))
|
||||
|
||||
notes = [await self._persist_draft(draft, created_by=created_by) for draft in drafts]
|
||||
logger.info(
|
||||
"[knowledge] file '%s' imported → %d note(s) [%s], status=%s, llm_fallback=%s",
|
||||
filename,
|
||||
len(notes),
|
||||
", ".join(n.get("title", "") for n in notes),
|
||||
normalized,
|
||||
llm_fallback,
|
||||
)
|
||||
return {"note": notes[0], "created": True, "notes": notes, "llm_fallback": llm_fallback}
|
||||
|
||||
# -- CRUD -------------------------------------------------------------
|
||||
async def list_notes(
|
||||
self,
|
||||
|
||||
Reference in new issue
Block a user