fix: harden wiki delivery and leaderboard performance

This commit is contained in:
吉鑫升 committed 2026-08-25 16:57:42 +08:00
1 parent 0c5e2e8cb0
commit 9f29ecae9e
28 files changed
+616 -68

No files matched your search

+24 -3
View File
@@ -636,13 +636,20 @@ Two `TypeDecorator`s keep ORM columns correct across the `sqlite` / `postgres` /
### Schema Migrations / Manual Column-Add SQL (`scripts/sql/`)
Alembic auto-migration is **disabled** ? `_run_alembic_upgrade(url)` is commented out in `persistence/engine.py`. On startup only `Base.metadata.create_all` runs, which creates **new tables** but never patches **existing** ones (no new columns, no new indexes). So a column added to an ORM model appears automatically on a brand-new deployment, but is **missing on any existing MySQL database** ? and code reading/writing that column will error there until the column is added by hand.
Gateway startup runs Alembic through `app.gateway.app._run_db_migrations` after ORM table creation. The migration graph must always have unique revision ids and exactly one head; `tests/test_migration_graph.py` enforces both. Historical duplicate revision `20260821_01` is repaired by the `20260821_03` / `20260821_04` fixed-question chain and merge revision `20260825_01`. Startup DDL has a bounded timeout and failures are logged without preventing the app from serving.
**Convention (required):** whenever code adds a field to an existing table (a new `mapped_column` on an ORM model), you must, in the same change:
1. Keep the matching alembic version file under `persistence/migrations/versions/` (guarded with `_has_table`/`_has_column`), and
2. Append the idempotent `ADD COLUMN` statement to **`scripts/sql/column_additions_mysql.sql`**, with a definition matching the ORM column exactly (type, length, nullability).
Operators upgrading an existing DB run the whole file (`USE <db>;` then execute) ? it is idempotent (existing columns are skipped via `information_schema.columns`), so it is safe to re-run. Index-only changes go in the sibling `scripts/sql/leaderboard_indexes_mysql.sql` (same idempotent pattern).
Operators upgrading an existing DB may run the whole file (`USE <db>;` then execute) as a recovery path; it is idempotent (existing columns are skipped via `information_schema.columns`). Index-only changes go in the sibling `scripts/sql/leaderboard_indexes_mysql.sql` (same idempotent pattern) and stay out of startup migrations because online index creation on large metric tables can delay service boot.
Leaderboard scheduled-run exclusion uses correlated `NOT EXISTS` against the
indexed `scheduled_task_runs.agent_run_id`; do not restore `NOT IN`. Legacy
scheduler detection for tool metrics is also a correlated existence check and
must not outer-join the complete `runs` table into metric aggregations. Keep the
created-at + dimension compound indexes in ORM metadata and the idempotent
MySQL index script synchronized.
### Sandbox System (`packages/harness/deerflow/sandbox/`)
@@ -1494,6 +1501,16 @@ Wiki directory, reconstruct hierarchy from `parent_slug`, `wiki_path`,
directory navigation on the same lightweight route rather than mounting the
WeKnora iframe.
Generated HashRouter URLs must retain `window.location.pathname` (for example
the `/web/` reverse-proxy prefix); never emit root-absolute `/#/...` citation,
reader, or public-embed links. Protected Wiki image rendering must use real HTTP
URLs, never `URL.createObjectURL`. Public readers use the published file proxy;
authenticated readers obtain a short-lived signed capability from
`/files/access-url` and load `/api/public/llmwiki/files/{token}`. The signed
payload is restricted to `kind=wiki_file`, mapping id, remote id, validated
protected path, and expiry, and responses provide an inline UTF-8 filename for
new-tab viewing and browser Save As.
Third-party pages embed published Wiki content through the chrome-free route
`/#/embed/knowledge/{mapping_id}` (optional exact-page `wikiSlug`). It defaults
to a recognized Wiki index page (title `索引`/`index`, page type `index`, or an
@@ -1526,7 +1543,11 @@ mapping id is hard to guess.
Owners manually force a full sync/vectorization through `POST
/api/llmwiki/knowledge-bases/{id}/vectorize`; status is available through `GET
/api/llmwiki/knowledge-bases/{id}/wiki-index-status`. The Wiki management page
/api/llmwiki/knowledge-bases/{id}/wiki-index-status`. That response includes
aggregate `progress` and per-page `pages` details; current-scan progress is
derived from `active_sync_id` / `last_seen_sync_id`, while each page exposes its
index state, vector count, timestamps, and bounded error. The Wiki management
page must show the progress bar and a status badge beside every article. It
also exports all remote processed Wiki pages via `GET
/api/llmwiki/knowledge-bases/{id}/wiki/export.xlsx`. Directory export must keep
the full path, current directory, parent path, and dynamic per-level columns;