fix: harden wiki delivery and leaderboard performance
This commit is contained in:
1 parent
0c5e2e8cb0
commit
9f29ecae9e
28 files changed
+616
-68
No files matched your search
@@ -636,13 +636,20 @@ Two `TypeDecorator`s keep ORM columns correct across the `sqlite` / `postgres` /
|
||||
|
||||
### Schema Migrations / Manual Column-Add SQL (`scripts/sql/`)
|
||||
|
||||
Alembic auto-migration is **disabled** ? `_run_alembic_upgrade(url)` is commented out in `persistence/engine.py`. On startup only `Base.metadata.create_all` runs, which creates **new tables** but never patches **existing** ones (no new columns, no new indexes). So a column added to an ORM model appears automatically on a brand-new deployment, but is **missing on any existing MySQL database** ? and code reading/writing that column will error there until the column is added by hand.
|
||||
Gateway startup runs Alembic through `app.gateway.app._run_db_migrations` after ORM table creation. The migration graph must always have unique revision ids and exactly one head; `tests/test_migration_graph.py` enforces both. Historical duplicate revision `20260821_01` is repaired by the `20260821_03` / `20260821_04` fixed-question chain and merge revision `20260825_01`. Startup DDL has a bounded timeout and failures are logged without preventing the app from serving.
|
||||
|
||||
**Convention (required):** whenever code adds a field to an existing table (a new `mapped_column` on an ORM model), you must, in the same change:
|
||||
1. Keep the matching alembic version file under `persistence/migrations/versions/` (guarded with `_has_table`/`_has_column`), and
|
||||
2. Append the idempotent `ADD COLUMN` statement to **`scripts/sql/column_additions_mysql.sql`**, with a definition matching the ORM column exactly (type, length, nullability).
|
||||
|
||||
Operators upgrading an existing DB run the whole file (`USE <db>;` then execute) ? it is idempotent (existing columns are skipped via `information_schema.columns`), so it is safe to re-run. Index-only changes go in the sibling `scripts/sql/leaderboard_indexes_mysql.sql` (same idempotent pattern).
|
||||
Operators upgrading an existing DB may run the whole file (`USE <db>;` then execute) as a recovery path; it is idempotent (existing columns are skipped via `information_schema.columns`). Index-only changes go in the sibling `scripts/sql/leaderboard_indexes_mysql.sql` (same idempotent pattern) and stay out of startup migrations because online index creation on large metric tables can delay service boot.
|
||||
|
||||
Leaderboard scheduled-run exclusion uses correlated `NOT EXISTS` against the
|
||||
indexed `scheduled_task_runs.agent_run_id`; do not restore `NOT IN`. Legacy
|
||||
scheduler detection for tool metrics is also a correlated existence check and
|
||||
must not outer-join the complete `runs` table into metric aggregations. Keep the
|
||||
created-at + dimension compound indexes in ORM metadata and the idempotent
|
||||
MySQL index script synchronized.
|
||||
|
||||
### Sandbox System (`packages/harness/deerflow/sandbox/`)
|
||||
|
||||
@@ -1494,6 +1501,16 @@ Wiki directory, reconstruct hierarchy from `parent_slug`, `wiki_path`,
|
||||
directory navigation on the same lightweight route rather than mounting the
|
||||
WeKnora iframe.
|
||||
|
||||
Generated HashRouter URLs must retain `window.location.pathname` (for example
|
||||
the `/web/` reverse-proxy prefix); never emit root-absolute `/#/...` citation,
|
||||
reader, or public-embed links. Protected Wiki image rendering must use real HTTP
|
||||
URLs, never `URL.createObjectURL`. Public readers use the published file proxy;
|
||||
authenticated readers obtain a short-lived signed capability from
|
||||
`/files/access-url` and load `/api/public/llmwiki/files/{token}`. The signed
|
||||
payload is restricted to `kind=wiki_file`, mapping id, remote id, validated
|
||||
protected path, and expiry, and responses provide an inline UTF-8 filename for
|
||||
new-tab viewing and browser Save As.
|
||||
|
||||
Third-party pages embed published Wiki content through the chrome-free route
|
||||
`/#/embed/knowledge/{mapping_id}` (optional exact-page `wikiSlug`). It defaults
|
||||
to a recognized Wiki index page (title `索引`/`index`, page type `index`, or an
|
||||
@@ -1526,7 +1543,11 @@ mapping id is hard to guess.
|
||||
|
||||
Owners manually force a full sync/vectorization through `POST
|
||||
/api/llmwiki/knowledge-bases/{id}/vectorize`; status is available through `GET
|
||||
/api/llmwiki/knowledge-bases/{id}/wiki-index-status`. The Wiki management page
|
||||
/api/llmwiki/knowledge-bases/{id}/wiki-index-status`. That response includes
|
||||
aggregate `progress` and per-page `pages` details; current-scan progress is
|
||||
derived from `active_sync_id` / `last_seen_sync_id`, while each page exposes its
|
||||
index state, vector count, timestamps, and bounded error. The Wiki management
|
||||
page must show the progress bar and a status badge beside every article. It
|
||||
also exports all remote processed Wiki pages via `GET
|
||||
/api/llmwiki/knowledge-bases/{id}/wiki/export.xlsx`. Directory export must keep
|
||||
the full path, current directory, parent path, and dynamic per-level columns;
|
||||
|
||||
Reference in new issue
Block a user