wsk/deerflow-code
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: keep offline start script minimal

Browse Source
This commit is contained in:
吉鑫升 committed 2026-07-21 14:46:58 +08:00
1 parent 5d2997b6a0
commit e571caaf2c
4 files changed
+65 -102
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

offline-backend-20260512/README.md
+22 -20
View File
@@ -1,13 +1,13 @@
# DeerFlow 离线后端部署补充
本目录提供一套与 `start_backend_offline.sh` 配套的 wheelhouse 流程，用于在联网机器准备 Python 依赖，再将整个发布目录复制到无网络服务器运行。
本目录提供与原始 `start_backend.sh` 配套的 wheelhouse。启动脚本保持原有结构，只新增一行 Redis Python 客户端的离线安装命令。
## 文件
- `offline-runtime-requirements.txt`：启动时补装的固定版本依赖。
- `offline-runtime-requirements.txt`：固定版本的完整依赖闭包。
- `prepare_wheelhouse.sh`：联网机器下载 Linux/Python 3.12 wheel，并验证传递依赖完整性。
- `start_backend_offline.sh`：从本地 wheelhouse 补装依赖后启动 Docker 后端。
- `wheelhouse/`：下载脚本生成；应与启动脚本一起复制到离线机。
- `start_backend.sh`：原始 Docker 启动脚本，加一行 Redis 缺包安装命令。
- `wheelhouse/`：22 个 wheel 和 `SHA256SUMS`，需要随发布目录一起复制。
## 联网机器准备
@@ -18,12 +18,13 @@ cd offline-backend-20260512
bash prepare_wheelhouse.sh
```
脚本固定下载：
直接依赖为：
- `croniter==6.2.2`
- `hindsight-client==0.6.2`
- `redis==5.2.1`
- 上述包在 Python 3.12/Linux x86_64 下的 19 个传递依赖；版本也已全部锁定在 `offline-runtime-requirements.txt`
上述依赖在 Python 3.12/Linux x86_64 下的 19 个传递依赖也已锁定在 `offline-runtime-requirements.txt`。
国内镜像示例：
@@ -37,28 +38,29 @@ ARM64 镜像需要改目标平台：
TARGET_PLATFORM=manylinux2014_aarch64 bash prepare_wheelhouse.sh
```
下载完成后，`wheelhouse/SHA256SUMS` 可用于离线机校验文件完整性。
## 离线机器启动
确保发布目录还包含 Docker 镜像 tar、`.env`、`config.yaml`、`skill_evolution.json` 和 `backend/`，然后运行。`extensions_config.json` 与 `skills/` 会优先从发布根目录读取，不存在时自动回退到 `backend/`：
保持原脚本要求的发布目录结构，包括 Docker 镜像 tar、`.env`、`config.yaml`、`extensions_config.json`、`skill_evolution.json`、`skills/`、`wheelhouse/`、`backend/uv.lock` 和 `backend/`。
可先校验 wheel：
```bash
cd offline-backend-20260512
bash start_backend_offline.sh
cd offline-backend-20260512/wheelhouse
sha256sum -c SHA256SUMS
cd ..
```
然后启动：
```bash
bash start_backend.sh
docker logs -f deerflow-backend
```
只检查目录、配置项、wheel 和 checksum，不操作容器：
```bash
PREFLIGHT_ONLY=1 bash start_backend_offline.sh
```
启动脚本只使用 `/app/wheelhouse`，不会访问 PyPI。它会验证 checksum，并在镜像虚拟环境缺包时一次性安装三个固定版本的运行时包。
原脚本已有 `croniter` 和 `hindsight-client` 的离线补装逻辑；新增的一行命令只负责在缺少 `redis` 模块时，从 `/app/wheelhouse` 安装 `redis==5.2.1`。
## Redis 说明
`redis==5.2.1` 只是 Python 客户端，不包含 Redis 服务端。`config.yaml` 中 `concurrency.redis.enabled: false` 时无需部署 Redis；启用后必须另行准备可访问的 Redis 服务或离线 Redis Docker 镜像。
`redis==5.2.1` 只是 Python 客户端，不包含 Redis 服务端。`config.yaml` 中 `concurrency.redis.enabled: false` 时无需部署 Redis；启用后必须另行准备 Redis 服务或离线 Redis Docker 镜像。
如果 Redis 运行在另一个容器，`redis://127.0.0.1:6379/0` 指向的是后端容器自身，不能直接使用。应将两个容器加入同一个 Docker network，并把 URL 改成类似 `redis://redis:6379/0`，同时为不同环境设置不同的 `namespace`。
如果 Redis 运行在另一个容器，`redis://127.0.0.1:6379/0` 指向后端容器自身，不能直接使用。应将两个容器加入同一个 Docker network，并将 URL 改为类似 `redis://redis:6379/0`，同时为不同环境配置不同的 `namespace`。
offline-backend-20260512/backend/CLAUDE.md
+1 -1
View File
@@ -998,7 +998,7 @@ Both the `view_image` tool and `ViewImageMiddleware` are registered whenever `vi
## Documentation
See `docs/` directory for detailed documentation:
- [../README.md](../README.md) - Offline Docker wheelhouse preparation and startup workflow. The pinned versions in `../offline-runtime-requirements.txt` and `../start_backend_offline.sh` must stay synchronized. The download target defaults to CPython 3.12 on Linux x86_64 and verifies dependency closure using wheelhouse only.
- [../README.md](../README.md) - Offline Docker wheelhouse preparation and startup workflow. The pinned versions in `../offline-runtime-requirements.txt` and `../start_backend.sh` must stay synchronized. `start_backend.sh` stays aligned with the deployment script and adds only the Redis wheel fallback line. The download target defaults to CPython 3.12 on Linux x86_64 and verifies dependency closure using wheelhouse only.
- [CONFIGURATION.md](docs/CONFIGURATION.md) - Configuration options
- [ARCHITECTURE.md](docs/ARCHITECTURE.md) - Architecture details
- [API.md](docs/API.md) - API reference
offline-backend-20260512/offline-runtime-requirements.txt
+1 -1
View File
@@ -1,5 +1,5 @@
# Direct runtime packages layered into the prebuilt offline backend image.
# Keep these three pins synchronized with start_backend_offline.sh.
# Keep these three pins synchronized with start_backend.sh.
croniter==6.2.2
hindsight-client==0.6.2
redis==5.2.1
offline-backend-20260512/start_backend_offline.sh → offline-backend-20260512/start_backend.sh
+41 -80
View File
@@ -7,20 +7,13 @@ IMAGE_TAG="${IMAGE_TAG:-deerflow-backend-offline:20260505}"
CONTAINER_NAME="${CONTAINER_NAME:-deerflow-backend}"
HOST_PORT="${HOST_PORT:-8001}"
IMAGE_TAR="${IMAGE_TAR:-${SCRIPT_DIR}/deerflow-backend-offline-image-20260505.tar}"
# Default timezone for the container. The offline image is built from a base
# layer that ships with UTC, which made scheduled tasks fire at the wrong wall
# clock and made log timestamps confusing for ops. ``Asia/Shanghai`` keeps the
# scheduler aligned with the operator's expectation; override with TZ=... when
# deploying to a different region.
CONTAINER_TZ="${CONTAINER_TZ:-${TZ:-Asia/Shanghai}}"
default_extensions_config="${SCRIPT_DIR}/extensions_config.json"
if [ ! -f "${default_extensions_config}" ]; then
default_extensions_config="${SCRIPT_DIR}/backend/extensions_config.json"
fi
EXTENSIONS_CONFIG_FILE="${EXTENSIONS_CONFIG_FILE:-${default_extensions_config}}"
default_skills_dir="${SCRIPT_DIR}/skills"
if [ ! -d "${default_skills_dir}" ]; then
default_skills_dir="${SCRIPT_DIR}/backend/skills"
fi
SKILLS_DIR="${SKILLS_DIR:-${default_skills_dir}}"
die() {
echo "ERROR: $*" >&2
exit 1
@@ -43,11 +36,6 @@ require_env_key() {
grep -Eq "^[[:space:]]*${key}=" "${SCRIPT_DIR}/.env" || die "required .env key not found: ${key}"
}
require_wheel() {
local pattern="$1"
compgen -G "${SCRIPT_DIR}/wheelhouse/${pattern}" >/dev/null || die "required wheel not found: wheelhouse/${pattern}"
}
require_cmd docker
if [ ! -f "${SCRIPT_DIR}/.env" ]; then
@@ -68,37 +56,21 @@ if [ ! -f "${SCRIPT_DIR}/config.yaml" ]; then
die "missing config.yaml"
fi
require_env_key AUTH_JWT_SECRET
require_env_key SCHEDULED_TASKS_ENABLED
require_env_key ELEMENT_ENABLED
require_env_key "AUTH_JWT_SECRET"
require_env_key "SCHEDULED_TASKS_ENABLED"
require_env_key "ELEMENT_ENABLED"
require_file "${SCRIPT_DIR}/config.yaml"
require_file "${EXTENSIONS_CONFIG_FILE}"
require_file "${SCRIPT_DIR}/extensions_config.json"
require_file "${SCRIPT_DIR}/skill_evolution.json"
require_file "${SCRIPT_DIR}/backend/pyproject.toml"
require_file "${SCRIPT_DIR}/backend/app/gateway/app.py"
require_file "${SCRIPT_DIR}/backend/packages/harness/deerflow/runtime/scheduler/element.py"
require_file "${SCRIPT_DIR}/backend/langgraph.json"
require_dir "${SKILLS_DIR}"
require_dir "${SCRIPT_DIR}/skills"
require_dir "${SCRIPT_DIR}/wheelhouse"
# Knowledge base directory, mounted into the container at /mnt/knowledge_base.
require_dir "${SCRIPT_DIR}/backend/knowledge/knowledge_base"
require_wheel 'croniter-6.2.2-*.whl'
require_wheel 'hindsight_client-0.6.2-*.whl'
require_wheel 'redis-5.2.1-*.whl'
if [ -f "${SCRIPT_DIR}/wheelhouse/SHA256SUMS" ]; then
require_cmd sha256sum
require_cmd sed
echo "Verifying wheelhouse checksums"
(cd "${SCRIPT_DIR}/wheelhouse" && sed 's/\r$//' SHA256SUMS | sha256sum -c -)
fi
mkdir -p "${SCRIPT_DIR}/data"
if [ "${PREFLIGHT_ONLY:-0}" = "1" ]; then
echo "Offline backend preflight passed. No container was changed."
exit 0
fi
if ! docker image inspect "${IMAGE_TAG}" >/dev/null 2>&1; then
require_file "${IMAGE_TAR}"
echo "Loading Docker image from ${IMAGE_TAR}"
@@ -110,29 +82,6 @@ if docker ps -a --format '{{.Names}}' | awk -v name="${CONTAINER_NAME}" '$0 == n
docker rm -f "${CONTAINER_NAME}" >/dev/null
fi
mount_args=(
-v "${SCRIPT_DIR}/backend/app:/app/backend/app"
-v "${SCRIPT_DIR}/backend/packages:/app/backend/packages"
-v "${SCRIPT_DIR}/backend/scripts:/app/backend/scripts"
-v "${SCRIPT_DIR}/backend/pyproject.toml:/app/backend/pyproject.toml:ro"
-v "${SCRIPT_DIR}/backend/langgraph.json:/app/backend/langgraph.json:ro"
-v "${SCRIPT_DIR}/config.yaml:/app/config.yaml:ro"
-v "${EXTENSIONS_CONFIG_FILE}:/app/extensions_config.json:ro"
-v "${SCRIPT_DIR}/skill_evolution.json:/app/skill_evolution.json:ro"
-v "${SKILLS_DIR}:/app/skills:ro"
-v "${SCRIPT_DIR}/wheelhouse:/app/wheelhouse:ro"
-v "${SCRIPT_DIR}/backend/knowledge/knowledge_base:/mnt/knowledge_base:ro"
-v "${SCRIPT_DIR}/data:/app/backend/.deer-flow"
)
# uv.lock is not needed by uv run --no-sync, but mount it when the release
# bundle contains one so the container exactly reflects the online build.
if [ -f "${SCRIPT_DIR}/backend/uv.lock" ]; then
mount_args+=(
-v "${SCRIPT_DIR}/backend/uv.lock:/app/backend/uv.lock:ro"
)
fi
echo "Starting backend container ${CONTAINER_NAME}"
docker run -d \
--name "${CONTAINER_NAME}" \
@@ -147,33 +96,45 @@ docker run -d \
-e DEER_FLOW_SKILL_EVOLUTION_CONFIG_PATH=/app/skill_evolution.json \
-e DEER_FLOW_SKILLS_PATH=/app/skills \
-e UV_LINK_MODE=copy \
"${mount_args[@]}" \
-v "${SCRIPT_DIR}/backend/app:/app/backend/app" \
-v "${SCRIPT_DIR}/backend/packages:/app/backend/packages" \
-v "${SCRIPT_DIR}/backend/scripts:/app/backend/scripts" \
-v "${SCRIPT_DIR}/backend/pyproject.toml:/app/backend/pyproject.toml:ro" \
-v "${SCRIPT_DIR}/backend/uv.lock:/app/backend/uv.lock:ro" \
-v "${SCRIPT_DIR}/backend/langgraph.json:/app/backend/langgraph.json:ro" \
-v "${SCRIPT_DIR}/config.yaml:/app/config.yaml:ro" \
-v "${SCRIPT_DIR}/extensions_config.json:/app/extensions_config.json:ro" \
-v "${SCRIPT_DIR}/skill_evolution.json:/app/skill_evolution.json:ro" \
-v "${SCRIPT_DIR}/skills:/app/skills:ro" \
-v "${SCRIPT_DIR}/wheelhouse:/app/wheelhouse:ro" \
-v "${SCRIPT_DIR}/backend/knowledge/knowledge_base:/mnt/knowledge_base:ro" \
-v "${SCRIPT_DIR}/data:/app/backend/.deer-flow" \
"${IMAGE_TAG}" \
sh -c 'set -eu
cd backend
sh -c 'cd backend
# Apply timezone inside the container. ``TZ`` alone is enough for Python /
# uvicorn log timestamps, but linking /etc/localtime keeps C-level tools
# (e.g. anything that reads /etc/localtime directly) on the same clock.
if [ -n "${TZ:-}" ] && [ -f "/usr/share/zoneinfo/${TZ}" ]; then
ln -sf "/usr/share/zoneinfo/${TZ}" /etc/localtime 2>/dev/null || true
echo "${TZ}" > /etc/timezone 2>/dev/null || true
fi
if [ "${OFFLINE_RUNTIME_INSTALL_DEPS:-1}" != "0" ]; then
if ! uv run --no-sync python -c "import croniter, hindsight_client, redis" >/dev/null 2>&1; then
echo "Installing offline runtime packages from /app/wheelhouse"
if [ "${SCHEDULER_INSTALL_DEPS:-1}" != "0" ]; then
uv run --no-sync python -c "import croniter" >/dev/null 2>&1 || {
PYTHON_BIN="$(uv run --no-sync python -c "import sys; print(sys.executable)")"
# Word splitting is intentional so .env may override this package list.
uv pip install \
--python "${PYTHON_BIN}" \
--no-index \
--find-links /app/wheelhouse \
${OFFLINE_RUNTIME_PIP_PACKAGES:-croniter==6.2.2 hindsight-client==0.6.2 redis==5.2.1}
fi
uv run --no-sync python -c "import importlib.metadata as m; names=(\"croniter\", \"hindsight-client\", \"redis\"); print(\"Offline runtime packages:\", \", \".join(f\"{name}={m.version(name)}\" for name in names))"
uv pip install --python "${PYTHON_BIN}" --no-index --find-links /app/wheelhouse ${SCHEDULER_PIP_PACKAGES:-croniter==6.2.2}
}
fi
PYTHONPATH=. uv run --no-sync python -m uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 --workers "${GATEWAY_WORKERS:-2}"'
if [ "${HINDSIGHT_INSTALL_DEPS:-1}" != "0" ]; then
uv run --no-sync python -c "import hindsight_client" >/dev/null 2>&1 || {
PYTHON_BIN="$(uv run --no-sync python -c "import sys; print(sys.executable)")"
uv pip install --python "${PYTHON_BIN}" --no-index --find-links /app/wheelhouse ${HINDSIGHT_PIP_PACKAGES:-hindsight-client==0.6.2}
}
fi
uv run --no-sync python -c "import redis" >/dev/null 2>&1 || uv pip install --python "$(uv run --no-sync python -c "import sys; print(sys.executable)")" --no-index --find-links /app/wheelhouse ${REDIS_PIP_PACKAGES:-redis==5.2.1}
PYTHONPATH=. uv run --no-sync python -m uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 --workers ${GATEWAY_WORKERS:-2}'
echo
echo "Backend started."
echo " URL: http://<server-ip>:${HOST_PORT}"
echo " Logs: docker logs -f ${CONTAINER_NAME}"
echo " Stop: docker rm -f ${CONTAINER_NAME}"
echo " Stop: ./stop_backend.sh"