diff --git a/offline-backend-20260512/README.md b/offline-backend-20260512/README.md index e77ac35a..09610f41 100644 --- a/offline-backend-20260512/README.md +++ b/offline-backend-20260512/README.md @@ -1,13 +1,13 @@ # DeerFlow 离线后端部署补充 -本目录提供一套与 `start_backend_offline.sh` 配套的 wheelhouse 流程,用于在联网机器准备 Python 依赖,再将整个发布目录复制到无网络服务器运行。 +本目录提供与原始 `start_backend.sh` 配套的 wheelhouse。启动脚本保持原有结构,只新增一行 Redis Python 客户端的离线安装命令。 ## 文件 -- `offline-runtime-requirements.txt`:启动时补装的固定版本依赖。 +- `offline-runtime-requirements.txt`:固定版本的完整依赖闭包。 - `prepare_wheelhouse.sh`:联网机器下载 Linux/Python 3.12 wheel,并验证传递依赖完整性。 -- `start_backend_offline.sh`:从本地 wheelhouse 补装依赖后启动 Docker 后端。 -- `wheelhouse/`:下载脚本生成;应与启动脚本一起复制到离线机。 +- `start_backend.sh`:原始 Docker 启动脚本,加一行 Redis 缺包安装命令。 +- `wheelhouse/`:22 个 wheel 和 `SHA256SUMS`,需要随发布目录一起复制。 ## 联网机器准备 @@ -18,12 +18,13 @@ cd offline-backend-20260512 bash prepare_wheelhouse.sh ``` -脚本固定下载: +直接依赖为: - `croniter==6.2.2` - `hindsight-client==0.6.2` - `redis==5.2.1` -- 上述包在 Python 3.12/Linux x86_64 下的 19 个传递依赖;版本也已全部锁定在 `offline-runtime-requirements.txt` + +上述依赖在 Python 3.12/Linux x86_64 下的 19 个传递依赖也已锁定在 `offline-runtime-requirements.txt`。 国内镜像示例: @@ -37,28 +38,29 @@ ARM64 镜像需要改目标平台: TARGET_PLATFORM=manylinux2014_aarch64 bash prepare_wheelhouse.sh ``` -下载完成后,`wheelhouse/SHA256SUMS` 可用于离线机校验文件完整性。 - ## 离线机器启动 -确保发布目录还包含 Docker 镜像 tar、`.env`、`config.yaml`、`skill_evolution.json` 和 `backend/`,然后运行。`extensions_config.json` 与 `skills/` 会优先从发布根目录读取,不存在时自动回退到 `backend/`: +保持原脚本要求的发布目录结构,包括 Docker 镜像 tar、`.env`、`config.yaml`、`extensions_config.json`、`skill_evolution.json`、`skills/`、`wheelhouse/`、`backend/uv.lock` 和 `backend/`。 + +可先校验 wheel: ```bash -cd offline-backend-20260512 -bash start_backend_offline.sh +cd offline-backend-20260512/wheelhouse +sha256sum -c SHA256SUMS +cd .. +``` + +然后启动: + +```bash +bash start_backend.sh docker logs -f deerflow-backend ``` -只检查目录、配置项、wheel 和 checksum,不操作容器: - -```bash -PREFLIGHT_ONLY=1 bash start_backend_offline.sh -``` - -启动脚本只使用 `/app/wheelhouse`,不会访问 PyPI。它会验证 checksum,并在镜像虚拟环境缺包时一次性安装三个固定版本的运行时包。 +原脚本已有 `croniter` 和 `hindsight-client` 的离线补装逻辑;新增的一行命令只负责在缺少 `redis` 模块时,从 `/app/wheelhouse` 安装 `redis==5.2.1`。 ## Redis 说明 -`redis==5.2.1` 只是 Python 客户端,不包含 Redis 服务端。`config.yaml` 中 `concurrency.redis.enabled: false` 时无需部署 Redis;启用后必须另行准备可访问的 Redis 服务或离线 Redis Docker 镜像。 +`redis==5.2.1` 只是 Python 客户端,不包含 Redis 服务端。`config.yaml` 中 `concurrency.redis.enabled: false` 时无需部署 Redis;启用后必须另行准备 Redis 服务或离线 Redis Docker 镜像。 -如果 Redis 运行在另一个容器,`redis://127.0.0.1:6379/0` 指向的是后端容器自身,不能直接使用。应将两个容器加入同一个 Docker network,并把 URL 改成类似 `redis://redis:6379/0`,同时为不同环境设置不同的 `namespace`。 +如果 Redis 运行在另一个容器,`redis://127.0.0.1:6379/0` 指向后端容器自身,不能直接使用。应将两个容器加入同一个 Docker network,并将 URL 改为类似 `redis://redis:6379/0`,同时为不同环境配置不同的 `namespace`。 diff --git a/offline-backend-20260512/backend/CLAUDE.md b/offline-backend-20260512/backend/CLAUDE.md index 839e9615..320b879a 100644 --- a/offline-backend-20260512/backend/CLAUDE.md +++ b/offline-backend-20260512/backend/CLAUDE.md @@ -998,7 +998,7 @@ Both the `view_image` tool and `ViewImageMiddleware` are registered whenever `vi ## Documentation See `docs/` directory for detailed documentation: -- [../README.md](../README.md) - Offline Docker wheelhouse preparation and startup workflow. The pinned versions in `../offline-runtime-requirements.txt` and `../start_backend_offline.sh` must stay synchronized. The download target defaults to CPython 3.12 on Linux x86_64 and verifies dependency closure using wheelhouse only. +- [../README.md](../README.md) - Offline Docker wheelhouse preparation and startup workflow. The pinned versions in `../offline-runtime-requirements.txt` and `../start_backend.sh` must stay synchronized. `start_backend.sh` stays aligned with the deployment script and adds only the Redis wheel fallback line. The download target defaults to CPython 3.12 on Linux x86_64 and verifies dependency closure using wheelhouse only. - [CONFIGURATION.md](docs/CONFIGURATION.md) - Configuration options - [ARCHITECTURE.md](docs/ARCHITECTURE.md) - Architecture details - [API.md](docs/API.md) - API reference diff --git a/offline-backend-20260512/offline-runtime-requirements.txt b/offline-backend-20260512/offline-runtime-requirements.txt index b493ed02..bcea964d 100644 --- a/offline-backend-20260512/offline-runtime-requirements.txt +++ b/offline-backend-20260512/offline-runtime-requirements.txt @@ -1,5 +1,5 @@ # Direct runtime packages layered into the prebuilt offline backend image. -# Keep these three pins synchronized with start_backend_offline.sh. +# Keep these three pins synchronized with start_backend.sh. croniter==6.2.2 hindsight-client==0.6.2 redis==5.2.1 diff --git a/offline-backend-20260512/start_backend_offline.sh b/offline-backend-20260512/start_backend.sh similarity index 54% rename from offline-backend-20260512/start_backend_offline.sh rename to offline-backend-20260512/start_backend.sh index 9e612b41..f6cfa872 100755 --- a/offline-backend-20260512/start_backend_offline.sh +++ b/offline-backend-20260512/start_backend.sh @@ -7,20 +7,13 @@ IMAGE_TAG="${IMAGE_TAG:-deerflow-backend-offline:20260505}" CONTAINER_NAME="${CONTAINER_NAME:-deerflow-backend}" HOST_PORT="${HOST_PORT:-8001}" IMAGE_TAR="${IMAGE_TAR:-${SCRIPT_DIR}/deerflow-backend-offline-image-20260505.tar}" +# Default timezone for the container. The offline image is built from a base +# layer that ships with UTC, which made scheduled tasks fire at the wrong wall +# clock and made log timestamps confusing for ops. ``Asia/Shanghai`` keeps the +# scheduler aligned with the operator's expectation; override with TZ=... when +# deploying to a different region. CONTAINER_TZ="${CONTAINER_TZ:-${TZ:-Asia/Shanghai}}" -default_extensions_config="${SCRIPT_DIR}/extensions_config.json" -if [ ! -f "${default_extensions_config}" ]; then - default_extensions_config="${SCRIPT_DIR}/backend/extensions_config.json" -fi -EXTENSIONS_CONFIG_FILE="${EXTENSIONS_CONFIG_FILE:-${default_extensions_config}}" - -default_skills_dir="${SCRIPT_DIR}/skills" -if [ ! -d "${default_skills_dir}" ]; then - default_skills_dir="${SCRIPT_DIR}/backend/skills" -fi -SKILLS_DIR="${SKILLS_DIR:-${default_skills_dir}}" - die() { echo "ERROR: $*" >&2 exit 1 @@ -43,11 +36,6 @@ require_env_key() { grep -Eq "^[[:space:]]*${key}=" "${SCRIPT_DIR}/.env" || die "required .env key not found: ${key}" } -require_wheel() { - local pattern="$1" - compgen -G "${SCRIPT_DIR}/wheelhouse/${pattern}" >/dev/null || die "required wheel not found: wheelhouse/${pattern}" -} - require_cmd docker if [ ! -f "${SCRIPT_DIR}/.env" ]; then @@ -68,37 +56,21 @@ if [ ! -f "${SCRIPT_DIR}/config.yaml" ]; then die "missing config.yaml" fi -require_env_key AUTH_JWT_SECRET -require_env_key SCHEDULED_TASKS_ENABLED -require_env_key ELEMENT_ENABLED +require_env_key "AUTH_JWT_SECRET" +require_env_key "SCHEDULED_TASKS_ENABLED" +require_env_key "ELEMENT_ENABLED" require_file "${SCRIPT_DIR}/config.yaml" -require_file "${EXTENSIONS_CONFIG_FILE}" +require_file "${SCRIPT_DIR}/extensions_config.json" require_file "${SCRIPT_DIR}/skill_evolution.json" require_file "${SCRIPT_DIR}/backend/pyproject.toml" require_file "${SCRIPT_DIR}/backend/app/gateway/app.py" require_file "${SCRIPT_DIR}/backend/packages/harness/deerflow/runtime/scheduler/element.py" -require_file "${SCRIPT_DIR}/backend/langgraph.json" -require_dir "${SKILLS_DIR}" +require_dir "${SCRIPT_DIR}/skills" require_dir "${SCRIPT_DIR}/wheelhouse" +# Knowledge base directory, mounted into the container at /mnt/knowledge_base. require_dir "${SCRIPT_DIR}/backend/knowledge/knowledge_base" -require_wheel 'croniter-6.2.2-*.whl' -require_wheel 'hindsight_client-0.6.2-*.whl' -require_wheel 'redis-5.2.1-*.whl' - -if [ -f "${SCRIPT_DIR}/wheelhouse/SHA256SUMS" ]; then - require_cmd sha256sum - require_cmd sed - echo "Verifying wheelhouse checksums" - (cd "${SCRIPT_DIR}/wheelhouse" && sed 's/\r$//' SHA256SUMS | sha256sum -c -) -fi - mkdir -p "${SCRIPT_DIR}/data" -if [ "${PREFLIGHT_ONLY:-0}" = "1" ]; then - echo "Offline backend preflight passed. No container was changed." - exit 0 -fi - if ! docker image inspect "${IMAGE_TAG}" >/dev/null 2>&1; then require_file "${IMAGE_TAR}" echo "Loading Docker image from ${IMAGE_TAR}" @@ -110,29 +82,6 @@ if docker ps -a --format '{{.Names}}' | awk -v name="${CONTAINER_NAME}" '$0 == n docker rm -f "${CONTAINER_NAME}" >/dev/null fi -mount_args=( - -v "${SCRIPT_DIR}/backend/app:/app/backend/app" - -v "${SCRIPT_DIR}/backend/packages:/app/backend/packages" - -v "${SCRIPT_DIR}/backend/scripts:/app/backend/scripts" - -v "${SCRIPT_DIR}/backend/pyproject.toml:/app/backend/pyproject.toml:ro" - -v "${SCRIPT_DIR}/backend/langgraph.json:/app/backend/langgraph.json:ro" - -v "${SCRIPT_DIR}/config.yaml:/app/config.yaml:ro" - -v "${EXTENSIONS_CONFIG_FILE}:/app/extensions_config.json:ro" - -v "${SCRIPT_DIR}/skill_evolution.json:/app/skill_evolution.json:ro" - -v "${SKILLS_DIR}:/app/skills:ro" - -v "${SCRIPT_DIR}/wheelhouse:/app/wheelhouse:ro" - -v "${SCRIPT_DIR}/backend/knowledge/knowledge_base:/mnt/knowledge_base:ro" - -v "${SCRIPT_DIR}/data:/app/backend/.deer-flow" -) - -# uv.lock is not needed by uv run --no-sync, but mount it when the release -# bundle contains one so the container exactly reflects the online build. -if [ -f "${SCRIPT_DIR}/backend/uv.lock" ]; then - mount_args+=( - -v "${SCRIPT_DIR}/backend/uv.lock:/app/backend/uv.lock:ro" - ) -fi - echo "Starting backend container ${CONTAINER_NAME}" docker run -d \ --name "${CONTAINER_NAME}" \ @@ -147,33 +96,45 @@ docker run -d \ -e DEER_FLOW_SKILL_EVOLUTION_CONFIG_PATH=/app/skill_evolution.json \ -e DEER_FLOW_SKILLS_PATH=/app/skills \ -e UV_LINK_MODE=copy \ - "${mount_args[@]}" \ + -v "${SCRIPT_DIR}/backend/app:/app/backend/app" \ + -v "${SCRIPT_DIR}/backend/packages:/app/backend/packages" \ + -v "${SCRIPT_DIR}/backend/scripts:/app/backend/scripts" \ + -v "${SCRIPT_DIR}/backend/pyproject.toml:/app/backend/pyproject.toml:ro" \ + -v "${SCRIPT_DIR}/backend/uv.lock:/app/backend/uv.lock:ro" \ + -v "${SCRIPT_DIR}/backend/langgraph.json:/app/backend/langgraph.json:ro" \ + -v "${SCRIPT_DIR}/config.yaml:/app/config.yaml:ro" \ + -v "${SCRIPT_DIR}/extensions_config.json:/app/extensions_config.json:ro" \ + -v "${SCRIPT_DIR}/skill_evolution.json:/app/skill_evolution.json:ro" \ + -v "${SCRIPT_DIR}/skills:/app/skills:ro" \ + -v "${SCRIPT_DIR}/wheelhouse:/app/wheelhouse:ro" \ + -v "${SCRIPT_DIR}/backend/knowledge/knowledge_base:/mnt/knowledge_base:ro" \ + -v "${SCRIPT_DIR}/data:/app/backend/.deer-flow" \ "${IMAGE_TAG}" \ - sh -c 'set -eu -cd backend + sh -c 'cd backend +# Apply timezone inside the container. ``TZ`` alone is enough for Python / +# uvicorn log timestamps, but linking /etc/localtime keeps C-level tools +# (e.g. anything that reads /etc/localtime directly) on the same clock. if [ -n "${TZ:-}" ] && [ -f "/usr/share/zoneinfo/${TZ}" ]; then ln -sf "/usr/share/zoneinfo/${TZ}" /etc/localtime 2>/dev/null || true echo "${TZ}" > /etc/timezone 2>/dev/null || true fi - -if [ "${OFFLINE_RUNTIME_INSTALL_DEPS:-1}" != "0" ]; then - if ! uv run --no-sync python -c "import croniter, hindsight_client, redis" >/dev/null 2>&1; then - echo "Installing offline runtime packages from /app/wheelhouse" +if [ "${SCHEDULER_INSTALL_DEPS:-1}" != "0" ]; then + uv run --no-sync python -c "import croniter" >/dev/null 2>&1 || { PYTHON_BIN="$(uv run --no-sync python -c "import sys; print(sys.executable)")" - # Word splitting is intentional so .env may override this package list. - uv pip install \ - --python "${PYTHON_BIN}" \ - --no-index \ - --find-links /app/wheelhouse \ - ${OFFLINE_RUNTIME_PIP_PACKAGES:-croniter==6.2.2 hindsight-client==0.6.2 redis==5.2.1} - fi - uv run --no-sync python -c "import importlib.metadata as m; names=(\"croniter\", \"hindsight-client\", \"redis\"); print(\"Offline runtime packages:\", \", \".join(f\"{name}={m.version(name)}\" for name in names))" + uv pip install --python "${PYTHON_BIN}" --no-index --find-links /app/wheelhouse ${SCHEDULER_PIP_PACKAGES:-croniter==6.2.2} + } fi - -PYTHONPATH=. uv run --no-sync python -m uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 --workers "${GATEWAY_WORKERS:-2}"' +if [ "${HINDSIGHT_INSTALL_DEPS:-1}" != "0" ]; then + uv run --no-sync python -c "import hindsight_client" >/dev/null 2>&1 || { + PYTHON_BIN="$(uv run --no-sync python -c "import sys; print(sys.executable)")" + uv pip install --python "${PYTHON_BIN}" --no-index --find-links /app/wheelhouse ${HINDSIGHT_PIP_PACKAGES:-hindsight-client==0.6.2} + } +fi +uv run --no-sync python -c "import redis" >/dev/null 2>&1 || uv pip install --python "$(uv run --no-sync python -c "import sys; print(sys.executable)")" --no-index --find-links /app/wheelhouse ${REDIS_PIP_PACKAGES:-redis==5.2.1} +PYTHONPATH=. uv run --no-sync python -m uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 --workers ${GATEWAY_WORKERS:-2}' echo echo "Backend started." echo " URL: http://:${HOST_PORT}" echo " Logs: docker logs -f ${CONTAINER_NAME}" -echo " Stop: docker rm -f ${CONTAINER_NAME}" +echo " Stop: ./stop_backend.sh"