diff --git a/offline-backend-20260512/README.md b/offline-backend-20260512/README.md new file mode 100644 index 00000000..e77ac35a --- /dev/null +++ b/offline-backend-20260512/README.md @@ -0,0 +1,64 @@ +# DeerFlow 离线后端部署补充 + +本目录提供一套与 `start_backend_offline.sh` 配套的 wheelhouse 流程,用于在联网机器准备 Python 依赖,再将整个发布目录复制到无网络服务器运行。 + +## 文件 + +- `offline-runtime-requirements.txt`:启动时补装的固定版本依赖。 +- `prepare_wheelhouse.sh`:联网机器下载 Linux/Python 3.12 wheel,并验证传递依赖完整性。 +- `start_backend_offline.sh`:从本地 wheelhouse 补装依赖后启动 Docker 后端。 +- `wheelhouse/`:下载脚本生成;应与启动脚本一起复制到离线机。 + +## 联网机器准备 + +默认目标是后端镜像使用的 CPython 3.12、Linux x86_64: + +```bash +cd offline-backend-20260512 +bash prepare_wheelhouse.sh +``` + +脚本固定下载: + +- `croniter==6.2.2` +- `hindsight-client==0.6.2` +- `redis==5.2.1` +- 上述包在 Python 3.12/Linux x86_64 下的 19 个传递依赖;版本也已全部锁定在 `offline-runtime-requirements.txt` + +国内镜像示例: + +```bash +PIP_INDEX_URL=https://pypi.tuna.tsinghua.edu.cn/simple bash prepare_wheelhouse.sh +``` + +ARM64 镜像需要改目标平台: + +```bash +TARGET_PLATFORM=manylinux2014_aarch64 bash prepare_wheelhouse.sh +``` + +下载完成后,`wheelhouse/SHA256SUMS` 可用于离线机校验文件完整性。 + +## 离线机器启动 + +确保发布目录还包含 Docker 镜像 tar、`.env`、`config.yaml`、`skill_evolution.json` 和 `backend/`,然后运行。`extensions_config.json` 与 `skills/` 会优先从发布根目录读取,不存在时自动回退到 `backend/`: + +```bash +cd offline-backend-20260512 +bash start_backend_offline.sh +docker logs -f deerflow-backend +``` + +只检查目录、配置项、wheel 和 checksum,不操作容器: + +```bash +PREFLIGHT_ONLY=1 bash start_backend_offline.sh +``` + +启动脚本只使用 `/app/wheelhouse`,不会访问 PyPI。它会验证 checksum,并在镜像虚拟环境缺包时一次性安装三个固定版本的运行时包。 + +## Redis 说明 + +`redis==5.2.1` 只是 Python 客户端,不包含 Redis 服务端。`config.yaml` 中 `concurrency.redis.enabled: false` 时无需部署 Redis;启用后必须另行准备可访问的 Redis 服务或离线 Redis Docker 镜像。 + +如果 Redis 运行在另一个容器,`redis://127.0.0.1:6379/0` 指向的是后端容器自身,不能直接使用。应将两个容器加入同一个 Docker network,并把 URL 改成类似 `redis://redis:6379/0`,同时为不同环境设置不同的 `namespace`。 diff --git a/offline-backend-20260512/backend/CLAUDE.md b/offline-backend-20260512/backend/CLAUDE.md index 1cb9bcd3..839e9615 100644 --- a/offline-backend-20260512/backend/CLAUDE.md +++ b/offline-backend-20260512/backend/CLAUDE.md @@ -998,6 +998,7 @@ Both the `view_image` tool and `ViewImageMiddleware` are registered whenever `vi ## Documentation See `docs/` directory for detailed documentation: +- [../README.md](../README.md) - Offline Docker wheelhouse preparation and startup workflow. The pinned versions in `../offline-runtime-requirements.txt` and `../start_backend_offline.sh` must stay synchronized. The download target defaults to CPython 3.12 on Linux x86_64 and verifies dependency closure using wheelhouse only. - [CONFIGURATION.md](docs/CONFIGURATION.md) - Configuration options - [ARCHITECTURE.md](docs/ARCHITECTURE.md) - Architecture details - [API.md](docs/API.md) - API reference diff --git a/offline-backend-20260512/backend/README.md b/offline-backend-20260512/backend/README.md index 40e7e03b..2ffcacd6 100644 --- a/offline-backend-20260512/backend/README.md +++ b/offline-backend-20260512/backend/README.md @@ -407,6 +407,7 @@ uv run pytest - [Configuration Guide](docs/CONFIGURATION.md) - [Architecture Details](docs/ARCHITECTURE.md) - [API Reference](docs/API.md) +- [Offline Docker wheelhouse deployment (ZH)](../README.md) - [File Upload](docs/FILE_UPLOAD.md) - [Path Examples](docs/PATH_EXAMPLES.md) - [Context Summarization](docs/summarization.md) diff --git a/offline-backend-20260512/offline-runtime-requirements.txt b/offline-backend-20260512/offline-runtime-requirements.txt new file mode 100644 index 00000000..b493ed02 --- /dev/null +++ b/offline-backend-20260512/offline-runtime-requirements.txt @@ -0,0 +1,27 @@ +# Direct runtime packages layered into the prebuilt offline backend image. +# Keep these three pins synchronized with start_backend_offline.sh. +croniter==6.2.2 +hindsight-client==0.6.2 +redis==5.2.1 + +# Fully pinned dependency closure for CPython 3.12 on Linux. Keeping the +# transitive packages here makes a newly generated wheelhouse reproducible. +aiohappyeyeballs==2.7.1 +aiohttp==3.14.2 +aiohttp-retry==2.9.1 +aiosignal==1.4.0 +annotated-types==0.7.0 +async-timeout==5.0.1 +attrs==26.1.0 +frozenlist==1.8.0 +idna==3.18 +multidict==6.7.1 +propcache==0.5.2 +pydantic==2.13.4 +pydantic-core==2.46.4 +python-dateutil==2.9.0.post0 +six==1.17.0 +typing-extensions==4.16.0 +typing-inspection==0.4.2 +urllib3==2.7.0 +yarl==1.24.5 diff --git a/offline-backend-20260512/prepare_wheelhouse.sh b/offline-backend-20260512/prepare_wheelhouse.sh new file mode 100755 index 00000000..81cff94a --- /dev/null +++ b/offline-backend-20260512/prepare_wheelhouse.sh @@ -0,0 +1,99 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +PYTHON_BIN="${PYTHON_BIN:-python3}" +REQUIREMENTS_FILE="${REQUIREMENTS_FILE:-${SCRIPT_DIR}/offline-runtime-requirements.txt}" +WHEELHOUSE_DIR="${WHEELHOUSE_DIR:-${SCRIPT_DIR}/wheelhouse}" +PIP_INDEX_URL="${PIP_INDEX_URL:-https://pypi.org/simple}" + +# The supplied backend image is CPython 3.12 on Debian x86_64. These values can +# be overridden when building an offline bundle for a different image/CPU. +TARGET_PLATFORM="${TARGET_PLATFORM:-manylinux2014_x86_64}" +TARGET_PYTHON_VERSION="${TARGET_PYTHON_VERSION:-312}" +TARGET_IMPLEMENTATION="${TARGET_IMPLEMENTATION:-cp}" +TARGET_ABI="${TARGET_ABI:-cp312}" + +die() { + echo "ERROR: $*" >&2 + exit 1 +} + +require_cmd() { + command -v "$1" >/dev/null 2>&1 || die "command not found: $1" +} + +require_wheel() { + local pattern="$1" + compgen -G "${WHEELHOUSE_DIR}/${pattern}" >/dev/null || die "wheel not downloaded: ${pattern}" +} + +require_cmd "${PYTHON_BIN}" +require_cmd sha256sum +[ -f "${REQUIREMENTS_FILE}" ] || die "requirements file not found: ${REQUIREMENTS_FILE}" +"${PYTHON_BIN}" -m pip --version >/dev/null 2>&1 || die "pip is not available for ${PYTHON_BIN}" + +mkdir -p "${WHEELHOUSE_DIR}" + +download_args=( + download + --dest "${WHEELHOUSE_DIR}" + --only-binary=:all: + --platform "${TARGET_PLATFORM}" + --python-version "${TARGET_PYTHON_VERSION}" + --implementation "${TARGET_IMPLEMENTATION}" + --abi "${TARGET_ABI}" +) + +echo "Downloading Linux/Python runtime wheels" +echo " requirements: ${REQUIREMENTS_FILE}" +echo " destination: ${WHEELHOUSE_DIR}" +echo " platform: ${TARGET_PLATFORM}" +echo " Python/ABI: ${TARGET_PYTHON_VERSION}/${TARGET_ABI}" + +"${PYTHON_BIN}" -m pip "${download_args[@]}" \ + --index-url "${PIP_INDEX_URL}" \ + --requirement "${REQUIREMENTS_FILE}" + +require_wheel 'croniter-6.2.2-*.whl' +require_wheel 'hindsight_client-0.6.2-*.whl' +require_wheel 'redis-5.2.1-*.whl' + +# Resolve the same requirements a second time using only wheelhouse. This +# catches a missing transitive dependency before the bundle reaches offline. +verify_dir="$(mktemp -d)" +cleanup() { + if [ -n "${verify_dir:-}" ] && [ -d "${verify_dir}" ]; then + rm -rf -- "${verify_dir}" + fi +} +trap cleanup EXIT + +verify_args=( + download + --dest "${verify_dir}" + --no-index + --find-links "${WHEELHOUSE_DIR}" + --only-binary=:all: + --platform "${TARGET_PLATFORM}" + --python-version "${TARGET_PYTHON_VERSION}" + --implementation "${TARGET_IMPLEMENTATION}" + --abi "${TARGET_ABI}" + --requirement "${REQUIREMENTS_FILE}" +) +"${PYTHON_BIN}" -m pip "${verify_args[@]}" >/dev/null + +shopt -s nullglob +wheels=("${WHEELHOUSE_DIR}"/*.whl) +[ "${#wheels[@]}" -gt 0 ] || die "wheelhouse is empty: ${WHEELHOUSE_DIR}" +( + cd "${WHEELHOUSE_DIR}" + sha256sum -- *.whl | LC_ALL=C sort > SHA256SUMS +) + +echo +echo "Wheelhouse is complete for the pinned runtime requirements." +echo " wheel count: ${#wheels[@]}" +echo " checksums: ${WHEELHOUSE_DIR}/SHA256SUMS" +echo "Copy the wheelhouse directory with the deployment bundle." diff --git a/offline-backend-20260512/start_backend_offline.sh b/offline-backend-20260512/start_backend_offline.sh new file mode 100755 index 00000000..1952f766 --- /dev/null +++ b/offline-backend-20260512/start_backend_offline.sh @@ -0,0 +1,178 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +IMAGE_TAG="${IMAGE_TAG:-deerflow-backend-offline:20260505}" +CONTAINER_NAME="${CONTAINER_NAME:-deerflow-backend}" +HOST_PORT="${HOST_PORT:-8001}" +IMAGE_TAR="${IMAGE_TAR:-${SCRIPT_DIR}/deerflow-backend-offline-image-20260505.tar}" +CONTAINER_TZ="${CONTAINER_TZ:-${TZ:-Asia/Shanghai}}" + +default_extensions_config="${SCRIPT_DIR}/extensions_config.json" +if [ ! -f "${default_extensions_config}" ]; then + default_extensions_config="${SCRIPT_DIR}/backend/extensions_config.json" +fi +EXTENSIONS_CONFIG_FILE="${EXTENSIONS_CONFIG_FILE:-${default_extensions_config}}" + +default_skills_dir="${SCRIPT_DIR}/skills" +if [ ! -d "${default_skills_dir}" ]; then + default_skills_dir="${SCRIPT_DIR}/backend/skills" +fi +SKILLS_DIR="${SKILLS_DIR:-${default_skills_dir}}" + +die() { + echo "ERROR: $*" >&2 + exit 1 +} + +require_cmd() { + command -v "$1" >/dev/null 2>&1 || die "command not found: $1" +} + +require_file() { + [ -f "$1" ] || die "required file not found: $1" +} + +require_dir() { + [ -d "$1" ] || die "required directory not found: $1" +} + +require_env_key() { + local key="$1" + grep -Eq "^[[:space:]]*${key}=" "${SCRIPT_DIR}/.env" || die "required .env key not found: ${key}" +} + +require_wheel() { + local pattern="$1" + compgen -G "${SCRIPT_DIR}/wheelhouse/${pattern}" >/dev/null || die "required wheel not found: wheelhouse/${pattern}" +} + +require_cmd docker + +if [ ! -f "${SCRIPT_DIR}/.env" ]; then + if [ -f "${SCRIPT_DIR}/.env.example" ]; then + cp "${SCRIPT_DIR}/.env.example" "${SCRIPT_DIR}/.env" + echo "Generated .env from .env.example. Please edit .env and run this script again." + exit 1 + fi + die "missing .env" +fi + +if [ ! -f "${SCRIPT_DIR}/config.yaml" ]; then + if [ -f "${SCRIPT_DIR}/config.yaml.example" ]; then + cp "${SCRIPT_DIR}/config.yaml.example" "${SCRIPT_DIR}/config.yaml" + echo "Generated config.yaml from config.yaml.example. Please edit model/API settings and run this script again." + exit 1 + fi + die "missing config.yaml" +fi + +require_env_key AUTH_JWT_SECRET +require_env_key SCHEDULED_TASKS_ENABLED +require_env_key ELEMENT_ENABLED +require_file "${SCRIPT_DIR}/config.yaml" +require_file "${EXTENSIONS_CONFIG_FILE}" +require_file "${SCRIPT_DIR}/skill_evolution.json" +require_file "${SCRIPT_DIR}/backend/pyproject.toml" +require_file "${SCRIPT_DIR}/backend/app/gateway/app.py" +require_file "${SCRIPT_DIR}/backend/packages/harness/deerflow/runtime/scheduler/element.py" +require_file "${SCRIPT_DIR}/backend/langgraph.json" +require_dir "${SKILLS_DIR}" +require_dir "${SCRIPT_DIR}/wheelhouse" +require_dir "${SCRIPT_DIR}/backend/knowledge/knowledge_base" +require_wheel 'croniter-6.2.2-*.whl' +require_wheel 'hindsight_client-0.6.2-*.whl' +require_wheel 'redis-5.2.1-*.whl' + +if [ -f "${SCRIPT_DIR}/wheelhouse/SHA256SUMS" ]; then + require_cmd sha256sum + echo "Verifying wheelhouse checksums" + (cd "${SCRIPT_DIR}/wheelhouse" && sha256sum -c SHA256SUMS) +fi + +mkdir -p "${SCRIPT_DIR}/data" + +if [ "${PREFLIGHT_ONLY:-0}" = "1" ]; then + echo "Offline backend preflight passed. No container was changed." + exit 0 +fi + +if ! docker image inspect "${IMAGE_TAG}" >/dev/null 2>&1; then + require_file "${IMAGE_TAR}" + echo "Loading Docker image from ${IMAGE_TAR}" + docker load -i "${IMAGE_TAR}" +fi + +if docker ps -a --format '{{.Names}}' | awk -v name="${CONTAINER_NAME}" '$0 == name { found=1 } END { exit(found ? 0 : 1) }'; then + echo "Removing existing container ${CONTAINER_NAME}" + docker rm -f "${CONTAINER_NAME}" >/dev/null +fi + +mount_args=( + -v "${SCRIPT_DIR}/backend/app:/app/backend/app" + -v "${SCRIPT_DIR}/backend/packages:/app/backend/packages" + -v "${SCRIPT_DIR}/backend/scripts:/app/backend/scripts" + -v "${SCRIPT_DIR}/backend/pyproject.toml:/app/backend/pyproject.toml:ro" + -v "${SCRIPT_DIR}/backend/langgraph.json:/app/backend/langgraph.json:ro" + -v "${SCRIPT_DIR}/config.yaml:/app/config.yaml:ro" + -v "${EXTENSIONS_CONFIG_FILE}:/app/extensions_config.json:ro" + -v "${SCRIPT_DIR}/skill_evolution.json:/app/skill_evolution.json:ro" + -v "${SKILLS_DIR}:/app/skills:ro" + -v "${SCRIPT_DIR}/wheelhouse:/app/wheelhouse:ro" + -v "${SCRIPT_DIR}/backend/knowledge/knowledge_base:/mnt/knowledge_base:ro" + -v "${SCRIPT_DIR}/data:/app/backend/.deer-flow" +) + +# uv.lock is not needed by uv run --no-sync, but mount it when the release +# bundle contains one so the container exactly reflects the online build. +if [ -f "${SCRIPT_DIR}/backend/uv.lock" ]; then + mount_args+=( + -v "${SCRIPT_DIR}/backend/uv.lock:/app/backend/uv.lock:ro" + ) +fi + +echo "Starting backend container ${CONTAINER_NAME}" +docker run -d \ + --name "${CONTAINER_NAME}" \ + --restart unless-stopped \ + -p "${HOST_PORT}:8001" \ + --env-file "${SCRIPT_DIR}/.env" \ + -e TZ="${CONTAINER_TZ}" \ + -e DEER_FLOW_PROJECT_ROOT=/app \ + -e DEER_FLOW_HOME=/app/backend/.deer-flow \ + -e DEER_FLOW_CONFIG_PATH=/app/config.yaml \ + -e DEER_FLOW_EXTENSIONS_CONFIG_PATH=/app/extensions_config.json \ + -e DEER_FLOW_SKILL_EVOLUTION_CONFIG_PATH=/app/skill_evolution.json \ + -e DEER_FLOW_SKILLS_PATH=/app/skills \ + -e UV_LINK_MODE=copy \ + "${mount_args[@]}" \ + "${IMAGE_TAG}" \ + sh -c 'set -eu +cd backend +if [ -n "${TZ:-}" ] && [ -f "/usr/share/zoneinfo/${TZ}" ]; then + ln -sf "/usr/share/zoneinfo/${TZ}" /etc/localtime 2>/dev/null || true + echo "${TZ}" > /etc/timezone 2>/dev/null || true +fi + +if [ "${OFFLINE_RUNTIME_INSTALL_DEPS:-1}" != "0" ]; then + if ! uv run --no-sync python -c "import croniter, hindsight_client, redis" >/dev/null 2>&1; then + echo "Installing offline runtime packages from /app/wheelhouse" + PYTHON_BIN="$(uv run --no-sync python -c "import sys; print(sys.executable)")" + # Word splitting is intentional so .env may override this package list. + uv pip install \ + --python "${PYTHON_BIN}" \ + --no-index \ + --find-links /app/wheelhouse \ + ${OFFLINE_RUNTIME_PIP_PACKAGES:-croniter==6.2.2 hindsight-client==0.6.2 redis==5.2.1} + fi + uv run --no-sync python -c "import importlib.metadata as m; names=(\"croniter\", \"hindsight-client\", \"redis\"); print(\"Offline runtime packages:\", \", \".join(f\"{name}={m.version(name)}\" for name in names))" +fi + +PYTHONPATH=. uv run --no-sync python -m uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 --workers "${GATEWAY_WORKERS:-2}"' + +echo +echo "Backend started." +echo " URL: http://:${HOST_PORT}" +echo " Logs: docker logs -f ${CONTAINER_NAME}" +echo " Stop: docker rm -f ${CONTAINER_NAME}"