104 lines
5.9 KiB
Markdown
104 lines
5.9 KiB
Markdown
# Build networking, cache and CI
|
|
|
|
### Build networking and cache
|
|
|
|
The Dockerfile retains the pinned Python base, locked Python dependencies,
|
|
`fonts-noto-cjk`, Debian repository signature checks, and offline artifact
|
|
SHA256 verification. Production uses `scripts/build-verify.sh`; CI uses `scripts/ci-compose-verify.py`.
|
|
Both build the same Dockerfile with BuildKit and retain `docker compose build --pull`.
|
|
|
|
APT mirror hosts are configurable through `DEBIAN_MIRROR` and
|
|
`DEBIAN_SECURITY_MIRROR` (hosts only, without scheme or path). Defaults are
|
|
`deb.debian.org`; HTTPS is used for both Debian and security repositories.
|
|
For mainland-China NAS builds, use the shared profile:
|
|
|
|
```sh
|
|
. ./deploy/build-cn.env
|
|
./scripts/build-verify.sh
|
|
```
|
|
|
|
CI loads this same profile, using Tsinghua TUNA for both repositories unless
|
|
Gitea repository variables `DEBIAN_MIRROR` / `DEBIAN_SECURITY_MIRROR` override
|
|
it. Production may also set the two hosts directly in `.env` when calling the
|
|
build script without the profile. An alternate mainland mirror is
|
|
`mirrors.ustc.edu.cn`; availability and throughput depend on the Runner network.
|
|
No fallback disables signature verification or removes packages.
|
|
|
|
Layer order stays: pinned base → APT/font dependencies → `requirements.lock`
|
|
→ pip dependencies → application/vendor source → build metadata.
|
|
A source-only change therefore reuses the APT and pip layers, provided the
|
|
same builder, base digest, mirror arguments, and optional offline artifacts
|
|
are used. Changing the lockfile rebuilds pip and later layers, not APT.
|
|
Changing the mirror or APT instructions rebuilds APT and later layers.
|
|
|
|
Previously an APT layer miss redownloaded indexes/packages, and pip used
|
|
`--no-cache-dir`. Now locked BuildKit mounts retain APT indexes, downloaded
|
|
`.deb` files, and pip downloads/wheels outside the final image. Debian's
|
|
`docker-clean` hook is disabled so it cannot discard cached packages.
|
|
Re-executed APT steps still run `apt-get update` and validate repository
|
|
signatures; cached packages and pip artifacts can avoid repeat large downloads.
|
|
The initial build after this Dockerfile change will miss dependency layers;
|
|
subsequent builds benefit. No measured speedup is claimed before CI completes.
|
|
|
|
Caches belong to the Docker builder, not the ephemeral CI workspace. Keep the
|
|
NAS Runner's builder storage across jobs; removing the builder, pruning caches,
|
|
or using a fresh Docker daemon loses the benefit. Cache mounts are disposable:
|
|
an empty cache still builds correctly. `--pull` remains enabled; a pinned base
|
|
digest check alone does not invalidate dependency layers. To deliberately
|
|
refresh installed APT versions, use `docker compose build --pull --no-cache`.
|
|
|
|
### CI with a host Docker socket
|
|
|
|
Run `python scripts/ci-compose-verify.py` after loading the build mirror profile.
|
|
The script derives a temporary Compose document from production settings, with
|
|
an absolute client-side build context, a UUID project name, a unique CI image
|
|
tag, and project-scoped `data` / `config` named volumes. It removes production
|
|
bind mounts, the fixed container name, host ports, and the production `.env`
|
|
reference. A fresh test token, mock driver, and disabled device writes are
|
|
explicitly configured. Docker initializes empty named volumes from the image's
|
|
owned `/data` and `/config` directories; the service remains UID/GID 10333.
|
|
|
|
Build contexts are sent by the Docker client, so the job workspace can remain
|
|
inside the job container. Runtime storage is managed by the daemon; no matching
|
|
NAS host `/workspace` directory is needed. HTTP readiness and the unmodified
|
|
nine-endpoint smoke script run inside the service with `compose exec -T`.
|
|
Restart is followed by another readiness check and the same smoke test.
|
|
|
|
A `finally` block removes the test containers, network, volumes, and unique
|
|
image tag on success or failure. The workflow records the private temporary
|
|
Compose path in `GITHUB_ENV` and repeats cleanup in an `always()` step, covering
|
|
normal cancellation/interruption. Failed cleanup retains the manifest so the
|
|
fallback can retry. Abrupt host/daemon loss or a forcibly terminated entire job
|
|
can prevent both cleanup paths; no CI script can guarantee cleanup in that case.
|
|
No global prune or production resource removal is performed. Keep builder
|
|
caches: only the run-specific image tag is removed.
|
|
|
|
### CI Job pip indexes versus Docker build indexes
|
|
|
|
The Job bootstrap runs `python scripts/ci-install.py` before Ruff/mypy/tests.
|
|
It needs only the Python standard library, so it runs before PyYAML or tooling
|
|
is installed. `CI_PIP_INDEX_URL` and `CI_PIP_FALLBACK_INDEX_URL` are optional
|
|
Gitea repository variables. Defaults are the TUNA HTTPS PyPI simple endpoint
|
|
and official `https://pypi.org/simple`. Each is attempted separately; the
|
|
fallback repeats the unchanged editable `.[test]` installation, including
|
|
PEP 517 build isolation and `setuptools>=75`. No simultaneous extra index is
|
|
used, no build dependency is removed, and both failures fail CI.
|
|
|
|
Logs show inherited index/config/proxy hosts and the selected index host for
|
|
each attempt, with URL credentials and query strings removed. Bootstrap
|
|
disables inherited pip config files via `PIP_CONFIG_FILE=/dev/null`, explicitly
|
|
sets the selected index, and clears no-index/extra-index/find-links, insecure
|
|
trusted-host, and no-build-isolation environment settings. Constraints, TLS
|
|
certificate environment settings, and proxy settings remain in effect.
|
|
|
|
These `CI_PIP_*` settings apply only to the Job's pip install. They are not
|
|
passed into Docker build arguments or persisted into the production image.
|
|
The Dockerfile keeps its own cached/offline dependency installation and logs
|
|
the Docker pip index host for online installs. Debian mirror variables affect
|
|
APT only and are loaded later in the Docker verification step.
|
|
|
|
If dependency installation fails, the always() cleanup step still executes.
|
|
It skips the Python/PyYAML cleanup script when no CI Compose manifest exists,
|
|
as no Docker verification resources have been created at that point.
|
|
|