Add opt-in production CD with daemon lock, verified rollback and safe planning
This commit is contained in:
Pigway
@@ -0,0 +1,169 @@
|
||||
# Gitea Actions production CD v1
|
||||
|
||||
## Audit and adoption boundary
|
||||
|
||||
Production `docker-compose.yml` is unchanged: AMD64 image, UID/GID 10333,
|
||||
`unless-stopped`, read-only rootfs, `/tmp` tmpfs, dropped capabilities, limited
|
||||
logs, `/data` and `/config` bind mounts, default private bridge, and configurable
|
||||
host bind/port. SQLite/images/generated/cache live under `/data`; accounts,
|
||||
vendor sessions and configuration live under `/config`. Neither is disposable.
|
||||
The optional photos mount is read-only and has supplementary group access.
|
||||
|
||||
Historical repository receipts mention `/volume2/docker/pigway-frame-flow/data`,
|
||||
`/volume2/docker/pigway-frame-flow/config`, `/volume1/photo`, port 18787, and
|
||||
container/project `pigway-frame-flow`. These are **confirmation candidates**, not
|
||||
script defaults. The deployment script accepts only explicitly configured paths
|
||||
that match the existing running container on the Runner's Docker daemon.
|
||||
It rejects different projects, ports, unknown mounts, host/custom networking,
|
||||
missing photos confirmation, or different user/restart policy. It does not
|
||||
create NAS host directories or migrate/adopt an unrelated container.
|
||||
|
||||
CI uses `ffci-*` projects, per-run image tags and named volumes, no host ports.
|
||||
Its cleanup does not refer to any production resource. Production uses fixed
|
||||
configured names and existing bind mounts, with `create_host_path: false`.
|
||||
|
||||
## Gitea configuration
|
||||
|
||||
Repository Settings → Actions → Variables (ordinary configuration):
|
||||
|
||||
| Variable | Value to explicitly confirm |
|
||||
| --- | --- |
|
||||
| `PROD_DEPLOY_ENABLED` | Initially `false`; change to `true` only after plan acceptance |
|
||||
| `PROD_PROJECT` | Existing Compose project label; candidate `pigway-frame-flow` |
|
||||
| `PROD_CONTAINER` | Existing container name; candidate `pigway-frame-flow` |
|
||||
| `PROD_IMAGE_REPO` | Fixed local image repository; suggested `pigway-frame-flow` |
|
||||
| `PROD_DATA_DIR` | Existing absolute NAS host data path; no default |
|
||||
| `PROD_CONFIG_DIR` | Existing absolute NAS host config path; no default |
|
||||
| `PROD_PHOTOS_DIR` | Existing read-only photos source if present; otherwise empty |
|
||||
| `PROD_WEB_BIND` | Exact existing Docker HostIp (`0.0.0.0` or NAS LAN IP etc.) |
|
||||
| `PROD_WEB_PORT` | Exact existing mapped port |
|
||||
| `PROD_STATE_VOLUME` | Dedicated new deployment journal volume, e.g. `pigway-frame-flow-cd-state` |
|
||||
| `PROD_MOCK` | Explicit `true`/`false`, consistent with intended production mode |
|
||||
| `PROD_WRITE_ENABLED` | Explicit `true`/`false`; preserve owner's existing write policy |
|
||||
|
||||
The state volume must be separate from application/CI storage. An existing
|
||||
volume without the matching `frameflow.deploy.project` label is rejected.
|
||||
Supplementary groups are read from the existing container; no host ACL changes.
|
||||
Keep existing `DEBIAN_MIRROR`, `DEBIAN_SECURITY_MIRROR`, `CI_PIP_INDEX_URL`, and
|
||||
`CI_PIP_FALLBACK_INDEX_URL` settings from Verify; CI/production builds share the
|
||||
Dockerfile, pinned base, lockfile, and BuildKit caches.
|
||||
|
||||
Repository Settings → Actions → Secrets:
|
||||
|
||||
`PRODUCTION_ENV`: multiline **unquoted** `KEY=value` entries, without shell
|
||||
expansion. At least:
|
||||
|
||||
```dotenv
|
||||
FRAME_FLOW_TOKEN=<existing Frame Flow administrator token>
|
||||
KOKONNA_API_KEY=<existing KoKonna key, or empty when mock>
|
||||
```
|
||||
|
||||
Optional additional runtime secret variables can be included. No key values are
|
||||
committed, passed to image builds, or printed. Nonsecret mock/write flags use
|
||||
Variables. Other existing runtime environment values are retained. The script
|
||||
sets commit/build time and internal data/config paths explicitly. Dollar signs
|
||||
in secret values are escaped before Compose interpolation. Snapshots containing
|
||||
old runtime secrets are stored only in a private journal file (0600, directory
|
||||
0700) on the dedicated Docker volume; Docker administrators can access it.
|
||||
Never upload that journal or a full Docker inspect as a public CI artifact.
|
||||
|
||||
## First safe trial
|
||||
|
||||
1. On NAS, inspect only the necessary fields (use existing Docker/sudo access):
|
||||
```sh
|
||||
docker inspect pigway-frame-flow --format '{{index .Config.Labels "com.docker.compose.project"}}'
|
||||
docker inspect pigway-frame-flow --format '{{json .Mounts}}'
|
||||
docker inspect pigway-frame-flow --format '{{json .HostConfig.PortBindings}} {{.HostConfig.NetworkMode}}'
|
||||
```
|
||||
Confirm the Variables against these actual values, including photos.
|
||||
2. Configure the Secret privately. Keep `PROD_DEPLOY_ENABLED=false`.
|
||||
3. Manually run **Verify Frame Flow** on `main`, `deploy_action=plan`.
|
||||
Verify must pass first. Plan checks resource identity, paths/port, required
|
||||
Secret keys, current `/ready`, smoke endpoints and commit. It performs no
|
||||
build, stop, restart, tagging, lock creation, volume creation, or data writes.
|
||||
If no production Variables exist, it reports configuration missing and does
|
||||
nothing; that is an unconfigured plan, not production acceptance.
|
||||
4. Review `PLAN ONLY` output and arrange a maintenance window for the first real
|
||||
deploy/restart. Set `PROD_DEPLOY_ENABLED=true`, manually run `deploy_action=deploy`.
|
||||
5. After `DEPLOY VERIFIED`, subsequent pushes to `main` run Verify → deploy.
|
||||
Pushes to other branches and pull requests never deploy. A manual run defaults
|
||||
to plan, even when automatic deployment is enabled.
|
||||
|
||||
The script intentionally supports updates to an existing healthy deployment,
|
||||
not first-time bootstrap. No real production deployment or rollback drill has
|
||||
been performed during implementation; those are first-run acceptance steps.
|
||||
|
||||
## Deployment transaction and rollback
|
||||
|
||||
1. Verify job must succeed completely, including isolated Docker tests/cleanup.
|
||||
2. Deploy job checks out the same revision and prepares tools. An atomic
|
||||
`docker create` reservation named `<PROD_CONTAINER>-deploy-lock` serializes
|
||||
production changes on the daemon. Job `concurrency` is also declared, but
|
||||
correctness does not depend on Gitea implementing it. A competing deploy
|
||||
fails safely instead of changing production. No other job may remove a lock
|
||||
whose owner label differs from its own.
|
||||
3. Re-read the current baseline under the lock; confirm ready/smoke. Record exact
|
||||
image ID, runtime Compose/environment, version/commit. Pin the old image with
|
||||
an immutable rollback tag so normal dangling-image cleanup cannot remove it.
|
||||
4. Refuse a stale checked-out `main` revision that no longer equals remote main.
|
||||
Build an immutable `release-<commit>-<owner>` tag locally, without a registry
|
||||
and without changing `latest` or CI image tags.
|
||||
5. Persist a `prepared` journal with old/new runtime configs. Create an additive
|
||||
SQLite online backup under `/data/deploy-backups/<owner>.db`. No data/config
|
||||
directory or file is removed, replaced or recursively chowned.
|
||||
6. `compose up -d --no-build --pull never` replaces only the configured service.
|
||||
Check `/ready`, the unchanged nine-endpoint read-only smoke test, and exact
|
||||
`/api/v1/version` commit. Only all checks passing produce `DEPLOY VERIFIED`.
|
||||
7. Any activation/verification failure runs `up` again with the saved exact old
|
||||
image ID and old runtime environment/mounts, then repeats ready/smoke/commit
|
||||
verification. A failed deploy remains a failed workflow even if rollback
|
||||
succeeds. If rollback fails, the journal/lock are retained, failure is explicit,
|
||||
and redacted scoped diagnostics are emitted.
|
||||
8. An `always()` step uses the recorded owner to recover an interrupted prepared
|
||||
transaction and release only its own lock. Abrupt host loss or termination of
|
||||
the entire runner can prevent both handlers: inspect the journal/lock before
|
||||
manually recovering; never automatically force-unlock an unknown owner.
|
||||
|
||||
Rollback is an **image/config rollback**, not an automatic database rewind.
|
||||
User data accumulated during deployment remains. The SQLite backup is for
|
||||
operator-reviewed recovery; destructive or backward-incompatible future schema
|
||||
changes must not be auto-deployed without a separate migration/backup plan.
|
||||
Baseline artifacts and additive backups are retained for the operator; there is
|
||||
no automated pruning of production rollback history or data.
|
||||
|
||||
After enabling CD, use its workflow/script as the production lifecycle owner.
|
||||
The old NAS checkout's Compose/.env files are not rewritten remotely by CD and
|
||||
may describe an older deployment. Do not concurrently recreate production from
|
||||
that old checkout. The repository's standalone Compose remains usable for an
|
||||
independent deployment; the private journal records the CD runtime configuration.
|
||||
|
||||
## Actual commit and manual rollback
|
||||
|
||||
```sh
|
||||
curl -fsS http://NAS_LAN_IP:PORT/api/v1/version
|
||||
# Check `commit`, `version`, and `build_time` against successful deployment output.
|
||||
docker inspect PRODUCTION_CONTAINER --format '{{.Image}}'
|
||||
```
|
||||
|
||||
For a manual rollback: keep CD enabled and run **Verify Frame Flow** on `main`
|
||||
with `deploy_action=rollback`. After Verify passes, the script locks production,
|
||||
loads the last journal's `previous` image/config and verifies recovery. This is
|
||||
last-baseline rollback, not an arbitrary image selector. Never use `down -v`,
|
||||
volume pruning, or delete data/config to roll back.
|
||||
|
||||
If a failed run retained a lock, inspect only its owner label:
|
||||
|
||||
```sh
|
||||
docker inspect PRODUCTION_CONTAINER-deploy-lock --format '{{index .Config.Labels "frameflow.deploy.owner"}}'
|
||||
```
|
||||
|
||||
With the documented Variables configured in a trusted Docker-enabled shell:
|
||||
|
||||
```sh
|
||||
FRAME_FLOW_DEPLOY_OWNER=<confirmed failed owner> CD_ACTION=recover python scripts/production-deploy.py
|
||||
```
|
||||
|
||||
`recover` retries that owner's journaled rollback and retains the lock again if
|
||||
recovery fails. If the very first attempt stopped before a journal was written,
|
||||
confirm that the original image is still running and healthy before an operator
|
||||
removes the stale lock. Do not delete a live lock or any persistence volume.
|
||||
Reference in New Issue
Block a user