pigway-frame-flow/docs/deployment/GITEA_CD.md

# Gitea Actions production CD v1

## Audit and adoption boundary

Production `docker-compose.yml` is unchanged: AMD64 image, UID/GID 10333,
`unless-stopped`, read-only rootfs, `/tmp` tmpfs, dropped capabilities, limited
logs, `/data` and `/config` bind mounts, default private bridge, and configurable
host bind/port. SQLite/images/generated/cache live under `/data`; accounts,
vendor sessions and configuration live under `/config`. Neither is disposable.
The optional photos mount is read-only and has supplementary group access.

Historical repository receipts mention `/volume2/docker/pigway-frame-flow/data`,
`/volume2/docker/pigway-frame-flow/config`, `/volume1/photo`, port 18787, and
container/project `pigway-frame-flow`. These are **confirmation candidates**, not
script defaults. The deploy action accepts only explicitly configured paths
that match the existing running container on the Runner's Docker daemon.
It rejects different projects, ports, unknown mounts, host/custom networking,
missing photos confirmation, or different user/restart policy. The deploy action does not create NAS host directories or migrate/adopt an unrelated
container. First-time directory creation is limited to the manual bootstrap below.

CI uses `ffci-*` projects, per-run image tags and named volumes, no host ports.
Its cleanup does not refer to any production resource. Production uses fixed
configured names and existing bind mounts, with `create_host_path: false`.

## Gitea configuration

Repository Settings → Actions → Variables (ordinary configuration):

| Variable | Value to explicitly confirm |
| --- | --- |
| `PROD_DEPLOY_ENABLED` | Initially `false`; change to `true` only after plan acceptance |
| `PROD_PROJECT` | Existing Compose project label; candidate `pigway-frame-flow` |
| `PROD_CONTAINER` | Existing container name; candidate `pigway-frame-flow` |
| `PROD_IMAGE_REPO` | Fixed local image repository; suggested `pigway-frame-flow` |
| `PROD_DATA_DIR` | Existing absolute NAS host data path; no default |
| `PROD_CONFIG_DIR` | Existing absolute NAS host config path; no default |
| `PROD_PHOTOS_DIR` | Existing read-only photos source if present; otherwise empty |
| `PROD_WEB_BIND` | Exact existing Docker HostIp (`0.0.0.0` or NAS LAN IP etc.) |
| `PROD_WEB_PORT` | Exact existing mapped port |
| `PROD_STATE_VOLUME` | Dedicated new deployment journal volume, e.g. `pigway-frame-flow-cd-state` |
| `PROD_MOCK` | Explicit `true`/`false`, consistent with intended production mode |
| `PROD_WRITE_ENABLED` | Explicit `true`/`false`; preserve owner's existing write policy |

The state volume must be separate from application/CI storage. An existing
volume without the matching `frameflow.deploy.project` label is rejected.
Supplementary groups are read from the existing container; no host ACL changes.
Keep existing `DEBIAN_MIRROR`, `DEBIAN_SECURITY_MIRROR`, `CI_PIP_INDEX_URL`, and
`CI_PIP_FALLBACK_INDEX_URL` settings from Verify; CI/production builds share the
Dockerfile, pinned base, lockfile, and BuildKit caches.

Repository Settings → Actions → Secrets:

`PRODUCTION_ENV`: multiline **unquoted** `KEY=value` entries, without shell
expansion. At least:

```dotenv
FRAME_FLOW_TOKEN=<existing Frame Flow administrator token>
KOKONNA_API_KEY=<existing KoKonna key, or empty when mock>
```

Optional additional runtime secret variables can be included. No key values are
committed, passed to image builds, or printed. Nonsecret mock/write flags use
Variables. Other existing runtime environment values are retained. The script
sets commit/build time and internal data/config paths explicitly. Dollar signs
in secret values are escaped before Compose interpolation. Snapshots containing
old runtime secrets are stored only in a private journal file (0600, directory
0700) on the dedicated Docker volume; Docker administrators can access it.
Never upload that journal or a full Docker inspect as a public CI artifact.

## First safe trial

1. On NAS, inspect only the necessary fields (use existing Docker/sudo access):
   ```sh
   docker inspect pigway-frame-flow --format '{{index .Config.Labels "com.docker.compose.project"}}'
   docker inspect pigway-frame-flow --format '{{json .Mounts}}'
   docker inspect pigway-frame-flow --format '{{json .HostConfig.PortBindings}} {{.HostConfig.NetworkMode}}'
   ```
   Confirm the Variables against these actual values, including photos.
2. Configure the Secret privately. Keep `PROD_DEPLOY_ENABLED=false`.
3. Manually run **Verify Frame Flow** on `main`, `deploy_action=plan`.
   Verify must pass first. Plan checks resource identity, paths/port, required
   Secret keys, current `/ready`, smoke endpoints and commit. It performs no
   build, stop, restart, tagging, lock creation, volume creation, or data writes.
   If no production Variables exist, it reports configuration missing and does
   nothing; that is an unconfigured plan, not production acceptance.
4. Review `PLAN ONLY` output and arrange a maintenance window for the first real
   deploy/restart. Set `PROD_DEPLOY_ENABLED=true`, manually run `deploy_action=deploy`.
5. After `DEPLOY VERIFIED`, subsequent pushes to `main` run Verify → deploy.
   Pushes to other branches and pull requests never deploy. A manual run defaults
   to plan, even when automatic deployment is enabled.

The deploy action supports updates to an existing healthy deployment. For first-time
creation use the explicit manual bootstrap flow below. No real production deployment or rollback drill has
been performed during implementation; those are first-run acceptance steps.

## Deployment transaction and rollback

1. Verify job must succeed completely, including isolated Docker tests/cleanup.
2. Deploy job checks out the same revision and prepares tools. An atomic
   `docker create` reservation named `<PROD_CONTAINER>-deploy-lock` serializes
   production changes on the daemon. Job `concurrency` is also declared, but
   correctness does not depend on Gitea implementing it. A competing deploy
   fails safely instead of changing production. No other job may remove a lock
   whose owner label differs from its own.
3. Re-read the current baseline under the lock; confirm ready/smoke. Record exact
   image ID, runtime Compose/environment, version/commit. Pin the old image with
   an immutable rollback tag so normal dangling-image cleanup cannot remove it.
4. Refuse a stale checked-out `main` revision that no longer equals remote main.
   Build an immutable `release-<commit>-<owner>` tag locally, without a registry
   and without changing `latest` or CI image tags.
5. Persist a `prepared` journal with old/new runtime configs. Create an additive
   SQLite online backup under `/data/deploy-backups/<owner>.db`. No data/config
   directory or file is removed, replaced or recursively chowned.
6. `compose up -d --no-build --pull never` replaces only the configured service.
   Check `/ready`, the unchanged nine-endpoint read-only smoke test, and exact
   `/api/v1/version` commit. Only all checks passing produce `DEPLOY VERIFIED`.
7. Any activation/verification failure runs `up` again with the saved exact old
   image ID and old runtime environment/mounts, then repeats ready/smoke/commit
   verification. A failed deploy remains a failed workflow even if rollback
   succeeds. If rollback fails, the journal/lock are retained, failure is explicit,
   and redacted scoped diagnostics are emitted.
8. An `always()` step uses the recorded owner to recover an interrupted prepared
   transaction and release only its own lock. Abrupt host loss or termination of
   the entire runner can prevent both handlers: inspect the journal/lock before
   manually recovering; never automatically force-unlock an unknown owner.

Rollback is an **image/config rollback**, not an automatic database rewind.
User data accumulated during deployment remains. The SQLite backup is for
operator-reviewed recovery; destructive or backward-incompatible future schema
changes must not be auto-deployed without a separate migration/backup plan.
Baseline artifacts and additive backups are retained for the operator; there is
no automated pruning of production rollback history or data.

After enabling CD, use its workflow/script as the production lifecycle owner.
The old NAS checkout's Compose/.env files are not rewritten remotely by CD and
may describe an older deployment. Do not concurrently recreate production from
that old checkout. The repository's standalone Compose remains usable for an
independent deployment; the private journal records the CD runtime configuration.

## Actual commit and manual rollback

```sh
curl -fsS http://NAS_LAN_IP:PORT/api/v1/version
# Check `commit`, `version`, and `build_time` against successful deployment output.
docker inspect PRODUCTION_CONTAINER --format '{{.Image}}'
```

For a manual rollback: keep CD enabled and run **Verify Frame Flow** on `main`
with `deploy_action=rollback`. After Verify passes, the script locks production,
loads the last journal's `previous` image/config and verifies recovery. This is
last-baseline rollback, not an arbitrary image selector. Never use `down -v`,
volume pruning, or delete data/config to roll back.

If a failed run retained a lock, inspect only its owner label:

```sh
docker inspect PRODUCTION_CONTAINER-deploy-lock --format '{{index .Config.Labels "frameflow.deploy.owner"}}'
```

With the documented Variables configured in a trusted Docker-enabled shell:

```sh
FRAME_FLOW_DEPLOY_OWNER=<confirmed failed owner> CD_ACTION=recover python scripts/production-deploy.py
```

`recover` retries that owner's journaled rollback and retains the lock again if
recovery fails. If the very first attempt stopped before a journal was written,
confirm that the original image is still running and healthy before an operator
removes the stale lock. Do not delete a live lock or any persistence volume.

## First-time production bootstrap (explicit manual action)

The confirmed NAS residue is a `created`, never-started container with dead
`/workspace/way/...` bind paths. It is not an adoption/rollback baseline.
No migration from those paths is performed. Production Compose remains independently usable.

Set these repository Actions **Variables** (not Secrets):

```text
PROD_DEPLOY_ENABLED=true
PROD_PROJECT=pigway-frame-flow
PROD_CONTAINER=pigway-frame-flow
PROD_IMAGE_REPO=pigway-frame-flow
PROD_DATA_DIR=/volume2/docker/pigway-frame-flow/data
PROD_CONFIG_DIR=/volume2/docker/pigway-frame-flow/config
PROD_WEB_BIND=192.0.2.10
PROD_WEB_PORT=18787
PROD_STATE_VOLUME=pigway-frame-flow-cd-state
PROD_MOCK=true
PROD_WRITE_ENABLED=false
```

Leave `PROD_PHOTOS_DIR` unset/empty. Existing mirror Variables still apply.
Enabling CD does not enable real frame writes. Before the first successful bootstrap,
ordinary main pushes report `production not bootstrapped / bootstrap required`.
They cannot create production. `bootstrap` additionally requires the Actions event
`workflow_dispatch`; the default action remains `plan`. Every action is gated by
successful complete Verify on main.

Create repository Actions **Secret** `PRODUCTION_ENV` as multiline bare `KEY=value`:

```dotenv
FRAME_FLOW_TOKEN=replace-with-a-long-random-private-token
KOKONNA_API_KEY=
```

Do not add shell quotes, `export`, interpolation or spaces around `=`. An existing
KoKonna key may be supplied but first bootstrap forcibly uses Mock with writes off.
Optional application secrets/settings may be added in the same format; bootstrap
always overrides Mock/write flags, data/config paths, timezone and build identity.
This token remains useful for API/smoke checks even if Web UI uses account login.
Do not paste the real Secret into logs or repository files.

### Safety and cleanup

Bootstrap builds the verified main revision first, then acquires the same atomic
Docker daemon deployment lock used by deploy. Host path checks run in a short-lived,
network-disabled helper with **read-only** NAS root mounted, not CI workspace paths.
Only explicitly configured data/config directories are subsequently mounted writable
for initialization, owned `10333:10333`, mode `0750`. Nonempty directories, symlinks
(including ancestor symlinks), overlapping paths and unknown journal resources fail closed.
Docker may create missing ancestor directories while creating these exact bind sources;
there is no recursive chmod/chown/delete on their parents.

A same-name residue can be removed **without force** only after checking all of:
Compose project/service labels; `Status=created`; `Running=false`;
`StartedAt=0001-01-01T00:00:00Z`; exactly two bind mounts `/data` and `/config`;
and both old host sources absent. A running, previously started, unknown or data-bearing
container is refused. An existing default network must be owned by this Compose
project and contain no endpoints other than the validated residue.

The production service uses `create_host_path=false`, the explicit NAS LAN bind,
non-root UID, `unless-stopped`, normal bridge network and existing resource/security
limits. Acceptance runs ready, all read-only smoke endpoints, version/commit,
preflight mount/port/project/user/restart checks, one restart, repeated checks,
Docker health and exact built-image ID before `BOOTSTRAP VERIFIED`.

On failure after transaction preparation, cleanup removes only this transaction's
owner-labelled new container and its newly created, empty, project-labelled network.
The previously validated never-started residue has no rollback value and is not recreated.
**Data/config files, images/build cache and the private CD journal volume are retained.**
No `down -v`, volume prune or removal of unrelated/CI resources is used.
Cleanup failure retains the lock so the always recovery step can retry using the journal.
A failed bootstrap journal prevents automatic retry: review preserved data and diagnostics
before another bootstrap; do not automatically empty those directories or delete the journal.
Abrupt host/runner loss still requires operator review of the lock/journal.
The journal is private (0700/0600) and may contain prior/candidate secrets; never export it
as an Actions artifact. First bootstrap has no prior healthy version to roll back to.

### First run and acceptance

1. In Gitea repository **Actions → Verify Frame Flow → Run workflow**, choose branch
   `main`, action **plan**. This is read-only and reports bootstrap required for the
   absent/created instance; it is not a bootstrap safety acceptance. Detailed host
   residue/directory checks happen under the lock in bootstrap.
2. Run the same workflow on `main` with action **bootstrap**. Verify runs in full first.
   No NAS bootstrap is executed merely by committing this implementation.
3. Require `BOOTSTRAP VERIFIED` in the deploy log. In Container Manager confirm
   container `pigway-frame-flow` is running/healthy, user 10333, restart unless-stopped,
   exact `/volume2/docker/pigway-frame-flow/{data,config}` mounts and
   `192.0.2.10:18787` binding (not loopback).
4. Open `http://192.0.2.10:18787/api/v1/version`; its `commit` must equal the full
   workflow checkout commit. Check `/ready` returns HTTP 200. The immutable image tag
   is `pigway-frame-flow:release-<commit>-<transaction>`; compare its image ID in
   Container Manager with the running container.
5. Keep `PROD_DEPLOY_ENABLED=true`. Subsequent main pushes Verify then deploy using
   this same runtime and journal mechanism. The first update records bootstrap as
   its rollback baseline. Repeating bootstrap against this instance is refused.
   Enable real-device read-only access and write access only through separate explicit
   operator decisions; no automatic change is made by bootstrap.

Docker tests are mocked locally. The first manual NAS Runner bootstrap, including
real host mounts and restart, remains the required live acceptance.

## Actions download transport

Verify and deploy use explicitly addressed, commit-pinned actions from
`https://gitea.com/actions/checkout` (v4) and
`https://gitea.com/actions/setup-python` (v5). This avoids the Runner default
GitHub action discovery transport, which failed with EOF/timeouts in runs 27/28.
All validation steps are unchanged. Python toolcache misses may still require
setup-python to download Python from its upstream release source; this change
does not claim to eliminate every external download. Action commits should be
reviewed and intentionally updated, not silently replaced by mutable mirror tags.