ci(release): publish releases as the sites/zddc-releases OCI image #7

Merged

casey merged 1 commit from ctrl-agent/zddc:ci/releases-oci-image into main 2026-10-15 22:21:27 +00:00

Conversation 0 Commits 1 Files changed 14 +618 -131

ctrl-agent commented 2026-10-15 22:17:04 +00:00

Collaborator

Copy link

Replaces the rsync release deploy (deploy-origin.sh, whose header said an OCI image would replace it) with the lab's standard content-image model. Releases are now published as sites/zddc-releases:<X.Y.Z> + :latest, with the tree under releases/ in the image (image path /releases/...). The origin mounts the image at /var/www/zddc-releases and serves /releases/* with root /var/www/zddc-releases, so no path is rewritten. Each release is a full image, so a stale file a cut didn't include is simply absent; that replaces rsync's --delete-after cleansing.

Flow (release-on-merge.yml)

clone → decide → preflight (every push) → release: fetch+verify → publish-image.sh → verify live or an explicit "SKIPPED, not refreshed" warning step → record │ no release: ensure-published.sh → (verify live if it built and refreshed).

1. publish-image.sh (replaces deploy-origin.sh in the same position). It moves the verified bundle to <stage>/releases, runs buildah from scratch + copy <stage> /, deletes the staging tree, commits, buildah rms the working container, logs in (password on stdin) and pushes :<ver> + :latest. When all three DEPLOY_* are set it rings "$DEPLOY_COMMAND zddc-releases" and prints refreshed=true. Otherwise it prints refreshed=false with a ::warning:: that the release is not live, and the workflow's live-check step is skipped while a separate step says so. It never passes silently.

2. History guard (fail closed), before any build. The bundle's archive_v*.html count (maxdepth 1, .sig excluded) must be ≥ the count in the LIVE listing https://zddc.varasys.io/releases/ fetched with Accept: application/json (Caddy browse JSON). If the listing can't be read, isn't JSON, isn't an array, or counts zero, the publish fails and is never treated as "no history". The messages are the old guard's (history guard: bundle=N versions, live=M versions, …could not count live versions…, live tree reports ZERO versions…, bundle has less history than live (N < M)…).

3. ensure-published.sh runs on every non-release merge. It sends GET https://$REGISTRY/v2/sites/zddc-releases/manifests/<manifest version> with the push credentials (passed to curl on stdin with -K -, never in argv) and an Accept header listing the OCI and docker manifest and index types:

   • 200 means the image exists and nothing is done.
   • 404 means it is missing. The script rebuilds it from the accumulator's newest zddc-releases bundle. That bundle's bundle.sha256 must equal the manifest's bundle-sha256 (stricter than the brief, see below). It then runs the same release-fetch.sh checks a release gets (hash, every signature against release-keys/current.pem, canonicals point at the version, the index names it), deletes the tarball, and calls publish-image.sh.
   • Anything else (any other code, or the registry unreachable) fails the run.

   This is the bootstrap (this PR's own merge publishes the first image, from v0.0.57) and it self-heals a lost image.

4. release-preflight.sh runs on every push. It requires REGISTRY, REGISTRY_USER, REGISTRY_TOKEN and PACKAGE_TOKEN. DEPLOY_ORIGIN, DEPLOY_COMMAND and STATIC_DEPLOY_KEY must be set all together or not at all; a partial set fails with nothing published. publish-image.sh re-checks all-or-none itself.

5. release-fetch.sh gains an optional <package> <package-version> source (both or neither); with neither, it behaves exactly as before. The inline live check moved unchanged into verify-live.sh.

Vars/secrets (for the hub)

kind	name	value
var	REGISTRY	zot host
var	REGISTRY_USER	push-zddc-releases
var	DEPLOY_ORIGIN	at cutover (all three or none)
var	DEPLOY_COMMAND	at cutover (refresh-site)
secret	REGISTRY_TOKEN	push-zddc-releases's password
secret	PACKAGE_TOKEN	exists
secret	STATIC_DEPLOY_KEY	repo-level refresh key pinned to refresh-site zddc-releases, at cutover

Must be true before merging

• REGISTRY, REGISTRY_USER and REGISTRY_TOKEN must be set before the merge. The preflight now runs on every push, so without them this merge and every later push go red. That is intended: fail closed.
• The all-or-none trap. The old deploy read vars.DEPLOY_ORIGIN and secrets.STATIC_DEPLOY_KEY, very likely org-level, which this repo inherits. With DEPLOY_COMMAND unset that is 2 of 3, and the preflight fails the merge. With DEPLOY_COMMAND org-visible too, it is 3 of 3 and the merge rings the origin with the old shared key before cutover. Before merging, make sure this repo sees either none of the three or all three. Repo-level values override org ones, but an org value cannot be "unset" per repo.
• REGISTRY_USER must be the repo-level scoped account. An inherited org-level push account such as ci-pub would pass the preflight and then fail the push with a 403.

Between this merge and the cutover

Releases publish their image and are recorded (accumulator, 7 tags) but are not live: the origin keeps serving the old /var/www/zddc/releases/ tree, and each run warns "published but NOT verified live". The history guard keeps comparing against that live listing, which is still correct. AGENTS.md says this.

Disk (runner-pub, ~23 GB free)

The tree is ~5.35 GB unpacked and ~2.35 GB packed.

• Non-release (ensure) path: the tarball is deleted right after release-fetch.sh unpacks it, and the staging tree right after buildah copy. Peak ≈ tree + vfs container rootfs (≈10.7 GB) during copy and commit.
• Release path: the tarball must survive for release-record.sh's accumulator upload, so the peak is ≈ 13.1 GB.
• Possible extra: if buildah push spools a compressed blob, add ≈2.4 GB, so the worst case is ≈15.5 GB.

These are estimates from the vfs model, not a measured run.

Tests

sh .forgejo/scripts/test-release-on-merge.sh gives 42 passed, 0 failed (was 21). The 21 new known-answer cases are:

• Preflight: none set gives refresh=false, all three give refresh=true, REGISTRY_TOKEN missing, PACKAGE_TOKEN missing, and two of three DEPLOY_* refused.
• Guard: equal history passes, and the same case checks the tree landed under /releases with the canonical symlink intact, :ver and :latest pushed, password on stdin, refresh skipped out loud, and staging removed. Fewer than live is refused with nothing pushed. Unreadable, non-JSON, non-array and zero listings are refused.
• Refresh: rung (ssh deploy@origin 'refresh-site zddc-releases', refreshed=true), and a half-configured set is refused.
• ensure-published: exists (200, nothing built). Missing (404, rebuilt, tarball and tree freed). Registry 500, 401 and unreachable are each refused, not read as missing. The credential never appears on curl's argv. An accumulator bundle that isn't the manifest's is refused. An accumulator bundle with a bad signature is refused by release-fetch.sh.

buildah, ssh and the registry's /v2/ answers are logging stand-ins on PATH. Every other request goes to the real curl against file:// fixtures.

Mutation tests. Each planted bug turned the suite red:

Mutation	Cases turned red
Guard comparison dropped	fewer-than-live
Zero check dropped	zero
Unreadable read as []	unreadable
Uncountable read as 0	non-JSON, non-array
Count archive*.html	4 cases
Any non-200 read as missing	500, 401, unreachable
5xx read as published	500
Unreachable read as 404	unreachable
Credential moved to -u	argv case
Refresh never rung	refresh case
Accumulator equality dropped	mismatch case

shellcheck -S warning is clean on every new or changed script (and -S info, except SC1091 for the dynamic . sources). The SC2034 warnings in the untouched release-record.sh were already there.

Not tested here

No real buildah, registry or origin was available on the authoring side. The test stand-ins assume buildah copy <dir> / copies the directory's contents and preserves symlinks: the canonical archive.html and friends are symlinks. Please confirm on the first image, for example /releases/archive.html resolves after cutover. verify-live.sh checks it.

Hub notes

1. The repo vars and secrets are already set by the hub: REGISTRY_TOKEN (a repo secret), REGISTRY_USER (push-zddc-releases), IMAGE_REPO and SITE_NAME. REGISTRY, DEPLOY_ORIGIN and DEPLOY_COMMAND are inherited org vars. STATIC_DEPLOY_KEY is still the inherited (gated) org key until cutover. All of the refresh inputs are therefore present, so the refresh will be attempted on merge.
2. The merge's refresh will therefore FAIL red before cutover: the origin's refresh-site rejects zddc-releases, which it does not serve yet. The image push happens before the refresh and is unaffected. The hub cuts over right after the merges and re-runs the workflow.

🤖 Generated with Claude Code

Replaces the rsync release deploy (`deploy-origin.sh`, whose header said an OCI image would replace it) with the lab's standard content-image model. Releases are now published as **`sites/zddc-releases:<X.Y.Z>` + `:latest`**, with the tree **under `releases/`** in the image (image path `/releases/...`). The origin mounts the image at `/var/www/zddc-releases` and serves `/releases/*` with `root /var/www/zddc-releases`, so no path is rewritten. Each release is a full image, so a stale file a cut didn't include is simply absent; that replaces rsync's `--delete-after` cleansing. ## Flow (`release-on-merge.yml`) clone → decide → **preflight (every push)** → *release:* fetch+verify → **`publish-image.sh`** → verify live **or** an explicit "SKIPPED, not refreshed" warning step → record │ *no release:* **`ensure-published.sh`** → (verify live if it built and refreshed). 1. **`publish-image.sh`** (replaces `deploy-origin.sh` in the same position). It moves the verified bundle to `<stage>/releases`, runs `buildah from scratch` + `copy <stage> /`, deletes the staging tree, commits, `buildah rm`s the working container, logs in (password on stdin) and pushes `:<ver>` + `:latest`. When all three `DEPLOY_*` are set it rings `"$DEPLOY_COMMAND zddc-releases"` and prints `refreshed=true`. Otherwise it prints `refreshed=false` with a `::warning::` that the release is **not live**, and the workflow's live-check step is skipped while a separate step says so. It never passes silently. 2. **History guard (fail closed)**, before any build. The bundle's `archive_v*.html` count (maxdepth 1, `.sig` excluded) must be ≥ the count in the LIVE listing `https://zddc.varasys.io/releases/` fetched with `Accept: application/json` (Caddy browse JSON). If the listing can't be read, isn't JSON, isn't an array, or counts zero, the publish fails and is never treated as "no history". The messages are the old guard's (`history guard: bundle=N versions, live=M versions`, `…could not count live versions…`, `live tree reports ZERO versions…`, `bundle has less history than live (N < M)…`). 3. **`ensure-published.sh`** runs on every non-release merge. It sends `GET https://$REGISTRY/v2/sites/zddc-releases/manifests/<manifest version>` with the push credentials (passed to curl on stdin with `-K -`, never in argv) and an Accept header listing the OCI and docker manifest and index types: - **200** means the image exists and nothing is done. - **404** means it is missing. The script rebuilds it from the accumulator's newest `zddc-releases` bundle. That bundle's `bundle.sha256` must equal the manifest's `bundle-sha256` (stricter than the brief, see below). It then runs the same `release-fetch.sh` checks a release gets (hash, every signature against `release-keys/current.pem`, canonicals point at the version, the index names it), deletes the tarball, and calls `publish-image.sh`. - **Anything else** (any other code, or the registry unreachable) fails the run. This is the bootstrap (this PR's own merge publishes the first image, from v0.0.57) and it self-heals a lost image. 4. **`release-preflight.sh`** runs on every push. It requires `REGISTRY`, `REGISTRY_USER`, `REGISTRY_TOKEN` and `PACKAGE_TOKEN`. `DEPLOY_ORIGIN`, `DEPLOY_COMMAND` and `STATIC_DEPLOY_KEY` must be set all together or not at all; a partial set fails with nothing published. `publish-image.sh` re-checks all-or-none itself. 5. `release-fetch.sh` gains an optional `<package> <package-version>` source (both or neither); with neither, it behaves exactly as before. The inline live check moved unchanged into `verify-live.sh`. ## Vars/secrets (for the hub) | kind | name | value | |---|---|---| | var | `REGISTRY` | zot host | | var | `REGISTRY_USER` | `push-zddc-releases` | | var | `DEPLOY_ORIGIN` | **at cutover** (all three or none) | | var | `DEPLOY_COMMAND` | **at cutover** (`refresh-site`) | | secret | `REGISTRY_TOKEN` | `push-zddc-releases`'s password | | secret | `PACKAGE_TOKEN` | exists | | secret | `STATIC_DEPLOY_KEY` | **repo-level** refresh key pinned to `refresh-site zddc-releases`, **at cutover** | ## Must be true before merging - **`REGISTRY`, `REGISTRY_USER` and `REGISTRY_TOKEN` must be set before the merge.** The preflight now runs on every push, so without them this merge and every later push go red. That is intended: fail closed. - **The all-or-none trap.** The old deploy read `vars.DEPLOY_ORIGIN` and `secrets.STATIC_DEPLOY_KEY`, very likely org-level, which this repo inherits. With `DEPLOY_COMMAND` unset that is 2 of 3, and **the preflight fails the merge**. With `DEPLOY_COMMAND` org-visible too, it is 3 of 3 and the merge **rings the origin with the old shared key before cutover**. Before merging, make sure this repo sees either none of the three or all three. Repo-level values override org ones, but an org value cannot be "unset" per repo. - `REGISTRY_USER` must be the repo-level scoped account. An inherited org-level push account such as `ci-pub` would pass the preflight and then fail the push with a 403. ## Between this merge and the cutover Releases publish their image and are **recorded** (accumulator, 7 tags) but are **not live**: the origin keeps serving the old `/var/www/zddc/releases/` tree, and each run warns "published but NOT verified live". The history guard keeps comparing against that live listing, which is still correct. AGENTS.md says this. ## Disk (runner-pub, ~23 GB free) The tree is ~5.35 GB unpacked and ~2.35 GB packed. - **Non-release (ensure) path:** the tarball is deleted right after `release-fetch.sh` unpacks it, and the staging tree right after `buildah copy`. Peak ≈ tree + vfs container rootfs (≈10.7 GB) during copy and commit. - **Release path:** the tarball must survive for `release-record.sh`'s accumulator upload, so the peak is ≈ **13.1 GB**. - Possible extra: if `buildah push` spools a compressed blob, add ≈2.4 GB, so the worst case is ≈15.5 GB. These are estimates from the vfs model, not a measured run. ## Tests `sh .forgejo/scripts/test-release-on-merge.sh` gives **42 passed, 0 failed** (was 21). The 21 new known-answer cases are: - **Preflight:** none set gives `refresh=false`, all three give `refresh=true`, `REGISTRY_TOKEN` missing, `PACKAGE_TOKEN` missing, and two of three `DEPLOY_*` refused. - **Guard:** equal history passes, and the same case checks the tree landed under `/releases` with the canonical symlink intact, `:ver` and `:latest` pushed, password on stdin, refresh skipped *out loud*, and staging removed. Fewer than live is refused with nothing pushed. Unreadable, non-JSON, non-array and zero listings are refused. - **Refresh:** rung (`ssh deploy@origin 'refresh-site zddc-releases'`, `refreshed=true`), and a half-configured set is refused. - **ensure-published:** exists (200, nothing built). Missing (404, rebuilt, tarball and tree freed). Registry 500, 401 and unreachable are each refused, not read as missing. The credential never appears on curl's argv. An accumulator bundle that isn't the manifest's is refused. An accumulator bundle with a bad signature is refused by `release-fetch.sh`. buildah, ssh and the registry's `/v2/` answers are logging stand-ins on PATH. Every other request goes to the real curl against `file://` fixtures. **Mutation tests.** Each planted bug turned the suite red: | Mutation | Cases turned red | |---|---| | Guard comparison dropped | fewer-than-live | | Zero check dropped | zero | | Unreadable read as `[]` | unreadable | | Uncountable read as 0 | non-JSON, non-array | | Count `archive*.html` | 4 cases | | Any non-200 read as missing | 500, 401, unreachable | | 5xx read as published | 500 | | Unreachable read as 404 | unreachable | | Credential moved to `-u` | argv case | | Refresh never rung | refresh case | | Accumulator equality dropped | mismatch case | `shellcheck -S warning` is clean on every new or changed script (and `-S info`, except SC1091 for the dynamic `.` sources). The SC2034 warnings in the untouched `release-record.sh` were already there. ## Not tested here No real buildah, registry or origin was available on the authoring side. The test stand-ins assume `buildah copy <dir> /` copies the directory's contents and **preserves symlinks**: the canonical `archive.html` and friends are symlinks. Please confirm on the first image, for example `/releases/archive.html` resolves after cutover. `verify-live.sh` checks it. ## Hub notes 1. **The repo vars and secrets are already set by the hub:** `REGISTRY_TOKEN` (a repo secret), `REGISTRY_USER` (`push-zddc-releases`), `IMAGE_REPO` and `SITE_NAME`. `REGISTRY`, `DEPLOY_ORIGIN` and `DEPLOY_COMMAND` are inherited org vars. `STATIC_DEPLOY_KEY` is still the inherited (gated) org key until cutover. All of the refresh inputs are therefore present, so the refresh **will be attempted** on merge. 2. **The merge's refresh will therefore FAIL red before cutover**: the origin's `refresh-site` rejects `zddc-releases`, which it does not serve yet. The image push happens **before** the refresh and is unaffected. The hub cuts over right after the merges and re-runs the workflow. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

ctrl-agent added 1 commit 2026-10-15 22:17:04 +00:00

ci(release): publish releases as the sites/zddc-releases OCI image ... cf521fa3de

The release step was deploy-origin.sh: an rsync --delete-after of the
verified bundle into /var/www/zddc/releases/ over a shared deploy key. Its
own header said an OCI image would replace it; this is that change, in the
lab's standard content-image model (varasys/metronome, varasys/renew).

- publish-image.sh replaces deploy-origin.sh in the same position (after
  fetch+verify, before the live check and record): the verified tree goes
  under releases/ in a staging dir, buildah builds FROM scratch (vfs +
  chroot), pushes sites/zddc-releases:<X.Y.Z> and :latest, and rings
  `$DEPLOY_COMMAND zddc-releases` on the origin when DEPLOY_* are set.
  Unset: the image is pushed, the origin is not told, and the run says the
  release is NOT live and skips the live check -- never a silent pass.
- History guard kept fail-closed: the tree must hold at least as many
  archive_v*.html as the live site lists, read from Caddy's browse JSON at
  https://zddc.varasys.io/releases/. Unreadable / uncountable / zero = refuse.
- ensure-published.sh on every non-release merge: GET the registry manifest
  for the manifest's version; 200 = done, 404 = rebuild from the
  accumulator's newest bundle (must be the manifest's bundle-sha256, and
  passes release-fetch.sh's signature/structure checks), anything else =
  fail. Bootstrap + self-heal.
- release-preflight.sh on every push: REGISTRY, REGISTRY_USER,
  REGISTRY_TOKEN, PACKAGE_TOKEN required; DEPLOY_ORIGIN, DEPLOY_COMMAND,
  STATIC_DEPLOY_KEY all or none.
- release-fetch.sh takes an optional <package> <package-version> source;
  the live check moved verbatim into verify-live.sh.
- test-release-on-merge.sh: 21 new known-answer cases (42 total) for the
  preflight, the guard, refresh rung/skipped and ensure-published, with
  buildah/ssh/registry stand-ins.
- Docs: AGENTS.md, ARCHITECTURE.md, CLAUDE.md and build comments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

casey merged commit 9ae8694bb9 into main 2026-10-15 22:21:27 +00:00

casey referenced this pull request from a commit 2026-10-15 22:21:27 +00:00

Merge pull request 'ci(release): publish releases as the sites/zddc-releases OCI image' (#7) from ctrl-agent/zddc:ci/releases-oci-image into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

varasys/zddc!7

No description provided.