Today v0.0.19 surfaced a real failure mode: varasys → codeberg push-
mirror is `sync_on_commit: true`, but a transient codeberg 504 mid-
push left 2 of 8 tags un-replicated. BMC chart's Dockerfile fetches
zddc-server-v<X.Y.Z> from codeberg (no egress to git.varasys.io),
so the bumped chart fired BMC pipelines that immediately failed at
`git fetch refs/tags/zddc-server-v0.0.19`. Mirror's next periodic
push (8h default) would self-heal — but by then dev was broken.
Make the stable-cut deterministic: before bumping the chart, force
the push-mirror via the Forgejo API and poll codeberg until all 8
lockstep tags are visible. Fail the job (and skip the chart bump)
if codeberg is genuinely unreachable after 5 min — operator triages
manually rather than triggering downstream builds against a stale
codeberg.
Uses ${{ github.token }} (Forgejo Actions auto-injected) for the
push_mirrors-sync API call. If that token turns out to lack admin
scope on this repo (Forgejo specifics around runner-token perms
vary), the failure will be a clear 401/403 on the curl — switch
to a dedicated CHART_FORGEJO_TOKEN-style secret then.
Local repro:
FORGEJO_TOKEN=$FORGEJO_TOKEN curl -X POST \
-H "Authorization: token $FORGEJO_TOKEN" \
https://git.varasys.io/api/v1/repos/VARASYS/ZDDC/push_mirrors-sync
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both notify-chart-dev.yml and the notify-chart-prod job in
deploy-release.yml were carrying ~80 lines of inline shell each,
duplicating the clone-bump-push flow. Extract to a single script:
.forgejo/scripts/notify-chart-bump.sh <beta|stable> [VERSION]
Three benefits:
1. **Locally testable**. The script is invocable directly:
CHART_FORGEJO_TOKEN=$FORGEJO_TOKEN \
.forgejo/scripts/notify-chart-bump.sh beta
No more "push to main and watch what the runner does" debug loop.
2. **Manual escape hatch**. When CI is broken, the same script is
how we recover. The 0.0.16-beta-1ddd331 chart bump preceding
this commit was performed via this very script.
3. **Runner-quirk-immune**. The previous three commits chased a
Forgejo runner v12.9.0 phantom-SIGPIPE bug that would only
surface under the runner's `bash -e -o pipefail {0}` wrapper.
A real script with its own `#!/usr/bin/env bash` and explicit
error handling sidesteps the wrapper entirely.
The workflow YAMLs shrink to checkout + run-script. No GITHUB_OUTPUT
plumbing, no inline if/else gates, no shell flag overrides. The
behavior is identical to the prior inline versions.
The chart repo (BMCD/tnd-zddc-chart) is mirrored Forgejo→GitHub
one-way (we set this up so the chart matches the same canonical-
on-Forgejo pattern as the public repos). When notify-chart-prod
and notify-chart-dev pushed directly to GitHub, the bump landed
on GitHub but Forgejo never got it — and the next time Forgejo's
push-mirror ran, it force-overwrote GitHub's bump with Forgejo's
older state. Symptom: prod stuck at v0.0.9 even after auto-bump
appeared to succeed; manual investigation showed Chart.yaml
appVersion was actually still 0.0.10 (the previous manual bump
that DID land on Forgejo).
Fix: clone+push to Forgejo (git.varasys.io/BMCD/tnd-zddc-chart)
instead of GitHub. Forgejo's mirror replicates to GitHub on the
next sync — going through the canonical-Forgejo path keeps both
sides in sync. Uses a new CHART_FORGEJO_TOKEN secret (separate
from CHART_GITHUB_TOKEN, which is no longer needed for these
workflows but kept for any future direct-GitHub use case).
Per the bake-in invariant: when no active beta exists, dev tracks
stable. Previously notify-chart-prod only bumped chart's main, so
a stable cut updated BMCD prod but left dev one cut behind until
the next beta or manual dispatch.
Now the job loops through {main, develop} and pushes the same
appVersion bump to both. Each branch push triggers its own BMCD
pipeline (pipeline-prod for main, pipeline-dev for develop), so
prod + dev both rebuild against the new ZDDC stable in parallel.
notify-chart-dev.yml continues to handle the beta-cut path
(advances develop ahead of main between stable cuts).
Multi-line git commit message bodies broke YAML parsing — pipe blocks
end on unindented lines, so the body lines starting at column 0 were
being interpreted by Forgejo's YAML parser as keys, yielding:
yaml: line 158: could not find expected ':'
Switch to repeated `-m` flags (one per paragraph). Same end result
in git log; valid YAML.
Closes the loop on the user-described workflow:
1. Iterate on tools / cut alpha → no chart involvement.
2. `./build beta` → embedded/ commits to ZDDC main →
notify-chart-dev.yml pushes a chart appVersion bump to
burnsmcd/tnd-zddc-chart's develop branch → BMCD pipeline-dev
fires automatically → dev image rebuilt with new beta bytes
baked in.
3. `./build release` → tag pushed → existing deploy-release.yml's
new notify-chart-prod job pushes a chart appVersion bump to
burnsmcd/tnd-zddc-chart's main branch → BMCD pipeline-prod
fires automatically → prod image rebuilt with new stable bytes.
The chart repo IS still committed to (one Chart.yaml line, auto-
generated by either workflow), but no human ever touches it for
routine ZDDC releases. The chart commits are idempotent (skip if
appVersion already at target) and clearly marked as bot-generated.
The truly chart-commit-free version would require either (a)
BMCD's private helm-deploy-latest reusable to accept --set overrides
we'd compute, or (b) bypassing it entirely with our own helm step.
Both are deeper changes than this PR; this is the simplest reliable
solution within the existing reusable.
Auth: a new repo-scoped Forgejo Actions secret CHART_GITHUB_TOKEN
holds the classic GitHub PAT (already provisioned for the
Forgejo→GitHub mirror; same token, repo+workflow scopes,
SAML-SSO authorized for burnsmcd). The bot identity is
'ZDDC Release Bot <noreply@zddc.varasys.io>'.
Tested behavior:
- Workflow files are added by THIS commit. Pushing this commit
does not fire either workflow (notify-chart-prod requires a
tag; notify-chart-dev requires changes under
zddc/internal/apps/embedded/). Safe to land before testing.
- First real test fires on the next ZDDC stable cut or beta cut.
Runner now runs in a quadlet container on caddy-net, so 127.0.0.1
is the runner's own loopback. Reach the Caddy container by name
('caddy') with --connect-to keeping SNI/Host as the public hostname
so the right vhost matches.
Also adds the tag trigger: push of zddc-server-v[0-9]+.[0-9]+.[0-9]+
auto-cuts a stable release. The lockstep set pushes six tags at once;
filtering on zddc-server-v* gives exactly one workflow run per cut.
Re-cutting at the tagged commit is safe — _promote_stable in
shared/build-lib.sh is idempotent re: tag creation.
Forgejo Actions workflow that runs ./build alpha|beta|release [version]
followed by ./deploy --releases. Uses the host-mode runner so the
behavior is identical to manual cuts. Tag-trigger added later once
the dispatch path is exercised.
