ZDDC/zddc/README.md

# zddc-server

A purpose-built HTTPS file server for ZDDC document archives. Designed to replace
`caddy file-server --browse` with features specific to ZDDC workflows.

## Features

- **High-performance static file serving** — ETag, conditional GET, Cache-Control
- **Cascading `.zddc` ACL** — email-based allow/deny lists evaluated bottom-up from requested directory to root
- **Caddy-compatible JSON listings** — the Archive Browser works without modification
- **Virtual `.archive` index** — resolve the earliest revision of any tracked document by URL
- **Filesystem watcher** — archive index updates automatically when files change
- **Flexible TLS modes** — self-signed, real certificates, or plain HTTP
- **Single static binary** — CGO-free, no runtime dependencies; cross-compiled to Linux/macOS/Windows

## Quick Start

zddc-server ships as a cross-compiled binary distributed via Codeberg release assets.

```sh
# Pick a tag from https://codeberg.org/VARASYS/ZDDC/releases (filter by zddc-server-v*)
curl -L -o zddc-server \
  https://codeberg.org/VARASYS/ZDDC/releases/download/zddc-server-vX.Y.Z/zddc-server-linux-amd64
chmod +x zddc-server

# Run against your archive root (HTTPS on :8443 with an in-memory self-signed cert)
ZDDC_ROOT=/srv/archive ./zddc-server
```

Or build from source (requires Go 1.24+):

```sh
git clone https://codeberg.org/VARASYS/ZDDC.git
cd ZDDC/zddc
go build -o zddc-server ./cmd/zddc-server
ZDDC_ROOT=/srv/archive ./zddc-server
```

For plain HTTP behind a reverse proxy, set `ZDDC_TLS_CERT=none` and `ZDDC_INSECURE_DIRECT=1` — see "TLS" below.

There is no Containerfile / Dockerfile / compose file in this repo. Two ways to run zddc-server in Kubernetes / containers:

- The example Helm charts under [`helm/`](../helm/) (`zddc-server-prod/` for stable / `zddc-server-dev/` for tracking main HEAD) compile zddc-server from source via init container — no image registry needed.
- Roll your own image: copy the static binary into a `FROM scratch` or `FROM alpine` base in a few lines.

## Environment Variables

| Variable | Default | Description |
|---|---|---|
| `ZDDC_ROOT` | *(required)* | Absolute path to the served file tree |
| `ZDDC_ADDR` | `:8443` | Bind address (host:port) |
| `ZDDC_TLS_CERT` | *(empty)* | Path to PEM certificate file. `none` = plain HTTP (no TLS); empty = generate self-signed |
| `ZDDC_TLS_KEY` | *(empty)* | Path to PEM private key file. Required when `ZDDC_TLS_CERT` is a file path; ignored otherwise |
| `ZDDC_INSECURE_DIRECT` | *(empty)* | Must be `1` when `ZDDC_TLS_CERT=none` and the bind address is non-loopback. Acknowledges that an authenticating reverse proxy is in front of zddc-server; without it, plain-HTTP non-loopback startup is refused |
| `ZDDC_LOG_LEVEL` | `info` | Log level: `debug`, `info`, `warn`, `error` |
| `ZDDC_INDEX_PATH` | `.archive` | URL path segment name for the virtual archive index |
| `ZDDC_EMAIL_HEADER` | `X-Auth-Request-Email` | HTTP request header containing the authenticated user's email (the oauth2-proxy / nginx auth-request convention) |
| `ZDDC_CORS_ORIGIN` | `https://zddc.varasys.io` | Comma-separated allowlist of origins permitted to make cross-origin requests. Empty value disables CORS entirely. Default lets ZDDC tools served from `zddc.varasys.io` (e.g. via the bootstrap pattern) call back into your deployed server. |

`ZDDC_TLS_CERT=none` disables TLS entirely (plain HTTP). Both cert and key must be set together when using real certificates.

### CORS

The default `ZDDC_CORS_ORIGIN=https://zddc.varasys.io` exists so the canonical
ZDDC tool builds (hosted at `zddc.varasys.io`) can call back into your
deployed `zddc-server` without extra configuration. If you self-host the
tools on your own domain (e.g. `tools.acme.com`), set:

```sh
ZDDC_CORS_ORIGIN=https://tools.acme.com
```

Multiple origins are comma-separated. To disable CORS entirely (e.g. when
all clients are same-origin), set `ZDDC_CORS_ORIGIN=` (empty value). The
middleware echoes the matched origin back per-request and sets
`Access-Control-Allow-Credentials: true` so the upstream-set
`X-Auth-Request-Email` header crosses the boundary.

## TLS

### Plain HTTP (no TLS)

Set `ZDDC_TLS_CERT=none` to run without TLS. Recommended when an upstream reverse proxy
(nginx, Caddy, Traefik) terminates external TLS and talks to zddc-server over plain HTTP
on a private network. zddc-server requires `ZDDC_INSECURE_DIRECT=1` for any non-loopback
bind in this mode — an explicit acknowledgement that an authenticating proxy sits in front:

```sh
ZDDC_ROOT=/srv/archive \
ZDDC_TLS_CERT=none \
ZDDC_ADDR=:8080 \
ZDDC_INSECURE_DIRECT=1 \
  ./zddc-server
```

When `ZDDC_TLS_CERT` / `ZDDC_TLS_KEY` are empty (or when using real certificates), zddc-server generates an ECDSA P-256
self-signed certificate in memory at startup. The certificate changes on every restart —
this is intentional and acceptable when an upstream reverse proxy terminates external TLS
and uses this server only for encrypted in-datacenter transport.

To use a real certificate (e.g. from Let's Encrypt or an internal CA):

```sh
ZDDC_ROOT=/srv/archive \
ZDDC_TLS_CERT=/etc/ssl/zddc/server.crt \
ZDDC_TLS_KEY=/etc/ssl/zddc/server.key \
  ./zddc-server
```

## Authentication

zddc-server does **not** perform authentication itself. It reads the user's email address
from a request header (default: `X-Auth-Request-Email`) that must be set by an upstream reverse proxy
(nginx, Caddy, Traefik, Azure Application Gateway, etc.) after authenticating the user.

If the header is absent, the user is treated as anonymous (empty email). A directory with
no `.zddc` rules is publicly accessible; a directory with an allowlist requires a matching
email.

## `.zddc` Access Control Files

Place a `.zddc` YAML file in any directory to control access. Rules cascade from parent
directories — child rules are appended to (not replaced by) parent rules.

```yaml
# Example .zddc file
acl:
  allow:
    - "*@mycompany.com"       # all users at mycompany.com
    - "contractor@partner.com" # specific external user
  deny:
    - "intern@mycompany.com"  # override: block this specific user
```

### ACL evaluation order

Rules are evaluated **bottom-up**: starting at the requested directory and walking
toward the root. The first explicit match (allow or deny) at any level wins.

1. Check deny patterns at the current level — if email matches → **403 Forbidden**
2. Check allow patterns at the current level — if email matches → **allow**
3. No match at this level → walk up to parent directory and repeat
4. If no `.zddc` files were found anywhere in the chain → **allow** (public, no rules)
5. If `.zddc` files exist but email matched nothing → **403 Forbidden** (not on any list)

This model supports three user tiers in a single tree:

| Level | Rule | Effect |
|---|---|---|
| Root | `allow: ["*@company.com"]` | All company users see everything |
| Project dir | `allow: ["team@company.com"]` | Restricts to the project team |
| Vendor subdir | `allow: ["vendor@ext.com"]` | Grants a third-party access to their folder only |

A vendor navigating to their subdirectory is allowed by the deepest matching rule,
even if a higher-level rule would deny them.

### Glob patterns

`*` matches any sequence of characters within one side of the `@` boundary:

| Pattern | Matches |
|---|---|
| `*@mycompany.com` | Any user at mycompany.com |
| `alice@*` | alice at any domain |
| `*` | Any non-empty email |
| `alice@example.com` | Exact match only |

### Directory visibility

Directories for which the user lacks access are **omitted** from JSON listings entirely —
they are neither listed nor queryable. A direct request to a denied path returns `403`.

### Reserved hidden segments

Two prefixes are filtered from listings under `ZDDC_ROOT`:

- **`.`-prefixed** (e.g. `/.devshell/`, `/Project-A/.internal/notes.md`) — excluded
  from listings **and** 404 on direct HTTP access. The recognized virtual prefixes
  (`.archive`, `.admin`) are explicitly permitted through. This lets operators store
  side-state (caches, dev-shell home dirs, snapshot staging) on the same volume
  that's served, without exposing it.
- **`_`-prefixed** (e.g. `/_template/`) — excluded from listings only. Direct URL
  access still works. Use this for operator-managed scaffolding the user shouldn't
  browse to but might link to (e.g. a `_template/` directory of stub-HTML examples
  to copy into project subdirs).

## Admin Debug Page

`zddc-server` exposes a built-in debug page at `/.admin/` for operators who can
push code/images but cannot `kubectl exec` into the running container. It surfaces:

- **`/.admin/whoami`** — every header on the current request, the configured email
  header name, the value observed at that name, and the resolved email. This is the
  first thing to look at when access logs show `email=anonymous` — it tells you
  exactly which (if any) header the upstream proxy is sending.
- **`/.admin/config`** — the resolved `Config` (env vars). Equivalent to
  `kubectl exec -- env | grep ^ZDDC_` for diagnosing chart / deployment overrides.
- **`/.admin/logs`** — recent log entries (last 500) from an in-memory ring buffer.
  Optional `?level=info|warn|error|debug` and `?since=<RFC3339>` query params.
  At `ZDDC_LOG_LEVEL=debug` every request also logs its full header map under
  `msg=request headers` — useful for diagnosing proxy / SSO header passthrough
  (e.g. confirming which header carries the email). Note: that dump includes
  auth tokens and cookies; only enable debug in trusted environments.
- **`/.admin/`** — HTML dashboard that fetches the three JSON endpoints client-side.

### Authorization

Authorization is via an `admins:` list in the **root** `.zddc` file (`<ZDDC_ROOT>/.zddc`).
Patterns use the same glob syntax as `acl.allow` / `acl.deny`:

```yaml
admins:
  - alice@mycompany.com
  - "*@admin.mycompany.com"
acl:
  allow:
    - "*@mycompany.com"
```

Only the root-level `admins` entry is honored — subdirectory `.zddc` files'
`admins` keys are ignored. Otherwise anyone with subtree write access could
elevate themselves.

If the root `.zddc` has no `admins` list (or no `.zddc` exists), every admin
endpoint returns **404** to every caller. Non-admin requests also receive 404
(not 403) so the existence of the admin page is invisible to unauthorized
callers.

### Forward-auth target for upstream proxies

`zddc-server` also exposes `GET /.auth/admin` — a machine-only endpoint that
returns **200** if the caller's resolved email is in the root `.zddc` `admins:`
list, **403** otherwise. No body, no redirect, no UI; it is a pure
authorization decision intended to be polled by an upstream proxy's
forward-auth directive (Caddy `forward_auth`, nginx `auth_request`, Traefik
`ForwardAuth`, etc.).

The intended use case is gating *adjacent* services on the same pod / host
that don't have their own ACL. Concretely: the dev-shell deployment runs
both `zddc-server` and `code-server` behind one Caddy listener; Caddy uses
`forward_auth` to ask `/.auth/admin` whether the caller is allowed to reach
`/devshell/*` (the IDE) before forwarding. zddc-server's own routes (`/`,
`/<project>/`, `/.archive/`, etc.) keep their existing `.zddc`-cascade ACL
and don't go through this endpoint.

```caddy
# example: protect /devshell/* with forward_auth on /.auth/admin
handle_path /devshell/* {
    forward_auth 127.0.0.1:9090 {
        uri /.auth/admin
        copy_headers X-Auth-Request-Email
    }
    reverse_proxy 127.0.0.1:8443  # code-server
}
```

The check is cheap (one map lookup against the cached `PolicyChain`); calling
it on every request is fine. Edits to `/srv/.zddc` propagate within the
fsnotify watcher's debounce window (~2 s) — no service restart needed.

### Caveats

- Logs are in-memory and lost on restart. The buffer holds the most recent 500
  records; for long-term audit, parse the stderr stream the way you already do.
- The page reads only configuration and request state — it does not modify anything.
- An interactive terminal is not yet available; that's planned as a follow-up
  behind a separate `ZDDC_ADMIN_TERM=1` env-var gate so it stays opt-in.

## Apps: virtual tool HTMLs

`zddc-server` virtually serves the five tool HTMLs (archive, transmittal,
classifier, mdedit, landing) at the appropriate paths. The current-stable
build of each tool is **baked into the binary at compile time** via
`//go:embed`; that's the default. No fetch happens out of the box.

### Where each tool is served

| App           | Available at                                                            |
|---------------|-------------------------------------------------------------------------|
| `archive`     | every directory (multi-project, project, archive, vendor)               |
| `classifier`  | any `Incoming`, `Working`, or `Staging` directory and its subtree       |
| `mdedit`      | any `Working` directory and its subtree                                 |
| `transmittal` | any `Staging` directory and its subtree                                 |
| `landing`     | only at the deployment root (the project picker)                        |

Outside these locations, the corresponding `<app>.html` URL returns 404.

### Override and version-pin

For any path, the resolution order is:

1. **Real file at the path** — operator drops `archive.html` (or any other)
   into a directory; the static handler serves it. Beats everything below.
2. **Closer-to-leaf `.zddc apps:` entry** — walks `.zddc` files leaf→root
   for an `apps.<app>` entry. The first match wins. Spec is one of:
   - `stable` / `beta` / `alpha` (canonical upstream channel)
   - `v0.0.4` / `v0.0` / `v0` (canonical upstream version pin)
   - `https://...` (full URL to a custom mirror)
   - `./local.html` / `/abs/path.html` (local file)
3. **Embedded** — the build-time HTML compiled into the binary.

URL sources are fetched once on first request and cached forever in
`<ZDDC_ROOT>/_app/<host>/<path>`. There is no background refresh and no
hash verification — to pull a new build, delete the cache file. Concurrent
misses for the same URL share one outbound fetch (singleflight). Direct
URL access to `/_app/...` is blocked at dispatch; cached HTMLs are served
only via the apps resolver.

If a configured URL fetch fails (network down, 5xx), the server falls back
to the embedded copy and emits a one-time WARN log per source. The
`X-ZDDC-Source` response header always reports what was served:
`fetch:URL`, `cache:URL`, `path:/abs`, or `embedded:<app>@<build>`.

### Example

```yaml
# <ZDDC_ROOT>/Project-A/.zddc
apps:
  classifier: alpha                                                # track alpha for this project
  archive: https://my-mirror.internal/zddc/archive_v0.0.4.html     # custom mirror, pinned
  mdedit: ./our-mdedit.html                                        # local fork
```

### Env vars

| Variable             | Default | Purpose                                                  |
|----------------------|---------|----------------------------------------------------------|
| `ZDDC_BUILD_VERSION` | `dev`   | String stamped into `X-ZDDC-Source: embedded:<app>@<v>`  |

The landing page fetches `GET /` (with `Accept: application/json`) to retrieve the list
of top-level project directories the requesting user has access to. It renders checkboxes
for each project and opens `archive.html?projects=Proj-A,Proj-B` when the user clicks
"Open Archive".

**Presets** (named project selections) are stored in the browser's `localStorage` — no
server-side state required.

**Shared URLs**: the `?projects=` parameter is preserved in the archive browser URL so
users can email direct links to a pre-filtered view. If the recipient does not have
access to a project listed in the URL, a warning banner is shown.

## Access Logging

Every HTTP request is logged as a structured `slog` entry at `INFO` level:

| Field | Description |
|---|---|
| `ts` | Request arrival timestamp (RFC3339) |
| `email` | User email from the configured header, or `anonymous` |
| `method` | HTTP method |
| `path` | URL path |
| `status` | HTTP response status code |
| `bytes` | Response body bytes written |
| `duration_ms` | Request duration in milliseconds |

Log output goes to `stderr`. Use `ZDDC_LOG_LEVEL=warn` to suppress access logs if needed,
or pipe `stderr` to a log aggregator.

## Virtual Archive Index (`.archive`)

Any URL path segment named `.archive` (configurable via `ZDDC_INDEX_PATH`) is intercepted
by the server and treated as a virtual document index.

The index is built at startup by scanning all transmittal folders under `ZDDC_ROOT`. It
maps each `(project, trackingNumber, revision, modifier)` tuple to the file from the
**chronologically earliest** transmittal folder within that project that contains it.

### Project scoping

The `.archive` index is **scoped to the project** — i.e. the first slash-separated
segment of the request's `.archive` context path. The same tracking number issued
under two different projects does NOT collide; each project's `.archive/` surfaces
only that project's documents.

A request to `/.archive/...` at the very root has no project segment to scope by
and returns **404 Not Found**. Stable references must always be project-rooted
(e.g. `/ProjectA/.archive/TRK-001.html`).

Within one project, two different files claiming to be the same `(tracking, rev)`
are an authoring mistake. The chronological winner still wins, but a `WARN`
log is emitted with both paths so the conflict can be diagnosed and corrected.

### URL patterns

| URL | Resolves to |
|---|---|
| `GET /Project/.archive/TRK-001.html` | Latest base revision of TRK-001 within Project |
| `GET /Project/.archive/TRK-001_A.html` | Base revision A of TRK-001 within Project |
| `GET /Project/.archive/TRK-001_A+C1.html` | Modifier C1 of revision A of TRK-001 within Project |
| `GET /Project/.archive/` | JSON listing of Project's resolvable entries |
| `GET /Project/sub/sub/.archive/TRK-001.html` | Same as the top-level Project listing — depth within a project doesn't change scope |
| `GET /.archive/...` | **404** — root has no project segment |

All successful responses are `302 Found` redirects to the actual file URL. ACL
is enforced on both the `.archive` context directory and the resolved target file.

### Why "earliest" transmittal?

Within one project, any file claiming to be `TRK-001_A (IFC)` should be identical
across transmittals (same content, same SHA-256). If the same tracking number and
revision appears in multiple transmittals, the first one received chronologically is
treated as the authoritative copy. A later arrival with a different file path is an
error condition; the server logs a `WARN` with both paths but does not change the
winner.

### Index refresh

The index refreshes automatically via an `fsnotify` filesystem watcher. Changes are
debounced by 2 seconds before the relevant transmittal folder is re-indexed.

> **Note for Azure Files**: Azure SMB mounts do not support `inotify`/`fsnotify` reliably.
> The watcher will log a warning and the index will only be updated by restarting the server.

## ZDDC Filename Convention

The server parses filenames following the ZDDC convention:

```
trackingNumber_revision (status) - title.extension
```

| Part | Format | Example |
|---|---|---|
| `trackingNumber` | No spaces or underscores | `123456-EL-SPC-2623` |
| `revision` | `~?[A-Z0-9]+(\+[CBNQ][0-9]+)?` | `A`, `~B`, `C+C1` |
| `status` | One of the valid status codes | `IFC`, `REC`, `---` |
| `title` | Free text | `Electrical Specification` |

Valid status codes: `IFA IFB IFC IFD IFI IFP IFR IFU REC RSA RSB RSC RSD RSI ---`

Transmittal folder format: `YYYY-MM-DD_trackingNumber (STATUS) - title`

## Integration with Archive Browser

The Archive Browser (`archive.html`) can connect to zddc-server in HTTP mode. The server
returns JSON directory listings in exactly the same format as Caddy's `file-server --browse`
— no changes to `archive/js/source.js` are needed.

To use: install `archive.html` at `ZDDC_ROOT/archive.html` (or any subdirectory) — either
the actual built tool downloaded by the self-contained install snippet, or one of the
six-line stubs from the project-subdir / track-upstream snippets that fetches it. Then
open it via the zddc-server URL; the app will auto-connect and scan the directory tree.

## Distribution

Each stable release is a Codeberg git tag (`zddc-server-vX.Y.Z`) with four pre-built binaries attached as release assets:

| File | Platform |
|---|---|
| `zddc-server-linux-amd64` | Linux (x86-64) |
| `zddc-server-darwin-amd64` | macOS (Intel) |
| `zddc-server-darwin-arm64` | macOS (Apple Silicon) |
| `zddc-server-windows-amd64.exe` | Windows (x86-64) |

All binaries are statically linked (CGO disabled), built with `-trimpath -ldflags="-s -w -X main.version=<ver>"`. No runtime dependencies.

Download URLs from Codeberg directly:

```
https://codeberg.org/VARASYS/ZDDC/releases/download/zddc-server-vX.Y.Z/zddc-server-linux-amd64
```

Browse all releases at <https://codeberg.org/VARASYS/ZDDC/releases>.

There is no alpha/beta channel for binary distribution. Active dev/soak happens via the [`helm/zddc-server-dev/`](../helm/zddc-server-dev/) chart, which builds zddc-server from source on every pod restart against any commit you point it at. There is no container image; if you want your own, copy the static binary into a `FROM scratch` or `FROM alpine` base in a few lines, or use one of the helm charts which compile from source via init container.

### Env-var contract (for chart consumers)

Downstream Helm charts and Compose files should set these explicitly:

| Variable | Typical value (behind ingress + SSO) | Purpose |
|---|---|---|
| `ZDDC_ROOT` | `/srv` | Path of the served archive (volume mount) |
| `ZDDC_TLS_CERT` | `none` | TLS terminated upstream |
| `ZDDC_INSECURE_DIRECT` | `1` | Acknowledge plain HTTP behind a trusted proxy |
| `ZDDC_ADDR` | `:8080` | Match service / probe port |
| `ZDDC_EMAIL_HEADER` | `X-Auth-Request-Email` | Header your auth proxy sets |
| `ZDDC_CORS_ORIGIN` | `https://your-host` | Origins permitted to call back into the server |

See "Environment Variables" above for the full list.

## Building from source

Requires Go 1.24+.

```sh
# Single binary for the host platform
(cd zddc && go build -o zddc-server ./cmd/zddc-server)

# All four release platforms (cross-compiled, statically linked)
./build   # at the repo root — silently skips if Go isn't on PATH
              # → outputs to zddc/dist/zddc-server-{linux,darwin,windows}-*
```

To run unit tests:

```sh
(cd zddc && go test ./...)
```

## Release tagging

`sh zddc/release.sh` is the canonical path. **Stable cuts only.** The script tags the commit, cross-compiles the four binaries (native Go), and uploads them as Codeberg release assets via the shared `publish-codeberg-release.sh` helper.

```sh
sh zddc/release.sh                       # patch-bump from latest clean stable tag
sh zddc/release.sh 0.1.0                 # explicit version
```

The script tags the commit but does NOT push — finish with `git push origin main` and `git push origin <tag>`.

Prerequisites:

- Go 1.24+ on PATH.
- `$CODEBERG_TOKEN` exported, scoped to write the VARASYS/ZDDC repo. Generate one at <https://codeberg.org/user/settings/applications>.

After the script returns successfully, the website's versions index doesn't need updating for zddc-server (it links out to the Codeberg release page directly). Just push:

```sh
git push origin main
git push origin zddc-server-vX.Y.Z
```

Single-developer / solo-release flow by design — no CI babysitting, no separate dashboard to debug. The script fails loudly and visibly on the developer's terminal if anything goes wrong.

### Versioning

Clean semver. Stable cuts get `<tool>-vX.Y.Z` tags. There are no alpha/beta channel tags for zddc-server — channel URLs are stable URLs by design (counters defeat that), and zddc-server has no static-asset distribution layer where channel mirrors would matter. Active dev runs via `helm/zddc-server-dev/`, which builds from source on each rollout.

The two existing `zddc-server-v0.0.8-alpha.1` and `zddc-server-v0.0.8-alpha.2` tags from a previous experiment stay as historical artifacts; no new alpha/beta tags are created going forward.

---

**Notes:**

- The `.archive` virtual path resolves ZDDC tracking numbers to their earliest-received revision
- ACL is enforced via bottom-up `.zddc` file evaluation