docs: AGENTS.md + ARCHITECTURE.md cover records audit + history

AGENTS.md:
- Form-data system: clarify that submission filenames now depend on
  whether a records: rule matches (composed tracking number) or not
  (legacy date+email scheme).
- Validator subset: mention the three Schema extensions (readOnly,
  pattern, x-labels) that survive YAML→JSON round-trip.
- Tables system: replace the speculative "Future per-row history"
  bullet with the implemented .history/<base>/<ts>-<sha8>.yaml layout.
- New section "Records, audit, and history": the three record-type
  shapes (MDL independent, RSK rows-of-deliverable, SSR party-folder
  identity), the two new .zddc keys (field_codes + records), the six
  audit fields, write ordering (history first, then live), strip-and-
  stamp anti-forgery, ?history=1 wire surface, record-vs-config gate,
  operator customization recipes.

ARCHITECTURE.md Form Renderer section:
- Note that record-typed writes route through WriteWithHistory rather
  than plain WriteAtomic.
- Distinguish records (audited, composed filenames, immutable history)
  from generic submissions (plain writes, free-form filenames).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
ZDDC 2026-05-19 10:26:45 -05:00
parent 3b2280de7f
commit 480cb0e4a3
2 changed files with 53 additions and 5 deletions

View file

@ -382,11 +382,15 @@ A schema-driven form renderer used to collect structured data into YAML files in
- `GET /<dir>/<id>.yaml.html` — render form pre-filled from `<id>.yaml`
- `POST /<dir>/<id>.yaml.html` — overwrite that submission → 200
**Storage**: spec at `<dir>/form.yaml`, submissions at `<dir>/<YYYY-MM-DD>-<email-sanitized>.yaml` (siblings of the spec). Copying `<dir>` elsewhere copies the spec plus every submission together. ACL applies via the existing `.zddc` cascade.
**Storage**: spec at `<dir>/form.yaml`. Submission filenames depend on whether the directory has a cascade-declared `records:` rule (see "Records, audit, and history" below):
- **No matching `records:` rule** — submissions land at `<dir>/<YYYY-MM-DD>-<email-sanitized>.yaml` (the legacy date+email scheme; still the path for ad-hoc operator-defined forms).
- **Matching `records:` rule** (mdl/rsk/ssr and operator-declared records) — filename is composed from body fields via the rule's `filename_format`; for rsk-style rules the server also auto-assigns a per-row sequence within the table-scope group.
Copying `<dir>` elsewhere copies the spec plus every submission together. ACL applies via the existing `.zddc` cascade.
**Round-trip**: v0 is form-as-truth — submission YAML is regenerated from form state on every save; comments in submissions are not preserved. File-as-truth mode (lossless YAML round-trip via the eemeli/yaml Document API) is a v1 feature, needed for hand-edited files like `.zddc` itself.
**Validator subset** (`zddc/internal/jsonschema/`): `type` (string/number/integer/boolean/array/object), `enum`, `minimum`, `maximum`, `minLength`, `maxLength`, `required`, `additionalProperties: false`, `properties`, `items`, `format` (`date`, `email`). NOT supported in v0: `$ref`, `$defs`, `if/then/else`, `oneOf`/`anyOf`/`allOf`, conditional visibility. The form-spec meta-schema enforces that authors stay in the supported subset.
**Validator subset** (`zddc/internal/jsonschema/`): `type` (string/number/integer/boolean/array/object), `enum`, `minimum`, `maximum`, `minLength`, `maxLength`, `pattern`, `required`, `additionalProperties: false`, `properties`, `items`, `format` (`date`, `email`). Schema also carries three client-facing extensions that survive round-trip but aren't enforced by the validator (the server enforces them via cascade or strip-on-write): `readOnly: true` (UI renders disabled), `x-labels: { code → label }` (paired display text for enum dropdowns). NOT supported in v0: `$ref`, `$defs`, `if/then/else`, `oneOf`/`anyOf`/`allOf`, conditional visibility. The form-spec meta-schema enforces that authors stay in the supported subset.
**Renderer subset** (`form/js/`): types listed above, enum (select / `ui:widget: radio`), `format: date|email`, textarea, nested objects, arrays of primitives, arrays of objects with add/remove rows. `ui:show-when` and reorder are v1.
@ -415,14 +419,56 @@ Read/aggregate counterpart to the form system. Renders a directory of YAML row f
- **Nested sub-tables**`<dir>/sub-list/table.yaml` is its own self-contained table at `<dir>/sub-list/table.html`. Composition, not violation.
- **Per-row attachments**`<dir>/<id>.attachments/file.pdf`. Natural sidecar pattern; the row YAML can reference its attachments by relative path.
- **Drafts / staging**`<dir>/.drafts/<id>.yaml` (dot-prefix → hidden from listings as well as from the table).
- **Future per-row history** — `<dir>/.history/<id>/<timestamp>.yaml` if/when version sidecars are added.
- **Per-row history** — `<dir>/.history/<base-without-ext>/<RFC3339Nano>-<sha8>.yaml`. Server-managed; one directory per record, one file per archived revision. See "Records, audit, and history" below.
**Default-MDL fallback at `archive/<party>/mdl/`**: when no `table.yaml` (or `form.yaml`) exists on disk in this exact location, the server serves embedded default bytes. The `mdl/` directory itself doesn't even need to exist — the URL renders the default MDL view fully virtually so a fresh archive surfaces the master document list without operator setup. Outside `archive/<party>/mdl/`, presence-based discovery is the rule.
**Default-MDL columns mirror the tracking-number components** documented at `zddc.varasys.io/reference.html#tracking-numbers`: `originator`, `phase`, `project`, `area`, `discipline`, `type`, `sequence`, `suffix` — each one a slot of the deliverable's permanent identifier — plus `title`, `plannedRevision`, `plannedDate`, `status`, `owner`. The form schema accepts free-text on every component (no enums or regex constraints) so projects pick their own conventions. Operators customize by dropping their own `table.yaml` + `form.yaml` into `archive/<party>/mdl/`; both files override the embedded defaults atomically (no merge — operator-supplied wins entirely). Source: `zddc/internal/handler/default-mdl.{table,form}.yaml`.
**Default-MDL columns mirror the tracking-number components** documented at `zddc.varasys.io/reference.html#tracking-numbers`: `originator`, `phase`, `project`, `area`, `discipline`, `type`, `sequence`, `suffix` — each one a slot of the deliverable's permanent identifier — plus `title`, `plannedRevision`, `plannedDate`, `status`, `owner`. The form schema accepts free-text on every component by default. Projects narrow the vocabulary via the cascade's `field_codes:` (see below) without rewriting the schema — operator overrides at `archive/<party>/mdl/{table,form}.yaml` still win atomically over the embedded defaults. Source: `zddc/internal/handler/default-mdl.{table,form}.yaml`.
**Adding a new table**: create a directory `<dir>/` and drop `table.yaml` (and optionally `form.yaml` for row editing) into it. No code change required. Visit `<dir>/table.html`.
## Records, audit, and history
The "records" subset of the tables system carries three guarantees the generic form/table flow doesn't: server-stamped audit fields, immutable per-record history, and cascade-driven filename composition. The mechanism lives in `zddc/internal/handler/history.go` (`WriteWithHistory`) and `zddc/internal/zddc/field_codes.go`. Three record types ship out of the box:
| Type | Folder | Filename | Identity carrier |
|---|---|---|---|
| **MDL** (deliverables) | `archive/<party>/mdl/` (many siblings) | Composed tracking number, e.g. `ACM-PRJ-EL-SPC-0001.yaml` | Body's component fields |
| **RSK** (risk register) | `archive/<party>/rsk/` (many siblings, multiple tables) | `<table-tracking>-<row>.yaml`, e.g. `ACM-PRJ-EL-RSK-0001-001.yaml` | Body's components + server-assigned row sequence |
| **SSR** (parties register) | `archive/<party>/ssr.yaml` (one per party folder) | Always literal `ssr.yaml` | Parent folder name (existing `name` strip/inject in `ssrhandler.go`) |
**Two new `.zddc` keys** carry the rules (see `zddc/internal/zddc/file.go` + `field_codes.go`):
- `field_codes:` — vocabulary for the components used in filename composition and constrained body fields. Each entry is a discriminated union over `kind: enum|pattern|free` (`{kind: enum, codes: {ACM: Acme Inc, …}}` / `{kind: pattern, pattern: "^[0-9]{4}$"}` / `{kind: free, description: "..."}`). Map-merge across the cascade (mirror of `apps:`) — a deeper level can narrow or replace a single code's vocabulary without dropping unrelated codes.
- `records:` — per-pattern rules keyed by filename basename (literal `ssr.yaml` or glob `*.yaml`). Each entry carries `filename_format` (composition template with `{field}` and `{field?}` placeholders), `field_defaults`, `locked`, plus `row_field` + `row_scope_fields` for RSK-style tables-of-rows. Filename-pattern scoping is what lets the SSR rule live at the party-folder level without affecting `mdl/`, `rsk/`, `received/`, etc., siblings.
Defaults are baked into `defaults.zddc.yaml`; `field_codes:` ships empty (every deployment writes its own vocabulary).
**Six server-managed audit fields** are injected on every write and stripped from incoming bodies before validation (snake_case to match `.zddc`'s existing `created_by:`):
- `created_at`, `created_by` — stamped on create; preserved untouched on every update
- `updated_at`, `updated_by` — refreshed on every write
- `revision``1` on create, `+1` per update
- `previous_sha` — first 8 hex chars of SHA-256 of the prior revision's bytes; absent on create. Forms a hash chain for tamper evidence
**History layout**: for any record at `<dir>/<base>.<ext>`, the prior version is archived at `<dir>/.history/<base>/<RFC3339Nano-UTC>-<sha8>.<ext>` before the live file is overwritten. Per-record subfolder under `.history/` keeps `readdir` cheap and makes party-folder rename move SSR history along atomically (the dot-folder is inside the party folder, so `os.Rename` carries it).
**Write ordering**: history first, then live. A crash between the two leaves the prior version safely archived; the retry is idempotent because the history filename is deterministic (timestamp + sha of prior bytes).
**Strip-and-stamp policy**: clients can't forge audit fields. `WriteWithHistory` strips all six keys from the incoming body BEFORE schema validation runs, then injects authoritative values from request context. A client that sends `created_by: eve@evil` finds it silently overwritten with the request principal.
**Wire surface**:
- `PUT /<record>.yaml` — routed through `WriteWithHistory` automatically when the basename matches a `records:` rule. Response echoes the stamped YAML as the body (Content-Type: application/yaml) so the tables client can update its row state without a re-GET.
- `GET /<record>.yaml?history=1` — JSON list of prior revisions: `[{revision, ts, by, sha, path}, …]`. ACL gates against the live record (read it → read its history).
**Record-vs-config distinction**: `WriteWithHistory` fires only for genuine record paths. The gate (`isRecordPath` in `fileapi.go`) excludes `table.yaml`, `form.yaml`, `.zddc`, and the spec naming variants `*.table.yaml` / `*.form.yaml`. Those bypass audit stamping (they're configuration, not data) and go through plain `WriteAtomic`.
**Operator customization**:
- To narrow a deployment's originator codes: write `field_codes: originator: {kind: enum, codes: {ACM: …, BET: …}}` at the project root `.zddc`.
- To add a new table type: declare a `records:` entry under the appropriate `paths:` level (or a sibling `.zddc` in the folder) with a `filename_format` referencing fields the body carries.
- To inspect a record's revision history: `curl https://<host>/<path>.yaml?history=1 -H 'Authorization: Bearer …'`.
Source: `zddc/internal/handler/history.go`, `zddc/internal/zddc/field_codes.go`, `zddc/internal/zddc/walker.go`, `zddc/internal/zddc/cascade.go`, `zddc/internal/zddc/defaults.zddc.yaml`. Tests: `zddc/internal/handler/history_test.go`.
## Implementation-vs-dependency policy
Match implementation cost to actual surface used. Reimplement focused subsets when a dep's surface area is much larger than what we consume; adopt for genuinely large specs (YAML parsing, etc.) where reimplementing is foolish. Examples in this codebase:

View file

@ -474,7 +474,9 @@ app.state.subscribe((property, newValue) => {
- `js/post.js` — POST + handle 200/201/422/403/409 responses
- `js/main.js` — boot: load context, mount root widget, wire submit
**Server-side counterpart:** `zddc/internal/handler/formhandler.go` recognizes `*.form.html` and `*.yaml.html` URLs, parses the spec, validates submissions via `zddc/internal/jsonschema/`, writes via `zddc.WriteAtomic`. Existence of `<name>.form.yaml` is the trigger; without it, the URL falls through to static-file serving.
**Server-side counterpart:** `zddc/internal/handler/formhandler.go` recognizes `*.form.html` and `*.yaml.html` URLs, parses the spec, validates submissions via `zddc/internal/jsonschema/`, writes via `zddc.WriteAtomic` (plain submissions) or `zddc/internal/handler/history.go` `WriteWithHistory` (record-typed YAML — mdl rows, rsk rows, ssr.yaml). Existence of `<name>.form.yaml` is the trigger; without it, the URL falls through to static-file serving.
**Record-vs-submission distinction.** "Records" are the three table-store types (mdl/rsk/ssr); everything else is a "submission." Records get server-stamped audit fields (`created_at`/`_by`, `updated_at`/`_by`, `revision`, `previous_sha`), an immutable per-record history at `<dir>/.history/<base>/<RFC3339Nano>-<sha8>.<ext>`, cascade-driven filename composition (via the `records:` + `field_codes:` `.zddc` keys), and per-folder field locking (e.g. type=RSK in rsk/). The mechanism intercepts at the file-API write path (`serveFilePut`): if `isRecordPath` matches, the call routes through `WriteWithHistory`; otherwise the historical `WriteAtomic` path is used. See AGENTS.md "Records, audit, and history" for the operator surface; `zddc/internal/handler/history.go` for the orchestration.
**Round-trip philosophy:** v0 is "form-as-truth" — submission YAML is regenerated from form state on every save. Hand-edits to submission files are not preserved across re-edit→re-submit. v1 will add an opt-in "file-as-truth" mode (eemeli/yaml Document API) for forms like `.zddc` itself where users hand-edit and comments must survive.