Sub-threshold finding from a focused security review of the CI URL
work — defense-in-depth even though it sits inside the documented
"trust upstream" boundary.
The mirror walker's purgeOrphans deletes local files that aren't in
the upstream's listing. It walked a dirPath built recursively from
upstream-supplied entry names and called os.Remove on the resolved
local path with no containment check. A hostile or compromised
upstream returning ".." in a directory listing could steer the
walker out of cache.root and into the parent — deleting whatever
matches the upstream's "expected to be there" filter in the wrong
directory.
A healthy master never produces such entries (listing.FromDirEntries
filters dot-prefix names), so the bug only fires under an actively
malicious or MITM'd upstream — confidence stayed below the report
threshold. But the fix is small and the cost of being wrong is real
deletion of files outside the cache, so it's worth doing.
Two layers:
1. walker.go walkDir filters upstream listing entries with name ==
"" / "." / ".." or containing "/" / "\" before recursing. Logs
a WARN with the dropped name so an operator can see if their
upstream is misbehaving.
2. purgeOrphans verifies the resolved localDir is contained under
s.cache.root (HasPrefix(root + sep) || == root) before
ReadDir+Remove. Logs a WARN and bails on mismatch.
Either layer alone would fix the original vector; both together
match the defense-in-depth pattern cachePathFor already follows for
single-file writes (line 506).
New TestWalker_HostileUpstreamCannotEscapeCacheRoot constructs a
fake upstream that returns a "../" entry in its listing, places a
sentinel file in the parent of cache.root, runs a mirror walk, and
asserts the sentinel survives. Both filter and containment guard
fire; the sentinel stays put.
Existing mirror tests unchanged — the filter only drops names that
shouldn't appear in healthy listings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
--mode mirror layers an access-triggered walker on top of the cache
pipeline. When an incoming request's URL falls under one of the
configured --mirror-subtree paths, the scheduler kicks off a recursive
walk of that subtree iff (a) no walk for that subtree is in flight and
(b) now - last_walk_at >= --mirror-min-interval (default 1h). Walks
run in a goroutine; the user's request never blocks on scheduling.
Why access-triggered: a naive "walk on a fixed timer" would produce
thundering-herd polls on a master from many vendor mirrors most of
which are idle most of the time. Demand-triggering means idle mirrors
generate zero upstream traffic until someone hits them; active
mirrors stay current as a side effect of normal use.
The walk:
1. Recursively fetches JSON listings under the subtree, persisting
each at <dir>/.zddc-listing.json so directory browsing works
offline for walked subtrees.
2. For each file, fires a conditional If-Modified-Since GET (bounded
parallelism; default 4 concurrent) — 304 no-op, 200 overwrites,
403/404 purges the local cache.
3. After enumeration, per-directory orphan purge: local files absent
from upstream's filtered listing are removed (handles upstream
deletes + ACL revocations).
State persists at <root>/.zddc-mirror-state.json as
{subtrees: {<path>: {last_walk_at}}}. In-flight tracking is in-memory
only — a crash mid-walk lets the next access retry without manual
cleanup. Subtree path matching is longest-prefix-wins; "/" is a
catch-all (full mirror, the default when --mode=mirror is set without
explicit --mirror-subtree).
The cache layer also gained directory-listing caching (independent of
mirror mode but enabled by it). Directories are now stored at
<dir>/.zddc-listing.<html|json> sidecars, varied by Accept header.
Hit/miss/offline semantics mirror the file pipeline. Phase 2's
limitation that directories always proxied live (no offline browse)
is now resolved for any directory the user has visited or that mirror
mode has walked.
Mirror scope falls out of auth: the walker uses the local instance's
bearer, so it sees exactly what the user can see at upstream. Admin
bearer → full mirror; vendor bearer → vendor's permitted subtree;
no code distinguishes the cases.
New flags (also as ZDDC_* env vars), ignored when --mode != mirror:
- --mirror-subtree <csv> — repeatable subtrees (comma-separated);
empty + --mode=mirror = "/" (full mirror)
- --mirror-min-interval <duration> — default 1h
Tests (15 new in walker_test.go, 3 new in cache_test.go): subtree
normalization, longest-prefix matching, root-as-catch-all, walk
fetches all files in scope, out-of-scope URLs are no-op, rate-
limiting prevents double-walks within min-interval, walks re-fire
after interval elapses, orphan purge removes local-only files,
state file survives restart, concurrent triggers don't double-walk,
end-to-end ServeHTTP-kicks-mirror-on-access, listing format varies
by Accept, listing offline serves stale, persisted state atomic
write + corrupt-input handling. Full suite + go vet clean.
Doc updates: zddc/README.md flags table gains the two new entries
plus a "Mirror mode (access-triggered subtree walker)" subsection
with trigger semantics and properties; the "What client mode is NOT,
yet" list shrinks accordingly. AGENTS.md env-var table gains the
two new entries. ARCHITECTURE.md "Master + proxy/cache/mirror"
section now documents the walker scheduler / walk algorithm / state
file in a "Mirror walker (access-triggered)" subsection.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
