The lazy repo lane: clone-free reads and commits against Artifacts

Status: shipped behind RepoDurableObject (PR #2262) · Owners: repos domain This is the running pressure-test + limitations document. Add findings here; do not let them live only in PR comments.

Why it exists#

The repo Durable Object used to equate every read with possessing the whole repository: any head move re-cloned the full pack into an in-memory filesystem (iterate/iterate: ~37 s), and every commitFiles cloned too. One pushed commit made the next board load pay ~37 s twice. The requirement driving the redesign: a repo much larger than iterate/iterate, holding thousands of tasks, has to stay efficient end to end.

Architecture#

                        RepoDurableObject
   ┌────────────────────────────────────────────────────────────┐
   │  readFile / listFiles / getFilesSnapshot({paths})          │
   │  commitFiles                                               │
   │        │ lazy-first, clone-lane fallback                   │
   │        ▼                                                   │
   │  lazy-repo-reader.ts     ←— freshness policy stays in the  │
   │   syncToHead(target)         DO: candidates are validated  │
   │   readHeadPaths/listHead     against the branch authority  │
   │   commitFiles → typed        (decideHeadResolution) before │
   │        │        outcome      and after every install       │
   │        ▼                                                   │
   │  repo-object-store.ts    — durable SQLite snapshot:        │
   │   objects (oid → chunked bytes, verified on read)          │
   │   manifest (branch, path → blob oid, mode)                 │
   │   dir trees (the `have`s), heads                           │
   │        │                                                   │
   │        ▼                                                   │
   │  git-wire.ts             — protocol v2: ls-refs, fetch,    │
   │   receive-pack, pack codec (deltas, strict validation)     │
   └────────────┬───────────────────────────────────────────────┘
                ▼
        Artifacts git endpoint ("gitty/1.0")

• One byte authority for lazy reads. Reads are manifest lookup → verified blob bytes, labeled with the head they came from, captured under one serialization barrier (readHeadPaths). The pre-existing byte-tree + clone machinery is untouched and serves as the fallback lane.
• Sync is delta-priced. want <target>, deepen 1, have <every known dir tree> — the server excludes unchanged subtrees recursively; blobs the exclusion swallows (rename/copy into a changed directory) hydrate by exact oid.
• Commits are built locally (blobs + only the changed ancestor-chain trees + commit), pushed as one small pack under a compare-and-swap, and installed into the store — read-your-write costs zero network.
• Outcomes are proofs. applied / rejected (remote provably unmoved — the only state that may fall back to the clone write lane) / indeterminate (reconciled against the ref; surfaces as an error rather than risking a double commit).
• Lifecycle is bounded by construction. installSnapshot prunes every object unreachable from any branch's live snapshot in the same transaction. No LRU, no budget: storage ≈ the working set.

The wire contract (all empirically probed against gitty/1.0)#

Measured (2026-07-22/23, dev-namespace Artifacts, full reader stack)#

Prod (deployed pair, before this branch merged — clone lane): the same iterate/iterate cold read was 36.9 s, re-paid after every push; warm 438 ms.

Production, after the merge (2026-07-23, tasks.iterate.workers.dev → os.iterate.com)#

Two findings from the prod pressure run:

• The 10,001st task broke the board — the vessel's single readFiles call hit the platform's 10,000-path cap and threw. Fixed in iterate/tasks (9c8366b): the board seed reads in 5,000-path chunks, two lanes at a time. The cap itself stays (it bounds one RPC's work honestly).
• Back-to-back commits on a GitHub-linked repo wait ~12 s — every commit schedules a full GitHub mirror push ON the repo DO's serialized write chain, so the next commit queues behind a clone+push to GitHub. Pre-existing main behavior (the lazy lane made it visible by making commits fast); prod /repos/iterate has additionally diverged from its GitHub source, so those mirror pushes cannot fast-forward. Follow-up candidates: move the mirror push off the write chain, or mirror from the lazy store without a clone.

Known limitations (ranked; documented, not hidden)#

1. Kernel-scale repos fail at the SERVICE, not the client. Artifacts' upload-pack returns HTTP 500 after ~73 s for the 281 MB-pack kernel snapshot — real git clone --depth 1 fails identically, so the clone lane is equally dead. The practical ceiling sits somewhere between a ~20 MB pack (works, fast) and a ~281 MB pack (server 500). Pushing the same repo INTO Artifacts worked (6 min), so ingest and serving limits differ. Follow-up if kernel-scale becomes real: manifest-first cold sync (batched tree-only wants — the arbitrary-oid want primitive already supports it) + blob-on-demand, which also sidesteps the isolate-memory point below.
2. Cold ingest buffers the pack ~3× in memory (HTTP response + demuxed pack + inflated objects). Fine to ~40-50 MB packs inside a 128 MB isolate; beyond that cold ingest would OOM and fall back to the clone lane (which has the same ceiling). Streaming pkt-line → incremental pack parse → store writes is the designed successor; the wire client isolates that change to one module.
3. Commit compile holds the full manifest in memory — O(repo paths) per commit (~10 k rows ≈ tens of ms; ~100 k ≈ ~100-300 ms). Tree building is already ancestor-only; the manifest map is the remaining O(repo) piece. SQLite-side diffing is the follow-up if profiles ever show it.
4. Reads serialize with installs (one chain). Correctness first; a reader/writer barrier is a measured-need optimization. Read latency at 10 k files (250 ms for the full set) says this is nowhere near binding.
5. Every read re-verifies SHA-1 (~µs/KB). At current sizes this is noise (4 ms for 74 files). A verified-once memo per isolate is a possible micro-optimization; deliberately not built without evidence.
6. Capabilities are asserted, not negotiated. The receive-pack capability line and pako's strm.avail_in are validated loudly at runtime and documented as gitty-specific; a server or pako change fails into the clone lane with a clear message.
7. edit() still uses the clone lane. Same recipe applies if it ever matters; it is the rare lane.

Test harness (no vitest-pool-workers anywhere)#

• Wire (git-wire.test.ts, 22): byte-golden fixtures from real git; delta chains incl. forward ref-delta and ofs-on-ref; corrupted trailer/version/bounds; three-state push report classification (applied / rejected / indeterminate on truncation).
• Reader + store (lazy-repo-reader.test.ts, 25): REAL SQLite transactions (BEGIN IMMEDIATE/ROLLBACK) with statement-level fault injection — install atomicity, pushed-then-install-fails, chunk corruption (mutated AND missing chunks) with quarantine + rehydration, corrupt-tree self-healing, multi-branch reachability pruning, gitlinks, empty-tree commits, final-state conflict validation, CAS conflict retry with true parents, transport-death reconciliation (applied / rejected / indeterminate), storage-bounded-after-5-commits.
• E2e (repo-lazy.itx.e2e.test.ts, real worker + real Artifacts): batched commit → RYW at the itx surface → listFiles → stacked commit → log sees lazy history → clone-lane edit interleaves → lazy commit on the clone-lane head.
• Live probes (scripts/probe-git-wire-live.ts, scripts/probe-lazy-reader-live.ts): the full stack against real Artifacts repos, env-gated.

Review provenance#

Three adversarial thermo-nuclear rounds (self-review + Codex gpt-5.6-sol, xhigh reasoning, independent runs). Round 1: BLOCK → single-byte-authority restructure, typed outcomes, bounded lifecycle, verified reads. Round 2: BLOCK → exhaustive DO commit boundary (double-commit hole closed), three-state push reports, serialized observations, self-healing trees, post-install authority re-check, final-state validation, forward-delta fix. Round 3: BLOCK → three P0s fixed: (1) ambiguous-push reconciliation re-sends the idempotent CAS pack instead of trusting eventually-consistent ref reads — a stale parent sighting can never manufacture rejected, and a CAS/ls-refs contradiction stays indeterminate; (2) the whole commit (capture → compile → push → install) runs as ONE chained operation, so a mid-commit sync can never interleave a newer snapshot under a stale delta; (3) contentHash publication reads {head, files} under one barrier and re-checks head identity + authority endorsement + raced records immediately before the put.

Accepted residual gap (Codex round-3 P1): no test crosses the private #commitFilesLazy → #commitFiles DO seam directly — the seam is an exhaustive typed switch whose variants are all reader-tested; a DO harness with injectable wire/store gates would close it if the seam ever grows.