Quarantine sick preview slots automatically

2026-07-18: preview-7 spent two hours throwing Durable Object storage operation exceeded timeout which caused object to be reset (749×, mostly CapabilityHostDurableObject, spread over 86 objects including ones born mid-storm, sticky across deploys, self-recovered untouched) while preview-1 absorbed strictly more load with 10. Verdict: transient Cloudflare storage-shard degradation — erase-data coverage was audited complete and accumulation disproven. It cost a full day of PR #2083's merge gate until the lease was moved by hand. Precedent: preview-6's ~200-error hour on 07-14.

Automate that manual save: before/after each preview run (or in the semaphore acquire path), count the storage-timeout signature for the slot's worker over the trailing hour via the Workers Observability query API; above a threshold (~50/hr), release the lease, mark the slot quarantined in semaphore for a few hours (they self-heal), and acquire another. Surface quarantines in pnpm preview status.

Was this page helpful?