Per-suite flake sentinels: specs + preview e2e
Status summary: Extends the flake pipeline's self-proof to the suites most likely to actually flake. Each suite gets its own deliberately ~10%-flaky, month-gated sentinel (distinct test names → distinct dashboard rows, so a suite whose sentinel reads 0% has broken plumbing), plus the two-line CI wiring that makes its records flow: FLAKE_RECORD_DIR on the lane + a flake-records-<suite> artifact upload. Ingestion is already suite-agnostic (#2582) — no server changes.
Checklist#
-
specs/flake-sentinel.spec.ts— playwright sentinel (verified through the real spec runner: expected-fail registration accepted, record line written; cosmetic wart: playwright shows the wrapper's file as the test location) -
apps/os/e2e/vitest/flake-sentinel.e2e.test.ts— preview e2e sentinel (verified under the e2e vitest config; it flaked on the verification run and stayed green, recording the flake-fail) - preview lanes:
FLAKE_RECORD_DIRper lane inscripts/preview/preview.ts(playwright →test-results/flake-records/specs, vitest e2e →.../preview-e2e), guard tests updated -
cloudflare-previews.yml:flake-records-specs+flake-records-preview-e2eartifact uploads (if: always(),if-no-files-found: ignore,overwrite: true) - docs/testing.md: note the per-suite sentinel convention + the stance that
createFlakereplaces retries only for tests that opted in (unwrapped specs keep playwright retries) - createFlake pins per-test
retry: 0on vitest (surfaced by the first real preview run: the e2e suite'sretry: {count:1, delay:5000}re-ran the sentinel after the wrapper's green throw — vitest retry fires before the.failsinversion — recording every green outcome twice; fixture child now runs with suite retry to pin it) - ingestion: drop the Depot run-status filter (also surfaced live: Depot's per-sha run status settles only after the LAST check completes, and that check's webhook beats the flip, so
status: ["finished","failed"]in ListRuns permanently skipped the last-completing check's artifacts — always the slow preview check, i.e. exactly these two new suites) - first real adoptee:
specs/mobile/chat-photos.spec.tswrapped withcreateFlake(defeated its playwright retry twice on 2026-09-03 —Timeout …ms exceededwaiting forgetByLabel('phone-screenshot.png'); pattern scoped to that signature, 120s wrapper deadline under the 240s spec timeout)
Post-merge#
- Confirm both new rows appear on the Flake dashboard issue after a preview run on main
Implementation log#
- Preview slot 17 was damaged (missing container classes, upstream Cloudflare gap); repaired by deleting the
os-preview-17worker via the Cloudflare API (no queue consumers were attached) and letting the bootstrap recreate it on the next deploy. No dedicated repair script exists —erase-datadeliberately leaves workers in place. - Run rs7dwp1l27 attempt 3 proved the wiring end to end: both sentinels ran, recorded, and uploaded (
flake-records-specs276B,flake-records-preview-e2e286B); the job's only real failure was an unrelated mobile chat-photos flake. - Those two artifacts were then never ingested — replaying the deployed ingestion steps locally (ListRuns/ListArtifacts/download/unzip/parse) all succeeded, which isolated the run-status race fixed above.