failingTest: expected-fail pins with a concrete failure assertion
Status summary#
Done, pending review. A failingTest helper replaces bare test.fails for pinned
platform bugs, then the quarantined facet source-version pin converts to it
— with a restructure that makes it immune to the coincidental-recycle false
alarm that got it quarantined (7+ red runs).
The problem with test.fails#
test.fails passes when the body fails for ANY reason and goes red when the
body succeeds. Two failure modes bit us:
- A body that starts failing for a DIFFERENT reason (infra error, typo) stays silently green — the pin no longer pins anything.
- A body that succeeds coincidentally (the facet pin's recycle-in-the-poll- window) goes red exactly like "bug fixed" — 7+ false alarms, each flipping an unrelated PR's CI.
The helper (Misha's sketch)#
failingTest({ failure: /SAME-BOOT STALENESS/ }, "facet picks up commits", async () => {
// asserts the DESIRED behavior; today it throws the matched error
});Semantics: body fails matching failure → test passes (bug still pinned).
Body fails with anything else → red ("expected failure to match X, got Y").
Body succeeds → red ("should have failed — if the bug is fixed, delete the
wrapper and keep the body as a plain test"). One care point from the sketch:
the success-path throw must live AFTER the try/catch, or the catch treats it
as a candidate failure.
Checklist#
-
failingTest(options, name, body)in shared test-support, with the assertion core (expectFailure) exported separately so it can be unit tested without nesting vitest. Options:{ failure: RegExp, timeout?: ms }. - Unit tests: matching failure passes; mismatching failure rejects naming both patterns; success rejects with the delete-the-wrapper message.
- Convert
userspace-facet-source-version.e2e.test.tsfromtest.skipback to a live pin viafailingTest, restructured for coincidence immunity: after each source commit, classify answers by BOOT — same-boot stale accumulates toward the concreteSAME-BOOT STALENESSerror (matched by the wrapper), same-boot new-revision means fixed (body succeeds → wrapper alerts), a DIFFERENT boot means the facet recycled coincidentally → the round is inconclusive, so commit a fresh revision against the now-running facet and observe again (bounded rounds; all rounds interrupted → a distinct loud error). The unchanged-source stability coverage in the same test comes back for free. - Update the quarantine note in
tasks/platform-stall-repros.md(thread 5's exit happens via coincidence-immunity now; provenance remains a nice-to-have, no longer the gate). - Short docs note in docs/testing.md next to the quarantine protocol: when to use failingTest vs test.fails vs skip.
- Verify the converted test against local dev: green (bug present, error matches) 60s run; green under failingTest proves the body threw the pinned SAME-BOOT STALENESS error
Implementation log#
(appended as work happens)