A regression suite is only worth the trust placed in it. This is a short account of taking one from "fails for reasons nobody investigates" to "every failure has a named cause", measured across its complete run history.
A Playwright and Pytest end-to-end suite validates a live GitHub Pages site on every deployment. Because the target is a real deployed page rather than a local build, the suite inherits every timing problem the deployment pipeline has, and there is no local reproduction to fall back on when a run goes red.
The pipeline waited for the site to answer HTTP 200 before launching a browser. That check is worthless as proof of freshness: the previous build answers 200 just as happily. Runs began while a deployment was still propagating, so early tests saw the old markup and later tests saw the new. The result was a locator timing out on an element that was never broken, in a run that could not be reproduced afterwards, because by then the CDN had settled.
What was changed. The gate now waits on two real signals instead of one meaningless one: the site repository must report no queued or in-progress Pages deployment, and the served ETag must repeat across consecutive polls, proving the edge has settled on one version. Only then does a browser context start.
A failure that only says "locator timed out" moves the work to whoever picks it up: check out the branch, install browsers, reproduce, guess. That is the expensive part of time-to-resolution, and it scales with the number of people who ever look at a red build.
What was changed. Every failure now ships its own evidence, attached to the report rather than left in a log: a full-page screenshot, the fully rendered DOM including script-injected nodes, a replayable Playwright trace for stepping through the run, and a structured root-cause verdict generated by an LLM from the post-failure DOM. The triage step is strictly advisory: a missing credential or an API error degrades to "no report" instead of turning a product failure into an infrastructure failure.
Each push queued a full suite run, so a burst of five commits paid for five executions of which only the last described a state anyone cared about.
What was changed. Concurrency groups scoped to workflow, event and ref, with cancel-on-supersede, so a burst collapses to roughly one execution. The same policy is deliberately inverted on a sibling API suite: there the scarce resource is a third-party request quota shared across all branches rather than compute, so runs serialize and in-flight runs are allowed to finish. The scarce resource dictates the policy.
Figures below are counted from the complete run history of the pipeline, not estimated.
| Measure | Result |
|---|---|
| Unexplained (flaky) failures | Zero across all 27 runs. Every failure resolved to a named cause; none was retried away or ignored. |
| Failures that were real product defects | 5 consecutive runs held red by one genuine mobile-layout bug: a 398px document inside a 390px viewport, from a missing box-sizing on stacked cells and a table sized by its intrinsic min-content width. The suite found it on its first execution against that viewport. |
| Failures that were infrastructure | 3 runs, every one the same cross-repository ordering race and every one diagnosed from the CI log alone with no local reproduction. Two had the site deployed ahead of the framework, one had the framework ahead of the site; in each case the error named the colliding elements outright. The lesson is in What transfers below. |
| Runs since, on push and manual dispatch | 8 of 8 green. |
| Suite cost | 61 tests, of which 10 are generated at collection time by crawling the site's own route graph. The deployment pipeline runs 59 of them: the two checks that reach a third party - resolving outbound links, and reading the published container image to confirm it still matches what the site claims - are deselected and run weekly instead, so no deploy waits on GitHub or Docker Hub. Roughly 90 seconds of wall clock per job including the deployment gate, dependency install and report generation. The two figures above are read back from this page on every run and compared against the suite that is executing, so they cannot drift out of date unnoticed. |
| Quality gate | Static analysis at a blocking 10.00/10, evaluated before any browser starts, so a lint regression cannot consume browser minutes. |