Roadmap Overview

Locked decisions

Decision	Value
Assumed baseline	You write automated tests already. New to test strategy, contracts, and owning a release decision.
Language / runtime	TypeScript 5.9 on Node 24, `strict`, `any` banned
UI and API runner	Playwright — one runner for browser journeys and API tests, so fixtures and reporting are shared
Why one runner	Two runners means two fixture systems, two reports and two answers to “did it pass”. The gate needs one.
Contracts	Consumer-driven contracts between the products; recorded stubs for third parties (payment provider, carriers)
Asynchronous assertions	Bounded polling against a stated deadline. `sleep` with a fixed duration is banned by a lint rule from step 4.
Time	A clock the test controls. Cut-off behaviour is not tested by waiting until 16:00.
Test data	Generated by factories, isolated per worker. Never a copy of production data, in any product.
Accessibility	`axe-core` in the suite for Beechwood, plus manual keyboard and screen-reader checks that no tool replaces
Performance	k6, from step 13, answering one written question per product
Proof of detection	The shipped defect catalogue. An acceptance test means “these defects, switched on, turn the suite red and name themselves”.
Flake policy	A flaky test is a defect with an owner and a deadline. Retries are allowed only where step 10 says, and they are always reported.
Pipeline	GitHub Actions. Sharded, selected, with a stated runtime budget from step 15.
Release gate	A build passes or is blocked with a stated reason, recorded. No verbal approvals.
Systems under test	The shipped products only. Never a real company’s system, never real patient data.
Quality	ESLint + Prettier, `tsc --noEmit` strict, applied to the test code because it is code
Ops	Docker Compose on one host — the three products, PostgreSQL 18, a message broker, and the stub servers
AI harness	`AGENTS.md` from step 4, evolving v1→v3

Note on step count: 18 numbered steps, 0 through 17. Steps 0–3 contain no code and produce the documents everything later is argued from.

Why this domain

Testing is the one engineering discipline whose output reports on itself. A suite that tests nothing passes, quietly, for years. Which means the interesting problems here are not about writing assertions:

You cannot test everything → deciding what not to test, and defending it
Green is not proof → seeded defects, detection rate, and the difference from coverage
Flake destroys trust faster than bugs do → a policy, an owner, a root cause per case
Some products have no screen → contracts, events, bounded waiting, a controlled clock
Some data may never be copied → generated data and per-worker isolation
Some checks are legally required → accessibility as a requirement, and evidence as a deliverable
Somebody signs the release → a gate with reasons, not a feeling
Pipelines rot → a runtime budget defended every step after 15
Bugs escape → the review that turns an escape into a test

The three example products

One repository, one release train, three products with different rules.

Product	What it is	The rule it exists to break
Trailhead Outfitters `trailhead`	An outdoor gear storefront — browse, basket, checkout, a third-party payment provider	The baseline, and the trap. Everything is clickable, so everything gets clicked. It teaches the opposite: set state through the API, assert the one thing the journey exists to prove, and keep tests independent enough to run on eight workers at once.
Beechwood Health `beechwood`	A patient portal for a group of clinics — appointments, letters, results	You may not copy production data, so the data has to be manufactured and still be realistic. Accessibility is a legal requirement, not a backlog item. And a release needs written evidence a non-engineer signs, which makes the report an output of the suite rather than a screenshot someone pastes into a chat.
Polaris Freight `polaris`	A dispatch service allocating consignments to carriers — no user interface at all	There is nothing to click, so a UI-shaped strategy has nothing to say. Work arrives as events, carriers answer minutes later, webhooks arrive twice and out of order, and a 16:00 cut-off changes what “correct” means. It forces contracts, bounded waiting, idempotency checks and a clock you own.

Questions to keep asking: if Trailhead’s checkout test fails, do you know whether it is the basket, the stock service or the payment stub? If Beechwood’s release record landed on a regulator’s desk, would it answer the question they asked? And if Polaris receives the same carrier webhook twice, does anything in your suite notice?

None of the three may be special-cased in the test code. A helper with if (product === 'beechwood') in it means the abstraction is wrong, not that Beechwood is unusual.

The proof rule

The centre of the material, and the thing that separates it from a tutorial about a runner.

A suite is proven by the defects it catches, not by passing.

all flags off

✓checkout completes
✓discount applied
✓stock decremented
✓order confirmed

no failure to read

What you knowNothing.

TRAIL-DISCOUNT-ROUNDING on

✓checkout completes
✗discount applied
✓stock decremented
✓order confirmed

order total was 48.59, expected 48.60

What you knowThis suite catches that bug, and names it.

A suite is proven by the defects it catches, not by passing. Every acceptance test in this material is stated as the right-hand panel.

The products ship with a defect catalogue — real bugs, each behind a named flag, each switchable from an environment variable or a seed:

Example defect	What it looks like in the product	What it punishes
`TRAIL-DISCOUNT-ROUNDING`	A multi-item discount rounds the wrong way, so the total is a penny out	Assertions on “checkout completed” rather than on the amount
`TRAIL-STOCK-OVERSELL`	Two simultaneous checkouts both take the last item	Sequential-only tests, and no test for a race anyone hit in production
`BEECH-LETTER-LEAK`	A patient letter is visible to the wrong clinic’s staff	No authorisation matrix — step 14
`BEECH-FOCUS-TRAP`	A modal that cannot be left with a keyboard	An accessibility check that only runs `axe-core` and stops
`POLARIS-DOUBLE-BOOK`	A carrier webhook delivered twice creates two consignments	Happy-path event tests and no idempotency check
`POLARIS-CUTOFF-OFF-BY-ONE`	An order at exactly 16:00 is treated as next-day	Tests that use the real clock and never hit the boundary
`POLARIS-CONTRACT-DRIFT`	A carrier stub returns a renamed field	Stubs written by hand and never checked against the real schema

Rules that keep the catalogue useful:

Read the flag name, never the implementation. Looking at the defect’s code before you build the test turns detection into transcription.
A defect the suite catches without naming is half a catch. “Checkout test failed” costs a developer an hour. “Order total was 48.59, expected 48.60” costs a minute.
Detection rate is a number you track from step 16. Defects caught, out of defects switched on. It is the honest version of coverage.
Add your own. Every escaped defect in step 17 becomes a catalogue entry and a test.

The quality system (target state)

Everything upstream of the gate is work. The release record is the part somebody outside the team ever sees.

A test carries the risk it exists for. A test nobody can explain is deleted at step 16, not kept for comfort.
Flake history lives with the result, not in somebody’s memory. Step 10 makes it a record.
The release record is the artefact of the whole material. It is what a portfolio shows and what an auditor reads.

Global guardrails (Verification Layer 5)

Guardrail	From step	What it catches
`npm run verify` — lint, types, unit tests, suite	4	Everything below
`tsc --noEmit` strict on the test code	4	Test helpers that quietly accept anything
The no-fixed-sleep lint rule	4	The most common cause of flake, before it is written
The defect-catalogue run	5	A suite that passes while a seeded defect is switched on
The repeat run (`--repeat-each`)	10	A test that passes nine times out of ten
The independence run (`--shuffle`, full parallel)	6	Tests that depend on each other’s leftovers
The runtime budget check	15	A pipeline drifting past the number you committed to
Required checks on the branch	15	A release that went out without the gate
CI on every push	4	The above, on a machine that is not yours

The independence run is the one that hurts the first time. Almost every suite written without it has order dependencies nobody knew about.

Reading the steps

Steps 0 to 3 have no code and are not optional. They are where the arguments you will have with a product manager are settled: what is not being tested, why the data is generated, and what “ready” means before there is a release to argue about.

Mode is not a suggestion. Test architecture, asynchronous assertions and flake diagnosis all produce code that looks reasonable and is quietly useless. An agent writes plausible tests very fast, and plausible tests that never fail are the whole problem.

Step 10 is the step. Every suite is green on the day it is written. Step 10 is the month later, when it fails twice a week for no reason and the team starts ignoring it. Nothing else in this material matters if that happens.