Depth

Step 8 — Polaris: testing a service with no interface

Story: As Polaris Freight, a consignment created at 15:59 goes out today and one created at 16:01 goes tomorrow, and I can prove it without waiting until four o’clock.

Mode: LEARN — asynchronous tests written by instinct are the second-largest source of flake, after shared state.

Why now: The suite works on a product with screens. Polaris has none, and everything the browser was doing for you — waiting, retrying, showing you the state — you now do yourself.

Concepts:

There is nothing to click, so the strategy has to start somewhere else: an event goes in, a carrier is called, an event comes out, a row changes. Those are your observation points, and choosing them is the design.
Bounded waiting, never sleeping. Poll for the condition with a stated deadline and a stated interval, and fail with what the state actually was at the end. “Timed out after 5s waiting for consignment status ALLOCATED, last saw PENDING” is a test that helps.
Eventual consistency is not an excuse. “It will be right eventually” with no deadline is an untestable claim. The deadline is a product decision; make someone state it, then assert it.
A clock the test controls. POLARIS-CUTOFF-OFF-BY-ONE lives at exactly 16:00:00. Testing a boundary by waiting for real time to reach it means the test runs once a day and is skipped forever after.
Ordering you do not control. Carrier webhooks arrive out of order and sometimes twice. Your tests have to produce that deliberately — deliver acknowledgement before dispatch, deliver the same webhook twice — because production will.
Idempotency is a test, not a hope. POLARIS-DOUBLE-BOOK is the catalogue entry for it. Send the same message twice and assert there is one consignment, not two.
Asserting absence. “No second consignment was created” has no event to wait for. You need a settling condition — a processed marker, a quiet period with a stated basis — and it must not be a sleep.
Failure injection. A carrier that times out, returns a 500, or answers after your deadline. Each has a correct behaviour, and each is a test.

Step 8 in one picture. Nothing here is observable by clicking, none of it can be tested by waiting for the real clock, and two of the three marks are the seeded defects POLARIS-CUTOFF-OFF-BY-ONE and POLARIS-DOUBLE-BOOK.

Libraries: Playwright request context, a broker client, a fake clock, the carrier stubs from step 4

Expected outcome: A Polaris suite covering allocation, the cut-off boundary at exactly 16:00, duplicate webhook delivery, out-of-order delivery, and three carrier failure modes. A bounded-wait helper with deadline, interval and a state-revealing failure message. A controlled clock. An absence-assertion pattern with a defensible settling condition.

Verification


L1 — Gating test	`ACC-08` — with `POLARIS-DOUBLE-BOOK` on, the duplicate-delivery test fails and reports two consignments; with `POLARIS-CUTOFF-OFF-BY-ONE` on, the boundary test fails and names the timestamp; no fixed sleep appears in the Polaris suite; the whole Polaris suite runs in under two minutes with no real waiting.
L2 — Manual checks	(a) Force every bounded wait to time out at once and read the failures. Any message that does not say what the state was is not finished. (b) Run the Polaris suite fifty times. Anything that fails once is step 10’s problem, and you have just found it early.
L4 — Anti-patterns	`AP-08-a`, `AP-08-b`, `AP-08-c`, `AP-08-d` — full text
Done when	`ACC-08` green, and no test in it depends on real time passing

Step 9 — Contracts, stubs, and the third party that changes

Story: As a developer deploying the orders service, I find out that I broke Polaris before it reaches production, not from a lorry.

Mode: LEARN — consumer-driven contracts are easy to implement backwards, and backwards they prove nothing.

Why now: Polaris is tested against stubs. Stubs written by hand agree with what you believed on the day you wrote them, and POLARIS-CONTRACT-DRIFT is what happens next.

Concepts:

The gap contract testing fills. Unit tests cover a service alone; end-to-end tests cover everything and are slow and flaky. Contracts cover the boundary — the only part two teams can break for each other — at unit-test speed.
Consumer-driven, and what that means in practice. The consumer states what it needs from a response; the provider proves it still supplies it. A contract written by the provider is documentation, not a test.
A stub that drifts is worse than no stub. It reports success while the real thing has changed. Verified contracts are what stop the stub from quietly becoming fiction.
Third parties you cannot verify against. The payment provider and the carriers will not run your contract tests. Recorded interactions plus a scheduled check against their sandbox is the honest compromise, and the schedule is the part people skip.
What belongs in a contract: the fields you actually use, their types, and the status codes you handle. Not the whole payload. A contract over every field fails on changes that do not affect you and gets ignored.
Versioning and the deploy question. “Can I deploy this?” is answerable when contracts are verified per version. Without that, two teams coordinate by chat message.
Where end-to-end tests survive. A small number, for the journeys that cross everything and matter most. Contracts do not replace them; they make it possible for there to be five instead of fifty.

Libraries: a consumer-driven contract tool, a stub server with recording, the existing stubs

Expected outcome: Contracts between the orders service and Polaris, and between Polaris and the carrier adapters, verified in CI on both sides. Recorded interactions for the payment provider and carriers, with a scheduled job checking the recordings against the sandbox. Stubs generated from contracts rather than hand-written. A documented list of the five end-to-end journeys that remain and why.

Verification


L1 — Gating test	`ACC-09` — with `POLARIS-CONTRACT-DRIFT` on, provider verification fails and names the field, while the hand-written stub would have passed; removing a field a consumer uses fails the provider build; the scheduled recording check fails when the sandbox response shape changes.
L2 — Manual checks	(a) Ask what happens if a carrier changes a field you do not use. Your contracts should not notice. If they do, they are too wide. (b) Try to answer “can I deploy the orders service right now?” using only the contract results. If you cannot, the versioning is incomplete.
L4 — Anti-patterns	`AP-09-a`, `AP-09-b`, `AP-09-c`, `AP-09-d` — full text
Done when	`ACC-09` green, and no stub in the repository was written by hand

Step 10 — Flake is a defect

Story: As a developer, when the pipeline is red I believe it, because a test that fails intermittently is treated like any other bug.

Mode: LEARN — the diagnosis matters more than the fix, and the usual fix makes it worse.

Why now: You now have enough tests for flake to appear. Every suite reaches this point; most handle it by adding retries and lose the suite within a year.

Concepts:

What flake costs. Not the minutes of a re-run. It is that a red build stops meaning anything, so a real failure gets re-run twice and merged. The suite is then worse than having none, because it takes time and gives false comfort.
The causes, in the order you will meet them: shared state between tests, waiting for a duration instead of a condition, real time and time zones, ordering assumptions in asynchronous flows, animation and focus timing, and an environment under load behaving differently. Each has a different fix, and none of them is a retry.
Detect before you argue. A nightly --repeat-each run on the full suite gives you a flake rate per test. Without that number every conversation is anecdote.
Quarantine is a deadline, not a bin. A flaky test moves out of the gate, gets an owner and a date, and deletion is the honest outcome if the date passes. A quarantine with no expiry becomes a graveyard of tests nobody trusts and nobody removes.
Retries, argued precisely. Retrying hides the one class of bug that presents as intermittent — a race condition — and a race is exactly what Trailhead’s TRAIL-STOCK-OVERSELL is. Where a retry is allowed, it is reported as a flake, counted, and never silent.
A flaky test is a real report. Sometimes the product is genuinely intermittent and the test is the only thing noticing. Rule that in or out before touching the test.
Flake rate is a tracked number with a target, reported in step 16 alongside detection rate.

A test fails one run in twenty

Which of these six is it? Answer before touching the test.

Shared state between testsPer-worker data, and a fixture instead of a shared variable
A wait on a durationWait for the condition, with a deadline
Real time, or a time zoneA clock the test controls
An ordering assumptionAssert on the outcome, and test the reordered delivery
Animation or focus timingWait for the settled state, not the transition
A load-dependent environmentBound the resource, or move the check to a cheaper level

The tempting fixadd a retryIt makes all six disappear from the report, and hides the one cause that is not a test problem at all — a real race in the product, likeTRAIL-STOCK-OVERSELL, which then ships.

Step 10. Flake is a defect with an owner, and every cause has a different fix.

Libraries: Playwright’s repeat and retry options, the reporter from step 4

Expected outcome: A nightly repeat run producing a per-test flake rate stored over time. A quarantine mechanism with an owner and an expiry, excluded from the gate and visible in the report. A written flake policy in AGENTS.md: detection, triage, the six causes with their fixes, when a retry is permitted and how it is reported. Every currently flaky test diagnosed by cause, not patched.

Verification


L1 — Gating test	`ACC-10` — the full suite passes a hundred-run repeat with zero non-quarantined failures; every quarantined test has an owner and an expiry date, and the build warns when one expires; a retried test appears in the report as a flake rather than a pass; with `TRAIL-STOCK-OVERSELL` on, the race test fails consistently rather than intermittently.
L2 — Manual checks	(a) For each flaky test you fixed, write down which of the six causes it was. Any you cannot classify, you have not diagnosed. (b) Ask a developer whether they trust the pipeline. Ask again in a month. That answer is the real metric.
L4 — Anti-patterns	`AP-10-a`, `AP-10-b`, `AP-10-c`, `AP-10-d` — full text
Done when	`ACC-10` green, and no test in the gate has an unexplained failure in its history

Harness impact: AGENTS.md v3 — the flake policy in full, the retry rule, and the instruction that an agent may never fix an intermittent failure with a retry or a wait.

Step 11 — Beechwood: accessibility, personal data, and evidence

Story: As a patient using a screen reader, I can book an appointment; and as the person signing the release, I have the evidence that this was checked.

Mode: BUILD — the checks are mechanical, and the judgement about what they miss is not.

Why now: After the suite works, because this adds a whole class of check to it. Before the gate, because Beechwood’s release conditions depend on the output.

Concepts:

Accessibility is a requirement here, not a preference. For a health service in most jurisdictions it is a legal obligation with a published standard, and “we will do it later” is a decision with a liability attached.
What axe-core finds and what it cannot. It catches contrast, missing labels, roles and structure — perhaps a third of real barriers. It cannot tell you that the focus order makes the form impossible, that an error message is announced to nobody, or that BEECH-FOCUS-TRAP locks a keyboard user inside a modal.
The manual checks that belong in every release: the whole journey by keyboard only, the same journey with a screen reader, and the page at 200% zoom. Half an hour, and it finds what the scanner cannot.
Automated checks go where a person will fix them — in the suite, failing the build on new violations, with existing ones recorded as a known baseline rather than ignored.
Personal data in the test estate. Beechwood’s manufactured data still looks like patient data. Screenshots, traces and reports hold whatever was on screen, so retention and access apply to your artefacts too.
Evidence as an output. The release record for Beechwood needs what ran, the accessibility result, the manual checks with who did them, and the accepted failures. Produced by the suite, not assembled by hand at 17:00.
Traceability. A regulator’s question is “show me this was tested”. A test referencing a risk ID referencing a requirement answers it in one step.

Libraries: axe-core integrated with Playwright, the reporter, a screen reader for the manual pass

Expected outcome: Automated accessibility checks on Beechwood’s main journeys, failing on new violations against a recorded baseline. A documented manual pass — keyboard, screen reader, 200% zoom — with a checklist and a place to record who ran it. Artefact retention and access rules applied to traces and screenshots. A generated Beechwood release record containing test results, accessibility status, manual checks and accepted failures, in one page of plain language.

Verification


L1 — Gating test	`ACC-11` — with `BEECH-FOCUS-TRAP` on, the automated checks pass and the documented keyboard pass catches it, and this outcome is written down as the reason the manual pass exists; a new contrast violation fails the build; the release record generates from a run with no hand editing.
L2 — Manual checks	(a) Book an appointment using only the keyboard, then again with a screen reader. Write down every point you were lost. (b) Give the release record to someone non-technical and ask whether they would sign it. Their questions are the missing fields.
L4 — Anti-patterns	`AP-11-a`, `AP-11-b`, `AP-11-c`, `AP-11-d` — full text
Done when	`ACC-11` green, and the release record answers “was this tested?” without you in the room

Step 12 — Exploratory testing, and a bug report that gets fixed

Story: As the quality engineer, I find the problems nobody wrote a test for, and my reports get fixed instead of closed as “cannot reproduce”.

Mode: LEARN — a practice, and the one place in this material where the skill is entirely yours.

Why now: Automation now covers what you thought of in step 1. This step is about everything else, and it needs a working system to explore.

Concepts:

Automated tests only check what you already imagined. Every significant defect that escapes was outside somebody’s imagination, and the only tool for that is a person looking with intent.
Exploratory is not unstructured. A charter states what you are exploring, for how long, and what would make it interesting: “Explore the Beechwood appointment cancellation flow for two hours, looking for states where a patient sees another clinic’s data.” Time-boxed, recorded, repeatable by someone else.
Session notes are the deliverable: what you tried, what you saw, what you did not get to, and the questions raised. A session with no notes cannot be reviewed, handed over, or counted.
Where to aim. Boundaries, the back button, two tabs, interrupted flows, the second-most-common path, error recovery, and anything the team says “nobody would do”.
A bug report that gets fixed has exact steps from a known state, the actual and expected result, the environment and build, evidence, and a one-line statement of why it matters. Anything less is negotiation.
“Cannot reproduce” is usually a missing precondition — a specific account, a leftover state, a timing. Finding it is part of the report, not the developer’s problem.
What becomes a test and what does not. A real risk that recurs becomes an automated test; a one-off in a feature being replaced next month does not. Deciding deliberately keeps the suite from bloating with everything you ever found.
Charters point at the gaps you wrote down in step 1. The “not automated” list is your exploration schedule.

Libraries: none — notes, a timer, and the products

Expected outcome: At least six completed exploratory sessions across the three products, each with a charter, a time box and session notes. A bug report template used for every finding. At least three defects found that the suite did not catch, each with a decision on whether it becomes a test and why. The “not automated” list from step 1 updated with what exploration revealed.

Verification


L1 — Gating test	`ACC-12` — six sessions with charters, time boxes and notes; at least three findings the automated suite missed, each with steps that reproduce from a stated starting state; each finding has a written automate-or-not decision with a reason.
L2 — Manual checks	(a) Hand a bug report to someone else and have them reproduce it from the text alone. Every question they ask is a missing field. (b) Re-read a session’s notes a week later. If you cannot tell what you did not get to, the notes are too thin.
L4 — Anti-patterns	`AP-12-a`, `AP-12-b`, `AP-12-c`, `AP-12-d` — full text
Done when	`ACC-12` green, and at least one finding is a defect the suite could not have found