← the Atlas

E2E Graduation — Faster Without Losing User Journeys

Analysis·evergreen·implemented·

How Kolu moved redundant browser assertions to lower layers while retaining every distinct user-level journey and making future removals auditable.

A test should live at the lowest layer that can honestly assert its promise, but “a lower test exists” is not proof that it asserts the same thing. #2064 applied that rule to all 58 feature files. The suite moved from 530 Gherkin declarations and 546 expanded executions to 494 declarations and 508 executions. Default CI moved from 541 to 503 Linux executions and from 540 to 502 Darwin executions: 38 browser executions removed without removing a distinct user journey.

live Gherkin suite 58 files · 530 declarations 508 expanded executions immutable inventory steps · backgrounds · tags expanded Examples values attempt timing every try · retry · duration Linux + Darwin baseline coverage-ledger gate exact historical revision · user promise · real defect surface collected replacement · human review note · browser survivor later change may remove exactly that E2E revision all required evidence present; no count target
The graduation gate. CI preserves scenario history, checks complete ledger rows, collects every named replacement test, and requires current browser survivors. Human review—not the schema—decides whether a lower replacement or stronger retained journey covers the same defect surface.

What was safe to graduate

The removed scenarios repeated assertions already owned more directly by pure logic, component, fork-free integration tests, or a strictly stronger retained browser journey. Each retirement names its historical scenario revision, user promise, defect surface, independent browser survivor, and—when its action is replace—lower-layer evidence with declared platforms. The lower tests include production reducers and state machines, real temporary files and watchers, real SQLite, short-lived Git repositories, and render-from-state component tests.

The perfection review restored candidates whenever the browser contributed a fact the lower test did not own. That kept the full Code-tab local / branch / browse behavior matrix, Markdown sanitization and rich-render composition, real agent-token lifecycle paint, terminal switcher dismissal and split exclusion, subterminal resize, intent slot selection, file-drop size and media boundaries, canvas close-pan, theme shuffling, recent-agent filtering, Kaval PID behavior, and the other distinct composition seams found during the sweep.

The result is deliberately conservative. A scenario stayed in E2E if its promise emerged from browser or system composition: deep links, printed URLs, mobile gestures and the soft keyboard, reconnect, xterm behavior, canvas geometry/WebGL, focus and chord journeys, worktree composition, or any boundary for which the lower test could remain green while the user path broke.

What the guardrail actually proves

Mechanism Source of truth Mechanical guarantee
Scenario inventory packages/tests/scenario-inventory.json Every revision fingerprints backgrounds, steps, tags, data tables, doc strings, and expanded Examples values. CI compares the current file with every committed version reachable from HEAD, so historical records cannot be edited or removed.
Coverage ledger packages/tests/coverage-ledger.yaml Every revision absent from the live suite has exactly one landed row naming its promise, defect surface, destination, and current browser survivors. A replace action must also name replacement evidence.
Test collection packages/tests/governance/ledger.ts Every named replacement and review-evidence test is currently collected, with at least one declared platform and a stated realism boundary. The platform list is a declaration, not proof that this check ran the test there.
Attempt timing Cucumber message stream → reports/e2e-timing.json Every attempt is retained with status, duration, retry verdict, final-attempt totals, and per-feature totals. A passing retry cannot erase the failed attempt’s cost.
Starting census packages/tests/baseline-census.json The original 58 / 530 / 546 suite and its 541 Linux / 540 Darwin default counts remain pinned.

The ledger is an audit index, not an equivalence oracle. CI proves immutable history, ledger completeness, active test collection, non-empty platform declarations, and retained current scenario identities. Human review verifies that each replacement exercises the same promise and defect surface; reviewEvidence.note records that judgment explicitly as non-executable evidence. This is more honest than a generic “counterfactual” sentence that no tool actually ran.

The runtime lesson

The exact macOS CI host exposed a harness defect before it supplied a useful baseline. Its interactive shell resolved /usr/bin/git, an Xcode command-line-tools stub, so strict commands failed and retried for more than an hour. The E2E dev shell now carries Nix’s Git, and the fixture shell restores the inherited Nix PATH after macOS /etc/zprofile. Command helpers again fail on nonzero status; intentionally interactive or failing commands use an explicit start-only step. This changed no product behavior, but made false-green setup failures impossible in the test harness.

The campaign therefore has a censored baseline—the original strict Petit suite did not settle within one hour—rather than a fabricated five-sample median. The final suite’s settled measurements live in docs/e2e-runtime-ralph-report.md.

The durable decision rule

For a future retirement:

  1. Name the exact immutable revision and user promise.
  2. Add the lowest-layer test that crosses the same real defect surface.
  3. Retain an independent browser scenario that still walks the user path.
  4. Record the human equivalence review without claiming CI performed it.
  5. Land a complete ledger row and remove only that revision.
  6. Measure the whole pipeline, and restore the browser proof if either coverage review or runtime evidence fails.

There is no target retirement count. The count is an output of evidence, never a quota.