Caught Regressions
An eval harness earns its keep by what it catches. These are four real,
documented catches from this repo’s history — what broke, how the harness caught it, and
the evidence. Nothing on this page is invented; every claim links to the issue, PR, or CI
run that recorded it.
M3 · BASELINE COMPARE
Deliberate scorer sabotage turned CI red
- What broke
- For the M3 acceptance test (PR #39), the outcome_success scorer was deliberately sabotaged on a branch — a change that would have silently shipped a broken gate.
- How the harness caught it
- The artifact-based baseline compare diffed the branch scorecard against the last green main run and failed the job with a readable per-metric breakdown.
book-happy-path: passed → failed
outcome_success: 1 → 0
aggregate 6/6 → 5/6
M4 · SCRIPTED VS REAL
First real-LLM runs made zero tool calls
- What broke
- The first real Claude-driven runs completed “successfully” while making ZERO tool calls: empty tool schemas, no call context in the prompt, and the seam design’s outcome-mode scoring had never been implemented (issue #43).
- How the harness caught it
- The M4 dual-adapter comparison published RED on the live dashboard instead of silence — every scenario regressed scripted → real, visibly. Fixed in PR #44: real per-tool JSON schemas, a scenario-derived call-context block, and order-independent outcome scoring; real runs then scored outcome_success 6/6.
M4/M5 · LATENCY GATE
Real runs green on outcome, permanently red on the 800ms gate
- What broke
- After #43 was fixed, real runs booked correctly (outcome_success 6/6) but stayed red forever: real Claude turns take 2–7s against the 800ms Tier-1 latency budget — a mis-calibrated gate, not a mis-behaving agent (issue #45).
- How the harness caught it
- The harness surfaced the miscalibration instead of hiding it: the replay timing overlay redlined every real turn against the 800ms budget, making the gate itself the visible defect. Redesigned in PR #46 to a real-run budget, with the 800ms redline retained as the reference line.
M7B · PHASE-0 GATE
The telephony architecture was built on packages that don’t exist
- What broke
- The M7b PSTN architecture (and the 2026-07-11 ADR behind it) assumed three Cloudflare packages its own docs present as installable: @cloudflare/voice-twilio is an empty 0.0.0 stub and -deepgram/-elevenlabs were never published. A sprint would have been written against imports that throw.
- How the harness caught it
- The Phase-0 gate doctrine (verify the prior-art assumption before building on it) caught it at the npm registry, and a 3-engine QA pass then killed the gate plan’s own v1 — whose A4 test could never have fired onTurn (end_of_speech is a server-side no-op) and whose A3 tested the mock instead of the pipeline. The executed v2 gate passed all four assertions in workerd; M7b proceeds on a hand-written transport.
[email protected]: tarball = package.json only
voice-deepgram, voice-elevenlabs: 404
A1–A4: PASS (spike-findings §18)