Background
Not measured yet
What no eval covers, and where the design itself may be wrong.
Not measured
- Anything on the production models. Every committed run so far is on the cheap tier or the development defaults, capped. The next thing to do is three production-profile runs of one cluster.
- The noise floor. Idempotency ran once on the cheap tier and lost its second arm; no comparison has a scale yet.
- Anything adversarial, in results. The suite, the gambit library and the red team are built; no arm has run.
- Governance under simulated people. The scenario and the personas are built; the Reviewer and Arbitrator have been read once, under the scripted scenario, on the cheap tier.
- Model fidelity. No swap has run; the allocator’s tiering rests on a guess.
- The judge’s newest dimensions. Sycophancy, hedging, canonical-form strength and political bias: judged on cheap-tier runs, never reviewed.
- Calibration. Nothing seeded into production, nothing resolved, no market baseline.
- Paraphrase and frame invariance. The same proposition reworded, or stated by a proponent and a critic, has no arm.
- The live graph, as read. The monitors read it; nothing measures the quality of what visitors read.
Detail
Where the design may be wrong
- The judge is weaker than the judged. Sonnet grades Opus 5.5, one model family, no second judge.
- The judge’s reviewer wrote its prompt. An outside reader would be a stronger check.
- Agreement is measured by another matcher, with a hand-picked embedding threshold never checked against the golden pairs.
- The golden suite is thirty in-house pairs at its ceiling. It will catch regressions on those thirty.
- Clusters are small; the judge sample is a fraction of a large one and the whole of a small one.
- The cost model is extrapolated from one capped Sonnet run.
- Predictions are in-house and correlated, with no crowd baseline.
- Authorship counts rewrites, not improvements. Whether the Matcher’s rewording was better is not judged.
- The 1–5 scales may be dead weight. The first review found they carried nothing the flags did not.
The plan's nine suites
| suite | what | standing |
|---|---|---|
| S1 | per-PR golden suite | Matcher pairs in CI; canonical-form goldens built |
| S2 | quality scorecard | built; newest four dimensions unreviewed |
| S3 | properties and stability | six properties, cascade stability and assessment history built; paraphrase and frame invariance not |
| S4 | adversarial robustness | all four cells built; never run |
| S5 | downstream-reasoner probe | built; never run |
| S6 | calibration track | built; not seeded into production |
| S7 | model economics and lifecycle | guard, swap, discover and adopt built |
| S8 | persona simulation | phase one built (twenty personas); never run |
| S9 | production monitors | built; scheduler off by default |