Minerval

← evals

EvalIs a graph well built?

Comparing runs

One run is one sample. A change counts only when it is larger than the spread between runs that changed nothing.

no group of runs exists yet; no verdict ever issuedthree runs per side

The rule

Noise band: three runs on each side; the difference between the two means counts only when it exceeds the sum of the two spreads.A: mean ± spreadB: mean ± spreadΔ of means: a result only if larger than spread A + spread B

Each side of a comparison is a group of about three runs. A metric’s value is the group mean; its noise is the sample spread. The difference of means counts only when it exceeds spread A plus spread B.

A side with one run gets its difference printed and no verdict. A verdict against one side’s spread alone is marked one-sided. Code: band.ts.

Detail

What it cannot show

Which side is better. Only whether the difference is real. Better is the judge’s and the reviewer’s question.

Run it
npm run corpus:compare -- A1.json,A2.json,A3.json B1.json,B2.json,B3.json
npm run corpus:compare -- db:<idA> db:<idB>         # single runs: deltas, no verdict