Minerval

← evals

EvalIs the pipeline stable?

Model discovery and adoption

Which models the providers now offer that the code does not know, and what a candidate buys per dollar before anyone changes a pin.

discover: network only · adopt: two runs

What it checks

Discover polls the three providers’ model lists and diffs them against everything the code registers (the model tables, the production pins, the pricing table): candidates the code has never heard of, registered models a provider no longer lists (the drift the per-PR guard cannot see), and prices that OpenRouter resells at a rate other than the one the code bills through. It opens no issues; --json is the hook for a workflow that would.

Adopt runs a candidate through the agent’s own suite with the judge pinned: for the The agent that decides whether a claim is already in the graph, possibly worded differently or as its denial.agent → the golden pairs on candidate and incumbent, plus a swap when a cluster is given; for the other agents a swap. It summarises quality per dollar in one line: adopt, hold, or reject. A person decides; the pin changes by pull request.

Detail

How adopt runs, step by step
  1. Resolve the incumbent. the agent's model after --profile, from the same config the run would use.
  2. Golden pairs twice. for the Matcher: the suite on the candidate and on the incumbent, each a child process, each filed and registered as a golden run.
  3. Swap. when --cluster is given: arm A on the incumbent, arm B on the candidate, snapshotted and compared by graph agreement.
  4. Summarise. pass-rate points per dollar and fidelity per dollar against fixed tolerances (a golden delta within 0.05, an F1 above 0.85 is close, below 0.6 is a reject) into one recommendation, registered as kind adopt.
What it cannot show
  • The golden delta is exact-match on a saturating task; the swap is one sample of each arm.
  • The Extractor, Steward and Curator have no per-decision golden set, so their verdict rests on fidelity and cost alone.
  • Discover’s idea of a candidate is a deliberately narrow list of vendors; pricing drift compares against OpenRouter’s resold rate, not the vendor’s page.
Run it
npm run models:discover -- --json
npm run corpus:adopt -- --agent=matcher --model=<id> --profile=production
npm run corpus:adopt -- --agent=steward --model=<id> --cluster=eggs --baseline=<snapshot>