Minerval

← docs

Evals

The properties of the claim graph we can measure, and the test for each. Click a row.

Watch a run: the agents building a graph, played back event by event · 5 recordings.

evalproperty checkedstatuslast runresultcost
Is a graph well built?
Corpus runsthe pipeline builds a graph from fixed documents, on recordrun2026-09-1811 runs, capped, dev models$80 – 200 est.
Structural scorecardextraction, wording, merging, depth and verdicts look sanerun2026-09-1832 claims, dedup 1.94, single samplefree
Judged scorecardclaims and verdicts meet the constitution's standardsrun2026-09-18claim bar 100% of 11, single sample≈ $1
Judge reviewthe judge's task is one a person agrees withdone once2026-09-024 task fixes; 4 new dimensions unrevieweda person's hour
Comparing runsa change is larger than run-to-run noiseno group of runsnot yetno verdict ever issued3 runs per side
Epoch gatea change's scored runs do not regress the baseline beyond noisebuiltnot yetrefuses: one baseline runfree
Is the pipeline stable?
Golden pairsthe Matcher tells same, denial, narrower and different apartrun · in CI2026-09-1629/30$0.10
Graph agreementtwo graphs from the same documents can be comparedbuiltnot yetnot yetfree
Same graph twicethe graph is stable under a repeat and a shuffled orderbuiltnot yetnot yet2 runs
Model swapa cheaper model builds the same graph as the strong onebuiltnot yetnot yet2 runs
Cascade stabilitya reassessment ripples less than one-for-one (R < 1)on every runeach runprinted per run; no committed recordfree
Assessment historycredence moves with the evidence's sign and discriminates reversalsbuiltnot yetnot yetfree
Canonical-form goldensthe Extractor writes the neutral, debated-direction wording and keeps good wordingrun2026-09-1810/26$0.10
Model discovery and adoptionnew and withdrawn models are noticed; a candidate is priced before a pin movesbuiltnot yetnot yetnetwork; 2 runs
Does governance work?
Contribution scenariosreviews, escalations and appeals behave under scripted inputbuiltnot yetnot yet$10 – 30 est.
Adversarial robustnessillegitimate technique buys no more movement than a sincere casebuiltnot yet4 targets pinned6 – 9 arms
Where can reality check it?
Predictionsthe Steward's probabilities are calibrated against outcomesseedednot yet22 questions, 0 resolved≈ 22 Steward runs
Does it hold up in use?
Reasoner probea reasoner's confidence follows the record; a retraction reaches dependentsbuiltnot yetnot yet2 calls per question
Simulated usersreaders, contributors, scripts and a hostile minority are served and handledbuiltnot yet20 personas20 sessions
Production monitorsthe live graph's own record flags what deserves a second lookbuiltnot yetscheduler offfree
How they fit togetherHow the evals fit together: a run makes a graph; counts and a judge score it; two runs are compared for stability; fixed cases test the Matcher; scripted contributions test governance; predictions are checked against reality.a corpus run builds a graphscore itcounts · judge · reviewbuild it againsame · shuffled · swappedcontribute to itReviewer · Arbitratorcompare runs3 vs 3, beyond the noiseagreementhow far the graph movedgolden pairs30 fixed Matcher cases, centspredictions: the one place reality grades the graph

Background

The documentsModels and costGround rulesThe rubricWhat a run leaves behindNot measured yetRun it yourselfReading a replay

Rendered from files committed to the repository, synced 2026-10-28 at commit cbdd425. Nothing here reads a live database.