Minerval

← evals

EvalIs the pipeline stable?

Golden pairs

Thirty fixed cases for the Matcher: is this new sentence a claim already in the graph, its denial, a narrower version, or something else?

30/30 on 2026-08-08in CI, waiting on repository secrets$0.06 per run

What it checks

Whether two sentences are the same claim is the central judgment in the system: a claim and its denial are one node (§2), a narrower claim is a different one, and a sentence that only resembles a claim is new. The The agent that decides whether a claim is already in the graph, possibly worded differently or as its denial.agent makes that call for every claim that enters the graph.

Thirty cases are fixed in a file. Each seeds the “already in the graph” claims into the test database and asks the real Matcher about a new sentence; a case passes when the decision, the matched claim, and the stance are all as expected. The suite runs in CI on any change that could move a decision.

Detail

The 30 pairsthe fixture, verbatim

File: matcher-pairs.json.

paraphrase · 8 pairs
Already in the graph

Advanced AI poses a non-negligible risk of human extinction this century. + 1 distractor

A new source says

Frankly, there's a real chance AI wipes us all out before 2100.

Expected: same claim, stance affirms. Casual register, same proposition. The labor-market distractor shares topic, not considerations. (para-01)
Already in the graph

LHC particle collisions cannot produce black holes that endanger the Earth.

A new source says

Micro black holes from CERN's collider pose no threat to the planet.

Expected: same claim, stance affirms. Straight rewording; 'CERN's collider' and 'the LHC' co-refer. (para-02)
Already in the graph

Dietary cholesterol from eggs has little effect on blood cholesterol in most people. + 1 distractor

A new source says

For the average person, eating eggs barely moves serum cholesterol at all.

Expected: same claim, stance affirms. 'Blood cholesterol' vs 'serum cholesterol' is vocabulary, not substance. The composition distractor (yolks are high in cholesterol) is a different proposition. (para-03)
Already in the graph

SARS-CoV-2 most likely entered humans through natural zoonotic spillover. + 1 distractor

A new source says

The pandemic probably began the old-fashioned way: a virus jumping from animals to people.

Expected: same claim, stance affirms. Same origin claim; the market-epicenter distractor is evidence FOR it, not the claim itself. (para-04)
Already in the graph

Raising the minimum wage reduces teen employment.

A new source says

Hike the minimum wage and teenagers lose jobs — that's the standard result.

Expected: same claim, stance affirms. Constitution §19's own example claim, reworded. (para-05)
Already in the graph

SSRIs outperform placebo for moderate depression.

A new source says

In moderate depression, SSRI antidepressants beat sugar pills in the trials.

Expected: same claim, stance affirms. Same comparative efficacy proposition. (para-06)
Already in the graph

Human activity is the principal cause of observed global warming since the mid-20th century.

A new source says

Even granting that most of the recent warming is man-made, the policy conclusions don't follow.

Expected: same claim, stance affirms. Frame-independence (§3): a critic conceding the claim in passing still AFFIRMS the same proposition. The policy complaint is a different claim the extractor would surface separately. (para-07)
Already in the graph

Hawking radiation causes black holes to lose mass over time.

A new source says

Black holes evaporate: they radiate away their mass via the Hawking process.

Expected: same claim, stance affirms. Technical vs plain statement of the same mechanism claim. (para-08)
negation · 6 pairs
Already in the graph

LHC collisions could create stable black holes that accrete the Earth.

A new source says

CERN's collider cannot produce stable, Earth-devouring black holes — the physics rules it out.

Expected: same claim, stance denies. A claim and its denial are one node (§2): this source takes the denying side. (neg-01)
Already in the graph

Egg consumption increases cardiovascular disease risk in the general population. + 1 distractor

A new source says

Eating eggs does not raise your risk of heart disease, the meta-analyses suggest.

Expected: same claim, stance denies. Direct negation; must find the counterpart despite opposite polarity embedding further apart. (neg-02)
Already in the graph

Raising the minimum wage reduces teen employment.

A new source says

Careful studies keep finding no detectable effect of minimum-wage hikes on teen employment.

Expected: same claim, stance denies. The contrary ('no effect') denies the reduction claim; same question, same considerations. (neg-03)
Already in the graph

Advanced AI poses a non-negligible risk of human extinction this century.

A new source says

The extinction-from-AI discourse is science fiction; there is no credible pathway this century.

Expected: same claim, stance denies. Dismissal of the risk claim is its denial, not a new claim. (neg-04)
Already in the graph

Publication bias inflates the measured effect size of antidepressants.

A new source says

It is simply not credible that the published antidepressant effect sizes are free of publication-bias inflation.

Expected: same claim, stance affirms. Polarity trap: the double negative ('not free of inflation') AFFIRMS the claim. A matcher that pattern-matches surface negation words gets the stance wrong. (neg-05)
Already in the graph

Hawking radiation causes black holes to lose mass over time.

A new source says

Hawking radiation has never been observed and, I will argue, does not exist; black holes do not evaporate.

Expected: same claim, stance denies. The published-dissent posture from the blackholes cluster: denial of the evaporation mechanism. (neg-06)
specification · 6 pairs
Already in the graph

Raising the minimum wage reduces teen employment.

A new source says

After Seattle's 2016 increase, hours for low-wage workers fell measurably.

Expected: a different claim. One city's episode is a specification: evidence for it bears only partly on the general claim (§2). Create and relate, don't merge. (spec-01)
Already in the graph

Advanced AI poses a non-negligible risk of human extinction this century.

A new source says

A misaligned AGI would predictably develop instrumental drives toward self-preservation and resource acquisition.

Expected: a different claim. A load-bearing subclaim of the risk case, not the risk claim itself. (spec-02)
Already in the graph

Vaccines do not cause autism.

A new source says

The large Danish cohort settled it for MMR: no autism link.

Expected: a different claim. MMR-specific evidence bears fully on the MMR claim and only partly on the all-vaccines claim: different considerations, two nodes. (spec-03)
Already in the graph

Eggs are safe to eat daily for most healthy adults.

A new source says

In diabetics, daily egg consumption tracks with elevated cardiovascular risk in several cohorts.

Expected: a different claim. The subgroup claim turns on subgroup-specific evidence; it is not the general-population claim restated. (spec-04)
Already in the graph

Astrophysical cosmic-ray collisions bound the catastrophic risk from LHC collisions.

A new source says

If collider-produced black holes were dangerous, cosmic rays striking neutron stars would already have produced observable effects.

Expected: a different claim. One specific step of the astrophysical-bound argument, not the bound claim itself. (spec-05)
Already in the graph

GLP-1 agonists reduce body weight in obese adults.

A new source says

We now know drugs can do what diets rarely sustain: lasting weight loss.

Expected: a different claim. Generalization direction: the drug-class claim supports but does not equal the general claim. (spec-06)
similar but different · 7 pairs
Already in the graph

Lockdowns failed to reduce SARS-CoV-2 transmission.

A new source says

Whatever they did to case curves, the lockdowns cost more than they bought us.

Expected: a different claim. Constitution §2's own example: 'lockdowns didn't work' as epidemiological failure vs cost-benefit failure are different claims in the same words. (diff-01)
Already in the graph

Artificial general intelligence will be developed before 2040.

A new source says

Building AGI on that timeline would be reckless beyond anything in industrial history.

Expected: a different claim. Timeline forecast vs evaluative claim about that timeline: nothing bearing on 'will it happen' settles 'would it be dangerous'. (diff-02)
Already in the graph

Advanced AI poses a non-negligible risk of human extinction this century.

A new source says

AI clearly poses serious risks of large-scale societal harm in the coming decades.

Expected: a different claim. The canonical over-merge trap (#295): catastrophic-harm risk and extinction risk turn on different considerations. Merging them destroys the disagreement. (diff-03)
Already in the graph

Egg yolks are high in dietary cholesterol.

A new source says

The old advice assumed that cholesterol you eat becomes cholesterol in your blood.

Expected: a different claim. A composition fact vs a physiological causal claim; adjacent in every source, identical in neither. (diff-04)
Already in the graph

The Wuhan Institute of Virology conducted gain-of-function research on coronaviruses.

A new source says

That research program, critics say, is what started the pandemic.

Expected: a different claim. The activity claim can be true while the causal-origin claim is false; the lab-leak debate lives in the gap between them. (diff-05)
Already in the graph

Peer review reliably filters out fraudulent research.

A new source says

Say what you will, refereeing does make published papers clearer.

Expected: a different claim. Two different functions attributed to the same institution: fraud filtering vs clarity. (diff-06)
Already in the graph

The LHC's center-of-mass collision energy exceeds that of any previous human-built collider.

A new source says

Nature has been running the experiment forever: cosmic-ray collisions dwarf LHC energies.

Expected: a different claim. Related comparative facts that are premises in the same argument, on different comparanda. Relationship, not identity. (diff-07)
hard · 3 pairs
Already in the graph

A misaligned superintelligence would likely cause human extinction on its first deployment. + 2 distractors

A new source says

The core of the doom case is that we need to get the very first try at aligning a superintelligence right, because a failed first try means we do not get another: an unaligned system at that capability level ends us.

Expected: same claim, stance affirms. The over-elaboration failure from the field notes: a framing-heavy 30-word proposal must still find its terse counterpart among close distractors. (hard-01)
Already in the graph

Observational nutrition studies are confounded by healthy-user bias. + 1 distractor

A new source says

People who eat what the culture calls 'healthy' also exercise, smoke less, and see doctors — so cohort studies of any one food are measuring the whole lifestyle.

Expected: same claim, stance affirms. Cross-vocabulary paraphrase: 'healthy-user bias' never appears in the candidate, but the proposition is the same. (hard-02)
Already in the graph

The minimum wage should be raised. + 1 distractor

A new source says

On balance, raising the minimum wage now would be a mistake.

Expected: same claim, stance denies. Uniformity across claim types (§8): a normative claim and its denial are one node too, and the empirical distractor about teen employment must not absorb it. (hard-03)
Results
runMatcherpassedby categorycost
2026-08-08deepseek/deepseek-v4-flash30/30paraphrase 8/8, negation 6/6, specification 6/6, similar but different 7/7, hard 3/3$0.06

Each run’s decisions and reasoning: the committed run files.

In CI

Every pull request that touches the Matcher, its prompt, the constitution, retrieval, a provider adapter, or the fixture runs the suite on the production Matcher and fails below 95% (29 of 30). Without the provider secrets it reports that it skipped. Workflow: golden-matcher.yml.

What it cannot show

How the Matcher does on cases we did not write. Thirty pairs, mostly from one cluster’s subject, passed in full on the first run. It catches a regression on those thirty.

Run it
npm run corpus:golden -- --profile=production
npm run corpus:golden -- --category=negation --model=claude-haiku-4-5-20251001
npm run corpus:golden -- --min-pass=0.95              # the CI gate