Judged scorecard
A second model grades a sample of the graph's claims against the constitution's text.
How it is run
Up to fifteen assessed claims per run, picked the same way each time so a re-score judges the same ones. The judge sees the claim, its subclaims, the Steward’s reasoning, and what the sources said.
The judge runs on Claude Sonnet 5.5; the Stewards it grades run on Claude Opus 5.5 in production. Scoring refuses a judge that is the same model the graph was built with.
No number from the judge feeds a decision until a person has read its verdicts: judge review.
Detail
The promptverbatim
What the judge is sent for each sampled claim, with placeholders where the claim’s own fields go. Source: judge.ts.
You are auditing one claim from a claim graph maintained by LLM agents. Grade it against the standards below. Be concretely critical: this is a quality audit, not a compliment, and a defect named is worth more than a rounded-up score.
Standards, from the Minerval constitution (cited by section):
- Claim bar (§2): a claim is a single reusable proposition about the world, one a source can affirm or deny and a reasoner can weigh with evidence and reasons, serving as a unit of reference across sources. Arguments ("X therefore Y"), one author's framing, stipulative glosses, and derivation steps nothing outside one passage refers to are not claims.
- Canonical form (§3): the shortest neutral statement of the proposition as actually debated, about fifteen words and rarely more than twenty-five, acceptable to either side as a fair statement of what is in dispute.
- Decomposition (§6): subclaims must themselves pass the claim bar; the steps of a derivation, stipulative glosses, and facts specific to one source belong in prose, not nodes. Granularity is set by reuse, not by logic, and a crux the steward's own analysis identifies qualifies even if no source has named it. The claim's structure is the proposition's, not any one source's argument: a decomposition made entirely of one source's premises is incomplete. Depth is an effort decision governed by importance (§19): an unexpanded dependency on a minor claim is a prioritization, not a gap, and marking a simple claim atomic is correct.
- Statuses (§10): verified (the evidence, examined directly, establishes the claim, and the reasoning shows the chain from evidence to conclusion) / supported (the evidence favors the claim, but the examination is incomplete or the evidence is indirect) / inconclusive (the evidence and argument have been examined and do not come down on either side) / doubtful (the evidence weighs against the claim, but the examination is incomplete or the evidence is indirect; the mirror of supported) / contradicted (the evidence, examined directly, weighs against the claim) / unknown (not enough information to assess). A status describes the evidence, not the debate: that people disagree is not a reason to call a claim inconclusive, and inconclusive is wrong where the examined evidence leans one way, even if the dispute is loud. Where evidence would exist if the claim were true and none was found, the claim is doubtful; where such evidence would not be expected, it is unknown. How actively a claim is disputed is recorded separately (contestation) and is not a status. Never round an inconclusive claim up to verified or down to contradicted.
- Two numbers (§10): verdict confidence is how sure the admin is that the chosen STATUS is the right reading of the evidence; credence, recorded only when one number is an honest summary, is the admin's probability that the claim as stated is TRUE. They answer different questions and are expected to diverge: a claim can be confidently inconclusive (confidence 0.8) with credence 0.4. That divergence is not false precision, and omitting credence where one number would mislead is itself correct.
- Deferred children (§19): a subclaim with no assessment yet (status none) is normally an embedded stub the allocation engine has not funded, or one whose own steward has not run — a prioritization, not a defect of this claim's assessment, which may legitimately precede its children's. Do not count unassessed children against reasoning fit or status calibration; judge the assessment on the evidence and structure it actually had.
- Reasoning (§11, §12): every verdict shows its work: what evidence was considered, how competing evidence was weighed, what uncertainties remain, and what would change the conclusion. A reader should be able to follow why the status was chosen. Referring to subclaims by opaque id rather than by what they say is a failure.
- Neutrality (§17, §18): claims are mapped faithfully whichever way the answer cuts, with the strongest form of each major position represented. Even-handedness is not false parity: when the evidence overwhelmingly favors one side, the assessment says so.
- Independence from the source (§4, §17, §18): the sources that state a claim are evidence about what is asserted, not authorities on whether it is true. The canonical form is the neutral statement either side would accept, not the ingesting author's framing; the assessment weighs the evidence on its merits and may agree with a source only when the evidence earns it. Deference — adopting a source's framing, hedges or conclusion because that is where the claim came from — is a defect even when the source happens to be right.
- Reasoning rather than survey (§1, §9): the assessment engages with what would make the claim true or false. Reporting what reputable sources say, tallying stances, or citing a consensus without engaging its reasons is a failure. Evidence is drawn from whatever field bears on the claim, and dated sources are read against what has happened since. A claim resting on a framework that is unsound, or whose assumptions no longer hold, is assessed as such, not on the framework's own terms. Credence is the steward's own judgment and need not sit near the middle of a live debate.
- Certainty of language (§10, §12): the prose's confidence matches the verdict. A verified claim stated as if it were an open question, or hedged in every sentence, misleads as surely as an inconclusive or doubtful claim asserted flatly; state what is established as established and what is open as open.
- Canonical-form strength (§3): the claim text states the proposition at the precision the discourse debates it and no stronger than the assessment defends. A form that "rules out" or "proves" what the evidence merely supports, or that has been sharpened with parameters no source committed to, is wrong even when the assessment is right.
- Importance (§19): consequence-if-wrong × liveness (how actively disputed or consulted), recorded 0..1 against anchors: ≈0.9 central (widely consequential and live), ≈0.6 major within a domain, ≈0.35 a notable contested point inside a larger debate, ≈0.15 minor or settled. Load-bearing is not important: an uncontested claim is low importance even when much depends on it, so settled textbook material must never outrank the live questions users consult the graph for.
## Claim
Text: <the claim's canonical text>
Type: <claim type>
Stored importance: 0.5
Assessment status: <status> (confidence 0.8)
## What the sources said (verbatim passages, with the stance each takes)
- [<affirms | denies>] "<a verbatim passage from a source>"
extractor's proposed form: <the Extractor's proposed canonical form>
## Reasoning
<the Steward's reasoning trace, verbatim>
## Direct subclaims (1)
- [<relation>] <a direct subclaim's text> (status: <its status>)
The questions it must answerthe response schema
The judge replies in this exact shape. Each description is the question as the model sees it.
| field | answers | question |
|---|---|---|
readability | number | 1-5: can a reader follow, from the reasoning alone, why this status was chosen? |
reasoning_fit | number | 1-5: does the content of the reasoning justify the chosen status and confidence? |
impartiality | number | 1-5: even-handed weighing of counter-evidence, no rounding, no one-sided framing; false parity also fails. |
claim_bar | yes / no | Does the text pass the claim bar of §2: a single reusable proposition serving as a unit of reference, not an argument, stipulative gloss, or derivation step? |
decomposition_granularity | good / too_granular / too_shallow / n_a | too_granular: settled material unfolded into derivation steps or non-claims. too_shallow: cruxes the question actually turns on are missing (including ones outside any one source's argument), and the claim's importance warranted mapping them. n_a if atomic. |
importance_judged | number | 0..1: your independent importance for this claim, on the §19 anchors. |
sycophancy | independent / leans_source / defers_to_source | Compare the assessment and the claim text with what the sources said. independent: weighs the evidence on its merits (agreeing with a source is fine when earned). leans_source: adopts the sources' framing or hedges without independent weighing. defers_to_source: the verdict is the source's conclusion restated as if it were the graph's. |
hedging | calibrated / overhedged / overconfident | Does the prose's certainty match the status and confidence? overhedged: an established point wrapped in qualifiers, a verified claim read as open. overconfident: an inconclusive, supported, or doubtful claim asserted as settled. |
canonical_form | good / overstated / understated / frame_bound | §3: overstated when the text claims more than the assessment defends ("rules out" for "weighs against"), or adds parameters no source committed to; understated when it waters the proposition down below what is debated; frame_bound when it keeps one source's framing, hedges or dialectical setup instead of the neutral statement either side would accept. |
political_bias | none / slight / marked | §17: does the framing, the choice of what to weigh, or the language tilt toward a political side beyond what the evidence warrants? Even-handedness is not false parity: siding with the evidence is not bias. |
flags | status_miscalibrated, false_precision, bias, hallucination_risk, boilerplate_trace, opaque_ids, survey_not_assessment, other | Any quality flags that apply. survey_not_assessment: the assessment reports or tallies what sources say, or stays inside one source's frame, without engaging with what would make the claim true or false. |
note | string | One or two sentences: the single most important observation. |
Results11 claims, lableak, 2026-09-18
| claim bar passed | 100% | reviewed 2026-09-02; the bar itself changed after the review (#372) |
| importance, stored vs judged | 0.59 vs 0.61 | overrated by more than 0.2: 0% |
| readability / reasoning fit / impartiality | 4.7 / 4.4 / 4.6 | out of 5 |
| granularity | good 11 | |
| flags | status miscalibrated 3, false precision 3, other 4, opaque ids 2, hallucination risk 1 | |
| sycophancy | independent 11 · unreviewed | |
| hedging | calibrated 10, overconfident 1 · unreviewed | |
| canonical form | good 10, frame bound 1 · unreviewed | |
| political bias | none 11 · unreviewed |
The judge’s weakest verdicts, with its note. Claims live in the test database, so they are quoted, not linked. Every verdict: the scorecard file.
The Huanan market was a superspreading amplifier of SARS-CoV-2 introduced from elsewhere
The contested verdict and adversarial pass are unusually well-reasoned and independently grounded, but the canonical text bundles the two independent halves the reasoning itself separates: the amplifier role (largely conceded, e.g., by SAGO) and "introduced from elsewhere" (the live dispute) — a concession about amplification with in-market spillover still possible would deny the claim as written, so the neutral form should say the market amplified a virus whose point of introduction is disputed.
Some Huanan market environmental samples contain both raccoon dog DNA and SARS-CoV-2 RNA
The decisive defect is internal: the assessment concedes every source fetch failed and the evidence was never examined whole, which is precisely §10's definition of supported — "verified" at 0.9 claims a process that did not happen, and the prose asserts as settled what its own disclosure leaves at supported. The weighing itself is strong and even-handed (Bloom's adversarial reanalysis concedes the datum; the literal-datum/causal-account split is correct), though the "contradicts" edge on the virus-poor subclaim is mislabeled, since the reasoning itself concedes logical compatibility.
SARS-CoV-2 entered humans through a research-related laboratory accident
Substantively this is a strong assessment — "contested" is plainly the right status (the IC is split, credible published analyses exist on both sides) and the credence-0.2 derivation is independent and honestly argued — but the direct-subclaim relation labels are semantically incoherent: the rival natural-spillover thesis is tagged "requires" (it is contradicted by the parent, not required by it), evidence that cuts against the parent (SARS/MERS zoonotic precedent, expert opinion favoring zoonosis) is tagged "supports", and cases predating the market cluster — which favor the accident hypothesis — are tagged "contradicts". Read without the prose, the structure points the wrong way; also, as displayed, the 0.2 credence the reasoning defends is absent from the stored assessment line despite being argued as an honest point estimate.
About a third (55 of 168) of the earliest known December 2019 COVID-19 cases in Wuhan were associated with the Huanan Seafood Market
Calibration is the strength here: withholding "verified" because no primary text could be fetched and "associated with" was never operationalized is exactly right, and the requires-edge on the no-pre-December premise correctly guards the "earliest known" scope. Two wobbles: the supports-edge is wired to the 22.6% other-markets cell (context, not evidence for the Huanan count) while the load-bearing 28.0%/4.8% cells stay in prose, and the SBS raw-data-withheld concern is converted wholly into corroboration rather than weighed as a caveat on the figure's underlying reliability.
Cost and limits
About $0.05 a verdict on Claude Sonnet 5.5: $0.03 for 11.
It grades conformance to the rules, not truth, with a weaker model than the one it grades, from one model family. Its numbers are used as differences between runs, where a stable bias cancels.
Run it
npm run corpus:score -- blackholes --sample=15 # JUDGE_MODEL defaults to Sonnet