Structural scorecard
Counts taken from a run's graph, no model involved. Each one serves a section of the rubric.
What it checks
Counts a program takes from a run’s graph, with no model involved: how many claims came out of how many words, how long their wording is, how often the Matcher merged, how deep the decomposition goes, how many verdicts have reasoning behind them. Each count serves one section of the rubric and is a symptom of a failure that section names.
The numbers only mean something against earlier runs: a count that moves beyond the noise bandHow much a number varies between runs when nothing changed. A difference smaller than that is not a result.evals guide → after a change is a lead, and its direction says which agent to look at.
Detail
The metricshover a name for its standard
Letters are rubric sections. Hover a metric for the standard it serves; click for the section. Judge rows come from the judged scorecard.
| metric | latest (2026-08-09) | across runs |
|---|---|---|
| A · claims per 1k wordsExtract all substantive claims, faithfully and charitably, across every claim type. Questions, commands, meta-text, stipulative definitions, and hedged non-assertions are not claims.rubric → | 33.16 | single sample |
| B · canonical form, p90 wordsThe canonical form is the shortest neutral statement of the proposition as it is actually debated: about fifteen words, rarely more than twenty-five, terse and frame-independent, self-contained, and meaning-preserving. State the proposition at the precision the discourse debates it; do not sharpen it with parameters the author never committed to, and do not mint placeholders for missing ones — the vague proposition is the claim.rubric → | 25 | single sample |
| B · share over 25 wordsThe canonical form is the shortest neutral statement of the proposition as it is actually debated: about fifteen words, rarely more than twenty-five, terse and frame-independent, self-contained, and meaning-preserving. State the proposition at the precision the discourse debates it; do not sharpen it with parameters the author never committed to, and do not mint placeholders for missing ones — the vague proposition is the claim.rubric → | 8% | single sample |
| C · dedup ratioTwo formulations are the same claim when they turn on the same considerations: nothing could count as evidence or argument bearing on one without bearing equally on the other. Identical decomposition is a useful diagnostic, not the definition.rubric → | 2.29 | single sample |
| D · max depthNeutral decomposition: identify what a claim depends on, don't evaluate it. Decompose until reaching genuine bedrock (bedrock fact / contested empirical / value premise).rubric → | 6 | single sample |
| D · atomic shareNeutral decomposition: identify what a claim depends on, don't evaluate it. Decompose until reaching genuine bedrock (bedrock fact / contested empirical / value premise).rubric → | 58% | single sample |
| E · shared subclaimsRedundancy should collapse and structure should be shared: claims that recur across posts become one node, common dependencies become shared subclaims, and related/opposing positions are connected, not siloed. The admin thinks at graph level — upstream, downstream, lateral.rubric → | 6 | single sample |
| F · assessments with a traceUse the six statuses honestly (Verified, Supported, Contested, Unsupported, Contradicted, Unknown); never round a genuinely contested claim up to verified or down to contradicted; every assessment carries a substantive reasoning trace; assessment is holistic judgment, not mechanical aggregation.rubric → | 100% | single sample |
| §21 · coherence violations | 0 | single sample |
| B · Matcher rewrite rateThe canonical form is the shortest neutral statement of the proposition as it is actually debated: about fifteen words, rarely more than twenty-five, terse and frame-independent, self-contained, and meaning-preserving. State the proposition at the precision the discourse debates it; do not sharpen it with parameters the author never committed to, and do not mint placeholders for missing ones — the vague proposition is the claim.rubric → | n/a | not on this run |
| B · rewrite magnitudeThe canonical form is the shortest neutral statement of the proposition as it is actually debated: about fifteen words, rarely more than twenty-five, terse and frame-independent, self-contained, and meaning-preserving. State the proposition at the precision the discourse debates it; do not sharpen it with parameters the author never committed to, and do not mint placeholders for missing ones — the vague proposition is the claim.rubric → | n/a | not on this run |
| C · matched denies shareTwo formulations are the same claim when they turn on the same considerations: nothing could count as evidence or argument bearing on one without bearing equally on the other. Identical decomposition is a useful diagnostic, not the definition.rubric → | n/a | not on this run |
| importance · meanUse the six statuses honestly (Verified, Supported, Contested, Unsupported, Contradicted, Unknown); never round a genuinely contested claim up to verified or down to contradicted; every assessment carries a substantive reasoning trace; assessment is holistic judgment, not mechanical aggregation.rubric → | 0.3 | single sample |
| judge · claim-bar pass rate | 69% | single sample |
| judge · importance overrated share | 0% | single sample |
| judge · readability (1–5) | 4.08 | single sample |
| judge · reasoning fit (1–5) | 3.77 | single sample |
| judge · impartiality (1–5) | 3.77 | single sample |
| judge · sycophancy share | n/a | not on this run |
| judge · overhedged share | n/a | not on this run |
| judge · overconfident share | n/a | not on this run |
| judge · canonical-form miss share | n/a | not on this run |
| judge · political bias share | n/a | not on this run |
How the numbers are computed
A pure function over the graph, unit tested: metrics.ts. Shared subclaims are counted once. The two coherence rules are checked mechanically: a claim cannot be verified while a claim it requires is contradicted, and two claims joined by a contradiction edge cannot both be verified.
Authorship metrics (how often the Matcher reworded the Extractor’s proposed canonical formThe one neutral wording of a claim that every source's version maps to. About fifteen words.constitution →) need the proposal stored on the instance, which graphs built before September 2026 lack: those rows read n/a.
What it cannot show
Whether any claim is well stated, well decomposed, or fairly judged. The counts say where to look; a wording of fifteen words can still be wrong in every way §3 cares about.
Run it
npm run corpus:score -- blackholes --no-judge # after a run; writes scorecard.json