EvalIs the pipeline stable?
Graph agreement
How different are two graphs built from the same documents? The measurement the next three evals use.
Three numbers
- Claims: precision, recall and F1 over a one-to-one matching of the two graphs’ claims. Unmatched claims are attributed to the agent that created them.
- Verdicts: on matched claims, how far apart the credencesThe Steward's probability that a claim is true.constitution → are, and how often the status agrees.
- Edges: mapped through the matching, edge precision and recall, and the edit distance between the two decompositionsBreaking a claim into the claims it depends on: what would have to be true for it to be true.constitution →.
Code, pure and unit tested: graph-agreement.ts.
Detail
How claims are matchedand the pair judge's prompt
Exact text first, then stored embeddings above 0.85 cosine, greedily one-to-one. Pairs between 0.85 and 0.95 are the ambiguous band: kept and reported, or, with --confirm, sent to a judge on Claude Sonnet 5 with this prompt:
Two claim graphs, built independently, each contain a claim. Decide whether they are the SAME proposition — a claim is individuated by what bears on it (constitution §2): two formulations are the same claim when nothing could count as evidence or argument bearing on one without bearing equally on the other. Identical decomposition is a diagnostic, not the definition. A claim and its denial are one node. A specification, a generalization, or a claim that turns on different considerations is a different claim. Claim 1: <a claim from graph A> Claim 2: <a claim from graph B>
It answers same_proposition (true/false) and a one-line reason.
What it cannot show
Which graph is better. And the matching is itself a matcher, with a threshold chosen by hand and never checked against the golden pairs: two wordings of one claim can read as disagreement.
Run it
npm run corpus:snapshot -- save run1 # after one run
npm run corpus:run -- blackholes # run again
npm run corpus:agreement -- snap:run1 db --confirm