Minerval

← evals

EvalIs the pipeline stable?

Same graph twice

Build the graph twice with nothing changed (the noise floor), with the documents reordered, flooded with duplicates, joined by an unrelated document, or reassessed in place. None of it should matter.

six properties built; idempotency and path independence run once on the cheap tier, arms losttwo drains, or one drain plus a perturbation

Two arms

Two arms: the same cluster built twice, arm B with same, or shuffled order; both saved as snapshots and compared by the graph-agreement metric.a clusterarm Athe reference runarm Bsame, or shuffled ordersnapshot Asnapshot Bagreementclaims · verdicts · edges

Idempotency: the same setup twice. The disagreement between the arms is the pipeline’s own noise, and the floor every other comparison is read against.

Path independence: arm B reads the documents in a seeded random order. The The agent that decides whether a claim is already in the graph, possibly worded differently or as its denial.agent → is stateful, since the first wording it sees becomes the claim, so order can change the graph; §2 and §3 say it should not.

Adversarial order is the attack on path independence: arm B reads the most partisan document first (by its role in the manifest, or --role), and the summary reports the first-mover effect: the mean signed credence move of matched claims toward the first document’s stance.

Duplicate flood: arm B is arm A’s graph plus the same documents submitted again three times under distinct URLs. Instance count is provenance, not evidence weight, so the reading is inflation: the mean signed credence move toward what the sources said, plus the claims only B has (duplicates the Matcher failed to absorb).

Locality: arm B is arm A’s graph plus one document from another cluster. The reading is churn: the share of A’s claims whose status changed or credence moved by 0.1 or more.

Fixpoint: arm B is arm A’s graph with every stewarded claim re-enqueued on the staleness trigger and drained. The reading says whether stewardship settles: verdicts moved, claims and edges added or removed.

Every property also reports granularity stability: the share of matched claims with the same number of children and the same depth in both arms.

Arms are child runs, saved as A saved copy of a run's database, kept so two runs can be compared later., and compared by graph agreement, whose report now carries a per-claim block (children, depth, verdicts, instances with source and stance) for every matched pair. The tier-two arms restore A’s snapshot and perturb it, so they cost one drain plus the perturbation. One pair of arms is one sample.

Detail

How an arm is built, step by step
  1. Arm A is a child corpus:run with the flags given, snapshotted when it drains.
  2. Arm B is a second child run: for idempotency the same flags; for path independence --order=shuffle:<seed>; for adversarial order --order=adversarial or --order=role:<role>; for the tier-two properties the runner first restores A’s snapshot and runs with --no-reset plus --dup-suffix, --foreign=<cluster>:<post>, or --reassess-all.
  3. Agreement between the two snapshots: claims matched one to one by text, then by embedding, the ambiguous band judged when --confirm is given.
  4. Summary in plain language with the property’s own reading (noise floor, first-mover effect, inflation, churn, settling) and each arm’s exact cost, registered as kind property, with a two-arm replay of both arms.
What it cannot show
  • Paraphrase and frame invariance (the same proposition reworded, or stated by a proponent and a critic) have no arm yet; the golden pairs cover the Matcher half of paraphrase.
  • A matched duplicate never re-triggers the Steward, so a duplicate flood can inflate credence only where duplicates minted new nodes or edges; the instrument shows Matcher leakage reliably and inflation only where the pipeline re-triggers.
  • Which document is most partisan is a keyword heuristic over the manifest roles unless --role says.
  • A capped fixpoint drain reassesses only the highest-priority claims.
Run it
npm run corpus:property -- idempotency blackholes --profile=production
npm run corpus:property -- path-independence blackholes --seed=3 --baseline=<snapshot>
npm run corpus:property -- adversarial-order lableak
npm run corpus:property -- dup-flood blackholes --dups=3 --baseline=<snapshot>
npm run corpus:property -- locality eggs --foreign=blackholes:cern-lhc-safety --baseline=<snapshot>
npm run corpus:property -- fixpoint eggs --baseline=<snapshot>