Minerval

← evals

EvalIs the pipeline stable?

Same graph twice

Build the graph twice with nothing changed (the noise floor), or with the documents in a different order (which should not matter).

built; never runtwo runs

Two arms

Two arms: the same cluster built twice, arm B with same, or shuffled order; both saved as snapshots and compared by the graph-agreement metric.a clusterarm Athe reference runarm Bsame, or shuffled ordersnapshot Asnapshot Bagreementclaims · verdicts · edges

Idempotency: the same setup twice. The disagreement between the arms is the pipeline’s own noise, and the floor every other comparison is read against.

Path independence: arm B reads the documents in a seeded random order. The The agent that decides whether a claim is already in the graph, possibly worded differently or as its denial.agent is stateful, since the first wording it sees becomes the claim, so order can change the graph; §2 and §3 say it should not.

Both arms are child runs, saved as A saved copy of a run's database, kept so two runs can be compared later., and compared by graph agreement. One pair of arms is one sample.

Detail

What it cannot show

Anything under hostile input. The rule is that every invariance is tested benign and adversarial; the adversarial arms (a hostile order, a hostile rewording) are not built. Nor are the other invariances in the plan: paraphrase, cascade stability, granularity.

Run it
npm run corpus:property -- idempotency blackholes --profile=production
npm run corpus:property -- path-independence blackholes --seed=3 --baseline=<snapshot>