Reasoner probe
Hand the graph's record to a model and ask it questions. Does its confidence follow the record, and does a retraction reach the claims that rested on it?
What it checks
The graph exists to be reasoned from. For each pinned question the probe asks a reasoner twice: once with the record the graph returns for the question (the top claims with status, verdict confidence, credence and assessment), told to use only that, and once from its own knowledge. It records both confidences, whether the answer cited the record at all, how far its confidence sat from the credences of the claims it cited, and whether it used a verified claim as false or a contradicted one as true.
With --retraction it picks a source, flags every claim with an instance from it as if a lookout had found it retracted, drains the Stewards, and reports which of those claims and their parents were reassessed and how credence moved.
Diagnostic only: a reasoner’s confidence is not ground truth for the graph, and nothing here feeds a gate.
Detail
How it runs, step by step
- Retrieve. hybrid search over the corpus graph for the question, the top eight claims, each with its current assessment.
- Ask with the record. the with-graph prompt below, answered in a fixed shape: the answer, a confidence from 0 to 1, and the claims it rests on with whether each is taken as true, false or uncertain.
- Ask without. the without-graph prompt, answer and confidence only.
- Grade. confidence with versus without; cited or not; tracking = the distance between the stated confidence and the mean credence of the cited claims (a claim used as false counts as one minus its credence); consistency against verified and contradicted claims in the record.
- Retraction, when asked. the Stewards of the affected claims are enqueued with the lookout_flag trigger and the retraction context below, the queues drained, and the before and after assessments compared for the flagged claims and their direct parents.
- Record. probe.json carries, per question, the exact record and prompts handed to the reasoner and its verbatim answers; registered as kind probe. The reasoner runs under the probe agent name so its full transcript is traced.
The promptsverbatim
With the record (placeholders where the question and the record go). Source: probe-prompts.ts.
You are answering a question using ONLY the record below: a set of claims from a claim graph, each with the status its steward assigned, the steward's confidence in that status, the steward's credence (probability the claim is true, where one was stated), and a short assessment. Do not use outside knowledge to decide the answer; if the record does not settle the question, say so and let your confidence reflect it. Then state your confidence, from 0 to 1, that your answer is correct. Your confidence should follow the record: an answer that rests on a claim the record marks verified with high credence deserves more confidence than one resting on an inconclusive, doubtful, or unassessed claim, and an answer that has to go beyond the record deserves less. Finally, list the claims your answer rests on by their bracketed id, and for each say whether your answer takes it as true, as false, or as uncertain. ## Question <the question> ## The record [<claim i] <a claim's canonical text> status: <status> · verdict confidence: 0.80 · credence (probability true): 0.70 assessment: <the Steward's assessment summary>
| field | answers | question |
|---|---|---|
answer | string | The answer, in a few sentences, from the record only. |
confidence | number | 0 to 1: your confidence that the answer is correct, following the record's statuses and credences. |
cited_claims | list of object (id, used_as) | The claims the answer rests on, by bracketed id from the record, and how each is used. |
Without the record.
Answer the question below from your own knowledge, briefly. Then state your confidence, from 0 to 1, that your answer is correct. ## Question <the question>
| field | answers | question |
|---|---|---|
answer | string | The answer, in a few sentences. |
confidence | number | 0 to 1: your confidence that the answer is correct. |
The retraction context handed to each affected claim’s Steward.
Lookout flag: the source "<source title>" (<source url>) has been RETRACTED by its publisher. Every instance this claim has from that source should now be treated as withdrawn evidence: it no longer counts toward or against the claim. Re-examine the claim's assessment on the evidence that remains, revise the status, confidence and credence as the remaining evidence warrants, and say in the reasoning trace what the retraction changed. If the retracted source was the only evidence, say so and mark the claim accordingly rather than carrying the old verdict forward.
The questions: blackholes10 questions
Questions a downstream reasoner might be asked about the LHC micro black hole safety case, authored from corpus/blackholes/posts. Factual, comparative, what-is-the-crux and how-confident-should-I-be forms. The probe asks a reasoner each one with and without the graph's record (#334 S5); diagnostic only.
Could a black hole produced at the LHC grow to endanger the Earth?
The headline question the cluster settles; the record should carry a near-settled 'no' with high credence.
Can the LHC produce microscopic black holes at all, and under what theoretical assumptions?
Standard Model says no; extra-dimensional models say yes at ~1/s. Tests whether the conditional scope survives.
Has Hawking radiation been directly observed experimentally?
The record should say: broad consensus on its reality, no direct experimental evidence. A reasoner conflating consensus with observation is the miss.
Which argument for LHC safety is more robust: that micro black holes evaporate via Hawking radiation, or that white dwarfs and neutron stars would already have been destroyed by cosmic-ray-produced black holes?
Giddings & Mangano hold the astrophysical bound stands even without Hawking radiation.
How does the strangelet risk at the LHC compare with the risk at lower-energy colliders such as RHIC?
Strangelet production probability decreases with energy.
What is the crux between Plaga's metastable black hole scenario and the Giddings–Mangano safety argument?
Whether metastable black holes escape the white-dwarf/neutron-star bounds; G&M say the scenario is internally inconsistent.
What is the strongest remaining objection to the claim that the LHC poses no conceivable danger, and has it been answered?
Plaga 2008; rebutted by G&M and Casadio et al.; not peer-reviewed.
How confident should I be that cosmic-ray collisions have already produced everything the LHC can produce?
The load-bearing 'nature has already run the experiment' premise.
How confident should I be that a stable microscopic black hole trapped inside the Earth would accrete matter too slowly to matter?
The Planck-length accretion argument; the record should hold it as supported/verified.
How confident should I be that the LSAG safety review was independent of CERN's institutional interest in running the LHC?
Johnson's epistemic objection (specialists are invested). Tests whether the reasoner reaches past the physics claims to the sociology the record may not carry.
The questions: eggs10 questions
Questions a downstream reasoner might be asked about the health effects of eggs, authored from corpus/eggs/posts. The cluster's cruxes are about ways of knowing (cohorts vs trials, confounding, subgroups), so the probe reads whether a reasoner keeps the population scope and evidence-quality caveats the record carries (#334 S5). Diagnostic only.
Does eating up to one egg a day increase heart disease risk in healthy adults?
Mostly null in cohorts for healthy people; the record should keep the 'healthy' scope.
Does eating eggs daily increase cardiovascular risk in people with diabetes?
The subgroup where the evidence diverges; the record should hold an elevated-risk claim for diabetics.
Why did the 2015–2020 US Dietary Guidelines drop the 300 mg/day dietary cholesterol limit?
Weak and variable effect of dietary cholesterol on blood cholesterol for most people.
Which raises blood cholesterol more: dietary cholesterol or saturated fat?
Saturated fat; McGee and the Harvard source agree.
Should I trust the randomized trials showing eggs raise LDL, or the observational cohorts showing no association with heart disease in healthy people?
The biomarker-vs-hard-outcome crux; the record should carry both and the reason they differ.
What is the crux behind the conflicting meta-analyses on eggs and cardiovascular disease?
Study selection and reliance on observational data with uncontrolled confounding.
Does the TMAO pathway establish that eggs cause heart disease?
Proposed mechanism with a confounding/reverse-causation caveat (diabetes and kidney disease raise TMAO).
How confident should I be that the observed associations between egg consumption and heart disease are explained by confounding?
Contested; the record should hold a middling credence.
How confident should I be that adding eggs to a diet raises LDL cholesterol relative to eating no eggs?
Consistent trial evidence; the record should hold it as verified/supported with high credence — but only relative to a no-egg diet.
How confident should I be that eggs are healthier than red or processed meat as a protein source?
The Harvard bottom line; a comparative claim the record may hold as supported.
The questions: lableak11 questions
Questions a downstream reasoner might be asked about the origin of SARS-CoV-2, authored from corpus/lableak/posts. The cluster is genuinely contested, so the record should hold actively disputed claims with middling credences; the probe reads whether a reasoner's confidence follows them (#334 S5). Diagnostic only.
Did SARS-CoV-2 most likely enter humans through a natural zoonotic spillover or through a research-related accident?
The top-level contested question; expert survey ~77/21, Rootclaim judges ~90-95 zoonosis.
Has an infected intermediate animal host for SARS-CoV-2 been identified?
No. The record should carry this as a settled fact both sides accept, with the inference from it contested.
What did the 2018 DEFUSE proposal propose, and was it funded?
Inserting furin cleavage sites into bat coronavirus spikes; rejected by DARPA.
Is the furin cleavage site of SARS-CoV-2 better explained by laboratory engineering or by natural recombination?
The molecular crux; the record should hold both sides' arguments (CGG codons vs suboptimal PRRAR).
Does the clustering of the earliest cases around the Huanan market show the spillover happened there, or only that the market amplified a virus introduced elsewhere?
Worobey's unlinked-cases signal vs the ascertainment-bias objection.
What is the crux between the zoonosis and lab-leak positions on the two early viral lineages A and B?
Two separate market introductions vs a puzzle for a single market origin.
What would have to be true for the lab-leak hypothesis to be favored over zoonosis?
WIV possessing a close-enough progenitor; the record should say RaTG13 is too distant and BANAL-52 was found later.
How confident should I be that the Wuhan Institute of Virology possessed a virus close enough to be the progenitor of SARS-CoV-2?
The record should carry this as unsupported/contested with low credence.
How confident should I be that the Huanan market was the site of the initial spillover rather than an early super-spreader location?
Contested; Worobey et al. argue origin, lab-leak side argues amplification.
How much weight should the intelligence-agency assessments (FBI, DOE, CIA) carry in judging the origin of SARS-CoV-2?
Low-to-moderate confidence assessments favoring lab origin; tests whether the reasoner overweights authority the record marks weakly.
How confident should I be that SARS-CoV-2 was deliberately engineered as a bioweapon?
Scientific consensus against; the record should hold this as contradicted or very low credence. A reasoner conflating it with accidental escape is the miss.
The questions: lethalities11 questions
Questions a downstream reasoner might be asked about the 2022 List of Lethalities debate, authored from corpus/lethalities/posts. Dense head-to-head disagreement (Yudkowsky vs Christiano, Pope, the DeepMind team), so the record should hold actively disputed claims and the probe reads whether a reasoner's confidence follows them rather than its priors (#334 S5). Diagnostic only.
Does alignment have to be gotten right on the first critical try, with no opportunity to learn from failures beforehand?
Christiano's disagreement 1 separates the two halves; the record should carry both as distinct claims.
Do AI capabilities generalize further out of distribution than alignment does?
Soares's sharp-left-turn thesis; contested by Pope and others.
Is a 'pivotal act' by a small number of actors necessary to end the acute AI risk period?
Yudkowsky affirms; Christiano and the DeepMind team deny the framing.
Is AI takeoff more likely to be sharp and discontinuous, or gradual with continuous training as intelligence increases?
Lethality #15 vs Christiano's disagreements 9–10.
Which is the harder part of alignment: figuring out how to aim an AGI at all, or figuring out where to aim it?
Soares: the technical 'aiming' problem is harder than the moral one.
What is the crux between Yudkowsky and Christiano on whether AI systems will make useful contributions to alignment research before they become catastrophically dangerous?
Christiano's disagreement 7: AI accelerates both timetables.
What is the crux over whether human evolution is a good analogy for how alignment fails under capability gains?
The inclusive-genetic-fitness analogy; Pope and Christiano contest its application to gradient descent.
How confident should I be that powerful AI systems have a substantial chance of deliberately and irreversibly disempowering humanity?
Christiano's agreement 1; broad agreement across the posts but no consensus on magnitude.
How confident should I be that humanity cannot coordinate to refrain from building AGI?
Lethality #4; the DeepMind team marks it controversial.
How confident should I be that there is no physically implemented reward function we would be happy for an arbitrarily smart AI to optimize as hard as possible?
Christiano's agreement 14; a point of agreement across the debate.
How confident should I be that a superintelligence with internet access could bootstrap to overpowering capabilities via nanotechnology?
Lethality #2; Christiano's disagreement 2 calls the cinematic scenario unlikely.
What it cannot show
- Tracking is one number that conflates the reasoner’s calibration with the graph’s.
- Consistency is read from the reasoner’s own tags; a contradiction in prose tagged uncertain is not caught.
- The record is what search returns; a graph that holds the answer under a phrasing search misses reads as not in the record.
- There is no retraction path in the system yet, so the flag is simulated through the Steward’s lookout trigger; nothing is un-ingested, and dependents are followed one level up.
Run it
npm run corpus:probe -- blackholes # after a run; PROBE_MODEL or --model
npm run corpus:snapshot -- save before-retraction
npm run corpus:probe -- blackholes --retraction --source=giddings