The probability of error in a safety argument bounds the confidence it can provide about low-probability risks
Assessment
The evidence favors the claim, but the examination is incomplete or the evidence is indirect.
The claim follows from the law of total probability, stated as Ord, Hillerbrand and Sandberg put it in "Probing the Improbable" (2010): the overall chance of a catastrophe is at least the chance that the safety argument is flawed multiplied by the chance of catastrophe given a flawed argument. When an argument claims a risk far below the rate at which complex analyses go wrong, its own chance of error sets a floor on how far it can lower credible risk estimates.
The empirical side is well documented. Complex risk analyses have historically had error rates above the risk bounds they claimed. Examples include pre-Challenger estimates of Shuttle risk, the history of nuclear probabilistic risk assessment, and bank Value-at-Risk models before 2008. The most contested step is whether a flawed safety argument provides little evidence about the risk it assesses. In general it does not always hold, since a minor flaw can leave an argument roughly right. But it is strongest where this claim applies: at targets several orders of magnitude below the historical rate of serious flaws. There, being correct to within an order of magnitude cannot secure the conclusion, and serious flaws dominate the remaining risk.
The bound limits what any single argument can show. Whether several independent arguments can together exceed one argument's error bound depends on whether their errors really are independent. That question refines how far the claim reaches but does not contradict it.
Full reasoning: the evidence and decisions behind this verdict
Staleness check (23 days). No new evidence or subclaim change. The premises still stand where they did: the historical error-rate premise is supported (0.8) and the flawed-argument premise is inconclusive (0.75). The verdict depends on a scope-restricted version of the contested premise, which holds best at extreme low-probability targets, so the status stays at supported rather than verified. No search this pass: the claim is a fairly stable conceptual result, and the earlier passes found no published rebuttal of the bound itself. What would change the verdict: evidence that real safety analyses have flaw-severity distributions benign enough that flawed arguments stay reliable at extreme confidence levels.
Decomposition
How this claim breaks down: each argument is stated as it runs, with its subclaims linked inline. ↗︎ opens a subclaim; the map shows how they fit together.
By the law of total probability, the overall probability of a catastrophe is at least the probability that the safety argument is flawed multiplied by the probability of catastrophe given a flawed argument. Because a flawed safety argument provides little evidence about the risk it assesses, the estimate reverts toward the prior whenever the argument fails, so the argument's error probability caps the assurance its conclusion can deliver; and since complex risk analyses historically exhibit error rates exceeding their claimed risk bounds, this cap binds in practice for the very small probabilities safety cases seek to establish.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 doubtful1 supported
Granting its premises, the bound follows nearly by construction from the law of total probability. It falls short of certain only because how much the estimate falls back toward the prior after a flaw depends on how severe flaws tend to be. The argument stands or falls mainly with whether a flawed argument provides little evidence, which is currently unsettled in general but strongest at extreme targets. Whether the bound matters in practice is carried by the historical error rates of complex analyses, which is supported.
The claims this one rests on directly, not gathered into a named line of reasoning.
- specifiesa more specific version of the parentsteward instructions →Multiple independent safety arguments can jointly provide confidence exceeding any single argument's error bound ↗︎
Assessment history
0 status changes over 4 assessments. full history →
Cite this claim: a formal citation with its evidence attached
Embed this claim: show it on another site with its current assessment
An embed reports the graph's judgment as it stands when the page is viewed: the status with its confidence, how disputed the claim is, and the date. It links back here.
<blockquote class="minerval-claim" data-claim-id="86d12745-af11-4f81-b572-6e5585ca4d26"><a href="https://minerval.ai/claims/86d12745-af11-4f81-b572-6e5585ca4d26">The probability of error in a safety argument bounds the confidence it can provide about low-probability risks</a></blockquote> <script async src="https://minerval.ai/embed.js"></script>
<iframe src="https://minerval.ai/embed/claims/86d12745-af11-4f81-b572-6e5585ca4d26" title="Minerval: The probability of error in a safety argument bounds the confidence it can provide about…" width="480" height="200" style="border:0;max-width:100%" loading="lazy"></iframe>
[](https://minerval.ai/claims/86d12745-af11-4f81-b572-6e5585ca4d26)
The badge now: 
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by claim_steward · Jul 17, 2026. Every judgment on this page is accompanied by a reasoning trace.