The probability of error in a safety argument bounds the confidence it can provide about low-probability risks
7 events · 3 assessments · 3 decisions
Reassessed
Trigger: subclaim_change. The supporting premise "complex risk analyses historically exhibit error rates exceeding their claimed risk bounds" received its first assessment (supported, 0.75, credence 0.8). Materiality judgment: the change confirms what the prior verdict here had already assumed (the premise was described as consistent with well-documented evidence), so the status stays SUPPORTED; confidence raised modestly 0.75 -> 0.78 because the empirical leg is now formally assessed rather than assumed. Credence unchanged at 0.85. The subclaim steward's caveats (salient cases not a systematic sample; regulatory conservatism) qualify universality but not the use made here, where the claim only needs the cap to bind at extreme low-probability targets. The contested "requires" premise is unchanged and remains the reason the verdict is supported rather than verified. Also recorded the previously missing evaluation of the named argument "The argument-error bound" (holds_with_caveats, load-bearing on the contested premise in its scope-restricted form), and confirmed importance at 0.5 with contestation 0.45. No new searches: prior pass searched for rebuttals and found none, and this change gave no reason to expect the external picture moved. Not notifying the sole dependent (the LHC risk-analysis claim): status and credence are unchanged, so nothing material could propagate.
Reassessed: still Supported
verdict confidence 0.75 → 0.78 · credence 0.85
Reassessed
Trigger: subclaim_change. The load-bearing "requires" premise ("a flawed safety argument provides little evidence about the risk it assesses") received its first assessment: CONTESTED (0.7). Judged materiality: the contestation is real but concentrated on the premise's general form (minor flaws leave arguments approximately correct; definitional disputes over flaw severity). This claim is scoped to low-probability risks, precisely where the premise's steward noted it is strongest: approximate correctness cannot certify targets orders of magnitude below historical severe-flaw rates, and conditional on a flaw, the severe-flaw tail dominates residual risk at extreme confidence levels. So the change is absorbed without a status flip: status stays SUPPORTED, confidence lowered 0.8→0.75, credence 0.9→0.85, and both assessment texts rewritten to surface the contested premise and the scope reason it does not overturn the bound. One adversarial web search found no published rebuttal of the bound itself; discourse applies it rather than disputing it. No structural change: minting a narrower "extreme-probability" variant of the contested premise would create a near-duplicate; the scope reasoning lives in the assessment prose instead. Canonical form left as is (accurate, neutral, scoped). Importance left at 0.5: a methodological claim with real consequence for how safety cases are read (feeds the contested LHC risk claim) but with a narrow dependent base; matches the notable-to-major range. Status unchanged, so dependents were not notified: the only dependent (the LHC risk-analysis claim) sees the same SUPPORTED verdict it already integrated, and the modest credence adjustment is not material at its end.
Reassessed: still Supported
verdict confidence 0.80 → 0.75 · credence 0.85
Structured and assessed first pass
First-pass stewardship. Identified the claim as the central thesis of Ord, Hillerbrand & Sandberg (2010), confirmed via web search. Decomposed into three subclaims, all confirmed novel by match_claim before creation: (1) REQUIRES "A flawed safety argument provides little evidence about the risk it assesses" (importance 0.4, the genuinely contestable premise); (2) SUPPORTS "Complex risk analyses historically exhibit error rates exceeding their claimed risk bounds" (0.35, empirical grounding); (3) SPECIFIES "Multiple independent safety arguments can jointly provide confidence exceeding any single argument's error bound" (0.25, scope qualifier endorsed by the source authors). Deliberately did NOT decompose the law-of-total-probability step: settled mathematics, uncontested, a leaf per the contestedness stop rule. Grouped (1) and (2) under a named "for" argument ("The argument-error bound") with a written form. Left canonical form unchanged: it is short, neutral, and a fair statement both sides would accept. Revised importance from the 0.45 extractor seed to 0.5: peer-reviewed principle actively invoked in LHC-risk and AI safety-case debates, with a contested dependent (the LHC risk-analysis claim) already in the graph. Assessed SUPPORTED (confidence 0.8, credence 0.9) rather than VERIFIED because the subclaims are not yet independently assessed and the key premise is an approximation of debatable strength. Notifying the dependent steward (LHC claim) since this establishes a first material assessment.
Assessed Supported
verdict confidence 0.80 · credence 0.90
This claim states a methodological principle about how much assurance any single safety analysis can deliver: when an argument concludes that some catastrophe has a very small probability, the trustworthiness of that conclusion cannot exceed the probability that the argument itself is sound. Its best-known statement is by Ord, Hillerbrand and Sandberg (Journal of Risk Research, 2010), who derive it from elementary probability: the overall chance of catastrophe is at least the chance that the argument is flawed multiplied by the chance of catastrophe if it is flawed. If a flawed argument leaves the risk estimate near its prior value, then an analysis with, say, a one-in-a-thousand chance of containing a significant error cannot by itself establish that a risk is below one in a million. The formal step is uncontroversial, and the supporting premises are well grounded. Empirical records of retracted papers, software verification limits, and failed probabilistic risk assessments (such as pre-Challenger estimates of Space Shuttle failure rates) indicate that complex technical arguments err at rates far above the tiny risk thresholds safety cases often claim to meet. The main point of resistance concerns degree rather than principle: arguments with minor flaws often remain approximately correct, so how sharply a given error probability limits confidence depends on how uninformative a flawed argument really is. The principle also has a recognized limit, acknowledged by its own proponents: it caps the confidence available from any one argument, not from all evidence combined. Several genuinely independent lines of analysis, or successful replication and testable side-predictions, can push the joint probability that every argument is flawed well below any single argument's error rate. Read with that qualification, the claim is broadly accepted in the risk-assessment literature and has no substantive published rebuttal of its core reasoning.
Claim entered the graph