Complex risk analyses historically exhibit error rates exceeding their claimed risk bounds
5 events · 2 assessments · 2 decisions
Reassessed
Trigger: subclaim_change on the contradicting subclaim (regulatory conservatism overstates risk), which received its first assessment as SUPPORTED (0.72/0.75) on the modest reading, with a domain distinction noting it concerns chemical/environmental/public-health screening rather than the engineering/financial probabilistic models behind the exceedance pattern. Judged materiality: confirmatory, not disruptive. The prior parent assessment already treated the conservatism point as credible and scope-limiting rather than pattern-contradicting. The subclaim resolving on the modest reading (rather than the stronger systematic-net-overstatement reading, which remains contested) plus the domain distinction narrows the contradicting subclaim's reach further. Status held at SUPPORTED; confidence nudged 0.75 -> 0.78 because the against-line's load-bearing premise landed weaker than a full net-overstatement finding would have; credence held at 0.80. No structural change (no missing dependency discovered). Updated assessment text and reasoning trace to fold in the now-assessed subclaim and the domain distinction; re-evaluated the 'Conservatism and selection effects' argument to reflect the modest-reading assessment and genre distinction, and re-recorded the 'Cross-domain track record' argument unchanged (its premises' standing did not change). Importance confirmed at 0.35 (notable) with contestation recorded at 0.55. No dependent notification: status unchanged and the confidence nudge is immaterial to the two dependents.
Reassessed: still Supported
verdict confidence 0.75 → 0.78 · credence 0.80
Structured and assessed first pass
First pass (structure_and_assess). Decomposition: created two named arguments. For: cross-domain track record (three new subclaims, all confirmed novel by the Matcher: pre-Challenger Shuttle estimates vs realized rate, importance 0.2 deferred stub; nuclear PRA core-damage frequency vs observed accident rate, 0.3; pre-2008 VaR understatement, 0.2 deferred stub). Against: conservatism and selection effects (one new subclaim, confirmed novel: regulatory assessments commonly use conservative assumptions that overstate risk, 0.35, contradicts). The selection-effect point and the peer-review error-rate calibration stay in prose (argument written form and reasoning trace) as they are steps/evidence rather than claims the discourse disputes as such, per the claim bar. Importance confirmed at 0.35 (notable: a supporting empirical premise for the Ord/Hillerbrand/Sandberg-style thesis with two dependents), contestation 0.45. Evidence pass: three web searches (Ord et al. paper; Shuttle estimates including NASA retrospective 1-in-9 for early flights; nuclear PRA predicted vs actuarial frequencies; conservatism literature). Verdict: SUPPORTED, confidence 0.75, credence 0.8, marginal yield 0.3 (a systematic calibration study of risk-analysis track records, e.g. the forecasting/cost-overrun literature, was not digested this pass and could sharpen the verdict). Canonical form kept: already terse, neutral, frame-independent. Both argument evaluations recorded (each holds with caveats; the dispute reduces to universal vs pattern reading). Notifying both dependents since this is the claim's first recorded assessment and both lean on it as support.
Assessed Supported
verdict confidence 0.75 · credence 0.80
The claim generalizes from a documented track record: in several independent, heavily analyzed domains, formal risk analyses have quoted failure probabilities that realized events then exceeded, sometimes by orders of magnitude. Before the Challenger accident, NASA management figures put Space Shuttle loss risk near 1 in 100,000 per flight; the program lost 2 of 135 flights, and NASA's own retrospective analysis put early-flight risk near 1 in 9. Nuclear probabilistic risk assessments predicted core-damage frequencies around one in a million reactor-years, while the actuarial rate implied by Three Mile Island, Chernobyl, and Fukushima is orders of magnitude higher. Bank value-at-risk models before 2008 assigned near-impossibility to loss sequences that then occurred repeatedly. A related calibration point supports the pattern from below: peer-reviewed technical work is retracted or found materially flawed at rates around one in ten thousand to one in a hundred, so any analysis claiming a risk bound of one in a billion is claiming a reliability its own error rate cannot support. The credible reservation is about generality rather than the cases themselves. Many regulatory and environmental risk assessments deliberately build in conservative assumptions and overstate risk, and the famous failures are salient precisely because they failed; no systematic sample of risk analyses and their outcomes exists. The claim is therefore well supported as a recurring historical pattern, and close to unavoidable for analyses claiming extremely small bounds, but it is not established as a universal property of complex risk analysis. A systematic calibration study across a defined population of risk analyses would sharpen or qualify the verdict.
Claim entered the graph