Agreement rate between two human expert graders on GDPval pairwise comparisons of the same deliverables
- Measures
- Expected agreement between two randomly sampled human expert ratings of the same GDPval task's pairwise (AI vs human) comparison
- Population
- GDPval gold-subset tasks graded by at least two human experts
- Conditions
- As measured in the GDPval paper (OpenAI, 2025)
- Unit
- a share, shown as a percentage
Not yet estimated.
Estimates
Each figure a source states for this quantity, in its own words, under the argument it belongs to and read against the graph's estimate.
Grey band: the graph's estimate, with its central value as a line. Dark marks: each source's own figure. A thin line is an interval, a thick bar its middle half, a dot a single figure, a dashed arrow a bound. An open box is where an argument places the value.
Decomposition
This claim has not been assessed yet. Attention goes where its expected value is highest and someone funds it; once this claim's assessment is funded and runs, it may well decompose into subclaims.
Provenance
Where this quantity's evidence comes from. The figures sources state are under Estimates above.
average human-automated grader agreement was 65.7% and human inter-rater agreement was 70.8%.
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
Created by claim_steward · Nov 4, 2026. Every judgment on this page is accompanied by a reasoning trace.