The Allen Institute for AI (Ai2) announced ToyBlocks on November 6, 2026, giving robotics researchers a controlled way to investigate a practical question: did a robot choose the wrong action, or fail to carry out the right one? For teams assessing automation, the distinction helps identify what to investigate next.
The public framework simulates an arm manipulating tabletop blocks, generates training demonstrations and supports tests of trained models. It offers a way to diagnose problems in a deliberately limited setting, not proof that a robot can handle an arbitrary workplace task.
Separate choosing a step from carrying it out
A planner chooses the next step and the block to act on; a controller turns that instruction into arm movements. In the project’s example system, Molmo 2 handles planning and ManiFlow controls movement.
ToyBlocks lets researchers replace either the planner or the controller with a reference component, then compare the outcomes. Supplying a reference plan helps examine movement; supplying a reference executor helps examine planning. Starting conditions and scoring need to remain comparable.
A successful swap points to a component worth investigating, but it is not an automatic verdict on the cause. The evaluation metrics explicitly record failures of the reference executor, including infeasible motions and unsuccessful motion plans. Researchers need to inspect those failures before attributing a problem to the learned planner.
A successful movement is not a completed task
Ai2 reports that its combined Molmo 2–ManiFlow system completed 15 of 30 physical three-block spelling trials, compared with 111 of 200 simulated trials. The physical scenes contained 4–10 blocks familiar from training. An operator stopping unsafe movement counted as a failed episode.
Ai2 separately reports 81.1% success for individual physical skills. That counts skill executions, not complete spelling tasks, and cannot identify the share of task failures caused by planning. The institute says more trials are needed to establish statistical confidence across tasks.
The practical lesson for an automation buyer is to ask for evidence covering the whole job. Thirty trials with familiar blocks do not establish reliability with unfamiliar objects in a warehouse or home.
A score needs a deadline and a finish line
In a separate evaluation of ManiFlow with reference plans, Ai2’s model card reports 26–28 successes out of 128 simulated episodes using a 30-second limit. Across four runs with a longer allowance—described as 194 seconds for this test, scaled to expert demonstrations—it reports 69–76 episodes reaching the goal. These results test the movement model, not the combined system from the physical trials.
The implication is that being able to finish and being able to finish in time are different questions. A generous allowance can help investigate capability; an operational deadline tests whether that capability is useful for the intended job. Neither score is complete without its time budget.
The finish line matters too. ToyBlocks distinguishes reaching the goal at any point from still satisfying it when the episode ends. For an object-arranging task, briefly getting the arrangement right is not the same as leaving it intact. A useful comparison should state which outcome it counts, alongside collision and stopping rules.
Unseen examples can still leave out hard cases
The released dataset’s documentation says it retains successful demonstrations and removes failed attempts. Difficult starting arrangements are therefore underrepresented, including in validation data used to assess performance. It also says the demonstrations contain no recovery from mistakes.
That creates a distinction for anyone reading a benchmark: examples withheld from training can still be a selected set of easier-to-solve situations. They do not automatically represent every situation a robot will encounter.
ToyBlocks can capture failure states and generate new expert demonstrations and recovery behaviors from them. That is a tool for designing further experiments, not evidence that recovery examples are already present in the released dataset.
What teams can use it for now
For a robotics team, the immediate use is a bounded comparison: hold the task and starting conditions steady, replace one component, and examine completion, timeouts and failure records. The evaluation guide warns that individual trajectories can vary between repeated runs even with fixed seeds. Repeated aggregate results are more informative than one favorable replay.
Access also requires resources. The setup guide specifies Linux, an NVIDIA GPU and a driver supporting CUDA 12.8. Public code does not remove compute, storage or integration costs; the documentation does not establish an all-in cost per experiment.
Teams choosing a different kind of virtual experiment can use the related guide to NVIDIA Cosmos for virtual robot trials, which explains the choice between predicting video from actions and varying a simulator’s imagery.
ToyBlocks answers a narrower development question: which part of a robot’s behavior deserves closer investigation? Using that answer to choose the next experiment is justified. A deployment decision still needs whole-task evidence under the objects, deadlines and physical conditions that deployment will involve.