Article

How IBMARL helps an AI team learn beyond imitation

For curious readers and robotics researchers, a simulated gamepad study shows how AI agents can learn from imperfect lessons and improve.

Editorial illustration for How IBMARL helps an AI team learn beyond imitation: a controlled task flows from input to output. Not documentary evidence.

One researcher, one Xbox gamepad, two simulated teammates: the lessons did not have to be perfect. In a preprint submitted on November 13, 2026, a University of Utah team reports that its IBMARL method helped the pair improve beyond the behaviour learned by copying. It kept imitation available for guidance while a predefined task score directed further learning.

The task was Buzzwire: two simulated agents connected to a ball had to guide it through a narrow corridor without touching the walls. One author steered a single teammate at a time. Only 13 of the 24 demonstration episodes reached the goal—yet the resulting lessons still provided a useful starting point.

For researchers training cooperative systems, the question is what to do after collecting those examples. The result suggests that demonstrations can help a team get started without setting its permanent ceiling. The crucial ingredient is a separate way to judge improvement, not copying alone.

How one person teaches a team

Trying to steer every member of a team simultaneously would give the human several jobs at once. The earlier Round-Robin Behavior Cloning method, or R2BC, divides the work: teach one teammate while the other follows its current learned behaviour, then switch. The imitation policies are updated between rounds.

A policy is simply a learned rule for choosing an action. Here, imitation learning means learning to copy demonstrated movements. Because the other teammate is already moving, the person demonstrates how to cooperate with the partner actually present—not how to perform a solo task in an empty arena.

IBMARL builds on that teaching method and earlier imitation-assisted reinforcement learning; it does not invent the principle of improving beyond examples. Its arXiv record also acknowledges an earlier workshop version, so November 13 dates this preprint submission, not the first appearance of the idea.

Why copying does not have to set the limit

Reinforcement learning means learning from scored outcomes. In this study, the researchers supply the reward in advance. Demonstrations help the team discover useful behaviour; they do not define the goal or teach the system what people value.

IBMARL keeps the imitation policy available to suggest actions during reward-guided training. Each agent gets two proposals: one from its imitation policy and another from the policy being improved. The system considers combinations across the team, so one agent can follow imitation while its partner tries its newer proposal.

The paper’s action-selection mechanism uses learned value estimators, called critics, to rate those combinations. During training, it samples among them according to their estimated value. These critics are predictive models, not human reviewers—and their estimates are not guarantees of success.

The learning rule also encourages imitation where the critics rate the team’s imitation proposal more highly. That encouragement is selective, rather than a permanent instruction to reproduce every recorded action. After training, the evaluated controllers act separately from their own observations, without the critics or imitation policies choosing their moves.

This makes teaching and improvement different jobs. The examples supply an initial route through the task; rewards provide a reason to try another. For researchers designing such experiments, the related guide to auditing reinforcement-learning rewards explains why a higher score is useful only when the scoring rule measures the intended task.

What the gamepad experiment actually showed

In the authors’ human-demonstration experiment, IBMARL exceeded the imitation-only baseline after a mean of about 123,000 simulator steps. The comparisons took about 286,000 steps for RLfD, which reuses recorded demonstrations, and 296,000 for RFT, which encourages matching demonstrated actions on a preset schedule.

Those averages cover five training runs with different random seeds, not five human participants. All five IBMARL runs crossed the baseline. One RLfD run never did within the 600,000-step budget; the repository’s calculation rules count a noncrossing run at that limit. The thresholds come from smoothed evaluation scores, not an exact instant when learning suddenly happened.

The important comparison is with the learned copy, not the person holding the gamepad. The result does not show that IBMARL beat its human teacher, that failed demonstrations are preferable, or that 24 lessons will suffice for another task.

Nor did the method win everywhere. In the paper’s separate four-task study using synthetic demonstrators, the replay-based comparison finished slightly higher on Navigation. The authors also found that mixing proposals across agents mattered particularly on Transport, but was neutral on Buzzwire. The gamepad result cannot be credited to that ingredient alone.

Fewer interactions did not mean a shorter full run

There is a revealing wrinkle in the authors’ runtime table. Full Buzzwire runs took roughly 2.0–2.1 hours for IBMARL variants, versus 1.6 hours for RLfD and 1.7 hours for RFT. Those runs used a 600,000-step budget and shared a GPU with other concurrent runs.

That does not contradict reaching a score threshold in fewer interactions. It answers a different question: how long the full training budget took. The figures do not establish elapsed time to the same quality, a lower electricity bill or a lower total cost.

A practical comparison should therefore record both interactions and elapsed time to a chosen score, alongside final performance and variation between runs. Fewer attempts may matter when collecting experience is expensive; extra computation still has to be counted.

What researchers can use—and still need to check

The public implementation supplies recorded results and frozen imitation policies. Its data instructions distinguish redrawing the supplied curves from rerunning demonstration-based training, which needs a separately hosted archive. New demonstrations use the R2BC collection setup. These are research materials, not a ready-made robot product; accessible code alone is not independent replication.

All IBMARL experiments reported here are simulated, with teams of two or three agents. The human evidence comes from one author on one task. Scoring every combination of imitation and learned proposals also becomes harder as teams grow. The paper establishes neither physical-robot reliability nor a general solution for large teams.

For a fresh experiment, an essential evaluation question is whether the final controller works when its training-time helpers are removed. For a different diagnostic problem, ToyBlocks separates robot planning from movement. And the study of AI-written neural game players versus ordinary code asks whether a neural controller is the most useful representation at all. Neither is evidence that IBMARL works outside its tested tasks.

For a new cooperative-learning experiment, the useful choice is whether demonstrations should remain active advisers during training, rather than only stored examples to replay. Compare those approaches on the same task and reward, then judge the standalone controllers. The gamepad supplies the lessons; it need not supply the limit.