Gemini Robotics 2 Makes the Persistent Session the Robot’s Control Plane
Google has put a persistent agent session above the robot.
With Gemini Robotics ER 2, that session can watch a camera feed, listen to a person, keep a multi-step goal in context, call robot tools, wait for the result, judge progress, and choose what happens next. Beneath it sit vision-language-action models such as Gemini Robotics 2 and On-Device 2—or any navigation API, third-party VLA, or human tele-operator a developer exposes as a tool.
That architecture is the important part of Google DeepMind’s July 30 launch. The polished humanoid clips show what the stack can attempt. The developer contract shows what Google wants to own: the high-level layer where intent becomes plans, plans become tool calls, and several machines share task state.
Access makes the strategy clearer. ER 2 is available through the Gemini API and Google AI Studio, while the Robotics 2 and On-Device 2 action models remain limited to partners or trusted testers. Google has opened the brain-shaped control plane before opening the motor policies.
The numbers also resist an easy victory lap. Google reports 57.4% accuracy for progress classification. Pickup from the floor reaches 45.7%. Four of five displayed five-finger tasks land at 44% success or lower. These are meaningful research gains, but they are not production reliability.
The useful builder conclusion is precise: ER 2 is a promising semantic supervisor and orchestration runtime, not a motor controller or functional-safety system. A deployable physical agent is the model plus a durable session, robot adapters, telemetry, deterministic completion checks, privacy controls, and an independent stop path.
Google’s family has three distinct layers.
Gemini Robotics 2 is the flagship vision-language-action model. It consumes observations and produces robot actions, spanning full humanoids, bi-arm platforms, and different hands or grippers. Google says one checkpoint was evaluated on Apptronik Apollo 2 robots with two hand configurations and on a Franka Duo with Robotiq grippers.
Gemini Robotics ER 2 is the embodied-reasoning model. It is based on Gemini 3.5 Flash, accepts interleaved text, image, video, and audio, and returns text. It can reason about space and time, use tools, inspect task progress, and orchestrate one or more robots.
Gemini Robotics On-Device 2 is the local action model. Its model card says it combines Gemini Robotics 1.5 technology with on-device Gemma models, accepts text, images, and proprioception, and outputs numerical actions.
The availability split matters as much as the technical split. ER 2 is public in preview. Robotics 2 is for early-access partners. On-Device 2 is for trusted testers.
That gives developers an open door into Google’s orchestration layer and a closed door around its scarce action data, embodiment tuning, and motor-policy implementation. A team can integrate ER 2 with its own executor today, then become more dependent on the session, tool schemas, traces, and operational conventions around Google’s API—even if it never receives Robotics 2.
The architecture is easiest to understand when the safety boundary is drawn first.
The learned stack can propose, monitor, and recover. The deterministic safety rail retains unconditional authority over motion.
ER 2 receives human instructions and multimodal observations. It maintains task state, generates a plan, and selects declared tools. A tool can wrap a VLA, a waypoint navigator, a gripper routine, a text-to-speech service, or a message to another robot. The application executes the call, returns the result, and gives ER 2 another observation.
Google’s open robotics samples make the missing middle visible. The repository includes an agent server, Boston Dynamics Spot integrations, and a Tinybot controller. It provides WebSockets, FastAPI services, tool dispatch, camera polling, and embodiment adapters. It does not provide ER 2 weights or a universal actuator contract.
That distinction echoes RohitAI’s earlier MiniCPM-Robot deployment audit: a checkpoint is one link in the loop. Builders still own timestamps, units, coordinate frames, controller behavior, command freshness, retries, authority, and the conditions under which motion must stop.
Google describes the ER 2 streaming endpoint as suited to latency-sensitive physical agents. The phrase is reasonable at the planning layer, but it needs a systems translation.
The official robotics streaming documentation specifies a stateful WebSocket session. Input can include raw 16-bit, 16 kHz PCM audio, text, and JPEG images at no more than one frame per second. Output is text. Camera frames alone do not trigger a reasoning turn; the application must pair them with text or audio. Google’s proactive-monitoring example sends the latest image followed by a heartbeat instruction once per second.
Physical actions have a second constraint: they must be declared as blocking functions. The model calls a tool, the application runs it, and ER 2 waits for a tool response before choosing the next action.
This produces an unusual but sensible split:
camera and audio intake asynchronous
semantic observation periodic, prompt-triggered
physical action selection serialized
joint and force control external
emergency stop independent
ER 2 can keep receiving observations while an action runs, but the public contract is not a high-frequency parallel motion bus. It is a stateful planner wrapped around serialized side effects.
That is why RohitAI’s “three clocks” framing for NVIDIA Cosmos 3 Edge applies here too. Semantic reasoning, policy inference, and deterministic control operate on different timing budgets. “Streaming” does not collapse them into one loop.
ER 2’s most interesting gains are temporal. A robot needs to know whether a grasp held, whether liquid is still pouring, whether a step finished, and whether it should retry. Static object recognition does not answer those questions.
Google reports that ER 2 classifies task progress into five 20-percentage-point bands with 57.4% accuracy. It finds a target moment with 91.3% success accuracy and 0.96 seconds mean absolute distance. In orchestration tests, it scores 60.0% with a real VLA, 42.9% with a simulated VLA, and 74.0% with a human tele-operator.
All of these figures are Google-reported. The launch material does not expose enough sample counts, confidence intervals, prompt settings, hardware details, or complete task distributions to reproduce the comparisons. Independent reporting corroborates the launch, not the benchmark results.
The 57.4% number deserves special attention. It is a step forward against Google’s displayed baselines, yet it also implies a 42.6% error rate on the evaluated five-bin classification. A system that advances to the next hazardous step whenever the model says “80–100% complete” will eventually advance too early.
The production pattern should be sensor fusion:
ER 2 progress estimate
+ gripper state
+ force and torque
+ joint position
+ PLC or controller flags
+ independent vision
+ deterministic completion predicate
= evidence strong enough to advance
This is the first non-obvious consequence of ER 2: better video reasoning does not remove the state machine. It gives the state machine another observer.
Google’s action-model results are impressive in breadth and uneven in success.
| Robot and task | Google-reported success | Builder interpretation |
|---|---|---|
| Apollo 2: pickup from table | 68.4% | Broad whole-body reach, with substantial failure still present |
| Apollo 2: pickup from floor | 45.7% | The hardest displayed whole-body pickup remains below half |
| Apollo 2: pickup from shelf | 76.3% | Promising within the reported setup, not a general warehouse rate |
| Five-finger hand: screw in bulb | 36% | Fine manipulation remains fragile |
| Five-finger hand: tie trash bag | 44% | Deformable-object handling is far from routine |
| Five-finger hand: use dustpan | 32% | Tool use exposes contact and coordination difficulty |
| Franka Duo: precise insertion | 89.6% | Structured two-finger work is the strongest displayed lane |
The advance is credible: one model checkpoint spans more bodies, whole-body motion, and finer manipulation. The unsupported leap would be to translate that breadth into “human-level dexterity.” Four of the five displayed five-finger tasks are at 44% or lower; the outlier is unscrewing a light bulb at 92%.
That pattern points to a second non-obvious insight. The value of Robotics 2 may initially be coverage, not autonomy: fewer separate policies for each body and task, faster setup for new work, and a broader recovery vocabulary. Production teams will still need constrained task definitions, fixtures, retries, and human takeover.
Google says On-Device 2 can adapt to a new bi-arm embodiment with fewer than 200 examples and a few hours of data. Its own model card narrows the evaluated scope to standing bi-arm manipulation and warns about out-of-distribution behavior and high-degree-of-freedom control. Fast adaptation lowers the entrance fee. It does not cancel the embodiment tax.
Google reports 97.9% safety-instruction-following accuracy and 93.0% accuracy for detecting whether a human is within one meter. It also introduced ASIMOV-Agentic to evaluate whether a reasoner can reject unsafe work, screen infeasible VLA calls, resolve uncertainty, and ask a person for help.
Those capabilities matter. Traditional interlocks know that a person crossed a light curtain; they do not know that “place the hot pan on the plastic tray” is a bad plan. A semantic supervisor can catch intent-level hazards before a motion command exists.
But Google’s own ER 2 model card tells developers not to use the robotics models in safety-critical applications, including healthcare, transportation, or settings where failure could cause injury or property damage. Google’s safety report also says certified hardware, redundancy, and real-time guarantees remain outside its evaluation.
There is no contradiction. Semantic safety and functional safety solve different problems.
Interpret intent, notice ambiguity, reason about constraints, monitor progress, ask for clarification, and recommend a pause or recovery.
Convert a bounded subtask into embodiment-specific motion, then return explicit status and telemetry to the session.
Enforce joint, speed, force, balance, workspace, proximity, stale-command, interlock, and emergency-stop limits without model permission.
The deployment rule is simple: ER 2 may call robot_stop(). A separate safety controller must be able to stop the robot even if ER 2 never calls it.
Google shows ER 2 coordinating robots through shared semantic context. One machine can report task state, another can receive a message, and the reasoner can hand off subtasks. That is useful, especially when different bodies have different reach, mobility, and grippers.
The public docs do not disclose a fleet-control protocol. They describe shared session state and developer-provided tools. The remaining problems are familiar to anyone who has run distributed infrastructure:
A model can reason about these questions. It cannot make the transport and controller semantics correct by thinking harder.
This is the third non-obvious insight: multi-robot intelligence will be constrained by coordination plumbing before it is constrained by conversational ability. If Google wants ER 2 to become a fleet brain, richer identity, lease, concurrency, and recovery primitives are likely to follow.
ER 2 ships through two public model IDs: gemini-robotics-er-2-preview and gemini-robotics-er-2-streaming-preview.
The standard endpoint supports caching, code execution, computer use, file search, function calling, Maps and Search grounding, structured outputs, thinking, URL context, and batch processing. It does not support the Live API.
The streaming endpoint supports the Live API, function calling, Search grounding, and thinking. It gives up caching, code execution, computer use, file search, structured output, Maps, URL context, and batch processing.
| Decision | Standard ER 2 | Streaming ER 2 |
|---|---|---|
| Best fit | Rich analysis, structured plans, offline video, tool-heavy workflows | Persistent feedback, live camera/audio, latency-sensitive orchestration |
| Session shape | Interactions API requests | Stateful WebSocket for task lifetime |
| Physical tools | Function calling | Blocking function calls only |
| Useful extras | Caching, structured output, code execution, file search, batch | Live API and continuous multimodal intake |
| Main operational risk | Stale world state between discrete requests | Session drops, heartbeat design, serialized tools, narrower capabilities |
The endpoint decision should follow the task. Do not choose streaming because a robot moves. Choose it when persistent feedback materially improves the workflow and your application can recover safely from a broken session.
Google says ER 1.6 will shut down at the end of August 2026. Replacing the model ID is the easy part.
According to Gemini API pricing, paid ER 2 input costs $2 per million tokens and output costs $10 per million, including thinking tokens. Batch is $1 input and $5 output. Streaming is $2 input and $10 output. ER 1.6 charged $1 for non-audio input and $5 for output.
Input and output prices therefore double for the common migration path. At the same time, continuous observation, heartbeat prompts, retries, multi-robot messages, and longer-lived sessions can increase usage.
Per-token arithmetic will understate physical-agent cost. The useful unit is:
reasoning and media tokens
+ heartbeat traffic
+ retries and recovery
+ VLA inference
+ edge compute
+ robot depreciation and energy
+ operator supervision
+ failed-task cleanup
= cost per safely completed task
Free-tier content may be used to improve Google’s products, while paid-tier content is not, according to the pricing page. The robotics privacy notice also requires notice and consent before identifiable people’s voice, image, or likeness is sent to Google, plus reasonable collection minimization such as face blurring where practical.
For robots in homes, shops, warehouses, and offices, camera governance is part of the product architecture. It cannot be postponed until after the demo works.
Start with one bounded workflow on the exact embodiment you intend to operate. Freeze the camera placement, controller firmware, tool schemas, model endpoint, prompt, safety limits, and network conditions. Then test the complete loop.
I would also demand missing deployment details from any On-Device 2 pilot: supported accelerator, model size, precision, memory, sustained action latency, power and thermal envelope, adaptation recipe, rollback path, and the exact safety controller used in evaluation.
The model family is in preview. The goal of an evaluation is not to prove that a robot can complete a chore once. It is to discover which failures remain bounded when the model, camera, network, tool, or human behaves differently from the happy path.
Three developments look likely.
First, embodied reasoners will become executor-neutral control planes. ER 2 already treats a real VLA, a simulated VLA, and human tele-operation as interchangeable tool categories. The high-level model will compete on temporal understanding, recovery, communication, and orchestration rather than direct joint control.
Second, Google will probably widen API access faster than VLA access. Public reasoners create integrations and application dependence; gated action models protect scarce physical data, partner relationships, and liability-sensitive deployment knowledge.
Third, robotics benchmarks will move from isolated success clips toward operational measures: progress calibration, time to detect failure, sensor-to-stop tail latency, intervention rate, recovery success, false-negative proximity events, session-loss behavior, and cost per completed task.
The harder prediction concerns multi-robot work. Google will need stronger concurrency or coordination primitives if shared sessions move beyond demos. Blocking-only tool calls are a reasonable launch contract, but parallel robots sharing space need explicit ownership and recovery semantics somewhere. If the API does not provide them, every serious customer will build a small fleet scheduler beside it.
Developers can use Gemini Robotics ER 2 in preview through the Gemini API and Google AI Studio. The flagship Gemini Robotics 2 VLA is limited to early-access partners, and On-Device 2 is available to trusted testers. Public access currently centers on reasoning and orchestration, not the Google action-model weights.
It is a high-level embodied-reasoning model. It plans, monitors, and calls robot tools. Low-level motion, actuator conversion, force control, balance, collision avoidance, and emergency stopping remain external responsibilities.
The public robotics Live API documentation specifies JPEG input at no more than one frame per second. Frames alone do not trigger a model response; the application must send text or audio, such as a periodic heartbeat prompt.
No. The scores measure semantic capabilities such as following safety instructions and detecting human proximity in Google’s evaluations. Google’s model card prohibits safety-critical use, and deployers still need certified hardware and deterministic safety controls appropriate to the robot and environment.
They should begin now because Google plans to shut ER 1.6 down at the end of August 2026. Treat the move as a new-model qualification: ER 2 changes capabilities, live-session options, behavior, and price. Re-run the complete physical-agent evaluation before production use.
Gemini Robotics 2 advances several fronts at once: whole-body control, hand dexterity, temporal video understanding, semantic safety, live orchestration, and collaboration between different machines.
Its largest near-term effect may happen one layer above motion. ER 2 gives developers a public, stateful interface where a robot’s goal, observations, tools, progress, and collaborators meet. That interface can sit above many executors, including ones Google does not own.
The benchmarks say the stack is improving. They also say it still needs guardrails, constrained workflows, sensor fusion, and human recovery. Floor pickup below 50% and progress classification at 57.4% are not embarrassing details to hide; they are the evidence that tells builders where the system belongs.
Google is making the robot session programmable. The teams that deploy it well will treat that session as a control-plane resource—with identity, authority, telemetry, budgets, privacy rules, and a clean way to end.
The machine can reason about when to stop. The system still needs a stop that does not depend on reasoning.