Google's Robot Brain Gets Eyes and a Team
Gemini Robotics ER 2 gives robots real-time video awareness, multi-step planning, and the ability to coordinate across machines.
By The ai-compute Desk
Google DeepMind has released Gemini Robotics ER 2, a model designed to act as a high-level brain for robots — letting them plan multi-step tasks, monitor their own progress through live video feeds, and hand off work to other robots when a single machine isn't enough. [1][2]
The model is now publicly available to developers via the Gemini API and Google AI Studio, and in private preview on Gemini Enterprise Agent Platform. [1] Developers can declare low-level control interfaces — like Vision-Language-Action (VLA) models or navigation APIs — as tools, and stream multimodal video, audio, or text directly into the model. [1]
One of robotics' hardest challenges is knowing when a task is done. [1] ER 2 attacks this with two new capabilities: progress classification and moment finding. For progress classification, each frame in a video feed is assigned to one of five completion brackets (0–20%, 20–40%, 40–60%, 60–80%, 80–100%), giving the robot continuous situational awareness without restarting an entire workflow if something goes wrong. ER 2 scores 57.4% accuracy on progress classification tasks, outperforming previous-generation models and competing frontier models. [1]
Moment finding — pinpointing the exact video frame when a critical event occurs, like knowing when to stop pouring coffee — is where ER 2 pulls further ahead. The model achieves 91.3% accuracy and a 0.96-second mean absolute distance on moment-finding tasks, competing closely with much larger model categories but delivering this precision at a fraction of the compute cost and 4x the execution speed. [1] That sub-second latency matters: a robot that pauses to think mid-pour is a robot that makes a mess.
To eliminate the jarring stop-and-think pauses common in earlier systems, ER 2 integrates with the Gemini Live API via a bidirectional streaming endpoint optimized for latency-sensitive tasks. [1] In a demo built with Boston Dynamics, ER 2 orchestrates Spot's navigation and manipulator APIs to fetch objects on a natural-language command — the robot doesn't freeze between instructions. [1]
[1] ER 2's multi-robot collaboration feature lets diverse machines communicate via a shared semantic understanding, handing off work to complete complex workflows a single robot could not do alone. [1] Google has demonstrated this with Apptronik's Apollo 2 and the Franka F3 Duo operating together under ER 2's orchestration. [1]
The shift is architectural as much as it is technical. Separating high-level reasoning from low-level motor execution means the brain can be swapped, upgraded, or retrained without touching the limbs — a division of labor that makes the whole system easier to evolve.
Safety has been folded into the benchmarking from the start. ER 2 outperforms its predecessor and other frontier models on both Safety Instruction Following and Human Proximity benchmarks, which evaluate adherence to physical constraints and spatial awareness around people. [1] Google reports the model successfully halts a humanoid robot when a person is nearby and autonomously resumes work only once the area is clear. [1]
Spatial reasoning has also been extended beyond the basics. Success and failure detection now runs on raw video feeds rather than static snapshots, catching mid-execution failures like spills or misalignments as they happen. [1] Instrument reading has expanded from circular dials to digital displays, linear scales, rulers, and liquid thermometers — tested across 10 different instrument types. [1]
Gemini Robotics ER 2 doesn't replace the robot's hands — it gives the hands a brain that can actually watch what they're doing. Whether that compounds into genuinely useful physical agents or remains a compelling benchmark story depends on how developers wire it into the messier realities of warehouses, kitchens, and factory floors. The API is open; the hard part starts now. [1]