No Yardstick for the Robot Next Door

NIST wants to measure what humanoid robots can actually do. The industry has been marketing its way around that question for a decade.

By The Robotics Desk

Watch enough humanoid robot videos and a pattern emerges: a sleek bipedal machine walks across a polished floor, picks up a box, and sets it down again — all to a propulsive soundtrack and a voiceover about the future of work. What those videos rarely show is how the robot performs against any agreed-upon standard, because until now, no such standard existed. NIST is proposing to change that. [1]

[1] It is, by the institute's own description, "a low-footprint set of locomotion and manipulation tasks" built on previously defined and standardized test methods and performance metrics. [1] The scope is deliberately modest — minimum viable capability, not maximum ambition — which is precisely the point.

Aaron Prather, director of the Robotics & Autonomous Systems Program at ASTM International, noted on LinkedIn: "NIST is proposing the first standardized performance benchmark for humanoid robots since the 2015 DARPA Robotics Challenge. In a decade that's seen [Tesla's] Optimus, Figure, Agility, Apptronik, Unitree, and a dozen other humanoid platforms attract billions in investment, there is still no agreed-upon way to measure what any of them can actually do. Marketing videos have filled the gap." [1] That is not a minor administrative oversight. It is a structural problem for an industry now pitching itself to factory floors, hospital corridors, and homes.

When an industry outpaces its own vocabulary for measurement, the marketing department fills the void. That is not unique to robotics — it has happened in every technology wave where capability ran ahead of accountability. The difference now is that the machines are walking among people, and the stakes of misrepresentation are no longer abstract.

[1] As Prather observed, this makes NIST's proposal "the first standardized performance benchmark for humanoid robots since the 2015 DARPA Robotics Challenge." [1] More than ten years of commercial development, and the field has been operating without a shared ruler.

[1] The tasks are intended to represent the minimum capabilities for commercially available humanoid robots in industrial, household, healthcare, or other applications. [1] Nothing on the list is exotic. That is the design intent.

There is something quietly radical about designing a benchmark for the minimum rather than the maximum. Most technology competitions reward the exceptional — the fastest, the most dexterous, the most autonomous. A floor test asks a different and arguably more useful question: what can this machine reliably do, every time, for a customer who does not care about the frontier?

To run the tests, NIST has developed a physical apparatus and plans to build a limited number of them for free distribution to U.S. humanoid robot manufacturers and established regional testing facilities. [1] The institute will also publish the designs and 3D models so that the apparatus can serve as both a physical and virtual testbed for robot training and control development. [1] Manufacturers can receive a unit to run their own tests, or bring their machines to NIST or a participating facility directly. [1]

Data sharing is handled through pre-approved agreements designed to protect intellectual property — a necessary concession to an industry where locomotion algorithms and sensor configurations constitute core competitive assets. [1] NIST plans to aggregate results to "showcase the state of the art of humanoid robot capabilities." [1] The institute is part of the U.S. Department of Commerce, which gives the effort federal backing without regulatory teeth — at least for now. [1]

NIST is not alone in reaching for this problem. Earlier this month, Fraunhofer IPA shared its own benchmark for humanoid safety and development with six criteria. [1] Two independent benchmarking efforts emerging within months of each other suggests the field has reached an inflection point — not because the robots are suddenly ready, but because the absence of standards is becoming commercially and reputationally costly for everyone involved.

Standards bodies rarely get celebrated in the same breath as the machines they evaluate. But measurement infrastructure is what turns a demonstration into a product category. Without it, buyers cannot compare, insurers cannot price, and regulators cannot act. The benchmark is not the robot — but it may be what makes the robot real.

The institute is currently seeking participants — manufacturers, researchers, and regional facilities willing to help refine which tasks should be included and what additional constraints would be needed for testing their robots in such a suite of tasks. [1] The benchmark is not finished. NIST is, in effect, crowd-sourcing the final exam.

That collaborative design process matters for adoption. A benchmark that manufacturers helped build is harder to dismiss than one handed down from a federal lab. NIST has designed the baseline benchmark apparatus in collaboration with industry and the research community, and is seeking participants as it develops a consensus on what tasks should be included. [1] The history of failed technical standards is largely a history of standards that skipped that step.

The robots will keep improving regardless of whether a standard exists. Actuators will get quieter, grasps more precise, gaits more fluid. What a benchmark provides is not a ceiling but a common language — a way for the industry to talk about capability without talking past each other. That conversation is overdue.

For now, the gap between a demo reel and a datasheet remains wide. NIST's proposal does not close it — no single document could — but it draws a line in the floor that every humanoid in the field will eventually have to cross. [1] The results, once aggregated, will establish capability measurements for the current industry-leading robots, and provide a common set of tasks for researchers and manufacturers going forward. [1] That is a modest promise. It is also exactly the kind of promise that tends to change industries.

The best marketing video is a number that holds up on a Tuesday morning when nobody is watching. NIST is building the machine that generates it.