RoboValue: A Benchmark for the Models That Tell Robots How Well a Task Is Going

A new arXiv paper, RoboValue, asks a question that sits underneath most robot-learning pipelines: how good are the models that tell a robot whether it is succeeding? These "value models" watch a robot's camera view and an instruction and estimate task progress, and their scores are used to train policies and to monitor execution. The Hugging Face paper page summarizes the authors' argument that existing evaluations only cover parts of this skill, so the reliability of the feedback is under-tested. RoboValue is their attempt at a fine-grained test across simulation and real hardware, and the code and data are public.

This is a research paper from a group we could not match to a top lab (the page does not state affiliations), so we cover it because it ships usable artifacts. The abstract gives no headline scores, so this post explains the design and the qualitative findings rather than a leaderboard.

TL;DR: RoboValue at a glance

What is a value model and why test it?

In reinforcement learning, a value function estimates how good a situation is: how much reward to expect from here. In robot manipulation, that idea becomes a learned model that looks at a video frame and the task instruction and answers: how far along is this task, and is it going well or badly? A robot folding a shirt, for example, should have a value that rises as the folds are completed and drops if the shirt slips.

Those estimates matter for two things. First, training: instead of a hand-coded reward, a value model can supervise a policy. Second, monitoring: a deployed robot can use it to notice it is stuck and retry or ask for help. If the value model is wrong, both fail quietly. A model that says "great progress" while the robot knocks the cup over teaches the wrong behavior.

That is the gap RoboValue targets. Existing tests, the authors say, check isolated abilities, so a model can look strong while hiding specific weaknesses.

The four capabilities

The benchmark evaluates models along four axes:

1. Task-state understanding: Can the model recognize what stage of the task the scene is in?
2. Temporal progress monitoring: Does its estimate move sensibly over time as the task advances?
3. Failure and recovery reasoning: Does it notice a mistake and recognize when the robot recovers?
4. Value consistency: Are estimates coherent across equivalent situations and phrasing?

It uses diagnostic trajectories, which are recorded executions designed to probe a specific weakness, and instruction variations, which change how the task is described to see whether the model truly grounds language. It tests zero-shot and one-shot settings and separates a change of robot body (embodiment shift) from a change of surroundings (environment shift), so a failure can be attributed to the right cause.

What it found

The paper page reports qualitative findings:

• Models make errors in fine-grained instruction grounding: they can miss which object or step the instruction refers to.
• They struggle with history-dependent progress tracking: the right value depends on what already happened, not just the current frame.
• They are weak at recovery assessment, judging whether a failed attempt has been repaired.
• This holds even for models with strong outcome and forward-progress scores, which means a single headline number can hide these gaps.
• The same semantic subtask can receive substantially different value gains across valid execution orders. If two valid orders of the same steps are scored differently, the supervision signal is noisy in a way that can mislead a policy.

The authors say the results point to specific targets for value supervision. We cannot report which models were tested or how they scored, because the page does not include scores; the project page and repository are the places to check, and we could not retrieve details from the project page.

Why the sim-and-real design matters

Robot benchmarks often live in simulation because it is cheap, then fail to transfer. RoboValue pairs 15 simulation tasks with 20 real-world manipulation tasks, so researchers can see whether a model's strengths survive contact with real cameras, lighting and objects. Splitting embodiment from environment shifts is the other useful choice: a model that fails when the arm changes but not when the room changes has a different problem from one that fails the other way.

For context on how other labs frame robot evaluation, see our coverage of the Artificial Analysis robotics benchmark, the LIBERO-PERM personalization benchmark, and the GPT-6 Astra robot-control benchmark. RoboValue differs in that it evaluates the judge, not the robot.

How to use it

Per the paper page, the artifacts are:

• The arXiv paper.
• A project page at robovalue-benchmark.github.io.
• Code in the RoboValue-Benchmark GitHub organization.
• A training dataset on Hugging Face, with an author comment saying training subsets are in LeRobot v3.0 format, which makes them loadable with the LeRobot tooling. That comment mentions "Full-Shot and One-Shot" subsets while the abstract describes zero-shot and one-shot evaluation, so the exact split names may differ; read the dataset card.

A practical way to use it: if you train or fine-tune a value model, run it through RoboValue and look at the per-capability breakdown rather than one aggregate. If a model is strong at forward progress but weak at recovery, you know not to rely on it for retry logic.

A worked example of why the findings matter

Imagine a robot told to "put the red cup on the tray, then pour water into it." A value model watching this should rise when the cup is placed, rise again when water is poured, and fall if the cup tips over. Now consider the failure modes the authors describe.

With weak instruction grounding, the model confuses the red cup with a red bowl on the table and awards progress for the wrong object. With weak history-dependent tracking, it sees a cup on a tray and a full cup and cannot tell whether the order was right, because the current frame looks the same whether water was poured before or after placement. With weak recovery assessment, the robot knocks the cup over, rights it and refills it, and the model keeps reporting low value long after the task is back on track, or reports high value while the spill is still there.

The finding about execution order is subtler. Suppose two valid orders exist, pour first and place second, or the reverse. If the model gives one order much more credit than the other for the same subtask, a policy trained on that signal will learn to prefer one ordering for reasons unrelated to success. That bias is invisible in an aggregate success-rate number, which is exactly the argument for a fine-grained diagnostic benchmark.

What to look for when the results are examined

Because the paper page does not list scores, a careful reader should look for specific things in the full paper and repository: which model families were tested, whether general video-language models were compared against value models trained for robotics, how large the gap is between zero-shot and one-shot, and whether failures cluster in the real-world tasks more than in simulation. A large simulation-to-real gap would suggest that cheap simulated evaluation is not enough for judging value models. A small gap would be good news for researchers who cannot afford many real-robot trials. Either way, the per-capability tables are more informative than any single summary number.

Caveats

• This is a preprint. We have not run the benchmark or reproduced any result.
• The paper page lists no affiliations, and we did not independently verify the authors' institutions.
• No scores appear on the page, so claims about which model is best cannot be made from it.
• A benchmark defines what counts as good; the 35 tasks are a sample of manipulation, not all of robotics.

What this means for builders

If you work on robot learning, value models are a quiet dependency, and silent errors there are expensive. RoboValue gives a shared way to ask where a given model fails. If you are a general AI builder, the lesson transfers: any learned judge, including LLM graders and reward models, deserves a capability-by-capability test, not a single score. Our coverage of Hao AI Lab's open robot-use agents and Reward AI's OM-1 robot foundation model shows where such judges plug in.

Questions people are asking

Is this a benchmark for robots or for AI models? For models that evaluate robots. It measures the judge, not the manipulator.

Does a good score mean a robot will succeed? No. It means the value model assesses progress and failure more reliably, which should help training and monitoring.

Why LeRobot format? LeRobot is a widely used open framework for robot learning data, so a common format lowers the effort to try the benchmark.

What is still unknown

• Which value models were evaluated and their scores.
• The authors' affiliations and any robot hardware details.
• Whether independent groups can reproduce the findings.
• How the benchmark handles future, larger models.

Related reading

• Artificial Analysis robotics benchmark: claimed vs verified
• LIBERO-PERM personalized manipulation benchmark
• GPT-6 Astra robot-control benchmark
• Hao AI Lab open robot-use agents
• Reward AI OM-1 robot foundation model
• NVIDIA Cosmos3-H-Dreams generative surgical simulator

Details are from the paper page on October 20, 2026; the paper is a preprint.