LIBERO-PeRM: Robots That Finish the Task Still Ignore Your Preferences, Benchmark Finds

A robot that finishes the job is not necessarily a robot that does it the way you want. LIBERO-PeRM, a new benchmark from researchers at UC Riverside, Yale, Georgia Tech, the University of Michigan and the University of Toronto, measures that gap directly. Its headline: the strongest zero-shot vision-language-action (VLA) policy tested completes 97.6 percent of ordinary tasks yet satisfies only about a quarter of motion and skill preferences.

The paper on arXiv describes a benchmark built on LIBERO, and the project website hosts the code, datasets and demos, released under the MIT license. This post explains what the benchmark tests, what it found, and what the authors say it cannot yet tell us.

TL;DR

What counts as a "preference"?

The authors separate what a robot should do (the task goal) from how it should do it (the preference). A preference never changes the goal: putting a bottle in a basket stays the same task whether the robot moves slowly or quickly. They propose a taxonomy of language-expressible preferences:

• Motion preferences constrain how a trajectory is executed, such as speed or keeping away from an object.
• Skill preferences change how a skill is performed, such as picking an object by its handle.
• User-profile preferences guide semantic choices, such as preferring dairy products or a certain drawer.

This mirrors how people actually talk to helpers: "gently," "by the handle," "I'm lactose intolerant, so grab the oat milk." Current robot benchmarks rarely test it, because they score only whether the goal was reached.

How is the benchmark built?

A data pipeline uses a vision-language model to make minimal layout edits and generate subtask plans, then calls heuristic skills built from human demonstrations and an OMPL motion planner. The pipeline also produces automatic preference-satisfaction metrics, giving each episode a discrete label from 0, one quarter, three quarters or 1. In a 15-person survey on a 10-point scale, demonstrations scored 9.11 for human-likeness and preferences scored 8.35 for real-world plausibility.

Five suites cover different questions:

1. LIBERO-SinglePref (100 tasks): learning and executing one preference.
2. LIBERO-PrefScale (100 tasks): how learning scales with demonstrations, layouts and tasks.
3. LIBERO-PrefConflict (48 tasks): conflicts between preferences, and between a preference and the task.
4. LIBERO-PrefCompose (64 tasks): four aligned conditions per layout: no preference, preference A, preference B, and both.
5. LIBERO-ComprePref: adds no tasks but rewrites prompts so the preference must be inferred from context.

Two metrics matter. Task success rate (SR) measures completion. Preference satisfaction (SA) averages the satisfaction label over episodes and counts only successful ones, so doing the wrong thing the right way earns nothing.

What did the experiments find?

Zero-shot, current VLAs fail at personalization. Six policies were tested: OpenVLA-OFT, pi0-fast, pi0.5, GR00T-N1.6, SmolVLA and UniVLA. The best, pi0.5, kept 97.6 percent success without preferences but reached only 24.38 percent satisfaction on motion preferences, 25.00 percent on skill preferences and 19.12 percent on user-profile tasks.

Fine-tuning makes preferences learnable, with a catch. LoRA fine-tuning of pi0.5 on single preferences raised all-preference success and satisfaction to 85.0 and 82.5. Mixed training, which blends preference and no-preference data, preserved 91.00 percent no-preference success versus 52.33 percent for sequential training. On ComprePref, mixed training reached 72.83 percent SR and 70.07 percent SA, compared with 44.00 and 42.64 for preference-only LoRA. But mixed training cut motion-preference success from 90.00 to 66.88 percent, a trade-off between keeping general skills and learning a style.

Data diversity beats repetition. At a fixed budget of 2.25 thousand demonstrations, more layouts helped more than more demonstrations per layout, and adding matched no-preference data also helped. For anyone collecting robot data, that points toward variety.

Generalization is weak. Preference-conditioned behavior was learned across settings, but joint success and satisfaction stayed much lower on unseen layouts with different tasks.

Composition and conflicts are hard. Composing two preferences reached 25.0 percent SR and SA. Decomposing, that is recovering one preference after training on a pair, gave 37.5 percent SR but 0.0 percent SA, showing that completing the task did not imply honoring the preference. Under two conflicting preferences, SR and SA fell from 73.0 and 63.4 to 66.0 and 47.8, with no clear dominant preference, and in task-preference conflicts 37.5 percent of trials were incomplete.

Training used LoRA on pi0.5, about 45,000 optimization steps per 4,000 rollout episodes on eight A6000 Ada GPUs at a learning rate of 5e-5, with results averaged over 10 evaluation runs per fixed checkpoint. The authors note that this captures evaluation randomness, not training-seed variance.

Why does this matter?

Home and workplace robots will be judged less on whether they can fold a towel than on whether they fold it the way you like, and whether they listen when you correct them. The benchmark's results suggest today's generalist policies are good at the average task and poor at individual style. That has product and safety implications: a robot told to "stay away from the child's toys" or "move slowly near the stove" needs reliable preference following, and a task-success metric will not reveal failures.

The finding that a preference-only fine-tune can destroy general ability (52.33 percent no-preference success under sequential training) echoes continual-learning problems in language models. For readers following robotics, it complements work such as Hao AI Lab's ORA harness, which controls robots through language-model tool calls, and the frontier-model robotics benchmark.

What are people likely to ask?

Why is 97.6 percent task success paired with about 24 percent preference satisfaction? Because the two are scored differently. Success asks whether the bottle ended up in the basket. Satisfaction asks whether it got there by the handle, slowly, or avoiding the cup. A policy trained to reach goals can do so by many paths and has no reason to pick the requested one unless it was trained on that signal. Zero-shot, the language instruction rarely changes the trajectory enough.

Is this a problem only for pi0.5? The paper tested six policies and reports the strongest, pi0.5, as the headline, which implies the others did worse on the same measure. The authors' conclusion is general: current VLAs struggle with zero-shot personalization.

Does language instruction help? The ComprePref suite tests exactly this by rewriting prompts so the preference is explicit or implicit. Mixed-trained policies reached 72.83 percent success and 70.07 percent satisfaction there, well above preference-only training at 44.00 and 42.64, so training mix matters more than phrasing alone.

Why does decomposition look so bad? Training on a pair and then asking for one member of the pair yielded 37.5 percent task success but 0.0 percent satisfaction. The robot completed tasks while behaving as if the missing preference did not exist, or while mixing the pair's behavior. That is the clearest example in the paper of success hiding failure, and it parallels what agent researchers see when a correct output conceals an unsafe action.

Can I use these tasks for a home-robot product? Not directly. The suites are simulated, with scripted skills and automatic metrics. They are a diagnostic for research and for regression testing, not evidence about kitchens or elder care.

What are the limits?

The authors are explicit:

• Only predefined, language-expressible, automatically measurable preferences are covered.
• Everything is in simulation, with no real-robot or longitudinal user study.
• Implicit preference inference, interaction history and non-linguistic signals are excluded.
• The composition suite uses only two preferences.
• The conflict suite does not test whether a policy can detect a conflict or ask a clarifying question.
• The human-likeness and plausibility scores come from a small 15-person survey.

Treat the numbers as a baseline for research, not a forecast of real-world behavior. The automatic satisfaction metrics also define what counts as honoring a preference, which narrows what "personalization" means here.

What can builders do with it?

1. Use it as a regression test. If you fine-tune a VLA, run SinglePref and PrefScale to see whether preference learning damages general ability.
2. Prefer diverse layouts. The scaling result suggests spending budget on scene variety.
3. Mix data deliberately. Blend preference and no-preference demonstrations, then check motion preferences specifically, since they were hit hardest.
4. Add clarification. The benchmark does not test asking the user; a policy that asks when preferences conflict would address the gap the authors leave open.
5. Reproduce first. The project provides one script per table and figure, covering fine-tuning through evaluation.

Related reading

• ORA: open robot-use agents with DeepSeek-V4.1-Flash
• Robotics benchmark of frontier models on arms and humanoids
• GPT-6 Astra on a robot control benchmark
• Gemini 4 Argon for robotics
• Egocentric data collection for robots, explained
• FDE-Gym: agents on messy customer delivery

Based on the LIBERO-PeRM paper and project page as of October 16, 2026; results are the authors' and have not been independently reproduced.