Artificial Analysis, the independent benchmarking outfit best known for its model Intelligence Index, has launched its first robotics test. The launch post on its X account says: "Welcome to the era of embodied AI," and describes AA-Robotics v0.1 as a research preview that measures frontier language model performance on understanding, planning and acting in the physical world. In v0.1 each model is given control of a 6-axis robot arm.
The headline numbers circulating from the launch: GPT-6 Astra scores 69, GPT-6.1 Sol 65 and Gemini 4 Argon 55 on the arm tasks, while no model reached 2 out of 100 on 11 humanoid tasks. This post separates what is claimed from what we could verify, then explains what it means for people building with these models.
AA-Robotics v0.1 at a glance
| Question | Answer |
|---|---|
| Who made it? | Artificial Analysis, announced on its official X account |
| What does it test? | Language models controlling a 6-axis robot arm, plus a separate humanoid set |
| Top arm score? | GPT-6 Astra 69, GPT-6.1 Sol 65, Gemini 4 Argon 55 (reported) |
| Humanoid result? | No model reached 2/100 on 11 tasks (reported) |
| Speed vs humans? | Roughly 100x slower, about 30 minutes per tabletop task (reported) |
| Cost vs humans? | Roughly 10x more expensive (reported) |
| Status? | Version 0.1 research preview |
| Independently replicated? | No |
What is claimed
Everything below comes from Artificial Analysis's launch and the posts that quoted it, including the evaluation partner account Val AI, which the circulating summaries credit with the humanoid result. We attribute each point to those parties rather than stating it as settled fact.
Arm scores. GPT-6 Astra placed first at 69, then GPT-6.1 Sol at 65 and Gemini 4 Argon at 55. The same summaries say the extra four points Astra earns over Sol cost $14.43 more per task, more than eight times Sol's price. Reported average cost per task was $1.93 for Sol and $16.36 for Astra.
Humanoid collapse. On 11 humanoid tasks where the model must keep the robot balanced and walking on its own, no model reached 2 out of 100. Reports add that Grok 4.7 and Gemini 4 Argon failed to move the robot at all in most episodes.
Slow and costly. Models can now manipulate an arm for simple tasks, but the reported averages are around 30 minutes per tabletop task, roughly 100x slower and 10x more expensive than a human baseline. Reaction time also varied: the GPT-6 models reportedly began moving after 75 to 80 seconds.
One commenter quoted in the same coverage argued that models such as Gemini 4 Argon may reason well about space but "spend the entire allotted time thinking." That is an interpretation of the failure, not a measured cause.
What we could verify
- The launch post exists on Artificial Analysis's official account and names the benchmark, the version, the research-preview label and the 6-axis arm setup. That is a primary source.
- Artificial Analysis describes itself as an independent evaluator at artificialanalysis.ai, and it has published other agentic and hardware results recently, including the HeyGen voice claim we checked.
- A dedicated robotics page on its website returned a 404 when we checked, so the full methodology, task list and raw episode logs were not available to us. We could not confirm the scoring formula, how many episodes per task were run, or whether the simulator or a real arm was used.
So the honest status is: the benchmark and its existence are verified; the specific scores, costs and timings are Artificial Analysis's own and not independently reproduced.
Why a robot-arm score is not a robot-product score
A language model driving an arm is a different problem from the vision-language-action models shipped by robotics labs. Here a general chat model receives observations, plans in text and issues commands, so it pays the full latency of reasoning before every move. That is why the numbers look so lopsided: a 75-second wait before the first movement would be unacceptable in a factory, but it is plausible for a model that thinks before acting.
It also explains the humanoid result. Balancing and walking are continuous, millisecond-scale control problems. Calling a frontier model once per step cannot keep a body upright, which is why dedicated controllers handle locomotion in real products. A score under 2 says "do not put a general LLM in the balance loop", not "humanoids are far off".
A robotic gripper copying a demonstrated grasp, illustrating how AA-Robotics tests tabletop manipulation
What people are asking
Is the arm test easy? The reported framing is "simple tasks", and even then a task takes about half an hour. We could not see how the score weighs speed, so a model may score well while still being far too slow for production. Read the 69 together with the timing figures, not instead of them.
Why did two models barely move the robot? The reports say Grok 4.7 and Gemini 4 Argon never moved it in most episodes. One plausible reading, offered by a commenter, is that they exhausted the time budget reasoning. Another is a tool-call formatting mismatch with the harness. Without episode logs, neither can be confirmed, which is exactly why the missing methodology page matters.
Does a low humanoid score mean humanoid robots are stalled? No. Commercial humanoid efforts use specialized low-level controllers and learned policies, and a language model is typically the planner on top. AA-Robotics v0.1 tests the harder, unusual case of the language model doing everything.
Why does Artificial Analysis care? The group built its reputation on independent measurement of models and, more recently, agentic inference hardware. Robotics is a natural next vertical, since most robotics results today are self-reported by the labs that built them. An independent harness that runs several vendors under identical conditions is valuable even when its first version is rough.
How it compares with earlier robotics evals we covered
This is not the first time a lab has pointed these models at a robot. Google showed Gemini 4 Argon for robotics with its own ERQA and simulation results, and OpenAI's team posted GPT-6 Astra on a YAM arm, where we flagged the claims as unverified. A separate robot control benchmark with Fable 5.1 and the RoboHarm result on harmful robot tasks round out the picture. What is new is an independent third party running one harness across vendors, which makes the cross-model ranking more meaningful than a vendor demo, even at version 0.1.
For context on the same models elsewhere, see the five-way model comparison and the cost-efficiency analysis of Sol and Astra, which show the same pattern: the top model costs far more for a small gain.
Which model for which job
| Need | Reasonable reading of the preview | Caveat |
|---|---|---|
| Best arm score | GPT-6 Astra (69) | About $16.36 per task reported |
| Best value | GPT-6.1 Sol (65 at $1.93 reported) | Four points behind |
| Spatial reasoning | Gemini 4 Argon (55) | Reportedly spends the time budget thinking |
| Humanoid control | None | All under 2/100 |
| Anything real-time | A dedicated controller | LLM latency is minutes, not milliseconds |
Scores for Grok 4.7 were not in the figures we saw beyond the report that it often failed to move the arm, so we do not rank it.
What builders should do
A robot hand reaching toward a small globe, representing embodied AI moving from chat into the physical world
- Treat 0.1 as a direction, not a verdict. A research preview can change scoring, tasks and weighting before v1.
- Budget for latency and cost. If you prototype with an LLM planner, assume tens of seconds before the first action and dollars per task, then plan caching, shorter reasoning and a fast low-level controller underneath.
- Separate planning from control. Use the language model for what to do and a purpose-built policy for how to move.
- Check the safety side. Giving a model actuators raises the stakes of errors and misuse. Agent permission layers such as AgentBeam, from the explainx.ai team, exist to stop AI agents before they take dangerous actions.
- Re-test on your own hardware. One arm in one setup is not your cell, your gripper or your lighting.
How to check the numbers yourself
Start with the pinned launch thread on the Artificial Analysis account and look for a methodology link, then compare the per-task cost and timing it lists against the figures quoted here. If the site later publishes episode logs, spot-check a few failed humanoid runs to see whether the robot fell, never started, or hit a timeout. Those three failure modes imply very different fixes, and the headline score hides them.
What to watch next
- Publication of the full methodology and task list, which would let others check the scoring.
- Whether Artificial Analysis adds more embodiments, a larger humanoid set, or open-weight models, where the on-device and open model ecosystem is moving fast.
- Whether vendors publish their own numbers on the benchmark, and whether they match.
- Any change in reaction time: the 75 to 80 second start delay is the easiest figure to improve and the most telling.
Figures are as reported on October 14 to 15, 2026 and may be revised as the benchmark moves past v0.1.