Updated October 18, 2026 · 6 min read

GPT-6 Astra wins the first robot-arm benchmark for LLMs — by taking 30 minutes and $16.36 per task

On October 14, 2026, Artificial Analysis released AA-Robotics v0.1: the first evaluation where frontier language models directly control a real 6-axis robot arm. GPT-6 Astra placed first at 69.4 — while averaging about 30 minutes and $16.36 per task against a 20-second human baseline. Here's what was actually tested, the full scorecard, and why "first place" and "production-ready" are very different things.

What AA-Robotics v0.1 tests

Most "embodied AI" leaderboards score perception or simulation. AA-Robotics v0.1 is different: it hands frontier models a real Hiwonder NexArm 6-axis robot arm (5 joints plus a gripper) on a physical desktop and asks them to complete everyday manipulation tasks. The setup:

• Four cameras — three around the workspace, one on the wrist — are the model's only eyes.
• The model gets almost nothing else: the arm's name, an HTTP API for communication, and the task description. It must perceive the scene, write and execute control code in an isolated runtime, and declare completion itself.
• Coding-agent harnesses: GPT models ran in Codex, Claude in Claude Code, Gemini in Antigravity.
• Zero-shot: no task-specific training for any model.
• Eight tabletop tasks modeled on everyday activities: press a red button, place a crocodile toy inside a rectangle, pour a drink into a cup, tidy four blocks into a box with the lid on, complete a shape-matching board, write "hello world" with a marker, build the tallest freestanding foam-block tower, and measure a box's dimensions to within 2 mm.

Artificial Analysis frames it as a bet on generalist agents: since early 2026 frontier models have become capable of directly manipulating robot arms, and the lab believes the future of robotics likely involves strong generalist models — even if, today, none of them is remotely production-ready.

The scores

Each attempt is scored out of 100 with partial credit for progress, so the composite is not a success rate:

• GPT-6 Astra — 69.4 (first place)
• GPT-6.1 Sol — 64.6
• Gemini 4 Argon — 55.3
• Claude Opus 5.5 — 48.8
• Claude Fable 5.1 — 46.3

Two details worth pausing on. First, Astra took perfect scores on four of eight tasks; second-place Sol took five — Astra's lead comes from partial credit on hard tasks, not from dominating them. Second, the two easiest tasks (pressing the button, placing the toy) were passed by all five models; beyond them the gap widens fast. Astra could pour a drink and write "hello world" but scored zero on measuring the box (no model exceeded 17 points on that ≤2 mm precision task); Sol completed block storage, shape matching, and writing but failed at pouring water; and no model earned a perfect score on the tower task — Sol even built a valid tower once, then kept adjusting it until the attempt was marked failed.

30 minutes and $16.36 per task

This is the finding that separates the leaderboard from reality:

• Time: models averaged roughly 30 minutes per task (Gemini 4 Argon was slowest at about 52 minutes). The human teleoperation baseline: about 20 seconds — nearly two orders of magnitude. Pressing a single button took Astra 2.2 minutes.
• Cost: Astra averaged $16.36 per task in API charges; the cheapest, GPT-6.1 Sol, cost $1.93 — 8.5× cheaper for only 4.8 fewer points. About 70% of Astra's cost came from visual context input, making perception the biggest efficiency bottleneck.
• Where the time goes: for Astra, roughly 20% understanding the environment and arm, about 59% action planning, and only ~13% actual arm motion. The physical movement often lasts seconds; the reasoning in between stretches to minutes.

Artificial Analysis's own conclusion: frontier language models are far too slow and expensive for any production robotics use today — a gap they expect to close quickly.

How the models fail: physics, not logic

The recorded failure clips are the most instructive part. An arm pushes an already-placed toy out of the target zone while trying to adjust it. A wrist collides with a tray, making a bottle harder to reach. A pour ignores the shifting liquid stream and spills the last drops outside the cup.

These are not reasoning errors — they are the continuous dynamics of the physical world: sliding, tipping, deformation, friction, collision. You cannot fully fix them by "thinking longer" at a high level; they demand real-time correction from proprioceptive and tactile feedback, not just re-planning from camera frames. Grasp reliability tells the same story: across six grasping tasks, Astra's grasp success rate was 45%, Claude Fable 5.1's 44%, and Sol's just 28% — even when the target was located, the gripper often closed beside the object or the object slipped after being lifted.

This fuels the ongoing industry debate between general-purpose LLMs and specialized Vision-Language-Action (VLA) "robot brains": generalist models can now move a real arm — but specialized systems, and true tactile feedback loops, still own the parts of the problem that involve touch.

What it means (and doesn't)

• It is a research preview. v0.1 covers eight desktop tasks and five models; Artificial Analysis explicitly labels it the simplest experiment they have designed, not a production benchmark.
• First place ≠ useful. At 30 minutes and $16.36 per task — versus ~20 seconds and roughly a tenth of the cost for a human — no model here is close to deployable in a home or warehouse.
• Scores reward progress, not completion. Partial credit means a 69.4 average can coexist with failing half the tasks outright.
• No tactile feedback. Camera-only perception is a deliberate design choice; it measures how far vision-and-code alone gets you, which is exactly why fine manipulation and measuring tasks crater.

The direction is the story: in a year, "LLMs controlling robot arms" went from demo videos to a quantified leaderboard where the flagship model of the field's current leader finishes most of an eight-task physical exam. For the neighboring results, see our guides to Astra as a robot brain, computer use on OSWorld 2.0, and the IKEA furniture-inspection benchmark.

Frequently asked questions

What is AA-Robotics v0.1?

A research preview released by Artificial Analysis on October 14, 2026 — the first benchmark of frontier LLMs controlling a real robot arm. Each model gets a Hiwonder NexArm 6-axis arm, four cameras, the arm's HTTP API docs, and a task description; it must write and execute control code zero-shot across eight tabletop tasks. GPT models ran in Codex, Claude in Claude Code, Gemini in Antigravity.

What score did GPT-6 Astra get on AA-Robotics?

69.4 out of 100 per task on average — first place, ahead of GPT-6.1 Sol (64.6), Gemini 4 Argon (55.3), Claude Opus 5.5 (48.8), and Claude Fable 5.1 (46.3). With partial credit counted, Astra earned perfect scores on four of eight tasks; Sol on five.

Why does a 69.4 score still not make Astra useful for real robots?

Speed and cost: about 30 minutes and $16.36 per task against a 20-second human baseline — nearly two orders of magnitude slower and far more expensive than human labor. Sol managed $1.93 per task just 4.8 points behind. Artificial Analysis says frontier models are far too slow and expensive for production robotics today.

How did the models fail the robot-arm tasks?

Physically: pushing placed objects out of position while adjusting them, colliding with trays, spilling the end of a pour, and low grasp success (Astra 45%, Sol 28%). Astra scored zero on the ≤2 mm box-measuring task, and no model topped 17 points there. These are continuous-dynamics problems that camera-only, re-plan-from-code approaches handle poorly.

Where does the time go if the motion itself is fast?

Reasoning: for Astra, about 20% of time went to understanding the scene and arm, roughly 59% to action planning, and only about 13% to actual arm motion. Even pressing one button took 2.2 minutes.

Related reading

For the other physical-world evaluations, see Astra as a robot brain, the IKEA furniture-inspection benchmark, and computer use on OSWorld 2.0. For context on reading benchmark claims honestly, see what the benchmark scores really mean.