S-Agent 2.0: Give a Coding Agent a 3D Perception CLI and It Beats SpatialClaw

S-Agent 2.0 makes an unusual claim: you do not need a specialized spatial model to get strong 3D reasoning, you need a general coding agent and the right command-line tools. The paper, from researchers at Nanyang Technological University, the University of Hong Kong and Ropedia (senior author Ziwei Liu), is described on its project page as adding "perception commands to a coding agent's existing shell, file system and execution loop." The code is on GitHub.

The headline: on MMSI-Bench and ReVSI, S-Agent 2.0 beats the specialized SpatialClaw system with both an open model and a closed one. That fits a theme we have followed in GPT-6 Astra's 3D builds and Qwen's BlenderWorld benchmark: agents get better at the physical world when you give them tools that return grounded measurements instead of asking a vision model to guess.

TL;DR: S-Agent 2.0 at a glance

How does S-Agent 2.0 work?

A vision-language model asked "how far is the chair from the door, and which way would I turn?" usually answers from a vague impression. S-Agent 2.0 turns that into a tool-use problem. The authors call the goal closing the "evidence loop": acquire scene evidence, store it in persistent scene memory, and get each reading back with a plain-language spatial interpretation.

S-Agent 2.0 framework: a general coding agent with a perception toolkit for spatial reasoning Image: Ropedia and S-Agent 2.0 authors

The toolkit is a set of shell commands the coding agent can call, grouped by job.

Because the commands live in the agent's ordinary shell, the agent can write code, check its own readouts, cross-check against the video and retry. A scene persists between commands, so a measurement made in step three is still available in step ten.

S-Agent 2.0 solving an office route question with recon, segment and side readouts for spatial reasoning Image: Ropedia and S-Agent 2.0 authors

The project page's example, an office route question from MMSI-Video-Bench, is solved with recon, segment, add_instance and side readouts in sequence: reconstruct, find the objects, place the observer, ask which side something is on.

The numbers

The paper evaluates five benchmarks: MMSI-Bench, MindCube and SPAR-Bench for multi-view reasoning, and ReVSI and MMSI-Video-Bench for video.

The tool gain is the improvement over the same agent running without the toolkit. The project page also states the toolkit lifts Qwen3.8-27B on ReVSI from 61.8 to 82.3.

One finding is more interesting than the leaderboard: how gains scale with a stronger model. The paper reports that on ReVSI SpatialClaw's tool gain shrinks from +13.2 to +2.7 on the stronger backbone, while S-Agent 2.0's stays at +9.6. The authors read that as evidence that their design keeps helping as the underlying model improves, rather than being a crutch for weak models.

What the ablations say

Removing parts of the system shows what matters most (drops on MMSI-Bench and ReVSI respectively):

• Model-perception interface removed: minus 8.8 and minus 5.9
• Scene memory removed: minus 3.5 and minus 2.5
• Interpreted readouts removed: minus 1.3 and minus 1.2

The biggest contributor is how the model perceives the scene through the tool interface. Persistent memory and plain-language interpretations of readings add smaller but consistent gains.

Where do spatial agents still go wrong?

The project page includes two failure figures that are worth reading before you trust any of the scores. In the first, SpatialClaw acquires the right evidence and still answers wrongly, while S-Agent 2.0 gets the same item right. That is the "evidence loop" argument in miniature: having a depth map or a segmentation mask is not the same as using it correctly. The second figure shows grounding failures in the new system itself, an imagined viewpoint placed in the wrong spot and a quantity read in the wrong sense.

Both failure types are easy to recognize if you have debugged a coding agent. The first is a reasoning error on top of correct data. The second is a coordinate or convention error: a left-right flip, a height measured from the wrong reference, a distance read in the wrong unit. The toolkit's design, with plain-language interpretations attached to each readout, targets exactly the second kind. The ablation supports this only modestly, since removing the interpreted readouts costs about 1.3 points on MMSI-Bench and 1.2 on ReVSI, much less than removing the perception interface.

The paper also shows one trajectory that cross-checks visual grounding, the stored observer and the readouts against one another. A pattern like this, where the agent verifies a measurement two ways before answering, is the behavior you want from any tool-using agent, and it is cheap to ask for in a prompt even outside this system.

Why a general coding agent?

The design choice is the story. Specialized spatial agents hard-wire a pipeline. S-Agent 2.0 bets that a coding agent already knows how to plan, write scripts, inspect outputs and recover from errors, so you only need to supply sensors. It then shows the same toolkit working across harnesses, naming OpenHands, Codex and Claude Code. If you have read our agent harness tier list reaction, this is a research-side data point that the harness matters less than the tools it can call. For another agent stack that bets on the same pattern, see OpenAI Codex's cloud agents and computer use.

This also fits a broader pattern of grounding agents with external measurement. Safety work is moving the same way, as in NVIDIA's open agent safety platform, where the agent's actions go through inspectable tools.

How to run S-Agent 2.0

The repository README gives this outline.

git clone --recurse-submodules https://github.com/Ropedia/S-Agent-2.0.git
cd S-Agent-2.0/agent
uv venv --python 3.12
uv pip install -e ".[spatial]"
bash scripts/build_runtime.sh
cd ../perception
bash setup.sh

Then start the perception GPU server (perception/launch_gpu_server.sh), the agent gateway (agent/gateway/launch_gateway.sh), and a LiteLLM endpoint for your backbone, either local vLLM or a hosted model through the provided configs. Point environment variables such as MMSI_PARQUET and REVSI_DIR at your benchmark data and run, for example, LLM_MODEL_NAME=qwen3.8-27b-fp8 scripts/run.sh mmsi all 16. Add --no-tools to run the same agent without the toolkit, which is how you reproduce the gain numbers. Results land in agent/jobs/.

Model weights and benchmark datasets are not included. You need a GPU for the perception server and you must create SpatialClaw's own environment following its docs.

The license catch

Read the license before building on this. The code under agent/ and docs/ is Apache-2.0. The perception/ folder wraps NVIDIA's SpatialClaw, which the repository says is under the NVIDIA Source Code License-NC, a non-commercial license. The frozen models have their own licenses, listed in the NOTICE file. For research and evaluation this is fine; for a product you would need to replace or relicense the perception layer.

What is not established

• Independent replication. All results are the authors', run on their configuration. The repository had 9 stars and one open issue when we looked, so community reproduction is minimal so far.
• Backbone names. The paper uses Qwen3.8-27B and Gemini-3.8-Flash as backbones but does not link model pages, so check exactly which checkpoints are meant before comparing.
• Real-world robustness. The benchmarks are curated multi-view and video questions. The paper's own failure figures show cases where the agent places an imagined viewpoint wrongly or reads a quantity in the wrong sense, so grounding errors remain.
• Cost and latency. The page we read does not report how many tool calls or how much time a question takes.

Open versus closed backbones

The two backbones tell different stories. With the open Qwen3.8-27B served locally in FP8, the toolkit lifts ReVSI from 61.8 to 82.3, a large jump that turns a mid-size open model into something competitive on video spatial questions. With the closed Gemini-3.8-Flash, the absolute scores are higher (74.1 and 90.5) but the tool gain on ReVSI is smaller at +9.6, because the base model is already stronger.

For builders choosing a stack, that suggests the toolkit is most valuable where your model is small, local or cheap, and still positive where it is strong. It also means you can run the whole system on a single workstation if you have the GPU memory for both the perception server and the backbone, which matters for privacy-sensitive video such as home or workplace footage.

What this means for builders

If you build embodied or video agents, the transferable idea is cheap: expose perception as CLI commands with human-readable outputs, keep a scene state on disk, and let a coding agent iterate. You can try that pattern without this repository, with any depth, segmentation and calibration models you already use, provided your licenses allow it. For robotics context see our coverage of robot world models.

Follow @explainx_ai for more agent research breakdowns.

Related reading

• GPT-6 Astra: 3D, Blender and Unreal community builds
• Qwen BlenderWorld benchmark for visual agents
• Agent harness tier list reaction
• OpenAI Codex cloud agents and computer use
• NVIDIA open agent safety platform
• Dream Labs Dream-1 robot world action model
• Official: S-Agent 2.0 project page, GitHub repository, paper on Hugging Face

Details are accurate as of October 21, 2026; check the repository for later changes.