Scene-Designer: An 8B Model That Designs 3D Rooms and Critiques Its Own Work

If you want a model that can lay out a bedroom, an office or a bathroom in 3D from a one-line prompt, the interesting question is no longer whether a large model can place objects. It is whether a small model can learn to tell a good layout from a bad one, and use that judgment to waste fewer steps. A Zhejiang University team has released Scene-Designer, an agentic framework built around exactly that idea, with code, an 8B checkpoint and a critique test set on GitHub and Hugging Face (paper, repo, project page).

The headline number: the learned critic picks the correct ranking of four candidate scenes 76% of the time, while GPT-5.2 manages 15% on the same test. This post covers what the system does, how it was trained, which claims hold up, and what you can actually run today.

The full preprint is on arXiv.

TL;DR: Scene-Designer at a glance

What problem does Scene-Designer solve?

Agentic scene builders such as I-Design, SceneSmith and SceneWeaver work by letting a language model call tools: create a floor plan, add a bed, move a lamp, check for collisions, repeat. Quality comes from a strong general model making good decisions. The paper's diagnostic study of 100 scenes generated by GPT-5.2 found that even strong models make physical and layout mistakes, and that fixing even simple errors often takes several unnecessary trial-and-error moves.

The authors argue two skills are missing. One is design: construct a scene that is physically valid, functionally correct (a desk has a chair, a window is not blocked) and layout-coherent. The other is critique: look at two or four finished scenes and say which is better. They call the combination "design intelligence".

Scene-Designer design intelligence overview: scene design and scene critique as two complementary abilities Image: ZJU-OmniAI

The point of linking them is practical. A good critic can be used as a reward, which is the same trick that made reward models central to modern post-training. If you want a primer on that family of methods, see our guides on agentic RL with forked rollouts and fine-tuning LLMs.

How was it trained?

Training happens in two stages on top of Qwen3-VL-8B-Instruct.

Stage 1: two datasets and joint SFT

Scene-OTA-35K teaches construction. The team ran GPT-5.2 inside the SceneSmith scaffold, kept scenes that a human judged high quality, and collected the raw tool-call trajectories. Raw trajectories are noisy: around 110 steps on average, much of it flailing. The authors propose Causal Trajectory Consolidation, which flags operations that neither move the scene closer to its final state nor reduce the error count, asks GPT-5.2 whether each flagged step contributed to a needed correction, and removes the ones that did not. Replaying the shortened trajectory must reproduce the same final scene. Average length drops from roughly 110 to 30 steps. The result is about 1K scenes and 35K Observation-Thought-Action steps with reasoning traces.

Scene-OTC-15K teaches critique. Instead of asking humans to rate scenes, the team perturbs good scenes on purpose, in three severities: mild layout deviations, functional degradation, and physical violations such as collisions and out-of-bounds placement. Because severity is known, the better scene in a pair is known automatically for two of the three construction strategies; the third needs manual annotation. This yields about 15K comparison examples with reasoning traces, plus held-out test sets of 94 two-scene comparisons and 55 four-scene rankings.

Joint SFT on both datasets produces Scene-Builder. The paper reports 2 epochs at a 1e-5 learning rate on 4 A800 GPUs.

Stage 2: Scene-GRPO

Scene-GRPO is a critic-guided variant of GRPO, the group-relative policy optimization method popular in reasoning-model training (we cover a real-world speedup in TRL AsyncGRPO). At each editing step the builder proposes four candidate actions from the same parent scene. A frozen critic supplies two local rewards:

• Branch-ranking reward: compares sibling scenes generated from the same parent, favoring better design decisions.
• Improvement reward: compares each child with its parent, discouraging edits that make no real progress.

Global rewards for trajectory length and object completeness keep scenes complete and efficient. Only the policy is updated. Learning rate is 1e-6, group size 4, with trajectories capped at 80 steps. The result is Scene-Designer.

Reinforcement learning illustrated as a ball following a curved track to a reward cup

What do the results show?

Critique: the largest gap

Two things stand out. First, a general model is no better than a coin-flip-plus at ranking four scenes. Second, adding the design data improved critique: the jointly trained critic beats the critique-only one by 25 points on four-scene ranking. The authors call this mutual reinforcement between design and critique, though it is one experiment on small test sets.

Generation: wins on most scores, not all

Scores are 0 to 10 for realism, functional correctness, layout coherence and completeness, judged by GPT-4 over 10 scenes per room type, for prompts of the form "Design me a bedroom." Selected rows from the paper's tables (realism / function / layout / completeness):

Scene-Designer reports zero out-of-bound objects and zero collisions in all four room types. But it does not sweep the table: SceneWeaver scores higher on functional correctness in the bedroom and living room (and, per the paper's other tables, the bathroom and office), and generates far more objects in the office (40 versus 23). The GRPO stage helps most on realism and layout: compared with the SFT-only Scene-Builder, realism improves by 0.7 to 1.6 points depending on room.

Qualitative comparison of agentic 3D scene generation methods across seven room prompts, Scene-Designer versus SceneWeaver and I-Design Image: ZJU-OmniAI

Efficiency: fewer wasted corrections

The paper's clearest practical claim is about trial and error. Compared with Scene-Builder, the share of interaction-orientation errors that needed five or more corrective moves fell from 40% to 7%, and window-accessibility blockages fell from 21% to 5%. If you pay per tool call or per token for an agent loop, that is the metric that translates to cost.

Critic at test time

The same critic, used at inference to pick among four candidate branches per step, adds roughly 0.3 to 0.5 points on each score and a few more objects per scene. It costs extra compute at generation time, so it is a quality-versus-latency choice.

What can you run today?

The repository lists two public resources: the Scene-Builder-Qwen3-VL-8B-Instruct full SFT checkpoint and the Scene-OTC-Bench dataset on Hugging Face. The README is explicit that the published weights are the SFT stage, before Scene-GRPO, so the headline GRPO numbers are not reproducible from released weights alone. Training scripts for SFT and Scene-GRPO are included, and the released RL entry point optimizes furniture construction; SFT and inference support floor plan, furniture, wall and ceiling stages.

The README quick start looks like this:

git clone https://github.com/ZJU-OmniAI/Scene-Designer.git
cd Scene-Designer
hf download XuanyuLiu/Scene-Builder-Qwen3-VL-8B-Instruct \
  --local-dir model/Scene-Builder-Qwen3-VL-8B-Instruct
python -m uvicorn src.tool_server:app --host 127.0.0.1 --port 18000
python -m src.rollout_client --server-url http://127.0.0.1:18000 \
  --model-dir model/Scene-Builder-Qwen3-VL-8B-Instruct \
  --prompt 'Design a warm bedroom with a bed, bedside storage, and a reading corner.' \
  --config scene_designer --max-steps-per-stage 30

Before that works you must follow the installation guide to install dependencies and a separately downloaded SceneSmith checkout, apply the integration patch, and set up geometry services and assets. The repo notes that third-party checkpoints, generators and scene assets keep their own licenses, and a separate model-license check is worth doing before any commercial use. Linux and Python 3.11 are expected. At the time of writing the repo had 12 GitHub stars and 8 commits, so expect rough edges.

What people are asking

Is this just another text-to-3D model? No. It does not generate meshes. It orchestrates tools in an existing scene-building scaffold, placing assets and checking constraints, which is why physical validity is checkable (collisions and bounds are counted programmatically).

Why does it matter for robotics? The paper motivates the work with embodied AI and spatial design. Synthetic, physically valid rooms are useful as simulation environments. For related robotics coverage see Nature Will's Natura-1 dexterous hands model and the Artificial Analysis robotics benchmark claims.

Is the agent pattern new? The shape (small model, frozen critic, branch ranking) resembles other agentic RL work, such as the RL environments on the Hugging Face Hub. The new part is the task-specific critic data built by perturbing good scenes.

Could it design CAD or product geometry? Not directly; it targets furniture-scale indoor layout. For parametric CAD with agents, see SolidCraft.

Caveats to keep in mind

• LLM-judged quality. Realism, function, layout and completeness are scored by GPT-4, on 10 scenes per room type. That is a small sample, and an LLM judge can favor certain styles.
• Authors' own critic benchmark. Scene-OTC-Bench has 94 pairs and 55 rankings, built by the same team that trained on the same perturbation recipe. A critic trained on perturbation severity may do well on perturbation-based tests without being a general design judge.
• Teacher dependence. Training data traces and the causal consolidation check rely on GPT-5.2, so the 8B model distills a stronger closed model.
• Released weights are SFT only. The GRPO gains are documented in the paper, but the checkpoint you can download stops before that stage.
• Preprint. The arXiv paper appeared on October 16, 2026 and has not been independently reproduced.

What this means if you build with AI

For builders, the lesson is less about bedrooms and more about the recipe: when you can automatically generate ordered quality differences (here, controlled perturbations), you can train a small critic cheaply, and a critic is both a reward model and a test-time selector. That is a pattern worth testing in your own domain, whether the artifact is a layout, a UI, or a plan. If you do experiment, start from the released critique benchmark and check whether your critic generalizes beyond the perturbations it was trained on.

Details reflect the arXiv preprint, project page and GitHub repo as of October 19, 2026; numbers are the authors' own and may change in later versions.

Related reading

• Prime Sandboxes and forked rollouts for agentic RL
• TRL AsyncGRPO: the 26x claim versus 3x measured
• What is fine-tuning? A complete guide
• Hugging Face Hub RL environments with OpenEnv and verifiers
• Natura-1 dexterous hands foundation model
• SolidCraft: open-source parametric CAD for agents
• Artificial Analysis robotics benchmark, claimed versus verified