NVIDIA Built a Talking Haunted House With Nemotron Open Models: How the Voice Agent Works

NVIDIA has published a build log for a talking haunted house, and underneath the Halloween joke it is one of the most detailed public write-ups of a complete, local, real-time voice agent we have seen this year. The project, called MonsterHouse, turns a garage door into a grumpy pirate-ship character that listens to trick-or-treaters, answers in a deep gravelly voice, moves a wooden mouth in sync, and warms up when a visitor says "please". The article went up on the NVIDIA Hugging Face blog on October 30, 2026, and it is written by an NVIDIA employee who says they tested the Bolt E4B model before its public release.

This post breaks down what is in the stack, the latency numbers the author gives, the design choices that matter for anyone building a voice agent, and what is and is not released.

TL;DR: the MonsterHouse stack

What does the voice pipeline look like?

The author describes a classic cascaded pipeline, the shape most production voice agents still use: speech recognition turns audio to text, a language model writes the reply, text to speech voices it, and a separate component animates the face. Voice activity detection and end-of-utterance handling decide when the character should talk. Everything is coordinated by the Nemotron Voice Agent, which the author calls a foundation for "listening, conversation and speech right out of the box".

Two details stand out. First, the pipeline "releases speakable text before an entire answer is finished", so the house starts talking while the rest of the sentence is still being generated. This is the standard trick for hiding language-model latency, and it matters more than raw model speed. Second, prepared greetings and candy jokes are stored as a text bank with cached audio recordings. Authored text skips fresh dialogue generation and saved audio skips synthesis, which is why cached replies come back in 0.72 seconds instead of 1.70.

If you are comparing approaches, our guide to Hugging Face's speech-to-speech voice agent stack covers the open-source cascaded route, while OpenAI's GPT Live 1 API is the hosted alternative that collapses the stages into one model.

Why Bolt E4B, and what did the author compare it with?

The language model is the interesting choice. The author says they tested "more than a dozen models" and landed on Nemotron 3.5 Bolt E4B, an open small language model "purpose-built for local conversational agents and interactive characters". They use the 4-bit quantized release, which is where the "4GB" in the name comes from, as the default response model, and keeps the BF16 checkpoint for comparison.

The stated reason is footprint. The author describes Bolt E4B as having a smaller footprint than the other frontrunners they tried, named as Qwen3 30B-A3B and Qwen 3.6 35B-A3B. That is an author comparison from one build, not a benchmark, so treat it as a practitioner's report: for a character that has to share two GPUs with a game engine and a text-to-speech model, a 4 GB model is attractive because it leaves compute for everything else.

This sits in a broader pattern. NVIDIA has been filling in the Nemotron family from the very large end, such as Nemotron 3 Ultra at 550B parameters, down through mid-size models like Nemotron 3.5 Lightning 30B-A3B. Bolt E4B is the small, speech-friendly end of that range. The Voice Agent repository includes Bolt E4B profiles for cloud, server and single-GPU deployments, and the model can be tried first on build.nvidia.com, where the author recommends conversational mode for short, speech-ready replies and disabling reasoning for simple exchanges.

How does a garage door get a personality?

A system prompt that says "be grumpy" is not enough, the author argues. The character has to follow a master conversation flow, respond to what visitors actually said, track the state of each encounter, and leave enough compute for the rest of the pipeline.

The design is a small state machine wrapped around the model:

• Visitors are first brushed off with lines like "I'm busy grumbling, go bother the neighbors!"
• Persistence wears the house down, and a "please" can break through immediately.
• "Thank you" earns a warm farewell, and once a visitor is won over, the house stays won over, so nobody has to negotiate with the same garage door twice.
• Seven moods, from grumpy and menacing to amused and delighted, are chosen by emotion tags that the prompt template asks the model to emit, and the text to speech engine uses them to pick the delivery.

To make the model hold the character, the author built a dataset of more than 5,000 multi-turn conversations, created and judged by a local agent and "a very large LLM", and trained a LoRA adapter for each candidate model that seemed good enough. Some took about an hour to train and some a couple. He then benchmarked models and adapters for quality, repetition and speed before listening to the results personally. The lesson for builders is the loop, not the numbers: generate reviewed conversations, train a small adapter per candidate model, measure repetition and timing, then judge by ear.

This is the same idea behind other small-model-for-a-narrow-job stories we have covered, such as a 4B model trained with reinforcement learning to beat GPT-6 on a game task: a tightly specified job lets a small model compete.

What are the latency numbers?

The post gives two figures, both medians for end of speech to the first audio:

The author adds that speech-end detection currently waits for 300 ms of silence, which counts toward both. Audio2Face-3D buffering adds a further 100 ms compensation to keep lip movement aligned with sound. Nothing here is a formal benchmark, and there is no word on how latency varies with reply length, but a 1.7 second reply on a local stack is in the range most people accept for conversation, and the cached path shows how much of the delay is synthesis and generation rather than recognition.

How does the face work?

Audio2Face-3D turns clean speech into mouth animation inside Unreal Engine 5. The author built a custom rig in Autodesk Maya, exported facial expression shapes, and let Audio2Face-3D drive them. The house has no eyes or eyebrows, which the author calls "surprisingly economical".

A useful detail: Audio2Face-3D prefers a clean, neutral voice that is not too deep or too high. The author wanted a demonic, echoing voice, so the clean speech goes to the animation and applies adjustable real-time effects to a second audio stream that goes out the speakers. Animation and sound are separate streams from the same source. That is a general pattern for any character: drive the face from clean audio, and treat the audience-facing voice as a separate effects chain.

How is the work split across hardware?

The build uses two GPUs. The language and voice pipeline runs on an NVIDIA RTX PRO 6000 Blackwell Max-Q. Unreal Engine graphics and Audio2Face-3D run on an NVIDIA RTX PRO 4000 Blackwell. The author says memory was not the issue, compute was, and the split keeps animation at a steady frame rate no matter how busy the agent pipeline is. If you are building a voice agent that also renders anything, isolating rendering from inference on separate devices is the practical takeaway.

What does the vision part add?

Nemotron 3 Nano Omni, a vision-language model, is an optional observer that describes costumes seen on camera, so the character has something specific to react to. YOLO and ByteTrack handle counting and tracking, which keeps the jobs separate and avoids asking the vision-language model to narrate every frame.

The author is candid about the evidence. In a test, Omni was given a video of five costumed sheep and its saved analysis identified pirate, vampire, witch, pumpkin and ghost outfits. The author calls that a synthetic test, "not evidence of a successful Halloween night", and says real driveway groups, costumes and night lighting still need testing. They also expect that costume identification and witty retorts may need more post-training, and notes that candy is not yet dispensed automatically.

What is released and what is not?

This matters for anyone hoping to download it. The article says "the MonsterHouse code itself is not packaged for release", but every NVIDIA model it uses is available today, and the Nemotron Voice Agent gets you most of the way. The author's suggested path:

1. Clone the Nemotron Voice Agent and run the Generic Assistant example, which gives you ASR, an LLM, Magpie TTS, voice activity detection and end-of-utterance handling.
2. Make Bolt E4B the brain using one of the included profiles.
3. Add a personality with a system prompt and a few connected example dialogues, and track conversation state.
4. Optionally train a LoRA adapter on reviewed conversations.
5. Add speech output with Magpie TTS Zeroshot and a short reference sample per mood.

The custom Unreal Engine scene, the control desk with its soundboard and mood controls, and the effects rules that tie fire and glow to delivery are described but not shared.

What people building voice agents should take from it

First, a cascaded local stack is now realistic for a single workstation. Second, small open models plus a LoRA can carry a character, provided you add structure around them: a conversation flow, state tracking, and emotion tags. Third, caching authored lines is the cheapest latency win available. Fourth, put rendering and inference on different GPUs.

There are caveats. This is a showcase by a vendor employee, on vendor models and vendor hardware, and the latency and model comparison numbers come from the author, not from independent testing. No Halloween-night results with real children are reported. Still, as an engineering write-up it is concrete, and it is one more sign that open voice stacks are moving from demos to something a hobbyist can assemble. For a broader view of where voice agents are heading commercially, see Sarvam's public voice agent launch.

What people are asking

Is Bolt E4B available now? The author says it is publicly released and that they tested it before release. Check the model card on Hugging Face and build.nvidia.com for the license and exact variants before shipping anything.

Do I need two GPUs? Not necessarily. The split was for smooth Unreal Engine animation. A voice-only agent can run on one GPU, and the Voice Agent includes a single-GPU deployment profile.

Is it better than a hosted voice API? Different tradeoff. The author wanted an off-grid, self-contained system with a fixed memory and compute budget, which a hosted API cannot give. A hosted model will likely have a stronger general brain and simpler setup.

Bottom line

MonsterHouse is a holiday stunt, but the write-up gives numbers and design choices that most voice-agent announcements leave out: a 4 GB open model, a LoRA trained on 5,000 conversations, 1.70 seconds to first audio, 0.72 seconds with caching, and a two-GPU split. If you build voice agents, it is worth reading the original and cloning the Voice Agent example this week.

Related reading

• Hugging Face speech-to-speech voice agent guide
• OpenAI GPT Live 1 API for voice agents
• NVIDIA Nemotron 3 Ultra 550B open-weight model
• NVIDIA Nemotron 3.5 Lightning 30B-A3B
• Sarvam voice agents public launch
• A 4B model beats GPT-6 on a narrow task

Details are accurate as of October 30, 2026, based on NVIDIA's Hugging Face blog post. Model availability and licenses can change.