NVIDIA Engineer Builds Interactive Haunted House Garage Door Using Open AI Models
Discover how an NVIDIA engineer built MonsterHouse, an interactive Halloween garage door powered by open Nemotron models, speech recognition, and Unreal Engine.
A creative project transformed a standard suburban garage door into a grumpy, talking pirate-themed building for Halloween. Named MonsterHouse, the interactive installation listens to trick-or-treaters, complains about visitors, and softens up only when people use good manners. The system relies entirely on a stack of open technologies running on a single workstation equipped with an NVIDIA RTX PRO 6000 Blackwell Max-Q and an RTX PRO 4000 Blackwell GPU.
The Technical Pipeline Behind the Monster
The project integrates multiple components to handle speech, thought, speech synthesis, and facial animation simultaneously. First, NVIDIA Nemotron Speech ASR converts incoming speech into text. For the reasoning engine, the creator selected the NVIDIA Nemotron 3.5 Bolt E4B 4GB model—a small, open language model tailored for local conversational agents. A custom LoRA adapter helps shape the model's cantankerous personality, which was trained on thousands of multi-turn dialogue examples.
Once the language model generates a reply, NVIDIA Magpie TTS Zeroshot synthesizes the audio using a short reference sample to establish the pirate voice. To save compute time and reduce latency, the setup utilizes a text bank and cached recordings for frequent remarks, dropping end-of-speech-to-audio times significantly. Clean audio feeds into NVIDIA Audio2Face-3D, which drives the facial animation inside Unreal Engine 5. Meanwhile, real-time vocal effects add a deep, echoing resonance to the speakers.
For visual awareness, the system incorporates NVIDIA Nemotron 3 Nano Omni, a vision-language model designed to describe costumes, paired with YOLO and ByteTrack to count and track people in the driveway. Developers looking to experiment with similar technologies can try top AI models cheaply through one API at https://apixoai.online.
What it means for developers
MonsterHouse demonstrates how modern open-source AI pipelines can be deployed locally for real-time interactive applications without relying on heavy cloud infrastructure. By breaking the architecture down into distinct modular blocks—speech recognition, small language models, zero-shot text-to-speech, and 3D facial animation—developers can construct complex conversational agents on a single workstation.
The project highlights the viability of small language models like Bolt E4B for edge deployment, proving that compact models paired with efficient quantization and prompt engineering can deliver responsive, context-aware characters. Furthermore, utilizing hybrid approaches such as audio caching and pre-rendered dialogue paths helps bypass performance bottlenecks, offering a practical blueprint for interactive installations, gaming NPCs, and voice-driven applications.
Source: nemotron-halloween — Hugging Face blog. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
