Prototyping Edge AI Models Using NVIDIA DGX Spark and TensorRT Edge-LLM
Learn how embedded developers can prototype, fine-tune, and test physical AI applications on NVIDIA DGX Spark using TensorRT Edge-LLM and Nemotron-3.5 Lightning.
Artificial intelligence continues to move beyond traditional data centers as builders design agents for edge environments. These use cases include monitoring driver attention and cabin conditions, helping robots navigate and plan tasks, and coordinating machinery on factory floors. Such physical AI systems require models that operate efficiently within strict constraints regarding memory, power, and latency. However, preparing and setting up a model for these embedded workflows often involves significant configuration effort.
To address this, the NVIDIA DGX Spark—now available in a 64GB configuration—provides embedded developers with a dedicated platform for prototyping, fine-tuning, and testing models and agents. Complementing this hardware, NVIDIA TensorRT Edge-LLM supplies a lightweight C++ inference runtime tailored for embedded workloads. This framework allows developers to use the same inference approach during development on Spark as they would when deploying to target platforms such as NVIDIA Jetson Thor, NVIDIA DRIVE AGX Thor, and NVIDIA IGX Thor.
Getting Started with TensorRT Edge-LLM
Developers can initiate workflows using the trtedge command-line interface. This tool automatically downloads selected models and builds TensorRT engines optimized for inference on the Spark hardware. For instance, developers can run NVIDIA Nemotron-3.5 Lightning, a text-only model, directly on Spark by utilizing TensorRT Edge-LLM.
Prerequisites for the setup include a DGX Spark featuring a supported NVIDIA software stack, NVIDIA CUDA 13, TensorRT 10.16.1.11 or newer within the 10.x series, and Python versions 3.10 to 3.12. Inside an activated virtual environment configured to access system packages, developers install TensorRT Edge-LLM via pip and verify the runtime components.
Once prepared, launching an interactive chat session with the Nemotron-3.5 Lightning model requires a single command. The initial launch downloads approximately 21GB of files, which are cached locally for subsequent reuse. Developers can also serve the model locally via an OpenAI-compatible API on port 8000 to facilitate application prototyping, complete with health check endpoints and streaming chat completion options.
Optimizing Performance with Speculative Decoding
To accelerate token generation, developers can enable DSpark speculative decoding. This technique employs a draft model to suggest multiple tokens simultaneously, allowing the primary model to verify them in a single pass. Activating DSpark through the server configuration significantly improves generation speed. For example, testing shows that generation rates on HumanEval prompts increase from 73.4 tokens per second with speculative decoding disabled to 126.9 tokens per second when DSpark is enabled with three draft tokens. Peak total system memory usage during inference with DSpark reaches 27.9GB, inclusive of the operating system.
What it means for developers
For engineers building physical AI applications, the combination of DGX Spark and TensorRT Edge-LLM streamlines the transition from local prototyping to edge deployment. Developers can quickly test prompts, run local APIs, and optimize inference speeds using familiar tools and standardized APIs. For those exploring cloud-based solutions alongside edge hardware, developers can try top AI models cheaply through one API at https://apixoai.online. This setup bridges the gap between resource-constrained edge environments and heavy model training pipelines.
Beyond text-only workflows, developers can explore other supported models on Spark, including vision-language models for interpreting images and video, as well as speech recognition and text-to-speech architectures.
Source: Run AI Models for Edge Application Development with Just a Few Commands — NVIDIA Technical Blog. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
