Distilling VLMs Into Lightweight Edge Detectors: A Performance and Cost Breakdown
Learn how distilling Vision-Language Models like Gemini into compact detectors like RF-DETR Nano dramatically cuts latency and inference costs for edge deployments.

Running vision-language models (VLMs) like Google Gemini on every frame of a continuous camera stream can quickly become prohibitively expensive and slow. To overcome these hurdles, computer vision developers are adopting a teacher-student model distillation approach. By using a powerful VLM to automatically annotate visual datasets, refining those labels with human review, and training a lightweight student detector like RF-DETR Nano, engineers can deploy efficient object detection directly on local edge devices.
The teacher-student distillation pipeline
The distillation workflow relies on using a VLM foundation model as a "teacher" to generate candidate labels and bounding boxes, which are subsequently reviewed before training a specialized "student" detector. Grounded in the broader model compression principles pioneered by Hinton, Vinyals, and Dean, this tutorial follows the Autodistill label-generation approach. Rather than transferring language capabilities, the process transfers spatial object detection supervision.
In a practical warehouse monitoring setup—identifying objects such as cardboard boxes and pallets—developers configure a Roboflow Workflow named dispatch-vlm-teacher utilizing a Gemini block (such as gemini-4-argon). The Gemini block automatically decodes detection responses into bounding boxes and object classes without requiring separate conversion steps. Crucially, an error_status=True output signals a decoding failure, preventing failed calls from being mistakenly logged as empty scenes.
Once the Gemini teacher generates candidate bounding boxes, human reviewers inspect and correct the labels inside Roboflow Annotate. Annotators fix overlapping or duplicate boxes, adjust borders to cover the visible extent of partially hidden items, and insert missing objects. The accepted annotations form the finalized dataset used to train RF-DETR Nano, a compact architecture within the RF-DETR model family, via hosted training.
After training, the student model is integrated into a workflow named rfdetr-student and deployed on an edge computer running Docker and the Roboflow Inference Server on port 9001 (http://localhost:9001). This eliminates the cloud VLM reliance during continuous video frame processing.
Benchmarking performance and cost reductions
A direct cloud API benchmark comparing the Gemini teacher workflow against the RF-DETR student workflow on a 100-image "Pallet Monitoring" test set highlights the massive operational differences between foundation VLMs and specialized edge models. The load test was configured at 1 request per second, planning 120 inferences across 2 minutes.
The benchmark data revealed stark contrasts across latency, reliability, and cost:
- Latency and Throughput: The Gemini teacher recorded a median (p50) request latency of 23.8 seconds, a p95 latency of 58.4 seconds, and a cold start duration of 13.6 seconds, achieving a throughput of 0.72 requests per second. In contrast, the RF-DETR student model achieved a median latency of just 409 milliseconds, a p95 latency of 1,062 milliseconds, and a cold start of 860 milliseconds, resulting in a throughput of 0.96 requests per second.
- Reliability: Under the test load, the Gemini teacher registered 2 failed requests and failed to fully keep up with the incoming request rate. The RF-DETR student model completed the run with 0 failed requests.
- Inference Costs: The cloud benchmark reported a cost of 5.19 credits per 1,000 inferences for the Gemini teacher (approximately 0.0052 credits per single-image inference, totaling 0.9 credits for the test run). The RF-DETR student cost just 0.12 credits per 1,000 inferences (approximately 0.00012 credits per inference, totaling 0.0216 credits overall).
What it means for developers
Distillation offers a clear pragmatic path for teams building real-time vision applications. Instead of choosing between expensive cloud foundation models and manual annotation from scratch, developers can combine both approaches. Utilizing VLMs to bootstrap dataset creation slashes manual labeling time, while deploying a distilled student model cuts per-inference cloud fees significantly and reduces median latency from nearly 24 seconds to under half a second.
For teams building early-stage prototypes or exploring multi-model AI architectures, access to foundation models remains essential during the dataset creation and teacher workflow setup phase. Developers looking to experiment with teacher models can try top AI models cheaply through one API at https://apixoai.online, simplifying integration before transferring supervision down to edge-ready detectors.
Ultimately, distilling VLMs into compact models like RF-DETR Nano allows hardware-adjacent applications—such as warehouse camera monitors—to run reliably on local edge computers without latency bottlenecks or runaway API costs.
Source: How to Distill a VLM Into a Small Object Detection Model — Roboflow Blog. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key

