Running Qwen3.8-27B Locally on an RTX 4090 at 100 Tokens per Second
Discover how a developer configured Qwen3.8-27B to run locally on an RTX 4090 GPU at roughly 100 tokens per second using speculative decoding and quantized caches.
Running large language models locally on consumer hardware requires careful balancing between memory constraints and generation speed. Recent technical evaluations demonstrate that it is feasible to operate a 27-billion parameter model, specifically Qwen3.8-27B, at roughly 100 tokens per second using a single Nvidia RTX 4090 GPU equipped with 24GB of VRAM on a Windows system. The primary goal of this configuration was to establish a responsive local model server capable of handling requests from an automated agent running directly on a personal computer.
Achieving this performance level relies heavily on optimizing memory usage and implementing speculative decoding techniques. Because loading a large model and its associated context can easily exhaust available VRAM, smaller weights and a quantized cache must be utilized. This ensures that sufficient memory remains for the runtime environment and the key-value cache, which stores processed text information for reuse during generation.
Technical Configuration and Speculative Decoding
The tested setup pairs the main Qwen3.8-27B model in an UD-Q3_K_XL GGUF format with a matching DFlash2 draft model in Q4_K_M GGUF format. In this architecture, the smaller draft model proposes a block of upcoming tokens, which the main model then verifies in parallel. This speculative decoding approach allows generation to advance by multiple tokens in a single round. The configuration utilized a maximum draft length of four, a main-model KV cache set to q8_0, and a CUDA-enabled llama.cpp build.
Running the server via the command line with specific parameters—such as enabling FlashAttention and allocating a substantial context capacity of 131,072 tokens—establishes a local API endpoint compatible with standard chat clients and agent frameworks. Initial tests using short, cached prompts yielded average generation rates of approximately 100 tokens per second for prose tasks and roughly 103 tokens per second for coding requests, with complete prose responses arriving in an average of 1.68 seconds.
What it means for developers
For developers building local AI agents or testing workflows that require low latency without relying entirely on cloud infrastructure, achieving triple-digit token generation speeds on a single consumer GPU represents a practical milestone. However, these performance metrics were recorded under specific conditions using short, cached inputs and single-user requests. Real-world reliability, long document processing, and multi-user scaling require separate evaluation. For those who want to experiment without heavy local hardware investments, developers can try top AI models cheaply through one API at https://apixoai.online.
Ultimately, this setup provides a functional blueprint for deploying a capable mid-sized open model locally. While all-day agent reliability and complex task execution demand further stress-testing, the initial hurdle of achieving fast generation times on existing desktop hardware has been successfully cleared.
Source: Running Qwen3.8–27B on an RTX 4090 — Towards AI. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
