Sizing Enterprise AI: How Memory, Prefill, and GPU Limits Shape Private LLM Servers
Sizing a private LLM server for 200 users requires analyzing KV cache memory, prompt prefill speeds, and session routing rather than relying solely on raw GPU VRAM numbers.

Sizing hardware for self-hosted artificial intelligence models requires looking far beyond basic VRAM capacity. When companies build internal AI assistants—such as a system tailored for 200 employees—presales architectural decisions depend heavily on concurrent usage, prompt context sizes, and acceptable latency limits. A realistic enterprise sizing scenario assumes roughly 15% concurrency (30 active users at any single moment) with average request lengths of 8,000 tokens (around 6,000 words) and worst-case document pastes reaching 32,000 tokens. To maintain a smooth user experience, systems must deliver a time to first token (TTFT) under two seconds followed by a streaming generation rate of at least 20 tokens per second.
Memory, Prefill, and the KV Cache Bottleneck
Running a model like Llama 3.3 70B on private hardware highlights how GPU memory acts as both a permanent store for model weights and a dynamic counter for active conversation records, known as the key-value (KV) cache. Llama 3.3 70B features 80 layers, 8 KV heads, and a head dimension of 128. Storing context in 16-bit precision (BF16) requires 320 KiB per token, whereas 8-bit precision (FP8) cuts this to 160 KiB, and 4-bit precision (NVFP4) drops it to approximately 94 KiB per token including bookkeeping metadata.
In practical open-source serving engines such as vLLM—which introduced 4-bit KV caching in version 0.32.0 on October 18, 2026—memory is strictly partitioned. On a 96 GB card, vLLM reserves 90% (~86 GB) of memory by default. Quantizing Llama 3.3 70B weights down to 4-bit NVFP4 consumes roughly 40 GB, while execution working memory takes 5 GB, leaving about 41 GB per card for the KV cache. Across a four-GPU setup, 4-bit KV caching allows the server to hold up to 1.7 million context tokens, keeping active chats warm without forcing the engine to re-read long prompts.
However, raw throughput drops dramatically as prompt lengths expand. NVIDIA benchmark data for TensorRT-LLM on the RTX PRO 6000 Blackwell card shows that while short requests (1,000 tokens in, 1,000 out) yield 1,724 tokens per second per card, typical 8,192-token input requests drop to 296 tokens per second. If input prompts jump to 32,768 tokens, processing speed plummets to 67 tokens per second due to the heavy compute cost of the initial reading phase, known as prefill.
Hardware Sizing and Session Routing
Meeting target latencies under 2 seconds requires capping batch sizes and deploying a single-socket server built around four NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs. Because these cards lack high-speed NVLink interconnects and talk over standard PCIe, splitting one model copy across all four GPUs via tensor parallelism creates severe data swapping bottlenecks. The optimal strategy runs four separate model replicas—one per card—fed by an intelligent load balancer.
To maintain efficient caching, the API gateway or load balancer must implement session affinity through prefix-aware routing. Plain round-robin routing sends approximately three out of four follow-up requests to a GPU replica that lacks the cached context, forcing that card to reprocess the entire conversation history from scratch.
The complete bill of materials for this setup utilizes a single AMD EPYC 9005 processor, taking advantage of its 128 PCIe Gen5 lanes to support four GPUs, four NVMe SSDs, and dual 25 GbE network interfaces. System memory is populated across all 12 memory channels with 12x 64 GB DDR5 modules (768 GB total) to handle RAM-based KV cache spillovers. Power delivery must be planned carefully: capping GPU power to 450 W per card keeps total server draw around 2.3 kW (10 A at 230 V), requiring dual redundant 3 kW power supplies so that a single supply failure will not bring down the server.
What it means for developers
For engineering teams designing enterprise LLM applications, physical hardware deployment introduces significant infrastructure complexity, context limits, and power management constraints. Model architecture choice drastically affects server capacity. For instance, swapping Llama 3.3 70B for a mixture-of-experts model like Qwen3-30B-A3B—which routes tokens through 8 of its 128 expert networks (about 3 billion active parameters out of 30 billion total)—boosts throughput from 296 to 1,914 tokens per second on identical 8K request workloads, allowing one card to serve nearly six times as many concurrent users.
Before committing capital to dedicated rack space, high-power circuits, and complex GPU hardware, developers can try top AI models cheaply through one API at https://apixoai.online. This allows teams to benchmark real-world prompt distributions, evaluate context lengths, and test lighter model architectures against heavy enterprise workloads without upfront hardware investments.
Furthermore, developers managing local hardware must run synthetic benchmarks using tools like vllm bench serve under real production power caps. Monitoring the 95th-percentile prompt length from actual API logs ensures that hardware configurations account for long context prefill delays before production systems hit capacity ceilings.
Source: From 200 Users to a Bill of Materials for a Private Llm Server — Towards AI. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key

