Understanding PCIe Performance in Modern AI Inference Infrastructure
A technical look at how PCIe interfaces impact AI inference workloads, where bottlenecks emerge, and how systems handle KV cache offloading and network integration.
While discussions surrounding AI inference hardware usually focus heavily on HBM memory and high-speed NVLink connections, the role of PCIe is frequently misunderstood. PCIe bridges the gap between GPUs and the rest of the server infrastructure, handling model weights, network traffic, system memory interactions, and control messages.
Understanding how data moves across these links is critical for building efficient infrastructure. Developers looking to test models can access top AI models cheaply through one API at https://apixoai.online.
Where PCIe Excels in Inference
PCIe infrastructure performs reliably when individual GPUs operate independently. Benchmarks from MLPerf v5.1 using Llama 2 70B demonstrate that 4-GPU servers equipped with H200 NVL PCIe cards achieve between 80% and 88% of the per-GPU speed delivered by fully interconnected NVLink systems. Because each GPU runs its own copy of the model, inter-GPU communication overhead remains minimal.
Loading model weights is another area where PCIe proves sufficient. Although large models require substantial data transfers, storage read speeds typically act as the primary bottleneck rather than the PCIe link itself. Furthermore, PCIe excels at managing Key-Value (KV) cache offloading. Reloading saved conversation histories from system memory over a Gen5 PCIe link is substantially faster than recomputing long prompts from scratch, supporting efficient memory management in frameworks like vLLM and SGLang.
Bottlenecks and Limitations
Despite its strengths, PCIe struggles when workloads require constant inter-GPU synchronization. Tensor parallelism splits model layers across multiple devices, forcing GPUs to execute frequent all-reduce sync operations for every generated token. On a 70B model, this translates to hundreds of sync cycles per step, causing communication delays to approach the duration of compute operations. Cross-socket traffic can further degrade bandwidth.
Performance also drops when systems attempt to stream weights or active KV cache data from outside GPU memory for every individual token, as PCIe bandwidth remains significantly lower than on-chip HBM. Additionally, modern networking requirements are outpacing older slots: 800 Gb/s network cards demand PCIe Gen6 lanes to run at maximum speed, requiring careful hardware matching.
What it means for developers
Software engineers deploying large language models must account for hardware topology to avoid unexpected latency spikes. Developers should verify that tensor-parallel groups and network cards share the same PCIe switch using tools like nvidia-smi topo -m and lspci -vv. Utilizing pipeline parallelism instead of tensor parallelism is often recommended for non-NVLink PCIe setups to minimize synchronization penalties, while pinned system memory should be utilized for efficient KV cache offloading.
Source: PCIe in AI Inference: Where It Shines and Where It Lags — Towards AI. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
