Understanding the KV Cache: How LLMs Trade Memory for Inference Speed
A deep dive into how Key-Value caching accelerates LLM autoregressive decoding, why it creates severe GPU memory bottlenecks, and how modern architectures adapt.
During autoregressive text generation, large language models produce output one token at a time. At each decoding step, the newly generated token must evaluate its relationship with every preceding token in the sequence. In an unoptimized transformer setup, processing a sequence requires recalculating the Key (K) and Value (V) vector representations for all past tokens on every single step. To eliminate this redundant computation, modern inference architectures rely on Key-Value (KV) caching.
How KV Cache Eliminates Redundant Attention Computations
Attention mechanisms rely on three primary components derived from an input matrix using learned projection matrices: Queries ($Q$), Keys ($K$), and Values ($V$). Conceptually, the Query functions as the question or target information being sought, the Key acts as the label used to establish whether a specific token is relevant, and the Value represents the actual data retrieved if a match occurs.
During model inference, the projection weights ($W_q$, $W_k$, and $W_v$) remain fixed. Attention weights are computed by taking the dot product of the Query with the transposed Keys, scaling by the square root of the key dimension ($d_k$), applying a softmax function, and multiplying the result by the Value matrix:
$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$
Because autoregressive generation only introduces one new Query at a time, the historical Key and Value representations do not change. Instead of discarding previously calculated representations and recomputing them from scratch at every step, the system stores past Keys and Values in memory. When a new token arrives, the model computes only that single token's $K$ and $V$, appends them to the cache, and computes attention using the full cached history.
The Memory Bottleneck in Extended Contexts
While caching avoids continuous recalculations, it transforms a computational challenge into a GPU memory challenge. Because every generated token adds new representations to the cache, memory consumption expands linearly alongside sequence length.
The volume of elements stored in a KV cache is proportional to $2 \times L \times T \times H_{KV} \times d_{head}$, where $L$ is the number of transformer layers, $T$ is the context length, $H_{KV}$ is the number of Key-Value heads, and $d_{head}$ is the head dimension. The factor of two accounts for retaining both Keys and Values.
For a standard 7-billion-parameter model featuring 32 layers, 32 attention heads, and a head dimension of 128 running in 16-bit floating point (FP16) at batch size 1, the memory demand grows significantly with context size:
- 4,000 tokens: approximately 1 GB of VRAM
- 32,000 tokens: approximately 8 GB of VRAM
- 128,000 tokens: approximately 32 GB of VRAM
At a 128,000-token context window, the KV cache alone requires roughly 32 GB of GPU memory before factoring in the space needed for model weights or general execution overhead.
What it means for developers
For developers managing deployment infrastructure, the linear memory footprint of the KV cache explains why long-context inference is challenging to scale. Standard Multi-Head Attention (MHA) allocates dedicated Key and Value projections for every attention head, which places intense pressure on hardware memory limits as inputs stretch into tens of thousands of tokens.
Because KV cache size scales directly with the number of Key-Value heads, researchers have shifted away from classic MHA in favor of architectures that share or compress these representations. These designs include Multi-Query Attention (MQA), which uses a single KV head; Grouped-Query Attention (GQA), which groups queries around a reduced set of KV heads; and Multi-Head Latent Attention (MLA). Alongside memory-access optimizations like Flash Attention, these architectural variations allow modern models to retain attention quality while significantly reducing memory overhead.
Developers looking to deploy and evaluate different model architectures can try top AI models cheaply through one API at https://apixoai.online. Understanding how caching parameters scale helps engineering teams better project hardware requirements, optimize batch sizes, and manage memory constraints when designing systems for long-context workloads.
Source: From Vanilla Transformers to Modern LLMs: The Evolution of Transformer Architecture — Part 2… — Towards AI. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
