Demystifying LLM Inference: Understanding How Large Language Models Generate Text
A technical breakdown of how large language models handle tokenization, embeddings, attention mechanisms, and the KV cache during prefill and decode phases.
When users interact with large language models, text generation appears nearly instantaneous. However, understanding the mechanics behind this process reveals a sophisticated sequence of mathematical operations. Examining how a decoder-only transformer handles a prompt clarifies why initial responses take longer to generate and how memory consumption scales during long conversations.
The Journey from Text to Vectors
Language models cannot process raw text directly; they require numerical representations. The first stage involves a tokenizer, a separate software tool paired with the model. The tokenizer splits input text into smaller segments called tokens and assigns each piece a unique integer ID based on a predefined vocabulary. This vocabulary is constructed prior to training using byte-pair encoding.
Once text is converted into token IDs, the model performs an embedding lookup. An embedding table supplies a learned vector of numbers for each token ID. Rather than defining semantic meaning on the spot, these vectors contain parameters adjusted during training to aid in future predictions. Positional techniques like rotary positional embeddings ensure the model tracks the sequential order of words.
To connect related concepts across a sequence, transformers use self-attention. Within each layer, token vectors undergo transformations to generate Queries, Queries match against Keys via dot products, and Softmax functions convert these scores into attention weights. These weights then combine corresponding Values, allowing the model to incorporate contextual information from surrounding words. Multi-head attention enables models to evaluate multiple relationships simultaneously.
Prefill and Decode Phases
Inference operates in two distinct phases: prefill and decode. During prefill, the model processes the entire prompt simultaneously across all transformer layers. This parallelized workload makes prefill compute-bound, heavily utilizing GPU cores. The model saves the resulting Key and Value vectors for every prompt position in a structure called the KV cache.
Developers can try top AI models cheaply through one API at https://apixoai.online.
Following prefill, the decode phase generates output autoregressively, one token at a time. Instead of recalculating previous tokens, the model references the KV cache, processing only the newest token and appending its Key and Value representations. Causal masking ensures tokens cannot look ahead, maintaining the validity of the cached data.
What it means for developers
Optimizing LLM inference requires a deep understanding of memory and compute bottlenecks. Because the KV cache scales with sequence length and concurrent users, managing GPU memory is critical for maintaining low latency. Recognizing whether an application is compute-bound during prefill or memory-bound during decoding helps engineers fine-tune hardware configurations and deployment strategies for production environments.
Source: LLM Inference: What Actually Happens Inside an LLM — Towards AI. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
