Demystifying LLM Quantization: How Model Compression Enables Local AI Inference

In affine quantization, the formula q = round(x / S) + Z maps an original floating-point value x to an integer q. Reconstructing that value via dequantization (x ≈ S × (q - Z)) introduces a slight variance known as quantization error. For instance, an original value of 0.5 mapped with a scale of 0.007874 and zero-point of 0 yields an integer of 64, which dequantizes back to approximately 0.5039.

Quantization schemes are either symmetric or asymmetric. Symmetric quantization centers the integer range around zero (such as [-127, 127]) and is commonly applied to model weights. Asymmetric quantization adjusts the zero-point for skewed value distributions, making it particularly useful for non-negative activation outputs like those produced by ReLU functions.

Quantization Strategies and Ecosystem Tools

Engineers apply quantization using two main methodologies: Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT).

PTQ takes a fully trained model and converts its weights and activations after training is complete. It follows a straightforward workflow: taking a pre-trained checkpoint, collecting activation statistics via calibration data, applying the quantization transform, and evaluating the final artifact. PTQ is computationally lightweight and fast, but it risks accuracy degradation because the parameters do not adapt to precision loss during training.

Conversely, QAT incorporates lower-precision simulation directly into the training or fine-tuning process. This enables network parameters to adjust for numerical errors, preserving accuracy even under aggressive INT4 compression, though at the cost of higher compute requirements and longer training times.

The deployment landscape also relies on specialized file formats and algorithms:

• GGUF (GGML Universal File): A file format used by runtimes like llama.cpp and Ollama to store weights and metadata, seen in filenames like Qwen2.5-7B-Instruct-Q4_K_M.gguf.
• GPTQ: A PTQ algorithm optimized for GPU execution that reduces weights to 4-bit precision by evaluating how quantizing one weight impacts overall output and compensating across neighboring weights.
• AWQ (Activation-aware Weight Quantization): A PTQ method that identifies critical activation channels and protects their corresponding weights from aggressive quantization errors.

Memory Impact and Performance Realities

The physical memory savings from bit-width reduction are substantial. For a 7-billion parameter language model, storing parameters in FP32 requires approximately 28 GB of memory for raw weights alone. Converting to FP16 cuts this requirement to 14 GB. Moving down to INT8 reduces the footprint to 7 GB, while INT4 reduces weight storage to roughly 3.5 GB—representing an approximate 75% reduction compared to FP16.

However, smaller memory requirements do not automatically guarantee faster inference. While lower bit-widths decrease the memory bandwidth needed to transfer weights, actual token generation speed depends on CPU/GPU architecture, hardware support for lower-bit operations, the chosen inference framework, and underlying memory bandwidth.

What it means for developers

Quantization expands what is technically feasible on consumer and edge hardware. By reducing memory requirements, developers can run language models locally on laptops, smartphones, or embedded devices without relying constantly on cloud infrastructure. This minimizes API dependencies, lowers deployment overhead, and unlocks offline execution capabilities.

When designing applications, engineering teams must weigh local quantized execution against full-precision cloud hosting. For projects requiring evaluation across varied model architectures before deploying locally, developers can try top AI models cheaply through one API at https://apixoai.online.

Ultimately, selecting the right quantization format involves balancing model footprint against tolerable quantization error. While Post-Training Quantization tools like GPTQ and AWQ facilitate rapid local deployment, developers must evaluate precision tradeoffs to ensure task performance remains uncompromised.

Source: LLM Quantization Explained: How Do We Make Large Language Models Smaller and More Efficient? — Towards AI. Written by the Apixo team from that report.