LLM.int8(): 8-Bit Matrix Multiplication for Transformers at Scale

A 175-billion parameter model stored in 16-bit floating point requires about 350 GB of GPU memory — just to hold the weights, before any activations or batch data. That exceeds the capacity of even the most expensive enterprise GPUs. In 2022, running inference on such a model required a multi-GPU cluster costing tens of thousands of dollars.

The feed-forward and attention projection layers account for 95% of all parameters and 65–85% of all computation. If we could quantize these layers from 16-bit to 8-bit integers, we would halve the memory footprint: 175 GB instead of 350 GB. This puts models like OPT-175B and BLOOM within reach of a single server with consumer GPUs.

But quantization is not free. Reducing precision means representing continuous values with fewer discrete bins. If the distribution of values is well-behaved, the rounding error is small. If it is not — if there are extreme outliers — quantization can destroy the information the model needs to function.

Quantization maps high-precision floating-point numbers to low-precision integers. The simplest method is absmax quantization: divide every value in a tensor by the largest absolute value, multiply by 127, and round to the nearest integer. This maps the tensor into the Int8 range $[-127, 127]$.

The idea is like rescaling a thermometer. If temperatures range from $-30°$ to $+30°$, you rescale so that $-30°$ maps to $-127$ and $+30°$ maps to $+127$. Every degree gets about 4 bins — plenty of resolution. But if one day hits $300°$, then $300°$ maps to $127$ and all normal temperatures are squeezed into a tiny range around zero. One outlier destroys the precision for everything else.

A more sophisticated method is zeropoint quantization, which shifts the distribution so that asymmetric ranges use the full $[-127, 127]$ interval. This helps with activations that are all positive (like ReLU outputs), since absmax would waste half its range on negative values that never appear.

However, zeropoint quantization requires special instructions that GPUs do not natively support efficiently, making it slower in practice. The paper focuses on absmax as the practical choice and enhances it with two innovations.

Standard quantization uses a single scaling constant for the entire tensor — one number to rule all values. If any single value is an outlier, it sets the scale for millions of other values. This is called tensor-wise quantization.

The key insight is that matrix multiplication can be viewed as a collection of independent inner products. Each inner product involves one row of the hidden states and one column of the weight matrix. Since they are independent, each can have its own scaling constant.

Think of it like grading exams in different classrooms. Tensor-wise quantization is like grading all students on the same curve — one genius student skews the scale for everyone. Vector-wise quantization is like giving each classroom its own curve. Each inner product gets an independent scale that fits its own range of values, so no single outlier can dominate the entire computation.

The paper's most important contribution may be its analysis of emergent outlier features. As transformers grow, something dramatic happens to the hidden states during inference: certain feature dimensions develop values with magnitudes 20 times larger than any other dimension.

These outliers are not random noise. They are highly systematic: in a 6.7B parameter transformer processing a 2048-token sequence, about 150,000 outlier values appear across all layers — but they concentrate in only 6 feature dimensions out of thousands. The same 6 dimensions, across every layer, every token, every sequence.

The emergence follows a phase transition. Below 6B parameters, outliers appear sporadically in about 25% of layers. Between 6B and 6.7B, a sharp shift occurs: outliers suddenly appear in 100% of layers and 75% of all sequence positions. This is the exact point where standard quantization methods collapse.

Why do these outliers matter so much? The paper measured what happens when you set outlier dimensions to zero before computing attention. The result is devastating: the top-1 softmax probability drops by more than 20%, and validation perplexity increases by 600–1000%. Yet these outliers represent only about 0.1% of all features.

By contrast, removing the same number of random (non-outlier) dimensions reduces the top-1 probability by at most 0.3% and increases perplexity by just 0.1%.

These few dimensions carry information that the entire model depends on. They act as a kind of information highway — critical channels through which the model routes its most important signals. Quantizing them carelessly is like cutting the main power cable to a building: a tiny physical change with catastrophic consequences.

Since outliers are systematic and sparse — the same few feature dimensions across all layers — the solution is elegant: separate them out. Before performing the matrix multiplication, identify the outlier columns (those with any value exceeding a threshold $\alpha = 6.0$), extract them into their own sub-matrices, and multiply those in 16-bit precision. Everything else — the 99.9% — stays in 8-bit.

The process works in four steps. First, scan the hidden state to find which feature dimensions contain outliers above the threshold. Second, decompose the input matrix and weight matrix: pull out the outlier columns into separate matrices. Third, perform 16-bit matrix multiplication on the outlier sub-matrices and 8-bit vector-wise quantized multiplication on the rest. Fourth, add the two results together in 16-bit.

This is computationally inexpensive because the outlier dimensions are so few — at most 7 out of thousands. The extra memory from 16-bit storage of these columns is about 0.1%.

LLM.int8() is the combination of vector-wise quantization and mixed-precision decomposition. Given 16-bit inputs and weights, the method proceeds as follows:

Step 1: Identify outlier feature dimensions — any column where at least one value has magnitude $\geq 6.0$.

Step 2: Decompose both the input and weight matrices into outlier and non-outlier sub-matrices along those feature dimensions.

Step 3: Multiply the outlier sub-matrices in 16-bit floating point. Independently, quantize the non-outlier sub-matrices using absmax vector-wise quantization (row-wise for inputs, column-wise for weights), perform Int8 matrix multiplication, and dequantize the Int32 result using the outer product of the scaling constants.

Step 4: Add the 16-bit outlier result and the dequantized 8-bit result to produce the final 16-bit output.

The method requires no retraining, no calibration data, and no changes to the model architecture. A pretrained 16-bit checkpoint can be loaded and converted on-the-fly.

The paper evaluates quantization methods across model sizes from 125M to 175B parameters using two metrics: C4 perplexity (language modeling quality) and zeroshot accuracy on benchmark tasks.

The results are striking. Standard absmax quantization begins degrading at 2.7B parameters and collapses by 13B — an 8-bit 13B model performs worse than an 8-bit 6.7B model, inverting the expected scaling trend. Zeropoint quantization holds longer but also fails at 13B. Vector-wise quantization alone delays the problem but cannot solve it.

LLM.int8() is the only method that preserves full 16-bit performance at every scale tested. At 175B parameters (OPT-175B), the 8-bit model matches the 16-bit model exactly on all benchmarks, including WinoGrande, HellaSwag, PIQA, and LAMBADA. The memory footprint drops by nearly 2x — from about 350 GB to about 175 GB — making it possible to run the model on hardware that previously could not support it.

Before LLM.int8(), running a 175B parameter model required 8 enterprise A100 GPUs with 80 GB each — hardware costing over $100,000. After LLM.int8(), the same model runs on 8 consumer RTX 3090 GPUs with 24 GB each. A model previously restricted to major corporations became accessible to academic labs.

The method was integrated into Hugging Face Transformers through the bitsandbytes library, reaching millions of practitioners with a single flag: load_in_8bit=True. This was not just a memory optimization — it was an accessibility revolution.

Perhaps more importantly, the outlier analysis changed how the field thinks about quantization. The discovery that emergent features concentrate in a handful of dimensions and are critical for model performance informed every major quantization method that followed, including GPTQ (4-bit post-training quantization) and QLoRA (quantized fine-tuning), both of which were co-authored by Tim Dettmers.