GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformers

GPT-3 has 175 billion parameters. Stored in 16-bit floating point (FP16), that is 326 GB of memory — more than any single GPU can hold. Running inference requires splitting the model across multiple expensive GPUs, making it inaccessible to most researchers and developers.

The fundamental bottleneck is not computation but memory bandwidth. During autoregressive generation, each token requires loading the entire model from memory. The speed of generation is limited by how fast you can read weights, not how fast you can multiply them. If you could store each weight in 4 bits instead of 16, you would need 4× less memory and could load weights 4× faster — a direct speedup for generation.

This is where quantization comes in: replacing high-precision numbers (16-bit floats) with low-precision ones (3–4 bit integers). The challenge is doing this without destroying the model's accuracy.

Rather than quantizing the entire model at once, GPTQ works layer by layer. For each linear layer with weight matrix $\mathbf{W}$ and a small set of calibration inputs $\mathbf{X}$ (just 128 samples), the goal is to find quantized weights $\hat{\mathbf{W}}$ that keep the layer's output as close as possible to the original. Formally:

Think of this like a relay race: if every runner keeps almost the same pace as the original team, the total time stays almost the same. One runner slowing down slightly does not ruin the race — but every runner collapsing would. The layer-wise approach ensures each "runner" (layer) maintains its pace.

GPTQ builds on Optimal Brain Quantization (OBQ), which itself extends the classic Optimal Brain Surgeon framework from pruning to quantization. OBQ's idea: quantize weights one at a time, and after each quantized weight, adjust all remaining full-precision weights to compensate for the rounding error. The adjustment direction is guided by the inverse Hessian $\mathbf{H}^{-1}$.

Concretely, when quantizing weight $w_q$ at position $q$, the quantization error is $w_q - \text{quant}(w_q)$, and the optimal adjustment to all remaining weights is:

The problem: OBQ processes each row of $\mathbf{W}$ independently and picks a different greedy order for each row. This means the Hessian inverse must be updated separately for every row of every layer. For a weight matrix of size $d_{\text{row}} \times d_{\text{col}}$, the runtime is $O(d_{\text{row}} \cdot d_{\text{col}}^3)$ — cubic in the column dimension. On ResNet-50 (25M parameters), this takes about an hour. On OPT-175B, it would take years.

GPTQ makes three modifications to OBQ that together reduce the runtime by over three orders of magnitude — enough to quantize OPT-175B in about four hours on one GPU.

Innovation 1 — Fixed column order. The authors discovered that on large, heavily-parameterized layers, the greedy ordering of OBQ gives only marginal improvement over a fixed arbitrary order. The reason is intuitive: in a layer with thousands of weights, the handful of "bad" rounding decisions caused by a suboptimal order are diluted among thousands of "good" compensations. By quantizing all rows in the same fixed column order, the Hessian inverse $\mathbf{H}_F^{-1}$ is shared across all rows. The inverse needs to be updated only $d_{\text{col}}$ times instead of $d_{\text{row}} \times d_{\text{col}}$ times, reducing runtime by a factor of $d_{\text{row}}$.

Innovation 2 — Lazy batch updates. Even with a fixed order, updating the entire remaining weight matrix after each single weight is memory-bandwidth-bound — too many memory reads and writes for too few floating-point operations. GPTQ batches the updates: it processes blocks of $B = 128$ columns at a time, keeping intermediate updates local to the block. Only after finishing a block does it apply a single large matrix update to the remaining weights. This transforms the workload from many small memory-bound operations into a few large compute-bound matrix multiplications that GPUs excel at.

Innovation 3 — Cholesky-based Hessian inverse. Repeatedly removing rows and columns from the inverse Hessian via Gaussian elimination accumulates numerical errors. GPTQ pre-computes the Cholesky decomposition of $\mathbf{H}^{-1}$ once, then reads off the needed inverse entries row by row. This is both faster and more numerically stable, adding a small dampening term $\lambda \mathbf{I}$ to handle near-singular Hessians.

Here is the full GPTQ procedure for one layer. The input is the weight matrix $\mathbf{W}$ and the Hessian $\mathbf{H} = 2\mathbf{X}\mathbf{X}^\top$ computed from calibration data. The output is the quantized matrix $\hat{\mathbf{W}}$.

The headline result: GPTQ quantizes OPT-175B and BLOOM-176B to 4 bits with a perplexity increase of less than 0.05 points on WikiText-2 — compared to round-to-nearest (RTN) which often increases perplexity by 1–2 points or more. At 3 bits, GPTQ still produces usable models where RTN completely fails with perplexity exploding by orders of magnitude.

Crucially, the quality gap between GPTQ and RTN grows with model size. On small models (125M parameters), RTN is competitive. But on OPT-175B, RTN at 4 bits gives a perplexity of 10.54 versus GPTQ's 8.37 (FP16 baseline: 8.34). The larger the model, the more GPTQ matters — precisely the regime where compression is most needed.

Beyond accuracy, GPTQ delivers real inference speedups. The authors developed custom GPU kernels that dequantize weights on-the-fly during matrix multiplication. Since autoregressive generation is memory-bandwidth-bound (not compute-bound), loading 4× fewer bytes per weight translates directly to faster generation.

The results: approximately 3.25× speedup on NVIDIA A100 and 4.5× on NVIDIA A6000 GPUs for end-to-end generative inference. More importantly, a 4-bit OPT-175B fits entirely in the 80 GB of a single A100 — the first time a 175-billion-parameter model ran on one GPU for generative tasks.

GPTQ quantizes weights only — activations remain in FP16. For most generative scenarios this is fine because the bottleneck is weight loading, not activation computation. But for batch inference with many simultaneous requests, activation quantization becomes important too, and GPTQ does not address this.

The method also does not accelerate the actual multiply operations because mainstream hardware lacks native support for mixed-precision formats like FP16 × INT4. The speedup comes entirely from reduced memory traffic. Finally, at 2-bit and ternary quantization, accuracy degrades noticeably — GPTQ pushes boundaries but does not eliminate the accuracy-compression tradeoff.