Choose a method

If your checkpoint is already quantized, the checkpoint decides the method. Load it as shown in Load a pre-quantized checkpoint. Otherwise, pick a method based on your hardware and what you need to do with the model after loading it.

Start from your goal

Each row recommends a first method to try. Check the full comparison table before you commit to one.

Goal	Start with	Why
Fit a model in less memory with the least setup	bitsandbytes	Quantizes to 4 or 8 bits while loading without calibration, runs on CUDA, XPU, HPU, and CPU, and supports saving and fine-tuning.
Fine-tune with QLoRA	bitsandbytes	Supports 4-bit NF4 with PEFT. HQQ, EETQ, GPTQ, SINQ, and Quanto also support fine-tuning.
Speed up inference with `torch.compile`	torchao	`generate` compiles torchao models automatically with a static cache, and torchao has int4, int8, and FP8 schemes for CUDA, XPU, and CPU.
Get the best 4-bit accuracy	GPTQ	Calibrates on a dataset while loading. To use AWQ or AutoRound instead, quantize with their libraries and load the checkpoint.
Run FP8 on Ada, Hopper, or Intel XPU	Fine-grained FP8	Needs CUDA compute capability 8.9 or higher, or an XPU. FBGEMM FP8 needs compute capability 9.0 or higher, or an XPU.
Run FP4 on Blackwell	NVFP4	Quantizes to NVIDIA’s 4-bit float format while loading. MXFP4 runs on compute capability 7.5 or higher, XPU, and CPU.
Run on Apple silicon	Metal	Quantizes to 2, 4, or 8 bits on MPS while loading. GGUF files also load on MPS.
Load checkpoints made with llm-compressor	compressed-tensors	Loads FP8, int4, and int8 checkpoints in the compressed-tensors format.
Compress below 4 bits	HQQ or SINQ	Quantize to 2 or 3 bits while loading without calibration. AQLM, VPTQ, and SpQR checkpoints go lower but must be quantized with their own libraries.

Show the full comparison table

Method	`quant_method`	Workflow	Bits	Hardware	Save	Fine-tune	Auto-compile
AQLM	`aqlm`	Pre-quantized only	1–2	CUDA, CPU	✅	✅ with `aqlm>=1.0.2`	❌
AutoRound	`auto-round`	Pre-quantized only	2–8	CUDA, XPU, CPU	✅	❌	❌
AWQ	`awq`	Pre-quantized only	4	CUDA, ROCm, XPU, CPU	✅ except ExLlama backends	✅ with `gptqmodel>=5.0.0`	❌
BitNet	`bitnet`	Pre-quantized only	1.58	CUDA	✅	Only `autobitlinear` in `online` mode	❌
bitsandbytes	`bitsandbytes`	While loading	4, 8	CUDA, XPU, HPU, CPU	✅	✅	❌
compressed-tensors	`compressed-tensors`	Pre-quantized only	Set by checkpoint	CUDA, XPU, MPS, CPU	✅	✅ except with FP8 kernels	✅
EETQ	`eetq`	While loading	8	CUDA	✅	✅	❌
FBGEMM FP8	`fbgemm_fp8`	While loading	8	CUDA (compute capability 9.0+), XPU	✅	❌	❌
Fine-grained FP8	`fp8`, `mxfp8`	While loading	8	CUDA (compute capability 8.9+), XPU	✅	❌	✅
Four Over Six	`fouroversix`	While loading	4	CUDA (fastest on Blackwell), CPU	✅	Only with `keep_master_weights=True`	❌
FP-Quant	`fp_quant`	While loading	4	CUDA, XPU	✅	Only with `store_master_weights=True`	❌
GGUF	`gguf`	Pre-quantized only	1–8	CUDA, MPS, XPU, CPU	❌	❌	✅
GPTQ	`gptq`	While loading, with calibration	2, 3, 4, 8	CUDA, ROCm, XPU, CPU	✅	✅	❌
HIGGS	`higgs`	While loading	2, 3, 4	CUDA	✅	❌	❌
HQQ	`hqq`	While loading	1, 2, 3, 4, 8	CUDA, XPU	✅	✅	❌
Metal	`metal`	While loading	2, 4, 8	MPS	✅	❌	❌
MXFP4	`mxfp4`	While loading	4	CUDA (compute capability 7.5+), XPU, CPU	✅	❌	❌
NVFP4	`nvfp4`	While loading	4	One Blackwell GPU	❌	❌	✅
Quanto	`quanto`	While loading	2, 4, 8	CUDA, MPS, XPU, CPU	❌	✅	❌
Quark	`quark`	Pre-quantized only	Set by checkpoint	CUDA, ROCm, CPU	❌	❌	❌
SINQ	`sinq`	While loading	2, 3, 4, 6, 8	One CUDA GPU, CPU	✅	✅	❌
SpQR	`spqr`	Pre-quantized only	3	CUDA	✅	❌	❌
torchao	`torchao`	While loading	4, 8	CUDA, XPU, CPU	✅	Only 8-bit schemes	✅
VPTQ	`vptq`	Pre-quantized only	1–8	CUDA	✅	❌	❌

Check memory with the quantization picker

The quantization picker estimates whether a model fits on your hardware at each precision, including the KV cache. Pick a model, your hardware, and what you need the method to do, and the Compare view lists the methods that match.

Benchmarks

The benchmarks below compare several methods on Llama 3.1 8B and 70B. They report accuracy, throughput in tokens per second, peak memory in GB, and quantization time.

Llama 3.1 70B in bf16 ran on 2 NVIDIA A100 80GB GPUs. FP8 methods ran on 1 NVIDIA H100 80GB GPU, and all other methods ran on 1 NVIDIA A100 80GB GPU. Throughput was measured at batch size 1, generating 64 tokens. Results include torch.compile and Marlin kernels where a method supports them.

The results group into three bit ranges.

Bits	Methods benchmarked	Memory vs. bf16	Accuracy
8	bitsandbytes int8, HQQ, Quanto, torchao, FP8	About 2x smaller	Very close to bf16
4	AWQ, GPTQ, HQQ, bitsandbytes NF4, SINQ	About 4x smaller	High. AWQ and GPTQ often lead but need calibration.
Below 4	VPTQ, AQLM, 2-bit GPTQ	More than 4x smaller	Noticeable drop, especially at 2 bits. AQLM and VPTQ take a long time to quantize.

Benchmark accuracy and speed on your own task and hardware before you pick a method for production.

Update on GitHub

Transformers

Choose a method

Start from your goal

Check memory with the quantization picker

Benchmarks