LLM Compressor

Use it when

• Your model is too large for available GPU VRAM and you need to quantize weights, activations, or KV cache before serving with vLLM
• You want to apply GPTQ, AWQ, SmoothQuant, FP8, or NVFP4 quantization to a Hugging Face model and save it in the compressed-tensors format vLLM can load

What it solves

Not the fit when

• Not an inference server; compressed checkpoints are served by vLLM
• Compression runs need GPU hardware and calibration passes for some algorithms
• model training from scratch
• serving inference itself
• cloud API cost reduction
• fine-tuning workflow management

Install

pip install llmcompressor

Invoke

from llmcompressor import oneshot; from llmcompressor.modifiers.quantization import QuantizationModifier; oneshot(model=model, recipe=QuantizationModifier(targets="Linear", scheme="FP8_BLOCK", ignore=["lm_head"])); model.save_pretrained(SAVE_DIR)  # then load in vLLM

Alternatives

No reviewed alternatives recorded yet.