AirLLM

Use it when

• A model you want to run locally is far larger than your GPU VRAM, for example 70B on a 4GB card, and you do not want to shrink it with quantization, distillation, or pruning
• You keep hitting CUDA out-of-memory errors loading a large model with transformers and need layer-by-layer streaming so only one layer occupies the GPU at a time

What it solves

Not the fit when

• production serving where tokens-per-second matters (disk streaming trades speed for memory)
• machines without substantial free disk space (models are decomposed and saved layer-wise)
• non-Apple-silicon Macs
• high-throughput production serving
• low-latency chat inference
• OpenAI-compatible HTTP endpoint
• multi-model API routing

Install

pip install airllm (optional: pip install -U bitsandbytes for 4bit/8bit block-wise compression; mlx and torch on Apple silicon for MacOS)

Invoke

from airllm import AutoModel; model = AutoModel.from_pretrained("<hf-repo-id>"); model.generate(input_ids.cuda(), ...) -- optionally compression='4bit' for up to 3x speedup

Alternatives

No reviewed alternatives recorded yet.