vllm

Use it when

• high throughput local model serving
• self-host LLM inference server offline

What it solves

Not the fit when

• runs well without GPU for large models

Install

pip install vllm  # GPU recommended

Invoke

vllm serve MODEL; OpenAI-compatible API for high throughput self-host to dodge provider rate limits.

Alternatives

No reviewed alternatives recorded yet.