GuideLLM

Use it when

• I need to load test an OpenAI-compatible LLM endpoint and find its maximum sustainable rate.
• I need TTFT, inter-token latency, throughput, and error metrics for an inference deployment.

What it solves

Not the fit when

• not an inference server or model host
• not for judging answer quality or replacing production monitoring
• credentials and usage costs for the benchmarked endpoint remain the operator's responsibility
• hosting an inference server
• model quality evaluation
• production monitoring

Install

pip install guidellm[recommended]

Invoke

guidellm run --backend kind=openai_http,target=http://localhost:8000 --profile kind=sweep --constraint kind=max_duration,seconds=30 --data kind=synthetic_text,prompt_tokens=256,output_tokens=128

Alternatives

No reviewed alternatives recorded yet.