LightEval

Use it when

• You fine-tuned or quantized a model and need to score it on standard benchmarks like MMLU, GSM8K, or GPQA to compare against baselines.
• You want one harness that can evaluate the same task suite across vllm, transformers, sglang, or hosted API endpoints, with sample-by-sample results for debugging.

What it solves

Not the fit when

• Windows; the retained README says it is completely untested and unsupported
• Assuming every backend works from the base install without its optional dependencies
• Assuming Hugging Face inference providers or other hosted endpoints are free
• Evaluating non-LLM systems or classic ML models
• Live production monitoring; it is an offline benchmark harness
• app-level prompt regression testing
• production llm tracing and observability
• human preference or A/B evaluation
• guardrail enforcement at inference time

Install

Install with `pip install lighteval`; install the extra dependencies required by the chosen local or hosted backend. The documented `hf-inference-providers` example also requires `hf auth login` and access to the selected provider/model.

Invoke

Run `lighteval eval "hf-inference-providers/openai/gpt-oss-20b" gpqa:diamond` for the documented hosted-provider example after Hugging Face authentication, or select a local `accelerate`, `vllm`, `sglang`, `nanotron`, TGI, or custom Python backend. Hosted endpoints and inference providers can incur usage charges.

Alternatives

No reviewed alternatives recorded yet.