LM Evaluation Harness

Use it when

• You fine-tuned or pretrained a model and need its scores on standard academic benchmarks (MMLU, HellaSwag, GSM8K) reported the same way papers and leaderboards report them
• You must compare several checkpoints or an API-hosted model against open models using identical public prompts and metrics so the numbers are actually comparable

What it solves

Not the fit when

• Multi-node evaluation through the built-in Hugging Face backend
• Chat APIs without logprobs for loglikelihood or multiple-choice tasks; those APIs are limited to generation-style tasks
• Broad multimodal evaluation; the README describes multimodal support as a prototype and points to lmms-eval
• Assuming hosted API evaluation is free; requests are billed and rate-limited by the selected provider
• Directly executing untrusted benchmark code; use the documented prediction-only path where applicable
• runtime output validation of LLM responses
• prompt regression testing for a production app
• multimodal benchmark coverage beyond prototype tasks
• multi-node distributed evaluation
• serving or hosting models

Install

git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness && cd lm-evaluation-harness && pip install -e . && pip install "lm_eval[hf]"

Invoke

Use the current CLI shape: `lm-eval ls tasks`, then `lm-eval run --model hf --model_args pretrained=<hf-model-id> --tasks hellaswag,mmlu --device cuda:0 --batch_size auto`. Install the backend extra that matches the target (`lm_eval[hf]`, `lm_eval[vllm]`, or `lm_eval[api]`); hosted API evaluations incur the provider's charges and capability limits.

Alternatives

No reviewed alternatives recorded yet.