LMMs-Eval

Use it when

• You need to benchmark a vision, video, or audio language model on standard suites like MMMU, MME, or VideoMME without hand-wiring each dataset and its post-processing
• Two runs of the same multimodal benchmark produce different numbers and you need one reproducible pipeline with confidence intervals and paired comparisons

What it solves

Not the fit when

• custom rubric-based grading of text-only chat quality
• production monitoring of deployed models
• the HTTP eval server has no built-in auth and is for trusted networks only
• caption benchmarks like coco require Java 1.8 for pycocoeval
• live production llm observability
• llm output guardrails at inference time
• prompt-level regression testing for app prompts
• fine-tuning pipeline management

Install

git clone https://github.com/EvolvingLMMs-Lab/lmms-eval.git && cd lmms-eval && uv pip install -e ".[all]"

Invoke

python -m lmms_eval --model qwen2_5_vl --model_args pretrained=Qwen/Qwen2.5-VL-3B-Instruct --tasks mme --batch_size 1 --limit 8

Alternatives

No reviewed alternatives recorded yet.