EvalScope

Use it when

• You need to score an OpenAI-compatible model endpoint or a local model on standard benchmarks like GSM8K, MMLU, or SWE-bench with a single command instead of writing a harness
• You need to stress test an LLM serving endpoint for TTFT, TPOT, and throughput and compare multiple models side by side in a dashboard or arena ranking

What it solves

Not the fit when

• Evaluating your own application's prompts against custom business assertions (it centers on standard benchmarks; custom datasets need extra configuration)
• API costs still accrue when the evaluated model is a paid endpoint
• Docker required for sandboxed agent and code benchmarks
• application-level prompt regression testing
• web UI regression testing
• production request tracing and observability

Install

pip install evalscope

Invoke

evalscope eval --model your-model-name --api-url $OPENAI_API_BASE_URL --api-key $OPENAI_API_KEY --eval-type openai_api --datasets gsm8k arc --limit 5

Alternatives

No reviewed alternatives recorded yet.