MTEB (Massive Text Embedding Benchmark)

Use it when

• You must choose an embedding model for retrieval, classification, or clustering and want reproducible benchmark scores on relevant tasks instead of vendor claims
• You trained or fine-tuned a custom embedding model and need to evaluate it against established baselines or submit it to the public leaderboard

What it solves

Not the fit when

• Does not improve a model; it measures models on benchmark tasks
• Benchmark scores on public tasks may not transfer to your private corpus
• Running large model evaluations requires substantial compute and dataset downloads
• serving embeddings in production
• vector storage and indexing
• end-to-end rag pipeline evaluation
• llm output quality judging

Install

pip install mteb (or uv add mteb)

Invoke

Python: model = mteb.get_model(name); tasks = mteb.get_tasks(tasks=["Banking77Classification.v2"]); mteb.evaluate(model, tasks=tasks). CLI: mteb run -m sentence-transformers/all-MiniLM-L6-v2 -t "Banking77Classification.v2" --output-folder results

Alternatives

No reviewed alternatives recorded yet.