MTEB-PT: A Text Embedding Benchmark for Brazilian Portuguese
2026-07-06 • Computation and Language
Computation and LanguageInformation RetrievalMachine Learning
AI summaryⓘ
The authors created MTEB-PT, a new benchmark with 22 Brazilian-Portuguese language tasks, to evaluate text embedding models without relying on translations. They tested 93 models of various sizes, including open-source and commercial ones, and found that several top models perform similarly well. Notably, some open, self-hosted models do as well as commercial ones, showing good Portuguese embeddings don't need to come from big companies. They also observed that how models rank on multilingual tests doesn't always predict their Portuguese performance. The authors provide their data and code publicly for others to use.
text embeddingsbenchmarkBrazilian Portuguesesemantic textual similaritymultilingual modelsmodel evaluationconfidence intervalsItem Response Theoryopen-source modelsranking correlation
Authors
Tardelli Ronan Coelho Stekel
Abstract
Text embeddings for Portuguese have no dedicated benchmark: evaluation rests on translated corpora such as English MS MARCO or on thin multilingual coverage, with native tasks scattered and unconsolidated. We introduce MTEB-PT, a benchmark of 22 native Brazilian-Portuguese tasks across seven categories (classification, multilabel classification, pair classification, semantic textual similarity, clustering, retrieval, and reranking), admitting only data created or found in Portuguese and excluding translations by construction. We evaluate 93 models spanning 23M to 27B parameters: 73 open-weight and 20 closed commercial APIs. Alongside the leaderboard we report a statistical layer for every headline comparison: per-task bootstrap confidence intervals, paired-bootstrap significance, a task- and instance-level discrimination analysis (how sharply each task separates models) adapted from Item Response Theory, and a cross-leaderboard correlation. Three findings stand out. The benchmark cleanly separates about a dozen tiers of models, though the top six are statistically too close to order. An openly licensed, self-hostable model reaches that leading tier, so strong Portuguese embedding quality does not require a commercial API. And a model's rank on the global multilingual leaderboard predicts its Portuguese rank only moderately (Spearman rho = 0.75 over 55 shared models; one model ranks 3rd there and 49th here), so a native benchmark measures something the multilingual boards do not. We release every task, our code, and a public leaderboard, so practitioners can choose Portuguese embedding models on native evidence.