Multilingual encoders struggle to produce consistent semantic codes across languages

Do Multilingual Encoders Produce Language-Consistent Semantic IDs?

Information RetrievalComputation and Language

Summary

Semantic IDs (SIDs) are special codes that represent product information in a compact way, used for searching and retrieving items. This paper examines whether a multilingual encoder can assign consistent SIDs to the same product described in different languages. The authors found that translations often get very different SIDs compared to the English original, meaning the system doesn’t reliably recognize them as the same item across languages. Adjusting training to balance languages or quantizer settings did not solve this issue, showing that more work is needed for language-consistent semantic coding.

What this means in practice

  • For multilingual search engineers: Improve cross-language search by understanding limitations of semantic ID consistency when using multilingual encoders for product retrieval.
  • For e-commerce platform developers: Design product matching systems that do not assume language-consistent semantic codes, avoiding errors in international catalog integration.

Tested on one dataset.

Authors

Abhinav Bohra, Anuj Bohra

Abstract

Semantic IDs (SIDs) compress item embeddings into discrete code sequences used in generative retrieval. We ask whether a multilingual encoder is sufficient for different-language renderings of the same product to receive language-consistent SIDs. Using Amazon ESCI listings rendered in English, Spanish, and Japanese, we test whether translations remain close to their English source, whether residual quantization is unusually sensitive to translation-induced movement, and whether multilingual or language-balanced quantizer fitting improves SID agreement. Multilingual E5 places translations measurably apart: under an English-heavy fit, a Japanese translation preserves the first SID code of its English counterpart in only 7.7% of cases, compared with 89.0% for an English rewording. Distance-matched product-directed controls produce nearly the same full-SID mismatch as translation, providing no evidence that the quantizer selectively amplifies language directions. Balancing the fitting mixture makes codebook use more uniform but further reduces cross-lingual prefix agreement: Spanish first-code consistency falls from 28.3% to 6.6%, while an English-only fit preserves it for 67.6% of Spanish translations. These results show that multilingual exposure and balanced codebook use alone do not guarantee language-consistent SIDs.