Multilingual product matching benchmark challenges existing methods
Billiger.de Products: A Bilingual Entity Matching Benchmark
Computation and Language
Summary
Product matching helps systems decide if two product listings are actually for the same item, which can be tricky with multiple languages and diverse product types. The authors created a new benchmark dataset featuring German and English product listings across 13 different categories including harder ones like clothing and furniture. Testing several matching algorithms and a large language model shows that this new dataset is more difficult than earlier English-only benchmarks. The results also reveal that some models work better on English data, while others handle both languages similarly well.
What this means in practice
- •For e-commerce engineers: Test product matching systems on a bilingual, multi-category dataset to improve matching accuracy across German and English product listings.
- •For machine learning developers: Use the benchmark to evaluate and enhance multilingual entity matching models, especially for challenging categories with many corner cases.
Authors
Aaron Steiner, Ksenia Elagin, Ralph Peeters, Johannes Knopp, Christian Bizer
Abstract
Existing product matching benchmarks primarily contain English-language product data and are often dominated by a single product category, such as electronics. This paper introduces Billiger.de Products, a bilingual German and English entity matching benchmark covering thirteen consumer product categories, including difficult-to-handle categories such as clothing and furniture. The benchmark data originates from the German price comparison platform billiger.de. Following the design of WDC Products, the benchmark offers multiple variants that differ in the fraction of corner cases, the size of the development set, and the fraction of entities unseen during training. An aligned English translation of every offer keeps all pairs, splits, and labels fixed, while cross-language test sets combine German and English records within individual pairs. We validate the benchmark using six supervised matchers and zero-shot GPT-5.2 on both language versions and the cross-language test sets. The validation shows the difficulty of the benchmark. The comparison of the results on the English version of the benchmark to the results on the German version shows that most matchers score on average higher on the English version. The difference is largest for RoBERTa and HierGAT, while the zero-shot LLM runs are largely insensitive to the language. Comparing the F1 scores achieved by PLM-based matchers on the English version of Billiger.de Products with their performance on existing English-language benchmarks, such as WDC Products and Abt-Buy, shows that Billiger.de Products is more difficult than these benchmarks.