Summary
Traditional online shopping search often struggles when users ask questions in everyday language or use different words than product listings. The authors treat product recommendation as a very large classification problem and teach a transformer model to predict products by learning from simpler TF-IDF similarity rules. Their method reaches high accuracy close to the original TF-IDF approach but works better on natural, conversational queries where keyword matching fails. They also explore how this approach could scale to millions of products and improve real-world e-commerce search systems.
What this means in practice
- •For e-commerce engineers: Improve product search relevance for conversational and intent-driven queries by replacing keyword matching with transformer-based classification models.$Commercial implications: Enables building smarter search engines that better understand user intent and recommend products within large catalogs, offering a competitive advantage.
- •For recommendation system developers: Integrate distilled lexical knowledge into neural models to enhance item-to-item similarity and recommendation quality in retail platforms.
Tested on one dataset.
Abstract
Traditional e-commerce search platforms rely heavily on inverted indices and token-level lexical matching algorithms (e.g., BM25 and TF-IDF), which frequently fail on conversational, intent-driven, or paraphrased user queries -- the classic vocabulary mismatch problem. We formulate conversational product recommendation as an Extreme Multi-Label Classification (XMLC) problem over an e-commerce catalog of N = 54,000 products spanning 27 balanced retail categories from the Amazon Reviews '23 benchmark. Using a pre-trained DistilBERT transformer encoder, we distill dense item-to-item similarity topologies (generated via TF-IDF cosine similarity over cumulative metadata with K = 50 nearest neighbours) into a deep contextual representation via a pseudo-label knowledge distillation framework. Evaluated on an exact 85/15 train/validation split (8,089 held-out products across C = 53,923 output classes) with strict self-exclusion enforced, the DistilBERT neural student achieves P@1 = 93.15%, P@5 = 90.08%, NDCG@10 = 0.8845, and MRR@10 = 0.9545, closely recovering the empirical ceiling established by the corrected TF-IDF teacher (P@1 = 98.10%, NDCG@10 = 0.9419, MRR@10 = 0.9882). Furthermore, a qualitative benchmark across ten structured natural language query archetypes -- encompassing situational, cross-category, paraphrased, and negative-constraint queries -- demonstrates that the transformer student generalises substantially beyond keyword matching, successfully resolving implicit user intent where lexical models fail completely. Finally, we analyse the architectural and memory scalability trade-offs of extreme classification projection layers at industrial catalog scale (> 10^6 items) and present a concrete deployment trajectory toward Dual-Encoder (Two-Tower) vector search. Code: https://github.com/Sunnidhya/Distilling-Lexical-Product-Associations-into-Deep-Transformers.