Multimodal embeddings enable flexible product clustering by instructions
CIM-Embed: Learning Instruction-Conditioned Multimodal Embeddings for Clustering
Artificial IntelligenceMachine Learning
Summary
Sometimes the same set of items can be grouped in different ways depending on what you want to find—for example, grouping products by color or by material. The authors propose a new method called CIM-Embed that creates special data summaries (embeddings) for each product based on instructions, like "group by color." This approach lets a computer organize items into groups without knowing in advance the number of groups. They tested their method on a large Amazon product dataset and showed it works much better than earlier general methods.
What this means in practice
- •For e-commerce catalog managers: Automatically create product groups based on flexible instructions like color or material to improve customer browsing experiences.$Commercial implications: Enables advanced, customizable product grouping features for e-commerce platforms to increase user engagement and sales.
- •For multimodal data engineers: Build embedding models that incorporate images, text, and instructions to produce task-specific data clusters without predefined group numbers.
Authors
Yinghua Xu, Pengyang Xie, Yanxiang Zhou, Chuanzhi Tong, Xiao Xiao, Chaoqun Feng, Xiaoqing Wang
Abstract
Multimodal objects can require different partitions under different instructions, as when products in an e-commerce catalog are grouped by color, material, or both. Retrieval-oriented embeddings may not recover these partitions, while generative clustering often relies on intermediate descriptions or iterative decisions. This paper presents CIM-Embed, a framework for learning instruction-conditioned multimodal embeddings for clustering. Supervised contrastive learning uses same-group positives and negatives from other groups, optionally including other tasks; heterogeneous structured batches accommodate tasks with different group counts. At inference, the model maps each object's image, text, and instruction directly to an embedding; HDBSCAN then forms a partition without the ground-truth cluster count. We construct ProductMIC from product metadata in Amazon Reviews 2023, comprising training data and an evaluation benchmark for instruction-conditioned clustering. The benchmark contains 5,373 products and 449 single-attribute and compositional grouping tasks spanning 50 leaf categories excluded from our training. With full-dimensional embeddings and HDBSCAN, direct within-task SupCon training achieves a mean per-task V-measure of 95.01, compared with 53.06 for WeMM 9B, the strongest general-purpose embedding baseline evaluated. Frozen product-trained encoders also improve out-of-domain conditional grouping, while controlled transfer experiments distinguish partition transfer from instruction response. We will release the code and dataset upon acceptance.