Multimodal learning boosts next place recommendation accuracy
LUMIR: Decoupled Multimodal Representation Learning from Heterogeneous Data for Next-POI Recommendation
Information RetrievalMachine LearningMultimedia
Summary
Predicting where someone will go next can be tricky, especially when information comes from different sources like text, images, or maps. The authors present LUMIR, a way to turn all these different types of information into a shared language-based format. This method then mixes these data in a smart way and uses the order of past visits to improve predictions. Tests show that LUMIR works better than earlier methods and its learned representations can be used with different recommendation systems.
What this means in practice
- •For location-based service developers: Build next-POI recommendation features using reusable multimodal representations that improve accuracy across diverse user data inputs.
- •For travel app engineers: Enhance travel apps by integrating a decoupled fusion method that links multiple data sources for better destination suggestions.
Authors
Zehui Wang, Wolfram Höpken, Dietmar Jannach
Abstract
As multimodal tourism-related information collected from heterogeneous data sources has become increasingly available, learning effective representations has attracted attention for improving recommendation performance. Existing approaches often tightly couple multimodal fusion with the architecture and optimization objective of a specific downstream model, which limits the generality and reusability of the learned representations. To address these limitations, we propose LUMIR, a decoupled multimodal representation learning framework based on contrastive learning. Multimodal information collected from heterogeneous data sources is first semantically unified into a shared textual representation space using large language models (LLMs). An interaction-aware fusion mechanism is then designed to model cross-modal interactions, while sequential signals are incorporated as a structural constraint to guide representation learning. To evaluate the learned representations, experiments are conducted on a multimodal point-of-interest (POI) dataset under the next-POI recommendation setting across multiple sequential recommendation backbones. The results show that LUMIR consistently outperforms existing multimodal fusion baselines in both accuracy-oriented and beyond-accuracy metrics, demonstrating that decoupling representation learning from downstream optimization provides an effective paradigm for learning reusable multimodal representations. The code is publicly available at https://github.com/wzehui/LUMIR.