Slim models speed up online recommendation predictions with little accuracy loss
Slim Foundation Models for Latency-Critical Recommendation Serving
Information RetrievalArtificial Intelligence
Summary
Online systems that suggest things to click on, like ads or products, rely on predicting how likely you are to click something. Big language models help these predictions but are too slow for real-time use. The authors created a smaller, faster version of such a model that keeps almost all the accuracy while making predictions up to 3.4 times faster. They do this by cutting the big model down, tuning it efficiently, and cleverly combining its output with regular recommendation software.
What this means in practice
- •For online advertisers: Run faster real-time ad click prediction systems using smaller language models without losing accuracy.$Commercial implications: Enables offering efficient online CTR prediction services that reduce latency and improve ad serving user experience.
- •For ecommerce platform engineers: Integrate lightweight semantic embeddings from compressed language models to enhance product recommendation speed and relevance.
Authors
Danyal Aftab, Anas Zafar, Assadig Abakr, Steven Davy
Abstract
Click-through rate (CTR) prediction is a core component of recommendation and advertising systems. Large language models (LLMs) offer rich semantic priors that improve prediction quality, particularly in cold-start and long-tail settings. Injecting a full-scale LLM into a CTR pipeline is, however, prohibitively expensive at inference time: existing LLM-enhanced CTR methods avoid this cost by caching LLM outputs offline rather than serving the LLM online. We propose a resource-efficient pipeline, SlimFM, that makes online LLM-based semantic modeling practical: (1) dependency-aware structured pruning compresses LLaMA3-8B by 44% to a 4.5B-parameter backbone; (2) LoRA-based field-aware fine-tuning trains the pruned model to produce discriminative field-level semantic embeddings from lightweight textual prompts; (3) a semantic-guided fusion module injects these embeddings into the feature interactions of a conventional CTR model. Across four datasets and six CTR backbones, SlimFM retains over 99% of uncompressed-model accuracy while reducing inference latency by up to 3.4x relative to existing LLM-enhanced baselines.