Slim models speed up online recommendation predictions with little accuracy loss

Slim Foundation Models for Latency-Critical Recommendation Serving

Information RetrievalArtificial Intelligence

Summary

Online systems that suggest things to click on, like ads or products, rely on predicting how likely you are to click something. Big language models help these predictions but are too slow for real-time use. The authors created a smaller, faster version of such a model that keeps almost all the accuracy while making predictions up to 3.4 times faster. They do this by cutting the big model down, tuning it efficiently, and cleverly combining its output with regular recommendation software.

What this means in practice

  • For online advertisers: Run faster real-time ad click prediction systems using smaller language models without losing accuracy.$Commercial implications: Enables offering efficient online CTR prediction services that reduce latency and improve ad serving user experience.
  • For ecommerce platform engineers: Integrate lightweight semantic embeddings from compressed language models to enhance product recommendation speed and relevance.

Authors

Danyal Aftab, Anas Zafar, Assadig Abakr, Steven Davy

Abstract

Click-through rate (CTR) prediction is a core component of recommendation and advertising systems. Large language models (LLMs) offer rich semantic priors that improve prediction quality, particularly in cold-start and long-tail settings. Injecting a full-scale LLM into a CTR pipeline is, however, prohibitively expensive at inference time: existing LLM-enhanced CTR methods avoid this cost by caching LLM outputs offline rather than serving the LLM online. We propose a resource-efficient pipeline, SlimFM, that makes online LLM-based semantic modeling practical: (1) dependency-aware structured pruning compresses LLaMA3-8B by 44% to a 4.5B-parameter backbone; (2) LoRA-based field-aware fine-tuning trains the pruned model to produce discriminative field-level semantic embeddings from lightweight textual prompts; (3) a semantic-guided fusion module injects these embeddings into the feature interactions of a conventional CTR model. Across four datasets and six CTR backbones, SlimFM retains over 99% of uncompressed-model accuracy while reducing inference latency by up to 3.4x relative to existing LLM-enhanced baselines.