AI summaryⓘ
The authors study how multimodal large language models learn from example image-text pairs during inference, a process called multimodal in-context learning (ICL). They note that this method can be slow and unstable because it depends heavily on how examples are presented. To fix this, the authors propose Hyper-ICL, which trains a lightweight adapter to mimic the effects of these examples without needing them at inference. Their approach also adjusts how much it adapts based on the current input and uses a special loss to align internal features with a demonstration-informed teacher model. Experiments show Hyper-ICL improves accuracy and consistency across several multimodal tasks compared to traditional methods.
Multimodal In-Context LearningMultimodal Large Language ModelsIn-Context DemonstrationsInference LatencyAttention MechanismLow-Rank AdapterQuery-Adaptive ModulationHyperbolic Anchor DistillationLorentz Geodesic DistanceVQAv2
Authors
Niloufar Alipour Talemi, Hossein Kashiani, Fatemeh Afghah
Abstract
Multimodal In-Context Learning (ICL) has emerged as a practical inference paradigm for Multimodal Large Language Models, where a small set of interleaved image-text In-Context Demonstrations (ICDs) conditions the model to solve new tasks. Despite its flexibility, multimodal ICL incurs high inference latency and suffers from instability due to sensitivity to demonstration formatting, ordering, and content. To address these limitations, we propose Hyper-ICL, a lightweight, training-based framework for demonstration-free multimodal ICL that reconstructs demonstration effects directly without requiring ICDs at inference time. Hyper-ICL learns a parameter-efficient low-rank logit-level adapter that calibrates attention distributions to better match demonstration-induced attention redistribution. To capture how demonstration influence varies across queries, we introduce a query-adaptive modulation mechanism that adaptively controls intervention strength at token level across layers and heads based on the current query. Finally, we propose a layer-wise hyperbolic anchor distillation loss that aligns intermediate student features to a demonstration-conditioned teacher via Lorentz geodesic distance. This loss encourages the student to reconstruct the demonstration-query relationships induced by ICDs. Extensive experiments across six different multimodal benchmarks (including VQAv2, OK-VQA, and COCO Caption) demonstrate that Hyper-ICL consistently improves accuracy and stability over vanilla ICL and existing state-of-the-art methods.