AI summaryⓘ
The authors present Video2LoRA, a new way to process videos for vision-language models more efficiently. Instead of analyzing every video frame during each query, their method creates a compact adapter that summarizes the whole video in one step. This adapter lets the model answer questions or describe videos without needing to repeatedly look at the video frames, saving a lot of time and computing power. Their tests show that Video2LoRA works just as well as traditional methods on multiple benchmarks and remains effective even with longer videos and higher resolution. Additionally, they found adapters made from different parts of a video can be combined, hinting at handling very long videos more easily.
video summarizationvision-language modelsLow-Rank Adaptation (LoRA)perceiver hypernetworkfrozen modelvideo captioningvideo question answeringtoken efficiencyadapter compositioninference speed
Abstract
Processing video in vision-language models is expensive: each frame occupies hundreds of tokens, and inference cost scales with every frame and every repeated query. We introduce Video2LoRA, a method for parametric video internalization. A perceiver hypernetwork reads the intermediate representations produced layer-by-layer as a frozen VLM encodes a video, and generates a Low-Rank Adaptation (LoRA) adapter in a single forward pass. Unlike standard LoRA fine-tuning, which requires iterative gradient updates, Video2LoRA predicts these weights directly from the video. Trained for SmolVLM2 500M and 2.2B on video summarization and captioning, Video2LoRA enables the same frozen VLM to answer queries from the adapter alone, with zero visual tokens in its context at query time. Video2LoRA is statistically non-inferior and equivalent to direct video-in-context inference across all five captioning benchmarks at both model scales, and across seven of eight video question answering benchmark-scale pairings. Although trained only on 12 frames at 384px, it remains stable up to 1,024 frames and 1024px, where direct video-in-context inference often degenerates. Across this sweep, it reduces answer-time visual-token load by up to 1,500x and query TTFT by 6-80x, while preserving video-faithful outputs. We also find that independently generated adapters for non-overlapping video segments can compose in rank space, suggesting a path toward chunked long-video internalization.