Model improves video object tagging detection and captioning with test time tricks
CapMap-MS-TTA: 3rd Place Solution for the MUMU Track of the 8th LSVOS Challenge at ECCV 2026
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
The paper addresses the challenge of making one model do three different video analysis jobs—tagging images, spotting objects with any label, and writing English descriptions—while keeping the model small and memory-light. The authors built on an existing model called Florence-2 and added some clever steps without retraining it, such as mapping caption words to official tags and testing multiple image sizes and flips. This approach boosted the model's performance and earned them third place in a competition. Their method shows it’s possible to improve complex tasks by smart adjustments rather than heavy retraining.
What this means in practice
- •For video analytics developers: Create single models that perform tagging, object detection, and captioning efficiently under strict memory limits using training-free techniques.
- •For content moderation teams: Use unified multimodal models to automatically label and describe video content on platforms with limited computational resources.
Authors
Chengfeng Qiu, Kaifeng Wei
Abstract
The MUMU track of the 8th Large-scale Video Object Segmentation (LSVOS) Challenge requires a single unified multimodal model to jointly solve image tagging (Task A), open-vocabulary object detection (Task B), and English captioning (Task C) under strict resource constraints (<=0.5B parameters and <=8 GB peak GPU memory). We present CapMap-MS-TTA, a training-free submission built on Microsoft Florence-2-base (~231M parameters), combining caption keyword mapping with multi-scale flip test-time augmentation. Task C uses the native <DETAILED_CAPTION> pathway with length/token sanitization. Task A maps the same detailed caption into the official quality/scene/event vocabularies via an expanded keyword lexicon with whole-word matching and a lightweight expand-hints stage. Task B runs Florence-2 open detection (<OD>) with multi-scale and horizontal-flip test-time augmentation (TTA), followed by label-aware non-maximum suppression (NMS). Without fine-tuning, the system improves our reproduced Florence-2 baseline from 15.16 to a best public score of 16.4815, and ranks 3rd on the final MUMU leaderboard.