Jag improves visual grounding with fast direct box prediction
Jag: Direct Box Prediction for Efficient Visual Grounding
Computer Vision and Pattern Recognition
Summary
Finding a specific object in a picture based on a description can be slow if computers guess box corners one by one. The authors show how their method, called Jag, predicts all box corners at once using a trained model, making the process faster and more accurate. Jag improves the model’s understanding of both the language and the image by training them together with the box prediction. This lets systems find objects precisely in one step, saving time and computing power.
What this means in practice
- •For computer vision developers: Speed up locating objects in images from language descriptions by using a direct box-predicting model.
- •For augmented reality engineers: Improve real-time object selection based on verbal cues with lower delay and hardware requirements.
Authors
Xiuyuan Zhu, Ke Lu, Hao Wu, Zijin Du, Dongming Zhang, Jian Xue
Abstract
Visual grounding combines image--language understanding with geometric prediction, yet autoregressive multimodal models often express a box as a coordinate sequence. Sequential decoding adds computation to the multimodal processing needed for localization. Adapting these representations to box geometry offers a route to direct prediction. We present Jag, which adapts a compact pretrained multimodal model for direct continuous box prediction. A lightweight regression head reads the last valid input-token state, conditioned on the image and referring expression, and predicts all four box coordinates together. Geometric supervision jointly adapts the language backbone, visual merger, and regression head, allowing localization losses to shape both the multimodal representations and the coordinate mapping. Inference retains image--language processing and returns the box in one forward pass, without coordinate generation or text parsing. Across benchmarks, Jag consistently improves grounding accuracy over its base model across the observed target sizes and expression groups. Component ablation show that jointly adapting the representations and head substantially improves accuracy over training the head alone. Against existing specialized grounding models, it achieves competitive accuracy with lower single-request latency and GPU memory use. These results show how supervision tailored to the geometric answer can specialize a compact pretrained multimodal model for accurate grounding and efficient direct prediction. Project page: https://xyzzzh.github.io/Jag/