Vision language model improves object box precision and speed
LOCUS: Locate Once, Check Until Settled. Draft-and-Verify Visual Grounding with a Box Head and Coordinate-Token Verification
Computer Vision and Pattern Recognition
Summary
Finding the exact location of an object described by words is tricky for vision-language models. The authors show that combining two different ways of finding boxes—a fast guess and a careful check—makes the model both more accurate and quicker at getting exact boundaries. Their method repeats just a few times to improve precision without slowing down too much. This new approach performs better than many existing models on common object location tests.
What this means in practice
- •For mobile app developers: Use improved object bounding box precision and faster visual grounding to enhance real-time image search or augmented reality features in mobile apps.
- •For autonomous vehicle teams: Improve object localization accuracy in vehicle perception systems by integrating a combined drafting and verification process for bounding box predictions.
Authors
Muhammad Talha
Abstract
Vision-language models (VLMs) ground a referring expression in one of two ways. They decode the box as coordinate tokens, which is precise but takes one sequential step per token, or they regress it from a hidden state with a small head, which takes one step but loses accuracy at high IoU. The two fail differently: a head usually finds the right object but places its edges loosely, while coordinate tokens place edges precisely. We therefore use the regression head as a speculative drafter for the model's own coordinate tokens. Locus writes the head's box as coordinate tokens and checks them with the same backbone by Jacobi iteration over a shared image prefix. The fixed point is the model's greedy coordinate-token answer, reached in about 5.60 sequential passes on average instead of 18.46 decoding steps; head and verifier are one set of weights, trained jointly. On the 8 RefCOCO, RefCOCO+ and RefCOCOg splits, verification raises Locus-2B's Acc@0.9 from 54.87 for its own head to 59.13, and the one-pass head is ahead at Acc@0.5 of every published model with at most 3B parameters except Florence-2-L. Against Jag, the strongest open regression-head model, at its own size and training budget, Locus ties at Acc@0.5 and, with typical acceptance, gains +4.52 points at Acc@0.9, with paired significance tests. Exact answers come 1.5x faster than vLLM greedy-decoding the same coordinate tokens with the same weights; a pure regression head remains faster still.