Object size mismatch causes failures in instance image retrieval
Beyond Resolution: Object-to-Image Ratio Mismatch in Instance Retrieval
Computer Vision and Pattern RecognitionInformation Retrieval
Summary
Images of the same object can look very different depending on how much of the image it fills. The researchers found that mismatches in the size of the object relative to the image usually cause retrieval systems to fail, more than just low image resolution. They tested this on many objects at different distances and showed that adjusting for object size or fine-tuning models can improve retrieval accuracy significantly. Their methods do not require changing the stored images, making improvements easier to deploy.
What this means in practice
- •For image search developers: Improve object retrieval accuracy by adjusting queries for object size differences without changing stored image databases.
- •For e-commerce platform engineers: Enhance product image search by accounting for varying object sizes in user queries and catalog images to reduce mismatches.$Commercial implications: Enables better product matching and search ranking for online marketplaces, improving customer experience and sales.
Authors
Boaz Meivar, Ofir Kedem, Amit Edenzon, Gal Chechik, Shai Avidan
Abstract
Visual instance retrieval often fails when the same object appears at different apparent sizes in the query and gallery. We show that the dominant cause is usually not resolution loss but object-to-image (O2I) ratio mismatch: the object occupies different fractions of the two images. On a controlled benchmark of 3,021 Objaverse objects rendered at five camera distances, more than 80% of the cross-distance degradation is attributable to O2I mismatch rather than resolution for 9 of 12 pretrained backbones; multi-scale architectures cut the resolution-only effect to single digits yet remain equally susceptible. The failure is also asymmetric: tight queries retrieve more reliably against wide gallery images than the reverse. Guided by this analysis, query-side scale augmentation and an OWLv2 crop reranker reach state of the art on ILIAS 100M (29.2 mAP@1000 before reranking, 42.0 after) without training or modifying the precomputed gallery index, and a LoRA fine-tune matches the query-side gains at a single forward pass, showing that O2I robustness is learnable.