Visual localization speeds up by caching image processing steps
LateMatch: Breaking the Interaction Bottleneck via Cacheable Matching for Efficient Visual Localization
Computer Vision and Pattern RecognitionRobotics
Summary
Visual localization helps devices like robots know exactly where they are by matching pictures they see to a big set of known images. The authors found that current deep learning methods do this by repeating lots of heavy calculations many times, making it slow and costly on small devices. They introduced LateMatch, a way to do most of the work once ahead of time and only do a small part live when the robot sees a new image. This cuts the effort needed in half and can be even more efficient with a technique called E-SToRE, which smartly reduces image details before matching with hardly any loss in accuracy.
What this means in practice
- •For robotics developers: Improve robot navigation systems to run efficient image matching for location finding on limited hardware.
- •For mobile device engineers: Enable faster and less power-hungry visual localization on smartphones and AR glasses by offloading part of image processing.
Authors
Mohammad Omama, Aeyan Ashraf, Sandeep Chinchali
Abstract
Visual localization is critical for embodied agents, but deploying state-of-the-art deep feature matchers on edge devices is fundamentally limited by their massive computational cost. Standard deep matchers tightly interleave intra-image context building and inter-image matching at every layer, creating a severe interaction bottleneck. Localizing a single query against K database images forces the system to execute the full network K independent times with no capacity for caching. To break this bottleneck, we advocate for a paradigm shift toward cacheable image matching. We realize this paradigm with LateMatch, a versatile approach that restructures existing matchers by deferring inter-image interaction to their final layers. By isolating intra-image processing into offline-cacheable Context Blocks, only a small set of Interaction Blocks must be executed online. To further optimize LateMatch, we introduce E-SToRE (Entropy-guided Spatial Token Reduction and Expansion). E-SToRE leverages the decreasing Shannon entropy of the late Interaction Blocks to dynamically reduce token counts just prior to cross-image interaction. Evaluated across HPatches, ScanNet, MegaDepth, ETH-3D, and Aachen, LateMatch reduces effective computational cost by over 2x compared to standard deep matchers, while E-SToRE yields an additional 10% to 14% savings with negligible degradation in accuracy.