Vision language model inputs can hide exact label-flipping errors
Certified Interface Aliases: Exact Collisions in Vision-Language Preprocessing, and When They Exist
Computer Vision and Pattern Recognition
Summary
When computers try to understand pictures and language together, they first shrink and change the picture before analyzing it. This shrinking can sometimes hide important details, making the computer think the picture means something different than it really does. The authors created a way to show exactly when and how this hiding happens, proving certain errors can't be fixed by just trying harder with language. They tested their method on different systems and found it helps identify when extra reasoning won't improve accuracy.
What this means in practice
- •For ai system engineers: Identify exact cases where image preprocessing destroys crucial visual information, improving debugging of vision-language verifiers and routers.
- •For computer vision developers: Screen preprocessing configurations to certify when label-flipping collisions can or cannot occur, avoiding wasted computation in multi-modal models.
Authors
Mert Onur Cakiroglu, Elham Buxton, Mehmet Dalkilic, Hasan Kurban
Abstract
Vision-language verifiers and routers must distinguish errors repairable by more reasoning from those caused by visual evidence never reaching the language model. This distinction lacks ground truth because annotators see full-resolution images while models receive preprocessed tensors. We introduce AliasForge to create cases where the relevant fact is provably absent from the interface. Fixed-point resampling makes the pre-rounding resize an exact integer linear map that can send nonzero integer perturbations to zero. Hiding a label-flipping perturbation there produces images with opposite step-correctness labels but bit-identical interface states. Every verifier therefore has the same output law on both members, giving pair-balanced accuracy exactly one half and zero gain from language-side repair. We prove that every fixed-point downscaler has such null vectors and bound their smallest size at the most common ratios, which rules out an 8-bit fit whenever the bound exceeds 255. From the resize configuration alone, a lattice criterion supplies realizable collisions and certifies their absence within the specified construction family. It resolves all 20 screened configurations, 17 as constructible and 3 as non-constructible. We construct certified pairs across three architectures and certify four additional processors, with zero decision-logit gap on all 18 scored pairs and none of the 18 controls. The pairs also screen routers that waste computation on re-attention or further reasoning. On natural items, per-item routing headroom exists, but no tested interface-only router improves over stopping. Our fiber ceiling bounds the headroom recoverable from the interface. Code: https://github.com/KurbanIntelligenceLab/aliasforge.