AI summaryⓘ
The authors study a problem in Vision Transformers where small but important image regions (called boundary minorities) are overlooked when summarizing visual information using simple averages. They show mathematically why these small regions can have a big influence even if their average contribution seems tiny. To fix this, they propose a method called BMFA that adaptively focuses more on these important small regions by refining how image patches are summarized. Experiments on synthetic data and real benchmarks confirm that BMFA improves the representation of critical regions without losing much accuracy. Their approach helps Vision Transformers better capture important details that are usually missed by coarse summarization.
Vision TransformerstokensGibbs massfree energymean summarizationlog-mean-expboundary minorityadaptive screeninghierarchical approximationImageNet
Abstract
Vision Transformers process spatially redundant tokens efficiently only when coarse token summaries preserve the evidence required by exponential attention aggregation. We identify a boundary-minority underestimation failure in which a spatially small, high-response region contributes dominant Gibbs mass while remaining nearly invisible to a block mean. We formalize the failure through the discrepancy between normalized log-mean-exp free energy and mean summarization, prove that minority Gibbs mass can remain non-vanishing as its spatial support and mean contribution vanish, and characterize the limitations of finite-order moment corrections. Building on the resulting analysis, we introduce Boundary-Minority Free-Energy Adaptive Screening (BMFA), which constructs a hierarchical piecewise-constant approximation and recursively refines blocks according to a computable lower-bound increment of local free energy. Controlled synthetic tests, COCO and LVIS diagnostic probes, closed-loop DeiT-Tiny evaluations, and ImageNet-1K experiments establish a consistent evidence chain. BMFA reduces the mean synthetic underestimate from 2.582 to 0.261 at a 5.794% leaf ratio, lowers the COCO image-edge mean gap from 2.254 to 0.526, and preserves 71.520% ImageNet Top-1 accuracy at a 55.861% leaf ratio. The current prototype evaluates selection quality after full QK computation; the reported leaf ratio therefore characterizes representation granularity rather than verified sparse-kernel speedup.