Small model predicts which images aid faster and accurate answers
Decide Before You Look: Learning Which Retrieved Memories Deserve Pixels
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Answering questions with many images can be slow because looking at each full image uses a lot of computer power. The authors show that usually only one or two images really help with the answer. They created PixelTriage, a small model that looks at tiny picture previews and some text to decide which images are worth loading in full detail. This method can speed up answering by almost three times without losing accuracy and works well across different memory systems and models.
What this means in practice
- •For mobile app developers: Reduce processing time and cost by selectively loading only necessary images for AI assistants that answer user questions with visual memories.
- •For digital archive managers: Speed up retrieval systems accessing large image databases by prioritizing which images to fully decode before answering queries.
Authors
Youxing LI
Abstract
Multimodal assistants answer questions from long-term memories that contain images. After retrieval, each retrieved image reaches the answering model either as pixels, at about a thousand visual tokens per image, or as a stored text proxy that often misses the detail the question asks about. We find that the benefit of pixels usually comes from one or two retrieved memories, and that it can be predicted before the answering model runs, without reading any full-resolution image. In PixelTriage, a plug-in placed after retrieval, a small model that does not generate text reads the dialogue, a short note and a thumbnail of each retrieved memory and predicts how much its pixels would add. It is trained on synthetic memory episodes labeled by a frozen 27B model that answers each question with and without each memory's pixels. With a 7B answering model, PixelTriage lies on the accuracy--cost frontier of M$^3$Exam, DMV and MemEye and uses 11--23\% of the visual tokens without a significant loss of accuracy. On DMV it answers 2.9 times faster than opening all images. It outperforms retrieval order and uniform down-sizing at equal budgets and transfers to other memory systems and to a 397B answering model.