Text aligned vision model improves fine detail puzzle reasoning and segmentation
TDDN: Text-aligned Diffused DINO Network for Puzzle Understanding
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Understanding images like puzzles needs very detailed visual perception, which existing vision language models often miss because they focus more on general meaning than fine details. The authors combined two different image representations into a new model called TDDN, which aligns better with language and captures fine visual details. This model matches or beats existing ones in finding good matches between images and text, and it is much better at image segmentation. They also created a new dataset to test puzzle-like image understanding, where their model performs twice as well as previous methods.
What this means in practice
- •For computer vision engineers: Integrate TDDN to enhance image segmentation and detailed visual reasoning in applications handling complex scenes or puzzles.
- •For natural language processing developers: Use TDDN's aligned text-image representations to improve accuracy in systems connecting language queries with precise image content.
Authors
Harsha Patnala, Debopriyo Banerjee, Ayush Sunil Munot, Somak Aditya
Abstract
Structured visual reasoning, such as image puzzles, demands fine-grained visual perception, an ability current Vision Language Models (VLMs) lack. VLMs built on CLIP-based ViT backbones trade fine-grained detail for high-level semantics, and we show this loss propagates downstream. To recover it, we fuse DINOv3 and CleanDIFT representations into a perception encoder (DiffusedDINO) and align it with RoBERTa-L, yielding a text-aligned model TDDN that preserves this perceptual advantage: with frozen backbones and only $\sim$590K alignment pairs, TDDN matches CLIP on image-text retrieval, surpassing it on three of four settings. It does so while more than tripling CLIP's dense-prediction accuracy (ADE20K 5.20 $\to$ 18.11 mIoU, COCO-Stuff 7.35 $\to$ 24.44), despite CLIP's massive training corpus. TDDN leads on segmentation benchmarks among general-purpose contrastive encoders, including SigLIP$\,$2. We further introduce Puzzle Perception, a segmentation and visual question answering dataset that probes fine-grained spatial understanding, on which TDDN doubles CLIP's segmentation accuracy (11.04 $\to$ 22.51 mIoU).