TDDN improves detailed image understanding and puzzle solving with better alignment
TDDN: Text-aligned Diffused DINO Network for Puzzle Understanding
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Current AI systems that connect images and text often miss small details in pictures, which is a problem when solving puzzles or other tasks needing careful visual attention. The authors introduce a new model called TDDN that combines two advanced image features and matches them with language features to keep these details. TDDN performs as well as or better than older models at finding matching images and text, while greatly improving detailed image segmentation. They also created a new puzzle-focused dataset where TDDN shows twice the accuracy of a popular model in understanding complex images.
Vision Language ModelsCLIPDINOv3Image segmentationVisual reasoningContrastive learningPuzzle perceptionRoBERTamIoUVisual question answering
Authors
Harsha Patnala, Debopriyo Banerjee, Ayush Sunil Munot, Somak Aditya
Abstract
Structured visual reasoning, such as image puzzles, demands fine-grained visual perception, an ability current Vision Language Models (VLMs) lack. VLMs built on CLIP-based ViT backbones trade fine-grained detail for high-level semantics, and we show this loss propagates downstream. To recover it, we fuse DINOv3 and CleanDIFT representations into a perception encoder (DiffusedDINO) and align it with RoBERTa-L, yielding a text-aligned model TDDN that preserves this perceptual advantage: with frozen backbones and only $\sim$590K alignment pairs, TDDN matches CLIP on image-text retrieval, surpassing it on three of four settings. It does so while more than tripling CLIP's dense-prediction accuracy (ADE20K 5.20 $\to$ 18.11 mIoU, COCO-Stuff 7.35 $\to$ 24.44), despite CLIP's massive training corpus. TDDN leads on segmentation benchmarks among general-purpose contrastive encoders, including SigLIP$\,$2. We further introduce Puzzle Perception, a segmentation and visual question answering dataset that probes fine-grained spatial understanding, on which TDDN doubles CLIP's segmentation accuracy (11.04 $\to$ 22.51 mIoU).