Papers for

multimodal app developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

D3Tree speeds up verifying draft token blocks during heavy loads

D3Tree: Learning to Prune Block-Diffusion Draft Trees for Efficient Verification

Abstract: Block-diffusion drafters propose a whole block of tokens in one forward pass, and draft trees built from their position-wise marginals increase the expected accepted length of speculative decoding. Every tree node occupies a query row in target verification, so under concurrent serving low-yield branches compete with other requests for verification capacity. Existing tree methods select verified nodes by draft probability under a fixed or cost-selected per-request budget, and thus always retain a prefix of the drafter's ranking. We present D3Tree, which learns which nodes of a block-diffusion draft tree are worth verifying. A tree-size head with 2H+2 parameters predicts each candidate's conditional survival from the draft state, the candidate embedding, and its draft log-probability. Thresholding the product of these scores along parent chains yields an ancestor-closed subtree whose size adapts to each request and decoding round. The head is trained on a frozen drafter with conditional labels on the full candidate tree; an optional joint mode also steers the drafter toward the retained reference path. At serving time, D3Tree verifies whole trees under light load and, under heavy load, allocates one batch-wide node budget by predicted survival, sized to the target's matrix-multiplication tiles. Retained trees share one ancestor-masked target pass, while construction, pruning, and scheduling run on the GPU and overlap with the host. Across six vision-language verifiers and 17 text and multimodal benchmarks, D3Tree is fastest in 70 of 72 load settings, with 3.2-4.0x speedup over autoregressive decoding for one request and 1.3-1.8x at 64 concurrent requests, where fixed-budget trees fall below autoregressive speed. Under a time-varying arrival trace, it has the lowest mean end-to-end latency in every arrival phase.

Sun 8 NovMachine LearningArtificial Intelligence
The gist
Verifying parts of generated text or images can be slow, especially when many requests come in at once. The authors developed D3Tree, which learns to pick only the most promising pieces from a block of suggestions to verify, adapting automatically to how busy the system is. This method makes the process much faster without losing quality, working well across many types of tasks. It helps speed up generating text and images especially when multiple users request outputs simultaneously.
Open → 2611.11371v1

Vision language models struggle to match attributes with objects in images

Stuck on Shuffle: On Attribute-Object Binding in Vision-Language Models

Abstract: Retrieval is a fundamental task in multimodal systems: given an image or text query, the goal is to find the most relevant images or texts in a collection, often using a shared embedding space. Reliable retrieval requires these embeddings to capture not only which concepts are present, but also how they are composed. One key compositional challenge is attribute--object binding: encoding which attributes belong to which objects. Existing benchmarks provide a limited view of this capability, typically covering a subset of modality pairings, simple two-object swaps, or small attribute sets. Consequently, it remains unclear how robustly current models preserve attribute--object bindings across different modality pairings and more complex settings. We therefore introduce SABER, a controlled benchmark for evaluating attribute--object binding across all four image and text modality pairings, spanning diverse attribute types and scenes with up to ten objects. Evaluating models on SABER reveals substantial binding failures in both CLIP-style dual encoders and recent MLLM-based embedders. These findings motivate us to ask how binding can be improved efficiently. While multimodal supervision may be effective, collecting suitable images is difficult and costly. Text, on the other hand, is well suited to binding supervision: it is cheap to generate, supports a broader range of attributes and enables precisely controlled hard negatives. We thus finetune an MLLM-based embedder exclusively on synthetic attribute-binding text. Despite the absence of image supervision, the gains transfer to cross-modal and image--image retrieval while preserving general multimodal embedding performance. Our model achieves SOTA results on SABER while also improving on negation and instance-retrieval benchmarks, demonstrating broader benefits of our binding-focused supervision.

Thu 15 OctComputer Vision and Pattern Recognition
The gist
Finding images or texts that match a description requires understanding not just what things are in the scene, but which details go with which objects. Current models often mix up attributes and objects, like pairing the wrong color with a shape, especially when dealing with complex scenes or different ways of asking. The authors created a new test called SABER to better measure this problem across images and text, finding that many models fail. They improved one model by training it only on specially made text that describes correct attribute-object matches and this helped the model do better even without seeing more images.
Open → 2610.22602v1