Orthogonal drift correction improves text to image prompt matching

ODC: Orthogonal Drift Correction for Improved Text-to-Image Semantic Alignment at Inference

Computer Vision and Pattern RecognitionComputation and LanguageMachine Learning

Summary

Text-to-image models sometimes create pictures that don’t exactly match the details in the description given. The authors developed a method called Orthogonal Drift Correction (ODC) that fixes this problem without changing the model itself or asking for more input. ODC looks at the difference between the original picture and the description, figures out what part is irrelevant, and adjusts the text instructions accordingly. This helps the model make pictures that better match complex or detailed text prompts.

What this means in practice

  • For digital artists: Generate images that more accurately follow complex text descriptions without retraining models or revising prompts manually.
  • For game developers: Improve automatic creation of visual content from textual scene descriptions to better match detailed creative requirements.

Authors

Kasra Kamali, Basim Azam, Tuan Duong Trinh, Jean Honorio, Naveed Akhtar

Abstract

Text-to-image models have achieved remarkable success in generating high-quality images from textual descriptions. However, they often struggle with "semantic drift," where the generated output fails to precisely align with complex or nuanced text prompts. While recent approaches have attempted to address semantic errors regarding attribute binding or object presence, there remains a gap for a more holistic method that addresses these issues by directly refining the text embeddings of the initial user prompt. In this work, we introduce Orthogonal Drift Correction (ODC), an inference-time guidance method designed to mitigate semantic drift without requiring model retraining or additional user inputs. ODC guides image generation through a two-stage process. In the first stage, it identifies the semantic drift by evaluating the initially generated image against the user prompt in a shared vision-language embedding space. It then isolates the component of this drift vector that is orthogonal to the prompt's direction and translates it back into text via a vocabulary-based surrogate mechanism. In the second stage, it produces refined text conditioning for a second generation pass by feeding both the initial text embedding and the re-embedded drift representation into an adaptive rank-reduced concept removal module. Our experiments demonstrate that ODC consistently improves prompt-image alignment, increasing CLIP scores by up to 5.4% over the baseline on the COALBench-300 and PartiPrompts benchmarks. Furthermore, on the T2I-CompBench complex subset, ODC improves the average 3-in-1 score by 4.5% over the competing methods. As a plug-and-play module, ODC offers a practical and efficient method for improving the reliability of text-to-image models. Code is available at https://github.com/kasrakamali/ODC.