Omni Woodpecker reduces errors in audio visual large language models
Omni-Woodpecker: Hallucination Correction via Evidence-Prior Rebalancing for Omni LLMs
Artificial IntelligenceComputation and LanguageComputer Vision and Pattern RecognitionMultimedia
Summary
Large language models that understand both sounds and images sometimes make mistakes by mixing up information from these two sources. The authors propose a new method called Omni-Woodpecker that improves how the model balances real evidence against its guesses inside its brain instead of just fixing mistakes at the output. This method spots conflicts between what it hears and sees, separates true facts from misleading assumptions, and adjusts the internal signals accordingly to reduce errors. Their tests show this approach improves performance on several benchmarks and works well with different model designs.
What this means in practice
- •For multimodal ai developers: Improve multimodal language model reasoning accuracy by reducing audio-visual hallucinations within fused representations during inference.
- •For video analysis teams: Enhance video-based AI systems by correcting conflicting evidence in audio and visual streams to yield more reliable outputs.
Authors
Zijian Liu, Chaoyou Fu, Yan Yang, Caifeng Shan, Ran He
Abstract
Recent Omni LLMs (OLLMs) have demonstrated remarkable capabilities in joint audio-visual reasoning, yet cross-modal hallucinations remain a major challenge. Existing mitigation methods mainly operate at the output level, overlooking the entanglement of grounding evidence and learned priors within fused representations. To address this issue, we propose Omni-Woodpecker (OWP), a training-free framework that shifts hallucination mitigation from output-level manipulation to conflict-conditioned representation rebalancing. Specifically, OWP first identifies cross-modal conflicts from complementary modality-specific views. It then employs orthogonal decomposition to separate grounding evidence from competing priors within the hidden states. Finally, OWP adaptively rebalances fused representations by suppressing misleading priors and restoring ignored evidence before decoding. Extensive experiments demonstrate the effectiveness and strong cross-architecture generalization of OWP. Using Qwen2.5-Omni-7B as the backbone, OWP improves performance by 12.9 and 8.6 points on CMM and AVHBench, respectively, while consistently achieving gains of 0.67-2.25 points across four general OLLM reasoning benchmarks. Furthermore, OWP generalizes effectively to VideoLLaMA2-AV, improving performance by 4.6 and 3.9 points on the same benchmarks. Code is publicly available at https://github.com/ZiiJianLiu/Omni-Woodpecker