Referring audio visual segmentation avoids overthinking with adaptive reasoning budgets

To Think or Not to Think: Pre-Decisional Reasoning Budgets for Referring Audio-Visual Segmentation

Multimedia

Summary

Too much thought can actually confuse computer systems when they try to find objects in videos by listening and looking, especially if the questions are simple. The authors found that systems can make mistakes when they think too long, so they built a way to predict when extra thinking will help or hurt. By checking a quick early guess, their method decides if more reasoning is needed and saves time and effort without losing much accuracy. This helps computers work faster and smarter by skipping unnecessary steps.

What this means in practice

  • For multimedia app developers: Create apps that identify and segment objects in videos with sound by deciding when to use short or long reasoning, improving efficiency without losing accuracy.
  • For robotics engineers: Improve robot perception systems by adaptively adjusting the length of audio-visual reasoning processes to avoid confusion on simple queries and save processing time.

Authors

Haojie Hu, Senda Chen, Ying Shen, Lin Zhang

Abstract

Multimodal reasoning systems increasingly assume that longer chain-of-thought (CoT) uniformly improves downstream grounding. We show that this assumption fails in referring audio-visual segmentation (Ref-AVS): simple queries can be harmed by excessive reasoning, creating an "overthinking trap" associated with phrase and localization drift, while genuinely ambiguous queries still benefit from multi-step reasoning. To systematically study this, we construct counterfactual reasoning-budget labels under a five-fold out-of-fold evaluation protocol. Our results recast adaptive Ref-AVS reasoning as a pre-decisional representation problem: before the first generated token, the generation-onset state contains linearly decodable information predictive of whether additional reasoning will improve segmentation. A linear probe predicts binary reasoning need with 70.1% accuracy, substantially outperforming text-only features and uncertainty-based proxy signals. As a router, the onset-state controller improves over downstream confidence routing and closely matches Short-then-Decide while using far fewer tokens. Proxy comparisons show that the onset representation contains predictive information not fully recovered by the evaluated hand-crafted features. Reusing this onset state for lightweight, single forward-pass routing reduces autoregressive token consumption by 60% while retaining about 96% of always-long performance. It further provides a cost-quality trade-off on $R^2$-AVSBench without target-label tuning, rather than an accuracy improvement over always-long. Error analysis links harmful long reasoning to phrase and localization drift, illustrating the benefit of bypassing unnecessary reasoning on simple queries. Ultimately, our results support efficient adaptive reasoning in Ref-AVS through pre-decisional internal representations.