Language models learn to balance quick answers with detailed reasoning
CounterRoute: Self-Routed Reasoning via Hierarchical Counterfactual Credit Assignment
Artificial Intelligence
Summary
Some language models often give long explanations even when a simple answer would be enough, which wastes time and computing power. The authors developed CounterRoute, a system that helps these models decide when to think deeply and when to answer directly. It learns to switch between thinking modes automatically by training itself using a special method called reinforcement learning. This approach makes the models more accurate and efficient across many different tasks, using fewer words and less computing time.
What this means in practice
- •For ai model developers: Improve large language model inference efficiency by integrating adaptive routing between direct and reasoning modes to reduce compute while maintaining accuracy.
- •For software engineers building conversational agents: Develop chatbots that decide autonomously when to provide quick answers or detailed explanations, improving response speed and user experience.
- •For data center optimization teams: Reduce computing costs by deploying language models that generate fewer tokens without sacrificing performance on common benchmarks.$Commercial implications: Enables cloud providers and AI service vendors to offer faster, cheaper NLP-based services by optimizing token generation.
Authors
Ruochen Jiao, Besnik Fetahu, Zhenyu Shi, Priyanka Nigam
Abstract
Reasoning-capable language models often produce long chains of thought when direct answers suffice, wasting inference compute. Many dual-mode models leave this choice to users. Automating it is challenging because routing targets evolve with the policy, initial mode preferences destabilize exploration, and sequence-level objectives entangle routing with response learning. We introduce CounterRoute, an online reinforcement-learning framework that jointly learns routing and modeconditioned responses in one shared policy directly from a native dual-mode checkpoint, without method-specific SFT warm-up. Paired current-policy counterfactual rollouts assign cross-mode credit only to the routing token, while within-mode GRPO trains response tokens. A paired-to-self-routed curriculum stabilizes early training with forced rollouts from both modes, then increases self-routed updates to improve autonomous routing. Across nine benchmarks, CounterRoute better balances accuracy and efficiency than heuristic and learned adaptive-routing methods. Relative to always-thinking checkpoints, it improves macro-average accuracy while reducing mean generated tokens by 51% for Qwen3-8B and 41% for Qwen3-14B. On instruction-following and commonsense benchmarks where direct answering is strong, think rates fall as low as 1% while response quality improves. Despite training only on math and instruction following, its routing behavior and response quality generalize to held-out coding, science, knowledge, and commonsense benchmarks.