Confidence-guided local credit correction improves multi-turn ai training

C3: Rethinking Process Supervision via Confidence-Guided Credit Correction in Multi-Turn Reinforcement Learning

Artificial Intelligence

Summary

Sometimes when teaching AI to have multi-step conversations or tasks, the AI gets rewarded even when it behaves badly in small ways during the process. The authors propose a method called Confidence-Guided Credit Correction, or C3, which helps identify and fix these small problem behaviors without changing the overall reward for success. This method adjusts the learning based on how confident the AI policy is in the problematic behaviors and improves performance on multiple tasks. They show it works better than just giving turn-level or overall rewards for behavior.

What this means in practice

  • For ai developers: Enhance multi-turn conversational AI training by correcting token-level behavior credit to reduce undesirable local actions without changing overall task rewards.
  • For chatbot platform engineers: Improve reinforcement learning workflows for chatbots by integrating local credit corrections guided by policy confidence to boost task success rates.

Authors

Ai Jian, Haoran Pan, Eryu Guo, Haojin Yang, Jinghan He, Xiaoyun Zhang, Jiangbo Pei, Peiyu Liu

Abstract

Large language models are increasingly trained as multi-turn agents that interact with external tools. However, outcome-only reinforcement learning can produce a form of misguided reinforcement: even on successful trajectories, undesirable local behaviors may receive positive token-level advantages and be favored by the policy update. To address this issue, we propose Confidence-Guided Credit Correction (C$^3$), a plug-in method for local credit correction. C$^3$ identifies spans corresponding to undesirable behaviors and selectively corrects positive credit after advantage estimation, with the correction strength adaptively controlled by the behavior policy's confidence in the corresponding behavior, while leaving the task reward and original credit estimation unchanged. Across multiple multi-turn benchmarks, C$^3$ improves performance under both {critic-based and critic-free reinforcement learning methods}. Injecting the same behavioral signals as turn-level or outcome rewards does not consistently improve performance. Our results show that local credit correction provides an effective form of process supervision for outcome-driven RL.