Llm training improves performance on multi-turn tasks with new weighting method
Turn-level Multiscale Density Ratio Estimation for LLM Agents
Artificial Intelligence
Summary
Large language models (LLMs) can get better at complicated tasks when trained to handle multiple steps or turns of a conversation. The authors found that existing training methods often focus only on single-step tasks and don’t work as well for these longer interactions. They created a new training approach that looks at the importance of each turn differently and pays special attention to where the model performs well or poorly across turns. Their tests show that this method helps LLMs think more clearly and consistently during longer conversations, even when the tasks are unfamiliar.
What this means in practice
- •For software developers: Improve chatbot responses in applications requiring extended conversations and reasoning with tools.
- •For customer support teams: Enhance automated support agents to better handle multi-step troubleshooting interactions with customers, including unfamiliar cases.
Authors
Zishuo Zhao, Kai Chen, Ao Li, Yuan Liu
Abstract
With the rapid development of Large language model (LLM), agent systems enhanced by LLMs show huge potential in being able to deal with complex tasks, especially involving multi-step thinking or interaction with tools. For applying LLM techniques with a well-designed agent paradigm, post-training of LLM in multiple agent scenarios is necessary to achieve better performance. Among the variable post-training techniques, alignment methods such as PPO, DPO, DIL, and GRPO become popular because many papers show a significant positive impact on the model's performance by punishing negative samples while keeping acceptable training complexity. However, most alignment methods address simple single-turn tasks, and there remains room for improvement for complex multi-turn tasks. We propose Turn-level Multiscale Density Ratio Estimation (tlm-DRE), which assigns different weights on corresponding turns and proposes asymmetric token-level training based on the positive-negative space gaps across multiple turns of tasks. The results of the experiment on a wide range of agent benchmarks show that the proposed method performs competitively compared to traditional alignment methods. The proposed training method enables LLMs to perform robustly in multi-turn reasoning tasks with both in-domain and out-of-domain conditions.